Punch Card Patterns Designed with GAN

Abstract

Knitting punch cards codify different stitch patterns into binary patterns, telling the machine when to change color or to generate different stitch types. This research utilizes Neural Networks (NN) and image-based Generative Adversarial Networks (GAN), with an image database of knitting punch cards, to generate new punch card designs. The hypothesis is that artificial intelligence will learn the basic underlying structures of the punch cards and the pattern makeup that is inherent across patterns of different styles and cultures. Different neural networks were utilized throughout the research, such as Neural Style Transfer (NST), AdaIN Style Transfers, and StyleGAN2. The results from these explorations offer different insights into pattern design and various outcomes of the different neural networks. Ultimately physically testing these punch card designs, these patterns were knit on a domestic knitting machine, resulting in novel fabrication and design techniques that are both digital and craft-based.

1 Introduction

Visual patterns are all around us in nature, mathematics, and textiles. These patterns are made of repetitive shapes and geometries. Patterns have often been associated with textiles specifically, as many pattern designs emerged based on the structure of weaving and knitting (Stewart 2015). Knitting uses a single yarn looped around itself in rows to create a textile. Multiple colored yarns are knit together to make decorative patterns. Similar to the Jacquard loom for weaving, knitting machines use punch cards as a basic binary pattern telling the machine to knit either color “A” or “B”. Most domestic punchcard knitting machines come with a set of standard punch cards, and more punch cards can be purchased separately. Images of these punch cards are easily found on the internet, supplying an available data set for this research. See Fig. 1.

Punch cards were initially invented for creating complex weaving patterns on the jacquard-loom. In developing early computers that utilize binary code, Charles Babbage and Ada Lovelace adapted these punch cards to run their Analytical Engine (Essinger 2015). This intertwined history of punch cards, computation, and textiles creates a deep theoretical and foundation for the continued research into technology and textile design.

Fig. 1.

Sample portion of punch card data set.

Using artificial intelligence to design new punch card knitting patterns combines these old and new textile creation methods. It looks at the heritage of computing as well as exploring the future of design with AI. Building upon the use of high-tech design tools applied to low-tech fabrication and crafted methods.

AI is modeled after the human brain and is successful at understanding patterns in data sets. This research explores a data set is of knitting punch card images. This research hypothesizes the generate new knitting designs using neural networks trained on a punch card database. These new patterns will be representative of a variety of styles, cultures, and histories. The success of these patterns will be tested on their viability to generate successful knits. As the patterns could exist as only virtual punch card images, the actual test is the physical constraints of the intended materials. Several different AI techniques were tested during the research, including Neural Style Transfer, AdaIN style transfer, and styleGAN2 training. The results are images of new “fake” punch cards. These were then translated into physical punch cards that could be used to fabricate physical test samples. The results begin to reflect on what can be learned from the knitting patterns designed with AI. Underlying structures of patterns emerge based on the input dataset. The human designer also curates the data set, which biased towards creating a productive knit pattern. Creating and intertwining our history and future of patterns and the techniques used to produce them. Ultimately, the importance of textiles in computation goes beyond the textile community as these results are beginning to discuss a larger question of design computation, pattern language, ornamentation, craft, and fabrication.

2 Context

Patterns are repetitive, symmetric, geometric, and balanced; the human brain is attracted to them. Gestalt theory illustrates the principles such as the laws of symmetry, figure-ground, similarity, and common fate as ways to describe how our minds begin to understand patterns as a whole before they recognize the specific elements (Koffka 2013). Psychologists are still studying the ways that our minds process these patterns. Since neural networks are modeled after how the human brain learns, artificial intelligence predictably should beable to understand the specific rhythms, symmetries, geometries, and spacing that make these knitting patterns.

Fig. 2.

Sample of fair isle knit pattern and corresponding punch card.

2.1 Patterns in Knitting

Punch card knitting patterns interoperate with a traditional knitting technique called Fair Isle. Its origins are credited to Scotland's Fair Isles, as it is a popular knitting pattern technique in that region. Fair Isle patterns are recognizable by their basic geometric shapes, small-scale repetition, mirroring, and simple color changes that never consist of more than two colors per row. See Fig. 2. While one color is used as activate stitches, the other colored yarn hangs or floats in the back. Floats should be no longer than three to five stitches in a successful Fair Isle pattern (Pulliam 2004). The switching between active and inactive yarn colors creates a pattern through pixel-like imagery as each stitch acts as a pixel of color across the textile design. The use of only two yarns makes this technique ideal for binary codification into punch cards. In contrast to domestic punch card knitting machines, CNC machines and hand knitting are capable of more complex patterns that include more than two colors or multiple stitch types. Therefore, this use of punch cards is not entirely low-tech or high-tech, as other methods are available. In this case, the use of knitting punch cards was due to the available data set, the historical and theoretical contexts, the simplicity of binary coding, and easy access to domestic knitting machines.

3 Computational Textile Design

Pixels, punch cards, and binary all sound like computer terms, yet they are all used to design and produce textiles. This is the intersection between computation and textile fabrication and design. As computational design methods have developed over the years, so have computational design techniques for textiles. There are many examples of computation used to weave, knit, and design patterns. Although there are yet few examples addressing Fair Isle knit punch card patterns, which creates an opportunity for this research exploration to fit within the context of computational textiles.

3.1 Precedent Examples of Computational Textile Design

Designers and artists have worked with algorithms and AI for knitting, sewing, and embroidery. These examples show some of the development and precedents for computational textile design. Genetic algorithms were a precursor to contemporary forms of AI, employing a metaheuristic approach to learning. Genetic algorithms were used in the research of lacemaking pattern design. These algorithms could learn the rule sets to create a knittable lace pattern. At each generation of the designs, the algorithm learned through a supervised training method what choices to make in the design to produce a knittable lace design (Ekart 2007).

Neural Networks have been used for various textile designs. Such as for the color selection and pattern design for new embroidery samplers. In this precedent research, a sentence was input into an Entertainment AI that adapted the sentence's content into the color selection and motifs for an embroidery sampler design (Smith 2017). Furthermore, knitting has been explored with generative AI design. Through the development of using a neural network to generate CNC knitting machine patterns. This example generates knitting patterns from images of unknown knit material by being trained on the structure of several sample knits and their corresponding patterns, resulting in a user-friendly interface called img2prog (Kaspar 2019). Hand knitting patterns are written out in a shorthand language, referred to as knit-speak. A natural language learning AI was trained on 500 patterns in the Sky-Knit project to develop new knit-speak directions for hand knitting patterns. Using the online community of Raverlry.com, these patterns were physically knit by artisans and artisans; the resultant designs were ultimately very strange looking (Shane 2019).

These examples show the development of computation within textiles and design. Eventually, creating actual physical manifestations of the crafted knitwork is essential. Currently, so many of AI designs happen and remain within the computer. Working with textiles allows the design from the computer to easily be fabricated physically and test the constraints of those designs.

3.2 AI Designed Punch Cards

Domestic knitting machines were a popular craft between the 1940s until the 1980s. Unfortunately, as a hobby, machine knitting has since decreased in popularity. Resultantly, many of the companies that sold knitting machines and punch cards no longer produce them. Fair Isle knit pattern punch cards have an almost 1:1 relationship with the image of the punch card. Making this an easy starting point to design AI knit patterns as one can visually see the potential design in the punch card results. Standard punch cards are 24 dots or stitches wide and 60 stitches long. They can be looped in the vertical direction, and patterns are repeated in the horizontal direction to create larger knit fabric pieces.

There are three main types of Fair Isle patterns; geometric patterns, organic floral patterns, and object-based or images. The database of knitting punch card images was generated by image scraping from Google. Eventually, these images were sorted manually to affirm the best quality of images for training. See Fig. 1. This sorting could generate bias, but also allowed a way to collect only clear legible punch card images, which would work best for pattern learning. Google images may also provide bias as the search results from what is available on the internet; perhaps specific patterns may appear more frequently than others due to trends and popularity. These types of bias could be considered as positive weighing of designs as it creates a way for some clarity and influences to be embedded into the data. With this in mind, to begin the research, different deep learning methods were used to design sample punch cards and ultimately to fabricate them into knit designs.

3.3 Neural Style Punch Card Designs

Neural Style Transfer (NST) was developed in 2015 (Gatys 2015). It utilizes a Convolutional Neural Network (CNN) to understand the underlying structure of an image separate from an image’s style. This network uses only two images to generate a new image. NST has been used to generate AI art, where one style of a well-known artist is applied to another image. In the case of knitting patterns, the goal was to apply the style of one knitting punch card pattern to the underlying structure of another pattern. The prediction is that the results might find a median between the two punch cards and express elements of both designs at once.

In the tests, punch cards with different pattern styles were used to see how the organic forms and geometric patterns would combine. When running the neural network, the number of iterations was adjusted to test the quality. A more refined image occurred with a higher number of iterations. While a lower number of iterations resulted in undefined dots and small dots that did not fit within the punch card grid. See Fig. 3. Secondly, adjusting the weight of the images allowed for more or control over the influence of style over structure. Several versions of the style transfer were run, with different knitting patterns. In this process, the human and machine collaboration is clear from adjusting the settings to get desirable results to the section of the images to use for style and structure inputs. The results achieved were based on what the designer felt was most successful at creating a legible punch card and displaying features from both inputs. This is important because the machine generates designs, but ultimately, the human and machine collaborate to achieve the best output. The final designs resulted in a structure from the content punch card that distorted and manipulated the patterns without any understanding of cultural significance or meaning behind the pattern. See Fig. 3.

Fig. 3.

Example result of neural style transfer.

3.4 AdaIN Style Transfer Punch Card Designs

Developed in 2017 as a faster alternative to NST, Adaptive Instance Normalization (AdaIN) uses a single feed-forward neural network to produce similar results (Huang 2017).

This network was used to generate another set of knitting punch card patterns. AdaIN only uses two image inputs for a style image and a content image. Although running much faster than NST, the results were less effective as the dots were more distorted and did not arrange to the original punch card grid structure. Moreover, there are fewer settings and controls to manipulate the results of AdaIN Style Transfers. The resulting images had an underlying grey shade from the input pattern’s geometry rather than a discrete dot matrix. See Fig. 4. Thus adding additional human manipulation necessary post AI production to interoperate this design into a useable punch card design. Thus, the grey shade is negated, but there are minor deviations and subtle shifts to this pattern from the style image. See Fig. 4.

Fig. 4.

Example results from AdaIN

3.5 StyleGAN2 Designed Punch Cards

StyleGAN2 was released in early 2020 by NVIDIA; it is an update to the earlier version of StyleGAN developed in 2018. Generative Adversarial Networks (GAN) consist of two neural networks, one to generate images and one to test the images (Karras 2020). StyleGAN2 learns the characteristic artifacts in a data set of images to produce new images. The GAN first generates images from a random noise pattern; the discriminator tests them and feeds back information to the generator to correct. Each epoch, the generator gets closer to the desired results until eventually, the generated images can fool the discriminator into believing that the image is real.

The results of image scraping from google was manually sorted into small data set of clear resolution punch cards of about 120 images. These were primarily curated on their legibility and not on the pattern content. This data set included various patterns, from floral, geometric and object-based designs such as cats and owls. The hypothesis did not want to depict one style or type of punch card in the first tests. However, to test the ability of the network to understand the underlying structure of the punch cards, such as the constraints in Fair Isle knitting to consist of floats no more than 3–5 stitches. Simultaneously, other design constraints, such as not having too many substantial areas of one color for aesthetic reasons. The pattern should also be repeatable; it would need to have balance across the card rather than be weighted to one side or the other. Furthermore, the knit is constructed in rows, each row can exist independently, but a successful pattern has vertical and horizontal repetition and geometry.

In order to generate a more extensive set of data from the few quality images collected, the punch cards were broken down into smaller sections. Typical punch cards are 24 dots wide and about 60 dots long; the 24 dot width defines the pattern in the machine. Meanwhile, the various lengths of images would cause issues in training; thus, cutting them into equal-sized images would produce a more controlled data set. Each image was cropped into several smaller square-proportioned images, resultantly about 24 dots by 24 dots. These images consisted of overlaps between them. Mirroring was also used, as well as cropping the images since punch cards do not necessarily have a front or back and can be fed into the knitting machine facing any direction. These approaches to manipulating the images multiplied the data set ten times, achieving a final set of close to 1200 punch card images for training. Since the research was focused on the knitting patterns' underlying structures, it was not essential to need the overall pattern and emphasis on the more minor relationships of spacing and localized dot matrix. The 24 dots width was kept consistent in the data set so that the results could then be recombined to create more extended patterns more similar to their original proportions.

Fig. 5.

Results from StyleGAN

The data set was uploaded to a base model of StyleGAN2, trained initially on bird illustrations, for the training. At around 1500 epochs, the images started to begin to look like new punch card designs. After that point, the model seemed to face mode collapse, where the punch cards generated began to look all self-similar, and the individual dot matrix was to be lost. This is most likely due to the original data set being too small and self-similar as well. Further investigation of this could be explored. Although at 1500 epochs, a set of 50 successful sample image results was downloaded. The human user judges the success of these images based on their appearance to look like a punch card and no longer have any reference to the original training set of birds and unique enough to have diverse results. Since these images were in the square format of the input images, three images close in aesthetic quality were selected and combined vertically to generate a more common proportioned punch card pattern. The images created varied widely, from ones with a high density of dots to relatively sparse ones. Some patterns seemed very random, while others had clear underlying diagonal patterns embedded within them. Although these appeared random and stochastic at a glance, some underlying structure was revealed upon further reading and inspection. Some patterns did emerge, such as checkered patterns and diagonal striping, and underlying vertical designs. These smaller repeated structures can be seen in many of the input patterns from the data set. See Fig. 5.

3.6 Physical Results

After the designs were digitally generated, physical punch cards were made. The image results from each method were not clear enough to directly use as a punch card and needed to be processed. Grasshopper for Rhino was used to trace the large, clear dots from the images into vector linework, which was then organized on the grid structure by moving these circles to the closest grid points. See Fig. 6. The patterns were laser cut out of thick Mylar to make them usable punch cards. They were then used to knit on a Brother KH836 Domestic Punch Card knitting machine with a standard 4.5 mm gauge. Since the punch card pattern is only 24 stitches wide, this would result in a small pattern of only four inches wide. Therefore, the pattern was set up to repeat once in width. This created an eight-inch by eight-inch test swatch, allowing it to knit once vertically through the pattern.

One of each of the designs were tested using two different colors of yarns. To visually and texturally make the pattern apparent. While physically knitting the patterns, a better understanding was developed of their successes and failures for generating Fair Isle knitting punch cards, as they could be tested with material constraints of the different yarn types as well.

3.7 Neural Style Transfer Knit

The NST pattern tested consisted of an evident pattern with large swatches of each color. When knitted, the pattern had a noticeable vertical structure with some variation that seemed organic. The pattern appeared very intriguing and exciting as a punch card design was less successful when knit. The spacing and gaps horizontally resulted in large floats, which are undesirable in Fair Isle knits. This is undesirable because it leaves extra yarn hanging on the back, weakening the structure and snags on things. Mainly this occurred when the pattern is knitted doublewide; therefore, perhaps the NN could not understand how the pattern would be used to repeat and did not consider this edge condition. The structure overall is recognizably vertical. Vertical patterns are successful because they do not usually create long floats, but perhaps the vertical design proportions of this resulting design were too large.

On the other hand, there is evident mirroring of the design, which is a typical feature in Fair Isle designs. Finally, when repeated, the relationship of the repeat is successful as there is not a clear seam line. The pattern has evident balance from left to right that there is no noticeable start or end to the pattern. See Fig. 7.

Fig. 6.

Translation of AdaIN training to knit results

3.8 AdaIN Style Transfer Knit

The AdaIN style. This design ultimately resulted in a more random pattern than the NST. The knitting was more successful as the structure was at a smaller scale, and there were no long lengths of floats. In addition, there was still an underlying diagonal and argyle type style despite that it was disrupted by some non-repetitive structure as well. When knit, the pattern was also repeated, and it had some shifts in the repeat, making the pattern not completely seamless yet was not a very noticeable edge to the pathen. See Fig. 7.

Fig. 7.

Results of NST, AdaIN, and StyleGAN2

3.9 StyleGAN2 Knit

The StyleGAN2 training resulted in various outputs, consisting of very dense dots to very sparse dots. The sparse dot patterns were going to an issue, as there were some rows with only one or two changes in color, resulting in very undesirable long floats.

The denser StyleGAN2 generated patterns were more successful punch cards as they had adequate spacing for short floats, all under six stitches. They also had a noticeable clear structure of diagonal pattern disrupted by some random stitches. This is possibly because the data set had a lot of diagonal patterns in it. However, the structure became less apparent when knitted because the diagonals were only a single stitch wide. As well as when knit, the pattern is squished, and the results are not as long as the input punch card. The yarn structure with this pattern resulted in making it appear so small that it was no longer clear diagonals but more of an overall descript dot hatch or fill between the two colors. In addition, because the structure was invisible when knit, it is difficult to tell that the pattern is repeated more than once horizontally, which can be quite successful. The pattern ultimately has a particular movement to it as it resonates between the different diagonals and checkered designs.

4 Conclusion

Each of the resulting tests developed unique patterns that never existed before. The results varied in success, yet there were clear underlying structures that each training method could understand and replicate. The training on the different networks proved always to make an easily repeatable pattern. The neural networks learned the patterns underlying structures, which results in noticeable styles from the existing dataset. These underlying structures worked to create visual appeal essential to the knit material's tectonics and the principles of patterns defined in gestalt theory. Ultimately, each of the neural networks had different positives and negatives.

The human and machine collaborate back and forth through the collection of data images and the curation of these images. The human is also controlling the weights of this input data and the number of epochs that the networks run. Ultimately, the knits are produced using domestic craft techniques and result in unique fabrication methods that integrate high-tech and low-tech processes.

For the development of this research, there are still opportunities to have more control over the data, such as inputting only geometric patterns or testing the knits as tuck or lace patterns rather than Fair Isle. These patterns also have further potential in how they can be utilized in fashion, décor, or architecture. These new patterns ultimately combine the structures and mish-mash the cultural meanings behind these patterns into something new and designed computationally. This work is reflecting and connecting our technological past and the design potentials of contemporary AI and technology.