PyTorch (and PyTorch Lightning) implementation of Neural Style Transfer, Pix2Pix, CycleGAN, and Deep Dream!
PyTorch (and PyTorch Lightning) implementation of Neural Style Transfer, Pix2Pix, CycleGAN, and Deep Dream!
A walkthrough of three models that paint, translate, and reimagine images — Neural Style Transfer, Pix2Pix, and CycleGAN.
Creativity is something we closely associate with what it means to be human. But with digital technology now enabling machines to recognize, learn from, and respond to humans, an inevitable question follows: Can machines be creative?
It could be argued that the ability of machines to learn what things look like, and then make convincing new examples marks the advent of creative AI. This tutorial will cover three different Deep Learning models to create novel arts, solely by code — Neural Style Transfer, Pix2Pix, and CycleGAN. They build on each other in a natural way: NST optimizes a single image against a pretrained network without any training of its own, Pix2Pix learns a paired mapping with a conditional GAN, and CycleGAN drops the paired-data requirement entirely.
Style Transfer is one of the most fun techniques in Deep learning. It combines two images, namely, a Content image (C) and a Style image (S), to create an Output image (G). The Output image has the content of image C painted in the style of image S.
Style Transfer uses a pre-trained Convolutional Neural Network to get the content and style representations of the image, but why do these intermediate outputs within the pre-trained image classification network allow us to define style and content representations?
A network trained on image classification has learned to convert raw pixels into a progressively richer internal representation of what's in the image. The activation maps of the first few layers represent low-level features like edges and textures; as we go deeper through the network, the activation maps represent higher-level features — objects like wheels, or eyes, or faces. Style Transfer incorporates three different kinds of losses:
Putting it all together:
$$J_{\text{Total}}(G) = \alpha \cdot J_{\text{Content}}(C, G) + \beta \cdot J_{\text{Style}}(S, G) + \gamma \cdot J_{\text{TV}}(G)$$
Let's delve deeper to know more profoundly what's going on under the hood!
Usually, each layer in the network defines a non-linear filter bank whose complexity increases with the position of the layer in the network. Content loss tries to make sure that the Output image G has similar content as the Input image C, by minimizing the L2 distance between their activation maps.
Practically, we get the most visually pleasing results if we choose a layer in the middle of the network — neither too shallow nor too deep. The higher layers in the network capture the high-level content in terms of objects and their arrangement in the input image but do not constrain the exact pixel values of the reconstruction very much. In contrast, reconstructions from the lower layers simply reproduce the exact pixel values of the original image.
Let $a(C)$ be the hidden layer activations which is a $N_h \times N_w \times N_c$ dimensional tensor, and let $a(G)$ be the corresponding hidden layer activations of the Output image. Finally, the Content Cost function is defined as follows:
$N_h$, $N_w$, $N_c$ are the height, width, and the number of channels of the hidden layer chosen. To compute the cost $J_{\text{Content}}(C, G)$, it might also be convenient to unroll these 3D volumes into a 2D matrix, as shown below.
The first image is the original one, while the remaining ones are the reconstructions when layers Conv_1_2, Conv_2_2, Conv_3_2, Conv_4_2, and Conv_5_2 (left to right and top to bottom) are chosen in the Content loss.
To understand it better, we first need to know something about the Gram Matrix. In linear algebra, the Gram matrix $G$ of a set of vectors $(v_1, \ldots, v_n)$ is the matrix of dot products, whose entries are $G(i, j) = v_i^\top v_j$. In other words, $G(i, j)$ compares how similar $v_i$ is to $v_j$. If they are highly similar, the outcome would be a large value, otherwise it would be low, suggesting a lower correlation. In Style Transfer, we can compute the Gram matrix by multiplying the unrolled filter matrix with its transpose as shown below:
The result is a matrix of dimension $(n_C, n_C)$ where $n_C$ is the number of filters. The value $G(i, j)$ measures how similar the activations of filter $i$ are to the activations of filter $j$. One important part of the gram matrix is that the diagonal elements such as $G(i, i)$ measure how active filter $i$ is. For example, suppose filter $i$ is detecting vertical textures in the image, then $G(i, i)$ measures how common vertical textures are in the image as a whole.
By capturing the prevalence of different types of features $G(i, i)$, as well as how much different features occur together $G(i, j)$, the Gram matrix $G$ measures the Style of an image. Once we have the Gram matrix, we minimize the L2 distance between the Gram matrix of the Style image S and the Output image G. Usually, we take more than one layer into account to calculate the Style cost as opposed to Content cost (which only requires one layer), and the reason for doing so is discussed later on in the post. For a single hidden layer, the corresponding style cost is defined as:
It acts like a regularizer that encourages spatial smoothness in the generated image (G). This was not used in the original paper proposed by Gatys et al., but it sometimes improves the results. For a 2D signal (or image), it is defined as follows:
What happens if we zero out the coefficients of the Content and TV loss, and consider only a single layer to compute the Style cost?
As many of you might have guessed, the optimization algorithm will now only minimize the Style cost. So, for a given Style image, we will see the different kinds of brush-strokes (depending on the layer used) that the model will try to enforce in the final generated image (G). Remember, we started with a single layer in the Style cost, so running the experiments for different layers would give different kinds of brush-strokes. Suppose the style image is the famous The Great Wave off Kanagawa shown below:
The brush-strokes that we get after running the experiment, taking different layers one at a time, are attached below.
These are brush-strokes that the model learned when layers Conv_2_2, Conv_3_1, Conv_3_2, Conv_3_3, Conv_4_1, Conv_4_3, Conv_4_4, Conv_5_1, and Conv_5_4 (left to right and top to bottom) were used one at a time in the Style cost.
The reason behind running this experiment was that the authors of the original paper gave equal weightage to the styles learned by different layers while calculating the Total Style Cost. Now, that's not intuitive at all after looking at these images, because we can see that styles learned by the shallower layers are much more aesthetically pleasing compared to what deeper layers learned. So, we would like to assign a lower weight to the deeper layers and higher to the shallower ones (exponentially decreasing the weightage could be one way).
The original Gatys et al. approach was slow — each new output image required hundreds of optimization steps against a frozen VGG. The follow-up literature has largely been about making it faster and then making it work for arbitrary styles at test time. Two papers capture that arc cleanly:
No code or results for either of these in this post — the writeups below are just for reference.
The key move: instead of optimizing one image at a time, train a separate feed-forward network to do the stylization in a single pass. The training objective is the same perceptual loss as Gatys et al. — feature reconstruction against a pretrained VGG (content) plus Gram-matrix matching across multiple VGG layers (style) — but the loss now serves as a target for a network being trained on a large image dataset, rather than as an objective for per-image optimization.
At test time, stylizing a new image is just one forward pass through the trained transformation network — orders of magnitude faster than Gatys et al. The trade-off is that you now have to train a separate network for every style: the model is locked to whatever style image was used during training. Quality is broadly comparable to optimization-based NST, slightly worse in some cases, but in exchange you get real-time inference.
This paper drops the "one network per style" constraint entirely. The key observation: Instance Normalization (which had been observed to work better than batch norm for style transfer networks) can be reinterpreted as performing a kind of style normalization — it removes per-instance style information by normalizing the mean and variance of feature maps. So if you make the normalization adaptive — feeding in the desired mean and variance from a style image rather than learning them — you can transfer arbitrary styles at test time.
Concretely, AdaIN is defined as:
$$\text{AdaIN}(x, y) = \sigma(y) \cdot \left( \frac{x - \mu(x)}{\sigma(x)} \right) + \mu(y)$$
where $x$ is the content feature map, $y$ is the style feature map, and $\mu$ and $\sigma$ are channel-wise mean and standard deviation computed across spatial locations. Aligning the channel-wise mean and variance of $x$ to those of $y$ transfers the style.
The architecture is a VGG encoder (fixed, pretrained on ImageNet) plus a learned decoder that roughly mirrors the encoder. Content and style images both get encoded; AdaIN is applied at an intermediate VGG layer; the result is decoded back to image space. The decoder is trained with a content loss (against the AdaIN output) and a style loss (matching mean and standard deviation of decoder activations to those of the style image at multiple VGG layers).
The result is what most modern "real-time arbitrary style transfer" demos use: one trained model, any style image at test time, a single forward pass.
Image-to-image translation is the task of taking one representation of a scene — an edge map, a semantic label map, or a daytime photograph — and producing another representation of the same scene: a sketch turned into a colored photo, a label map turned into a street view, a day
No open issues yet, or sync has not completed.