Generate Anime using CartoonGAN and TensorFlow 2.0

What images would you choose to transform if you could turn any of them into anime using AI?

In this post, I will briefly introduce a CartoonGAN implemented in TensorFlow 2.0 Alpha by my friend mnicnc404 and me.

I will also demo a TensorFlow.js app which allows you to generate anime on your browser directly.

CartoonGAN (original paper) was published in 2018 CVPR and is a Generative Adversarial Network (GAN) which attempts to transform real world images into cartoon-style images.

The ideas demonstrated in the paper are very interesting and the authors also show good results using two of my favorite Japanese animator styles: Shinkai Makoto and Miyazaki Hayao.

Therefore, I decided to build a TensorFlow.js app to allow anyone to try CartoonGAN with ease.

Choose your favorite anime style and upload one image, it's just that simple.

Under the hood, TensorFlow.js runs in your browser whilst:

• downloading the pretrained models
• cartoonizing uploaded images

To get the best performance, it is highly recommended that you:

• try the app on a desktop/laptop rather than mobile devices (to speed up transformation)
• try the app when network speed is fast（to minimalize loading time）

Try CartoonGAN yourself¶

Without further ado, here is the tfjs app:

How does your anime look?

If the app seems to be stuck:

• Loading Models: simply means it's still downloading the model
• Cartoonizing images: it might indicate there is no sufficient computation power on your device

You can still generate anime in the next section using TensorFlow if tfjs doesn't work for you.

Generate anime using TensorFlow 2¶

In addition to TensorFlow.js, we also implemented CartoonGAN using TensorFlow 2.0 Alpha. If you want to transform larger images and/or gifs, you can run this Colab notebook:

Google Colaboratory is a cloud Jupyter notebook environment allowing anyone to start their machine learning projects with free GPU available.

In this notebook, everything you need is set up for you:

1. Build TensorFlow 2.0 environment
2. Clone our github repo and download pretrained models
3. Download arbitrary images on the web
4. Transform the images using CartoonGAN

You get the idea. All you have to do is open the notebook and generate your anime. It's that simple.

Gallery: some anime we generated¶

Is better to share one's happiness than to enjoy it alone. In this section, we simply share some anime generated with CartoonGAN by ourselves.

For easy comparison, every image below is divided into 4 parts where:

• top-left: original real world image
• top-right: Shinkai Makoto style
• bottom-left: Miyazaki Hayao style
• bottom-right: Hosoda Mamoru style

Click the arrows on the left/right to view different results:

Marvel movies, cute cats, scenery or pop idols, the sky is the limit. When trained, we can use CartoonGAN to combine everything with anime.

Using Python script in our github repo, you can generate these results with just one command:

python cartoonize.py \
    --styles shinkai hayao hosoda

You can imagine apps with similar features will become much more powerful in the near future. Everyone will be able to generate their own anime using various styles, while animators will be able to test their new ideas and draw drafts much more faster than ever before.

Generative models like CartoonGAN can show us many possibilities and inspire us.

Train your own CartoonGAN¶

You may noticed that we've put a lot of focus on actual application rather than the algorithmic details for CartoonGAN. That's because:

• I suspect most readers will be more interested in how to use CartoonGAN as a service rather than how to train a model themselves
• There are already lots of great learning resources on how to train your GANs on the web

For those who are interested in implementation details of CartoonGAN, we suggest you take a look at our github repo. If you have your own datasets, you can even train your own CartoonGAN simply in one command:

Our project can also be used to learn the latest version of TensorFlow since everything is implemented in Tensorflow 2.0 alpha:

• Use tf.keras to implement custom layers and GAN
• Use tf.data to properly load and process large amount of images
• Write custom train logic and make computation faster using tf.function
• Use TensorBoard to monitor model performance in real-time

As some of you already knew, it is not easy to train a GAN. You must keep monitoring your model's performance to decide how to adjust your hyper-parameters or even model architecture accordingly.

In our training script, TensorBoard is integrated perfectly so you can monitor your model's performance easily:

In addition to metrics and loss functions, it is good practice to keep an eye on the images generated by GAN during training as well.

Using our script, monitoring generated images on TensorBoard is a no-brainer:

Although we only record CartoonGAN-specific metrics and images here, you can easily apply same technique to monitor any other models you like.

Finally, in order to get a sense of how CartoonGAN is doing during the training phase, we can save some images and use them as a validation set.

The idea here is to "test" CartoonGAN using same images periodically so that we can observe how it improves over time:

Even before the training is finished, you can tell that CartoonGAN somehow learns to transform real world images into cartoon-like images with clear edges and smooth color shading.

Training a GAN like CartoonGAN is far more difficult than training a simple classifier. But in our github repo, we have tried our best to make the TensorFlow code clear and easy to understand, hoping it will help more people to get started.

Needless to say, you will be able to get a better understanding of our code if you already know some basics. In the next section, I will list some good learning resources for your reference.

Recommended learning resources¶

Recently, deep learning-based generative models are very popular and GAN is one of the most exciting research areas. The CartoonGAN we saw in this post is just a simple application in the ever-growing GAN zoo.

For those who want to learn more about GAN, I recommend the following resources:

• MIT 6.S191 Deep Generative Models
• GAN Lab let you train GANs on browser
• Andrej Karpathy's online GAN demo
• TensorFlow offical tutorial teach you how to generate MNIST using DCGAN
• Synced: Reproducing Japanese Anime Styles With CartoonGAN AI
• Open Questions about Generative Adversarial Networks

Research on generative models and GAN are evolving rapidly, but I believe you will be able to implement some simple GANs and understand code written by others after studying these learning resources.

To sum up¶

AI technology and applications like CartoonGAN should be more accessable to everyone, not only researchers or machine learning practitioners.

With this in mind, I built the project to let you create your own anime in no time and hope it can encourge some of you to explore more about GAN and generative models, and create more interesting AI applications in the future.

Many thanks to the authors of CartoonGAN, TensorFlow/TensorFlow.js teams and my friend mnicnc404.

Oh! Also don't forget to share what anime you generate! You can find me on Twitter or Facebook. Let the world be filled with anime! :D