High-Resolution Neural Texture Synthesis | Two Minute Papers #221

High-Resolution Neural Texture Synthesis | Two Minute Papers #221

Dear Fellow Scholars, this is Two Minute Papers
with Károly Zsolnai-Fehér. Deep Learning means that we are working with
neural networks that contain many inner layers. As neurons in each layer combine information
from the layer before, the deeper we go in these networks, the more elaborate details
we’re going to see. Let’s have a look at an example. For instance, if we train a neural network
to recognize images of human faces, first we’ll see an edge detector, and as a combination
of edges, object parts will emerge in the next layer. And in the later layers, a combination of
object parts create object models. Neural texture synthesis is about creating
lots of new images based on an input texture, and these new images have to resemble, but
not copy the input. Previous works on neural texture synthesis
focused on how different features in a given layer relate to the ones before and after
it. The issue is that because neurons in convolutional
neural networks are endowed with a small receptive field, they can only look at an input texture
at one scale. So for instance, if you look here, you see
that with previous techniques, trying to create small-scale details in a synthesized texture
is going to lead to rather poor results. This new method is about changing the inputs
and the outputs of the network to be able to process these images at different scales. These scales range from coarser to finer versions
of the same images. Sound simple enough, right? This simple idea makes all the difference
– here, you can see the input texture, and here is the output. As you can see, it has different patterns
but has very similar properties to the input, and if we zoom into both of these images,
we see that this one is able to create beautiful, high-frequency details as well. Wow, this is some really, really crisp output. Now, it has to be emphasized that this means
that the statistical properties of the original image are being mimiced really well. What it doesn’t mean is that it takes into
consideration the meaning of these images. Just have a look at the synthesized bubbles
or the flowers here. The statistical properties of the synthesized
textures may be correct, but the semantic meaning of the input is not captured well. In a future work, it would be super useful
to extend this algorithm to have a greater understanding of the structure and the symmetries
of the input images into consideration. The source code is available under the permissive
MIT license, so don’t hold back those crazy experiments. If you have enjoyed this episode and you think
the series provides you value or entertainment, please consider supporting us on Patreon. One-time payments and cryptocurrencies like
Bitcoin or Ethereum are also supported, and have been massively successful, I am really
out of words, thank you so much. The details are available in the video description. Thanks for watching and for your generous
support, and I’ll see you next time!

Comments (37)

  1. If you are interested in one-time payments, look below. Thank you very much for your generous support!
    Bitcoin: 13hhmJnLEzwXgmgJN7RB6bWVdT7WkrFAHh
    Ethereum: 0x002BB163DfE89B7aD0712846F1a1E53ba6136b5A

    Our Patreon page with the details:

  2. Wow, this is another jump in the right direction.

  3. Wow, this is actually revolutionary

  4. In the future, AI will be able to produce entire movies and video games

  5. The creation of AGi is probably life's purpose:

  6. progress, aimed at creating the illusion of a reality so perfect that philosophy should answer where it is going …

  7. Great to see how quickly AI is evolving! Keep up the awesome work!

  8. Papers like this is the reason I am subscribed to this channel, wow!

  9. I wonder how much bigger Wang or edge tiles need to be than the examples this network receives, to produce similar results

  10. The video games of the future are going to look amazing thanks to AI.

  11. I really like your comment that in future work they should make NN be able to understand underlying symmetries in the image. In retrospect it's very obvious, that in our brain there is some special mechanism to reconginze symmetries and to recognize even slightest deviations from them. Maybe that's the reason why so often the output from NNs looks so ugly to our eyes.

  12. Hey, Károly! Would you support Monero (XMR) too? Would love to contribute 🙂

  13. Thanks for subtitles!!!!

  14. Sounds like experimenting with this in the wavelet domain could be quite interesting, as that lends itself very easily to multiresolution operations.

  15. Super interesting! I am only just starting to learn more about AI and deep learning and in particular am interested in how it could be integrated into rendering in game development. Once we have something trained to understand the subject matter, context etc this could be very useful in making game worlds with unique details? Like moss on rocks

  16. I think it could do some sort of outline recognition and understanding it's properties. Would fix the bubbles at least..)


  18. Put a Jupiter texture map for the input.

  19. I'm very excited to see where this is going.

  20. if you're a geologist the generated texture looks fake af.

  21. Regular videos : 360p
    2 minute papers : 1080p60

  22. This would be so nice in nuke for creating clean plates of footage

  23. The link is not working. Can you fix it?

  24. Dear Udemy Scholars…

  25. The generated crowd and flower images are right in the uncanny valley. shudder

  26. I didn't understand the video.
    Does it mean that you feed in different varieties of course and fine images into the neural network instead of just the regular image?

  27. Hey, can you do a video about Capsule Networks? Thanks for all the effort!

  28. Nice work, huge improvement!

  29. So is this basically a clever way of texture compression?

  30. what iss the specific song name at the outro ?

  31. Would pretty interesting imho to add an adversative NN dedicated to macro geometrical structures (at different level too). Working at different levels seems also a very good idea !

  32. New paper has been released called "Non-Stationary Texture Synthesis by Adversarial Expansion". That largely addresses the problems discussed here.

  33. Man you are the best, please continue!!

Comment here