NVIDIA’s Minecraft AI: Feels Like Magic! 🌴 …Also, 1 Million Subs! 🥳

Dear Fellow Scholars, this is Two Minute  Papers with Dr. Károly Zsolnai-Fehér. We just hit a million subscribers! I can  hardly believe that so many of you Fellow   Scholars are enjoying the Papers! Thank you so  much for all the love! In a previous episode,  

We explored an absolutely insane idea. The idea  was to unleash a learning algorithm on a dataset   that contains images and videos of cities,  then take a piece of video footage from a game,   and translate it into a real movie. It  is an absolute miracle that this works,  

And it not only works, but it works reliably and  interactively. And it also works much better than   its predecessors. Now, we discussed that the  input video game footage is pretty detailed. And I was wondering, what if we don’t create  the entire game in such detail. What about,  

Creating just the bare minimum,  a draft of the game if you will   and let the algorithm do the heavy lifting.  Let’s call this world to world translation!   So, is world to world translation  possible, or is this science fiction? Fortunately, scientists at NVIDIA and Cornell  University thought of that problem and came  

Up with a remarkable solution. But, the first  question is – what form should this draft take?   And they say, it should be a Minecraft world, or  in other words, a landscape assembled from little   blocks. Yes, that is simple enough indeed. So this  goes in. And now, let’s see what comes out. Oh my!

It created water, it understands the concept of  an island, and it created a beautiful landscape,   also, with vegetation. Insanity. It even seems to have some concept   of reflections, although they will need  some extra work to get perfectly right.

But, what about artistic control? Do we get  this one solution, or can we give more detailed   instructions to the technique? Yes we can! Look at  that. Since the training data contains desert and   snowy landscapes too, is also supports them as  outputs. Whoa, this is getting wild. I like it.

And it even supports interpolation, which  means that we can create one landscape   and ask the AI to create a  blend between different styles.   We just look at the output animations, and pick  the one that we like best. Absolutely amazing. What I also really liked is that  it also supports rendering fog.  

But this is not some trivial fog technique,  no-no, look how beautifully it occludes the trees.   If we look under the hood, oh my! I am a  light transport researcher by trade, and boy,   am I happy to see the authors having done  their homework. Look, we are not playing  

Games here, the technique contains bona-fide  volumetric light transmission calculations. Now, this is not the first technique to perform  this kind of world to world translation.   What about the competition? As you see, there  are many prior techniques here, but there is  

One key issue that almost all of them share.  So, what is that? Oh yes, much like with the   other video game papers, the issue is the lack of  temporal coherence, which means that the previous   techniques don’t remember they it did a few images  earlier, and may create a drastically different  

Series of images. And the result is this kind of  flickering that is often a deal-breaker regardless   of how good the technique is otherwise. Look,  the new method does this significantly better. This could help level generation for computer  games, creating all kinds simulations, and  

If it improves some more, these could maybe even  become backdrops to be used in animated movies.   Now, of course, this is still not perfect,  some of the outputs are still blocky. But, with this method, creating virtual worlds  has never been easier. I cannot believe that we  

Can have a learning-based algorithm where the  input is one draft world, and it transforms it   to a much more detailed and beautiful one. Yes,  it has its limitations, but just imagine what   we will be able to do two more papers down the  line. Especially given that the quality of the  

Results can be algorithmically measured, which is  a godsend for comparing this to future methods.   And for now, huge congratulations to NVIDIA  and Cornell University for this amazing paper. Thanks for watching and for your generous  support, and I’ll see you next time!