Nature VR

Making the natural treasures of our world accessible to everyone. The project will take 2d images of natural scenes, and attempt to texture 3d Unity environments, using a complex of machine learning approaches for interpolating texture and depth from static images.

Motivation

• https://psmag.com/environment/living-near-a-forest-is-linked-to-better-brain-health?source=TDB&via=FB_Page

Terrain generation models

• https://github.com/universome/alis

2d –> 3d models

• Google PHORHUM - “…Observing that 3D supervision alone is not sufficient for high fidelity color reconstruction, we introduce patch-based rendering losses that enable reliable color reconstruction on visible parts of the human, and detailed and plausible color estimation for the non-visible parts…” - https://phorhum.github.io/

• NVIDIA’s New AI: Enhance! - https://www.youtube.com/watch?v=e0yEOw6Zews

• “AI can turn a collection of 2D images into an explorable 3D world” - https://www.newscientist.com/article/2294737-ai-can-turn-a-collection-of-2d-images-into-an-explorable-3d-world/ - https://www.youtube.com/watch?v=BBmcsyB7aDw

• https://www.reddit.com/r/MachineLearning/comments/ruaew1/r_learning_3d_representations_from_2d_images/

• https://www.reddit.com/r/MachineLearning/comments/pskwhy/r_3d_annotation_of_arbitrary_objects_in_the_wild/

• “AI Generates 3D high-resolution reconstructions of people from 2D images

  Introduction to PIFuHD” https://www.youtube.com/watch?v=ajWtdm05-6g

• “PIFuHD: High-Resolution 3D Human Digitization (CVPR2020 Oral, 5min Presentation)”
  https://www.youtube.com/watch?v=uEDqCxvF5yc

• https://github.com/facebookresearch/pifuhd

• https://colab.research.google.com/drive/1cPntk-PyiXnxiUfsfw5teV8UrdPoi0e4?usp=sharing

Existing Nature VR applications

• “Get Lost In Nature With Luke” https://www.oculus.com/experiences/rift/3507999942601704/ - https://www.reddit.com/r/virtualreality/comments/nubeqv/get_lost_in_nature_with_luke_how_is_this_level_of/ - https://www.youtube.com/watch?v=j4RX_TnJH0k
• “Nature Treks VR” - https://www.oculus.com/experiences/quest/2616537008386430

Related

• https://www.reddit.com/r/InternetIsBeautiful/comments/nuc3ef/this_website_developed_by_nvidia_lets_you_draw/h0xbuhp/
• https://github.com/cyrildiagne/instagram-3d-photo
• https://www.reddit.com/r/MediaSynthesis/comments/fq6zyx/front2back_ai_reconstructs_3d_shapes_from_2d/
• https://www.reddit.com/r/MachineLearning/comments/fc9l4p/n_turn_2d_photos_into_3d_using_convolutional/
• https://github.com/NVIDIAGameWorks/kaolin/
• https://smaerdlatigid.github.io/3D-Photo-Viewer/
• https://www.reddit.com/r/replications/comments/e17lr4/turning_2d_into_3d_with_ai_depth_map/

• https://arxiv.org/abs/1811.07605 sparse 3d to 3d point cloud

• Eric Guérin, Julie Digne, Eric Galin, Adrien Peytavie, Christian Wolf, et al.. Interactive Example- Based Terrain Authoring with Conditional Generative Adversarial Networks. Transactions on Graph- ics (Proceedings of Siggraph Asia 2017), ACM, 2017, 36, pp.228 - 228. <10.1145/3130800.3130804>.

• BBC Earth - Live in VR - https://www.youtube.com/watch?v=k9OkENZwzL4

• https://www.reddit.com/r/virtualreality/comments/a340lk/research_at_nvidia_the_first_interactive_ai/

• https://github.com/SyntonicApps/unity-vrtk-examples - https://www.youtube.com/watch?v=aJhPm5h1pl4

• https://github.com/llSourcell/How-to-Generate-Art-Demo
  • https://medium.com/towards-data-science/neural-networks-and-the-future-of-3d-procedural-content-generation-a2132487d44a - “Knowing that, I went over to http://terrain.party, a website where you can download elevation data from the Earth.”

• https://github.com/jiajunwu/3dinn - “Wu, J., Xue, T., Lim, J. J., Tian, Y., Tenenbaum, J. B., Torralba, A., & Freeman, W. T. (2016). Single image 3D interpreter network. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (Vol. 9910 LNCS). http://doi.org/10.1007/978-3-319-46466-4_22”

• https://www.reddit.com/r/virtualreality/comments/65ctm9/vr_biking_in_the_mountains_with_vr_rider_early/
• http://variety.com/2017/digital/news/adobe-6dof-vr-video-algorithms-1202394491/
  • http://zhilichen.com/projects/sixdofvr/vr360.pdf
• https://research.googleblog.com/2017/07/using-deep-learning-to-create.html