Render out Spatial Experiences with ease

mahi_sreekumar

Hey fellow CG folk,

I don't know if I can call myself that yet tbh. But here I am.

So this is going to seem like a shameless plug but honestly, I need your help! My team has been developing a new web app called Creator and we launched it last week. Creator enables any user to build custom spatial experiences with the help of generative AI. It's still a work in progress and that's where I need your help. As CG artists, this tool is for you and I would love for you to try Creator out and tell me what you did and didn't like! Really hoping you would try Creator here! 

Here is a simple experience I built:

If you are interested in building such experiences but need to some help, check out this step-by-step tutorial covering how you can build a spatial experience on Creator!

https://www.youtube.com/watch?v=tOIEgxruvSE&list=PLlVJ58hGJoHdcf8C8N_NCG2hzDl1CXSP5

Eric Chadwick

How exactly is generative AI used by your service? I couldn't find much in your docs except about using AI for processing NeRF assets?

There has been much discussion here (and much consternation, frankly) about the unethical harvesting of datasets used to train generative AI systems. So it would be helpful to hear more info about how you're using it, and how it has been sourced.

mahi_sreekumar

Hey Eric! I understand your interest in wanting to know more about our Gen AI pipeline and where it is used. I hope the explanation I provide answers your query. 

On using Gen-AI:

• We are using GenAI for using our single image-to-3D service and text-to-3D service, which is based on diffusion models.
• Our NeRF processing is based on a learning-based pipeline.
• Our text-2-room service will be public soon as well which also uses Gen-AI

On datasets:

• We make use of both open-sourced 3D datasets, and our custom in-house dataset, which our 3d team has built over time.

Hope this clears things up. 

Eric Chadwick

Thanks for the reply, mahi.

Unfortunately, if your service is replying on diffusion models, then it has likely been trained using web-scraped datasets (for example LAION and CIFAR-10) which infringe copyrights when they're subsequently used for commercial purposes. 

For example LAION is not for commercial use, see 
https://laion.ai/blog/laion-400-open-dataset/#license

License

We distribute the metadata dataset (the parquet files) under the most open Creative Common CC-BY 4.0 license, which poses no particular restriction. The images are under their copyright.

An intriguing look at how diffusion models memorize scraped data.
https://www.marktechpost.com/2023/07/22/what-did-you-feed-on-this-ai-model-can-extract-training-data-from-diffusion-models/

I have a friend working at Avataar, so it makes me a bit uncomfortable calling this out. However as a working artist I cannot condone the use of these types of unethically-sourced AI Gen models. And speaking on behalf of Polycount, we're strongly against anything that infringes on artist copyrights.

mahi_sreekumar

Thanks, Eric.

We completely understand your perspective, especially in light of recent developments. We also believe one shouldn’t enable infringement of copyrighted data. The community is working hard on finding ways to train generative models without allowing data memorization. Hopefully, these breakthroughs will come sooner rather than later.

At Avataar, our core philosophy revolves around enhancing the capabilities of 3D artists. We firmly believe in the synergy between human creativity and the power of AI, and want to achieve it without infringing on artists’ copyrighted data.

We train our AI models, using 3D data released under the Creative Commons license, permitted for our use-case, alongside our in-house data.  I checked with our team internally and they confirmed that our models don’t reproduce copyrighted data.

Having said that, as AI and copyright law are evolving, hopefully, the communities will come to a consensus on how training data should be used and attributed, especially as AI continues to play a larger role in content generation

Eric Chadwick

Yeah it's a hard problem for these image generation models.

For better results, they need to train on a large database of image-metadata pairings, but the actual number of open-source assets is fairly small.

I've created open-source assets. But I realize not everyone can nor wants to do this.