June 21, 2024

Physical Interfaces + Real-time Diffusion

I have explored diffusion pipelines in the past, primarily using img2img for the creative opportunities they provide. My first deep dive into this field was focused on using Stable Diffusion in conjunction with ControlNet, TemporalKit and Ebsynth on NeRF (Neural Radiance Fields) renders. NeRF allowed me to generate a wide range of viewpoints and camera motions for natural or urban scenes, offering opportunities to experiment with different visual narratives. Meanwhile, Stable Diffusion enabled me to explore new visual concepts and be transported to different worlds. And the mix of these two technologies allows me to maintain control over my creative direction. At that time, all my explorations were non-real-time pipelines—rendering a video from NeRF and then processing these videos with Stable Diffusion to output another video.

We are now reaching a point where models and their associated pipelines are becoming more performant and faster, approaching real-time interactive generation. This is where I’m getting really excited.

Here is a real-time output of an img2img diffusion pipeline, using Stream Diffusion and SDXL Turbo. Being able to interact with the pipeline using my hands and actual clay, and seeing sculptures emerge from my actions, is mesmerizing. The goal is of course not to replace or to copy the creative process or the mastery of human individuals, but to engage with these on a different level. These tools allow me to explore new visual concepts, trigger a reflection process, and again to be transported to different worlds.

Using generative AI, it is crucial to acknowledge the ethical and legal aspects of using models trained on datasets we don't own, control, or fully understand. It’s important to recognize a dataset's limitations and the need for ongoing work to mitigate copyright infringement and biases. All this is vital to ensure ethical AI development and responsible innovation.

This hands-on approach, is a core part of our creative process at Dpt.. Continually learning, understanding, and experimenting with these technologies enables us to navigate more effectively in this rapidly evolving space in order to create captivating experiences.