July 14, 2024

Physical Interfaces + Real-time Diffusion Overview

In the past few weeks, I deep dived into an exploration revolving around the use of physical interfaces to feed and interact with a real-time img2img diffusion pipeline using Stream Diffusion and SDXL Turbo. This exploration is not only a personal journey but also aligns with the innovative work we are doing at Dpt. with the same technology. This synergy between personal and professional exploration is why I love my work.

Interacting in real-time with a system has always been my guiding principle. Initially, I fed pre-rendered video to the diffusion pipeline, but now I can use a real-time video feed. Of course, this allows for ‘the mirror’ interaction (face filter), exploring various effects from humorous to eerie, but what really captivated me is to use my hands, objects, art supplies, tools, and light to create images and scenes.

Physical Media + Tools

I experimented with clay, manipulating different types and colors along with a selection of prompts. I used a magnifying glass, tracked in real-time, to focus the diffusion process on specific areas. Combining these tools created a dynamic and inspiring experience. Using magic clay to layer shapes and colors as a base for revealing landscapes and hidden worlds, and the magnifying glass to focus and reveal these details, was particularly effective.

Physical Light

I used light as my method of interaction with the img2img diffusion. This approach felt special right away. There was something magical about holding a physical light source and seeing it influence the generated visuals. I iterated on this technique with themes like Rococo architecture, flowers, Brutalist architecture, hidden worlds, and origami landscapes.

Rococo...

Brutalism...

I brought to the surface not just images but memories. Revisiting my hometown of Avignon, living in the ancient town center, and the Palais des Papes just five minutes away on foot. Though I now live in Montreal, using only my flashlight, I revisited some parts of the Palais des Papes at night.

Ink + Hybrid Formats

I also used ink in milk as a means of physical interaction with the diffusion pipeline. As I drop ink into milk, shapes come alive instantly. By learning to manipulate the combination of physical and digital elements, I steered the generated output toward my areas of interest. These iterations extended beyond ink in milk to include the format in which these elements are contained: a circular plate or a triptych of small stainless steel trays. These formats provide a structured yet flexible framework to explore themes and narratives across multiple visual spaces. It's magical.

Some of that last iteration has been captured in this insightful article by Fast Company's Jesus Diaz.

"Perhaps Bruyère’s work is a sign that there’s hope for artificial intelligence and the human brain to create together, rather than becoming adversarial entities in the AI hamster race."

I found his insights and descriptions particularly engaging and reflective of my work.