ViZARTS Encounters IV: "Reality Jump" - Virtual Production ICVFX using AI generated 3D Locations

These scenes are the results of the efforts of a crew of filmmakers and techies who joined forces and collaborated intensively during three days at our ViZARTS "AI & Virtual Production for Film makers" Workshop at Aalborg University's Medialogy education (Copenhagen. Denmark, December 3-6. 2024)
The theme was "Show me a World, I haven't seen!" and the challenge was for each of the four crews to explore how generative AI could help them produce exterior and interior shots (Approx. 1.5 hours shooting time for each location).
The crews worked to achieve Final Pixels In-Camera and in-camera visual effects ( ICVFX ), by using physical props and effects, a simple rear-projection screen, Unreal and Vive camera tracking.

The goal of this workshop was to let participants experiment with how any location - real or imaginary - now can be re-created or generated using an AI prompt (or a real photo).
All crews used text prompts to create 2D location images generated with the help from Midjourney. These 2D images were then adjusted a bit (using ComfyAI) and transformed into a high-resolution 3D Depth map (using Stable Diffusion), which could then be used as a background behind the actors, while achieving a parallax effect when the camera was moving.

Here is more from the Behind-the-Scenes:
   • ViZARTS: "Reality Jump" - A Quest for AI-G...  

If you are interested in receiving invitations for future AI & VP workshops, then please sign up for the ViZARTS Newsletters here (Approx 4 per year):
https://vizarts.aau.dk

Or stay tuned here:
  / vizarts  

ViZARTS is supported by:
Nordisk Film Fonden
Aalborg University - The Medialogy Education
Amunet
Storyline Studios CPH
Upright Music
SAMSUNG / SMILE Lab

--------

ViZARTS presents:
Reality Jump

Credits:

Team 1 - “Somewhere Never”
Victor Harder Hesel - Director
Dirk Brüel - Director of Cinematography
Ola Persson - Editor
Eigil Hedegaard - Acting

Team 2 - “Toy Store Story”
Christian Grønvald - Director
Morten Øbro - Director of Cinematography
Eigil Hedegaard - Acting

Team 3 - “Himmelstigen”
Tara Salas - Director
Sine Vadstrup Brooker - Director of Cinematography
Susanne Wejp-Olsen - Production Design
Morten Holst - Acting
Mette Løjberg- Acting

Team 4 - “The B-Day Showdown”
Jacob Østergaad Johansson – Director of Cinematography
David Wittrock Rathje - Production Design
Felix Drott - Edit
Morten Holst - Acting

Pre-production Team:
Peter de Neergaard
Salle Salée
Jesper Duelund Hansen
Michael G. Rose
Jacob Stig Olsson
Jonas Abrahamsen

Medialogy Volunteers:
Felix Jordan Hasal
Alejandro Lenin Rosado Garcia
Nicolai Villas Gajda Østergaard

The ViZARTS Crew:
Ida Marie Bredegaard - Concept Artist / Sketch to AI pipeline
Andreas Jin Sum - Supporting AI Crew / Runner / Intern / Boom
Nicolai Lind Bouquin - Supporting AI Crew
Abderrahman Mhadden - Technical Support / Reality Jump machine operator
Mathias Ramsø Thomsen - Technical Lead (Unreal Brainbar / Space Command)
Aishah Hussain - Unreal Artist / ViZARTS manager
Ani Kreyer - BTS cinematographer and editor
Henrik Schønau Fog - ViZARTS Founder / Executive producer

AI tools used:
Midjourney - For generating 2D locations based on prompts
Comfy UI - For adjusting 2D location images
Stable Diffusion - For generating 3D and 2.5D depth maps from 2D images

Sound Effects:
Upright Music
Pixabay

Special Thanks to:
Storyline Rentals: Arri Alexa 35 Gear
Upright Music: Music and Sound Effects
Samsung: Monitors, Tablets and Smartphones

Aalborg University, Copenhagen Campus:
Erik Petersen
Stig Møllenberg
Frank Petersen
Lene Hastrup Jensen
Susan Biensø
Colleagues, Students and Staff at Medialogy / CREATE