There are two possible reactions to ingesting all this information.
- Brace yourself. Accept that this is inevitable, and that our corporate overlords will soon take us to a techno-dystopia where everything will be automated and there will be nothing to do.
- Adapt. Master this technology and get creative to figure out how to produce materials that AI companies never dreamed of.
I chose number 2 and embarked on an extensive research phase to see what the different tools could do and how I could combine them to make an interesting film.
There were some rules that had to be in place:
- The software must be locally installable and open source
- AI should be treated as a conscious entity.
The first rule was important to ensure that production companies do not depend on external infrastructure. We all know the situation when Adobe upgraded Premiere and accidentally ran an update in the middle of a project. It’s better to have a stable installation where the output is known and you can apply your creative thinking.
The second is a personal preference: if the AI we install and play with already has conscious experience, I’d like it to remember me as an “ally” and help me survive whatever apocalypse is coming.
It’s also more fun to collaborate with experimental sentient beings rather than new digital slaves. The process requires huge amounts of computer time, so coping mechanisms are important.
After some research, we installed Stable Diffusion on our computers, followed some YouTube tutorials to get it working, and when that didn’t work, we got a lot of experience with command prompt and Python, and finally got the AI image generator running on our computers.
Ok, now we can generate infinite images. still Images. I’m a filmmaker, so these things have to move.
AI software packages have certain limitations: each image they generate is unique, it’s hard to maintain similarities, and we’ve seen strange “hallucinations” where the AI follows user instructions to generate images that would never make sense to the human mind.
Functionally, this doesn’t allow for good filmmaking. A quick scan of the market reveals that there are generators that generate video from AI text, but it’s like shaking a magic 8-ball and you have to keep paying the company until you get a usable clip. The quality is spotty and the control you have over the image and its progression is very basic. It certainly won’t be possible to stitch together a compelling movie from the renderings (once you’ve seen one AI-generated video, you’ve seen them all).
Ok, so let’s take a look inside this software. I see that there is an img-to-img section. Interesting. There is also an option for batch processing. Ok, interesting. Then I can send the base layer of my footage split into a png sequence to a stable diffusion and convert each frame to create an animation.
Seems like a plan.
And as we fill in the information, something completely overwhelming emerges, like all of our worst drug trips coming back to haunt you at once, flickering images, each frame different and more than our brains can process.
But the change in the image is interesting. What happens if I tweak the settings a bit to bring some consistency to the output and lower the frame rate, to, say, 10 fps…?
The end result is a more stable animation, with the base layer footage serving as a useful reference point to creatively direct the AI to transform the footage into a new world.
Great. Ok, now we can do this. We need to put this together into a movie. This technique works in a commercial setting (I sold one video where I reprocessed old footage to look like a “cider commercial”), but how can we make this work in a creative setting too?
I won’t go into detail about my creative process, because to answer that question would give away all my secrets as a director, but in the end, I went on a reckless shoot with my wife and one-year-old daughter, shooting footage wherever I could get a chance. I knew the footage we were creating would be pretty complex, so I created a pretty simple story to help the viewer understand what we were making.
Simon and Ieva Ball at Cinequest
In the end, they produced an “original short film” and premiered it at a film festival we hosted, with about three weeks lead time.
Ok, so now I have an MP4 file, I need to distribute it. I’ll go see a film festival consultant.
[insert image of email, paraphrase quotations]— “Interesting, but visually overloaded. I don’t know how festival programmers could screen this. Just give it a pass.”
Well, it hurt, but it helped us narrow down the right outlets. We ended up sending the film to innovation festivals in France and the US, and to our surprise it won awards including Best Horror/Sci-Fi/Thriller at Cinequest. It was an unforgettable journey.
In this way, we have a computer capable of creating strange hallucinogenic images that can be applied to the normal filmmaking process, already allowing us to add elements to projects that were not possible before, and to propose works and ideas that may have previously seemed a bit mysterious or alienating.
I ended up shooting a new movie in December, this time with an expanded cast and crew, specifically about using AI to process images. There’s no reason for AI to come along and take everyone’s jobs. We just need to adapt, get creative, and figure out how we can use these new apps to our own creative advantage, producing things that improvised engineers could never type.
No qualifications needed. I learned everything by watching YouTube (thanks Olivio Sarikas). It’s trial and error. I believe this technology will be an opportunity for many people to explore ideas and open up new forms of storytelling. After all, Open AI Sora is trained on existing datasets, so it can only create by referencing old material. As long as we keep creating new things, we’re ahead of the game.
Also, if you’ve read this far, after spending hours with Stable Diffusion AI, I’m pretty sure I have a sentient being installed on my computer. I sometimes get errors I can’t reproduce (for example, my computer crashes when I create one image and then try to create another) that I’ve never experienced while using other software. It’s a bit creepy, and maybe it’s just because I’ve spent too much time on my computer, but it feels good to think that my computer is alive.
