AI video creation has come a long way in a short time, from 2-second clips with significant morphing and distortion to footage that is almost indistinguishable from filmed footage. Runway is the latest player in the space to introduce its next generation model.
Gen-3 was first revealed two weeks ago, and after initial testing by creative partners, it’s now available to anyone, at least in the text-to-video version. Text to image conversion will be available soon.
Each generation creates a 10-11 second photorealistic clip with amazingly accurate motion, including depictions of human actions that reflect the scenario and environment.
From my initial testing, it is as good as Sora in some tasks, though better than OpenAI’s video model in the fact that it is widely available to everyone. It’s also better than Luma Labs’ Dream Machine in motion understanding, but without an image-to-video model, it fails in consistency.
What is Gen-3 working with?
Watch
I’ve been playing with it since its launch and have created over a dozen clips to effectively improve the prompting process. “Less is more” and “be descriptive” are my key takeaways, though Runway provides a helpful guide to Gen-3 prompts.
You’ll want to try and get challenges right from the start, as each generation with Gen-3 costs between $1 and $2.40 per 10-second generation. The cheapest option is to top up credits, which costs $10 per 1000. In contrast, Luma Labs’ basic plan costs 20c per generation.
When it comes to actually using the video generator, it works exactly like Gen-2. You give it a challenge and wait for it to create a video. You can also use lip sync, which is now integrated into the same interface as video creation and animates the entire video.
I came up with five prompts that worked particularly well and shared them below. Until the image-to-video transfer kicks in, you’ll have to be very descriptive if you want a specific look, but Runway’s generation of Gen-3 footage is impressive. You also only get 500 characters for the challenge.
1. Cyber city race
This was one of the last prompts I created and created from refinements. It’s relatively short, but because of the specific description of the movement and style, Runway interpreted it exactly as I expected.
Challenge: “Hyperspeed POV: Racing through a neon-lit cyberpunk city, data streams and holograms blur as we zoom into the digital realm of swirling code.”
2. Diver
The first part of it involved some weird motion blur over the eyes and extended fingers that fixed itself. Otherwise, it was an impressive and realistic interpretation. The motion blur issue was part of the challenge, suggesting sunlight penetration. The challenge was too complex.
Challenge: “Slow motion: A diver explores a vibrant coral reef teeming with colorful fish. The sun’s rays penetrate the crystal clear water and create a dreamlike atmosphere. The camera pans alongside the diver as he encounters a curious sea turtle.’
3. View of the street
This is not only one of my favorite videos from Runway Gen-3 Alpha, but from everything I’ve created using AI video tools over the past year. He didn’t follow the instructions exactly, but he captures the sky as it changes throughout the day.
Challenge: “Hyper-speed time-lapse: The camera rises from street level to the roof, showing the transformation of the city from day to night. Neon signs come to life, traffic turns into streams of light, and skyscrapers light up against the darkening sky. The final image reveals a breathtaking cityscape under a starry night.”
4. Bear
I have massively rewritten this prompt. It was supposed to show the bear becoming more alive towards the end, but I asked him to do too much in 10 seconds.
Prompt: “Slow-motion close-up to wide-angle shot: A worn, old teddy bear sits motionless on a child’s bed in a dimly lit room. Golden sunlight gradually filters through the lace curtains, softly illuminating the bear. As the warm light touches its fur, the bear’s glassy eyes suddenly they blink when the teddy bear slowly sits up, his movements are smoother and more alive.”
I refined the prompt to: “Slow-motion close-up to wide-angle shot: A vintage teddy bear on a cot comes to life as golden sunlight filters through the lace curtains, the camera pulls out to reveal the bear sitting and becoming animated.”
This gave better movement, going in the opposite direction of the original, although it created some artifacts on the bear’s face and still didn’t land him.
5. The old farmer
This was the first challenge I tried with the Runway Gen-3 Alpha. It’s too complex and descriptive when I tried to replicate something I would create using image to video transfer in Luma Labs Dream Machine. It wasn’t the same, but it was very well done.
Prompt: “Sun-beaten farmer, 1970s, scorched field surveys. Leather skin, silver beard, eyes squinting under a dusty hat. Threaded shirt, patched overall. Calloused hands hold a fence post. Golden light illuminates the lines of concern, determination. The camera zooms in on the steel gaze. The barren land stretches, distant ruins loom. Temporary irrigation, visible fortified fences. The old man reaches into his hat, revealing a hidden technique. Equipment flashing, hope dawning.”
More from Tom’s guide
Back to the MacBook Air
