Meta, under the leadership of Mark Zuckerberg, is diving into the emerging field of AI video generation with a new tool called Movie Gen. This video generator can produce reasonably realistic videos from brief text prompts, making it a versatile asset for both Hollywood filmmakers and everyday users on platforms like Instagram. However, access to Movie Gen remains restricted to Meta itself for the time being. Notably, this generator can also create audio, establishing it as one of the most sophisticated deepfake tools available today.
In a recent blog post, Meta showcased some of Movie Gen’s capabilities with several example videos. One featured an adorable baby hippo gracefully swimming just beneath the water’s surface, seemingly unfazed as it held its breath. Other examples included penguins clad in “Victorian” attire, complete with ill-fitting sleeves and skirts that don’t accurately represent the historical era. Another amusing clip showed a woman DJing next to a cheetah, blissfully unaware of the potential danger posed by the distracted big cat.
The competition in AI-generated video is heating up. This year, Microsoft introduced VASA-1, while OpenAI teased its own video generator, Sora, which promises “realistic” outputs from simple text prompts. However, despite the initial excitement in February, Sora has not yet launched. Meta’s Movie Gen stands out with additional features, including the ability to edit existing videos using text prompts, generate videos from images, and add AI-generated sound to the final product.
One particularly innovative aspect of Movie Gen is its video editing suite, which can enhance both generated and real-world videos. Meta claims that its technology preserves the integrity of the original content while adding new elements—be it backgrounds or costumes for characters. Users can even incorporate photos of individuals into the generated scenes, further enhancing personalization.
Meta already possesses advanced music and sound generation models. The company demonstrated how its 13B parameter audio generator could layer sound effects and background scores onto video content. For example, a simple input like “rustling leaves and snapping twigs” could accompany a video of a snake slithering through a forest. However, this audio generator currently has a 45-second limit, so it won’t be crafting full-length films just yet.
As for availability, it seems Movie Gen won’t be rolling out to the public anytime soon. Meta’s chief product officer, Chris Cox, shared on Threads that the technology is still expensive to operate, and the generation time is too lengthy for practical use at this stage.
In a detailed whitepaper, Meta outlined the architecture of Movie Gen, which consists of multiple foundational models. The company’s most powerful video model boasts a whopping 30B parameters and can handle a maximum context length of 73,000 video tokens. The accompanying audio generator, with its 13B parameters, can convert both video to audio and text to audio.
When comparing Movie Gen to competitors, it’s challenging to draw direct parallels. OpenAI’s Sora, for instance, reportedly employs a unique data methodology involving “patches” akin to tokens used in GPT models. Unlike many firms in the AI space, Meta continues to disclose data related to its AI tools, a practice that has diminished as the industry becomes increasingly commercialized. However, details on the training data for Movie Gen remain sparse. It’s likely that user-generated videos from Facebook contributed to the dataset, along with images captured by the Meta Ray-Ban smart glasses.
For now, users can’t access Movie Gen directly. Other platforms, like RunwayML’s Gen 3, offer limited access for creating short clips, although users must eventually pay for additional tokens. Reports from earlier this year revealed that Runway sourced its training data from countless YouTube videos, often without obtaining permission from content creators.
Meta asserts that it collaborated closely with filmmakers and video producers in the development of Movie Gen and plans to continue these partnerships as the project evolves. There have been indications that film studios are increasingly engaging with AI companies, with indie powerhouse A24 recently exploring collaborations with venture capital firms focused on AI, including some with ties to OpenAI. On the other hand, Meta is reportedly negotiating with notable Hollywood figures like Judi Dench and Awkwafina to utilize their voices for future AI projects.
As the AI video landscape expands, Movie Gen positions Meta at the forefront, offering tools that could reshape how content is created and consumed. The company’s emphasis on collaboration with industry professionals reflects a commitment to integrating AI technology meaningfully into the filmmaking process. While the wait for public access continues, the potential implications of such technology are immense, suggesting a future where creativity and AI intertwine more seamlessly than ever before.
In the rapidly evolving world of AI-generated media, the question remains: how will the industry adapt to these innovative tools? As Meta and other companies push forward, it’s clear that the interplay between traditional filmmaking and AI-generated content is only just beginning.
