ByteDance's new AI turns photos into movie sequences; Mistral launches a content moderation API; Walt Disney forms new AI business unit; Saudi to invest $100b in AI through "Project Transcendence"

UK launches business-oriented AI safety platform; VCs warn AI startups to prioritize revenue in 2025; there's a strong community of women building AI startups; AI chatbots are the new priests

Nov 08, 2024

∙ Paid

TikTok’s parent company ByteDance has just previewed X-Portrait 2, a new portrait animation model built in cooperation with researchers from Tsinghua University, China’s best-ranked university in science and technology.

The model is an evolution of the first-generation X-Portrait technology (which was presented earlier this year at SIGGRAPH) and provides similar functionality to Runway’s Act-One tool, though the comparison with the latter is not entirely fair as Act-One generates a full video sequence, not just AI avatars.

X-Portrait 2 works by taking two inputs: a static portrait image and a driving video. It uses the facial expressions and speech patterns of the human present in the driving video to animate the other human in the static image, generating expressive and realistic videos as an output.

What’s remarkable about X-Portrait 2 is how it is able to capture even the more subtle facial expressions from the inputs and preserve them in the outputs, thanks to improvements in its expression encoder which has been pre-trained on a large dataset likely sourced from user generated content on TikTok or Douyin.

X-Portrait 2 relies on a generative diffusion model to accurately render the facial expressions and head movements in the output video, including pouting, tougue-out, cheek-puffing and frowning. Diffusion models have become the norm in image and video generation over the last 12-24 months because they offer greater control, flexibility, and are less susceptible to model collapse compared to GANs.

The other noticeable innovation of X-Portrait 2 compared to its predecessor is the ability to disentangle appearance and motion, allowing the new model to focus solely on facial expressions and then faithfully transfer those expressions across different filmmaking techniques and styles, for example from live-action shots to animation clips. Therefore, the researchers argue that X-Portrait 2 could be deployed as a fast iteration tool in the video production process of Hollywood-quality animations or movies, saving valuable time and resources for visual effects teams.

X-Portrait 2 is one of several portrait animation solutions I’ve seen trending on social media over the last month. It feels like many of the companies that launched general purpose video generators earlier this year (Kuaishou’s Kling AI, MiniMax’s Video-01, Hailuo AI, Luma AI’s Dream Machine, Pika or Runway’s Gen-3 Alpha) are now looking to optimize their technology for more specific use cases such as AI avatars because they sense there could be a market for this technology among professional-quality video creators.

Something to keep in mind is that these image-to-video approaches don't fully generate lip-sync performances from a synthetic voice in a way that platforms such as Synthesia do, since they rely on an existing, expressive video to drive the character’s facial movements. And, as with other general purpose video models, there are open questions about how well they generalize in real scenarios since the results presented are always cherry picked to illustrate the best outputs.

It’s also noteworthy that more than half of the companies listed above come from China, which is culturally more open toward digital humans and also has a thriving animation and interactive entertainment community that produces high-quality user-generated content optimized for social sharing.

And now, here are this week’s news:

❤️Computer loves

Our top news picks for the week - your essential reading from the world of AI

VentureBeat: Mistral AI takes on OpenAI with new moderation API, tackling harmful content in 11 languages
Bloomberg: AI Will Transform Medicine. There's Just One Catch
FT: UK government launches new AI safety platform for businesses
Bloomberg: Saudis Plan $100 Billion AI Powerhouse to Rival UAE’s Tech Hub
TechCrunch: AI startups will need ‘quality of revenue’ to raise in 2025, seed VCs warn
Business Insider: For women building AI startups, finding community has been the key to success
Washington Post: A deepfake showed MLK Jr. backing Trump. His daughter calls it ‘vile.’
Axios: What AI knows about you
The Information: The Generative AI Spending of 50 Companies, From Coke to Walmart
New York Times: An ‘Interview’ With a Dead Luminary Exposes the Pitfalls of A.I.
WSJ: The Giant Supercomputer Built to Transform an Entire Country—and Paid For by Ozempic
Business Insider: AI chatbots are the new priests
FT: AI’s huge power needs give oil majors incentive to invest in renewables, says Adnoc boss
Reuters: Walt Disney forms business unit to coordinate use of AI, augmented reality

⚙️Computer does

AI in the wild: how artificial intelligence is used across industry, from the internet, social media, and retail to transportation, healthcare, banking, and more

🧑‍🎓Computer learns

Interesting trends and developments from various AI fields, companies and people

Keep reading with a 7-day free trial

Subscribe to Computerspeak by Alexandru Voica to keep reading this post and get 7 days of free access to the full post archives.