VideoPoet: A Large Language Model for Zero-Shot Video Generation
Dan Kondratyuk*, Lijun Yu*, Xiuye Gu*, José Lezama*, Jonathan Huang*, Grant Schindler, Rachel Hornung, Vighnesh Birodkar, Jimmy Yan, Ming-Chang Chiu, Krishna Somandepalli, Hassan Akbari, Yair Alon, Yong Cheng, Josh Dillon, Agrim Gupta, Meera Hahn, Anja Hauth, David Hendon, Alonso Martinez, David Minnen, Mikhail Sirotenko, Kihyuk Sohn, Xuan Yang, Hartwig Adam, Ming-Hsuan Yang, Irfan Essa, Huisheng Wang, David A. Ross, Bryan Seybold*, Lu Jiang* (*Equal contribution)
Published in ICML (Best Paper Award, Oral), 2024
We present VideoPoet, a language model capable of synthesizing high-quality video, with matching audio, from a large variety of conditioning signals. VideoPoet employs a decoder-only transformer architecture that processes multimodal inputs – including images, videos, text, and audio. The training protocol follows that of Large Language Models (LLMs), consisting of two stages: pretraining and task-specific adaptation. During pretraining, VideoPoet incorporates a mixture of multimodal generative objectives within an autoregressive Transformer framework. The pretrained LLM serves as a foundation that can be adapted for a range of video generation tasks. We present empirical results demonstrating the model’s state-of-the-art capabilities in zero-shot video generation, specifically highlighting VideoPoet’s ability to generate high-fidelity motions.
@inproceedings{kondratyuk2023videopoet,
title={VideoPoet: A Large Language Model for Zero-Shot Video Generation},
author={Kondratyuk, Dan and Yu, Lijun and Gu, Xiuye and Lezama, Jos{\'e} and Huang, Jonathan and Hornung, Rachel and Adam, Hartwig and Akbari, Hassan and Alon, Yair and Birodkar, Vighnesh and Cheng, Yong and Chiu, Ming-Chang and Dillon, Josh and Essa, Irfan and Gupta, Agrim and Hahn, Meera and Hauth, Anja and Hendon, David and Martinez, Alonso and Minnen, David and Ross, David and Schindler, Grant and Sirotenko, Mikhail and Sohn, Kihyuk and Somandepalli, Krishna and Wang, Huisheng and Yan, Jimmy and Yang, Ming-Hsuan and Yang, Xuan and Seybold, Bryan and Jiang, Lu},
booktitle=ICML,
year={2024}
}