Project Overview

LiveCC, first released on April 25, 2025, by the National University of Singapore's Show Lab and ByteDance, is a video-based large language model project focused on large-scale streaming speech transcription. The project aims to train the first video-based large language model with real-time commenting capabilities using an innovative video-automatic speech recognition (ASR) streaming method, achieving state-of-the-art (SOTA) performance in both streaming and offline benchmarks. Related research papers are available. LiveCC: Learning Video LLM with Streaming Speech Transcription at ScaleIt has been included in CVPR 2025.

This tutorial uses a single RTX A6000 card as the resource.

Project Examples

Run steps

1. After starting the container, click the API address to enter the Web interface

2. Once you enter the web page, you can interact with the model

If "Bad Gateway" is displayed, it means the model is initializing. Since the model is large, please wait about 1-2 minutes and refresh the page.

This tutorial provides two module tests: Real-Time Commentary and Conversation modules.

Do not switch models frequently to avoid resource congestion.

The functions of each module are as follows:

Real-Time Commentary

🖌️ If you see a high-quality project, please leave a message in the background to recommend it! In addition, we have also established a tutorial exchange group. Welcome friends to scan the QR code and remark [SD Tutorial] to join the group to discuss various technical issues and share application effects↓

Citation Information

The citation information for this project is as follows:

@inproceedings{livecc,
    author       = {Joya Chen and Ziyun Zeng and Yiqi Lin and Wei Li and Zejun Ma and Mike Zheng Shou},
    title        = {LiveCC: Learning Video LLM with Streaming Speech Transcription at Scale},
    booktitle    = {CVPR},
    year         = {2025},
}