2026-10-14T16:01:03Z https://www.scidb.cn/oai

10.57760/sciencedb.00zor 2026-09-13T19:56:01Z

2026-09-13 A Latent-Processed Multi-Source Audiovisual Dataset for Video Generation Model Training doi:10.57760/sciencedb.00zor en 1. OverviewThis dataset is a large-scale preprocessed dataset designed for training video generation models. It aims to provide high‑quality training data for tasks such as text‑/speech‑driven video generation and talking‑head video synthesis. All data have been pre‑processed into latent representations required for video generation model training, allowing users to directly use them for training without repeatedly performing the time‑consuming visual encoder forward computation.2. Data Sources and Generation ProcessThis dataset is compiled from the following three open‑source datasets through systematic data cleaning and preprocessing:avspeech‑visual‑audio: an open‑source dataset containing multimodal aligned data of speech, vision, and audio, providing paired talking‑head videos and corresponding speech signals.ResponseNet: a dataset designed for video generation and dialogue response tasks, containing rich video‑text paired content.ID‑LoRA‑TalkVid: a dataset focused on identity‑preserving talking‑head video generation, including talking‑head videos from multiple identities and scenarios.The three original datasets were processed through the following pipeline to produce the final dataset:Low‑quality data filtering: removal of samples with low resolution, blurry frames, audio‑video asynchrony, or inappropriate duration (too short or too long).Format unification: conversion of videos from different sources into a unified encoding format and resolution.Latent pre‑computation: all video frames are encoded into latent representations using a pre‑trained visual encoder (e.g., VAE), generating latent files that can be directly fed into diffusion models or video generation models, significantly reducing computational overhead in subsequent training.3. File Structure and Naming ConventionsThe dataset directory structure is as follows:textAIReady/ ├── Raw_data/ │ └── train/ │ ├── avspeech‑visual‑audio_aa │ ├── avspeech‑visual‑audio_ab │ └── ... (split compressed archive parts) └── split_avatar_data/ └── train/ ├── split_avspeech‑visual‑audio_aa ├── split_avspeech‑visual‑audio_ab └── ... (split compressed archive parts) Naming conventions:DirectoryFile PrefixDescriptionAIReady/Raw_data/train/avspeech‑visual‑audio_Preprocessed latent files of the original multimodal data (speech + vision + audio), stored as split compressed archivesAIReady/split_avatar_data/train/split_avspeech‑visual‑audio_Latent files of avatar video data after human‑segmentation / matting processing, stored as split compressed archivesThe suffixes _aa, _ab, _ac, ... indicate the sequential parts of the split archives. All parts together form a complete dataset archive.4. Data Loading and Usage InstructionsThe dataset is stored in split‑compressed format (tar + split). After downloading all files, users can extract them using the following commands:Extract Raw_data:bashcd AIReady/Raw_data/train/ cat avspeech‑visual‑audio_* | tar -zxvf - -C ./ Extract split_avatar_data:bashcd AIReady/split_avatar_data/train/ cat split_avspeech‑visual‑audio_* | tar -zxvf - -C ./ After extraction, the data are latent tensor files (in .pt or .npy format) for each video sample. These can be directly loaded into deep learning frameworks such as PyTorch for video generation model training or fine‑tuning.5. Value and Application ScenariosThe core value of this dataset includes:Lowered training barriers: latent pre‑computation via a visual encoder is already done, so users do not need to set up and run the encoder themselves, saving significant computational resources and time.Multi‑source integration: high‑quality samples from three open‑source datasets are combined, providing high diversity in terms of speakers, scenes, and multimodal alignment.Ready‑to‑use format: standardised latent tensors that can be directly plugged into training pipelines of mainstream video generation models (e.g., Stable Video Diffusion, AnimateDiff, ID‑LoRA, etc.).This dataset is suitable for, but not limited to, the following research and application areas:Text‑ / speech‑driven video generation model trainingTalking‑head video synthesisIdentity‑preserving video generationMultimodal alignment and video understanding research6. Data Usage StatementThis dataset is a secondary compilation and preprocessing derived from the three open‑source datasets: avspeech‑visual‑audio, ResponseNet, and ID‑LoRA‑TalkVid. Users are requested to comply with the license terms and citation requirements of the respective original datasets when using this dataset. This compiled dataset is released solely for academic research purposes. video generation; Multimodal Learning; Latent Representation; Digital Human Ming Wang EMBARGO https://creativecommons.org/licenses/by-nc-sa/4.0/ dataset Science Data Bank