Introducing SpatialVID: A Large-Scale Video Dataset with Spatial Annotations
SpatialVID is a new large-scale framework-for-long-form-video-understanding-using-self-verification-reinforcement-learning/”>temporal-prompting-matters-in-referring-video-object-segmentation-time-aware-prompts-for-more-accurate-object-localization-in-videos/”>video-segmentation-model/”>video-agentic-reasoning-and-its-implications-for-video-ai/”>exploring-vocap-enhancing-video-analysis-with-object-captioning-and-segmentation/”>video dataset designed to advance video understanding-fast-feature-field-f3-a-new-predictive-representation-of-events-and-its-implications-for-predictive-analytics/”>understanding research. It offers a wealth of spatial annotations alongside video clips, providing researchers with unprecedented opportunities for innovation.
Executive Summary
Scale and Scope: SpatialVID comprises 1.8 million video clips (approximately 3,600 hours) across 24 diverse domains, including sports, crowds, urban scenes, and nature.[1]
Rich Spatial Annotations: Each frame includes 3D pose data, 2D/3D bounding boxes, pixel-level segmentation masks, depth maps, and calibrated camera intrinsics/extrinsics.[2]
Access and Workflow: Researchers can register on the official portal for access. Licensing options include a Creative Commons–like baseline license (CC-BY 4.0 Non-Commercial) and a commercial license.[3] Data is available for download in shard packages.
Reproducible Pipelines: A public repository provides data loaders, augmentation utilities, baseline models, and a Dockerized environment for reproducibility.[4]
Benchmarks and Impact: Baseline I3D and 3D-ResNet models achieved ~72.4% top-1 accuracy. Pretraining on SpatialVID boosted this to ~76.8%. Additional metrics include a 3D pose PCKh@0.5 of ~82.3% and a depth RMSE of ~0.24m.[5]
Analytics and Trust-Building: Future plans include dashboards to monitor usage and model performance, fostering responsible use.
Data Access, Licensing, and Reproducible Pipelines
SpatialVID data is accessible through a registered portal. Licensing options include CC-BY 4.0 Non-Commercial and a commercial license. Data is organized into shards for efficient download. All publications using SpatialVID must provide proper attribution.
Data Formats, Packaging, and Versioning
The dataset prioritizes consistent data formats and packaging for streamlined research. Here’s a breakdown:
- Video Format: MP4 or AV1-encoded clips at 30 fps, audio stripped.
- Annotations: Per-frame JSON (2D/3D poses, boxes), per-frame PNG masks, per-frame 16-bit PNG depth maps.
- Camera Parameters: Intrinsics/Extrinsics in YAML/JSON.
- Packaging: Shards by domain/time window; manifest.json with metadata.
- Metadata: Domain labels, scene types, weather, lighting, sensor modality.
- Quality: QC score (0-1.0) and manually validated frames.
Reproducible Pipelines and Baselines
The spatialvid-sdk repository provides data loaders, augmentation pipelines, and evaluation scripts for various tasks (action recognition, pose estimation, depth inference). Pre-configured baseline models (I3D, S3D-G, 3D ResNet) and Docker images facilitate reproducibility.
Benchmarks and Practical Evaluation
SpatialVID offers richer spatial annotations than datasets like Kinetics-700, AVA, and Charades, enabling advanced video understanding tasks. Annotation quality indicators include average per-frame IoU for masks (~0.86), pose estimation PCKh@0.5 (~0.82), depth RMSE (~0.24m), and inter-annotator agreement (kappa) ~0.72 for keypoints.[6]
I3D baseline on SpatialVID achieved ~72.4% top-1 accuracy; pretraining boosted this to ~76.8%. Depth and pose tasks show complementary gains when fused with RGB features.
Pros, Cons, and Use Cases
Pros: Rich, multi-modal annotations; clear licensing and reproducible pipelines; scalable data packaging.
Cons: Extremely large data footprint; licensing nuances for commercial use; baseline adoption requires GPU resources.
Use Cases: Robotics, autonomous systems, AR/VR, improved action recognition, multi-modal video synthesis.
Leave a Reply