Module video_preprocess 

Video → patch tensor, the temporal sibling of crate::vision_preprocess.

What a video is, to this encoder

Qwen3-VL’s ViT has NO temporal attention. Frames fuse inside a patch: the flattened patch dimension is C × temporal_patch_size × patch² , so a patch already spans tp frames’ worth of pixels. A still image fills that axis by REPLICATING itself tp times (see preprocess_image) — the axis was always there, and a still is the degenerate case of a video.

So a video of n frames becomes grid_t = n / tp TEMPORAL GROUPS, each group a full grid_h × grid_w patch plane built from tp consecutive frames. Each group is shaped exactly like a preprocessed still, which is why the encoder needs no change at all: the groups ride the existing per-image path and only the bookkeeping downstream knows they belong to one item.

What DOES differ is position. An image holds MRoPE’s T coordinate constant across its whole pad run; a video advances T once per group. That is the reason grid_t is carried rather than groups being flattened into independent images, and it is why videos get their own pad token.

Container support

Two backends, chosen by MAGIC BYTES rather than the declared MIME:

• GIF decodes in-process, pure Rust, always available, no dependency.
• Everything else (MP4/MOV, WebM/Matroska, AVI — H.264, H.265, VP9, AV1) goes to ffmpeg as a subprocess, which is opt-in.

Sniffing the bytes rather than trusting the label means a client that sends an mp4 as video/gif, or as application/octet-stream, still gets the right decoder. See video_decode_ffmpeg for why a subprocess rather than a linked decoder, and issue #515.

Structs

PreprocessedVideo

A decoded, ready-to-encode video.

Constants

DEFAULT_FPS

Frames per second to sample at, when the caller has no better idea. Matches the fps: 2 every Qwen3-VL video_processor block declares.

DEFAULT_MAX_FRAMES

DEFAULT_MIN_FRAMES

Sampling floor and ceiling, also from the checkpoints’ own video processor (min_frames: 4, max_frames: 768). The floor matters more than it looks: with temporal_patch_size = 2, fewer than 2 frames cannot fill a single temporal group, and a 1-frame “video” would silently become a still.

Functions

decode_frames

Decode every frame of a container, choosing a backend by what the bytes actually are.

preprocess_video

Full pipeline: a base64 data: URI holding an animated container becomes temporal groups of patches.

sample_indices

Pick which frame indices to keep so the clip plays at fps.