Video lab

AnimateDiff locally: memory diagnosis and an 8-shot bench

Role: Memory diagnosis, pipeline rebuild, bench and metrics
Tools: Diffusers 0.39
AnimateDiff
Lightning (4 pas)
SD 1.5
PyTorch (MPS)
Real-ESRGAN
OpenCV
CLIP
Status: Functional

Two shots from the 20 Oct 2026 bench: forest mist and night street. 16 frames at 512², 4 steps, upscaled ×2 and interpolated to 24 fps.

per 16-frame 512² shot (333 s on 19 Oct): ≈ 40 s
frames bit-identical between two runs: 16/16
faster with memory-budget attention chunking: ×4.6
bench shots generated without failure: 8/8

The problem: Producing short animated shots locally on a 16 GB Mac, reusing SD 1.5, without memory or chance deciding the outcome.
For whom: Personal research project: the portfolio’s video building block, at zero cost.
The outcome: A 16-frame shot in about 40 s (333 s the day before), bit-for-bit reproducible, and a measured 8-shot bench: speed, temporal coherence, CLIP.

Context

Reusing SD 1.5 to produce short animated shots locally, with no paid service, with a log for every run.

The 20 Oct rebuild adds tested safeguards, then measures: speed, memory, temporal coherence, text adherence, on eight cinema shots.

What I built

Attention chunked to a memory budget (exact computation, tested against full attention): 38 s versus 175 s with the old slicing, at equal settings.
Initial noise drawn on CPU and latent fingerprints after every step: two runs with the same seed give 16 bit-identical frames.
Memory guard (footprint, swap, free memory) with a clean stop, and a JSON log written for every run.
Post-production: tiled Real-ESRGAN ×2 with margins, interpolation to 24 fps, H.264 MP4; temporal-coherence metrics and CLIP score computed on CPU.

Architecture 6 blocks

01 · InputShotPrompt, preset, seed; typed configuration
02 · ModelSD 1.5 + AnimateDiffLightning 4 steps by default, guoyww 16 steps optional
03 · ToolBudgeted attentionExact computation, chunked to memory
04 · ToolMemory guardFootprint, swap, free memory; clean stop
05 · ToolPostReal-ESRGAN ×2, 24 fps, MP4 and GIF
06 · OutputMetricsSSIM, registration, flicker, sharpness, CLIP

Pipeline 5 steps

DiagnosisMemory and attention profiled on the first script (333 s per shot).
RebuildEvery memory case becomes a tested safeguard (114 tests).
ReferenceBit-for-bit reproducible, 38 s.
Bench8 Lightning shots, then 2 guoyww shots.
Post×2, 24 fps, metrics.

Decisions the choice, the reason, the alternative ruled out

D1
Chunk attention to a memory budget rather than fixed slices.
Same exact result, 38 s against 175 s at equal settings: the old slicing was the real cause of the slowness.
Ruled outSlicedAttnProcessor, used on 19 Oct.
D2
Draw noise on CPU and track latent fingerprints at every step.
Two runs with the same seed are now bit-identical; a divergence is located to the step.
D3
Keep Lightning 4 steps as default after comparing with guoyww 16 steps.
Lightning: 40 s instead of 313 s and better frame-to-frame coherence; guoyww follows the text slightly better.
D4
Write a JSON log for every run, including when a safeguard stops the computation.
Every run leaves a usable trace for diagnosis.

Results

Lightning bench, 8 shots (4 steps, CFG 1, seed 2026, 16 frames at 512²)

Shot	Generation	Frame-to-frame SSIM ↑	Motion (px/frame)	CLIP ↑
Car on asphalt	39.7 s	0.77	8.6	29.7
Night street	39.6 s	0.73	3.9	29.9
Portrait	41.9 s	0.63	3.7	28.6
Forest mist	41.2 s	0.63	1.4	31.3
Watch (product)	40.7 s	0.54	10.7	29.7
Coffee steam	42.1 s	0.49	18.5	28.2
Basalt	39.6 s	0.43	16.7	28.4
Waves	39.1 s	0.33	39.9	31.2

Frame-to-frame SSIM reads together with motion: a frozen shot is perfectly “coherent”. The first clip, before the rebuild, scored 0.15.

Lightning 4 steps vs guoyww 16 steps (CFG 7.5), same shots

Shot	Module	Generation	SSIM ↑	CLIP ↑
Basalt	Lightning	39.6 s	0.43	28.4
Basalt	guoyww	313.2 s	0.21	31.0
Car on asphalt	Lightning	39.7 s	0.77	29.7
Car on asphalt	guoyww	313.1 s	0.32	30.1

Lightning is the default: eight times faster and more coherent from frame to frame; guoyww follows the text slightly better.

Deliverables 3

labo package: configuration, generation, resumable batches, post-production, metrics, reports.
8-shot Lightning bench and guoyww comparison, with result tables generated from the files.
24 fps clips, contact sheets and logs for every attempt; 114 CPU tests.

Environment

Hardware and measurement setup

Generated on a 16 GB Mac (PyTorch MPS, Diffusers 0.39): 16-frame shots at 512², upscaled ×2 and interpolated to 24 fps.
SD 1.5 and AnimateDiff-Lightning under the CreativeML OpenRAIL-M licence.

Reproduce README commands

Private code: read access under NDA, or a 30-minute live demo. Request access →

python -m labo doctor
python -m labo batch plans/bench_lightning.json
python -m labo post --root sorties/bench_lightning --upscale 2 --interp 24
python -m labo metrics --root sorties/bench_lightning
python -m labo report sorties/bench_lightning
python -m labo compare sorties/reference/ref-a sorties/reference/ref-b