Regression Testing AI-Generated Remotion Video, Frame by Frame

Remotion’s core contract is strict: given the same composition props and the same frame number, the renderer always produces the same pixels. That determinism is what makes Remotion genuinely testable. Unlike a canvas animation driven by requestAnimationFrame or a CSS transition that fires whenever the browser gets around to it, a Remotion composition is a pure function of (props, frame) → image. You can verify video output the same way you verify any pure function: by asserting on outputs for known inputs.

The problem surfaces when AI agents generate or modify Remotion composition code. LLM-generated TypeScript is syntactically valid far more often than it is semantically correct. An agent asked to “add a 20-frame fade-in to the title card” might produce code that compiles cleanly and renders without errors, but places the animation at the wrong frame offset, uses an unclamped interpolate range that produces negative opacity values before frame 0, or quietly adjusts a spring’s damping constant while refactoring surrounding code. None of these failures show up as TypeScript errors or runtime exceptions; they only surface when someone watches the rendered file.

A regression pipeline closes that gap. The key design decision is which frames to capture. Naive pipelines sample at uniform intervals (every 30 frames, or at each wall-clock second). This sounds thorough but routinely misses the failure modes that AI-generated code actually produces, because those failures cluster at animation phase boundaries, not between them.

---

Why Fixed Sampling Misses the Bugs That Matter

Consider a spring-driven element entry: a card that scales from 0 to 1 over roughly 20 frames, then holds steady for a 90-frame content window, then slides out over 15 frames. If your test captures frames 0, 30, 60, 90, and 120, you sample during the hold phase and during the slide-out, but you never sample at frame 20, where the spring overshoot peaks. An AI agent that accidentally increased stiffness from 200 to 400 will produce a visibly sharper snap that your sampled test never sees.

The same problem applies to Sequence boundaries. If an agent moves a <Sequence from={30}> to from={32} (an easy off-by-one when restructuring layout code), elements appear two frames late. A test sampling at frames 0 and 60 sees neither the absence at frame 30 nor the appearance at frame 32, so it passes cleanly while the client catches the problem in production.

The fix is to capture frames at event phase boundaries: the exact frames where animation state transitions occur. Those boundaries are knowable statically, before rendering, because Remotion compositions are code.

---

Defining Event Phases

An event phase is any frame where the animation state meaningfully changes: an element enters, a spring settles, a Sequence starts or ends, an exit begins. You can derive these mechanically from your composition structure.

// src/testing/phases.ts

export type Phase = {
  label: string;
  frame: number;
};

// Compute the frame at which a spring reaches `threshold` proximity to 1.0.
// Remotion's spring() is asymptotic — it never literally reaches 1.0,
// so "settled" means within 1% by default.
export function springSettleFrame(
  fps: number,
  config: { stiffness: number; damping: number; mass?: number },
  threshold = 0.01,
  maxFrames = 300
): number {
  const { spring } = require('remotion'); // imported at call site in real code
  for (let f = 0; f < maxFrames; f++) {
    const value = spring({ frame: f, fps, config });
    if (Math.abs(1 - value) < threshold) return f;
  }
  return maxFrames;
}

// Build a phase list for a composition with a standard entry/hold/exit structure.
export function buildPhases(options: {
  fps: number;
  durationInFrames: number;
  entrySpringConfig: { stiffness: number; damping: number };
  contentFrom: number;      // frame where content Sequence starts
  contentDuration: number;  // in frames
  exitFrom: number;         // frame where exit animation starts
}): Phase[] {
  const settle = springSettleFrame(options.fps, options.entrySpringConfig);
  return [
    { label: 'first-frame', frame: 0 },
    { label: 'entry-midpoint', frame: Math.floor(settle / 2) },
    { label: 'entry-settled', frame: settle },
    { label: 'content-start', frame: options.contentFrom },
    { label: 'content-end', frame: options.contentFrom + options.contentDuration - 1 },
    { label: 'exit-start', frame: options.exitFrom },
    { label: 'last-frame', frame: options.durationInFrames - 1 },
  ];
}

The springSettleFrame utility is worth examining. With a stiffness of 200 and damping of 26 at 30 fps, settlement (within 1%) lands around frame 18. With stiffness 400 and the same damping, it drops to roughly frame 12. An agent that doubles stiffness to “make things snappier” changes the settlement frame, and any assertion anchored to frame 18 now runs during a different part of the animation than intended. This is exactly the divergence you want the pipeline to catch.

---

The Render Harness

The harness bundles the project once, then renders stills at each phase frame. Bundling is the expensive step; rendering individual frames via renderStill is fast, typically under 200 ms per frame.

// src/testing/harness.ts
import path from 'path';
import fs from 'fs/promises';
import { bundle } from '@remotion/bundler';
import { getCompositions, renderStill, selectComposition } from '@remotion/renderer';
import { buildPhases, type Phase } from './phases';

const ENTRY_POINT = path.resolve(__dirname, '../../src/index.ts');
const OUT_DIR = path.resolve(__dirname, '../../test-frames');

async function runHarness(compositionId: string, phases: Phase[]): Promise<void> {
  // Bundle once; reuse the serve URL across all renderStill calls.
  const serveUrl = await bundle({
    entryPoint: ENTRY_POINT,
    // Pass through your existing webpack config if you have overrides.
    webpackOverride: (config) => config,
  });

  const comp = await selectComposition({
    serveUrl,
    id: compositionId,
    // These must match what the composition expects at runtime.
    inputProps: { title: 'Regression Test', accentColor: '#0ea5e9' },
  });

  await fs.mkdir(OUT_DIR, { recursive: true });

  for (const { label, frame } of phases) {
    const output = path.join(OUT_DIR, `${compositionId}--${label}--f${frame}.png`);
    await renderStill({
      composition: comp,
      serveUrl,
      output,
      frame,
      // Use 'png' for lossless comparison. JPEG introduces encoding artifacts
      // that make pixel diffing unreliable at tight thresholds.
      imageFormat: 'png',
      // Chromium scale factor — keep this at 1 for consistent pixel counts.
      scale: 1,
    });
    console.log(`rendered ${label} (frame ${frame}) → ${path.basename(output)}`);
  }
}

One subtlety: inputProps must be stable between runs. If an AI agent modifies the composition’s props interface and your test still passes the old shape, Remotion will silently use default values, and your golden images will have been generated with the old defaults, making the comparison meaningless. Pin inputProps in a dedicated fixture file and import it in both the harness and your composition’s defaultProps.

// src/testing/fixtures.ts
import type { MyCompositionProps } from '../compositions/MyComposition';

export const regressionProps: MyCompositionProps = {
  title: 'Regression Test',
  accentColor: '#0ea5e9',
  // Every prop the composition accepts must be explicitly set here.
  // If an agent adds a new optional prop with a conditional branch,
  // the default (undefined) state becomes part of your regression baseline.
  showBadge: false,
};

---

Asserting Animation Invariants

Beyond pixel comparison, you can assert structural invariants directly against the composition’s animation math. These catch a class of bugs faster than pixel diffs because they don’t require a golden image; they describe logical expectations.

// src/testing/invariants.ts
import { spring, interpolate } from 'remotion';

// A composition that fades in over 20 frames must not be visible at frame 0.
export function assertEntryInvisibleAtOrigin(fps: number) {
  const opacity = interpolate(0, [0, 20], [0, 1], {
    extrapolateLeft: 'clamp',
    extrapolateRight: 'clamp',
  });
  if (opacity !== 0) {
    throw new Error(`Expected opacity 0 at frame 0, got ${opacity}`);
  }
}

// Verify that the spring reaches 99% settlement within the expected window.
// If an agent changes damping, this assertion fails before you even render.
export function assertSpringSettles(
  fps: number,
  config: { stiffness: number; damping: number },
  byFrame: number
) {
  const value = spring({ frame: byFrame, fps, config });
  if (value < 0.99) {
    throw new Error(
      `Spring not settled at frame ${byFrame}: got ${value.toFixed(4)}. ` +
      `Check stiffness/damping config — agent may have modified it.`
    );
  }
}

// Sequences must start exactly where specified.
// An off-by-one in `from` is a common AI agent error.
export function assertSequenceStartVisible(
  fps: number,
  sequenceFrom: number,
  entrySpringConfig: { stiffness: number; damping: number }
) {
  // At the first frame of a Sequence, the local frame seen by children is 0.
  const localFrame = 0;
  const scale = spring({ frame: localFrame, fps, config: entrySpringConfig });
  // scale at local frame 0 is always near 0 — the point is that the sequence
  // IS active at sequenceFrom, meaning children receive frame 0, not -1.
  // Structural assertion: sequenceFrom must be a positive integer.
  if (!Number.isInteger(sequenceFrom) || sequenceFrom < 0) {
    throw new Error(`Sequence from must be a non-negative integer, got ${sequenceFrom}`);
  }
}

The interpolate Clamping Pitfall

This is the single most common semantic error in AI-generated Remotion code. interpolate without explicit clamp options uses 'extend' for both extrapolation directions by default, which means it produces values outside the output range when frame falls outside the input range.

// What an AI agent often generates — compiles, renders, looks wrong:
const opacity = interpolate(frame, [10, 30], [0, 1]);
// At frame 0: opacity = interpolate(0, [10, 30], [0, 1]) = -0.5  ← negative!
// At frame 50: opacity = 2.0  ← above 1, element "over-opaque" relative to stack

// What it should be:
const opacity = interpolate(frame, [10, 30], [0, 1], {
  extrapolateLeft: 'clamp',   // clamp to 0 for frame < 10
  extrapolateRight: 'clamp',  // clamp to 1 for frame > 30
});

Add a static analysis step to your CI that greps for bare interpolate( calls not followed by an options object. This catches the omission before you even bundle:

# In your CI script, before running the render harness:
BARE_INTERPOLATE=$(grep -rn "interpolate(" src/compositions/ \
  | grep -v "extrapolateLeft" \
  | grep -v "// verified-no-clamp-needed")

if [ -n "$BARE_INTERPOLATE" ]; then
  echo "ERROR: unclamped interpolate calls found:"
  echo "$BARE_INTERPOLATE"
  exit 1
fi

The // verified-no-clamp-needed escape hatch lets you intentionally use extending behavior (for perspective transforms, for example) without triggering the check.

---

Pixel Diffing Against Golden Images

Once you have rendered PNGs at each phase frame, compare them against stored reference images using pixelmatch:

// src/testing/diff.ts
import fs from 'fs/promises';
import path from 'path';
import { PNG } from 'pngjs';
import pixelmatch from 'pixelmatch';

const GOLDEN_DIR = path.resolve(__dirname, '../../test-golden');
const DIFF_DIR = path.resolve(__dirname, '../../test-diffs');

export async function diffFrame(
  actualPath: string,
  label: string
): Promise<{ diffPixels: number; total: number }> {
  const goldenPath = path.join(GOLDEN_DIR, path.basename(actualPath));

  const [actualData, goldenData] = await Promise.all([
    fs.readFile(actualPath),
    fs.readFile(goldenPath),
  ]);

  const actual = PNG.sync.read(actualData);
  const golden = PNG.sync.read(goldenData);

  if (actual.width !== golden.width || actual.height !== golden.height) {
    throw new Error(
      `Dimension mismatch for ${label}: ` +
      `actual ${actual.width}×${actual.height}, ` +
      `golden ${golden.width}×${golden.height}`
    );
  }

  const diff = new PNG({ width: golden.width, height: golden.height });

  const diffPixels = pixelmatch(
    golden.data,
    actual.data,
    diff.data,
    golden.width,
    golden.height,
    {
      // 0.1 is tight but necessary — Remotion renders are pixel-deterministic
      // on the same Chromium version. Loosen to 0.15 only if CI uses a
      // different OS than where goldens were generated (font rendering varies).
      threshold: 0.1,
      includeAA: false, // anti-aliasing pixels do not count as differences
    }
  );

  if (diffPixels > 0) {
    await fs.mkdir(DIFF_DIR, { recursive: true });
    const diffPath = path.join(DIFF_DIR, path.basename(actualPath));
    await fs.writeFile(diffPath, PNG.sync.write(diff));
  }

  return { diffPixels, total: golden.width * golden.height };
}

A zero-threshold comparison is too brittle for cross-platform CI, since macOS and Linux Chromium render subpixel text slightly differently. A threshold of 0.1 (10% color distance per channel) catches real layout and timing regressions while tolerating the subpixel variance you see between an M-series Mac and a GitHub Actions Ubuntu runner.

When an agent’s change legitimately improves the composition and you want to accept the new output as the new baseline, run:

# Update goldens for a specific composition only — never mass-update.
cp test-frames/HeroBanner--*.png test-golden/
git add test-golden/
git commit -m "update golden frames: HeroBanner after contrast fix"

Mass-updating all goldens without reviewing the diffs is how regressions sneak through. The mandatory per-composition update forces a human to look at the diff image before accepting it.

---

Wiring It Into CI

A GitHub Actions job that runs the full pipeline:

# .github/workflows/video-regression.yml
name: Video Regression

on:
  pull_request:
    paths:
      - 'src/compositions/**'
      - 'src/testing/**'

jobs:
  regression:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'

      - run: npm ci

      # Cache the Remotion bundle across runs — bundle output is deterministic
      # for the same source files, so a content hash cache key is safe.
      - uses: actions/cache@v4
        with:
          path: .remotion-bundle-cache
          key: remotion-bundle-${{ hashFiles('src/**') }}

      - name: Run static interpolate check
        run: |
          BARE=$(grep -rn "interpolate(" src/compositions/ | grep -v "extrapolateLeft" | grep -v "verified-no-clamp-needed" || true)
          if [ -n "$BARE" ]; then echo "$BARE"; exit 1; fi

      - name: Render phase frames
        run: npx ts-node src/testing/run-harness.ts

      - name: Diff against golden
        run: npx ts-node src/testing/run-diff.ts

      - name: Upload diffs on failure
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: regression-diffs
          path: test-diffs/
          retention-days: 7

The paths filter ensures the regression job only runs when composition code actually changes, not on doc edits or config tweaks.

---

Wrapping Up

The pipeline described here works because it leans into what Remotion already gives you: determinism. Every frame is computable, every animation value is inspectable as plain TypeScript before rendering, and every rendered PNG is bit-stable given the same Chromium version and scale factor.

The practical rules that come out of this:

• Capture frames at event phase boundaries, not at fixed intervals. Springs, Sequence offsets, and interpolate ranges all produce bugs that live between uniform sample points.
• Check spring settlement before rendering. A changed stiffness or damping shifts your entire timeline, and catching it statically is faster than catching it in a pixel diff.
• Lint for unclamped interpolate calls in CI, since this is the most common AI-agent error and catching it statically avoids burning render time.
• Update goldens per-composition with a human diff review first. Mass-accepting new baselines undoes the whole pipeline.

If you’re starting from existing production compositions rather than greenfield code (including templates like those in the RenderComp catalog), the buildPhases utility adapts cleanly to any composition that exposes its durationInFrames and spring configs as known constants. Start with the three most critical phases (first frame, spring-settled, last frame), add the others as your agent-generated code gets more complex.