Catching Remotion Render Regressions with SSIM and Pixel-Diff

Remotion’s fundamental promise is that a composition is a pure function: given a frame number and a set of input props, it produces exactly one image. This determinism is what makes programmatic video attractive — but it also means that regressions are silent and invisible. A refactored spring() call with a slightly different stiffness value, a CSS transform that rounds to the wrong pixel, a Sequence offset shifted by one frame — none of these throw errors. The video still renders. It just looks wrong.

Unit tests and TypeScript types validate logic, not pixels. A component can pass every type check while producing a layout that is visually broken at frame 45. The only way to catch this class of regression is to render actual frames and compare them against a known-good baseline. That means pixel-level comparison tooling, and two methods dominate production usage: exact pixel-diff (via pixelmatch) and perceptual similarity scoring (via SSIM). They solve different problems and belong in the same pipeline.

This article walks through building a regression harness from scratch using @remotion/renderer, pixelmatch, and ssim-js. You’ll see how to pick probe frames strategically, how to set thresholds that catch real regressions without crying wolf on anti-aliasing noise, and how to wire the whole thing into a GitHub Actions workflow that fails loudly on visual drift.

---

Why renderStill Instead of Extracting from a Rendered Video

Before touching comparison logic, get the extraction step right. The instinct is to render the full video and then extract frames with ffmpeg. Resist it. Codec compression is a second variable: H.264 in-loop deblocking, B-frame ordering, and chroma subsampling all modify pixel values in ways that shift between ffmpeg versions. Your test fails not because the composition changed, but because CI runs a different ffmpeg build than your laptop.

@remotion/renderer exposes renderStill, which renders a single composition frame directly to a PNG, bypassing any codec. PNG is lossless. The pixel values you write into your golden directory are exactly what the Remotion canvas produced — no intermediate encoding.

// scripts/capture-goldens.ts
import { bundle } from '@remotion/bundler';
import { renderStill, selectComposition } from '@remotion/renderer';
import path from 'path';
import fs from 'fs';

const BUNDLE_CACHE = path.join(__dirname, '../.remotion-cache');
const GOLDEN_DIR = path.join(__dirname, '../test/goldens');

// Probe frames: capture at transition points, not uniformly.
// For a 90-frame composition at 30fps:
//   frame 0  → initial state (nothing animated yet)
//   frame 15 → mid-spring (highest velocity, most drift risk)
//   frame 29 → settled state (spring should be at rest)
//   frame 60 → mid-sequence transition
const PROBE_FRAMES = [0, 15, 29, 60];

async function captureGoldens(compositionId: string) {
  const serveUrl = await bundle({
    entryPoint: path.join(__dirname, '../src/index.ts'),
    webpackOverride: (config) => config,
    // Re-use the webpack bundle between runs to avoid rebuilding on every CI job.
    cacheEnabled: true,
    cachePath: BUNDLE_CACHE,
  });

  const composition = await selectComposition({
    serveUrl,
    id: compositionId,
    inputProps: {},
  });

  fs.mkdirSync(path.join(GOLDEN_DIR, compositionId), { recursive: true });

  for (const frame of PROBE_FRAMES) {
    const outputPath = path.join(GOLDEN_DIR, compositionId, `frame-${frame}.png`);
    await renderStill({
      composition,
      serveUrl,
      frame,
      output: outputPath,
      inputProps: {},
      // Chromium flags that matter for determinism:
      // --disable-lcd-text prevents subpixel anti-aliasing from varying by screen density.
      // --force-device-scale-factor=1 prevents HiDPI scaling from changing pixel counts.
      chromiumOptions: {
        disableWebSecurity: false,
        headless: true,
      },
      scale: 1,
    });
    console.log(`Captured golden: ${outputPath}`);
  }
}

captureGoldens('MyComposition').catch(console.error);

Commit the PNG files into your repository. They are small (a 1280×720 frame is typically 200–400 KB as PNG), and having them in version control means you can see exactly which frame changed and when in your git history.

---

Pixel-Diff: Fast, Exact, and Intentionally Fragile

pixelmatch compares two images pixel-by-pixel and returns the count of pixels that differ beyond a configurable color-distance threshold. It is fast (pure JavaScript, no native dependencies) and produces a diff image that makes regressions visually obvious. Its fragility is a feature for certain use cases: if you want a test that fails when even one pixel moves, pixel-diff is the right tool.

// test/visual-regression.test.ts
import pixelmatch from 'pixelmatch';
import { PNG } from 'pngjs';
import fs from 'fs';
import path from 'path';

const GOLDEN_DIR = path.join(__dirname, 'goldens');
const ACTUAL_DIR = path.join(__dirname, 'actuals');
const DIFF_DIR = path.join(__dirname, 'diffs');

function loadPNG(filePath: string): PNG {
  return PNG.sync.read(fs.readFileSync(filePath));
}

function compareFramePixelDiff(
  compositionId: string,
  frame: number,
  // threshold: per-pixel color distance tolerance, 0–1.
  // 0.0 = exact match required; 0.1 = allows ~10% color-channel deviation.
  // For Remotion canvas output, 0.05 handles sub-pixel anti-aliasing on text
  // while still catching layout shifts of ≥1px.
  threshold = 0.05,
  // maxDiffPixels: absolute pixel budget. A 1280×720 frame has 921,600 pixels.
  // 500 allows for minor font-hint variation without letting a broken animation through.
  maxDiffPixels = 500,
): void {
  const goldenPath = path.join(GOLDEN_DIR, compositionId, `frame-${frame}.png`);
  const actualPath = path.join(ACTUAL_DIR, compositionId, `frame-${frame}.png`);

  const golden = loadPNG(goldenPath);
  const actual = loadPNG(actualPath);

  if (golden.width !== actual.width || golden.height !== actual.height) {
    throw new Error(
      `Dimension mismatch at frame ${frame}: ` +
      `golden is ${golden.width}×${golden.height}, ` +
      `actual is ${actual.width}×${actual.height}`
    );
  }

  const { width, height } = golden;
  const diff = new PNG({ width, height });

  const diffPixels = pixelmatch(
    golden.data,
    actual.data,
    diff.data,
    width,
    height,
    {
      threshold,
      // includeAA: true counts anti-aliased pixels as differences.
      // Set to false so that text rendering variation between Chromium versions
      // does not flood the diff count with false positives.
      includeAA: false,
    }
  );

  if (diffPixels > maxDiffPixels) {
    // Write the diff image before throwing so CI artifacts show what changed.
    fs.mkdirSync(path.join(DIFF_DIR, compositionId), { recursive: true });
    const diffPath = path.join(DIFF_DIR, compositionId, `frame-${frame}-diff.png`);
    fs.writeFileSync(diffPath, PNG.sync.write(diff));

    throw new Error(
      `Frame ${frame} exceeded pixel budget: ${diffPixels} pixels differ ` +
      `(max allowed: ${maxDiffPixels}). Diff written to ${diffPath}`
    );
  }
}

describe('MyComposition visual regression', () => {
  const PROBE_FRAMES = [0, 15, 29, 60];

  for (const frame of PROBE_FRAMES) {
    it(`frame ${frame} matches golden`, () => {
      compareFramePixelDiff('MyComposition', frame);
    });
  }
});

The per-pixel threshold of 0.05 is not arbitrary. pixelmatch uses the YIQ color space internally and measures color distance as a fraction of the maximum possible distance. At 0.05, a pixel that shifts by roughly 12 units on any single channel (out of 255) will be flagged. That is small enough to catch a spring whose damping was changed from 100 to 95 — the settled position is the same, but the overshoot at frame 15 differs by several pixels.

---

SSIM: Perceptual Similarity for Resilient Baselines

Pixel-diff is brittle when the change is expected and harmless: upgrading Chromium, changing a font weight, or enabling GPU rasterization. All of these shift individual pixel values without changing what the viewer sees. SSIM (Structural Similarity Index Measure) measures luminance, contrast, and structure similarity across local 8×8 windows, producing a score between 0 and 1 where 1 is perceptually identical.

SSIM handles anti-aliasing variation gracefully because a single pixel shift inside an 8×8 window barely moves the local structure score. A broken animation — a Sequence that fires 10 frames late, a gradient that renders at the wrong opacity — moves SSIM scores dramatically even when it looks subtle to the eye.

// test/ssim-compare.ts
import ssim from 'ssim-js';
import sharp from 'sharp';
import fs from 'fs';
import path from 'path';

interface ImageData {
  data: Uint8ClampedArray;
  width: number;
  height: number;
  channels: 3 | 4;
}

async function loadAsImageData(filePath: string): Promise<ImageData> {
  // Use sharp to normalize: convert to raw RGBA regardless of source format.
  const { data, info } = await sharp(filePath)
    .ensureAlpha()     // guarantee 4 channels (RGBA) for consistent comparison
    .raw()
    .toBuffer({ resolveWithObject: true });

  return {
    data: new Uint8ClampedArray(data.buffer),
    width: info.width,
    height: info.height,
    channels: 4,
  };
}

async function compareFrameSSIM(
  goldenPath: string,
  actualPath: string,
  // SSIM threshold: 0.98 is a practical production floor.
  // 0.98–1.00: imperceptible or no change.
  // 0.95–0.98: minor visual difference (subpixel text, slight gradient shift).
  // Below 0.95: likely a real regression — layout broken, animation phase wrong.
  minScore = 0.98,
): Promise<void> {
  const [golden, actual] = await Promise.all([
    loadAsImageData(goldenPath),
    loadAsImageData(actualPath),
  ]);

  // ssim-js computes MSSIM (mean SSIM over all windows).
  // windowSize defaults to 8; increasing to 11 matches the original paper
  // but adds negligible sensitivity improvement for video frames.
  const result = ssim(golden, actual, { windowSize: 8, k1: 0.01, k2: 0.03 });

  if (result.mssim < minScore) {
    throw new Error(
      `SSIM score ${result.mssim.toFixed(4)} below threshold ${minScore}. ` +
      `Golden: ${goldenPath} — Actual: ${actualPath}`
    );
  }
}

export { compareFrameSSIM };

The k1 and k2 constants (0.01 and 0.03) are the standard values from Wang et al. (2004). Do not tune them. They stabilize the denominator to avoid division by zero in flat regions; changing them does not meaningfully improve sensitivity for typical video content, and it makes your results incomparable to published benchmarks.

Combining Both Methods

The strongest harness runs both checks. Pixel-diff acts as the tight guardrail on frames where exact fidelity matters (the first and last frame of an animation, for example). SSIM acts as the broad perceptual check on frames with complex motion where individual pixel variance is expected:

// test/visual-regression.test.ts (combined)
import { compareFrameSSIM } from './ssim-compare';

describe('MyComposition — combined regression', () => {
  // Strict pixel-diff for boundary frames where nothing should move.
  it('frame 0 is pixel-exact', () => {
    compareFramePixelDiff('MyComposition', 0, 0.0, 0);
  });

  it('frame 89 is pixel-exact', () => {
    compareFramePixelDiff('MyComposition', 89, 0.0, 0);
  });

  // SSIM for mid-animation frames where spring physics introduces acceptable variance.
  it('frame 15 is perceptually identical', async () => {
    await compareFrameSSIM(
      path.join(GOLDEN_DIR, 'MyComposition/frame-15.png'),
      path.join(ACTUAL_DIR, 'MyComposition/frame-15.png'),
      0.97, // slightly relaxed: mid-spring peak has more rendering variance
    );
  });

  it('frame 29 is perceptually identical', async () => {
    await compareFrameSSIM(
      path.join(GOLDEN_DIR, 'MyComposition/frame-29.png'),
      path.join(ACTUAL_DIR, 'MyComposition/frame-29.png'),
      0.98,
    );
  });
});

---

Probe Frame Selection Strategy

Uniform sampling (every 10 frames) wastes time and misses the moments where regressions actually occur. Remotion compositions have structural seams: Sequence entry points, spring peaks, and interpolate clamped regions. These are the frames to probe.

For a composition driven by spring({ frame, fps: 30, config: { stiffness: 80, damping: 15 } }):

import { spring } from 'remotion';

// Find the frame where spring velocity peaks (first derivative maximum).
// For stiffness=80, damping=15, fps=30: peak is around frame 8-10.
// Settling (value within 0.001 of 1.0) happens around frame 40-50.
// Test at: first frame, velocity peak, settling frame, final frame.

function findSpringPeak(stiffness: number, damping: number, fps: number): number {
  let maxVelocity = 0;
  let peakFrame = 0;
  let prev = 0;

  for (let f = 0; f < 120; f++) {
    const value = spring({ frame: f, fps, config: { stiffness, damping } });
    const velocity = Math.abs(value - prev);
    if (velocity > maxVelocity) {
      maxVelocity = velocity;
      peakFrame = f;
    }
    prev = value;
  }
  return peakFrame;
}

// findSpringPeak(80, 15, 30) → 9
// findSpringPeak(200, 20, 30) → 4
// Use these frame numbers in your PROBE_FRAMES array.

This calculation is cheap enough to run at golden-capture time and embed the frame numbers directly in your test configuration.

---

CI Integration

Render the actual frames during CI using the same renderStill script, then compare against the committed goldens:

# .github/workflows/visual-regression.yml
name: Visual Regression

on: [pull_request]

jobs:
  regression:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'

      - run: npm ci

      # Restore the webpack bundle cache. Bundling is the slowest part (~30s).
      # The cache key includes package-lock.json so it invalidates on dependency changes.
      - uses: actions/cache@v4
        with:
          path: .remotion-cache
          key: remotion-bundle-${{ hashFiles('package-lock.json') }}
          restore-keys: remotion-bundle-

      # Chromium requires these system libraries on Ubuntu.
      - name: Install Chromium dependencies
        run: npx puppeteer browsers install chrome

      - name: Render actual frames
        run: npx ts-node scripts/render-actuals.ts

      - name: Run visual regression tests
        run: npm test -- --testPathPattern=visual-regression

      # Upload diff images as artifacts so reviewers can see exactly what changed.
      - uses: actions/upload-artifact@v4
        if: failure()
        with:
          name: visual-regression-diffs
          path: test/diffs/
          retention-days: 7

The render-actuals.ts script mirrors capture-goldens.ts but writes to test/actuals/ instead of test/goldens/. The goldens are never regenerated in CI — only locally, after a deliberate visual review, with npm run update-goldens.

---

Edge Cases That Will Surprise You

Font rendering. Chromium’s text rendering varies between operating systems. If your composition renders text, the golden frames captured on macOS will fail SSIM checks on Linux CI. Solve this by running Chromium with --disable-font-subpixel-positioning and --disable-lcd-text (both settable via chromiumOptions.additionalArgs in renderStill), which forces grayscale anti-aliasing. It is slightly less sharp than production rendering but consistent across platforms.

Math.random() in compositions. If any component calls Math.random() outside of a useMemo seeded by useCurrentFrame, frames are non-deterministic and no comparison method will work. Replace with a seeded PRNG keyed on the frame number. mulberry32 is 5 lines and produces uniform output: const rand = mulberry32(frame * 2654435761); rand();.

interpolate output range clamping. interpolate([0, 30], [0, 1]) clamps by default. If you refactor and accidentally write interpolate([0, 30], [0, 1], { extrapolateRight: 'extend' }), every frame past 30 changes. SSIM on frame 60 will catch this; a test that only probes frames 0–30 will miss it entirely. Always include at least one probe frame from the settled region past the last keyframe.

Scale factor and output dimensions. renderStill has a scale option that multiplies the composition dimensions. A scale of 2 on a 1280×720 composition produces a 2560×1440 PNG. If you change scale between golden capture and actual render, the dimension mismatch check at the top of compareFramePixelDiff will fire immediately — which is exactly the right failure mode. Hardcode scale: 1 in both scripts and never pass it via environment variable.

---

Wrapping Up

The practical setup that works in production:

• Use renderStill from @remotion/renderer for golden capture and actual rendering — never extract frames from encoded video.
• Probe at structurally meaningful frames: initial state, spring velocity peak, settled state, transition points. Five to eight frames per composition is enough.
• Run pixelmatch with threshold: 0.05 and a small absolute pixel budget on boundary frames where nothing should move. Zero tolerance on the first and last frame.
• Run SSIM with a floor of 0.98 on mid-animation frames where per-pixel variance is expected. Drop to 0.97 for frames at the spring peak.
• Write diff PNGs to a CI artifact directory unconditionally on failure — the diff image is far more actionable than a pixel count in a log line.
• Disable LCD text and subpixel font positioning in Chromium flags to neutralize cross-platform font rendering differences.

This harness adds roughly 20–60 seconds to a CI run depending on composition complexity and concurrency. That cost is small compared to the alternative: a broken animation reaching a client, invisible to every automated check that ran before it. The same approach scales naturally to larger template libraries — including ones like those in the RenderComp catalog — where dozens of compositions need to stay visually stable across Remotion version upgrades.