Visual Regression Testing for Remotion with BackstopJS

Remotion’s core promise is that video is a pure function of time: given the same frame number and the same props, the renderer will always produce the same pixels. That determinism is exactly what makes visual regression testing not just possible, but unusually reliable compared to testing a conventional web UI. A traditional web page renders differently across font hinting backends, OS versions, and GPU drivers. A Remotion composition, running through the same headless Chromium at the same frame, resolves to a stable pixel grid you can diff with confidence.

The gap in most Remotion workflows is that this determinism goes untested. A designer tweaks an interpolate ease curve, a merge conflict corrupts a spring stiffness value, or a dependency upgrade shifts sub-pixel antialiasing, and the breakage only surfaces in a rendered video no one watches frame-by-frame. Unit tests with Jest can catch logic errors; TypeScript catches type errors. Neither catches the frame at t=0.5 of your spring animation painting two pixels too high.

BackstopJS solves the diffing half of this problem, but it was designed around navigating URLs and screenshotting live pages. The trick is to keep BackstopJS’s comparison engine and reporting infrastructure while replacing the screenshot-of-a-URL step with renderStill(), Remotion’s programmatic single-frame renderer. The result is a suite where each “screenshot” is a lossless PNG rendered by Remotion itself, and BackstopJS handles the pixel diffing, threshold management, and HTML diff reports.

How the Pipeline Fits Together

The setup has two phases, each using a different tool:

1. renderStill() from @remotion/renderer renders specific key frames to PNG files on disk. Run it once to establish a reference baseline, then again after changes to produce the test frames.
2. BackstopJS handles frame comparison, configured with a per-scenario referenceUrl and url pointing at a thin local file server. Both endpoints serve pre-rendered PNGs wrapped in a minimal HTML shell. BackstopJS’s Puppeteer engine captures those images at exact viewport dimensions and diffs them.

The critical design choice is to never let BackstopJS drive the Remotion preview server live. The Remotion Studio at localhost:3000 introduces race conditions (fonts load asynchronously, <Video> poster frames vary, and React hydration timing is not guaranteed frame-accurate). Pre-rendering with renderStill() removes all of that.

Bootstrapping the Bundle

Before renderStill() can render anything, Remotion needs a Webpack bundle. Use bundle() from @remotion/bundler, then getCompositions() to discover composition metadata (duration, fps, width, height) without hard-coding it.

// scripts/build-bundle.ts
import { bundle } from '@remotion/bundler';
import { getCompositions } from '@remotion/renderer';
import path from 'path';

export async function buildBundle() {
  const serveUrl = await bundle({
    entryPoint: path.resolve('./src/index.ts'),
    // If you have a custom webpack config, pass webpackOverride here.
    // Avoid publicPath overrides — they break asset resolution in headless Chrome.
  });

  const compositions = await getCompositions(serveUrl, {
    inputProps: {},
    // Pass your default props if compositions require them at discovery time.
  });

  return { serveUrl, compositions };
}

bundle() returns a serveUrl. In local mode this is a file:// path pointing at the built output folder. Pass it directly to renderStill() and getCompositions(). In CI, where you may want reproducible byte-identical bundles, set the publicPath to an empty string and cache the .remotion/ cache directory between runs.

Choosing Frames Worth Testing

The naive approach (testing frame 0 and frame durationInFrames - 1) misses the most interesting visual states. Animations are usually in motion between those poles, and that motion is where regressions hide.

A practical frame selection heuristic:

• Frame 0: The initial composition state before any animation fires.
• The spring settle frame is where a spring with default parameters (damping: 10, stiffness: 100, mass: 1) reaches >0.999 of its target value. At 30 fps this is approximately frame 28; scale it to your composition’s fps with Math.round((28 / 30) * fps).
• Sequence boundary frames catch off-by-one timing errors: one frame before a <Sequence from={N}> starts and the first frame of the Sequence. Errors between from and from - 1 are invisible except at exactly these frames.
• Math.floor(durationInFrames / 2) as a midpoint catches easing direction inversions.
• Final frame: durationInFrames - 1, the resting state.

// scripts/frame-scenarios.ts
export interface FrameScenario {
  compositionId: string;
  frame: number;
  label: string;
  inputProps?: Record<string, unknown>;
}

interface CompositionMeta {
  id: string;
  durationInFrames: number;
  fps: number;
}

export function scenariosFor(
  comp: CompositionMeta,
  sequenceBoundaries: number[] = [], // pass explicit Sequence `from` values here
  inputProps: Record<string, unknown> = {}
): FrameScenario[] {
  const { id, durationInFrames, fps } = comp;

  // Spring settle at default params, scaled to fps
  const springSettle = Math.min(Math.round((28 / 30) * fps), durationInFrames - 1);

  const candidates = [
    0,
    springSettle,
    Math.floor(durationInFrames * 0.25),
    Math.floor(durationInFrames * 0.5),
    durationInFrames - 1,
    // Include frame-before and frame-of each Sequence boundary
    ...sequenceBoundaries.flatMap(f => [Math.max(0, f - 1), f]),
  ];

  // Deduplicate and sort
  const frames = [...new Set(candidates)]
    .filter(f => f >= 0 && f < durationInFrames)
    .sort((a, b) => a - b);

  return frames.map(frame => ({
    compositionId: id,
    frame,
    // Zero-pad to 4 digits so filenames sort lexicographically
    label: `${id}--f${String(frame).padStart(4, '0')}`,
    inputProps,
  }));
}

Rendering Frames with renderStill

renderStill() accepts a browserInstance that can be shared across renders, which cuts the total runtime by eliminating repeated Chrome launches. Open it once with openBrowser() and pass it through.

For CI, gl: 'swiftshader' is the most portable GPU backend. It runs in sandboxed environments like GitHub Actions runners that have no real GPU. Locally, angle gives slightly sharper text rendering. Keep these in sync between local and CI runs, or you will get false positives from font hinting differences.

// scripts/render-frames.ts
import { openBrowser, renderStill, selectComposition } from '@remotion/renderer';
import fs from 'fs';
import path from 'path';
import type { FrameScenario } from './frame-scenarios';

export async function renderFrames(
  scenarios: FrameScenario[],
  serveUrl: string,
  outputDir: string,
  glBackend: 'swiftshader' | 'angle' = 'swiftshader'
): Promise<void> {
  await fs.promises.mkdir(outputDir, { recursive: true });

  // One browser instance for all renders — saves ~400ms per frame
  const browserInstance = await openBrowser('chrome', {
    chromiumOptions: {
      gl: glBackend,
      // Do not enable disableWebSecurity unless your assets require cross-origin.
    },
  });

  try {
    for (const scenario of scenarios) {
      const composition = await selectComposition({
        serveUrl,
        id: scenario.compositionId,
        inputProps: scenario.inputProps ?? {},
      });

      const outputPath = path.join(outputDir, `${scenario.label}.png`);

      await renderStill({
        composition,
        serveUrl,
        output: outputPath,
        frame: scenario.frame,
        inputProps: scenario.inputProps ?? {},
        imageFormat: 'png',   // PNG only — JPEG compression introduces lossy artifacts
                               // that cause false positives on every diff run
        browserInstance,
        onBrowserLog: () => {}, // suppress console.log noise from the composition
      });

      console.log(`  ✓  ${scenario.label}  →  ${path.relative(process.cwd(), outputPath)}`);
    }
  } finally {
    await browserInstance.close(false);
  }
}

Run this twice: once before your change to populate frames/reference/, once after to populate frames/test/.

The Frame Server

BackstopJS expects to navigate to a URL and screenshot a page. We serve each PNG as the only content in a zero-margin HTML document. When Puppeteer sets its viewport to exactly the composition’s width × height, the screenshot is a pixel-identical copy of the PNG, with no scaling, no border, and no scrollbar.

// scripts/frame-server.ts
import http from 'http';
import fs from 'fs';
import path from 'path';

export function startFrameServer(framesDir: string, port: number): http.Server {
  const server = http.createServer((req, res) => {
    // URL format: /<label>.png — strip leading slash
    const filename = decodeURIComponent(req.url ?? '/').replace(/^\//, '');
    const filePath = path.join(framesDir, filename.endsWith('.png') ? filename : `${filename}.png`);

    if (!fs.existsSync(filePath)) {
      res.writeHead(404, { 'Content-Type': 'text/plain' });
      res.end(`Not found: ${filename}`);
      return;
    }

    const png = fs.readFileSync(filePath);
    const base64 = png.toString('base64');

    res.writeHead(200, { 'Content-Type': 'text/html; charset=utf-8' });
    res.end(`<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8">
    <style>
      /* Zero margin — the PNG must fill the viewport exactly */
      * { margin: 0; padding: 0; overflow: hidden; }
      body { background: #000; }
      img { display: block; }
    </style>
  </head>
  <body>
    <img
      id="frame"
      src="data:image/png;base64,${base64}"
      width="${getWidth(filePath)}"
      height="${getHeight(filePath)}"
    >
  </body>
</html>`);
  });

  server.listen(port);
  return server;
}

// Read PNG dimensions from the IHDR chunk (bytes 16–24) without a library
function getWidth(filePath: string): number {
  const buf = fs.readFileSync(filePath);
  return buf.readUInt32BE(16);
}
function getHeight(filePath: string): number {
  const buf = fs.readFileSync(filePath);
  return buf.readUInt32BE(20);
}

BackstopJS Configuration

BackstopJS receives a scenario per frame. The referenceUrl points at the reference server; url points at the test server. The viewport must match your composition dimensions exactly. Mismatched viewports cause Puppeteer to add scrollbars, which break the diff.

{
  "id": "remotion-regression",
  "viewports": [
    { "label": "comp-1920x1080", "width": 1920, "height": 1080 }
  ],
  "onReadyScript": "backstop_engine_scripts/onReady.js",
  "scenarios": [
    {
      "label": "TitleCard--f0000",
      "url": "http://localhost:4456/TitleCard--f0000",
      "referenceUrl": "http://localhost:4455/TitleCard--f0000",
      "misMatchThreshold": 0.15,
      "requireSameDimensions": true,
      "delay": 0
    },
    {
      "label": "TitleCard--f0028",
      "url": "http://localhost:4456/TitleCard--f0028",
      "referenceUrl": "http://localhost:4455/TitleCard--f0028",
      "misMatchThreshold": 0.15,
      "requireSameDimensions": true,
      "delay": 0
    }
  ],
  "paths": {
    "bitmaps_reference": "backstop_data/bitmaps_reference",
    "bitmaps_test": "backstop_data/bitmaps_test",
    "html_report": "backstop_data/html_report",
    "ci_report": "backstop_data/ci_report"
  },
  "engine": "puppeteer",
  "report": ["CI"],
  "engineOptions": {
    "args": ["--no-sandbox", "--disable-setuid-sandbox"]
  },
  "asyncCaptureLimit": 1,
  "asyncCompareLimit": 10
}

The onReadyScript waits for the image to fully decode before Puppeteer fires the screenshot. Without this, you occasionally catch the blank frame before the base64 image renders:

// backstop_engine_scripts/onReady.js
module.exports = async (page) => {
  await page.waitForSelector('img#frame');
  await page.evaluate(() =>
    new Promise((resolve, reject) => {
      const img = document.getElementById('frame');
      if (img.complete && img.naturalWidth > 0) return resolve();
      img.addEventListener('load', resolve);
      img.addEventListener('error', reject);
      setTimeout(reject, 8000);
    })
  );
};

Set misMatchThreshold per scenario, not globally. Text-heavy frames with sharp edges tolerate 0.05–0.15. Frames heavy with spring-driven opacity blends can go up to 0.5 without masking real regressions, because antialiasing variation between Chrome versions can shift a few percent of semi-transparent pixels.

Wiring It All Together in CI

A minimal GitHub Actions job that runs the full suite:

# .github/workflows/visual-regression.yml
name: Remotion Visual Regression

on: [pull_request]

jobs:
  visual-regression:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'

      - run: npm ci

      # Render reference frames from main branch
      - name: Render reference frames
        run: |
          git stash
          npx ts-node scripts/render-frames.ts --phase reference --gl swiftshader
          git stash pop

      # Render test frames from the PR branch
      - name: Render test frames
        run: |
          npx ts-node scripts/render-frames.ts --phase test --gl swiftshader

      # Start both servers and run BackstopJS
      - name: Run BackstopJS diff
        run: |
          npx ts-node scripts/start-servers.ts &
          sleep 2
          npx backstop test --config backstop.json
        env:
          REFERENCE_PORT: 4455
          TEST_PORT: 4456

      - name: Upload diff report
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: backstop-diff-report
          path: backstop_data/html_report/

The --gl swiftshader flag is non-negotiable for GitHub Actions. GitHub’s Ubuntu runners have no hardware GPU, and Chrome will fall back to software rendering regardless, but specifying swiftshader explicitly gives you deterministic rasterization rather than whatever fallback Chrome picks. Leaving it unspecified produces occasional one-pixel-off diffs for text with subpixel hints.

Cache the frames/reference/ directory between runs using a cache key derived from your main branch SHA. This eliminates the re-render of references on every PR and cuts CI time roughly in half for large template suites.

Wrapping Up

The payoff of this setup is that Remotion’s determinism becomes a testing guarantee. Once your reference frames are committed, any code change that shifts pixels (a tweaked spring config, a wrong interpolate output range, a changed default prop) will fail the CI job with a visual diff report showing exactly which frame broke and by how many pixels.

A few practical decisions to make upfront:

• Start with your most frequently shipped templates. Templates like those in the RenderComp catalog with data-driven text layouts benefit most, because a stray number format change in inputProps can reflow an entire title card.
• For threshold calibration, run a baseline pass with misMatchThreshold: 0 and observe natural drift between two identical renders. Anything above zero in that baseline is noise to absorb in your threshold, not ignore.
• Five to eight frames per composition is usually enough. More frames increase CI time without proportionally increasing defect detection; the failing frame is almost always at the spring settle point or a Sequence boundary, not at the midpoint.
• Re-run --phase reference deliberately after intentional visual changes, never automatically. Automating reference updates removes the human approval step that makes visual regression testing meaningful.

The grunt work is in the initial frame selection and threshold calibration. Once those are dialed in, the suite runs unattended and surfaces the class of bugs that are otherwise invisible until a client spots them in a delivered file.