TikTok-Style Animated Captions in Remotion with Whisper Transcription

Word-by-word animated captions — the style where each spoken word pops and changes color as it is said — have become the default visual grammar of short-form video. Scroll through TikTok, Reels, or Shorts for five minutes and nearly every talking video uses them.

The built-in caption tools in mobile editors work fine for one video at a time. But they lock you into a manual workflow: import the clip, wait for transcription, fix errors on a phone keyboard, pick from a fixed set of styles, export, repeat. At any real volume, that loop is the bottleneck.

Remotion turns the whole thing into code. Combined with a local Whisper transcription step, you get a pipeline that takes an audio file in and produces a styled, captioned vertical video out — no cloud transcription services, full control over typography, and a caption style you define once and apply to every video. This guide builds that pipeline end to end.

---

Why Word-Level Captions Boost Short-Form Retention

Most short-form video is watched with the sound off — in feeds, in public, during autoplay. Captions are not an accessibility nicety in this context; they are the primary channel for your message.

Word-level captions outperform static subtitle blocks for a few concrete reasons:

• They create constant micro-motion. A new visual event every 200–400 milliseconds gives the viewer’s eye something to track — exactly the stimulus cadence short-form feeds train people to expect.
• The karaoke highlight binds audio to visuals. With sound on, the highlighted word matches the spoken word, making the video feel tightly produced even if it is a single static shot.
• They force conciseness. Showing one to four words at a time makes pacing problems visible immediately.

Technically, word-level captions are a close cousin of kinetic typography: split text into atoms, give each atom its own timing, animate each independently. If you want to go deeper on per-word and per-character animation patterns, see our kinetic typography guide. The difference here is that the timing does not come from a stagger formula — it comes from a transcription with real word timestamps. Which is where Whisper comes in.

---

Transcribing Audio Locally with Whisper (@remotion/install-whisper-cpp)

Remotion ships an official package, @remotion/install-whisper-cpp, that downloads and runs whisper.cpp — a fast C++ port of OpenAI’s Whisper speech recognition model — entirely on your machine. No API keys, no upload, no per-minute metering, and it works offline. For a batch pipeline, local transcription also means deterministic costs: the only resource you spend is CPU time.

Install it into your Remotion project:

npx remotion add @remotion/install-whisper-cpp

Then create a Node.js script that installs whisper.cpp, downloads a model, and transcribes your audio. One important requirement: whisper.cpp expects a 16-bit, 16 kHz WAV file, so convert your source audio with FFmpeg first.

// transcribe.mjs — run with: node transcribe.mjs
import path from 'path';
import fs from 'fs';
import { execSync } from 'child_process';
import {
  installWhisperCpp,
  downloadWhisperModel,
  transcribe,
  toCaptions,
} from '@remotion/install-whisper-cpp';

const whisperPath = path.join(process.cwd(), 'whisper.cpp');

await installWhisperCpp({ to: whisperPath, version: '1.5.5' });
await downloadWhisperModel({ model: 'medium.en', folder: whisperPath });

// Whisper requires a 16-bit, 16 kHz WAV file
execSync(
  'ffmpeg -i public/voiceover.mp3 -ar 16000 public/voiceover.wav -y',
);

const whisperCppOutput = await transcribe({
  model: 'medium.en',
  whisperPath,
  whisperCppVersion: '1.5.5',
  inputPath: path.join(process.cwd(), 'public', 'voiceover.wav'),
  tokenLevelTimestamps: true,
});

// Convert whisper.cpp output into Remotion's Caption format
const { captions } = toCaptions({ whisperCppOutput });

fs.writeFileSync(
  path.join(process.cwd(), 'public', 'voiceover-captions.json'),
  JSON.stringify(captions, null, 2),
);

Two options matter most here:

• tokenLevelTimestamps: true is the whole point. It makes whisper.cpp compute accurate per-token timestamps (via its --dtw feature, available from whisper.cpp 1.5.5) — without it, token timing is too imprecise for word-by-word highlighting.
• Model choice trades speed for accuracy. base.en is fast enough for iterating on styles; medium.en is noticeably more accurate for final output. For non-English audio, use a multilingual model like medium or large-v3 (more on that below).

The toCaptions() helper post-processes whisper.cpp’s raw output into a clean array of Caption objects — the format every other caption utility in Remotion understands.

---

Parsing and Structuring Captions with @remotion/captions

The @remotion/captions package defines the shared Caption type and the utilities that operate on it:

import type { Caption } from '@remotion/captions';

type Caption = {
  text: string;
  startMs: number;
  endMs: number;
  timestampMs: number | null;
  confidence: number | null;
};

The key utility for our purposes is createTikTokStyleCaptions(). It groups a flat list of word-level captions into pages — the short groups of one to four words that appear on screen together — while preserving per-word timing as tokens inside each page. The combineTokensWithinMilliseconds option is your main creative knob: lower values produce fewer words per page (more aggressive, word-by-word pacing), higher values produce calmer multi-word pages.

Here is a component that loads the captions JSON, builds pages, and renders each page in its own <Sequence>:

import { useCallback, useEffect, useMemo, useState } from 'react';
import {
  AbsoluteFill,
  Sequence,
  staticFile,
  useDelayRender,
  useVideoConfig,
} from 'remotion';
import { createTikTokStyleCaptions } from '@remotion/captions';
import type { Caption } from '@remotion/captions';
import { CaptionPage } from './CaptionPage';

// Higher = more words per page, lower = more word-by-word
const SWITCH_CAPTIONS_EVERY_MS = 900;

export const Captions: React.FC = () => {
  const { fps } = useVideoConfig();
  const [captions, setCaptions] = useState<Caption[] | null>(null);
  const { delayRender, continueRender, cancelRender } = useDelayRender();
  const [handle] = useState(() => delayRender());

  const fetchCaptions = useCallback(async () => {
    try {
      const res = await fetch(staticFile('voiceover-captions.json'));
      const data = (await res.json()) as Caption[];
      setCaptions(data);
      continueRender(handle);
    } catch (e) {
      cancelRender(e);
    }
  }, [continueRender, cancelRender, handle]);

  useEffect(() => {
    fetchCaptions();
  }, [fetchCaptions]);

  const { pages } = useMemo(() => {
    return createTikTokStyleCaptions({
      captions: captions ?? [],
      combineTokensWithinMilliseconds: SWITCH_CAPTIONS_EVERY_MS,
    });
  }, [captions]);

  if (!captions) {
    return null;
  }

  return (
    <AbsoluteFill>
      {pages.map((page, index) => {
        const nextPage = pages[index + 1] ?? null;
        const startFrame = (page.startMs / 1000) * fps;
        const endFrame = Math.min(
          nextPage ? (nextPage.startMs / 1000) * fps : Infinity,
          startFrame + (SWITCH_CAPTIONS_EVERY_MS / 1000) * fps,
        );
        const durationInFrames = endFrame - startFrame;

        if (durationInFrames <= 0) {
          return null;
        }

        return (
          <Sequence
            key={index}
            from={startFrame}
            durationInFrames={durationInFrames}
          >
            <CaptionPage page={page} />
          </Sequence>
        );
      })}
    </AbsoluteFill>
  );
};

The useDelayRender() hook holds the render until the JSON has loaded, so no frames are captured with missing captions. Each page’s <Sequence> ends either when the next page starts or after the maximum display window, whichever comes first.

If you already have subtitles as .srt files — from an editor, a client, or a previous workflow — you can skip Whisper entirely and convert them with parseSrt() from the same package. Note that SRT files carry line-level timing, so word highlighting will be approximate unless the source was word-timed.

---

Building the Word-Highlight (Karaoke) Component

Each TikTokPage carries a tokens array, and each token has fromMs and toMs — the window during which that word is being spoken. Inside a page component, convert the current frame back into absolute milliseconds and check which token is active:

import React from 'react';
import {
  AbsoluteFill,
  spring,
  useCurrentFrame,
  useVideoConfig,
} from 'remotion';
import type { TikTokPage } from '@remotion/captions';

const HIGHLIGHT_COLOR = '#39E508';

export const CaptionPage: React.FC<{ page: TikTokPage }> = ({ page }) => {
  const frame = useCurrentFrame();
  const { fps } = useVideoConfig();

  // Frame inside this Sequence, converted to absolute milliseconds
  const absoluteTimeMs = page.startMs + (frame / fps) * 1000;

  return (
    <AbsoluteFill style={{ justifyContent: 'center', alignItems: 'center' }}>
      <div
        style={{
          fontFamily: '-apple-system, "Segoe UI", Roboto, sans-serif',
          fontSize: 72,
          fontWeight: 800,
          textAlign: 'center',
          whiteSpace: 'pre',
          color: '#ffffff',
          textShadow: '0 4px 20px rgba(0, 0, 0, 0.75)',
        }}
      >
        {page.tokens.map((token) => {
          const isActive =
            token.fromMs <= absoluteTimeMs && token.toMs > absoluteTimeMs;

          // Frames elapsed since this token became active
          const tokenStartFrame =
            ((token.fromMs - page.startMs) / 1000) * fps;
          const pop = spring({
            frame: frame - tokenStartFrame,
            fps,
            config: { mass: 0.5, stiffness: 200, damping: 14 },
          });

          return (
            <span
              key={token.fromMs}
              style={{
                display: 'inline-block',
                color: isActive ? HIGHLIGHT_COLOR : '#ffffff',
                scale: isActive ? 1 + pop * 0.15 : 1,
              }}
            >
              {token.text}
            </span>
          );
        })}
      </div>
    </AbsoluteFill>
  );
};

Three details make or break this component:

1. whiteSpace: 'pre' — token text includes its leading space (" world", not "world"). Without pre, those spaces collapse and words run together.
2. display: 'inline-block' — transforms are ignored on plain inline elements, so the scale pop would silently do nothing without it.
3. The spring is keyed to the token’s start. Passing frame - tokenStartFrame into spring() means every word gets a fresh pop the instant it becomes active, rather than one spring running from the start of the page.

This is the complete karaoke effect: the active word turns green and pops about 15% larger, then settles as the next word takes over.

---

Styling for Vertical Video: Safe Zones, Emphasis Words, and Emoji

For a 1080×1920 vertical composition, where you put the captions matters as much as how they look. Platform UI overlays eat into the frame: account name and sound title near the top, the caption box and action buttons near the bottom, and an engagement rail down the right edge. Practical safe-zone values that hold up across TikTok, Reels, and Shorts:

• Keep captions below ~220px from the top and above ~320px from the bottom
• Keep ~120px clear on the right edge for the like/comment/share rail
• The sweet spot for caption blocks is the horizontal band around 55–70% of frame height — below the subject’s face, above the platform UI

import { AbsoluteFill } from 'remotion';
import type { TikTokPage } from '@remotion/captions';
import { CaptionPage } from './CaptionPage';

const EMPHASIS_WORDS = new Set(['free', 'never', 'instantly', 'stop']);
const EMPHASIS_COLOR = '#FFD400';

export const isEmphasis = (tokenText: string): boolean => {
  const clean = tokenText.trim().toLowerCase().replace(/[^\p{L}\p{N}]/gu, '');
  return EMPHASIS_WORDS.has(clean);
};

export const SafeZoneCaptionLayer: React.FC<{ page: TikTokPage }> = ({
  page,
}) => {
  return (
    <AbsoluteFill>
      <div
        style={{
          position: 'absolute',
          left: 64,
          right: 120, // clear of the engagement rail
          top: '58%', // below the face, above the bottom UI
          display: 'flex',
          justifyContent: 'center',
        }}
      >
        <CaptionPage page={page} />
      </div>
    </AbsoluteFill>
  );
};

Emphasis words are the second signature of the TikTok caption style: certain words stay permanently colored (often yellow or red) regardless of the karaoke highlight. Maintain a set of emphasis words as a prop, normalize each token before lookup as above, and give matches a distinct color and heavier pop. Because captions are data, you can also auto-emphasize by rule — numbers, negations, superlatives.

Legibility rules of thumb at 1080px width: font size 64px or larger, weight 700+, and always a contrast device — a soft textShadow as in the component above, a solid rounded background chip, or a stroke built by layering a duplicate of the text with WebkitTextStroke behind the fill layer.

Fonts: the system font stack in the examples renders consistently in Remotion’s Chromium-based renderer with zero setup. If you want a specific brand font, self-host it — put the .woff2 file in public/ and register it with an @font-face rule or Remotion’s local font loading. Avoid runtime dependencies on third-party font CDNs in a render pipeline; a network hiccup mid-render becomes a corrupted batch.

Emoji work out of the box: Chromium renders system emoji fonts, so you can inline them in token text or append them from a keyword map ("money" → 💰). Keep them to one per page — they occupy roughly two characters of width and pull attention hard.

---

Handling Japanese and Multilingual Caption Layouts

Everything above assumes English, but the pipeline is language-agnostic with a few adjustments.

Model and language selection. The .en model variants are English-only. For Japanese or any other language, download a multilingual model (medium or large-v3 give the best results) and pass the language option to transcribe() — e.g. language: 'ja' — or pass language: 'auto' to auto-detect. (If you leave the option unset, whisper.cpp falls back to its own default, English.) Explicit is better for batch pipelines anyway, since auto-detection can misfire on short clips with music intros.

Token granularity is different. Japanese has no spaces, and Whisper emits sub-word tokens — a single word can arrive as two or three tokens, each with its own timestamp. Highlighting every token individually produces a flickery, stuttering effect. Two fixes work well:

• Lower combineTokensWithinMilliseconds to around 500–800 so each page is a short phrase, and highlight the entire page as the spoken unit instead of individual tokens.
• Or merge adjacent tokens whose gap is under ~50ms into a single display token before rendering, keeping per-token highlighting at word-ish granularity.

Layout. With no spaces, the browser will happily break a line mid-word. Keep pages short enough to fit one line — roughly 8–12 characters at 72px on a 1080px frame after padding. Since Japanese tokens carry no leading spaces, the whiteSpace: 'pre' requirement also falls away. Use a Japanese-capable font stack, again system fonts or a self-hosted @font-face:

fontFamily:
  '"Hiragino Kaku Gothic ProN", "Yu Gothic", "Noto Sans JP", sans-serif',

Kanji are visually denser than Latin glyphs, so a slightly smaller size (60–66px) with the same heavy weight usually reads better. The karaoke component itself needs no other changes — Caption timing is milliseconds either way.

---

Batch Pipeline: From an Audio Folder to Captioned Shorts

Now the payoff: turning a folder of voiceover files into a folder of captioned vertical videos with one command. Two changes make the composition batch-friendly.

First, pass captions as input props instead of fetching JSON — no useDelayRender() needed, and each render is fully self-describing. Second, use calculateMetadata to size each video’s duration from its own captions, so a 20-second voiceover produces a 21-second video automatically. (For a deeper look at dynamic duration, see the calculateMetadata guide.)

// Root.tsx
import { Composition } from 'remotion';
import type { CalculateMetadataFunction } from 'remotion';
import type { Caption } from '@remotion/captions';
import { CaptionedShort } from './CaptionedShort';

export type CaptionedShortProps = {
  captions: Caption[];
  audioFile: string;
};

const FPS = 30;

const calculateMetadata: CalculateMetadataFunction<CaptionedShortProps> = ({
  props,
}) => {
  const lastCaption = props.captions[props.captions.length - 1];
  const endMs = lastCaption ? lastCaption.endMs : 0;
  return {
    // One second of tail room after the last word
    durationInFrames: Math.ceil((endMs / 1000) * FPS) + FPS,
  };
};

export const RemotionRoot: React.FC = () => {
  return (
    <Composition
      id="CaptionedShort"
      component={CaptionedShort}
      fps={FPS}
      width={1080}
      height={1920}
      durationInFrames={300}
      defaultProps={{ captions: [], audioFile: 'voiceover.mp3' }}
      calculateMetadata={calculateMetadata}
    />
  );
};

The driver script transcribes each file, then renders with @remotion/renderer. Bundle once, render many:

// batch.mjs
import fs from 'fs';
import path from 'path';
import { execSync } from 'child_process';
import { bundle } from '@remotion/bundler';
import { renderMedia, selectComposition } from '@remotion/renderer';
import { transcribe, toCaptions } from '@remotion/install-whisper-cpp';

const whisperPath = path.join(process.cwd(), 'whisper.cpp');
const audioDir = path.join(process.cwd(), 'public', 'audio');
const files = fs.readdirSync(audioDir).filter((f) => f.endsWith('.mp3'));

const serveUrl = await bundle({
  entryPoint: path.join(process.cwd(), 'src', 'index.ts'),
});

for (const file of files) {
  const base = file.replace(/\.mp3$/, '');
  const wavPath = path.join(audioDir, `${base}.wav`);
  execSync(`ffmpeg -i "${path.join(audioDir, file)}" -ar 16000 "${wavPath}" -y`);

  const whisperCppOutput = await transcribe({
    model: 'medium.en',
    whisperPath,
    whisperCppVersion: '1.5.5',
    inputPath: wavPath,
    tokenLevelTimestamps: true,
  });
  const { captions } = toCaptions({ whisperCppOutput });

  const inputProps = { captions, audioFile: `audio/${file}` };

  const composition = await selectComposition({
    serveUrl,
    id: 'CaptionedShort',
    inputProps,
  });

  await renderMedia({
    composition,
    serveUrl,
    codec: 'h264',
    outputLocation: path.join(process.cwd(), 'out', `${base}.mp4`),
    inputProps,
  });

  console.log(`Rendered ${base}.mp4`);
}

Inside CaptionedShort, play the voiceover with <Audio src={staticFile(audioFile)} /> from @remotion/media and render the caption layer on top. Transcription is the slow step, but it happens once per file, before rendering starts, and the results can be cached as JSON alongside the audio.

From here the extensions are incremental: add a background video layer, swap codecs, or move rendering to the cloud. If you are wiring this into a full social publishing workflow, the Instagram Reels and TikTok automation guide covers format targeting and the surrounding pipeline.

---

Caption-Ready Templates in a Vertical Social Kit

If you build this pipeline from scratch, the transcription and paging logic takes an afternoon — the polish takes much longer. Production caption templates accumulate details that are invisible until they are missing: safe-zone presets per platform, emphasis-word props, stroke and chip style variants, springs tuned so page transitions never overlap, multilingual layout modes, and prop shapes designed to accept inputProps straight from a batch script.

That is the layer a template kit solves. The RenderComp library includes vertical social templates with caption systems already built in — word-highlight components, safe-zone layouts, and emphasis styling, shipped as editable TypeScript source you can drop your Whisper JSON into. Browse the collection at rendercomp.com and go from an audio folder to publishable shorts on day one.

---

FAQ

Q: Which Whisper model should I use?

base.en for iterating on caption styles — it transcribes quickly and timing accuracy is fine for layout work. Switch to medium.en for final renders; the accuracy jump is noticeable on proper nouns and fast speech. For non-English audio use the multilingual medium or large-v3.

Q: I already have SRT subtitle files. Do I still need Whisper?

No. Use parseSrt() from @remotion/captions to convert them into Caption[] and feed them into the same paging and rendering components. SRT timing is per subtitle line, not per word, so the karaoke highlight will step through lines rather than words unless the source was word-timed.

Q: Whisper misspelled a brand name. How do I fix it?

Edit the JSON. The transcription output is a reviewable, correctable artifact — a find-and-replace over the captions file, or a small dictionary-based fixup pass in your batch script, fixes recurring terms across every video at once.

Q: Do word-level captions slow down rendering?

Not meaningfully. The expensive step is transcription, which happens once before rendering. At render time the captions are static JSON, and each frame evaluates a handful of token comparisons and springs — trivial next to frame capture and encoding.

Q: My captions never show up in the rendered video, but they work in Studio. Why?

Almost always a loading race: the captions were fetched without useDelayRender(), so frames were captured before the JSON arrived. Either hold the render as shown above, or sidestep the problem by passing captions via inputProps as in the batch pipeline.

---

Summary

The complete TikTok-style caption pipeline in Remotion:

1. Transcribe audio locally with @remotion/install-whisper-cpp, using tokenLevelTimestamps: true, and convert the output with toCaptions()
2. Group words into pages with createTikTokStyleCaptions(), tuning combineTokensWithinMilliseconds for pacing
3. Render each page in a <Sequence> and highlight the active token by comparing the current time against each token’s fromMs/toMs
4. Add a token-keyed spring() pop, emphasis-word styling, and safe-zone positioning for vertical frames
5. For batch output, pass captions as inputProps, size duration with calculateMetadata, and loop selectComposition() + renderMedia() over your audio folder

---

Explore caption-ready vertical video templates at RenderComp →

The RenderComp library includes vertical social templates with word-highlight caption systems, platform safe-zone layouts, and emphasis styling built in — all editable TypeScript source, designed to accept your Whisper caption JSON as props.