Creating Lyric Videos and Music Videos with Remotion
Creating Lyric Videos and Music Videos with Remotion
Lyric videos are one of the most practical application areas for programmatic video. The content is structured data — timestamped words and lines — and the visual treatment follows repeatable patterns: text appears in sync with the audio, words highlight one by one, and the background responds to the music’s energy. This is exactly the kind of problem that Remotion solves better than any design tool. Instead of keyframing every word by hand in After Effects, you define the logic once and let the data drive the render.
This guide covers the full production pipeline: structuring your lyrics data, syncing text to audio timestamps, animating word-by-word highlights, generating background visuals programmatically, adding a background music track, and batch-rendering across an entire album.
The Lyrics Data Model
The foundation of any synced lyric video is a well-structured data file. A JSON array of lyric events is the standard approach:
[
{ "start": 0.0, "end": 2.4, "text": "We are the ones who dream" },
{ "start": 2.4, "end": 4.8, "text": "Beyond the fading light" },
{ "start": 4.8, "end": 7.2, "text": "We carry what remains" },
{ "start": 7.2, "end": 9.6, "text": "Into the endless night" }
]
Each entry has a start and end time in seconds, and a text string. For word-level karaoke highlighting you need a more granular format:
[
{
"start": 0.0,
"end": 2.4,
"words": [
{ "start": 0.0, "end": 0.4, "text": "We" },
{ "start": 0.4, "end": 0.7, "text": "are" },
{ "start": 0.7, "end": 0.9, "text": "the" },
{ "start": 0.9, "end": 1.2, "text": "ones" },
{ "start": 1.2, "end": 1.5, "text": "who" },
{ "start": 1.5, "end": 2.4, "text": "dream" }
]
}
]
Word-level timestamps can be generated automatically using forced alignment tools like WhisperX, Gentle, or even the OpenAI Whisper API with word-level timestamps enabled. For a track where the artist controls the data, the JSON can be authored manually in a few minutes per song.
Converting Timestamps to Frames
Remotion’s timeline is measured in frames, not seconds. The conversion is straightforward:
const timestampToFrame = (seconds: number, fps: number): number => {
return Math.round(seconds * fps);
};
You will use this constantly. Here is the pattern for wrapping a lyric line inside a <Sequence>:
import { Sequence, useVideoConfig } from "remotion";
const { fps } = useVideoConfig();
{lyrics.map((lyric, i) => (
<Sequence
key={i}
from={timestampToFrame(lyric.start, fps)}
durationInFrames={timestampToFrame(lyric.end - lyric.start, fps)}
>
<LyricLine lyric={lyric} />
</Sequence>
))}
The <Sequence> component handles all the visibility logic: the child only renders when the current frame falls within the from to from + durationInFrames range. This means LyricLine does not need any useCurrentFrame() logic for its appearance and disappearance — Sequence handles that automatically.
Word-by-Word Highlight Animation
Inside a LyricLine component, you iterate over the words and use another set of <Sequence> elements to drive the highlight timing:
import { Sequence, useCurrentFrame, interpolate } from "remotion";
export const LyricLine: React.FC<{ lyric: LyricEntry }> = ({ lyric }) => {
const frame = useCurrentFrame(); // frame 0 = start of this Sequence
return (
<div
style={{
display: "flex",
flexWrap: "wrap",
gap: 8,
justifyContent: "center",
fontFamily: '-apple-system, "Segoe UI", Roboto, sans-serif',
}}
>
{lyric.words.map((word, i) => {
const wordStart = timestampToFrame(word.start - lyric.start, fps);
const wordEnd = timestampToFrame(word.end - lyric.start, fps);
const isActive = frame >= wordStart;
const highlight = interpolate(
frame,
[wordStart, wordStart + 4],
[0, 1],
{ extrapolateLeft: "clamp", extrapolateRight: "clamp" }
);
return (
<span
key={i}
style={{
fontSize: 56,
fontWeight: 700,
color: isActive
? `rgba(255, 220, 50, ${0.4 + highlight * 0.6})`
: "rgba(255, 255, 255, 0.5)",
textShadow: isActive
? `0 0 ${20 * highlight}px rgba(255, 220, 50, 0.8)`
: "none",
transition: "none",
lineHeight: 1.3,
}}
>
{word.text}
</span>
);
})}
</div>
);
};
The key insight here is that useCurrentFrame() inside a child of <Sequence> returns the frame relative to the start of that Sequence. So when LyricLine is rendered inside a <Sequence from={60}>, useCurrentFrame() returns 0 at frame 60 of the composition. This makes all the per-word timing arithmetic relative and composable.
Line Entrance and Exit Animations
Beyond word highlighting, lyric lines benefit from smooth entrance and exit transitions. The standard approach:
export const LyricLine: React.FC<{ lyric: LyricEntry; durationFrames: number }> = ({
lyric,
durationFrames,
}) => {
const frame = useCurrentFrame();
const { fps } = useVideoConfig();
const enterDuration = 12;
const exitStart = durationFrames - 12;
const opacity = interpolate(
frame,
[0, enterDuration, exitStart, durationFrames],
[0, 1, 1, 0],
{ extrapolateLeft: "clamp", extrapolateRight: "clamp" }
);
const translateY = interpolate(
frame,
[0, enterDuration],
[24, 0],
{ extrapolateLeft: "clamp", extrapolateRight: "clamp" }
);
return (
<div
style={{
opacity,
transform: `translateY(${translateY}px)`,
position: "absolute",
bottom: "20%",
left: 0,
right: 0,
textAlign: "center",
padding: "0 10%",
}}
>
{/* word highlighting JSX from above */}
</div>
);
};
This produces a clean fade-up entrance and fade-out exit for each lyric line, independent of the word highlighting animation layered on top.
Background Visuals: Particles and Color Fields
A static background makes lyric videos feel flat. Remotion’s rendering pipeline runs in a browser context, which means you have full access to CSS, Canvas 2D, and SVG for generating dynamic backgrounds. Here is a simple particle field pattern driven by the current frame:
const PARTICLE_COUNT = 40;
export const ParticleBackground: React.FC = () => {
const frame = useCurrentFrame();
const { width, height } = useVideoConfig();
const particles = Array.from({ length: PARTICLE_COUNT }, (_, i) => {
// Deterministic pseudo-random values per particle
const seed = i * 137.508;
const x = (Math.sin(seed) * 0.5 + 0.5) * width;
const baseY = (Math.cos(seed * 2.1) * 0.5 + 0.5) * height;
const speed = 0.2 + (i % 5) * 0.08;
const y = ((baseY + frame * speed) % height);
const size = 2 + (i % 4) * 1.5;
const opacity = 0.1 + (Math.sin(frame * 0.02 + seed) * 0.5 + 0.5) * 0.3;
return { x, y, size, opacity };
});
return (
<svg
width={width}
height={height}
style={{ position: "absolute", top: 0, left: 0 }}
>
{particles.map((p, i) => (
<circle
key={i}
cx={p.x}
cy={p.y}
r={p.size}
fill="white"
opacity={p.opacity}
/>
))}
</svg>
);
};
For gradient backgrounds that shift with the music’s mood, you can map lyric sections to color palettes and interpolate between them:
const sectionColors = [
["#1a1a2e", "#16213e"], // Verse 1
["#0f3460", "#533483"], // Chorus
["#1a1a2e", "#16213e"], // Verse 2
["#e94560", "#533483"], // Bridge
];
const sectionIndex = Math.floor(frame / (durationInFrames / sectionColors.length));
const [bg1, bg2] = sectionColors[Math.min(sectionIndex, sectionColors.length - 1)];
Adding the Background Music Track
Remotion’s <Audio> component handles the music track. It accepts a src prop pointing to any audio file that can be loaded in the browser:
import { Audio } from "remotion";
export const LyricVideo: React.FC<{ audioSrc: string; lyrics: LyricEntry[] }> = ({
audioSrc,
lyrics,
}) => {
return (
<>
<Audio src={audioSrc} />
<ParticleBackground />
{lyrics.map((lyric, i) => (
<Sequence
key={i}
from={timestampToFrame(lyric.start, fps)}
durationInFrames={timestampToFrame(lyric.end - lyric.start, fps)}
>
<LyricLine lyric={lyric} durationFrames={timestampToFrame(lyric.end - lyric.start, fps)} />
</Sequence>
))}
</>
);
};
The <Audio> component plays back in the preview and is included in the final render. Remotion mixes audio using FFmpeg during the render step, so the output MP4 contains both the video frames and the audio track perfectly synchronized.
For volume control and fade-ins, <Audio> accepts a volume prop that can be an interpolate() expression:
<Audio
src={audioSrc}
volume={(f) =>
interpolate(f, [0, 30], [0, 1], { extrapolateRight: "clamp" })
}
/>
Rendering at 1080p and 4K
For YouTube uploads, 1920×1080 at 30fps is the baseline. For 4K music videos intended for high-resolution displays or licensing, 3840×2160 at 24fps or 30fps is standard. Your Remotion composition simply needs the correct dimensions:
<Composition
id="LyricVideo-1080p"
component={LyricVideo}
durationInFrames={durationFromAudio}
fps={30}
width={1920}
height={1080}
/>
<Composition
id="LyricVideo-4K"
component={LyricVideo}
durationInFrames={durationFromAudio}
fps={30}
width={3840}
height={2160}
/>
Since all sizing in your component is relative to scale = Math.min(width, height), the 4K render will simply be sharper with no additional work. Keep in mind that 4K renders take roughly 4x as long as 1080p. For production, render 1080p locally for review and trigger 4K on a cloud render farm or Remotion Lambda.
Batch Rendering an Entire Album
The real efficiency gain in using Remotion for lyric videos comes from batch rendering. An album with 12 tracks can be rendered overnight with a single script:
// render-album.ts
import { renderMedia, selectComposition } from "@remotion/renderer";
import tracks from "./tracks.json";
for (const track of tracks) {
const composition = await selectComposition({
serveUrl: bundleUrl,
id: "LyricVideo-1080p",
inputProps: {
audioSrc: track.audioSrc,
lyrics: track.lyrics,
},
});
await renderMedia({
composition,
serveUrl: bundleUrl,
codec: "h264",
outputLocation: `out/${track.slug}.mp4`,
inputProps: {
audioSrc: track.audioSrc,
lyrics: track.lyrics,
},
});
console.log(`Rendered: ${track.title}`);
}
Each track’s lyrics and audio path live in tracks.json. The script iterates through the array, renders each video sequentially, and saves the outputs to out/. Run it with npx tsx render-album.ts or include it in a CI job that triggers when the lyrics JSON is updated.
YouTube Video Format Best Practices
For lyric videos destined for YouTube, a few technical details matter:
Codec: H.264 (AAC audio) is the most compatible. Remotion uses this by default with codec: "h264".
Bitrate: For 1080p lyric videos with moderate motion, 8–12 Mbps produces excellent quality. Pass --crf=18 for high quality or use videoBitrate: "10M" in the renderMedia config.
Container: MP4 (.mp4) uploads fastest and processes quickest on YouTube. Avoid MOV for upload even though it is technically supported.
Audio: 44.1kHz stereo AAC at 192kbps or higher. If your source audio is 48kHz (common in music production), FFmpeg will resample automatically.
End card space: YouTube recommends leaving the last 20 seconds as lower-complexity visuals to accommodate the end card UI overlay. In your composition, you can fade to a title card or album art during this window.
RenderComp Lyric Video Templates
RenderComp includes a set of ready-to-use lyric video templates built on this exact architecture — particle backgrounds, word-level karaoke highlighting, section-based color palettes, and inputProps schemas that accept a lyrics JSON and audio file path. Whether you are producing videos for a single artist or running a lyric video production service, the templates give you a production-grade starting point. Visit rendercomp.com to explore.
FAQ
Q1: How do I generate word-level timestamps automatically from an audio file?
The most accessible option is the OpenAI Whisper API with timestamp_granularities: ["word"]. Open-source alternatives include WhisperX (Python, runs locally) and the Gentle forced aligner. For music with vocals, forced alignment generally outperforms transcription-based approaches since the lyrics are known in advance.
Q2: Can Remotion handle audio files longer than a few minutes? Yes. Remotion has no inherent audio length limit. A 4-minute song renders fine. For very long files (10+ minutes), ensure your system has adequate RAM since the full audio is loaded into the browser context during rendering.
Q3: My audio and video are slightly out of sync in the rendered output. What causes this?
Sync issues usually come from incorrect FPS calculations when converting timestamps to frames. Double-check that you are using the composition’s actual fps value from useVideoConfig() rather than a hardcoded constant. Also verify that durationInFrames for the composition matches the audio length (in seconds × fps, rounded to an integer).
Q4: Can I add audio reactivity — making visual elements respond to the audio amplitude?
Not natively within Remotion’s standard APIs. The recommended approach is to pre-analyze the audio waveform and amplitude data using a tool like Essentia or Meyda, export the amplitude data as a JSON array, and then use useCurrentFrame() to index into that array at render time. This gives you frame-accurate audio reactivity without any runtime audio analysis.
Q5: What is the best font choice for lyric videos?
Since Remotion renders in a browser context, you can use any font that is installed on the render machine. System fonts (-apple-system, "Segoe UI", Roboto) are always safe. For custom typography, load the font using a local @font-face declaration in a CSS file that Remotion bundles — avoid external CDN font links, as they introduce network dependency into your render pipeline.
Q6: How do I handle multi-language lyrics with CJK characters?
CJK characters render correctly when the system has appropriate fonts installed. On macOS, "Hiragino Kaku Gothic ProN" handles Japanese; on Windows, "Yu Gothic" is the standard choice. Add these to your fontFamily stack alongside the Latin fallbacks.
Q7: Can I add a music visualizer / waveform bar chart to the background?
Yes. Pre-analyze the audio to extract per-frame amplitude data, store it as a JSON array, then render bar charts using SVG <rect> elements sized by the amplitude value at the current frame. This is a common pattern and performs well in Remotion’s renderer since it is pure SVG, not Canvas.
Now available
Get 1,000+ Remotion Templates
Pay once — no subscription. Lifetime updates. TypeScript-first.
View pricing →