Streaming

Architecture

The streaming system is designed with a robust architecture, encapsulated within a core player function. The key responsibilities include:

• Video Uploading: Handles video file uploads, segments them for streaming, generates thumbnails, and prepares the video for playback.

• Video Transcoding: Uses FFmpeg for transcoding video segments into streaming-compatible formats.

• Video Playback: Uses the Media Source Extensions (MSE) API for dynamic playback of transcoded video segments.

• Resource Management: Efficiently manages system resources, including FFmpeg instances and Web Workers.

Key Components and Their Functionalities

Dependencies and Imports

The system relies on specific dependencies to execute various tasks:

• @custom/ffmpeg: A browser-compatible FFmpeg module optimized for video transcoding.

Video Uploading with uploadToIndexDb Function

The video upload phase involves the following steps:

1. File Handling:

   • Creates and mounts a virtual filesystem in FFmpeg (WORKERFS) to store the uploaded video.

2. Metadata Extraction:

   • Extracts key metadata (e.g., duration, audio presence) using FFmpeg probes.

3. Thumbnail Generation:

   • Captures a single frame from the video to generate a thumbnail image.

4. Video Segmentation:

   • Splits the video into smaller segments (e.g., 5-second chunks) to enable efficient streaming.

   • Each segment is stored as an individual file in the virtual filesystem.

5. JSON Metadata Creation:

   • Compiles a JSON object detailing the video segments' metadata, including duration, segment count, and audio presence.

Video Playback with transcodeFileToMediaSource Function

This function manages the playback phase, dynamically transcoding and streaming video segments.

1. Media Source Initialization:

   • Creates a MediaSource object and attaches it to the video element for MSE-based streaming.

2. Job Management:

   • Maintains a queue of transcoding jobs for video segments.

   • Uses FFmpeg to transcode each segment into a suitable format (e.g., MP4 with H.264 codec).

3. Buffer Management:

   • Appends transcoded segments to the sourceBuffer of the MediaSource.

   • Adjusts buffering logic dynamically based on playback position.

4. Event Handling:

   • Monitors video and buffer events (e.g., play, pause, seeking) to manage playback state and user interactions.

5. Error Handling and Recovery:

   • Implements mechanisms to retry failed transcoding tasks, ensuring a solid playback experience.

FFmpeg Commands

FFmpeg is central to the system, performing tasks such as thumbnail generation, video segmentation, transcoding, and merging.

Thumbnail Generation

await ffmpegs[0].exec([
  '-i', name,
  '-ss', `${pictureFrame}`,
  '-threads', '4',
  '-vframes', '1',
  'thumbnail_generated.jpg',
]);

• Purpose: Captures a single frame from the video.

• Key Parameters:

  • -i: Input file.

  • -ss: Timestamp for the frame.

  • -threads: Number of threads for processing.

  • -vframes 1: Outputs a single frame.

Video Segmentation

await ffmpegs[0].exec([
  '-fflags', '+genpts',
  '-i', name,
  '-threads', '4',
  '-f', 'segment',
  '-segment_time', '5',
  '-segment_time_delta', '0.01',
  '-c', 'copy',
  '-reset_timestamps', '1',
  `${fileName}_%03d.${ext}`,
]);

• Purpose: Splits the video into 5-second segments.

• Key Parameters:

  • -segment_time: Duration of each segment.

  • -reset_timestamps: Resets timestamps for synchronization.

  • -c copy: Avoids re-encoding to preserve quality.

Transcoding for Playback

await ffmpeg.exec([
  '-nostats',
  '-loglevel', 'error',
  '-threads', '4',
  '-i', inputFileChunk,
  '-fflags', '+genpts',
  '-an',
  '-movflags', 'frag_every_frame+empty_moov+default_base_moof',
  '-preset', 'ultrafast',
  '-c:v', 'libx264',
  '-crf', `${cfrValue}`,
  '-copyts', `/temp_video_${job.id}.mp4`,
], undefined, { signal });

• Purpose: Transcodes video segments for streaming.

• Key Parameters:

  • -c:v libx264: Specifies the H.264 codec.

  • -preset ultrafast: Optimizes for speed.

  • -movflags: Configures the MP4 container for MSE compatibility.

Audio Transcoding and Merging

await ffmpeg.exec([
  '-nostats',
  '-loglevel', 'error',
  '-threads', '4',
  '-i', `/temp_video_${job.id}.mp4`,
  '-i', `/temp_audio_${job.id}.aac`,
  '-c:v', 'copy',
  '-c:a', 'aac',
  '-b:a', '256k',
  '-movflags', 'frag_every_frame+empty_moov+default_base_moof',
  '-copyts',
  '-af', 'aresample=async=1',
  '-shortest', outputFile,
], undefined, { signal });

• Purpose: Merges video and audio streams into a final MP4 file.

• Key Parameters:

  • -c:a aac: Specifies AAC codec for audio.

  • -af aresample=async=1: Ensures audio-video sync.

Concatenating Video Segments for Download

await ffmpegs[0].exec([
  '-f', 'concat',
  '-safe', '0',
  '-i', 'concat_list.txt',
  '-c', 'copy',
  '-fflags', '+bitexact',
  file.name,
]);

• Purpose: Combines all video segments for user download.

• Key Parameters:

  • -f concat: Enables concatenation of video files.

Job Queue Management

Initialization

• Jobs Array: Stores job objects representing video segments.

let jobs = [];

• Chunk Parameters: Define segment durations and manage timing.

let chunkStart = 0;
let durationLeft = duration;
let chunkDurationSize = 5;

Creating Job Objects

Each job includes metadata about the segment:

let job = {
  id: index,
  chunkStart,
  chunkDuration,
  state: 'queued',
  outputData: null,
};
jobs.push(job);

Concurrent Processing

• Concurrent FFmpeg Instances:

let ffmpegCount = 1;

• Job Assignment:

let job = jobQueue.shift();
job.state = 'running';

Appending Data to SourceBuffer

• Event Listener:

sourceBuffer.addEventListener('updateend', async () => {
  let job = await getCompletedJob(ii++);
  if (job) sourceBuffer.appendBuffer(job.outputData);
});

• Stream Finalization

Signals end-of-stream to the video element:

mediaSource.endOfStream();

Directory Structure for Download

Video Folder:

• Original file

• 0kb marker file

• Thumbnail file

• Preview folder:

  • Segments named sequentially (e.g., 1, 2, 3).

  • metadata.json.

This ensures a comprehensive and efficient streaming system while supporting user download capabilities.