> For the complete documentation index, see [llms.txt](https://docs.zus.network/zus-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.zus.network/zus-docs/system-overview/storage/file-operations/streaming.md).

# Streaming

## Architecture

The streaming system is designed with a robust architecture, encapsulated within a core player function. The key responsibilities include:

* **Video Uploading**: Handles video file uploads, segments them for streaming, generates thumbnails, and prepares the video for playback.
* **Video Transcoding**: Uses FFmpeg for transcoding video segments into streaming-compatible formats.
* **Video Playback**: Uses the Media Source Extensions (MSE) API for dynamic playback of transcoded video segments.
* **Resource Management**: Efficiently manages system resources, including FFmpeg instances and Web Workers.

### Key Components and Their Functionalities

#### Dependencies and Imports

The system relies on specific dependencies to execute various tasks:

* **`@custom/ffmpeg`**: A browser-compatible FFmpeg module optimized for video transcoding.

#### Video Uploading with `uploadToIndexDb` Function

The video upload phase involves the following steps:

1. **File Handling**:
   * Creates and mounts a virtual filesystem in FFmpeg (`WORKERFS`) to store the uploaded video.
2. **Metadata Extraction**:
   * Extracts key metadata (e.g., duration, audio presence) using FFmpeg probes.
3. **Thumbnail Generation**:
   * Captures a single frame from the video to generate a thumbnail image.
4. **Video Segmentation**:
   * Splits the video into smaller segments (e.g., 5-second chunks) to enable efficient streaming.
   * Each segment is stored as an individual file in the virtual filesystem.
5. **JSON Metadata Creation**:
   * Compiles a JSON object detailing the video segments' metadata, including duration, segment count, and audio presence.

#### Video Playback with `transcodeFileToMediaSource` Function

This function manages the playback phase, dynamically transcoding and streaming video segments.

1. **Media Source Initialization**:
   * Creates a `MediaSource` object and attaches it to the video element for MSE-based streaming.
2. **Job Management**:
   * Maintains a queue of transcoding jobs for video segments.
   * Uses FFmpeg to transcode each segment into a suitable format (e.g., MP4 with H.264 codec).
3. **Buffer Management**:
   * Appends transcoded segments to the `sourceBuffer` of the `MediaSource`.
   * Adjusts buffering logic dynamically based on playback position.
4. **Event Handling**:
   * Monitors video and buffer events (e.g., play, pause, seeking) to manage playback state and user interactions.
5. **Error Handling and Recovery**:
   * Implements mechanisms to retry failed transcoding tasks, ensuring a solid playback experience.

### FFmpeg Commands

FFmpeg is central to the system, performing tasks such as thumbnail generation, video segmentation, transcoding, and merging.

#### Thumbnail Generation

```javascript
await ffmpegs[0].exec([
  '-i', name,
  '-ss', `${pictureFrame}`,
  '-threads', '4',
  '-vframes', '1',
  'thumbnail_generated.jpg',
]);
```

* **Purpose**: Captures a single frame from the video.
* **Key Parameters**:
  * `-i`: Input file.
  * `-ss`: Timestamp for the frame.
  * `-threads`: Number of threads for processing.
  * `-vframes 1`: Outputs a single frame.

#### Video Segmentation

```javascript
await ffmpegs[0].exec([
  '-fflags', '+genpts',
  '-i', name,
  '-threads', '4',
  '-f', 'segment',
  '-segment_time', '5',
  '-segment_time_delta', '0.01',
  '-c', 'copy',
  '-reset_timestamps', '1',
  `${fileName}_%03d.${ext}`,
]);
```

* **Purpose**: Splits the video into 5-second segments.
* **Key Parameters**:
  * `-segment_time`: Duration of each segment.
  * `-reset_timestamps`: Resets timestamps for synchronization.
  * `-c copy`: Avoids re-encoding to preserve quality.

#### Transcoding for Playback

```javascript
await ffmpeg.exec([
  '-nostats',
  '-loglevel', 'error',
  '-threads', '4',
  '-i', inputFileChunk,
  '-fflags', '+genpts',
  '-an',
  '-movflags', 'frag_every_frame+empty_moov+default_base_moof',
  '-preset', 'ultrafast',
  '-c:v', 'libx264',
  '-crf', `${cfrValue}`,
  '-copyts', `/temp_video_${job.id}.mp4`,
], undefined, { signal });
```

* **Purpose**: Transcodes video segments for streaming.
* **Key Parameters**:
  * `-c:v libx264`: Specifies the H.264 codec.
  * `-preset ultrafast`: Optimizes for speed.
  * `-movflags`: Configures the MP4 container for MSE compatibility.

#### Audio Transcoding and Merging

```javascript
await ffmpeg.exec([
  '-nostats',
  '-loglevel', 'error',
  '-threads', '4',
  '-i', `/temp_video_${job.id}.mp4`,
  '-i', `/temp_audio_${job.id}.aac`,
  '-c:v', 'copy',
  '-c:a', 'aac',
  '-b:a', '256k',
  '-movflags', 'frag_every_frame+empty_moov+default_base_moof',
  '-copyts',
  '-af', 'aresample=async=1',
  '-shortest', outputFile,
], undefined, { signal });
```

* **Purpose**: Merges video and audio streams into a final MP4 file.
* **Key Parameters**:
  * `-c:a aac`: Specifies AAC codec for audio.
  * `-af aresample=async=1`: Ensures audio-video sync.

#### Concatenating Video Segments for Download

```javascript
await ffmpegs[0].exec([
  '-f', 'concat',
  '-safe', '0',
  '-i', 'concat_list.txt',
  '-c', 'copy',
  '-fflags', '+bitexact',
  file.name,
]);
```

* **Purpose**: Combines all video segments for user download.
* **Key Parameters**:
  * `-f concat`: Enables concatenation of video files.

### Job Queue Management

#### Initialization

* **Jobs Array**: Stores job objects representing video segments.

```javascript
let jobs = [];
```

* **Chunk Parameters**: Define segment durations and manage timing.

```javascript
let chunkStart = 0;
let durationLeft = duration;
let chunkDurationSize = 5;
```

#### Creating Job Objects

Each job includes metadata about the segment:

```javascript
let job = {
  id: index,
  chunkStart,
  chunkDuration,
  state: 'queued',
  outputData: null,
};
jobs.push(job);
```

#### Concurrent Processing

* Concurrent FFmpeg Instances:

```javascript
let ffmpegCount = 1;
```

* Job Assignment:

```javascript
let job = jobQueue.shift();
job.state = 'running';
```

#### Appending Data to `SourceBuffer`

* Event Listener:

```javascript
sourceBuffer.addEventListener('updateend', async () => {
  let job = await getCompletedJob(ii++);
  if (job) sourceBuffer.appendBuffer(job.outputData);
});
```

* Stream Finalization

Signals end-of-stream to the video element:

```javascript
mediaSource.endOfStream();
```

### Directory Structure for Download

Video Folder:

* `Original file`
* `0kb marker file`
* `Thumbnail file`
* `Preview folder`:
  * Segments named sequentially (e.g., `1`, `2`, `3`).
  * `metadata.json`.

This ensures a comprehensive and efficient streaming system while supporting user download capabilities.