> For the complete documentation index, see [llms.txt](https://docs.zus.network/zus-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.zus.network/zus-docs/system-overview/storage/file-operations/streaming.md).

# Streaming

## Architecture

The streaming system is designed with a robust architecture, encapsulated within a core player function. The key responsibilities include:

* **Video Uploading**: Handles video file uploads, segments them for streaming, generates thumbnails, and prepares the video for playback.
* **Video Transcoding**: Uses FFmpeg for transcoding video segments into streaming-compatible formats.
* **Video Playback**: Uses the Media Source Extensions (MSE) API for dynamic playback of transcoded video segments.
* **Resource Management**: Efficiently manages system resources, including FFmpeg instances and Web Workers.

### Key Components and Their Functionalities

#### Dependencies and Imports

The system relies on specific dependencies to execute various tasks:

* **`@custom/ffmpeg`**: A browser-compatible FFmpeg module optimized for video transcoding.

#### Video Uploading with `uploadToIndexDb` Function

The video upload phase involves the following steps:

1. **File Handling**:
   * Creates and mounts a virtual filesystem in FFmpeg (`WORKERFS`) to store the uploaded video.
2. **Metadata Extraction**:
   * Extracts key metadata (e.g., duration, audio presence) using FFmpeg probes.
3. **Thumbnail Generation**:
   * Captures a single frame from the video to generate a thumbnail image.
4. **Video Segmentation**:
   * Splits the video into smaller segments (e.g., 5-second chunks) to enable efficient streaming.
   * Each segment is stored as an individual file in the virtual filesystem.
5. **JSON Metadata Creation**:
   * Compiles a JSON object detailing the video segments' metadata, including duration, segment count, and audio presence.

#### Video Playback with `transcodeFileToMediaSource` Function

This function manages the playback phase, dynamically transcoding and streaming video segments.

1. **Media Source Initialization**:
   * Creates a `MediaSource` object and attaches it to the video element for MSE-based streaming.
2. **Job Management**:
   * Maintains a queue of transcoding jobs for video segments.
   * Uses FFmpeg to transcode each segment into a suitable format (e.g., MP4 with H.264 codec).
3. **Buffer Management**:
   * Appends transcoded segments to the `sourceBuffer` of the `MediaSource`.
   * Adjusts buffering logic dynamically based on playback position.
4. **Event Handling**:
   * Monitors video and buffer events (e.g., play, pause, seeking) to manage playback state and user interactions.
5. **Error Handling and Recovery**:
   * Implements mechanisms to retry failed transcoding tasks, ensuring a solid playback experience.

### FFmpeg Commands

FFmpeg is central to the system, performing tasks such as thumbnail generation, video segmentation, transcoding, and merging.

#### Thumbnail Generation

```javascript
await ffmpegs[0].exec([
  '-i', name,
  '-ss', `${pictureFrame}`,
  '-threads', '4',
  '-vframes', '1',
  'thumbnail_generated.jpg',
]);
```

* **Purpose**: Captures a single frame from the video.
* **Key Parameters**:
  * `-i`: Input file.
  * `-ss`: Timestamp for the frame.
  * `-threads`: Number of threads for processing.
  * `-vframes 1`: Outputs a single frame.

#### Video Segmentation

```javascript
await ffmpegs[0].exec([
  '-fflags', '+genpts',
  '-i', name,
  '-threads', '4',
  '-f', 'segment',
  '-segment_time', '5',
  '-segment_time_delta', '0.01',
  '-c', 'copy',
  '-reset_timestamps', '1',
  `${fileName}_%03d.${ext}`,
]);
```

* **Purpose**: Splits the video into 5-second segments.
* **Key Parameters**:
  * `-segment_time`: Duration of each segment.
  * `-reset_timestamps`: Resets timestamps for synchronization.
  * `-c copy`: Avoids re-encoding to preserve quality.

#### Transcoding for Playback

```javascript
await ffmpeg.exec([
  '-nostats',
  '-loglevel', 'error',
  '-threads', '4',
  '-i', inputFileChunk,
  '-fflags', '+genpts',
  '-an',
  '-movflags', 'frag_every_frame+empty_moov+default_base_moof',
  '-preset', 'ultrafast',
  '-c:v', 'libx264',
  '-crf', `${cfrValue}`,
  '-copyts', `/temp_video_${job.id}.mp4`,
], undefined, { signal });
```

* **Purpose**: Transcodes video segments for streaming.
* **Key Parameters**:
  * `-c:v libx264`: Specifies the H.264 codec.
  * `-preset ultrafast`: Optimizes for speed.
  * `-movflags`: Configures the MP4 container for MSE compatibility.

#### Audio Transcoding and Merging

```javascript
await ffmpeg.exec([
  '-nostats',
  '-loglevel', 'error',
  '-threads', '4',
  '-i', `/temp_video_${job.id}.mp4`,
  '-i', `/temp_audio_${job.id}.aac`,
  '-c:v', 'copy',
  '-c:a', 'aac',
  '-b:a', '256k',
  '-movflags', 'frag_every_frame+empty_moov+default_base_moof',
  '-copyts',
  '-af', 'aresample=async=1',
  '-shortest', outputFile,
], undefined, { signal });
```

* **Purpose**: Merges video and audio streams into a final MP4 file.
* **Key Parameters**:
  * `-c:a aac`: Specifies AAC codec for audio.
  * `-af aresample=async=1`: Ensures audio-video sync.

#### Concatenating Video Segments for Download

```javascript
await ffmpegs[0].exec([
  '-f', 'concat',
  '-safe', '0',
  '-i', 'concat_list.txt',
  '-c', 'copy',
  '-fflags', '+bitexact',
  file.name,
]);
```

* **Purpose**: Combines all video segments for user download.
* **Key Parameters**:
  * `-f concat`: Enables concatenation of video files.

### Job Queue Management

#### Initialization

* **Jobs Array**: Stores job objects representing video segments.

```javascript
let jobs = [];
```

* **Chunk Parameters**: Define segment durations and manage timing.

```javascript
let chunkStart = 0;
let durationLeft = duration;
let chunkDurationSize = 5;
```

#### Creating Job Objects

Each job includes metadata about the segment:

```javascript
let job = {
  id: index,
  chunkStart,
  chunkDuration,
  state: 'queued',
  outputData: null,
};
jobs.push(job);
```

#### Concurrent Processing

* Concurrent FFmpeg Instances:

```javascript
let ffmpegCount = 1;
```

* Job Assignment:

```javascript
let job = jobQueue.shift();
job.state = 'running';
```

#### Appending Data to `SourceBuffer`

* Event Listener:

```javascript
sourceBuffer.addEventListener('updateend', async () => {
  let job = await getCompletedJob(ii++);
  if (job) sourceBuffer.appendBuffer(job.outputData);
});
```

* Stream Finalization

Signals end-of-stream to the video element:

```javascript
mediaSource.endOfStream();
```

### Directory Structure for Download

Video Folder:

* `Original file`
* `0kb marker file`
* `Thumbnail file`
* `Preview folder`:
  * Segments named sequentially (e.g., `1`, `2`, `3`).
  * `metadata.json`.

This ensures a comprehensive and efficient streaming system while supporting user download capabilities.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.zus.network/zus-docs/system-overview/storage/file-operations/streaming.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.