Upload
Last updated
Last updated
Uploading files involves chunked file transfers to multiple blobbers using erasure coding (EC). This approach ensures fault tolerance, redundancy, and efficient retrieval.
The system uses n:p EC ratio, where:
n = Number of data blobbers (storing actual data)
p = Number of parity blobbers (storing error correction data)
This method provides faster upload/download speeds through parallel processing and ensures data reliability.
The upload mechanism follows a chunked approach for handling large files efficiently. Here is step by step process:
Chunk Preparation
Read the file in 64 KB chunks. Ensure each chunk has n * 64 KB fragments.
If the last fragment is smaller, pad it with null bytes.
Erasure Encoding
Compute p parity-shards using Reed-Solomon encoding.
This ensures data redundancy and fault tolerance.
Shard Distribution
Each shard (data or parity) is sent to a respective blobber.
n + p upload requests are created.
Temporary Storage on Blobbers
Blobbers store chunks in a temporary directory.
Subsequent fragments are appended to the same file.
Chunk Upload Confirmation
Once all fragments are uploaded successfully, the file is considered fully uploaded.
The Merkle root ensures data integrity and is uniquely computed for each blobber. Lets talk about Merkle Root Calculation Process
Chunk Division: Each 64 KB chunk is split into 64-byte blocks, creating 1024 blocks.
Leaf Hash Calculation: Each index i (0-1023) has m number of 64-byte blocks (continuous data). Compute the hash of these m blocks → This forms the Merkle tree leaf.
Merkle Tree Construction: The Merkle root is derived from these 1024 leaf hashes.
Unique Blobber Merkle Roots: Since each blobber stores different shards, their Merkle roots differ.
Challenges ensure file integrity and availability by periodically verifying blobber-stored data. Here is how challenges are issued.
Block Count Tracking
Each uploaded file has num_of_blocks
, representing the total 64 KB chunks stored per blobber.
A directory's num_of_blocks
is the sum of its children's num_of_blocks
.
Random Block Selection for Challenge
The network selects a random block_num using:
The random seed is generated by the blockchain.
Fetching Object Path
The challenge queries GetObjectPath() to find:
File reference
Merkle root
Write markers history
Blobber’s Response to Challenge
Check if a challenge exists for the blobber.
Retrieve the required block.
Recalculate the Merkle tree.
Submit verification proof to the validator.
Validation and Rewards
Validators verify the challenge response.
If successful:
Blobber earns rewards from the challenge pool.
If failed:
Stake is penalized (if blobber fails multiple challenges).
To improve efficiency, dynamic configurations can be considered:
File Size
Block Size
Merkle Chunk Size
Optimizations
< 1 KB
1 KB
1 Byte
Fewer Merkle leaves
> 1 GB
50 MB
50 KB
Reduced computational load
By dynamically adjusting chunk sizes, the system can optimize performance based on:
File size
Data-shards count
Token economy impact
Computational efficiency