Download

In a decentralized storage network, downloading a file efficiently while ensuring data integrity, security, and performance presents several challenges:

• Consensus on file correctness: Data is distributed across multiple blobbers, requiring consensus on the correct file hash before reconstruction.

• Efficient data retrieval: Downloading from multiple sources must be optimized for speed and reliability.

• Integrity verification: Ensuring downloaded data is correct and unmodified using cryptographic proofs.

• Handling encrypted files: Securely decrypting data while preserving access control.

• Erasure coding reconstruction: Recovering the original file from distributed data shards.

Process

To overcome these challenges, the download process follows a structured approach:

Fig1: Download Process

1. Consensus on File Hash

• Before downloading, consensus is reached on the actual file hash across all blobbers.

• A subset of blobbers is selected based on successful validation of stored data.

• The consensus threshold is set to data_shards, as this is the minimum required to decode erasure-encoded data.

2. Parallel Block-Based Download

• Data is downloaded in blocks of 64KB * data_shards, ensuring efficient retrieval.

• Each blobber is requested to send 100 blocks per request by default.

• Performance tracking: The first request is timed, and a subset of blobbers is selected based on response time to optimize speed.

• Downloading occurs in parallel from multiple blobbers, significantly improving speed compared to a single-provider approach.

3. Erasure Decoding and Decryption

Erasure Coding Reconstruction

• The system uses Reed-Solomon erasure coding to reconstruct the original file from data_shards out of data_shards + parity_shards.

• Missing or slow blobbers do not impact file recovery as long as the threshold is met.

Decryption (If Encrypted)

• Each block is decrypted using the user's private key.

• Decryption follows AES-GCM (Authenticated Encryption with Associated Data) to ensure both confidentiality and authenticity.

• If the file was shared using Proxy Re-Encryption (PRE), the downloaded data is re-encrypted by blobbers for the recipient, and the recipient decrypts it using their private key.

4. Integrity Verification

Validation Merkle Proofs

• Each block’s correctness is verified using Merkle Proofs provided by blobbers.

• Blobbers generate a Validation Merkle Proof for each block, which is verified against the Validation Merkle Root Hash calculated during the upload.

• This ensures that data integrity is maintained and that no blobber has modified or provided incorrect data.

File Hash Verification

• Once all blocks are retrieved and decoded, the final reconstructed file hash is computed.

• The computed hash is compared against the consensus-agreed file hash to ensure correctness.

• Any mismatch results in rejection of the downloaded data.

5. Authorization for Shared Files

• Users can download shared files using an Auth Ticket, which ensures proper access control.

• The Auth Ticket contains the necessary metadata and cryptographic permissions for the recipient.

• If the file was encrypted, Proxy Re-Encryption (PRE) enables blobbers to re-encrypt data without exposing plaintext.