Download

In a decentralized storage network, downloading a file efficiently while ensuring data integrity, security, and performance presents several challenges:

  • Consensus on file correctness: Data is distributed across multiple blobbers, requiring consensus on the correct file hash before reconstruction.

  • Efficient data retrieval: Downloading from multiple sources must be optimized for speed and reliability.

  • Integrity verification: Ensuring downloaded data is correct and unmodified using cryptographic proofs.

  • Handling encrypted files: Securely decrypting data while preserving access control.

  • Erasure coding reconstruction: Recovering the original file from distributed data shards.

Process

To overcome these challenges, the download process follows a structured approach:

Fig1: Download Process

1. Consensus on File Hash

  • Before downloading, consensus is reached on the actual file hash across all blobbers.

  • A subset of blobbers is selected based on successful validation of stored data.

  • The consensus threshold is set to data_shards, as this is the minimum required to decode erasure-encoded data.

2. Parallel Block-Based Download

  • Data is downloaded in blocks of 64KB * data_shards, ensuring efficient retrieval.

  • Each blobber is requested to send 100 blocks per request by default.

  • Performance tracking: The first request is timed, and a subset of blobbers is selected based on response time to optimize speed.

  • Downloading occurs in parallel from multiple blobbers, significantly improving speed compared to a single-provider approach.

3. Erasure Decoding and Decryption

Erasure Coding Reconstruction

  • The system uses Reed-Solomon erasure coding to reconstruct the original file from data_shards out of data_shards + parity_shards.

  • Missing or slow blobbers do not impact file recovery as long as the threshold is met.

Decryption (If Encrypted)

  • Each block is decrypted using the user's private key.

  • Decryption follows AES-GCM (Authenticated Encryption with Associated Data) to ensure both confidentiality and authenticity.

  • If the file was shared using Proxy Re-Encryption (PRE), the downloaded data is re-encrypted by blobbers for the recipient, and the recipient decrypts it using their private key.

4. Integrity Verification

Validation Merkle Proofs

  • Each block’s correctness is verified using Merkle Proofs provided by blobbers.

  • Blobbers generate a Validation Merkle Proof for each block, which is verified against the Validation Merkle Root Hash calculated during the upload.

  • This ensures that data integrity is maintained and that no blobber has modified or provided incorrect data.

File Hash Verification

  • Once all blocks are retrieved and decoded, the final reconstructed file hash is computed.

  • The computed hash is compared against the consensus-agreed file hash to ensure correctness.

  • Any mismatch results in rejection of the downloaded data.

5. Authorization for Shared Files

  • Users can download shared files using an Auth Ticket, which ensures proper access control.

  • The Auth Ticket contains the necessary metadata and cryptographic permissions for the recipient.

  • If the file was encrypted, Proxy Re-Encryption (PRE) enables blobbers to re-encrypt data without exposing plaintext.

Last updated