Download Implementation

This page describes download implementation along with inclusion of validation root and provided io.Reader interface which can be used to download or resume download. But first lets discuss upload.

Consider an allocation A created with 3 blobbers of which two is data-shards (Ds) and one is parity-shard (Dp). Let client upload data of size 256 KB. Here is a simple description of how upload works in the above case.

Note: Block size of data is 64 KB. Chunk-size and block-size are used interchangeably in gosdk and blobber. Here I will be using block-size as being unit of block which is 64KB and chunk size as data-shards * block-size.

1. Client first calculates chunk size which is data-shards * block-size = 128 KB

2. From this chunk, client puts this data in an array of size 3 (total-blobbers) as [ 64 KB, 64 KB, nil ]. Lets name this array as Arr.

3. nil is to be calculated using erasure encoding which can calculate parity data for any given parity shards. Note that the client can use any sort of encoding but we recommend erasure encoding and gosdk implements erasure encoding only.

4. After parity shard is calculated array looks as [64 KB, 64 KB, 64 KB].

5. Client uploads Arr[0] to blobber-1, Arr[1] to blobber-2 and Arr[2] to blobber-3.

6. The client repeats this process to another chunk.

7. In blobber, it simply appends data as uploaded by the client.

8. The client would also continue to calculate FixedMerkleTree(FMT) and Validation Tree(VT) while processing data for each blobber. FMT and VT are different for each blobber because as for blobber it does not concern whether the client is associated with other blobbers or not.

9. The client will also calculate actual-file-hash which simply is the hash of whole file data.

10. The actual file hash is signed by the client(allocation owner), let's name it ActualFileHashSignature.

11. ActualFileHashSignature + ValidationRoot is hashed and signed, let's name it as ValidationSignature.

12. These metadata is used in the calculation of allocation root which is added into the writemarker. Client thus signs this writemarker and sends to blobber as a commitment to the uploaded data.

Necessity of step 10 and 11 will be discussed later.