Upload flow

This page provides an overview of upload process on the client side(gosdk)

Upload Flowchart

Above are the functions which are called during upload-

1. StartMultiUpload :- This function is used by zboxcli to start the upload process, here we create the upload request, open the file and set the required file meta fields we need to pass it to DoMultiOperation , the process of creating this request takes less than 1ms.

2. DoMultiUpload :- This function is responsible for processing the upload request(can take multiple requests), it starts by creating a connection object which is a network call and is created on the blobbers and then we call CreateChunkedUpload this builds the request further to perform chunked upload and takes around 2ms.

3. Process :- We start processing the request now, this function is responsible for creating all the shards and uploading them to the blobbers, checking the consensus at each step.

4. ReadChunks :- We start by reading the chunks until EOF, number of chunks read in one read call depends on number of data shards, it will be default chunk size (64KB)*no_data_shards, we read this data write it to hasher(for file hash calculation) and apply erasure encoding, we do this step for chunkNumber times provided by the client. This process is taking around 630-722ms for EC51 per 500 chunks.

5. BuildForm :- Once we have read the data and appended it to fileShards for each blobber we start the upload part, it starts with building the formData for each blobber. We read the data from fileShard(in memory) and write it to multipart form file, validation and merkle tree, if this is the final chunk we also finalize the hasher i.e get validation root,merkle root and file hash to send to blobber. This takes around 250ms per blobber and is invoked in separate go routine for each blobber.

6. SendUploadRequest :- This function sends the multipart formData to the blobber and takes around 350ms as this is asynchronous(blobber processes the request in background)

7. WriteMarkerLock :- After uploading all the data to blobber, we start the commit process by locking the writeMarker(network call) for commit, this process takes around 25ms

8. CheckAllocStatus :- We check the allocation version(network call) for auto repair and rollback, and takes around 7ms

9. CommitMarker :- We prepare the commit request for each blobber in the mask and get the affected object paths and latestWM(network call) to calculate the allocation root and populate other fields of writemarker , once generated we send the CommitWrite request to blobber. The whole process takes around 160ms for EC51, this can be higher for lower no of data shards.

Note- Above timings are for EC51( 5 data shards and 1 parity shard)