Allocations

Contract between the client and Züs System about storing their data

Introduction

To host your data on Züs system, there should be some kind of contract to agree on the terms of service, which are:

• Size limit.

• Expiration time.

• Number of data shards and parity shards.

• Write price range

• Nominating the preferred blobbers to host your data.

We call this contract an Allocation. In this page we'll discuss operations related to the allocation and how the data related to the allocations flow through the system.

Allocation Structure

type StorageAllocation struct {
	// ID is unique allocation ID that is equal to hash of transaction with
	// which the allocation has created.
	ID string `json:"id"`
	// Tx keeps hash with which the allocation has created or updated.
	Tx string `json:"tx"`

	DataShards        int                     `json:"data_shards"`
	ParityShards      int                     `json:"parity_shards"`
	Size              int64                   `json:"size"`
	Expiration        common.Timestamp        `json:"expiration_date"`
	Owner             string                  `json:"owner_id"`
	OwnerPublicKey    string                  `json:"owner_public_key"`
	Stats             *StorageAllocationStats `json:"stats"`
	DiverseBlobbers   bool                    `json:"diverse_blobbers"`
	PreferredBlobbers []string                `json:"preferred_blobbers"`
	// Blobbers not to be used anywhere except /allocation and /allocations table
	// if Blobbers are getting used in any smart-contract, we should avoid.
	BlobberAllocs    []*BlobberAllocation          `json:"blobber_details"`
	BlobberAllocsMap map[string]*BlobberAllocation `json:"-" msg:"-"`

	// Flag to determine if anyone can extend this allocation, meaning increase its size
	ThirdPartyExtendable bool `json:"third_party_extendable"`

	// FileOptions to define file restrictions on an allocation for third-parties
	// default 00000000 for all crud operations suggesting only owner has the below listed abilities.
	// enabling option/s allows any third party to perform certain ops
	// 00000001 - 1  - upload
	// 00000010 - 2  - delete
	// 00000100 - 4  - update
	// 00001000 - 8  - move
	// 00010000 - 16 - copy
	// 00100000 - 32 - rename
	FileOptions uint16 `json:"file_options"`

	WritePool currency.Coin `json:"write_pool"`

	// Requested ranges.
	ReadPriceRange  PriceRange `json:"read_price_range"`
	WritePriceRange PriceRange `json:"write_price_range"`

	// StartTime is time when the allocation has been created. We will
	// use it to check blobber's MaxOfferTime extending the allocation.
	StartTime common.Timestamp `json:"start_time"`
	// Finalized is true where allocation has been finalized.
	Finalized bool `json:"finalized,omitempty"`
	// Canceled set to true where allocation finalized by cancel_allocation
	// transaction.
	Canceled bool `json:"canceled,omitempty"`

	// MovedToChallenge is number of tokens moved to challenge pool.
	MovedToChallenge currency.Coin `json:"moved_to_challenge,omitempty"`
	// MovedBack is number of tokens moved from challenge pool to
	// related write pool (the Back) if a data has deleted.
	MovedBack currency.Coin `json:"moved_back,omitempty"`
	// MovedToValidators is total number of tokens moved to validators
	// of the allocation.
	MovedToValidators currency.Coin `json:"moved_to_validators,omitempty"`

	// TimeUnit configured in Storage SC when the allocation created. It can't
	// be changed for this allocation anymore. Even using expire allocation.
	TimeUnit time.Duration `json:"time_unit"`
}

Allocation flow in Züs System

Fig1: Allocation flow

1. The allocation is managed mainly in the blockchain network, so all the allocation operations are run as transactions to the Storage Smartcontract by the wallet of the allocation owner.

2. However, to manage your stored files that belong to the allocation, you need to communicate with the blobbers, thus the blobbers needs to know about the allocations they're involved in.

3. That's why the blobbers have a worker that synchronizes the allocation information with the blockchain periodically and store in the Blobber Metadata DB. The blobber uses the allocation data to authorize access and retrieve file references from the Blobber Metadata DB.

4. Also if you shared a file, the users who can access that file will be communicating with the blobbers to downloads and read the shared files (as discussed in the File Sharing page).

Allocation tokenomics

Allocation cost

Allocation cost is calculated as the write cost of the whole size of each data/parity shard on its corresponding blobber.

cost := currency.Coin(0)

// BlobberAllocations represent the relationship and terms between an allocation and a hosting blobber
for _, blobberAllocation := range allocation.BlobberAllocations {
    shardCost := currency.Multiply(blobberAllocation.WritePrice, blobberAllocation.Size)
    cost = currency.AddCoin(cost, shardCost)
}

return cost

Cancelation Charge

Cancelation charge applies on the user in the following 2 cases:

• At finalization/cancelation of the allocation.

• At removal of a blobber from the allocation.

In case of total finalization/cancelation, a base cancelation charge is calculated as a globally configured percentage of the allocation cost mentioned above.

cancelationCharge := currency.Multiply(cost, cancellationFraction)

To check all global storage configuration values, check the storage configuration endpoint in Sharders API.

If this base cancelation charge is less than the amount of used tokens of the Write Pool of the allocation (mainly used to pay for challenges) then no fee is deducted. Otherwise, a cancelation reward is then calculated for each blobber of the allocation as a percentage of the base cancelation charge based on its passing rate and write price, and then sent to them. The total of these calculated rewards will be the total of the cancelation charge that will be deducted from the allocation Write Pool.

totalCancelationCharge := currency.Coin(0)

for _, blobberAllocation := range allocation.BlobberAllocations {
    // totalWritePrice is the sum of write price for all BlobberAllocations
    blobberWritePriceWieght := float64(blobberAllocation.WritePrice) / float64(totalWritePrice)
    // passRate is how many challenges the blobber passes out of its total challenges that belong to this allocation
    cancelationReward = currency.Float64ToCoin(float64(cancelationCharge) * blobberWritePriceWeight * blobberAllocation.passRate)
    
    totalCancelationCharge = currency.AddCoin(totalCancelationCharge, cancelationReward)
}

return totalCancelationCharge

In case of blobber removal, the same calculation happens but taking into consideration just the blobber that is removed, not all allocation blobbers.

Operations on allocations

In this section, we'll be demonstrating the different operations that can be conducted on the allocations. We'll be using ZboxCLI as a client to communicate with the network, but you can use any communication method offered by Züs (like SDKs or public HTTP APIs).

If you want a simple yet powerful GUI to create and manage your allocations on Züs system, we encourage you to use Blimp or Vult apps of the Züs Ecosystem.

Creating an allocation

To create an allocation, you can use zboxcli newallocation command.

zbox newallocation --data <data-shards-count> --parity <parity-shards-count> ... 

Check ZboxCLI README for full usage docs of the command.

What happens under the hood is simply that:

• The client (in this case zboxcli, via Zboxcore SDK) computes the parameters of the allocation creation transaction using information from Global Storage Configuration. It also requests the chain for a list of ids of blobbers that are eligible to host the allocation. If the blobbers are not enough (they should be equal to parity shards + data shards), the creation request fails.

• After the client build the transaction data, it submits the transaction to the new_allocation_request function of the storage smart contract. The process of transaction submission is explained in more details in Using the Network > Smart Contracts.

• The smart contract function executes the following:

  • Validates the request:

    • Data and parity shards should be positive numbers.

    • Number of blobber ids provided should be equal to the sum of data and parity shards.

    • Price ranges (read/write) should be correct (min <= max).

    • Size of the allocation is not less than a globally configured allocation size minimum.

    • Checks all the blobbers are active.

    • Validating auth tokens for Restricted Blobbers, which are preferred blobbers that require authentication.

    • Checks if the locked tokens are enough to cover allocation cost.

  • Adds the allocation size to the blobber offers, which is then when a new allocation is being requested.

  • Create a new write pool for the allocation and add the lock tokens amount to it.

  • Creates and saves a new Challenge Pool to the MPT. Challenge pool can be thought as a pool or container for tokens that will be used to pay challenge reward to the blobbers passing challenges.

  • Inserts the allocation to the MPT.

• If the request is successful, the allocation ID is returned to the user, otherwise, the user will see the error. The allocation ID is set to the hash of the creation transaction.

Listing user's allocations

To list all your allocations, you can use zboxcli listalloctions command.

zbox listallocations

Check ZboxCLI README for full usage docs of the command.

What happens under the hood is:

The client (in this case zboxcli, via Zboxcore SDK) communicates with the Sharer API to retrieve a list of all the allocation objects associated with your wallet.

Updating an allocation

To update the allocation settings, you can use zboxcli updateallocation command.

zbox updateallocation

The owner can update the following about the allocation settings:

• Replace an existing blobber with a new one by mentioning the new blobber in the AddBlobberId field of the transaction data (check full listing in Storage S.C. page) and the one to remove in RemoveBlobberId. They should be provided together.

• Increase the size of the allocation, by setting the Size parameter of the transaction to a positive value.

• Change the file permissions options, by setting the FileOptions field of the transaction request.

• Make the allocation third-party-extendable, meaning non-owner clients can expand its size. You can do this by setting up SetThirdPartyExtendable of the transaction payload.

• Transfer allocation ownership (will be mentioned later).

Third party (non-owner) clients can also extend the size of an allocation if it's already set to be third-party-extendable by the owner.

What happens under the hood when trying to update the allocation is simply that the client (in this case zboxcli, via Zboxcore SDK) submits a transaction to the blockchain (as mentioned in Using the Network > Smart contracts) to the update_allocation_request function of the storage smart contract. The smart contract will validate the issuer of the function as well as the payload validity and run the required changes. The allocation txn hash is updated to the hash of the update transaction.

Finalizing an allocation

When the transaction is expired, either the owner or one of the blobbers of the allocation can issue a transaction to finalize the allocation. To finalize the allocation, you can zbox alloc-fini command:

zbox alloc-fini --allocation_id ...

Check ZboxCLI README for full usage docs of the command.

Finalizing the allocation includes:

• Settle all open challenges for the allocation, setting the unexpired ones to successful and the expired ones to failure, and updating the stats of the related blobbers.

• Unlock the locked tokens for write operations on this allocation for all the stake pools of the involved blobbers.

• Return all locked tokens in challenge pools to the owner's wallet.

• Pays all pending reward/penalty for the settled challenges to the involved blobbers.

• Pay cancelation charge if the the cancelation charge is larger than the amount of tokens payed for challenges.

• Update stats of the involved blobbers.

What happens under the hood is similar to the update allocation request, the client submit a transaction to the blockchain (as mentioned in Using the network > Smart contracts) to the finalize_allocation function of the Storage S.C. The smart contract will validate the issuer of the function as well as the payload validity and run the required changes.

Canceling an allocation

Canceling is similar to finalization, with a couple of differences:

• The cancelled allocation should NOT be expired, unlike finalization.

• Only the allocation owner can issue cancellation transaction to the allocation.

To cancel the allocation, you can use the zbox alloc-cancel command:

zbox alloc-cancel --allocation_id <allocation_id> ....

Check ZboxCLI README for full usage docs of the command.

Canceling the allocation have exactly the same effect on blobber balances and token pools as finalization mentioned above.

What happens under the hood is similar to the update allocation request, the client submit a transaction to the blockchain (as mentioned in Using the network > Smart contracts) to the cancel_allocation function of the Storage S.C. The smart contract will validate the issuer of the function as well as the payload validity and run the required changes.

Transfer an allocation to another account

For some use cases, users may need to exchange ownership of the allocation. Users can do this using the zbox transferallocation command:

zbox transferallocation --allocation <allocation_id> --new_owner <owner_id> --new_owner_key <new-owner-key> ....

Check ZboxCLI README for full usage docs of the command.

Transferring the ownership is simply an allocation update request to change the ownerId of the allocation. It uses the same smart contract function as Update allocation.