Storage Maintenance and Troubleshooting
How to keep your data safe and troubleshoot errors
Despite the great features that Züs offers to its customers, issues can occasionally happen to your allocation. This page should always come to the rescue! Here you'll find the possible issues your allocation can encounter and how you can take steps to resolve them.
Introduction to the allocation
The allocation in its essence is a contract of storage service between you (as a data owner) and the storage providers (blobbers) on Züs network, that will be hosting your data. The allocation defines the terms os storage service like:
Write price for your data.
Size of storage capacity allocated for your data.
Number of data shards hosting your data. Data shards are storage servers that your data will be split among those shards.
Number of parity shards used for your data. Parity shards are storage servers that will be used as backup if data shards fail.
Also the allocations is used in the blockchain transactions related to your data, as linkage between the storage system and the blockchain.
Using this approach, we can acheive high privacy, availability and durability of your data, using data and parity shards and splitting the blockchain system from the storage systems.
Synchronization failure among the blobbers
Problem Description
Sometimes the write operations you perform on your data (like upload, move, copy, delete or rename) are applied on some of the blobbers hosting your data but not the others, which makes the image of the allocation inconsistent among its blobbers. In this case, there are 2 scenarios:
The blobbers missed the operation is more than the number of your data shards + 1 extra redundant parity shard. In this scenario, the operation will be unsuccessful, and the blobbers who applied the operation will rollback.
The blobbers missed the operation is less than the number of your data shards + 1 extra redundant parity shards. In this scenario, the operation will be successful, however, the blobbers who missed the operation will have an inconsistent image of the allocation.
Detection (How to detect that this problem happened)
In the first scenario, the operation will fail but you'll not see anything related to the rollback process, since it happens automatically in the background.
In the second scenario, once you try to perform any other write operation, you'll see the following popup message
Resolution
Let's discuss the resolution in the previous 2 scenarios:
In the first scenario, you'll most probably not need to do anything, since the blobbers who applied the failed operation will rollback this process before it's committed.
In the second scenario, you'll need to repair the allocation, which is simply to download the latest changes on the blobbers which ran the process successfully and upload them on the blobbers who failed to apply. Once you enter blimp, you'll find a popup window notifying you that the allocation needs repair. You can either accept the repair, or be reminded later of it.
Recoverable allocation blobbers outage
Problem Description
This problem happens when a minority of the blobbers hosting your allocation are down. We define the minority as less or equal blobbers than the number of parity shards, so for example, if your allocation has 2 data shards and 3 parity shards, if 2 of the 5 blobbers are down, this is considered a minority, since 2 is less than the number of parity shards.
In this case, your allocation will still be functional, despite losing much of its power with each shard going down. Losing a shard makes the allocation less available and more prone to go out of service if more blobbers go down.
Problem Detection
Once you open one of the storage management apps (Vult, Blimp or Chalk), it will conduct an auto status check to your allocation. Based on this check, if your allocation suffers from this issue, you'll see a popup window as shown in the image below:
We don't have currently a method of notifying the allocation owners of such issues by emails or any other notification channel, that's why we strongly advise users to log into their accounts using the apps frequently, and we suggest once every 2 days.
Resolution
To fix such issue, you'll need to replace the failing blobbers with other healthy ones. Click "Replace Blobber" and you'll be redirected to the Replace Blobber page.
First, you'll need to determine the blobber you need to remove, you can choose only 1 blobber to replace at a time.
Next, you need to choose the new blobber to replace the removed blobber. You choose from just the blobbers managed by Züs, or just any blobber, using the Blobber brand selection field. You can also filter only the restricted blobbers that you are authorized to use, by toggling the button saying "Show Restricted Blobbers only".
After choosing the blobber you want, click "Confirm" to finish the replacement process.
Unrecoverable allocation blobbers outage
Problem Description
In contrast to the previous problem, this problem happens when majority of the blobbers hosting the allocation goes down, which we define as more blobbers than the number of parity shards of the allocation, causing the allocation to be completely nonfunctional, since all the write operations will fail, also puts your data in danger of being lost.
Problem Detection
Once you open one of the storage management apps (Vult, Blimp or Chalk), it will conduct an auto status check to your allocation. Based on this check, if your allocation suffers from this issue, you'll see a popup window as shown in the image below:
Resolution
We strongly advise our users not to leave their allocations without proper maintenance, and we suggest a check once every 2 days not to miss any warning.
Unfortunately, in this case, and since part of your data may have been lost, you will not be able to restore the allocation, or the part of your data that is lost. The resolution here is to cancel this allocation then open a new one. Click "Cancel Allocation" then "Confirm Cancellation" in the popup to cancel the allocation and pay the cancellation fees.
Last updated
