Panics/Errors

Debug the running network

ISSUES -

• Chain Stuck

  • sharder/miner issue

• Allocation failed to create

  • blobber issue

• Web-apps CORS issue

  • 0box is down if nginx configuration file is perfect.

SHARDER / MINER PANIC

• whenever sharder/miner fails with a panic it is marked as red and so further debugging is required.

• After 2-3 days sharder/miner might crashed due to panic which make chain stuck when all sharder/miner show same issue.

• sharder/miner can keep on restarting by themself after a panic which make chain slow & sometimes stuck when all sharder show same issue.

• sharder is not able to connect with postgres can make sharder unavailable

HOW TO CHECK CHAIN IS STUCK -

• Enter the network link from the browser.

• refresh the page 4-5 times and check the Round from the Round Health column as the number should keep on increasing.

• For sharder Healthcheck, you can use this script as well.

• For miner Healthcheck, you can use this script as well.

1. Login into the server

2. Enter the following command to check panic

docker logs << container_name >>

1. Share the panic with developers on Slack Channels.

BLOBBER & VALIDATOR PANIC

Use Zwalletcli for checking blobber details - https://github.com/0chain/zboxcli#list-blobbers

For Blobbers healthcheck, you can use this script.

The Problems we can face if blobber have issue -

• We can't create allocation or freestorage allocation.

• we can't upload data

Below error resembles there is some issue with blobbers -

not enough blobbers to honor the allocation

There can be two reasons due to which blobber can become unavailable -

1. Blobber Panic

For blobber Healthcheck, you can use this script as well.

• Login into the server

• Enter the following command to check panic

docker logs << container_name >>

• Share the panic with developers on Slack Channels.

1. Stake Blobbers

This section stakes blobber & validator of the network mentioned in the config.yaml file.

Below attached is the script for staking blobbers & validators -

#!/bin/bash
echo
echo "===================================================================="
echo "STAKING BLOBBERS AND VALIDATORS USING ZWALLETCLI & ZBOXCLI BINARIES"
echo "===================================================================="
echo
exec 6>&1 # Save stdout to file descriptor 6
function retry {
    local n=1
    local max=3
    local delay=5
    while true; do
    "$@" && break || {
        if [[ $n -lt $max ]]; then
            ((n++))
            echo "Command failed. Attempt $n/$max:"
            sleep $delay;
        else
            exec 1>&6 6>&- # Revert output to stdout instead of build.log and close file descriptor 6
            echo "failed to run $@"
            cat build.log && echo "::error title=Failed to stake the blobbers";
            echo "exiting blobber stake process..."
            exit 1
        fi
    }
    done
}
export HOME="/root"
retry ./zwallet create-wallet --configDir . --config ./config.yaml --wallet ../ignore > build.log 2>&1
for i in {1..12}
do
    retry ./zwallet faucet --methodName "pour" --input "{}" --tokens 9 --configDir . --config ./config.yaml --wallet ../ignore > build.log 2>&1
done
# Wait for 6 blobbers to register for 60 seconds...
count=0
while [[ $count -le 5 ]]
do
  retry ./zwallet getblobbers --silent --configDir . --config ./config.yaml --wallet ../ignore | grep -o "[a-z0-9]\{64\}" > build.log
  num=$(cat build.log | wc -l)
  echo "List Blobbers [$count] output:"
  cat build.log
  echo
  [[ $num -eq 6 ]] && break
  sleep 10
  ((count++))
done
if [[ $num -lt 6 ]]
then
  echo "::error title=Number of blobbers registered is less than 6:: Only ${num}/6 blobbers are registered. Exiting run..."
  cat build.log
  exit 1
fi
readarray -t blobber_ids < build.log
for id in "${blobber_ids[@]}"
do
    retry ./zbox sp-lock --blobber_id $id --tokens 5 --configDir . --config ./config.yaml --wallet ../ignore > build.log 2>&1
done

# Validator staking
count=0
while [[ $count -le 5 ]]
do
  retry ./zbox ls-validators --silent --configDir . --config ./config.yaml --wallet ../ignore | grep "id" | grep -o "[a-z0-9]\{64\}" > build.log
  num=$(cat build.log | wc -l)
  echo "List Validators [$count] output:"
  cat build.log
  echo
  [[ $num -eq 6 ]] && break
  sleep 10
  ((count++))
done
if [[ $num -lt 6 ]]
then
  echo "::error title=Number of Validators registered is less than 6:: Only ${num}/6 validators are registered. Exiting run..."
  cat build.log
  exit 1
fi
readarray -t validator_ids < build.log
for id in "${validator_ids[@]}"
do
    retry ./zbox sp-lock --validator_id $id --tokens 5 --configDir . --config ./config.yaml --wallet ../ignore > build.log 2>&1
done

0BOX PANIC

If the issue if 404 not found status code then 0box is not deployed on the server. However, if it's 502 Bad Gateway status code then 0box container crashed due to panic or db is not connected to it.

• Login into the server

• Enter the following command to check panic

docker logs << container_name >>

• Share the panic with developers on Slack Channels.