S3FS Setup and Usage

S3FS allows you to mount an S3-compatible server as a file system on your local machine using the FUSE-based S3FS utility.

This section provides step-by-step instructions for setting up and using S3FS with your ZS3Server.

Video Tutorial

Step 1: Create the Password File

1. Create a .passwd-s3fs file containing your ACCESS_KEY_ID and SECRET_ACCESS_KEY:

   echo ACCESS_KEY_ID:SECRET_ACCESS_KEY > ${HOME}/.passwd-s3fs

   Example:

   echo test1:rootroot > ${HOME}/.passwd-s3fs

2. Update the file permissions for security:

   chmod 600 ${HOME}/.passwd-s3fs

Step 2: Create a Mount Point

1. Create a directory to use as the mount point:

   mkdir -p /mnt/s3fs

Step 3: Install Tools

Option A: Using MinIO Client (mc)

The MinIO Client allows you to interact with your MinIO server.

The MinIO Server provides an S3-compatible API and supports core S3 features. Choose one of the following installation methods for Linux:

RPM (RHEL-based Systems)

• Download the latest stable MinIO RPM package. Install the package using dnf:

wget https://dl.min.io/server/minio/release/linux-amd64/archive/minio-20241218131544.0.0-1.x86_64.rpm -O minio.rpm
sudo dnf install minio.rpm

DEB (Debian/Ubuntu Systems)

• Download the appropriate .deb package from the MinIO Downloads Page. Install the package using dpkg:

wget https://dl.min.io/server/minio/release/linux-amd64/archive/minio_20241218131544.0.0_amd64.deb -O minio.deb
sudo dpkg -i minio.deb

Binary Installation (Universal Method)

• Download the MinIO Server binary. Make the binary executable and move the binary to a directory in your system PATH:

wget https://dl.min.io/server/minio/release/linux-amd64/minio
chmod +x minio
sudo mv minio /usr/local/bin/

Installing MinIO Client (mc)

The MinIO Client allows command-line interaction with your MinIO server.

1. Download the MinIO Client binary:

   wget https://dl.min.io/client/mc/release/linux-amd64/mc

2. Make the binary executable:

   chmod +x mc

3. Move the binary to a directory in your system PATH:

   sudo mv mc /usr/local/bin/mc

4. Verify the installation:

   mc --version

5. Set an alias for your server:

mc alias set zcn https://<BLIMP_DOMAIN> <ACCESS_KEY> <SECRET_KEY> --api S3v2

• Replace <BLIMP_DOMAIN> with the domain link from the Blimp output.

• Replace <ACCESS_KEY> and <SECRET_KEY> with your credentials.

Example:

mc alias set zcn https://blimpibh5l.zus.network shahnawaz rootroot --api S3v2

1. Verify the alias setup by listing buckets:

mc ls zcn

Option B: Using AWS CLI

If you prefer AWS CLI over MinIO:

Install AWS CLI

Follow installation from AWS CLI Docs.

Set Environment Variables

export AWS_ACCESS_KEY_ID=devuser
export AWS_SECRET_ACCESS_KEY=Admin@123

Create a Bucket

aws --endpoint-url https://blimpsogec.zus.network s3 mb s3://s3fs

Step 4: Create and List Buckets

1. Use MinIO Client to create a bucket:

   ./mc mb zcn/s3fs

2. List the contents of the S3 server to verify:

   ./mc ls zcn

   The bucket s3fs should now be visible.

Step 5: Mount the Server

1. Mount the S3 bucket as a local file system:

   s3fs mybucket /path/to/mountpoint -o passwd_file=${HOME}/.passwd-s3fs -o url=https://url.to.s3/ -o use_path_request_style,allow_other,umask=000,complement_stat,nocopyapi,norenameapi

   Replace the placeholders:

   • mybucket with the name of your bucket (s3fs).

   • /path/to/mountpoint with /mnt/s3fs.

2. Example using Blimp URL:

   s3fs s3fs /mnt/s3fs -o passwd_file=${HOME}/.passwd-s3fs -o url=https://blimp7fyct.zus.network -o use_path_request_style,allow_other,umask=000,complement_stat,nocopyapi,norenameapi

Step 6: Verify the Mount

1. Check the mounted file system:

   df -ha

2. Navigate to the mount point:

   cd /mnt/s3fs
   ls

Step 7: Test the Setup

1. Create a file in the mounted directory:

   touch file1.txt

2. List the contents to verify the file:

   ls

Step 8: Verify on Blimp

1. Open the Blimp dashboard. Navigate to the bucket associated with the mounted directory (s3fs).

2. Confirm that the file (file1.txt) and other contents of the mounted directory are visible on Blimp.

Additional Examples

File Operations Examples

# Write a file
echo "I use Züs!" > /mnt/s3fs/my-tools.txt

# Copy a file
cp ~/my-files/testfile /mnt/s3fs/

# Move a file
mv ~/my-files/testfile /mnt/s3fs/

# Delete a file
rm /mnt/s3fs/testfile

All file operations will reflect in your ZS3 bucket via Blimp.

Docker Integration

Docker Run Example

docker run -v /mnt/s3fs/data/postgres:/var/lib/postgresql/data postgres:15

Docker Compose Example

services:
  postgres:
    image: postgres:15
    volumes:
      - /mnt/s3fs/data/postgres:/var/lib/postgresql/data

Additional Volume Mappings

• Grafana Loki Logs: /mnt/s3fs/loki

• Sharder Blocks: /mnt/s3fs/sharder${SHARDER}/data/blocks

• Miner RocksDB: /mnt/s3fs/miner${MINER}/data

Troubleshooting

Check Mount

df -ha | grep s3fs

Check Logs (Ubuntu)

tail -f /var/log/syslog | grep s3fs

Enable Debug Logging

s3fs s3fs /mnt/s3fs -o passwd_file=${HOME}/.passwd-s3fs -o dbglevel=info -f -o curldbg

Unmount ZS3

umount -l /mnt/s3fs

Additional Notes

• The S3FS utility uses FUSE (Filesystem in Userspace) to mount S3 servers as a local file system.

• The s3fs command includes options for security and compatibility:

  • -o passwd_file=${HOME}/.passwd-s3fs: Specifies the password file.

  • -o url=https://url.to.s3/: Points to the S3-compatible server URL.

  • -o use_path_request_style,allow_other,umask=000,complement_stat: Enables compatibility and allows other users to access the mounted file system.

• Compression: If you are using compression, it's recommended to use the MinIO Client for efficient file operations.