ZS3 - Object Storage API Support
Add S3-compatible API to your allocation
Last updated
Add S3-compatible API to your allocation
Last updated
S3-compatible API to access object storage providers are now so widely adopted that it became a standard. That's why we offer support for S3-compatible API for accessing your Züs allocations, by deploying an intermediate server that provides an S3-compatible API and connects to the blobbers to get data from allocations to serve the S3 requests.
In this page we'll be discussing the flow of this feature, the structure of the components used to provide support for the S3-compatible API, the tools you can use to access this S3-compatible API and some use cases utilizing this API.
The main concepts behind supporting S3-compliant requests is mainly the ZS3 server, which is an intermediate server facing the client apps and accepting S3-compliant requests from them, then communicating with the blobbers using our Go SDK. This server should be owned and managed by the allocation owner, where each allocation should have a separate server. In this section we'll be going through the steps by which this feature is made possible:
Using Blimp webapp by Züs, you can create an allocation that supports S3-compliant API, and get directions on how you can deploy the ZS3-server on your own machines. For more information on this step, please use the guide of the Blimp app here.
If you want to deploy this server manually without using Blimp, check the guide in ZS3 Server repo.
Once your allocation is created and the server is deployed, you'll have a default account by which you can access your ZS3 server as admin. When you use Blimp to access your data using the S3-compatible methods (as mentioned in the next steps), you're prompted to add a secret key that you'll then use to access your data using CLI and APIs.
You may want to add more users who can use your ZS3 server. To do this, you'll need to use the Admins HTTP API exposed by the ZS3 server. This API is secure by the admin credentials, which you can find once you finish the creation of your ZS3 Server in "Using CLI" section, as part of the example curl command you can use to communicate with the HTTP API.
Example of this command (with the credentials part in double square parts)
Now that you have an example command with the credentials attached, let's see how you can add new users. All what you need to do is to send an HTTP request to the Add User endpoint of the Admin API of your server, using the preferred HTTP client. Here is an example using curl
Let's break down the command above:
curl is the tool we use to send the http request.
Since no HTTP Method is determined, GET is used by default.
https://blimpsogec.zus.network is the base url of your server, you'll find it in the example command in Blimp.
/minioclient/admin is the Admin endpoint, which accepts all requests related to the admin.
action=addUser is the action parameter which determines the action to be performed by the admin.
userAccessKey=newuser&userSecretKey=Strong_password are the parameters of this action, which are the access key and the secret key of the user to be added.
accessKey=devyetii&secretAccessKey=Admin@123 are the credentials of the admin.
If the request is successfully handled, the following response will be shown in the terminal:
Otherwise, you'll see a different response with the error message and "Success": false
More information about Admin API in the ZS3 Usage section.
The ZS3 server accepts S3-compliant requests from authorized users through the S3 API. The request should contain: the action name - the parameters of the action based on its definition - the credentials of the user.
Now you have 2 users, the admin user and the user you added in the step above. Only the admin credentials can be used to authorize access to the Admin API, but you can use the added credentials to access your data through the S3 API.
More information about S3 API in the ZS3 Usage section.
To access your data using the S3-compliant API, you can use either Blimp UI or CLI tools like mc or aws-cli or just send HTTP requests using curl , Postman or any other tool from you preference.
Let's check an example using curl to send a request to upload a file using the CreateObject action of the S3 API. Run the following command in the terminal in order to create a new bucket named "my-data"
Replace the URL and the keys with your own keys and run the command. If the request is successful, you'll see the following response:
S3 organizes files as objects contained inside buckets, however, blobbers store files in a typical directory-based tree-level file system. ZS3 does this translation between the two storage models, it turns users block-stroage-based requests into directory-based requests to the blobber. For example, when the user creates a bucket, ZS3 sends a request for the blobbers to create a root-level directory with the name of the bucket, and all requests based on this bucket will be sent to the blobber as a typical file system request of this file under the directory representing the bucket.
After doing the mapping, ZS3 Server communicates with the blobbers in order to fulfil the users requests. It uses GoSDK, the SDK Züs offers in Golang, to build the requests, which in turn uses the multi-operation Blobber API to upload, download and list storage content.
For more information on GoSDK, refer to the SDKs.
ZS3 server gathers the response from the blobbers (using GoSDK mentioned in the previous section), then responds to the user with the requested data.
These point conclude how ZS3 server makes it possible to access your allocation data using an S3-compatible API.
In the next section, we'll discuss how ZS3 is structured.
ZS3 Server is based on MinIO S3 server which provide a framework for S3 storage supporting. ZS3 server consists of:
MinIO server: the main component, receives requests from the users and maps them to the blobbers, usually on port 9000.
Logsearch API + Postgres DB: Acts as a real-time logs API which MinIO uses to push audit logs.
MinIO client: A test client used for testing the connection.
After deploying your ZS3 Server, you can use either of the following tools:
Once you finish deploying your ZS3 server via Blimp and creating your allocation, you'll be redirected to this page, which you can use to run S3 operations on your allocation. For more information on the operations of this page, check Blimp documentation here.
aws-cli is a tool provided by Amazon Web Services (AWS) as a native tool to access their resources, including AWS S3. Since aws-cli can be used to access AWS S3, it can also be used to access your ZS3 deployment, just by changing the URL of the service and adjusting the configuration as follows:
Run the following command and enter the prompted configuration as follows:
To run any command, add endpoint_url parameter as follows (example of the command to list contents of a bucket):
That's it! For more information about the operations you can perform using aws-cli check the README file of the ZS3 Server repo here.
mc - CLI toolmc (MinIO client) is a CLI tool associated with the MinIO platform, which provides a modern alternative to UNIX commands such as ls, cat, cp, mirror, diff etc. It supports both directory-based filesystems and Amazon S3 compatible cloud storage services.
Here are the steps to use mc CLI tool to access your ZS3 deployment:
First, you need to download and install it. Check this official guide to download it.
Then, configure it by setting an alias to your ZS3 server to simplify its usage later on:
To perform any file operation, use the following example:
That's it! For more information about the operations you can perform using mc check the README file of the ZS3 Server repo here.
You can connect directly to your ZS3 server using our HTTP API. Check the available actions and other parameters of the API here in the ZS3 Server repo.
Deployed with the ZS3 server is an external service, called Logsearch API, which gathers audit logs of the ZS3 server (written by MinIO) using Websockets and provides an API to simplify access to these logs.
This API is available on port 8080 on the machine you used to deploy the ZS3 server. For more information about the usage of this API, check the README file of this API in the ZS3 repo.
We offer 2 features that contribute to the robustness of the ZS3 server. First is the data encryption, which allows secure data storage on the blobbers by encrypting the data using the ZS3 server owner's wallet, ensuring secure storage. The other is compression which optimizes blobber communication requests by compressing the data (using lz4 compression algorithm) on the ZS3 server before sending it to the blobbers. To enable those features, use the configuration file for ZS3: .zcn/zs3server.json and edit it as follows:
Disaster recovery is process of replicating data from secondary to primary in case of primary failure. To set up disaster recovery, you need to have replication setup between primary and secondary zs3servers. In case of primary failure, you can use the following command to sync data from secondary to primary.
Check the following use cases to utilize the interoperability of the S3-compatible API to use your Züs allocation everywhere!
Backup and Restore: Backup and restore with better carbon footprint, security and performance. Check full use-case documentation here.
AI data lake: Use FUSE based S3 cloud storage to achieve performance and low egress cost. Check full use-case documentation here.
Private data vault: Prevent losing your business due to a breach and GDPR fine. Simply migrate and use Zus s3 instead. No code change. Check data migration documentation here.
Private cloud storage: Build your own storage cloud with just a few clicks instead.
