Two Commit

In a decentralized storage system, inconsistencies may arise when different blobbers maintain different states during a connection commit. This can happen when some blobbers successfully receive a write marker and commit a new state (S1), while others remain in the previous uncommitted state (S0), resulting in a broken state.

If an allocation repair is attempted in such a scenario, there may not be enough blobbers to reconstruct the committed state. Since committed changes are already merged, rolling back to a previous consistent state is not possible under the existing mechanism.

This issue may occur due to:

• Blobber crashes during a connection commit.

• Partial commit operations where only a subset of blobbers receives the commit request.

To solve this, we introduce a double-commit approach, which enables a rollback mechanism while maintaining consistency across blobbers.

Solution: Double-Commit Approach

To allow recovery of a previous correct state (S0) even if a new state (S1) has been partially committed, a new pre-committed state is introduced. This ensures that all blobbers have a synchronized state before finalizing a commit.

1. Introduce a pre-committed state (S1): Before finalizing a commit, changes are first stored in a pre-commit directory. The filesystem maintains a staged version that can be rolled back if needed.

2. Decoupling of Client-Blobber Interactions: Client-to-Blobber interactions (e.g., data uploads and connection commits) are now independent of Blobber-to-Blockchain interactions (e.g., write marker submissions and challenge completions).

3. Write Marker Chaining for Challenges: Blobbers store only the latest allocation version and do not maintain full historical versions. A rollback mechanism is implemented using empty write markers.

Implementation Details

Direct Flow (Normal Commit Process)

1. User uploads two files (f1, f2).

2. User commits with a write marker (W1). Files are moved to the pre-commit directory (version 1).

3. User uploads another file (f3) and deletes f2.

4. User commits with a new write marker (W2). f1 and f2 are moved to the file store with version 1. f3 and the deletion record of f2 (stored in the reference table) are moved to the pre-commit directory (version 2).

5. User uploads another file (f4).

6. User commits with write marker (W3). f2 is deleted from the file store, and f3 is moved to the main filesystem (version 2). f4 is moved to the pre-commit directory, updating the state to version 3.

Rollback Flow (Handling Inconsistencies)

1. User uploads two files (f1, f2).

2. User commits with write marker (W1). Files are moved to the pre-commit directory (version 1).

3. User uploads f3 and deletes f2.

4. User commits with write marker (W2). f1 and f2 are moved to the file store (version 1). f3 and the deletion of f2 (reference table record) are moved to the pre-commit directory (version 2).

5. User uploads f4.

6. User initiates a rollback using an empty write marker (W2e). The pre-commit directory is discarded, and the system reverts to version 1.

7. User uploads f3 again and commits with write marker (W3). The allocation is now correctly synchronized.

Allocation Versioning

To synchronize state across all blobbers, a common allocation version must be introduced. This is achieved using the timestamp of the write marker.

• Users must issue write markers with the exact same timestamp across blobbers when committing changes.

• The version of an allocation is determined by the write marker timestamp assigned to the pre-commit directory.

Empty Write Marker

An empty write marker is used to rollback an allocation to a previous committed state while ensuring blockchain integrity. It is a modified version of a write marker with zero file size.

type WriteMarker struct {
	AllocationRoot         string           `json:"allocation_root"` // previous allocation root
	PreviousAllocationRoot string           `json:"prev_allocation_root"` // previous allocation root (unchanged)
	FileMetaRoot           string           `json:"file_meta_root"` // unchanged
	AllocationID           string           `json:"allocation_id"` 
	Size                   int64            `json:"size"` // zero (empty marker)
	BlobberID              string           `json:"blobber_id"`
	Timestamp              common.Timestamp `json:"timestamp"` // must match the precommit timestamp
	ClientID               string           `json:"client_id"`
	Signature              string           `json:"signature"`
}

• The timestamp of the empty write marker must match the precommit directory version to be rolled back.

• After rollback, the last committed version is restored.

• The empty write marker is still committed to the blockchain, maintaining ledger integrity.

Challenge Protocol

The existing write marker chain model is replaced by a write marker tree structure to account for rollbacks.

• If a validator challenges a rolled-back allocation root, the blobber must provide a proof structure.

• The validator verifies that:

  • The blobber no longer holds the challenged allocation version.

  • The blobber has moved to a newer version based on a common ancestor.

• This ensures that rolled-back states are correctly handled without disrupting challenge validation.

Pre-Committed Section

Currently, temporary changes are stored in the getAllocTempDir directory. The new double-commit protocol introduces a getAllocPreCommitDir directory to maintain pre-committed states (S1) separately.

• When changes are pre-committed, they are moved from getAllocTempDir to getAllocPreCommitDir.

• This isolated storage allows rollbacks without affecting committed data.

Pre-Commit Event

The pre-commit event is the central mechanism in the double-commit protocol. It modifies the commit process as follows:

1. Instead of applying changes immediately, pre-commit stores them in getAllocPreCommitDir.

2. Changes are only finalized on commit when moving from pre-commit to the actual file system.

CommitWrite Process

When a client commits a write, the blobber performs two key actions:

1. Finalize the previous pre-committed data: Apply changes using connectionObj.ApplyChanges(), moving files from getAllocPreCommitDir to the main file store.

2. Pre-commit the current data: Move speculative changes from getAllocTempDir to getAllocPreCommitDir.

Rollback Mechanism

To revert to a previous committed state (S0), a rollback operation is introduced.

• Rollback removes the pre-committed state (S1) and restores the last committed state.

• Pre-committed changes are not permanently deleted, as they may be required for blockchain challenges.

• Instead, these changes are relocated from getAllocPreCommitDir to a temporary directory for potential use in validation.

Data Manipulation with Pre-Committed State

All file (/v1/file) and directory (/v1/dir) operations now interact with the pre-committed state instead of the previously merged state.

• The system builds file state based on connectionObj.ApplyChanges(), ensuring consistency across blobbers.

• This prevents inconsistencies caused by partial commits and ensures all blobbers maintain a synchronized state.