| PolarSPARC |
Hands-on with Garage
| Bhaskar S | 12/07/2025 |
Overview
The distributed object storage Simple Storage Service (or S3 for short) was made popular by Amazon in the late 2000s in their cloud offering. Ever since, it has gained traction even in the private cloud environments of the Enterprises.
The idea of the distributed object store S3 is to provide highly available, durable, scable, and performant object storage, where objects (documents, images, videos, etc) are stored into containers called Buckets.
Ever wondered if one can have such a storage setup in one's home lab ???
The answer is a resounding YES and this is where the open source distributed object store Garage comes to the rescue !!!
In other words, Garage is a highly performant, open source distributed object storage solution, which is compatible to the S3 offering from Amazon, with support for all the core features of S3.
Installation and Setup
The installation is on a Ubuntu 24.04 LTS based Linux desktop.
Ensure that Docker is installed and setup on the Linux desktop. Else, refer to the article HERE for help.
Assuming that we are logged in as alice and the current working directory is the home directory /home/alice, we will setup a directory structure by executing the following command in a terminal window:
$ mkdir -p /home/alice/garage{/meta,/data}
To pull and download the docker image for Garage (which includes the server and client), execute the following command:
$ docker pull dxflrs/garage:v2.1.0
The following would be the typical output:
v2.1.0: Pulling from dxflrs/garage bdaa3b9908c7: Pull complete Digest: sha256:4c9b34c113e61358466e83fd6e7d66e6d18657ede14b776eb78a93ee8da7cf6a Status: Downloaded newer image for dxflrs/garage:v2.1.0 docker.io/dxflrs/garage:v2.1.0
To install the AWS cli, execute the following command:
$ pip install awscli
To create a shared secret token for the node(s) in the cluster to communicate with each other via RPC, execute the following command:
$ openssl rand -hex 32
The following would be the typical output:
950db9bdc218ecff796a7e90337a3b3bd61e8a711d05ef1583277bfeac05a7d8
To create a secret token for the administrative access, execute the following command:
$ openssl rand -base64 32
The following would be the typical output:
eDjTaNDrzt316shbYST2uevZJ+i7PR6QQpb17EaIVZQ=
To check everything is ok with the docker image for Garage, execute the following command:
$ docker run --rm --name ps-garage dxflrs/garage:v2.1.0 /garage help
The following would be the typical output:
garage v2.1.0 [features: k2v, lmdb, sqlite, consul-discovery, kubernetes-discovery, metrics, telemetry-otlp, bundled-libs]
Structure for secret values or paths that are passed as CLI arguments or environment variables, instead of in the config
file
USAGE:
garage [OPTIONS] <SUBCOMMAND>
FLAGS:
--help Prints help information
-V, --version Prints version information
OPTIONS:
--admin-token <admin-token>
Admin API authentication token, replaces admin.admin_token in config.toml when running the Garage daemon
[env: GARAGE_ADMIN_TOKEN=]
--admin-token-file <admin-token-file>
Admin API authentication token file path, replaces admin.admin_token in config.toml and admin-token when
running the Garage daemon [env: GARAGE_ADMIN_TOKEN_FILE=]
--allow-world-readable-secrets <allow-world-readable-secrets>
Skip permission check on files containing secrets [env: GARAGE_ALLOW_WORLD_READABLE_SECRETS=]
-c, --config <config-file>
Path to configuration file [env: GARAGE_CONFIG_FILE=] [default: /etc/garage.toml]
--metrics-token <metrics-token>
Metrics API authentication token, replaces admin.metrics_token in config.toml when running the Garage daemon
[env: GARAGE_METRICS_TOKEN=]
--metrics-token-file <metrics-token-file>
Metrics API authentication token file path, replaces admin.metrics_token in config.toml and metrics-token
when running the Garage daemon [env: GARAGE_METRICS_TOKEN_FILE=]
-h, --rpc-host <rpc-host>
Host to connect to for admin operations, in the format: <full-node-id>@<ip>:<port> [env: GARAGE_RPC_HOST=]
-s, --rpc-secret <rpc-secret>
RPC secret network key, used to replace rpc_secret in config.toml when running the daemon or doing admin
operations [env: GARAGE_RPC_SECRET=]
--rpc-secret-file <rpc-secret-file>
RPC secret network key, used to replace rpc_secret in config.toml and rpc-secret when running the daemon or
doing admin operations [env: GARAGE_RPC_SECRET_FILE=]
SUBCOMMANDS:
admin-token Operations on admin API tokens
block Low-level node-local debug operations on data blocks
bucket Operations on buckets
convert-db Convert metadata db between database engine formats
help Prints this message or the help of the given subcommand(s)
json-api Directly invoke the admin API using a JSON payload. The result is printed to `stdout` in JSON
format
key Operations on S3 access keys
layout Operations on the assignment of node roles in the cluster layout
meta Operations on the metadata db
node Operations on individual Garage nodes
offline-repair Offline reparation of node data (these repairs must be run offline directly on the server
node)
repair Start repair of node data on remote node
server Run Garage server
stats Gather node statistics
status Get network status
worker Manage background workers
Let us assume that IP address of the Linux desktop is 192.168.1.25.
The following is the configuration file (in TOML format) located under /home/alice/garage:
# ### Garage TOML # replication_factor = 1 metadata_dir = "/var/lib/garage/meta" data_dir = "/var/lib/garage/data" rpc_bind_addr = "192.168.1.25:3901" rpc_public_addr = "192.168.1.25:3901" rpc_secret = "950db9bdc218ecff796a7e90337a3b3bd61e8a711d05ef1583277bfeac05a7d8" [s3_api] s3_region = "garage" api_bind_addr = "192.168.1.25:3900" [s3_web] bind_addr = "192.168.1.25:3902" root_domain = ".localhost" [admin] api_bind_addr = "192.168.1.25:3903" admin_token = "eDjTaNDrzt316shbYST2uevZJ+i7PR6QQpb17EaIVZQ="
This completes the installation and setup for the Garage hands-on demonstration.
Hands-on with Garage
For the hands-on demonstration, we will use Garage in a Single Node mode (versus deploying in a multi node cluster mode).
To run the Garage object storage service, execute the following command in the terminal window:
$ docker run --rm --name ps-garage --network host -u $(id -u $USER):$(id -g $USER) -v /home/alice/garage/garage.toml:/etc/garage.toml -v /home/alice/garage/meta:/var/lib/garage/meta -v /home/alice/garage/data:/var/lib/garage/data dxflrs/garage:v2.1.0
The following would be the typical output:
2025-12-07T17:04:42.731339Z INFO garage::server: Loading configuration... 2025-12-07T17:04:42.731577Z INFO garage::server: Initializing Garage main data store... 2025-12-07T17:04:42.731604Z INFO garage_model::garage: Opening database... 2025-12-07T17:04:42.731620Z INFO garage_db::lmdb_adapter: Opening LMDB database at: /var/lib/garage/meta/db.lmdb 2025-12-07T17:04:42.731865Z INFO garage_model::garage: Initializing RPC... 2025-12-07T17:04:42.731881Z INFO garage_model::garage: Initialize background variable system... 2025-12-07T17:04:42.731883Z INFO garage_model::garage: Initialize membership management system... 2025-12-07T17:04:42.731891Z INFO garage_rpc::system: Generating new node key pair. 2025-12-07T17:04:42.731970Z INFO garage_rpc::system: Node ID of this node: 57cea5a6f9b6b453 2025-12-07T17:04:42.731985Z INFO garage_rpc::layout::manager: No valid previous cluster layout stored (IO error: No such file or directory (os error 2)), starting fresh. 2025-12-07T17:04:42.732023Z INFO garage_rpc::layout::helper: ack_until updated to 0 2025-12-07T17:04:42.732110Z INFO garage_model::garage: Initialize block manager... 2025-12-07T17:04:42.732440Z INFO garage_model::garage: Initialize admin_token_table... 2025-12-07T17:04:42.732630Z INFO garage_model::garage: Initialize bucket_table... 2025-12-07T17:04:42.732808Z INFO garage_model::garage: Initialize bucket_alias_table... 2025-12-07T17:04:42.732992Z INFO garage_model::garage: Initialize key_table_table... 2025-12-07T17:04:42.733235Z INFO garage_model::garage: Initialize block_ref_table... 2025-12-07T17:04:42.733439Z INFO garage_model::garage: Initialize version_table... 2025-12-07T17:04:42.733633Z INFO garage_model::garage: Initialize multipart upload counter table... 2025-12-07T17:04:42.733833Z INFO garage_model::garage: Initialize multipart upload table... 2025-12-07T17:04:42.734017Z INFO garage_model::garage: Initialize object counter table... 2025-12-07T17:04:42.734233Z INFO garage_model::garage: Initialize object_table... 2025-12-07T17:04:42.734443Z INFO garage_model::garage: Load lifecycle worker state... 2025-12-07T17:04:42.734459Z INFO garage_model::garage: Initialize K2V counter table... 2025-12-07T17:04:42.734689Z INFO garage_model::garage: Initialize K2V subscription manager... 2025-12-07T17:04:42.734694Z INFO garage_model::garage: Initialize K2V item table... 2025-12-07T17:04:42.734897Z INFO garage_model::garage: Initialize K2V RPC handler... 2025-12-07T17:04:42.734944Z INFO garage::server: Initializing background runner... 2025-12-07T17:04:42.734951Z INFO garage::server: Spawning Garage workers... 2025-12-07T17:04:42.734999Z INFO garage_model::s3::lifecycle_worker: Starting lifecycle worker for 2025-12-07 2025-12-07T17:04:42.735004Z INFO garage::server: Initialize Admin API server and metrics collector... 2025-12-07T17:04:42.735327Z INFO garage_model::s3::lifecycle_worker: Lifecycle worker finished for 2025-12-07, objects expired: 0, mpu aborted: 0 2025-12-07T17:04:42.766245Z INFO garage::server: Launching internal Garage cluster communications... 2025-12-07T17:04:42.766268Z INFO garage::server: Initializing S3 API server... 2025-12-07T17:04:42.766272Z INFO garage::server: Initializing web server... 2025-12-07T17:04:42.766290Z INFO garage::server: Launching Admin API server... 2025-12-07T17:04:42.766322Z INFO garage_api_common::generic_server: S3 API server listening on http://192.168.1.25:3900 2025-12-07T17:04:42.766328Z INFO garage_web::web_server: Web server listening on http://192.168.1.25:3902 2025-12-07T17:04:42.766377Z INFO garage_api_common::generic_server: Admin API server listening on http://192.168.1.25:3903 2025-12-07T17:04:42.766376Z INFO garage_net::netapp: Listening on 192.168.1.25:3901
To set an alias for the Garage client command(s), execute the following command in a new terminal window (will refer to it as T2):
$ alias garage="docker exec -ti ps-garage /garage"
Executing the above command generates no output.
To check the status of the Garage node(s), execute the following command in the terminal window T2:
$ garage status
The following would be the typical output:
2025-12-07T17:05:39.299606Z INFO garage_net::netapp: Connected to 192.168.1.25:3901, negotiating handshake... 2025-12-07T17:05:39.340869Z INFO garage_net::netapp: Connection established to 57cea5a6f9b6b453 ==== HEALTHY NODES ==== ID Hostname Address Tags Zone Capacity DataAvail Version 57cea5a6f9b6b453 vader 192.168.1.25:3901 NO ROLE ASSIGNED v2.1.0
Before one can use Garage to create S3 bucket(s) or store object(s) in a bucket, one needs to first assign and then apply a Garage node's rack layout (what capacity, what datacenter zone, etc).
To initialize the layout of a specific node (using the node ID) with 1 GB of storage with a zone of dc1, execute the following command in the terminal window T2:
$ garage layout assign -z dc1 -c 1G 57cea5a6f9b6b453
The following would be the typical output:
2025-12-07T17:18:26.763097Z INFO garage_net::netapp: Connected to 192.168.1.25:3901, negotiating handshake... 2025-12-07T17:18:26.803850Z INFO garage_net::netapp: Connection established to 57cea5a6f9b6b453 Role changes are staged but not yet committed. Use `garage layout show` to view staged role changes, and `garage layout apply` to enact staged changes.
To apply the layout as a specific version, execute the following command in the terminal window T2:
$ garage layout apply --version 1
The following would be the typical output:
2025-12-07T17:19:55.519133Z INFO garage_net::netapp: Connected to 192.168.1.25:3901, negotiating handshake... 2025-12-07T17:19:55.559867Z INFO garage_net::netapp: Connection established to 57cea5a6f9b6b453 ==== COMPUTATION OF A NEW PARTITION ASSIGNATION ==== Partitions are replicated 1 times on at least 1 distinct zones. Optimal partition size: 3.9 MB Usable capacity / total cluster capacity: 1000.0 MB / 1000.0 MB (100.0 %) Effective capacity (replication factor 1): 1000.0 MB dc1 Tags Partitions Capacity Usable capacity 57cea5a6f9b6b453 [] 256 (256 new) 1000.0 MB 1000.0 MB (100.0%) TOTAL 256 (256 unique) 1000.0 MB 1000.0 MB (100.0%) New cluster layout with updated role assignment has been applied in cluster. Data will now be moved around between nodes accordingly.
To display information about the current layout of the single Garage node, execute the following command in the terminal window T2:
$ garage layout show
The following would be the typical output:
2025-12-07T17:22:11.748539Z INFO garage_net::netapp: Connected to 192.168.1.25:3901, negotiating handshake... 2025-12-07T17:22:11.789824Z INFO garage_net::netapp: Connection established to 57cea5a6f9b6b453 ==== CURRENT CLUSTER LAYOUT ==== ID Tags Zone Capacity Usable capacity 57cea5a6f9b6b453 [] dc1 1000.0 MB 1000.0 MB (100.0%) Zone redundancy: maximum Current cluster layout version: 1
To create a S3 bucket called spark-datasets, execute the following command in the terminal window T2:
$ garage bucket create spark-datasets
The following would be the typical output:
2025-12-07T17:23:20.090566Z INFO garage_net::netapp: Connected to 192.168.1.25:3901, negotiating handshake... 2025-12-07T17:23:20.131847Z INFO garage_net::netapp: Connection established to 57cea5a6f9b6b453 ==== BUCKET INFORMATION ==== Bucket: d9d5f73f64cf820732ebaccff0fd07b726f1e89a981ac669532e231fc1bf2525 Created: 2025-12-07 17:23:20.132 +00:00 Size: 0 B (0 B) Objects: 0 Website access: false Global alias: spark-datasets ==== KEYS FOR THIS BUCKET ==== Permissions Access key Local aliases
To list all the S3 bucket(s) in the single Garage node, execute the following command in the terminal window T2:
$ garage bucket list
The following would be the typical output:
2025-12-07T17:23:59.942798Z INFO garage_net::netapp: Connected to 192.168.1.25:3901, negotiating handshake... 2025-12-07T17:23:59.983833Z INFO garage_net::netapp: Connection established to 57cea5a6f9b6b453 ID Created Global aliases Local aliases d9d5f73f64cf8207 2025-12-07 spark-datasets
To display information about the S3 bucket spark-datasets, execute the following command in the terminal window T2:
$ garage bucket info spark-datasets
The following would be the typical output:
2025-12-07T17:25:22.659976Z INFO garage_net::netapp: Connected to 192.168.1.25:3901, negotiating handshake... 2025-12-07T17:25:22.700841Z INFO garage_net::netapp: Connection established to 57cea5a6f9b6b453 ==== BUCKET INFORMATION ==== Bucket: d9d5f73f64cf820732ebaccff0fd07b726f1e89a981ac669532e231fc1bf2525 Created: 2025-12-07 17:23:20.132 +00:00 Size: 0 B (0 B) Objects: 0 Website access: false Global alias: spark-datasets ==== KEYS FOR THIS BUCKET ==== Permissions Access key Local aliases
Before one can perform any operation on any S3 bucket, one needs to have an access key.
To create an access key called spark-datasets-key, execute the following command in the terminal window T2:
$ garage key create spark-datasets-key
The following would be the typical output:
2025-12-07T17:26:56.562798Z INFO garage_net::netapp: Connected to 192.168.1.25:3901, negotiating handshake... 2025-12-07T17:26:56.603968Z INFO garage_net::netapp: Connection established to 57cea5a6f9b6b453 ==== ACCESS KEY INFORMATION ==== Key ID: GK0cda775ac76876cd4317d737 Key name: spark-datasets-key Secret key: ae9c072c1edbb4b9018dc7c3f5fa63de1e9bd1526c26292cfb6a2b86f66f3e1a Created: 2025-12-07 17:26:56.604 +00:00 Validity: valid Expiration: never Can create buckets: false ==== BUCKETS FOR THIS KEY ==== Permissions ID Global aliases Local aliases
Make a note of the Key ID (GK0cda775ac76876cd4317d737) and the Secret key (ae9c072c1edbb4b9018dc7c3f5fa63de1e9bd1526c26292cfb6a2b86f66f3e1a).
To display information about the access key spark-datasets-key, execute the following command in the terminal window T2:
$ garage key info spark-datasets-key
The following would be the typical output:
2025-12-07T17:28:36.693999Z INFO garage_net::netapp: Connected to 192.168.1.25:3901, negotiating handshake... 2025-12-07T17:28:36.734839Z INFO garage_net::netapp: Connection established to 57cea5a6f9b6b453 ==== ACCESS KEY INFORMATION ==== Key ID: GK0cda775ac76876cd4317d737 Key name: spark-datasets-key Secret key: (redacted) Created: 2025-12-07 17:26:56.604 +00:00 Validity: valid Expiration: never Can create buckets: false ==== BUCKETS FOR THIS KEY ==== Permissions ID Global aliases Local aliases
To allow specific operations (read, write, etc) on using the access key called spark-datasets-key, execute the following command in the terminal window T2:
$ garage bucket allow --read --write --owner spark-datasets --key spark-datasets-key
The following would be the typical output:
2025-12-07T17:32:23.995073Z INFO garage_net::netapp: Connected to 192.168.1.25:3901, negotiating handshake... 2025-12-07T17:32:24.035835Z INFO garage_net::netapp: Connection established to 57cea5a6f9b6b453 ==== BUCKET INFORMATION ==== Bucket: d9d5f73f64cf820732ebaccff0fd07b726f1e89a981ac669532e231fc1bf2525 Created: 2025-12-07 17:23:20.132 +00:00 Size: 0 B (0 B) Objects: 0 Website access: false Global alias: spark-datasets ==== KEYS FOR THIS BUCKET ==== Permissions Access key Local aliases RWO GK0cda775ac76876cd4317d737 spark-datasets-key
Once again, to display information about the access key spark-datasets-key, execute the following command in the terminal window T2:
$ garage key info spark-datasets-key
The following would be the typical output:
2025-12-07T17:32:48.546789Z INFO garage_net::netapp: Connected to 192.168.1.25:3901, negotiating handshake... 2025-12-07T17:32:48.587860Z INFO garage_net::netapp: Connection established to 57cea5a6f9b6b453 ==== ACCESS KEY INFORMATION ==== Key ID: GK0cda775ac76876cd4317d737 Key name: spark-datasets-key Secret key: (redacted) Created: 2025-12-07 17:26:56.604 +00:00 Validity: valid Expiration: never Can create buckets: false ==== BUCKETS FOR THIS KEY ==== Permissions ID Global aliases Local aliases RWO d9d5f73f64cf8207 spark-datasets
We will now proceed to configure the AWS cli to use the appropriate access key details. Once AWS cli is configured, we will notice two files in the directory /home/alice/.aws - one called config and the other called credentials.
To configure the AWS cli, execute the following command in the terminal window T2:
$ aws configure
The following shows the interaction with the user and the details entered:
AWS Access Key ID [None]: GK0cda775ac76876cd4317d737 AWS Secret Access Key [None]: ae9c072c1edbb4b9018dc7c3f5fa63de1e9bd1526c26292cfb6a2b86f66f3e1a Default region name [None]: garage Default output format [None]: json
To list all the S3 buckets using the AWS cli, execute the following command in the terminal window T2:
$ aws s3 ls --endpoint-url http://192.168.1.25:3900
The following would be the typical output:
2025-12-07 12:23:20 spark-datasets
To copy a file called /tmp/iris.parquet the S3 bucket spark-datasets using the AWS cli, execute the following command in the terminal window T2:
$ aws s3 cp /tmp/iris.parquet s3://spark-datasets --endpoint-url http://192.168.1.25:3900
The following would be the typical output:
upload: /tmp/iris.parquet to s3://spark-datasets/iris.parquet
To list all object(s) in the S3 bucket spark-datasets using the AWS cli, execute the following command in the terminal window T2:
$ aws s3 ls s3://spark-datasets --endpoint-url http://192.168.1.25:3900
The following would be the typical output:
2025-12-07 13:32:07 2446 iris.parquet
To create a new S3 bucket called test-bucket using the AWS cli, execute the following command in the terminal window T2:
$ aws s3 mb s3://test-bucket --endpoint-url http://192.168.1.25:3900
The following would be the typical output:
make_bucket failed: s3://test-bucket An error occurred (AccessDenied) when calling the CreateBucket operation: Forbidden: Access key GK0cda775ac76876cd4317d737 is not allowed to create buckets
With this, we conclude the hands-on demonstration of how one can leverage Garage as an S3 object storage !!!
References