> ## Documentation Index
> Fetch the complete documentation index at: https://developers.stellar.org/llms.txt
> Use this file to discover all available pages before exploring further.

# Data Lake Storage Format

> How Galexie names and compresses the ledger files in a data lake, as specified by SEP-54.

Galexie stores ledger metadata in an object store as compressed XDR files.

Unless you are building for a highly specialized case, use the [ingest SDK](https://developers.stellar.org/docs/data/indexers/build-your-own/ingest-sdk) or [Stellar RPC](https://developers.stellar.org/docs/data/apis/rpc/admin-guide/data-lake-integration.md) rather than reading these files directly. Both tools hide the layout described here, so your application does not have to manage it.

Read this page if one of these applies to you:

* You read ledgers in bulk, for example for a backfill or an analytics job, and want to fetch objects from the bucket in parallel without an RPC server in the path.
* Your application is not written in Go, so it cannot use the ingest SDK, and you do not want to run Stellar RPC for history.
* You build a data store that other Stellar tools must be able to read.

[SEP-54: Ledger Metadata Storage](https://github.com/stellar/stellar-protocol/blob/master/ecosystem/sep-0054.md) is the specification for this format. This page summarizes the specification and shows the layout of the public Stellar data lake.

## Object keys

Galexie writes all ledger files under one configurable path in the bucket. Inside that path, each file sits in a partition directory:

```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
<ledgers-path>/<partition>/<batch>.xdr.zst
```

Galexie omits the partition directory when [`files_per_partition`](https://developers.stellar.org/docs/data/indexers/build-your-own/galexie/admin_guide/configuring#data-organization-optional) is `1`.

Both the partition name and the batch name start with 8 hexadecimal digits. That value is `0xFFFFFFFF` minus the first ledger sequence of the range. The subtraction reverses the order, so a listing returns the newest ledgers first. A client finds the latest ledger with one listing request and no index.

The partition name then repeats the first and last ledger sequence of the partition:

```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
FFFFFFFF--0-15
```

The batch name repeats the first and last ledger sequence of the batch. When a batch holds one ledger, the name carries that one sequence only:

```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
FFFFFFFD--2-3.xdr.zst
FFFFFFFD--2.xdr.zst
```

Put together, the full key of the first batch above is:

```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
<ledgers-path>/FFFFFFFF--0-15/FFFFFFFD--2-3.xdr.zst
```

> [!NOTE]
>
> A data store does not necessarily hold the whole ledger history. List the oldest partition to find where its history starts.

## File contents

Each ledger file is a [Zstandard](https://facebook.github.io/zstd/) compressed [`LedgerCloseMetaBatch`](https://github.com/stellar/stellar-xdr/blob/curr/Stellar-exporter.x) XDR value:

```cpp theme={"theme":{"light":"github-light","dark":"github-dark"}}
// Batch of ledgers along with their transaction metadata
struct LedgerCloseMetaBatch
{
    // starting ledger sequence number in the batch
    uint32 startSequence;

    // ending ledger sequence number in the batch
    uint32 endSequence;

    // Ledger close meta for each ledger within the batch
    LedgerCloseMeta ledgerCloseMetas<>;
};
```

The `ledgersPerBatch` value in the [configuration file](#configuration-file) sets the number of ledgers in each batch. Batch ranges start at sequence 0, but Galexie exports from ledger 2, so the first batch of a data store can hold fewer ledgers than the rest. Refer to the [XDR](https://developers.stellar.org/docs/learn/fundamentals/data-format/xdr.md) documentation for more information on the encoding.

## Configuration file

SEP-54 requires a JSON configuration object at `<ledgers-path>/.config.json`. A client reads it to learn the batch and partition sizes before it builds any object key.

| Property | Type | Description |
| - | - | - |
| `networkPassphrase` | string | The passphrase of the Stellar network that produced the ledgers. |
| `version` | string | The version of the configuration file schema. |
| `compression` | string | The compression algorithm. Only `zstd` is supported. |
| `ledgersPerBatch` | integer | The number of ledgers in each batch file. |
| `batchesPerPartition` | integer | The number of batch files in each partition directory. |

The Galexie [configuration file](https://developers.stellar.org/docs/data/indexers/build-your-own/galexie/admin_guide/configuring.md#data-organization-optional) names the last two values `ledgers_per_file` and `files_per_partition`.

Data stores written by Galexie versions before v23.0.0 do not hold this file, and their object keys end in `.xdr.zstd` instead of `.xdr.zst`. A client that reads one must get the batch and partition sizes from its own configuration. The Go `datastore` package does that when the file is absent.

For example, the Pubnet data store in the [public Stellar data lake](#the-public-stellar-data-lake) holds one ledger per batch and 64,000 batches per partition:

```json title="v1.1/stellar/ledgers/pubnet/.config.json" theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
  "networkPassphrase": "Public Global Stellar Network ; September 2015",
  "version": "1.0",
  "compression": "zstd",
  "ledgersPerBatch": 1,
  "batchesPerPartition": 64000
}
```

With those sizes, ledger 50000000 falls in the partition that starts at ledger 49984000, so its object key is:

```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
v1.1/stellar/ledgers/pubnet/FD054DFF--49984000-50047999/FD050F7F--50000000.xdr.zst
```

## The public Stellar data lake

A Stellar ledger data lake is available through the [AWS Open Data program](https://registry.opendata.aws/aws-public-blockchain). The bucket is public and needs no credentials. The Pubnet ledgers sit in one data store:

```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
s3://aws-public-blockchain/v1.1/stellar/ledgers/pubnet
```

Download ledger 50000000 from the example above and print it as JSON with [`stellar xdr decode`](https://developers.stellar.org/docs/tools/cli/stellar-cli#stellar-xdr-decode):

```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -s https://aws-public-blockchain.s3.amazonaws.com/v1.1/stellar/ledgers/pubnet/FD054DFF--49984000-50047999/FD050F7F--50000000.xdr.zst \
  | zstd -dc \
  | stellar xdr decode --type LedgerCloseMetaBatch --input single --output json-formatted
```

Testnet is reset regularly, and each reset starts a new data store. The data store for each reset sits in a directory named after the reset date:

```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
s3://aws-public-blockchain/v1.1/stellar/ledgers/testnet/<reset-date>
```

List `v1.1/stellar/ledgers/testnet/` to find the current reset directory, and use that directory as the `<ledgers-path>`. Some older reset directories have no `.config.json`. Read those as described in [Configuration file](#configuration-file).

## Object metadata

Galexie also attaches metadata to each ledger object. The metadata lets a client filter objects without downloading them. Data stores that cannot hold user-defined object metadata, such as a plain file system, omit it.

| Key | Description |
| - | - |
| `start-ledger` | The first ledger sequence in the batch. |
| `end-ledger` | The last ledger sequence in the batch. |
| `start-ledger-close-time` | The Unix timestamp of the close time of the first ledger. |
| `end-ledger-close-time` | The Unix timestamp of the close time of the last ledger. |
| `protocol-version` | The protocol version of the last ledger. |
| `core-version` | The version of Stellar Core that produced the ledgers. |
| `network-passphrase` | The passphrase of the Stellar network that produced the ledgers. |
| `compression-type` | The compression algorithm of the object. |
| `version` | The version of Galexie that wrote the object. |

On Amazon S3 these arrive as `x-amz-meta-` response headers:

```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -sI https://aws-public-blockchain.s3.amazonaws.com/v1.1/stellar/ledgers/pubnet/FD054DFF--49984000-50047999/FD050F7F--50000000.xdr.zst
```

## Data integrity

The format carries no proof that the ledgers in a data store are the ledgers the network closed. Verifying that is outside the scope of SEP-54. Read from a data store you run, or from an operator you trust.