Data Lake Storage Format - Stellar Developer Docs

Galexie stores ledger metadata in an object store as compressed XDR files. Unless you are building for a highly specialized case, use the ingest SDK or Stellar RPC rather than reading these files directly. Both tools hide the layout described here, so your application does not have to manage it. Read this page if one of these applies to you:

You read ledgers in bulk, for example for a backfill or an analytics job, and want to fetch objects from the bucket in parallel without an RPC server in the path.
Your application is not written in Go, so it cannot use the ingest SDK, and you do not want to run Stellar RPC for history.
You build a data store that other Stellar tools must be able to read.

SEP-54: Ledger Metadata Storage is the specification for this format. This page summarizes the specification and shows the layout of the public Stellar data lake.

Object keys

Galexie writes all ledger files under one configurable path in the bucket. Inside that path, each file sits in a partition directory:

<ledgers-path>/<partition>/<batch>.xdr.zst

Galexie omits the partition directory when files_per_partition is 1. Both the partition name and the batch name start with 8 hexadecimal digits. That value is 0xFFFFFFFF minus the first ledger sequence of the range. The subtraction reverses the order, so a listing returns the newest ledgers first. A client finds the latest ledger with one listing request and no index. The partition name then repeats the first and last ledger sequence of the partition:

FFFFFFFF--0-15

The batch name repeats the first and last ledger sequence of the batch. When a batch holds one ledger, the name carries that one sequence only:

FFFFFFFD--2-3.xdr.zst
FFFFFFFD--2.xdr.zst

Put together, the full key of the first batch above is:

<ledgers-path>/FFFFFFFF--0-15/FFFFFFFD--2-3.xdr.zst

A data store does not necessarily hold the whole ledger history. List the oldest partition to find where its history starts.

File contents

Each ledger file is a Zstandard compressed LedgerCloseMetaBatch XDR value:

// Batch of ledgers along with their transaction metadata
struct LedgerCloseMetaBatch
{
    // starting ledger sequence number in the batch
    uint32 startSequence;

    // ending ledger sequence number in the batch
    uint32 endSequence;

    // Ledger close meta for each ledger within the batch
    LedgerCloseMeta ledgerCloseMetas<>;
};

The ledgersPerBatch value in the configuration file sets the number of ledgers in each batch. Batch ranges start at sequence 0, but Galexie exports from ledger 2, so the first batch of a data store can hold fewer ledgers than the rest. Refer to the XDR documentation for more information on the encoding.

Configuration file

SEP-54 requires a JSON configuration object at <ledgers-path>/.config.json. A client reads it to learn the batch and partition sizes before it builds any object key.

Property	Type	Description
`networkPassphrase`	string	The passphrase of the Stellar network that produced the ledgers.
`version`	string	The version of the configuration file schema.
`compression`	string	The compression algorithm. Only `zstd` is supported.
`ledgersPerBatch`	integer	The number of ledgers in each batch file.
`batchesPerPartition`	integer	The number of batch files in each partition directory.

The Galexie configuration file names the last two values ledgers_per_file and files_per_partition. Data stores written by Galexie versions before v23.0.0 do not hold this file, and their object keys end in .xdr.zstd instead of .xdr.zst. A client that reads one must get the batch and partition sizes from its own configuration. The Go datastore package does that when the file is absent. For example, the Pubnet data store in the public Stellar data lake holds one ledger per batch and 64,000 batches per partition:

v1.1/stellar/ledgers/pubnet/.config.json

{
  "networkPassphrase": "Public Global Stellar Network ; September 2015",
  "version": "1.0",
  "compression": "zstd",
  "ledgersPerBatch": 1,
  "batchesPerPartition": 64000
}

With those sizes, ledger 50000000 falls in the partition that starts at ledger 49984000, so its object key is:

v1.1/stellar/ledgers/pubnet/FD054DFF--49984000-50047999/FD050F7F--50000000.xdr.zst

The public Stellar data lake

A Stellar ledger data lake is available through the AWS Open Data program. The bucket is public and needs no credentials. The Pubnet ledgers sit in one data store:

s3://aws-public-blockchain/v1.1/stellar/ledgers/pubnet

Download ledger 50000000 from the example above and print it as JSON with stellar xdr decode:

curl -s https://aws-public-blockchain.s3.amazonaws.com/v1.1/stellar/ledgers/pubnet/FD054DFF--49984000-50047999/FD050F7F--50000000.xdr.zst \
  | zstd -dc \
  | stellar xdr decode --type LedgerCloseMetaBatch --input single --output json-formatted

Testnet is reset regularly, and each reset starts a new data store. The data store for each reset sits in a directory named after the reset date:

s3://aws-public-blockchain/v1.1/stellar/ledgers/testnet/<reset-date>

List v1.1/stellar/ledgers/testnet/ to find the current reset directory, and use that directory as the <ledgers-path>. Some older reset directories have no .config.json. Read those as described in Configuration file.

Object metadata

Galexie also attaches metadata to each ledger object. The metadata lets a client filter objects without downloading them. Data stores that cannot hold user-defined object metadata, such as a plain file system, omit it.

Key	Description
`start-ledger`	The first ledger sequence in the batch.
`end-ledger`	The last ledger sequence in the batch.
`start-ledger-close-time`	The Unix timestamp of the close time of the first ledger.
`end-ledger-close-time`	The Unix timestamp of the close time of the last ledger.
`protocol-version`	The protocol version of the last ledger.
`core-version`	The version of Stellar Core that produced the ledgers.
`network-passphrase`	The passphrase of the Stellar network that produced the ledgers.
`compression-type`	The compression algorithm of the object.
`version`	The version of Galexie that wrote the object.

On Amazon S3 these arrive as x-amz-meta- response headers:

curl -sI https://aws-public-blockchain.s3.amazonaws.com/v1.1/stellar/ledgers/pubnet/FD054DFF--49984000-50047999/FD050F7F--50000000.xdr.zst

Data integrity

The format carries no proof that the ledgers in a data store are the ledgers the network closed. Verifying that is outside the scope of SEP-54. Read from a data store you run, or from an operator you trust.

​Object keys

​File contents

​Configuration file

​The public Stellar data lake

​Object metadata

​Data integrity

Object keys

File contents

Configuration file

The public Stellar data lake

Object metadata

Data integrity