> ## Documentation Index
> Fetch the complete documentation index at: https://developers.stellar.org/llms.txt
> Use this file to discover all available pages before exploring further.

# Getting Started

> stellar-etl-airflow GitHub repository.

[stellar-etl-airflow GitHub repository](https://github.com/stellar/stellar-etl-airflow/tree/master)

## GCP Account Setup

[The SDF](https://developers.stellar.org/docs/learn/glossary.md#stellar-development-foundation-sdf) runs Hubble in GCP using Composer and BigQuery. To follow the same deployment you will need to have access to a GCP project. Instructions can be found in the [Get Started](https://cloud.google.com/docs/get-started) documentation from Google.

> [!NOTE]
>
> BigQuery and Composer should be available by default. If they are not you can find instructions for enabling them in the [BigQuery](https://cloud.google.com/bigquery?hl=en) or [Composer](https://cloud.google.com/composer?hl=en) Google documentation.

## Create GCP Composer Instance to Run Airflow

Instructions on bringing up a GCP Composer instance to run Hubble can be found in the [Installation and Setup](https://github.com/stellar/stellar-etl-airflow?tab=readme-ov-file#installation-and-setup) section in the [stellar-etl-airflow](https://github.com/stellar/stellar-etl-airflow) repository.

> [!NOTE]
>
> Hardware requirements can be very different depending on the Stellar network data you require. The default GCP settings may be higher/lower than actually required.

## Configuring GCP Composer Airflow

There are two things required for the configuration and setup of GCP Composer Airflow:

* Upload DAGs to the Composer Airflow Bucket
* Configure the Airflow variables for your GCP setup

For more detailed instructions please see the [stellar-etl-airflow Installation and Setup](https://github.com/stellar/stellar-etl-airflow?tab=readme-ov-file#installation-and-setup) documentation.

### Uploading DAGs

Within the [stellar-etl-airflow](https://github.com/stellar/stellar-etl-airflow) repo there is an [upload\_static\_to\_gcs.sh](https://github.com/stellar/stellar-etl-airflow/blob/master/upload_static_to_gcs.sh) shell script that will upload all the DAGs and schemas into your Composer Airflow bucket.

This can also be done using the [gcloud CLI or console](https://cloud.google.com/storage/docs/uploading-objects) and manually selecting the dags and schemas you wish to upload.

### Configuring Airflow Variables

Please see the [Airflow Variables Explanation](https://github.com/stellar/stellar-etl-airflow?tab=readme-ov-file#airflow-variables-explanation) documentation for more information about what should and needs to be configured.

## Running the DAGs

To run a DAG all you have to do is toggle the DAG on/off as seen below

![Toggle DAGs](https://mintcdn.com/stellardevelopmentfoundation/hmWBKBicbGG4DTu6/img/hubble/airflow_dag_toggle.png?fit=max\&auto=format\&n=hmWBKBicbGG4DTu6\&q=85\&s=2253809395f671705eede6478f252d5b)

More information about each DAG can be found in the [DAG Diagrams](https://github.com/stellar/stellar-etl-airflow?tab=readme-ov-file#dag-diagrams) documentation.

## Available DAGs

More information can be found [here](https://github.com/stellar/stellar-etl-airflow/blob/master/README.md#public-dags)

### History Table Export DAG

[This DAG](https://github.com/stellar/stellar-etl-airflow/blob/master/dags/history_tables_dag.py):

* Exports part of sources: ledgers, operations, transactions, trades, effects and assets from Stellar using the data lake of LedgerCloseMeta files
  * Optionally this can ingest data using captive-core but that is not ideal nor recommended for usage with Airflow
* Inserts into BigQuery

![](https://mintcdn.com/stellardevelopmentfoundation/nXKshiEJBuAoHF5T/img/hubble/history_table_export.png?fit=max\&auto=format\&n=nXKshiEJBuAoHF5T\&q=85\&s=c32b186b8820b093a2417a0ec051b7f8)

### State Table Export DAG

[This DAG](https://github.com/stellar/stellar-etl-airflow/blob/master/dags/state_table_dag.py)

* Exports accounts, account\_signers, offers, claimable\_balances, liquidity pools, trustlines, contract\_data, contract\_code, config\_settings and ttl.
* Inserts into BigQuery

![](https://mintcdn.com/stellardevelopmentfoundation/nXKshiEJBuAoHF5T/img/hubble/state_table_export.png?fit=max\&auto=format\&n=nXKshiEJBuAoHF5T\&q=85\&s=a097df8775d95c6e50c5d7db180ae49b)

### DBT Enriched Base Tables DAG

[This DAG](https://github.com/stellar/stellar-etl-airflow/blob/master/dags/dbt_enriched_base_tables_dag.py)

* Creates the DBT staging views for models
* Updates the enriched\_history\_operations table
* Updates the current state tables
* (Optional) warnings and errors are sent to slack.

![](https://mintcdn.com/stellardevelopmentfoundation/fqZrpgzoDdQoCWJz/img/hubble/dbt_enriched_base_tables.png?fit=max\&auto=format\&n=fqZrpgzoDdQoCWJz\&q=85\&s=18a28f777073fd92d0a0842139e2b489)