# Pull Data

Source: https://www.hotdata.dev/docs/pull-data
Site index: https://www.hotdata.dev/llms.txt

Hotdata pulls data from systems you connect — databases, warehouses, lakes, streams, and SaaS APIs. Add a **source** in your workspace, then create an **ingest** that loads tables from it into an instant database; for data your systems push directly, see [Push Data](/docs/push-data).

## Finding a source type

Every source has a **type** (`postgres`, `stripe`, `buckets`) that belongs to a **family** (`sql`, `rest`, `filesystem`, …). The family determines the shape of the connection config; the type supplies the vendor's defaults on top of it. The catalog lives in the CLI, not on this page:

```sh
hotdata ingest sources types            # every type, with its family
hotdata ingest sources types stripe     # one type: its family and description
hotdata ingest sources fields           # what each family asks for
hotdata ingest sources fields sql       # config, credentials, and selector fields
```

Creating a source names the family; on a terminal the rest is prompted for:

```sh
hotdata ingest sources add --family sql --display-name prod-orders
```

`hotdata ingest sources types` is the authoritative list — 168 types across the eight families below. The tables further down are the vendors Hotdata ships presets for; when a row here and the CLI disagree, the CLI is right.

### Families

| Family | Required config | Reads |
| :-- | :-- | :-- |
| `sql` | `dialect` | Tables in a SQL database or warehouse |
| `rest` | `base_url` | JSON resources from an HTTP API |
| `filesystem` | `provider`, `root_uri` | Parquet, CSV, and JSONL files in an object store |
| `iceberg` | — | Apache Iceberg tables via a REST, Glue, Polaris, or Unity catalog |
| `delta` | `provider`, `root_uri` | A Delta Lake table by URI |
| `ducklake` | `catalog` | Tables in a DuckLake catalog |
| `kafka` | `bootstrap_servers` | Kafka topics, including Debezium CDC change logs |
| `derived` | `source_database_id` | A table built from another table in your own workspace |

A few familiar names sit in a different family than you might guess: **Neon**, **Supabase**, and **PlanetScale** are `rest` types that read each vendor's management API. To ingest the Postgres or MySQL database behind one of them, use the `sql` family with the `postgres` or `mysql` type.

See the [CLI reference](/docs/cli-reference#sources) for the full source and ingest command surface.

## Connection parameters

Every source is described in three parts. **Config** is the non-secret half of a connection — host, root URI, catalog. **Credentials** are the secret half, supplied separately so a token can be rotated without restating the connection. The **selector** lives on the ingest, not the source: it names the subset to read, which is how one source backs many ingests.

Config is fixed after creation — changing the server, root, or catalog a source points at means creating a new source rather than editing the existing one.

SaaS sources are `rest` presets: `base_url` and the auth shape come from the preset, and you supply only the credential it names — `api_key`, `token`, `username`/`password`, or `client_id`/`client_secret`. The per-source details for everything else live in the tables below; the full schema for any family is `hotdata ingest sources fields <family>`.

## Databases

All `sql` types. Config takes `host`, `port`, `database`, `default_schema`, and engine-specific `options`; credentials are `username` and `password`, or a full `connection_string` DSN (`postgresql://user:pass@host:5432/db`) that replaces both. The selector is expressed through `ingest create` — `--schema` and a repeatable `--table`, a `WHERE` clause via `--sql`, or the engine's own dialect verbatim via `--raw-sql`.

| Source | Description |
| --- | --- |
| PostgreSQL | **PostgreSQL** — open-source relational database with strong SQL support and extensibility. Type `postgres`. |
| Neon | **Neon** — serverless Postgres with branching and autoscaling. Type `neon` (`rest`) reads the management API; use `postgres` for the database itself. |
| Supabase | **Supabase** — managed Postgres, auth, storage, and APIs for applications. Type `supabase` (`rest`) reads the management API; use `postgres` for the database itself. |
| MySQL | **MySQL** — popular relational database for web and application workloads. Type `mysql`. |
| PlanetScale | **PlanetScale** — serverless MySQL platform with branching and connection pooling. Type `planetscale` (`rest`) reads the management API; use `mysql` for the database itself. |
| MariaDB | **MariaDB** — open-source, MySQL-compatible relational database. Type `mariadb`. |
| Microsoft SQL Server | **Microsoft SQL Server** — enterprise relational database (MSSQL). Type `mssql`. |
| Oracle Database | **Oracle Database** — enterprise relational database for transactional and analytical workloads. Type `oracle`. |

## Data warehouses

| Source | Description |
| --- | --- |
| Snowflake | **Snowflake** — cloud data warehouse for large-scale analytics and data engineering. Type `snowflake`. |
| BigQuery | **Google BigQuery** — serverless data warehouse for petabyte-scale analytics. Type `bigquery`; the credential is `credentials_json`, a service-account key, and `database` names the project. |
| MotherDuck | **MotherDuck** — serverless DuckDB in the cloud. Type `motherduck`; the credential is `motherduck_token`, and there is no host. |
| Databricks | **Databricks** — lakehouse platform for analytics, ETL, and AI. Type `databricks`; the credential is `access_token`. |
| Amazon Redshift | **Amazon Redshift** — cloud data warehouse on AWS. Type `redshift`. |

## Data lakes

| Source | Description |
| --- | --- |
| Iceberg | **Apache Iceberg** — open table format for large-scale data lakes. Type `iceberg`: config `catalog_config` (`uri`, `warehouse`, `region`); the credential is `token`, or `credential` as `client_id:client_secret`. The ingest selects `tables`, with an optional `row_filter`. |
| DuckLake | **DuckLake** — DuckDB's open lakehouse format, storing table data as Parquet with catalog metadata in a SQL database. Type `ducklake`: config `catalog` (a connection string — the password travels separately as `catalog_password`) and optional `storage`. The ingest selects `tables`. |
| Delta Lake | **Delta Lake** — open table format storing table data as Parquet with a transaction log. Type `delta`: config `provider` and `root_uri`; credentials are the store's own. Each ingest selects one `table_path`. |
| AWS Glue | **AWS Glue** — Apache Iceberg tables registered in an AWS Glue Data Catalog. Type `glue`: supply `aws_access_key_id`/`aws_secret_access_key`, or omit them to use the ambient AWS credential chain. |
| Apache Polaris | **Apache Polaris** — Apache Polaris / Snowflake Open Catalog over the Iceberg REST protocol. Type `polaris`; the credential is `token`, or `credential` as `client_id:client_secret`. |
| Unity Catalog | **Databricks Unity Catalog** — Unity Catalog tables via its Iceberg REST endpoint. Type `unity_catalog`; the credential is `token`. |
| Tigris | **Tigris** — S3-compatible object storage. Connect it with the `buckets` type (`filesystem` family): `provider: s3`, plus an `endpoint_url` pointing at your Tigris endpoint. |

## Files, streams, and derived tables

These types have no vendor of their own — they read whatever you point them at.

| Type | Family | Reads | Parameters |
| :-- | :-- | :-- | :-- |
| `buckets` | `filesystem` | Parquet, CSV, and JSONL files in S3, GCS, or Azure buckets | Config `provider` (`s3` \| `gs` \| `az`), `root_uri`, and for S3-compatible stores an `endpoint_url` and `region`. Credentials are the store's own — `aws_access_key_id`/`aws_secret_access_key`, `gs_token`, or the Azure account name and key; omit them for a public bucket. The ingest selects `file_format`, a `prefix` under the root, and a `glob` such as `**/*.parquet`. |
| `parquet` | `filesystem` | Parquet files in S3, GCS, or Azure buckets | As `buckets`, with the format fixed to Parquet. |
| `kafka` | `kafka` | Kafka topics — a bounded pull of the currently-available messages | Config `bootstrap_servers`, with `security_protocol` and `sasl_mechanism` as the cluster requires; credentials `sasl_username`/`sasl_password`. The ingest selects `topics` and a `start` of `earliest` or `latest`. |
| `debezium` | `kafka` | Debezium CDC on Kafka. The envelope is unwrapped and the ingest lands a change log — one row per event, with `__op`, `__ts_ms`, and `__deleted` — not current state | As `kafka`. |
| `api` | `rest` | Any REST API: you describe the client and the resources it exposes | Config `base_url`, an auth shape, and non-secret `headers`; the credential matches the auth shape — `token`, `api_key`, `username`/`password`, or `client_id`/`client_secret`. The ingest selects `resources`, with an optional `where` filter and `limit`. |
| `derived` | `derived` | A table built from another table in your own workspace | Config `source_database_id` and `source_schema`; no credentials. The ingest selects a `table` and a `cursor_column` to resume on — `NOT NULL`, never decreasing — with a `tiebreaker_column` unless the cursor is unique per row. |

## Public & open data

Many open, government, scientific, and sample APIs ship as `rest` types — most need no credentials. Anything not listed here can be reached with the generic `api` type by describing the endpoints yourself.

| Source | Description |
| --- | --- |
| FRED | **FRED** — US economic series from the St. Louis Fed. |
| Data.gov | **Data.gov** — US government open-data catalog: dataset search, orgs, publishers. |
| US Treasury FiscalData | **US Treasury FiscalData** — US national debt and Treasury interest rates. |
| World Bank | **World Bank** — World Bank indicators (GDP, population, ...). |
| Finnhub | **Finnhub** — Stock market data and US symbols. |
| CoinGecko | **CoinGecko** — Crypto prices and market caps. |
| CoinLore | **CoinLore** — Coinlore crypto tickers (no key). |
| Coinbase Exchange | **Coinbase Exchange** — Coinbase spot market data (products, BTC-USD stats). |
| NASA | **NASA** — NASA near-earth objects and picture of the day. |
| openFDA | **openFDA** — FDA drug adverse events and food recalls. |
| USGS Earthquakes | **USGS Earthquakes** — Recent earthquakes worldwide (USGS). |
| Open-Meteo | **Open-Meteo** — Weather forecasts, no key required. |
| Carbon Intensity | **Carbon Intensity** — UK grid carbon intensity (no key). |
| OpenAlex | **OpenAlex** — Scholarly papers and institutions (OpenAlex). |
| GBIF | **GBIF** — Biodiversity occurrences and species (GBIF). |
| UK Police | **UK Police** — UK police forces and neighbourhoods. |
| Wikimedia Pageviews | **Wikimedia Pageviews** — Wikipedia most-viewed articles. |
| Open Library | **Open Library** — Open Library book search. |
| Art Institute of Chicago | **Art Institute of Chicago** — Art Institute of Chicago collection. |
| Nager.Date | **Nager.Date** — Public holidays worldwide (no key). |
| RandomUser | **RandomUser** — Random user profiles (demo API). |
| Hacker News | **Hacker News** — Hacker News stories and front page. |
| TVmaze | **TVmaze** — TV shows directory (no key). |
| Jikan | **Jikan** — Top anime from MyAnimeList (no key). |
| ESPN NBA | **ESPN NBA** — NBA scoreboard, teams and news (ESPN). |
| TheSportsDB | **TheSportsDB** — Leagues and teams across sports (free tier). |
| balldontlie | **balldontlie** — NBA stats: teams, players and games since 1946. |
| Chess.com | **Chess.com** — Chess.com public player and streamer data. |
| Open Brewery DB | **Open Brewery DB** — US breweries directory (no key). |
| Frankfurter | **Frankfurter** — Daily FX exchange rates (ECB). |
| Open Notify | **Open Notify** — ISS position and astronauts in space (no key). |
| PokéAPI | **PokéAPI** — Pokémon data (demo API). |
| SWAPI | **SWAPI** — Star Wars films and characters (demo API). |
| Rick and Morty | **Rick and Morty** — Rick and Morty characters (demo API). |
| Cat Facts | **Cat Facts** — Random cat facts (demo API). |
| Fruityvice | **Fruityvice** — Fruit nutrition facts (demo API). |
| JSONPlaceholder | **JSONPlaceholder** — Fake posts and comments (demo API). |
| DummyJSON | **DummyJSON** — Fake products and users (demo API). |

## Further reading

- [Core Concepts](/docs/core-concepts) — the object model and authentication.
- [Push Data](/docs/push-data) — the loading model, upload sequence, and load modes.
- [CLI Reference](/docs/cli-reference#sources) — the full source and ingest command surface.