Pull Data

Hotdata pulls data from systems you connect — databases, warehouses, lakes, streams, and SaaS APIs. Add a source in your workspace, then create an ingest that loads tables from it into an instant database; for data your systems push directly, see Push Data.

Finding a source type

Every source has a type (postgres, stripe, buckets) that belongs to a family (sql, rest, filesystem, …). The family determines the shape of the connection config; the type supplies the vendor's defaults on top of it. The catalog lives in the CLI, not on this page:

hotdata ingest sources types            # every type, with its family
hotdata ingest sources types stripe     # one type: its family and description
hotdata ingest sources fields           # what each family asks for
hotdata ingest sources fields sql       # config, credentials, and selector fields

Creating a source names the family; on a terminal the rest is prompted for:

hotdata ingest sources add --family sql --display-name prod-orders

hotdata ingest sources types is the authoritative list — 168 types across the eight families below. The tables further down are the vendors Hotdata ships presets for; when a row here and the CLI disagree, the CLI is right.

Families

Family	Required config	Reads
`sql`	`dialect`	Tables in a SQL database or warehouse
`rest`	`base_url`	JSON resources from an HTTP API
`filesystem`	`provider`, `root_uri`	Parquet, CSV, and JSONL files in an object store
`iceberg`	—	Apache Iceberg tables via a REST, Glue, Polaris, or Unity catalog
`delta`	`provider`, `root_uri`	A Delta Lake table by URI
`ducklake`	`catalog`	Tables in a DuckLake catalog
`kafka`	`bootstrap_servers`	Kafka topics, including Debezium CDC change logs
`derived`	`source_database_id`	A table built from another table in your own workspace

A few familiar names sit in a different family than you might guess: Neon, Supabase, and PlanetScale are rest types that read each vendor's management API. To ingest the Postgres or MySQL database behind one of them, use the sql family with the postgres or mysql type.

See the CLI reference for the full source and ingest command surface.

Connection parameters

Every source is described in three parts. Config is the non-secret half of a connection — host, root URI, catalog. Credentials are the secret half, supplied separately so a token can be rotated without restating the connection. The selector lives on the ingest, not the source: it names the subset to read, which is how one source backs many ingests.

Config is fixed after creation — changing the server, root, or catalog a source points at means creating a new source rather than editing the existing one.

SaaS sources are rest presets: base_url and the auth shape come from the preset, and you supply only the credential it names — api_key, token, username/password, or client_id/client_secret. The per-source details for everything else live in the tables below; the full schema for any family is hotdata ingest sources fields <family>.

Databases

All sql types. Config takes host, port, database, default_schema, and engine-specific options; credentials are username and password, or a full connection_string DSN (postgresql://user:pass@host:5432/db) that replaces both. The selector is expressed through ingest create — --schema and a repeatable --table, a WHERE clause via --sql, or the engine's own dialect verbatim via --raw-sql.

Source	Description
	PostgreSQL — open-source relational database with strong SQL support and extensibility. Type `postgres`.
	Neon — serverless Postgres with branching and autoscaling. Type `neon` (`rest`) reads the management API; use `postgres` for the database itself.
	Supabase — managed Postgres, auth, storage, and APIs for applications. Type `supabase` (`rest`) reads the management API; use `postgres` for the database itself.
	MySQL — popular relational database for web and application workloads. Type `mysql`.
	PlanetScale — serverless MySQL platform with branching and connection pooling. Type `planetscale` (`rest`) reads the management API; use `mysql` for the database itself.
	MariaDB — open-source, MySQL-compatible relational database. Type `mariadb`.
	Microsoft SQL Server — enterprise relational database (MSSQL). Type `mssql`.
	Oracle Database — enterprise relational database for transactional and analytical workloads. Type `oracle`.

Data warehouses

Source	Description
	Snowflake — cloud data warehouse for large-scale analytics and data engineering. Type `snowflake`.
	Google BigQuery — serverless data warehouse for petabyte-scale analytics. Type `bigquery`; the credential is `credentials_json`, a service-account key, and `database` names the project.
	MotherDuck — serverless DuckDB in the cloud. Type `motherduck`; the credential is `motherduck_token`, and there is no host.
	Databricks — lakehouse platform for analytics, ETL, and AI. Type `databricks`; the credential is `access_token`.
	Amazon Redshift — cloud data warehouse on AWS. Type `redshift`.

Data lakes

Source	Description
	Apache Iceberg — open table format for large-scale data lakes. Type `iceberg`: config `catalog_config` (`uri`, `warehouse`, `region`); the credential is `token`, or `credential` as `client_id:client_secret`. The ingest selects `tables`, with an optional `row_filter`.
	DuckLake — DuckDB's open lakehouse format, storing table data as Parquet with catalog metadata in a SQL database. Type `ducklake`: config `catalog` (a connection string — the password travels separately as `catalog_password`) and optional `storage`. The ingest selects `tables`.
	Delta Lake — open table format storing table data as Parquet with a transaction log. Type `delta`: config `provider` and `root_uri`; credentials are the store's own. Each ingest selects one `table_path`.
	AWS Glue — Apache Iceberg tables registered in an AWS Glue Data Catalog. Type `glue`: supply `aws_access_key_id`/`aws_secret_access_key`, or omit them to use the ambient AWS credential chain.
	Apache Polaris — Apache Polaris / Snowflake Open Catalog over the Iceberg REST protocol. Type `polaris`; the credential is `token`, or `credential` as `client_id:client_secret`.
	Databricks Unity Catalog — Unity Catalog tables via its Iceberg REST endpoint. Type `unity_catalog`; the credential is `token`.
	Tigris — S3-compatible object storage. Connect it with the `buckets` type (`filesystem` family): `provider: s3`, plus an `endpoint_url` pointing at your Tigris endpoint.

Files, streams, and derived tables

These types have no vendor of their own — they read whatever you point them at.

Type	Family	Reads	Parameters
`buckets`	`filesystem`	Parquet, CSV, and JSONL files in S3, GCS, or Azure buckets	Config `provider` (`s3` \| `gs` \| `az`), `root_uri`, and for S3-compatible stores an `endpoint_url` and `region`. Credentials are the store's own — `aws_access_key_id`/`aws_secret_access_key`, `gs_token`, or the Azure account name and key; omit them for a public bucket. The ingest selects `file_format`, a `prefix` under the root, and a `glob` such as `*/.parquet`.
`parquet`	`filesystem`	Parquet files in S3, GCS, or Azure buckets	As `buckets`, with the format fixed to Parquet.
`kafka`	`kafka`	Kafka topics — a bounded pull of the currently-available messages	Config `bootstrap_servers`, with `security_protocol` and `sasl_mechanism` as the cluster requires; credentials `sasl_username`/`sasl_password`. The ingest selects `topics` and a `start` of `earliest` or `latest`.
`debezium`	`kafka`	Debezium CDC on Kafka. The envelope is unwrapped and the ingest lands a change log — one row per event, with `__op`, `__ts_ms`, and `__deleted` — not current state	As `kafka`.
`api`	`rest`	Any REST API: you describe the client and the resources it exposes	Config `base_url`, an auth shape, and non-secret `headers`; the credential matches the auth shape — `token`, `api_key`, `username`/`password`, or `client_id`/`client_secret`. The ingest selects `resources`, with an optional `where` filter and `limit`.
`derived`	`derived`	A table built from another table in your own workspace	Config `source_database_id` and `source_schema`; no credentials. The ingest selects a `table` and a `cursor_column` to resume on — `NOT NULL`, never decreasing — with a `tiebreaker_column` unless the cursor is unique per row.

Public & open data

Many open, government, scientific, and sample APIs ship as rest types — most need no credentials. Anything not listed here can be reached with the generic api type by describing the endpoints yourself.

Source	Description
	FRED — US economic series from the St. Louis Fed.
	Data.gov — US government open-data catalog: dataset search, orgs, publishers.
	US Treasury FiscalData — US national debt and Treasury interest rates.
	World Bank — World Bank indicators (GDP, population, ...).
	Finnhub — Stock market data and US symbols.
	CoinGecko — Crypto prices and market caps.
	CoinLore — Coinlore crypto tickers (no key).
	Coinbase Exchange — Coinbase spot market data (products, BTC-USD stats).
	NASA — NASA near-earth objects and picture of the day.
	openFDA — FDA drug adverse events and food recalls.
	USGS Earthquakes — Recent earthquakes worldwide (USGS).
	Open-Meteo — Weather forecasts, no key required.
	Carbon Intensity — UK grid carbon intensity (no key).
	OpenAlex — Scholarly papers and institutions (OpenAlex).
	GBIF — Biodiversity occurrences and species (GBIF).
	UK Police — UK police forces and neighbourhoods.
	Wikimedia Pageviews — Wikipedia most-viewed articles.
	Open Library — Open Library book search.
	Art Institute of Chicago — Art Institute of Chicago collection.
	Nager.Date — Public holidays worldwide (no key).
	RandomUser — Random user profiles (demo API).
	Hacker News — Hacker News stories and front page.
	TVmaze — TV shows directory (no key).
	Jikan — Top anime from MyAnimeList (no key).
	ESPN NBA — NBA scoreboard, teams and news (ESPN).
	TheSportsDB — Leagues and teams across sports (free tier).
	balldontlie — NBA stats: teams, players and games since 1946.
	Chess.com — Chess.com public player and streamer data.
	Open Brewery DB — US breweries directory (no key).
	Frankfurter — Daily FX exchange rates (ECB).
	Open Notify — ISS position and astronauts in space (no key).
	PokéAPI — Pokémon data (demo API).
	SWAPI — Star Wars films and characters (demo API).
	Rick and Morty — Rick and Morty characters (demo API).
	Cat Facts — Random cat facts (demo API).
	Fruityvice — Fruit nutrition facts (demo API).
	JSONPlaceholder — Fake posts and comments (demo API).
	DummyJSON — Fake products and users (demo API).