# Hotdata — all 59 pages > Give every agent an isolated, ephemeral environment to search, query, join, and analyze data without waiting on shared compute. Run SQL, vector, full-text, and geospatial workloads across your existing data platform. Generated from https://www.hotdata.dev. Each page below also answers on its own at its `Source:` URL with `.md` appended, and https://www.hotdata.dev/llms.txt lists them all without their contents. --- # Quick Start Source: https://www.hotdata.dev/docs/quick-start Site index: https://www.hotdata.dev/llms.txt Query any connected data source — from the terminal, a Python notebook, or inside an AI agent. Pick the integration that fits your workflow, then follow the steps below to authenticate and run your first query. [CLI](/docs/cli-reference) [Python SDK](/docs/python-sdk) [Rust SDK](/docs/rust-sdk) [Agent Skills](/docs/agent-skills) [Ibis](/docs/ibis) [LangChain](/docs/langchain) [dlt](/docs/dlthub) ## Fast path: create, load, and query ```bash hotdata databases create \ --name airbnb \ --catalog airbnb \ --table listings hotdata databases load \ --catalog airbnb \ --table listings \ --url https://hotdata.dev/data/sf-airbnb-listings.parquet hotdata query \ "SELECT COUNT(id) AS total_rows FROM airbnb.public.listings" ``` ## 1) Install the CLI Install walkthrough ([YouTube](https://youtu.be/LLo3A329FH4)): ```bash brew install hotdata-dev/tap/cli ``` Verify the installation: ```bash hotdata --help ``` ## 2) Authenticate Authenticate via browser: ```bash hotdata auth login ``` A browser window will open for you to sign in and authorize the CLI. Verify you're logged in: ```bash hotdata auth status ``` ## 3) Managed databases **Managed databases** are Hotdata-owned catalogs you populate with parquet files. Create them on demand, load data, query immediately, and delete when done. Managed database tutorial ([YouTube](https://www.youtube.com/watch?v=QMOURDIVgYo)): Create a database and declare the tables you plan to load: ```bash hotdata databases create \ --name mydb \ --catalog mydb \ --table orders \ --table customers ``` Load a parquet file from a local path or URL: ```bash # From a local file hotdata databases load \ --catalog mydb \ --table orders \ --file orders.parquet # From a URL hotdata databases load \ --catalog mydb \ --table orders \ --url https://hotdata.dev/data/sf-airbnb-listings.parquet ``` Query the loaded table — managed tables are addressed as `..`, where `` is the alias you set with `--catalog`: ```bash hotdata query \ "SELECT * FROM mydb.public.orders LIMIT 10" ``` List databases and their tables: ```bash hotdata databases list hotdata databases tables mydb ``` Delete a table or the whole database when you're done: ```bash hotdata databases tables remove orders --database mydb hotdata databases remove mydb ``` ## 4) Query your data ### Basic query ```bash hotdata query "SELECT id FROM mydb.public.orders LIMIT 5" ``` The database is resolved automatically from the catalog-qualified table name (`mydb.public.orders`), so no extra flag is needed. Use `-o table|json|csv` to change the output format, or `--database ` to target a specific managed database by its id. ### Analytical functions Window functions for rankings, running totals, and row comparisons: ```bash hotdata query " SELECT id, amount, sum(amount) OVER ( ORDER BY id ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW ) AS running_total FROM mydb.public.orders LIMIT 10 " ``` ```bash hotdata query " SELECT date, symbol, price, lag(price) OVER ( PARTITION BY symbol ORDER BY date ) AS prev_price FROM mydb.public.stock_prices " ``` ### Full-text search Create a [full-text index](/docs/cli-reference#search) on the text column, then search it by name — the search type is inferred from the index: ```bash hotdata search create articles_body \ --type text \ --from mydb.public.articles \ --column body hotdata search "machine learning" \ --index articles_body \ --select id,title,body \ --limit 10 ``` ### Vector search Create a [vector index](/docs/cli-reference#search) — `--provider` auto-embeds the text column server-side. Then search it by name; the server resolves the embedding model from the index metadata: Vector search demo ([YouTube](https://youtu.be/JEONO7N06-4)): ```bash hotdata search create documents_body \ --type vector \ --from mydb.public.documents \ --column body \ --provider hotdata search "documents about machine learning" \ --index documents_body \ --limit 10 ``` Equivalent SQL when you already have a query vector: ```bash hotdata query " SELECT id, title, l2_distance(embedding, ARRAY[0.1, -0.2, 0.5]) AS dist FROM mydb.public.documents ORDER BY dist ASC LIMIT 10 " ``` For cosine similarity, use `cosine_distance`; for inner product, use `negative_dot_product`. ## See also - [CLI Reference](/docs/cli-reference) — Full CLI documentation - [Agent Skills](/docs/agent-skills) — Let Claude Code and Cursor run hotdata commands for you - [API Reference](/docs/api-reference) — HTTP API for automation and integrations - [Data Sources](/docs/data-sources) — Supported ingest sources --- # Core Concepts Source: https://www.hotdata.dev/docs/core-concepts-overview Site index: https://www.hotdata.dev/llms.txt ## Introduction Hotdata gives you on-demand OLAP databases you can create, populate with parquet, query, and destroy — all through a single API call. There is no infrastructure to provision and no schema to migrate. The primary use case: an agent or application creates a database for a specific request, loads data into it, runs analytical queries (including vector search, full-text search, and geospatial), then discards the database when done. The whole lifecycle can happen in seconds. You can also pull data in from your existing databases and warehouses — Postgres, Snowflake, BigQuery, and others. Add the source once, then run an ingest that loads its rows into a managed database, where you query it like any other managed table. ## Two ways to get data in ### Managed databases (on demand) The fastest path. Create a database via the API, declare tables, upload parquet files, and start querying. Everything is provisioned on demand — there is no server to manage. ```bash # Create a database and load data in under 30 seconds hotdata databases create \ --catalog mydb \ --table orders hotdata databases load \ --catalog mydb --table orders \ --url https://example.com/orders.parquet hotdata query "SELECT COUNT(*) FROM mydb.public.orders" ``` Managed databases persist until you delete them, or until an optional `expires_at` you set at creation time. This makes them ideal for agent workflows, per-request analytics, and exploratory work where you need real compute on temporary data. See [CLI Reference — Databases](/docs/cli-reference#databases) and [API Reference — Databases](/docs/api-reference/databases). ### Ingest sources (existing systems) Pull data in from your existing databases and warehouses. Add an ingest source once — the config and credentials for an external system — then create an ingest that reads it and writes rows into a managed database. Once loaded, that data lives in the managed database and is queried the same way as any parquet you uploaded. Supported sources: Postgres, MySQL, Snowflake, BigQuery, MotherDuck, and more. See [Data Sources](/docs/data-sources). ```bash # Add a source, create an ingest into a managed database, then query it hotdata ingest sources add --family sql --display-name mydb-src hotdata ingest create \ --source mydb-src \ --sql "SELECT * FROM public.orders" \ --database-id # id from `databases create`; catalog alias mydb hotdata query "SELECT COUNT(*) FROM mydb.public.orders" ``` The ingested tables are addressed as `..
`, exactly like the tables you load from parquet. ## How it fits together ```text ╔═ hotdata ═══════════════════════════════════════╗ ╔══════════╗ ║ workspace ║░ ║ ║ API ║ ┏━━━━━━━━━━━━━━┓ ┏━━━━━━━━━━━━━━━┓ ║░ ║ client ║────▶║ ┃ ingest source┃ ingest ┃ managed db ┃ ║░ ║ ║ ║ ┃ (external) ┃━━━━━━━━━▶┃ (on demand) ┃ ║░ ╚══════════╝ ║ ┃ - postgres ┃ ┃ - parquet ┃ ║░ ║ ┃ - snowflake ┃ ┃ - ephemeral ┃ ║░ ║ ┃ - bigquery ┃ ┃ - any SQL ┃ ║░ ║ ┗━━━━━━━━━━━━━━┛ ┗━━━━━━━┳━━━━━━━┛ ║░ ║ ▼ ║░ ║ ┏━━━━━━━━━━━━━━━┓ ║░ ║ ┃ query engine ┃ ║░ ║ ┗━━━━━━━━━━━━━━━┛ ║░ ╚═════════════════════════════════════════════════╝░ ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ ``` Parquet uploads and ingested rows both land in a managed database; every query runs against managed databases. ## Organization A boundary for users and workspaces. All activity is scoped within an organization, including access control, resource limits, and usage tracking. Organizations isolate teams and environments while sharing a common governance layer. See [API Reference — Workspaces](/docs/api-reference/workspaces). ## Workspace An isolated execution environment provisioned on demand. Each workspace runs independently with its own compute, storage, and security boundary. Workspaces persist until explicitly deleted, allowing agents and applications to create and use them without affecting other workloads. See [API Reference — Workspaces](/docs/api-reference/workspaces) and [CLI Reference — Workspaces](/docs/cli-reference#workspaces). ## Managed Databases Hotdata-owned OLAP databases you create via the API, populate with parquet (or with rows pulled from an ingest source), and query immediately. Managed databases have no external dependency at query time — you define the schema, load the data, and Hotdata handles the rest. Key properties: - **Created on demand** — a single API call or CLI command is all you need - **Loaded from parquet** — upload a file or point to a URL; no schema migration required - **Any SQL** — analytical queries, window functions, vector search, full-text, geospatial, all in one engine - **Ephemeral or persistent** — set `expires_at` for automatic cleanup, or delete explicitly Tables inside a managed database are addressed as `..
` in SQL, where `` is the alias set with `--catalog` at create time. See [CLI Reference — Databases](/docs/cli-reference#databases) and [API Reference — Databases](/docs/api-reference/databases). ## Ingest sources & ingests An **ingest source** is a configuration for an external system (Postgres, Snowflake, BigQuery, SaaS APIs, and more) — its connection details and credentials. Sources are created within a workspace through controlled, read-only access, and each has a unique ID. An **ingest** reads from a source and writes rows into a managed database. You choose what to pull (a query, specific tables, or everything) and where it lands; a one-off ingest runs once, and a scheduled or continuous ingest keeps the destination table up to date. Nothing is queryable until it has been ingested — external data becomes a normal managed table once loaded. See [Data Sources](/docs/data-sources) and [CLI Reference — Ingest](/docs/cli-reference#ingest). ## Secrets Credentials used by ingest sources (passwords, tokens, API keys). Secrets are securely stored and scoped to a workspace. Values are never returned by read APIs and are injected only at execution time. This prevents leakage while allowing dynamic access to external systems. Supply them inline when you create an ingest source — see [Data Sources](/docs/data-sources). ## Saved Queries Reusable query definitions that can be executed multiple times. They capture logic without storing results, making them useful for standard transformations, recurring analysis, and agent workflows. Saved queries can be versioned and combined with managed databases to build repeatable patterns. See [API Reference — Saved Queries](/docs/api-reference/saved-queries). ## Persisted Results Every query result is automatically persisted as a parquet file. These results can be re-queried instantly, filtered, or joined without re-running the original query. This enables iterative workflows where each step builds on previous results. Persisted results also support time-based comparisons and replay. See [API Reference — Results](/docs/api-reference/results) and [CLI Reference — Results](/docs/cli-reference#query-run-history-results). ## Vector Search Uses USearch (HNSW) for approximate nearest-neighbor search over embeddings. SIMD-accelerated distance computation delivers high-throughput similarity search on CPUs, with no GPU dependency. Designed for real-time retrieval of embeddings (text, images, etc.) with low latency. See [SQL Reference — Vector search](/docs/sql#vector-search), [CLI Reference — Search](/docs/cli-reference#search), and [API Reference — Indexes](/docs/api-reference/indexes). ## Full text search Built-in BM25-ranked full-text retrieval. Text is indexed for fast evaluation of term relevance, with support for phrase matching, token weighting, and ranking across large text corpora. Eliminates the need for a separate search system while maintaining strong relevance and performance. See [SQL Reference — Full-text search](/docs/sql#full-text-search), [CLI Reference — Search](/docs/cli-reference#search), and [API Reference — Indexes](/docs/api-reference/indexes). ## Geospatial Queries Native support for spatial data types and operations such as distance calculations, containment checks, intersections, and bounding boxes. Enables location-aware filtering and joins within the same execution engine. Works alongside other query types, allowing spatial constraints to be combined with analytical, vector, and text queries. See [SQL Reference — Geospatial functions](/docs/sql#geospatial-functions). ## OLAP (Analytical Queries) Supports fast aggregations, filtering, and group-by operations over large datasets. Execution is vectorized and columnar, enabling efficient use of CPU and memory. Designed for analytical workloads where latency and throughput both matter. See [SQL Reference — Aggregate functions](/docs/sql-functions-aggregate), [SQL Reference — Window functions](/docs/sql-functions-window), and [API Reference — Query](/docs/api-reference/query). ## Hybrid Queries Combines multiple query types in a single execution plan. For example: full-text search → vector similarity → relational filtering → geospatial constraints → final row retrieval. This avoids coordinating multiple systems and keeps execution within a single low-latency path. See [API Reference — Query](/docs/api-reference/query) and [SQL Reference](/docs/sql). ## Joining Across Results Query results can be treated as tables and queried again. This allows joining across: - previous query outputs - different data sources - time-based snapshots This model supports iterative computation, where each step refines the result without recomputing from the original data. It enables complex workflows to be expressed as a sequence of lightweight, composable queries. See [API Reference — Query](/docs/api-reference/query), [API Reference — Databases](/docs/api-reference/databases), and [SQL Reference — SELECT syntax](/docs/sql#select-syntax). --- # Data Sources Source: https://www.hotdata.dev/docs/data-sources Site index: https://www.hotdata.dev/llms.txt Hotdata supports pulling data from a wide range of databases, warehouses, lakes, and SaaS. Add an ingest source in your workspace, then run an ingest to load tables from any of the sources below into a managed database. ## Databases | Source | Description | | --- | --- | | PostgreSQL | **PostgreSQL** — open-source relational database with strong SQL support and extensibility. | | Neon | **Neon** — serverless Postgres with branching and autoscaling. | | Supabase | **Supabase** — managed Postgres, auth, storage, and APIs for applications. | | MySQL | **MySQL** — popular relational database for web and application workloads. | | PlanetScale | **PlanetScale** — serverless MySQL platform with branching and connection pooling. | | MariaDB | **MariaDB** — open-source, MySQL-compatible relational database. | | Microsoft SQL Server | **Microsoft SQL Server** — enterprise relational database (MSSQL). | | Oracle Database | **Oracle Database** — enterprise relational database for transactional and analytical workloads. | ## Data warehouses | Source | Description | | --- | --- | | Snowflake | **Snowflake** — cloud data warehouse for large-scale analytics and data engineering. | | BigQuery | **Google BigQuery** — serverless data warehouse for petabyte-scale analytics. | | MotherDuck | **MotherDuck** — serverless DuckDB in the cloud. | | Databricks | **Databricks** — lakehouse platform for analytics, ETL, and AI. | | Amazon Redshift | **Amazon Redshift** — cloud data warehouse on AWS. | ## Data lakes | Source | Description | | --- | --- | | Iceberg | **Apache Iceberg** — open table format for large-scale data lakes (REST or AWS Glue catalog). | | DuckLake | **DuckLake** — DuckDB's open lakehouse format, storing table data as Parquet with catalog metadata in a SQL database. | | Delta Lake | **Delta Lake** — open table format storing table data as Parquet with a transaction log. | | Tigris | **Tigris** — S3-compatible object storage (buckets via S3-style credentials). | ## Productivity | Source | Description | | --- | --- | | Airtable | **Airtable** — low-code database and collaboration platform for bases, grids, and automations. | | Calendly | **Calendly** — scheduling and calendar integration for meetings and appointments. | | Coda | **Coda** — docs-as-apps platform combining documents, spreadsheets, and apps. | | Linear | **Linear** — issue tracking and project management for software teams. | | Monday | **Monday** — work OS for project management, workflows, and team collaboration. | | Notion | **Notion** — all-in-one workspace for notes, wikis, databases, and project management. | | Slack | **Slack** — messaging and collaboration for teams, channels, and integrations. | | Lattice | **Lattice** — people management and performance platform. | | Luma | **Luma** — event and community platform. | ## Data and analytics | Source | Description | | --- | --- | | Census | **Census** — reverse ETL and data activation from warehouse to business tools. | | Dask | **Dask** — parallel computing library for analytics at scale via distributed DataFrames. | | Datadog | **Datadog** — observability platform for metrics, logs, traces, and APM. | | Sentry | **Sentry** — error tracking and performance monitoring for applications. | | dbt Cloud | **dbt Cloud** — transformation layer and orchestration for the modern data stack. | | Enigma | **Enigma** — public data platform for commercial and government datasets. | | Metabase | **Metabase** — open-source BI and analytics for self-service dashboards and queries. | | New Relic | **New Relic** — full-stack observability for applications, infrastructure, and logs. | | Statsig | **Statsig** — experimentation and feature flags platform for product teams. | | Sumo Logic | **Sumo Logic** — cloud-native SIEM and log analytics for security and DevOps. | | AirNow | **AirNow** — EPA air quality and air pollution data API. | | Beam | **Beam** — data integration and ETL platform. | | Crunchbase | **Crunchbase** — company, investor, and funding data. | | Eppo | **Eppo** — experimentation and causal inference platform. | | Firebolt | **Firebolt** — cloud data warehouse for analytics. | | Fiserv | **Fiserv** — financial services and payment processing data. | | Fusegraph | **Fusegraph** — data platform and integration. | | Grants.gov | **Grants.gov** — US federal grants and funding opportunities. | | Marimo | **Marimo** — reactive Python notebooks for data science. | | Market API | **Market API** — market and financial data. | | Mapbox | **Mapbox** — maps, geocoding, and location APIs. | | MyParcel | **MyParcel** — shipping and logistics for e-commerce. | | Octopus Energy | **Octopus Energy** — energy and utility data. | | OpenCorporates | **OpenCorporates** — global company and corporate data. | | PropertyData | **PropertyData API** — real estate and property information. | | Regrid | **Regrid** — nationwide parcel, addressing, and land grid data. | | Sample CSV | **Sample CSV** — sample CSV datasets for testing (no auth). | | Yelp | **Yelp** — local business reviews and recommendations. | | Zillow | **Zillow** — real estate listings and property data. | | Airbyte | **Airbyte** — Airbyte connections, sources, destinations and jobs. | | Fivetran | **Fivetran** — Fivetran groups, connectors, users and roles. | | n8n | **n8n** — n8n workflows and executions. | ## Public & open data Many open, government, scientific, and sample APIs are available through the generic REST connector — most need no credentials. | Source | Description | | --- | --- | | FRED | **FRED** — US economic series from the St. Louis Fed. | | Data.gov | **Data.gov** — US government open-data catalog: dataset search, orgs, publishers. | | US Treasury FiscalData | **US Treasury FiscalData** — US national debt and Treasury interest rates. | | World Bank | **World Bank** — World Bank indicators (GDP, population, ...). | | Finnhub | **Finnhub** — Stock market data and US symbols. | | CoinGecko | **CoinGecko** — Crypto prices and market caps. | | CoinLore | **CoinLore** — Coinlore crypto tickers (no key). | | Coinbase Exchange | **Coinbase Exchange** — Coinbase spot market data (products, BTC-USD stats). | | NASA | **NASA** — NASA near-earth objects and picture of the day. | | openFDA | **openFDA** — FDA drug adverse events and food recalls. | | USGS Earthquakes | **USGS Earthquakes** — Recent earthquakes worldwide (USGS). | | Open-Meteo | **Open-Meteo** — Weather forecasts, no key required. | | Carbon Intensity | **Carbon Intensity** — UK grid carbon intensity (no key). | | OpenAlex | **OpenAlex** — Scholarly papers and institutions (OpenAlex). | | GBIF | **GBIF** — Biodiversity occurrences and species (GBIF). | | UK Police | **UK Police** — UK police forces and neighbourhoods. | | Wikimedia Pageviews | **Wikimedia Pageviews** — Wikipedia most-viewed articles. | | Open Library | **Open Library** — Open Library book search. | | Art Institute of Chicago | **Art Institute of Chicago** — Art Institute of Chicago collection. | | Nager.Date | **Nager.Date** — Public holidays worldwide (no key). | | RandomUser | **RandomUser** — Random user profiles (demo API). | | Hacker News | **Hacker News** — Hacker News stories and front page. | | TVmaze | **TVmaze** — TV shows directory (no key). | | Jikan | **Jikan** — Top anime from MyAnimeList (no key). | | ESPN NBA | **ESPN NBA** — NBA scoreboard, teams and news (ESPN). | | TheSportsDB | **TheSportsDB** — Leagues and teams across sports (free tier). | | balldontlie | **balldontlie** — NBA stats: teams, players and games since 1946. | | Chess.com | **Chess.com** — Chess.com public player and streamer data. | | Open Brewery DB | **Open Brewery DB** — US breweries directory (no key). | | Frankfurter | **Frankfurter** — Daily FX exchange rates (ECB). | | Open Notify | **Open Notify** — ISS position and astronauts in space (no key). | | PokéAPI | **PokéAPI** — Pokémon data (demo API). | | SWAPI | **SWAPI** — Star Wars films and characters (demo API). | | Rick and Morty | **Rick and Morty** — Rick and Morty characters (demo API). | | Cat Facts | **Cat Facts** — Random cat facts (demo API). | | Fruityvice | **Fruityvice** — Fruit nutrition facts (demo API). | | JSONPlaceholder | **JSONPlaceholder** — Fake posts and comments (demo API). | | DummyJSON | **DummyJSON** — Fake products and users (demo API). | ## Development & ops | Source | Description | | --- | --- | | CircleCI | **CircleCI** — CI and CD platform for building, testing, and deploying applications. | | Codecov | **Codecov** — code coverage reporting and analytics for test quality. | | Cloudflare | **Cloudflare** — DNS, CDN, security, and edge network services. | | Convex | **Convex** — reactive backend with real-time database and serverless functions. | | Cribl | **Cribl** — observability pipeline for routing, transforming, and controlling data. | | CrowdStrike | **CrowdStrike** — endpoint protection, threat intelligence, and incident response. | | GitHub | **GitHub** — source code hosting, version control, and software collaboration. | | Jira | **Jira** — issue tracking and project management for agile teams (Atlassian). | | LaunchDarkly | **LaunchDarkly** — feature flags and experimentation platform. | | Railway | **Railway** — deployment platform for applications and databases. | | Temporal | **Temporal** — durable workflow execution for microservices orchestration. | | Apache Kafka | **Apache Kafka** — distributed event streaming platform. | | Bugsnag | **Bugsnag** — error monitoring and application stability. | | Snyk | **Snyk** — developer-first application security and dependency scanning. | | Aikido Security | **Aikido Security** — cloud security posture and vulnerability management. | | Drata | **Drata** — compliance automation for SOC 2, HIPAA, and ISO. | | Ghostinspector | **Ghostinspector** — automated browser testing and monitoring. | | Incident.io | **Incident.io** — incident management and response platform. | | Instatus | **Instatus** — status page and downtime monitoring. | | Vanta | **Vanta** — security compliance and automation. | ## Customer and sales | Source | Description | | --- | --- | | Apollo | **Apollo** — sales intelligence and engagement platform for prospecting. | | Clay | **Clay** — data enrichment and prospecting for sales and recruiting. | | HubSpot | **HubSpot** — CRM, marketing, sales, and customer service platform. | | NetSuite | **NetSuite** — cloud ERP for finance, inventory, and operations. | | Twilio | **Twilio** — programmable messaging, voice, and customer engagement APIs. | | Zendesk | **Zendesk** — customer service software for support tickets and help centers. | ## HR and recruiting | Source | Description | | --- | --- | | Ashby | **Ashby** — recruiting OS, scheduling, and candidate pipeline for high-growth teams. | | BambooHR | **BambooHR** — HRIS, onboarding, and employee records for growing companies. | | Deel | **Deel** — global payroll, compliance, and contractor management. | | Greenhouse | **Greenhouse** — applicant tracking, structured hiring, and recruiting analytics. | | HiBob | **HiBob** — HR, time off, and people analytics for modern mid-size teams. | | Lever | **Lever** — recruiting ATS, nurture, and reporting for talent teams. | | Namely | **Namely** — HR, payroll, and benefits for mid-market organizations. | | Personio | **Personio** — HR, recruiting, and payroll for European businesses. | | Remote | **Remote** — global HR, payroll, and employer of record services. | | Workable | **Workable** — recruiting and applicant tracking for hiring teams. | ## Finance & billing | Source | Description | | --- | --- | | Brex | **Brex** — corporate cards, spend management, and business banking. | | Chargebee | **Chargebee** — subscription billing, invoicing, and revenue operations. | | Mercury | **Mercury** — banking and treasury for startups and growing companies. | | Plaid | **Plaid** — financial account linking, auth, and open banking data. | | Ramp | **Ramp** — corporate cards, bill pay, and expense automation. | | Stripe | **Stripe** — payments, billing, and financial infrastructure for the internet. | | Xero | **Xero** — cloud accounting, payroll, and small-business finance. | | Zuora | **Zuora** — subscription monetization, billing, and revenue recognition. | ## Legal | Source | Description | | --- | --- | | Clio | **Clio** — cloud practice management for law firms. | | DocuSign | **DocuSign** — electronic signatures and agreement lifecycle management. | | Dropbox Sign | **Dropbox Sign** — e-signatures and document workflows (HelloSign). | | Ironclad | **Ironclad** — contract lifecycle management and workflow automation. | | Litify | **Litify** — legal practice management and intake on Salesforce. | ## AI and ML | Source | Description | | --- | --- | | Cohere | **Cohere** — enterprise AI platform for embeddings and generation. | | Chroma | **Chroma** — embedding database and retrieval for LLM applications. | | Exa AI | **Exa AI** — neural search for semantic similarity and retrieval. | | LangDB | **LangDB** — database for AI applications and vector search. | | LangGraph | **LangGraph** — framework for building stateful, multi-actor AI agents. | | LangSmith | **LangSmith** — observability and debugging for LLM applications. | | Milvus | **Milvus** — open-source vector database for similarity search and AI. | | Modal | **Modal** — serverless GPU and compute for ML workloads. | | Pinecone | **Pinecone** — managed vector database for embeddings and retrieval. | | Qdrant | **Qdrant** — vector similarity search engine for ML applications. | | Weaviate | **Weaviate** — open-source vector database with hybrid and semantic search. | ## Blockchain and web3 | Source | Description | | --- | --- | | Algorand | **Algorand** — blockchain and cryptocurrency data. | | Bitcoin | **Bitcoin** — blockchain and cryptocurrency data. | | Open Ethereum | **Open Ethereum** — Ethereum blockchain data. | ## Search & content | Source | Description | | --- | --- | | Algolia | **Algolia** — hosted search API for applications and e-commerce. | | Firecrawl | **Firecrawl** — web scraping and content extraction API. | --- # CLI Reference Source: https://www.hotdata.dev/docs/cli-reference Site index: https://www.hotdata.dev/llms.txt ## Install **Homebrew** ```sh brew install hotdata-dev/tap/cli ``` **Shell (macOS, Linux)** ```sh curl -fsSL https://github.com/hotdata-dev/hotdata-cli/releases/latest/download/hotdata-cli-installer.sh | sh ``` **From source** (requires Rust) ```sh cargo install --path . ``` Or download a binary from [Releases](https://github.com/hotdata-dev/hotdata-cli/releases). Update with `hotdata manage upgrade`. ## Connect Authenticate via browser: ```sh hotdata auth login ``` This launches a browser window where you can sign in and authorize the CLI. To create a new account: ```sh hotdata auth register # GitHub OAuth; add --email for email + password ``` Check status or sign out: ```sh hotdata auth status hotdata auth logout ``` Alternatively, pass an API key directly: ```sh hotdata --api-key ``` Or set the `HOTDATA_API_KEY` environment variable (also loaded from `.env` files): ```sh export HOTDATA_API_KEY= hotdata ``` API key priority (lowest to highest): config file → `HOTDATA_API_KEY` env var → `--api-key` flag. ## Command reference The full command surface. The top level has eight groups — `auth`, `workspaces`, `databases`, `query`, `jobs`, `ingest`, `search`, and `manage`. Run `hotdata --help` for the complete flags on any of them. | Command | Description | | :-- | :-- | | `auth login` | Log in via browser | | `auth register` | Create a new account via browser (GitHub OAuth; `--email` for email + password) | | `auth logout` | Remove authentication for a profile | | `auth status` | Show authentication status | | `workspaces list` | List all workspaces | | `workspaces use` | Set the default workspace | | `databases list` | List managed databases in the workspace | | `databases count` | Count managed databases in the workspace | | `databases show` | Show details for a managed database | | `databases create` | Create a new managed database | | `databases fork` | Fork a database into a new, independent database | | `databases attach` | Attach a catalog so its tables are queryable | | `databases detach` | Detach a previously attached catalog | | `databases use` | Set the current (default) database | | `databases unset` | Clear the current database | | `databases remove` | Delete a database and all its tables | | `databases load` | Load a parquet file or saved result into a table | | `databases tables list` | List tables in a database | | `databases tables show` | Show column definitions for a table | | `databases tables load` | Load parquet/result into a table (create or replace) | | `databases tables remove` | Delete a table from a database | | `databases context list` | List named contexts in a database | | `databases context show` | Print context content to stdout | | `databases context pull` | Download context to `./.md` | | `databases context push` | Upload `./.md` as named context | | `databases query` | Execute a SQL query against a database | | `databases query status` | Check a running query and retrieve results | | `databases queries list` | List query runs | | `databases results get` | Show a stored query result by ID | | `databases results list` | List stored query results | | `query ""` | Execute a SQL query (shortcut for `databases query`) | | `query status` | Check a running query and retrieve results | | `jobs list` | List background jobs (active by default) | | `jobs ` | Show one background job | | `ingest create` | Create a load definition | | `ingest list` | List the ingests in the workspace | | `ingest show` | Show one ingest: state, selector, destination, schedule | | `ingest pause` | Stop an ingest (cancel the active run and future runs) | | `ingest resume` | Clear a stop and let the schedule dispatch again | | `ingest schedule` | Change when a scheduled/continuous ingest runs next | | `ingest logs` | List the runs of one ingest | | `ingest run` | Show one run: status, snapshots, timings | | `ingest remove` | Delete an ingest and release its destination table | | `ingest sources test` | Check a config and credentials without creating anything | | `ingest sources add` | Create a datasource and its first config version | | `ingest sources list` | List the datasources in the workspace | | `ingest sources show` | Show one datasource: state, config, discovery | | `ingest sources update-config` | Append a config version (rotate credentials) | | `ingest sources remove` | Delete a datasource | | `ingest sources types` | Browse the catalog of source types | | `ingest sources fields` | Show the fields a source family accepts | | `search "" --index ` | Run a full-text or vector search against an index | | `search create` | Create a search index over a table column | | `search list` | List search indexes | | `search show` | Show one search index by name | | `search remove` | Remove a search index by name | | `search embeddings list` | List embedding providers | | `search embeddings show` | Show one embedding provider | | `search embeddings add` | Create a new embedding provider | | `search embeddings update` | Update an embedding provider | | `search embeddings remove` | Delete an embedding provider | | `manage usage` | Show workspace usage: queries, bytes scanned, stored bytes | | `manage completions` | Generate shell completions (`bash`, `zsh`, `fish`) | | `manage upgrade` | Upgrade the CLI to the latest release | | `manage skills install` | Install/update the agent skill into agent directories | | `manage skills status` | Show the agent skill's installation status | | `manage skills list` | List installed skills (alias for `status`) | ## Global options | Option | Description | | :-- | :-- | | `--api-key ` | API key (overrides env var and config) | | `--no-input` | Disable interactive prompts; error instead | | `-v, --version` | Print version | | `-h, --help` | Print help | Most commands also accept `-w, --workspace-id ` and `-o, --output ` (query, search, and results also support `csv`). ## Workspaces ```sh hotdata workspaces list hotdata workspaces use [] ``` - `list` shows all workspaces with a `*` marker on the active one. - `use` switches the active workspace. Omit the ID for interactive selection. - The active workspace is the default for all commands that accept `-w`. ## Databases Managed databases are Hotdata-owned catalogs you populate with parquet files. Tables are addressed as `..
` in SQL, where `` is the alias set with `--catalog` at create time. ```sh hotdata databases list [--limit ] [--cursor ] [-o table|json|yaml] hotdata databases count hotdata databases show hotdata databases create \ [--catalog ] \ [--name
`, or `..
` when `--alias` is omitted. - `-d/--database` selects the database to attach into (defaults to the current database). - `detach` removes a previously attached catalog. ### Load parquet into a table ```sh hotdata databases load --catalog --table
--file hotdata databases load --catalog --table
--url hotdata databases load --catalog --table
--upload-id hotdata databases load --catalog --table
--result-id ``` - `--catalog` is the alias set at create time; `--schema` defaults to `public`. - Sources: `--file` uploads from a local path; `--url` downloads a remote parquet file; `--upload-id` uses a pre-staged upload from `POST /v1/uploads`; `--result-id` loads a saved query result (which must belong to the target database). - Load replaces the table contents on each call. ### Manage tables ```sh hotdata databases tables [] hotdata databases tables list [--database ] [--schema

] [--table

] [--limit ] hotdata databases tables show hotdata databases tables load

[--database ] [--schema ] (--file|--url|--upload-id|--result-id) hotdata databases tables remove
[--database ] [--schema ] ``` - `tables` (or `tables list`) lists the tables in a database; `--database` defaults to the current database. - `show` prints column definitions for `catalog.schema.table` (or `schema.table` with an active database). - `tables load` creates or replaces a table from parquet or a saved result, addressing the database by `--database` rather than by `--catalog`. - `tables remove` drops a table from the database. ### Context Sync named Markdown context files with a managed database — persistent notes and schema documentation scoped to a database, useful for giving agents durable context. ```sh hotdata databases context list [-d ] [--prefix

] hotdata databases context show [-d ] hotdata databases context pull [--force] [--dry-run] [-d ] hotdata databases context push [--dry-run] [-d ] ``` - `pull` downloads context to `./.md` (`--force` overwrites an existing file); `push` uploads `./.md` to the database as named context. - `` follows SQL identifier rules; a trailing `.md` is ignored (e.g. `USER.md` → `USER`). ### Query run history & results ```sh hotdata databases queries list [--status ] [--limit ] [--cursor ] hotdata databases queries hotdata databases results list [--limit ] [--offset ] hotdata databases results get [-o table|json|csv] ``` - `queries list` shows past runs with status, timing, row count, and a truncated SQL preview (default limit 20); `--status` filters by run status (comma-separated, e.g. `running,failed`). View a run by ID for full metadata (timings, `result_id`, SQL). - Every query result is stored automatically — retrieve rows with `results get ` (the `result-id` printed in a query's footer) without re-running the query. ## Query `query` is a top-level shortcut into `databases query`. ```sh hotdata query "" \ [-d ] \ [--dialect hotsql|duckdb|postgres|snowflake] \ [-o table|json|csv] hotdata query status ``` - Default output is `table`, which prints results with row count and execution time. - `-d/--database` runs against a specific managed database (defaults to the current database set via `databases use`). It is sent as the `X-Database-Id` header, so it takes the database **id** — it does not resolve a name or catalog alias. - `--dialect` names the SQL the query is written in. Anything other than `hotsql` (the default) is transpiled to HotSQL server-side before it runs — read-only queries only. See [SQL dialects](/docs/sql#sql-dialects). - Long-running queries fall back to async execution and return a `query_run_id`. Poll it with `hotdata query status `. - Exit codes for `query status`: `0` succeeded, `1` failed, `2` still running (poll again), `3` succeeded but the result is a truncated preview. ## Ingest An **ingest** is a saved load definition: it reads from an external **source** and writes rows into a managed database. Add the source first (`hotdata ingest sources add`), then create an ingest that reads it. Selector and destination are fixed at creation; creating an ingest starts nothing — the scheduler dispatches every run, so watch it with `hotdata ingest logs ` rather than expecting a run id back. ```sh # Create a load definition (source + what to read + where it lands) hotdata ingest create --source --sql "SELECT * FROM .

" --database-id hotdata ingest create --source --all --database-id hotdata ingest create --source --table ... --type scheduled --every 1h --database-id # Inspect and control ingests hotdata ingest list [--datasource-id ] [--type ] [--state ] hotdata ingest show hotdata ingest logs [] [--status ] [--wait] [--wait-timeout ] hotdata ingest run [--wait] hotdata ingest pause hotdata ingest resume hotdata ingest schedule [--every ] [--next now|] hotdata ingest remove ``` - `create` needs a `--source` (a `ds_…` id or a display name) and a selector. Selector shorthands: `--sql` (SQL sources: `SELECT FROM [.]
[WHERE …] [LIMIT n]`), `--raw-sql` (a verbatim query in the source's own dialect), `--all`, `--table`/`--topic`/`--table-path` (per family), or `--selector` for raw family JSON. Destination flags: `--database-id`, `--dest-table` / `--dest-table-prefix`, `--dest-schema` (default `public`), and `--write-mode replace|upsert`. `--type` is `one-time` (default), `scheduled`, or `continuous` (`--stream`); recurring types need `--every` or `--schedule`. - There is no `run-now` verb — bring a run forward with `hotdata ingest schedule --next now`. - `pause` cancels the active run and stops future ones; `resume` clears the stop but runs nothing immediately (one-time ingests can't be resumed — create a new one). - `logs` lists an ingest's runs (newest first); `run ` shows one run. Both accept `--wait` to watch until a terminal state (exit `0` succeeded, `1` failed/cancelled, `2` still queued/running). - `remove` soft-deletes the ingest and releases its destination table; the table, its data, and the source are never deleted. ### Sources A **source** (datasource) stores an external system's connection config and credentials. It's surfaced under `ingest sources` so a source and the ingests that read it share one command tree. ```sh hotdata ingest sources types [] hotdata ingest sources fields [] hotdata ingest sources test --family [--config ] [--credentials ] [--bucket-url ] hotdata ingest sources add [--family ] [--display-name ] [--config ] [--credentials ] hotdata ingest sources list [--family ] [--state ] hotdata ingest sources show hotdata ingest sources update-config [--config ] [--credentials ] hotdata ingest sources remove ``` - Families: `sql`, `filesystem`, `iceberg`, `delta`, `ducklake`, `kafka`, `rest`. Use `sql` for any SQL dialect (the dialect goes in the config) and `filesystem` for buckets. - `types` browses the catalog of source types and their families; `fields ` shows the config, credentials, and selector fields a family accepts (generated by the service, so it matches exactly what the API accepts). - `test` checks a config and credentials without creating anything — `valid` means the shape was accepted, `probed` means the source was actually contacted. - `add` returns a stable `ds_…` id and loads no data; on a terminal it prompts for the source type and fields (skip with `--no-input` and `--config`). Config and credentials are JSON — inline, `@file.json`, or `@-` for stdin; keep secrets out of argv with `@file`. - `update-config` appends a config version — rotate credentials by passing the same config plus new `--credentials`. - `remove` soft-deletes a source; its ingests must be removed first. ## Search Full-text (BM25) and vector search over an indexed table column, plus index and embedding-provider management. ```sh hotdata search "" \ --index \ [-d ] \ [--select ] \ [--limit ] \ [-o table|json|csv] ``` - The search type (full-text vs vector) is inferred from the index. `--index` addresses the index by name (from `hotdata search list`); `-d/--database` is the database the index lives in (defaults to the active database). - Full-text results are ordered by relevance (descending); vector results by distance (ascending). `--select` limits the returned columns (comma-separated, defaults to all). ### Manage indexes ```sh hotdata search create [] --type text|vector|sorted --from --column \ [--metric l2|cosine|dot] \ [--provider ] \ [--dimensions ] [--output-column ] [--description "..."] \ [--async] hotdata search list [--schema ] [--table ] hotdata search show [-d ] hotdata search remove [-d ] ``` - `--type`: `text` (BM25 full-text), `vector` (similarity), or `sorted` (range/equality filters). The index name is derived from table, column, and type when omitted. - `--metric` applies to vector indexes. `--provider` enables server-side auto-embeddings on a text column; `--dimensions` overrides the embedding output dimensions and `--output-column` names the generated column (default `{column}_embedding`). - `--async` submits index creation as a background job — poll with `hotdata jobs `. ### Embedding providers Embedding providers are the models behind vector search. ```sh hotdata search embeddings list hotdata search embeddings show hotdata search embeddings add --name --provider-type local|service \ [--config '{"model":"..."}'] \ [--provider-api-key | --secret-name ] hotdata search embeddings update [--name ] [--config ] [--provider-api-key | --secret-name ] hotdata search embeddings remove ``` - `--provider-type local` uses a local embedding model; `service` calls an external API (e.g. OpenAI). - `--provider-api-key` auto-creates a managed secret for the provider's API key; `--secret-name` references an existing secret instead. ## Jobs ```sh hotdata jobs list \ [--job-type ] \ [--status ] \ [--all] \ [--limit ] [--offset ] \ [-o table|json|yaml] hotdata jobs ``` - `list` shows only active jobs (`pending` and `running`) by default; use `--all` to see every job. - `--job-type` accepts: `data_refresh_table`, `data_refresh_connection`, `create_index`, `managed_load`. - `--status` accepts: `pending`, `running`, `succeeded`, `partially_succeeded`, `failed`. ## Manage Account, configuration, and CLI maintenance. ```sh hotdata manage usage [--since ] [-o table|json|yaml] hotdata manage completions bash|zsh|fish hotdata manage upgrade hotdata manage skills install [--project] hotdata manage skills status ``` - `usage` shows workspace usage: query count, bytes scanned, and stored bytes. `--since` counts usage from an RFC 3339 timestamp (e.g. `2026-06-01T00:00:00Z`); defaults to the current billing window. - `completions` prints a shell completion script; `upgrade` updates the CLI to the latest release. - `skills install` installs or refreshes the hotdata agent skill into agent directories (Claude Code, Cursor, etc.); `--project` installs into the current project instead of globally. See [Agent Skills](/docs/agent-skills). ## Configuration Config is stored at `~/.hotdata/config.yml`, keyed by profile (default: `default`). Override the config directory with `HOTDATA_CONFIG_DIR`. | Variable | Description | | :-- | :-- | | `HOTDATA_API_KEY` | API key (overrides the config file) | | `HOTDATA_WORKSPACE` | Default workspace ID for the current process | | `HOTDATA_DATABASE` | Default managed database for the current process | | `HOTDATA_API_URL` | Override the API endpoint (default `https://api.hotdata.dev/v1`) | | `HOTDATA_APP_URL` | Override the app URL used for browser auth | ## See also - [Quick Start](/docs/quick-start) — Install, authenticate, and run your first query - [API Reference](/docs/api-reference) — Full HTTP API documentation - [Data Sources](/docs/data-sources) — Supported source types --- # SQL Reference Source: https://www.hotdata.dev/docs/sql Site index: https://www.hotdata.dev/llms.txt Hotdata's native SQL is **HotSQL** — standard SQL with extensions for analytics, time-series, and complex types. If you know Postgres, you already know most of it. You can also write queries in PostgreSQL, DuckDB, or Snowflake syntax (see [SQL dialects](#sql-dialects)). ## SQL dialects Hotdata accepts queries in four SQL dialects — **HotSQL** (the native dialect), **PostgreSQL**, **DuckDB**, and **Snowflake**. Set the `dialect` field on the query request to write in a non-HotSQL dialect, and Hotdata translates it to HotSQL before running — so queries written for another engine run without a rewrite. Only read-only queries are accepted, and an unrecognized dialect is rejected. | `dialect` | Dialect | | --- | --- | | `hotsql` (default) | HotSQL | | `postgres` | PostgreSQL | | `duckdb` | DuckDB | | `snowflake` | Snowflake | ```json { "sql": "SELECT IFF(amount > 100, 'big', 'small') AS bucket FROM orders", "dialect": "snowflake" } ``` ## SELECT syntax The core query form is: ``` SELECT ... FROM ... [ WHERE ... ] [ GROUP BY ... ] [ HAVING ... ] [ ORDER BY ... ] [ LIMIT ... ] ``` ### WITH (CTEs) Use `WITH` for common table expressions: ```sql WITH top_customers AS ( SELECT customer_id, sum(amount) AS total FROM orders GROUP BY customer_id ORDER BY total DESC LIMIT 10 ) SELECT * FROM top_customers; ``` ### FROM clause Reference tables by schema and name. Tables live in your managed databases. Use fully qualified names when needed: `catalog.schema.table` or `schema.table`. ### WHERE clause Filter rows with conditions. Supports `AND`, `OR`, `NOT`, comparison operators (`=`, `!=`, `<`, `>`, `<=`, `>=`), `IN`, `BETWEEN`, `LIKE`, `IS NULL`, `IS NOT NULL`, and subqueries with `EXISTS`, `IN`, `ANY`, `ALL`. ### JOIN clause Join tables with `INNER`, `LEFT`, `RIGHT`, `FULL`, `CROSS` joins. Use `ON` for join conditions: ```sql SELECT o.id, c.name FROM orders o JOIN customers c ON o.customer_id = c.id WHERE o.created_at >= '2025-01-01'; ``` ### GROUP BY and HAVING Group rows and filter groups: ```sql SELECT department, count(*) AS headcount, avg(salary) AS avg_salary FROM employees GROUP BY department HAVING count(*) > 5 ORDER BY avg_salary DESC; ``` ### ORDER BY and LIMIT Sort and limit results: ```sql SELECT * FROM events ORDER BY created_at DESC LIMIT 100; ``` `ORDER BY` supports `ASC`, `DESC`, and multiple columns. `LIMIT` can optionally include `OFFSET` for pagination. ### UNION Combine result sets with `UNION` (distinct) or `UNION ALL` (keeps duplicates): ```sql SELECT id, name FROM users UNION ALL SELECT id, name FROM archived_users; ``` ### Subqueries Subqueries are supported in `SELECT`, `FROM`, `WHERE`, and `HAVING`: ```sql SELECT * FROM orders WHERE customer_id IN (SELECT id FROM customers WHERE tier = 'premium'); ``` ## Data types Standard SQL types are supported: | Category | Types | |----------|-------| | Numeric | `TINYINT`, `SMALLINT`, `INT`, `INTEGER`, `BIGINT`, `DECIMAL`, `FLOAT`, `DOUBLE`, `REAL` | | Character | `CHAR`, `VARCHAR`, `STRING` | | Date/Time | `DATE`, `TIME`, `TIMESTAMP`, `INTERVAL` | | Boolean | `BOOLEAN` | | Binary | `BINARY`, `VARBINARY` | | Complex | `ARRAY`, `STRUCT`, `MAP` | ## Operators ### Comparison `=`, `!=`, `<>`, `<`, `>`, `<=`, `>=`, `IS NULL`, `IS NOT NULL`, `BETWEEN ... AND ...`, `LIKE`, `IN (...)` ### Logical `AND`, `OR`, `NOT` ### Numerical `+`, `-`, `*`, `/`, `%` (modulo) ### Bitwise `&`, `|`, `^` (bitwise XOR), `~` (bitwise NOT), `<<`, `>>` ### String `||` for concatenation ## Functions HotSQL includes a comprehensive function library. Each category has a dedicated reference with signatures, arguments, and runnable examples: - [Math](/docs/sql-functions-math) - [String](/docs/sql-functions-string) — text, regular-expression, and hashing functions - [Date & time](/docs/sql-functions-datetime) - [Conditional & utility](/docs/sql-functions-conditional) - [Aggregate](/docs/sql-functions-aggregate) - [Window](/docs/sql-functions-window) - [Array, struct & map](/docs/sql-functions-array) - [JSON](/docs/sql-functions-json) — path access with `->`, `->>`, and `?` - [Spatial](/docs/sql-functions-spatial) ### Aggregates: FILTER and WITHIN GROUP Aggregates accept a `FILTER (WHERE ...)` clause, and ordered-set aggregates (`percentile_cont`, `approx_percentile_cont`) accept `WITHIN GROUP (ORDER BY ...)`: ```sql SELECT sum(amount) FILTER (WHERE status = 'completed') AS completed_revenue, percentile_cont(0.5) WITHIN GROUP (ORDER BY latency) AS median_latency FROM orders; ``` ### Window functions Any aggregate or window function runs over a frame with `OVER (PARTITION BY ... ORDER BY ...)`: ```sql SELECT id, amount, sum(amount) OVER (ORDER BY id ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS running_total, rank() OVER (ORDER BY amount DESC) AS amount_rank FROM transactions; ``` ### Expanding arrays `unnest(array)` expands an array into rows: ```sql SELECT id, unnest(tags) AS tag FROM articles; ``` ## Geospatial functions Hotdata includes spatial functions that follow PostGIS-style `ST_` naming, for querying geometry data. Geometries are planar (Cartesian)—there is no geography type, and coordinates are always interpreted as x = easting/longitude. Buffering and overlay operations (`ST_Buffer`, `ST_Union`, `ST_Intersection`, `ST_Difference`, `ST_MakeValid`, ...) and coordinate reprojection (`ST_Transform`) are available. For geodesic measurements in metres on lon/lat data, use the `_Spheroid` family (`ST_Area_Spheroid`, `ST_Distance_Spheroid`, `ST_DWithin_Spheroid`, ...) or `ST_DistanceSphere`. ### Constructing geometries Create points and geometries from coordinates or text: ```sql -- Create a 2D point SELECT ST_MakePoint(-122.4194, 37.7749) AS sf_point; -- Create from Well-Known Text (WKT) SELECT ST_GeomFromText('POINT(-122.4194 37.7749)') AS point; SELECT ST_GeomFromText('POLYGON((0 0, 1 0, 1 1, 0 1, 0 0))') AS polygon; -- Create from GeoHash SELECT ST_PointFromGeoHash('9q8yy') AS point; ``` ### Geometry accessors Extract properties and coordinates: ```sql SELECT ST_X(location) AS lon, ST_Y(location) AS lat, ST_Area(boundary) AS area_sq_units, ST_Length(path) AS path_length, ST_GeometryType(geom) AS geom_type FROM locations; ``` ### Spatial relationships Test how geometries relate to each other: ```sql -- Find stores within 10 km of a point (ST_DistanceSphere returns metres) SELECT id, name FROM stores WHERE ST_DistanceSphere( ST_GeomFromText('POINT(-122.4194 37.7749)'), location ) < 10000; -- Points contained in a polygon (e.g. delivery zones) SELECT order_id, ST_AsText(destination) FROM orders WHERE ST_Within(destination, delivery_zone); -- Intersecting geometries (e.g. overlapping regions) SELECT a.id, b.id FROM regions a JOIN regions b ON ST_Intersects(a.geom, b.geom) AND a.id < b.id; ``` `ST_Distance` and `ST_DWithin` are planar—distances in the coordinates' own units (degrees for lon/lat data). For metres on lon/lat data use `ST_DistanceSphere` (great-circle) or `ST_Distance_Spheroid` / `ST_DWithin_Spheroid` (WGS84 geodesic). ### Geometry processing Compute derived geometries: ```sql -- Convex hull of a geometry SELECT id, ST_ConvexHull(geom) AS hull FROM regions; -- Simplify a polygon (Douglas-Peucker) SELECT ST_Simplify(complex_geom, 0.001) AS simplified FROM boundaries; -- Centroid of a geometry SELECT ST_Centroid(polygon) AS center FROM parcels; -- Buffer a point by 1 unit SELECT ST_Buffer(location, 1.0) AS service_area FROM stores; -- Merge overlapping regions SELECT ST_Union(a.geom, b.geom) FROM regions a JOIN regions b ON ST_Intersects(a.geom, b.geom); ``` ### Coordinate reprojection `ST_Transform` reprojects between coordinate reference systems. The CRS travels in the call (there is no per-geometry SRID); coordinates are always x = easting/longitude: ```sql -- lon/lat (EPSG:4326) to Web Mercator metres (EPSG:3857) SELECT ST_Transform(location, 'EPSG:4326', 'EPSG:3857') AS mercator FROM stores; ``` Hotdata's spatial functions use PostGIS-style `ST_` names; alternate spellings from other SQL dialects are accepted as aliases where they differ (e.g. `ST_Distance_Sphere` for `ST_DistanceSphere`). See the [Spatial functions reference](/docs/sql-functions-spatial) for the full set of 117 functions, and the [PostGIS documentation](https://postgis.net/documentation/) for the semantics of individual functions. ## Vector search Hotdata provides vector similarity search with two approaches: **semantic search** on text columns using an embedding provider, and **direct vector search** on pre-computed embedding columns. When a vector index exists, queries are transparently rewritten into HNSW index calls—no special syntax needed. ### Semantic search with vector_distance `vector_distance(column, 'search text')` is the simplest way to perform semantic search. It takes a text column and a search string, automatically generates an embedding using the column's configured embedding provider, and computes the distance using the metric defined in the vector index. ```sql SELECT id, title, vector_distance(description, 'machine learning frameworks') AS dist FROM conn.public.articles ORDER BY dist ASC LIMIT 10; ``` **Requirements:** - A [vector index](#vector-indexes) on the column with an embedding provider configured - The column must be a text column (`VARCHAR` / `TEXT`) Behind the scenes, `vector_distance` resolves the embedding provider from the vector index metadata, calls it to embed your search text, and rewrites the query to the appropriate distance function (e.g. `cosine_distance`) on the generated embedding column. You don't need to know which distance metric or embedding column is used—the index configuration handles it. ### Distance functions For direct control over vector search—when you already have embedding vectors or want to specify the exact distance metric—use the distance functions directly on vector columns. All are lower-is-closer; use `ORDER BY ... ASC`: | SQL function | Metric | Use case | |---|---|---| | `l2_distance(column, query)` | L2 (Euclidean) | General-purpose distance | | `cosine_distance(column, query)` | Cosine | Angle between vectors (normalized embeddings) | | `negative_dot_product(column, query)` | Inner product | Maximum inner product search | These functions accept a vector column and a literal query vector (`ARRAY[...]`): ```sql SELECT id, title, cosine_distance(embedding, ARRAY[0.1, -0.2, 0.5, ...]) AS dist FROM documents ORDER BY dist ASC LIMIT 10; ``` **With a vector index:** queries matching `ORDER BY distance_fn(col, ARRAY[...]) ASC LIMIT k` are transparently rewritten into an HNSW index call for fast approximate nearest-neighbor search. **Without a vector index:** the distance functions still work via brute-force scan over all rows. This is useful for small tables or ad-hoc exploration, but much slower on large datasets. A metric mismatch (e.g. `cosine_distance` on an L2-built index) falls back silently to brute-force. `ORDER BY ... DESC` is also never rewritten. ### vector_search table function `vector_search` is a table function for semantic search that returns full rows. Like `vector_distance`, it auto-embeds your search text using the configured embedding provider — but returns results as a table you can query directly. ``` vector_search('catalog.schema.table', 'column', 'search text', k) ``` | Parameter | Type | Description | |---|---|---| | table | string literal | Fully-qualified table name: `'catalog.schema.table'` | | column | string literal | Text column with a vector index and embedding provider | | query | string literal | Search text to embed and search for | | k | integer | Number of nearest neighbors to return | Returns all table columns plus a `_distance` column (`Float32`). Lower distance means more similar. **Requirements:** A [vector index](#vector-indexes) on the column with an embedding provider configured. ```sql -- Semantic search returning full rows SELECT id, title, _distance FROM vector_search('mydb.public.articles', 'description', 'machine learning', 10) ORDER BY _distance ASC; ``` ```sql -- With filtering and aggregation SELECT category, COUNT(*) AS cnt, AVG(_distance) AS avg_dist FROM vector_search('mydb.public.articles', 'description', 'data science', 50) WHERE category IN ('ml', 'statistics') GROUP BY category ORDER BY avg_dist; ``` ### vector_search_vector table function `vector_search_vector` is the direct ANN search table function — you provide the query vector yourself. It requires a vector index but no embedding provider. ``` vector_search_vector('catalog.schema.table', 'column', ARRAY[...], k) ``` | Parameter | Type | Description | |---|---|---| | table | string literal | Fully-qualified table name: `'catalog.schema.table'` | | column | string literal | Vector column with a HNSW index | | query | `ARRAY[...]` literal | Query vector | | k | integer | Number of nearest neighbors to return | Returns all table columns plus a `_distance` column (`Float32`). **Requirements:** A [vector index](#vector-indexes) on the column. ```sql SELECT id, title, _distance FROM vector_search_vector('mydb.public.documents', 'embedding', ARRAY[0.1, -0.2, 0.5, ...], 10) ORDER BY _distance ASC; ``` ### Choosing between vector search methods | Method | Input | Requires embedding provider | Requires index | Returns | |---|---|---|---|---| | `vector_distance(col, 'text')` | text | yes | yes | scalar (`Float32`) — use in SELECT/ORDER BY | | `vector_search('table', 'col', 'text', k)` | text | yes | yes | table (all columns + `_distance`) | | `vector_search_vector('table', 'col', ARRAY, k)` | vector | no | yes | table (all columns + `_distance`) | | `l2_distance` / `cosine_distance` / `negative_dot_product` | vector | no | optional (HNSW if present, brute-force otherwise) | scalar (`Float32`) — use in SELECT/ORDER BY | Use `vector_search` or `vector_distance` when you have text and want automatic embedding. Use `vector_search_vector` or the distance functions when you already have vectors. ### WHERE clause filtering Scalar `WHERE` conditions are absorbed and applied **adaptively** at execution time when a vector index is present: | Selectivity | Strategy | Behavior | |---|---|---| | > 5% | In-graph filtering | HNSW filtered search skips non-passing nodes; returns exactly k results | | ≤ 5% | Brute-force subset | HNSW bypassed; exact distances over the valid subset; heap-select top-k | The 5% cutoff is a fixed engine default (not currently configurable per table). ```sql SELECT id, title, cosine_distance(embedding, ARRAY[...]) AS dist FROM documents WHERE category = 'technical' ORDER BY dist ASC LIMIT 10; ``` ### Behavior and limitations - **expansion_search** (ef_search): the beam width during HNSW traversal, a fixed engine default of 64 (higher would improve recall at the cost of speed). - **brute_force_selectivity_threshold**: the selectivity cutoff below which brute-force is used instead of HNSW—a fixed engine default of 0.05 (5%). - **Literal query vectors**: The optimizer only rewrites distance functions when the query vector is a compile-time literal (`ARRAY[...]`). Use `vector_distance` for text-based search. - **`SELECT *` with distance functions**: When the table has generated embedding columns, `SELECT *` expands to include them. Since the HNSW index doesn't serve embedding columns, the query falls back to brute-force. Select specific columns to use the index, or use `vector_search` / `vector_search_vector` which always use HNSW. - **Stacked filters**: Only one `Filter → TableScan` layer is absorbed. ## Full-text search Hotdata provides BM25 full-text search via the `bm25_search` table function. It performs ranked text retrieval against a column with a BM25 index. ### bm25_search ``` bm25_search('catalog.schema.table', 'column', 'query text' [, limit]) ``` | Parameter | Type | Description | |---|---|---| | table | string literal | Fully-qualified table name: `'catalog.schema.table'` | | column | string literal | Text column to search | | query | string literal | Search query text | | limit | integer (optional) | Maximum results to return. Default: `1000` | Returns all columns from the base table plus a `score` column (`Float32`) containing the BM25 relevance score. Higher scores indicate stronger relevance. Rows are **not** returned in score order—add an explicit `ORDER BY score DESC` to rank them. **Requirements:** A [BM25 index](#bm25-indexes) on the target column. ```sql -- Search articles for "machine learning", return top 20 by relevance SELECT id, title, body, score FROM bm25_search('mydb.public.articles', 'body', 'machine learning', 20) ORDER BY score DESC; ``` ```sql -- Combine full-text search with additional filters SELECT id, title, score FROM bm25_search('conn.public.docs', 'content', 'kubernetes deployment', 100) WHERE category = 'infrastructure' ORDER BY score DESC LIMIT 10; ``` > **Note:** `bm25_search` retrieves its top-`limit` matches first, then any `WHERE` filters are applied to that set. When combining with filters, pass a higher `limit` to `bm25_search` than the final number of rows you need so relevant rows aren't filtered out of a too-small candidate set. ## Managed table layout Managed database tables can declare a **storage layout**—how their rows are partitioned and sorted on disk—when the table is created. Unlike an index, the layout is intrinsic to how the table's data is written. It is fixed at table creation (via the [managed table API](/docs/api-reference/databases#add-table-to-database-default-catalog)), cannot be changed afterward, and is not set from SQL. - **Partitioning** — one or more partition keys, each a column plus a transform: `identity`, `year`, `month`, `day`, or `hour`. Transforms compose, so partitioning by `year(ts)` and `month(ts)` places each calendar month in its own partition. A `WHERE` filter on a partition column lets the planner skip whole partitions. - **Sort order** — one or more sort keys (a column plus optional `asc` / `desc` and null ordering). Rows are written in this order, so range filters can prune row groups and an `ORDER BY` on the sort keys is satisfied without an explicit sort step. Because the layout is applied at write time, it also organizes data for [key-based mutations](/docs/api-reference/databases#load-database-table-from-inline-data-upload-or-query-result) (`upsert`, `update`, `delete`) on partitioned tables. This is complementary to [sorted indexes](#sorted-indexes): a sorted index is a secondary, pre-sorted copy of cached data, whereas the declared layout governs the table's primary storage. ## Indexes Hotdata supports three index types to accelerate different query patterns. Indexes are created via the [API](/docs/api-reference/indexes) or [CLI](/docs/cli-reference#search)—not SQL. Indexes are built over managed-database tables, so load the data first (upload parquet or run an ingest). **CLI** (managed-database tables): `hotdata search create --type text|vector|sorted --from ..
--column ` Indexes are stored in the catalog and used automatically when the query planner finds a beneficial match. Use `GET .../indexes` to list indexes and `DELETE .../indexes/{index_name}` to drop one. ### Sorted indexes Pre-sorted Parquet copies of cached data, ordered by the columns you specify. Accelerate range queries, ordering, and point lookups. ```json { "index_name": "idx_created_at", "columns": ["created_at"], "index_type": "sorted" } ``` A sorted index is selected when: 1. **Filter match** — a `WHERE` clause filters on the index's leading sort column. Sorted Parquet enables efficient row-group pruning for range and equality predicates. 2. **Sort elimination** — an `ORDER BY` on the index columns is satisfied without an explicit sort step. ```sql -- Index on created_at: row-group pruning for range filter SELECT * FROM events WHERE created_at >= '2025-01-01' AND created_at < '2025-02-01'; -- Index on sku: sort elimination SELECT * FROM products ORDER BY sku LIMIT 100; ``` ### Vector indexes HNSW indexes for approximate nearest-neighbor search on vector columns. Require exactly one column. **On a pre-computed embedding column** (float array): ```json { "index_name": "docs_embedding_idx", "columns": ["embedding"], "index_type": "vector", "metric": "cosine" } ``` **On a text column with auto-embedding:** when the source column is text, Hotdata automatically generates embeddings using an embedding provider. This enables [`vector_distance()`](#semantic-search-with-vector_distance) for semantic search. If you omit `embedding_provider_id`, the system embedding provider is used automatically. If no system provider is configured, you must pass one explicitly. The `metric` also defaults to the provider's configured metric when not specified. ```json { "index_name": "docs_description_idx", "columns": ["description"], "index_type": "vector" } ``` Or with explicit provider and metric: ```json { "index_name": "docs_description_idx", "columns": ["description"], "index_type": "vector", "metric": "cosine", "embedding_provider_id": "embp_abc123", "dimensions": 1536 } ``` The embedding provider generates a new column (default: `{column}_embedding`) containing the vector embeddings. Configure embedding providers via the [Embedding Providers API](/docs/api-reference/embedding-providers). | Field | Required | Description | |---|---|---| | `metric` | no | Distance metric: `l2`, `cosine`, or `dot`. Defaults to `l2` for float array columns, or the provider's metric for text columns with auto-embedding | | `embedding_provider_id` | no | Embedding provider ID. Defaults to the system provider for text columns | | `output_column` | no | Name for the generated embedding column. Default: `{column}_embedding` | | `dimensions` | no | Embedding dimensions (model-dependent) | **CLI example:** ```sh hotdata search create --type vector --from mydb.public.documents \ --column embedding --metric cosine ``` ### BM25 indexes Full-text search indexes using BM25 scoring. Required for [`bm25_search()`](#bm25_search). ```json { "index_name": "articles_body_idx", "columns": ["body"], "index_type": "bm25" } ``` **CLI example:** ```sh hotdata search create --type text --from mydb.public.articles \ --column body ``` ### Point lookups Queries that fetch a single row by equality (`WHERE col = value` with `LIMIT 1`) are classified as point lookups. Parquet bloom filter reads skip row groups that cannot contain the value, reducing I/O. ```sql SELECT * FROM users WHERE id = 12345; ``` ### Index lifecycle Indexes are invalidated when the underlying table is purged or resynced. After a data refresh, indexes are automatically rebuilt from the new cached data. If a rebuild fails, drop and recreate the index via the API. ## Information schema Introspect the catalog by querying the `information_schema` views: - `information_schema.tables` — tables in scope - `information_schema.columns` — columns and their types - `information_schema.schemata` — schemas - `information_schema.catalogs` — catalogs ```sql SELECT table_schema, table_name FROM information_schema.tables ORDER BY table_schema, table_name; ``` `DESCRIBE table` returns a single table's schema: ```sql DESCRIBE conn.public.orders; ``` The `information_schema.views`, `routines`, and `parameters` views exist for tool compatibility but are empty. `SHOW TABLES` / `SHOW COLUMNS` / `SHOW FUNCTIONS` are not enabled—use the views above or `DESCRIBE`. You can also list tables and schemas via the API or MCP. ## EXPLAIN Use `EXPLAIN` to inspect the query plan: ```sql EXPLAIN SELECT * FROM orders WHERE created_at > '2025-01-01'; ``` `EXPLAIN ANALYZE` runs the query and returns execution metrics (timing, rows, etc.). Because it executes the plan, read-only enforcement still applies—you cannot run a write by wrapping it in `EXPLAIN`; the wrapped statement is inspected and rejected. ## Further reading - [Function references](#functions) — every category, with signatures and runnable examples - [PostGIS documentation](https://postgis.net/documentation/) — semantics of the `ST_` spatial functions (Hotdata implements a planar-geometry subset) - HNSW — approximate nearest-neighbor search --- # Math functions Source: https://www.hotdata.dev/docs/sql-functions-math Site index: https://www.hotdata.dev/llms.txt Reference for math functions in [HotSQL](/docs/sql). Names, signatures, and examples match the engine exactly. ### abs Returns the absolute value of a number. ``` abs(numeric_expression) ``` **Arguments** - `numeric_expression`: Numeric expression. ```sql > SELECT abs(-5); +----------+ | abs(-5) | +----------+ | 5 | +----------+ ``` ### ceil Returns the nearest integer greater than or equal to a number. ``` ceil(numeric_expression) ``` **Arguments** - `numeric_expression`: Numeric expression. ```sql > SELECT ceil(3.14); +------------+ | ceil(3.14) | +------------+ | 4.0 | +------------+ ``` ### cot Returns the cotangent of a number. ``` cot(numeric_expression) ``` **Arguments** - `numeric_expression`: Numeric expression. ```sql > SELECT cot(1); +---------+ | cot(1) | +---------+ | 0.64209 | +---------+ ``` ### factorial Factorial of a non-negative integer. Errors if the argument is negative or the result overflows. ``` factorial(numeric_expression) ``` **Arguments** - `numeric_expression`: Numeric expression. ```sql > SELECT factorial(5); +---------------+ | factorial(5) | +---------------+ | 120 | +---------------+ ``` ### floor Returns the nearest integer less than or equal to a number. ``` floor(numeric_expression) ``` **Arguments** - `numeric_expression`: Numeric expression. ```sql > SELECT floor(3.14); +-------------+ | floor(3.14) | +-------------+ | 3.0 | +-------------+ ``` ### gcd Returns the greatest common divisor of `expression_x` and `expression_y`. Returns 0 if both inputs are zero. ``` gcd(expression_x, expression_y) ``` **Arguments** - `expression_x`: First numeric expression. - `expression_y`: Second numeric expression. ```sql > SELECT gcd(48, 18); +------------+ | gcd(48,18) | +------------+ | 6 | +------------+ ``` ### isnan Returns true if a given number is +NaN or -NaN otherwise returns false. ``` isnan(numeric_expression) ``` **Arguments** - `numeric_expression`: Numeric expression. ```sql > SELECT isnan(1); +----------+ | isnan(1) | +----------+ | false | +----------+ ``` ### iszero Returns true if a given number is +0.0 or -0.0 otherwise returns false. ``` iszero(numeric_expression) ``` **Arguments** - `numeric_expression`: Numeric expression. ```sql > SELECT iszero(0); +------------+ | iszero(0) | +------------+ | true | +------------+ ``` ### lcm Returns the least common multiple of `expression_x` and `expression_y`. Returns 0 if either input is zero. ``` lcm(expression_x, expression_y) ``` **Arguments** - `expression_x`: First numeric expression. - `expression_y`: Second numeric expression. ```sql > SELECT lcm(4, 5); +----------+ | lcm(4,5) | +----------+ | 20 | +----------+ ``` ### log Returns the base-x logarithm of a number. Can either provide a specified base, or if omitted then takes the base-10 of a number. ``` log(base, numeric_expression) log(numeric_expression) ``` **Arguments** - `base`: Base numeric expression. - `numeric_expression`: Numeric expression. ```sql > SELECT log(10); +---------+ | log(10) | +---------+ | 1.0 | +---------+ ``` ### nanvl Returns the first argument if it's not _NaN_. Returns the second argument otherwise. ``` nanvl(expression_x, expression_y) ``` **Arguments** - `expression_x`: Numeric expression to return if it's not _NaN_. Can be a constant, column, or function, and any combination of arithmetic operators. - `expression_y`: Numeric expression to return if the first expression is _NaN_. Can be a constant, column, or function, and any combination of arithmetic operators. ```sql > SELECT nanvl(0, 5); +------------+ | nanvl(0,5) | +------------+ | 0 | +------------+ ``` ### pi Returns an approximate value of π. ``` pi() ``` ### power Returns a base expression raised to the power of an exponent. ``` power(base, exponent) ``` **Arguments** - `base`: Numeric expression. - `exponent`: Exponent numeric expression. ```sql > SELECT power(2, 3); +-------------+ | power(2,3) | +-------------+ | 8 | +-------------+ ``` ### random Returns a random float value in the range [0, 1). The random seed is unique to each row. ``` random() ``` ```sql > SELECT random(); +------------------+ | random() | +------------------+ | 0.7389238902938 | +------------------+ ``` ### round Rounds a number to the nearest integer. ``` round(numeric_expression[, decimal_places]) ``` **Arguments** - `numeric_expression`: Numeric expression. - `decimal_places`: Optional. The number of decimal places to round to. Defaults to 0. ```sql > SELECT round(3.14159); +--------------+ | round(3.14159)| +--------------+ | 3.0 | +--------------+ ``` ### signum Returns the sign of a number. Negative numbers return `-1`. Zero and positive numbers return `1`. ``` signum(numeric_expression) ``` **Arguments** - `numeric_expression`: Numeric expression. ```sql > SELECT signum(-42); +-------------+ | signum(-42) | +-------------+ | -1 | +-------------+ ``` ### trunc Truncates a number to a whole number or truncated to the specified decimal places. ``` trunc(numeric_expression[, decimal_places]) ``` **Arguments** - `numeric_expression`: Numeric expression. - `decimal_places`: Optional. The number of decimal places to truncate to. Defaults to 0 (truncate to a whole number). If `decimal_places` is a positive integer, truncates digits to the right of the decimal point. If `decimal_places` is a negative integer, replaces digits to the left of the decimal point with `0`. ```sql > SELECT trunc(42.738); +----------------+ | trunc(42.738) | +----------------+ | 42 | +----------------+ ``` --- # String functions Source: https://www.hotdata.dev/docs/sql-functions-string Site index: https://www.hotdata.dev/llms.txt Reference for string functions in [HotSQL](/docs/sql). Names, signatures, and examples match the engine exactly. ### ascii Returns the first Unicode scalar value of a string. ``` ascii(str) ``` **Arguments** - `str`: String expression. ```sql > select ascii('abc'); +--------------------+ | ascii(Utf8("abc")) | +--------------------+ | 97 | +--------------------+ > select ascii('🚀'); +-------------------+ | ascii(Utf8("🚀")) | +-------------------+ | 128640 | +-------------------+ ``` **Related:** `chr` ### bit_length Returns the bit length of a string. ``` bit_length(str) ``` **Arguments** - `str`: String expression. **Related:** `length`, `octet_length` ### btrim Trims the specified trim string from the start and end of a string. If no trim string is provided, all spaces are removed from the start and end of the input string. ``` btrim(str[, trim_str]) ``` **Arguments** - `str`: String expression. - `trim_str`: String expression to operate on. Can be a constant, column, or function, and any combination of operators. _Default is a space._ **Related:** `ltrim`, `rtrim` ### character_length Returns the number of characters in a string. ``` character_length(str) ``` **Arguments** - `str`: String expression. ```sql > select character_length('Ångström'); +------------------------------------+ | character_length(Utf8("Ångström")) | +------------------------------------+ | 8 | +------------------------------------+ ``` **Related:** `bit_length`, `octet_length` ### chr Returns a string containing the character with the specified Unicode scalar value. ``` chr(expression) ``` **Arguments** - `expression`: String expression. ```sql > select chr(128640); +--------------------+ | chr(Int64(128640)) | +--------------------+ | 🚀 | +--------------------+ ``` **Related:** `ascii` ### concat Concatenates multiple strings together. ``` concat(str[, ..., str_n]) ``` **Arguments** - `str`: String expression. - `str_n`: Subsequent string expressions to concatenate. **Related:** `concat_ws` ### concat_ws Concatenates multiple strings together with a specified separator. ``` concat_ws(separator, str[, ..., str_n]) ``` **Arguments** - `separator`: Separator to insert between concatenated strings. - `str`: String expression to operate on. Can be a constant, column, or function, and any combination of operators. - `str_n`: Subsequent string expressions to concatenate. ```sql > select concat_ws('_', 'data', 'fusion'); +--------------------------------------------------+ | concat_ws(Utf8("_"),Utf8("data"),Utf8("fusion")) | +--------------------------------------------------+ | data_fusion | +--------------------------------------------------+ ``` **Related:** `concat` ### contains Return true if search_str is found within string (case-sensitive). ``` contains(str, search_str) ``` **Arguments** - `str`: String expression. - `search_str`: The string to search for in str. ```sql > select contains('the quick brown fox', 'row'); +---------------------------------------------------+ | contains(Utf8("the quick brown fox"),Utf8("row")) | +---------------------------------------------------+ | true | +---------------------------------------------------+ ``` ### decode Decode binary data from textual representation in string. ``` decode(expression, format) ``` **Arguments** - `expression`: Expression containing encoded string data - `format`: Same arguments as [encode](#encode) **Related:** `encode` ### digest Computes the binary hash of an expression using the specified algorithm. ``` digest(expression, algorithm) ``` **Arguments** - `expression`: String expression. - `algorithm`: String expression specifying algorithm to use. Must be one of: - md5 - sha224 - sha256 - sha384 - sha512 - blake2s - blake2b - blake3 ```sql > select digest('foo', 'sha256'); +------------------------------------------------------------------+ | digest(Utf8("foo"),Utf8("sha256")) | +------------------------------------------------------------------+ | 2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae | +------------------------------------------------------------------+ ``` ### encode Encode binary data into a textual representation. ``` encode(expression, format) ``` **Arguments** - `expression`: Expression containing string or binary data - `format`: Supported formats are: `base64`, `base64pad`, `hex` **Related:** `decode` ### ends_with Tests if a string ends with a substring. ``` ends_with(str, substr) ``` **Arguments** - `str`: String expression. - `substr`: Substring to test for. ### find_in_set Returns a value in the range of 1 to N if the string str is in the string list strlist consisting of N substrings. ``` find_in_set(str, strlist) ``` **Arguments** - `str`: String expression to find in strlist. - `strlist`: A string list is a string composed of substrings separated by , characters. ```sql > select find_in_set('b', 'a,b,c,d'); +----------------------------------------+ | find_in_set(Utf8("b"),Utf8("a,b,c,d")) | +----------------------------------------+ | 2 | +----------------------------------------+ ``` ### initcap Capitalizes the first character in each word in the input string. Words are delimited by non-alphanumeric characters. ``` initcap(str) ``` **Arguments** - `str`: String expression. **Related:** `lower`, `upper` ### left Returns a specified number of characters from the left side of a string. ``` left(str, n) ``` **Arguments** - `str`: String expression. - `n`: Number of characters to return. **Related:** `right` ### levenshtein Returns the [`Levenshtein distance`](https://en.wikipedia.org/wiki/Levenshtein_distance) between the two given strings. ``` levenshtein(str1, str2) ``` **Arguments** - `str1`: String expression to compute Levenshtein distance with str2. - `str2`: String expression to compute Levenshtein distance with str1. ```sql > select levenshtein('kitten', 'sitting'); +---------------------------------------------+ | levenshtein(Utf8("kitten"),Utf8("sitting")) | +---------------------------------------------+ | 3 | +---------------------------------------------+ ``` ### lower Converts a string to lower-case. ``` lower(str) ``` **Arguments** - `str`: String expression. ```sql > select lower('Ångström'); +-------------------------+ | lower(Utf8("Ångström")) | +-------------------------+ | ångström | +-------------------------+ ``` **Related:** `initcap`, `upper` ### lpad Pads the left side of a string with another string to a specified string length. ``` lpad(str, n[, padding_str]) ``` **Arguments** - `str`: String expression. - `n`: String length to pad to. If the input string is longer than this length, it is truncated (on the right). - `padding_str`: Optional string expression to pad with. Can be a constant, column, or function, and any combination of string operators. _Default is a space._ ```sql > select lpad('Dolly', 10, 'hello'); +---------------------------------------------+ | lpad(Utf8("Dolly"),Int64(10),Utf8("hello")) | +---------------------------------------------+ | helloDolly | +---------------------------------------------+ ``` **Related:** `rpad` ### ltrim Trims the specified trim string from the beginning of a string. If no trim string is provided, spaces are removed from the start of the input string. ``` ltrim(str[, trim_str]) ``` **Arguments** - `str`: String expression. - `trim_str`: String expression to trim from the beginning of the input string. Can be a constant, column, or function, and any combination of arithmetic operators. _Default is a space._ **Related:** `btrim`, `rtrim` ### md5 Computes an MD5 128-bit checksum for a string expression. ``` md5(expression) ``` **Arguments** - `expression`: String expression. ```sql > select md5('foo'); +----------------------------------+ | md5(Utf8("foo")) | +----------------------------------+ | acbd18db4cc2f85cedef654fccc4a4d8 | +----------------------------------+ ``` ### octet_length Returns the length of a string in bytes. ``` octet_length(str) ``` **Arguments** - `str`: String expression. ```sql > select octet_length('Ångström'); +--------------------------------+ | octet_length(Utf8("Ångström")) | +--------------------------------+ | 10 | +--------------------------------+ ``` **Related:** `bit_length`, `length` ### overlay Returns the string which is replaced by another string from the specified position and specified count length. ``` overlay(str PLACING substr FROM pos [FOR count]) ``` **Arguments** - `str`: String expression. - `substr`: Substring to replace in str. - `pos`: The start position to start the replace in str. - `count`: The count of characters to be replaced from start position of str. If not specified, will use substr length instead. ```sql > select overlay('Txxxxas' placing 'hom' from 2 for 4); +--------------------------------------------------------+ | overlay(Utf8("Txxxxas"),Utf8("hom"),Int64(2),Int64(4)) | +--------------------------------------------------------+ | Thomas | +--------------------------------------------------------+ ``` ### regexp_count Returns the number of matches that a [regular expression](https://docs.rs/regex/latest/regex/#syntax) has in a string. ``` regexp_count(str, regexp[, start, flags]) ``` **Arguments** - `str`: String expression. - `regexp`: Regular expression. - `start`: - **start**: Optional start position (the first position is 1) to search for the regular expression. Can be a constant, column, or function. - `flags`: Optional regular expression flags that control the behavior of the regular expression. The following flags are supported: - **i**: case-insensitive: letters match both upper and lower case - **m**: multi-line mode: ^ and $ match begin/end of line - **s**: allow . to match \n - **R**: enables CRLF mode: when multi-line mode is enabled, \r\n is used - **U**: swap the meaning of x* and x*? ```sql > select regexp_count('abcAbAbc', 'abc', 2, 'i'); +---------------------------------------------------------------+ | regexp_count(Utf8("abcAbAbc"),Utf8("abc"),Int64(2),Utf8("i")) | +---------------------------------------------------------------+ | 1 | +---------------------------------------------------------------+ ``` ### regexp_instr Returns the position in a string where the specified occurrence of a POSIX regular expression is located. ``` regexp_instr(str, regexp[, start[, N[, flags[, subexpr]]]]) ``` **Arguments** - `str`: String expression. - `regexp`: Regular expression. - `start`: - **start**: Optional start position (the first position is 1) to search for the regular expression. Can be a constant, column, or function. Defaults to 1 - `N`: - **N**: Optional The N-th occurrence of pattern to find. Defaults to 1 (first match). Can be a constant, column, or function. - `flags`: Optional regular expression flags that control the behavior of the regular expression. The following flags are supported: - **i**: case-insensitive: letters match both upper and lower case - **m**: multi-line mode: ^ and $ match begin/end of line - **s**: allow . to match \n - **R**: enables CRLF mode: when multi-line mode is enabled, \r\n is used - **U**: swap the meaning of x* and x*? - `subexpr`: Optional Specifies which capture group (subexpression) to return the position for. Defaults to 0, which returns the position of the entire match. ```sql > SELECT regexp_instr('ABCDEF', 'C(.)(..)'); +---------------------------------------------------------------+ | regexp_instr(Utf8("ABCDEF"),Utf8("C(.)(..)")) | +---------------------------------------------------------------+ | 3 | +---------------------------------------------------------------+ ``` ### regexp_like Returns true if a [regular expression](https://docs.rs/regex/latest/regex/#syntax) has at least one match in a string, false otherwise. ``` regexp_like(str, regexp[, flags]) ``` **Arguments** - `str`: String expression. - `regexp`: Regular expression. - `flags`: Optional regular expression flags that control the behavior of the regular expression. The following flags are supported: - **i**: case-insensitive: letters match both upper and lower case - **m**: multi-line mode: ^ and $ match begin/end of line - **s**: allow . to match \n - **R**: enables CRLF mode: when multi-line mode is enabled, \r\n is used - **U**: swap the meaning of x* and x*? ### regexp_match Returns the first [regular expression](https://docs.rs/regex/latest/regex/#syntax) matches in a string. ``` regexp_match(str, regexp[, flags]) ``` **Arguments** - `str`: String expression. - `regexp`: Regular expression to match against. Can be a constant, column, or function. - `flags`: Optional regular expression flags that control the behavior of the regular expression. The following flags are supported: - **i**: case-insensitive: letters match both upper and lower case - **m**: multi-line mode: ^ and $ match begin/end of line - **s**: allow . to match \n - **R**: enables CRLF mode: when multi-line mode is enabled, \r\n is used - **U**: swap the meaning of x* and x*? ### regexp_replace Replaces substrings in a string that match a [regular expression](https://docs.rs/regex/latest/regex/#syntax). ``` regexp_replace(str, regexp, replacement[, flags]) ``` **Arguments** - `str`: String expression. - `regexp`: Regular expression to match against. Can be a constant, column, or function. - `replacement`: Replacement string expression to operate on. Can be a constant, column, or function, and any combination of operators. - `flags`: Optional regular expression flags that control the behavior of the regular expression. The following flags are supported: - **g**: (global) Search globally and don't return after the first match - **i**: case-insensitive: letters match both upper and lower case - **m**: multi-line mode: ^ and $ match begin/end of line - **s**: allow . to match \n - **R**: enables CRLF mode: when multi-line mode is enabled, \r\n is used - **U**: swap the meaning of x* and x*? ### repeat Returns a string with an input string repeated a specified number. ``` repeat(str, n) ``` **Arguments** - `str`: String expression. - `n`: Number of times to repeat the input string. ```sql > select repeat('data', 3); +-------------------------------+ | repeat(Utf8("data"),Int64(3)) | +-------------------------------+ | datadatadata | +-------------------------------+ ``` ### replace Replaces all occurrences of a specified substring in a string with a new substring. ``` replace(str, substr, replacement) ``` **Arguments** - `str`: String expression. - `substr`: Substring expression to replace in the input string. Substring expression. - `replacement`: Replacement substring expression. ```sql > select replace('ABabbaBA', 'ab', 'cd'); +-------------------------------------------------+ | replace(Utf8("ABabbaBA"),Utf8("ab"),Utf8("cd")) | +-------------------------------------------------+ | ABcdbaBA | +-------------------------------------------------+ ``` ### reverse Reverses the character order of a string. ``` reverse(str) ``` **Arguments** - `str`: String expression. ### right Returns a specified number of characters from the right side of a string. ``` right(str, n) ``` **Arguments** - `str`: String expression. - `n`: Number of characters to return. **Related:** `left` ### rpad Pads the right side of a string with another string to a specified string length. ``` rpad(str, n[, padding_str]) ``` **Arguments** - `str`: String expression. - `n`: String length to pad to. If the input string is longer than this length, it is truncated. - `padding_str`: String expression to pad with. Can be a constant, column, or function, and any combination of string operators. _Default is a space._ **Related:** `lpad` ### rtrim Trims the specified trim string from the end of a string. If no trim string is provided, all spaces are removed from the end of the input string. ``` rtrim(str[, trim_str]) ``` **Arguments** - `str`: String expression. - `trim_str`: String expression to trim from the end of the input string. Can be a constant, column, or function, and any combination of arithmetic operators. _Default is a space._ **Related:** `btrim`, `ltrim` ### sha224 Computes the SHA-224 hash of a binary string. ``` sha224(expression) ``` **Arguments** - `expression`: String expression. ```sql > select sha224('foo'); +----------------------------------------------------------+ | sha224(Utf8("foo")) | +----------------------------------------------------------+ | 0808f64e60d58979fcb676c96ec938270dea42445aeefcd3a4e6f8db | +----------------------------------------------------------+ ``` ### sha256 Computes the SHA-256 hash of a binary string. ``` sha256(expression) ``` **Arguments** - `expression`: String expression. ```sql > select sha256('foo'); +------------------------------------------------------------------+ | sha256(Utf8("foo")) | +------------------------------------------------------------------+ | 2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae | +------------------------------------------------------------------+ ``` ### sha384 Computes the SHA-384 hash of a binary string. ``` sha384(expression) ``` **Arguments** - `expression`: String expression. ```sql > select sha384('foo'); +--------------------------------------------------------------------------------------------------+ | sha384(Utf8("foo")) | +--------------------------------------------------------------------------------------------------+ | 98c11ffdfdd540676b1a137cb1a22b2a70350c9a44171d6b1180c6be5cbb2ee3f79d532c8a1dd9ef2e8e08e752a3babb | +--------------------------------------------------------------------------------------------------+ ``` ### sha512 Computes the SHA-512 hash of a binary string. ``` sha512(expression) ``` **Arguments** - `expression`: String expression. ```sql > select sha512('foo'); +----------------------------------------------------------------------------------------------------------------------------------+ | sha512(Utf8("foo")) | +----------------------------------------------------------------------------------------------------------------------------------+ | f7fbba6e0636f890e56fbbf3283e524c6fa3204ae298382d624741d0dc6638326e282c41be5e4254d8820772c5518a2c5a8c0c7f7eda19594a7eb539453e1ed7 | +----------------------------------------------------------------------------------------------------------------------------------+ ``` ### split_part Splits a string based on a specified delimiter and returns the substring in the specified position. ``` split_part(str, delimiter, pos) ``` **Arguments** - `str`: String expression. - `delimiter`: String or character to split on. - `pos`: Position of the part to return (counting from 1). Negative values count backward from the end of the string. ```sql > select split_part('1.2.3.4.5', '.', 3); +--------------------------------------------------+ | split_part(Utf8("1.2.3.4.5"),Utf8("."),Int64(3)) | +--------------------------------------------------+ | 3 | +--------------------------------------------------+ ``` ### starts_with Tests if a string starts with a substring. ``` starts_with(str, substr) ``` **Arguments** - `str`: String expression. - `substr`: Substring to test for. ### strpos Returns the starting position of a specified substring in a string. Positions begin at 1. If the substring does not exist in the string, the function returns 0. ``` strpos(str, substr) ``` **Arguments** - `str`: String expression. - `substr`: Substring expression to search for. ### substr Extracts a substring of a specified number of characters from a specific starting position in a string. ``` substr(str, start_pos[, length]) ``` **Arguments** - `str`: String expression. - `start_pos`: Character position to start the substring at. The first character in the string has a position of 1. If the start position is less than 1, it is treated as if it is before the start of the string and the (absolute) number of characters before position 1 is subtracted from `length` (if given). For example, `substr('abc', -3, 6)` returns `'ab'`. - `length`: Number of characters to extract. If not specified, returns the rest of the string after the start position. ### substr_index Returns the substring from str before count occurrences of the delimiter delim. If count is positive, everything to the left of the final delimiter (counting from the left) is returned. If count is negative, everything to the right of the final delimiter (counting from the right) is returned. ``` substr_index(str, delim, count) ``` **Arguments** - `str`: String expression. - `delim`: The string to find in str to split str. - `count`: The number of times to search for the delimiter. Can be either a positive or negative number. ```sql > select substr_index('www.apache.org', '.', 1); +---------------------------------------------------------+ | substr_index(Utf8("www.apache.org"),Utf8("."),Int64(1)) | +---------------------------------------------------------+ | www | +---------------------------------------------------------+ > select substr_index('www.apache.org', '.', -1); +----------------------------------------------------------+ | substr_index(Utf8("www.apache.org"),Utf8("."),Int64(-1)) | +----------------------------------------------------------+ | org | +----------------------------------------------------------+ ``` ### to_hex Converts an integer to a hexadecimal string. ``` to_hex(int) ``` **Arguments** - `int`: Integer expression. ```sql > select to_hex(12345689); +-------------------------+ | to_hex(Int64(12345689)) | +-------------------------+ | bc6159 | +-------------------------+ ``` ### translate Performs character-wise substitution based on a mapping. ``` translate(str, from, to) ``` **Arguments** - `str`: String expression. - `from`: The characters to be replaced. - `to`: The characters to replace them with. Each character in **from** that is found in **str** is replaced by the character at the same index in **to**. Any characters in **from** that don't have a corresponding character in **to** are removed. If a character appears more than once in **from**, the first occurrence determines the mapping. ```sql > select translate('twice', 'wic', 'her'); +--------------------------------------------------+ | translate(Utf8("twice"),Utf8("wic"),Utf8("her")) | +--------------------------------------------------+ | there | +--------------------------------------------------+ ``` ### upper Converts a string to upper-case. ``` upper(str) ``` **Arguments** - `str`: String expression. **Related:** `initcap`, `lower` ### uuid Returns [`UUID v4`](https://en.wikipedia.org/wiki/Universally_unique_identifier#Version_4_%28random%29) string value which is unique per row. ``` uuid() ``` ```sql > select uuid(); +--------------------------------------+ | uuid() | +--------------------------------------+ | 6ec17ef8-1934-41cc-8d59-d0c8f9eea1f0 | +--------------------------------------+ ``` --- # Date & time functions Source: https://www.hotdata.dev/docs/sql-functions-datetime Site index: https://www.hotdata.dev/llms.txt Reference for date & time functions in [HotSQL](/docs/sql). Names, signatures, and examples match the engine exactly. ### current_date Returns the current date in the session time zone. The `current_date()` return value is determined at query time and will return the same date, no matter when in the query plan the function executes. ``` current_date() ``` ### current_time Returns the current time in the session time zone. The `current_time()` return value is determined at query time and will return the same time, no matter when in the query plan the function executes. ``` current_time() ``` ### date_bin Calculates time intervals and returns the start of the interval nearest to the specified timestamp. Use `date_bin` to downsample time series data by grouping rows into time-based "bins" or "windows" and applying an aggregate or selector function to each window. For example, if you "bin" or "window" data into 15 minute intervals, an input timestamp of `2023-01-01T18:18:18Z` will be updated to the start time of the 15 minute bin it is in: `2023-01-01T18:15:00Z`. ``` date_bin(interval, expression, origin-timestamp) ``` **Arguments** - `interval`: Bin interval. - `expression`: Time expression to operate on. Can be a constant, column, or function. - `origin-timestamp`: Optional. Starting point used to determine bin boundaries. If not specified defaults 1970-01-01T00:00:00Z (the UNIX epoch in UTC). The following intervals are supported: - nanoseconds - microseconds - milliseconds - seconds - minutes - hours - days - weeks - months - years - century ```sql -- Bin the timestamp into 1 day intervals > SELECT date_bin(interval '1 day', time) as bin FROM VALUES ('2023-01-01T18:18:18Z'), ('2023-01-03T19:00:03Z') t(time); +---------------------+ | bin | +---------------------+ | 2023-01-01T00:00:00 | | 2023-01-03T00:00:00 | +---------------------+ 2 row(s) fetched. -- Bin the timestamp into 1 day intervals starting at 3AM on 2023-01-01 > SELECT date_bin(interval '1 day', time, '2023-01-01T03:00:00') as bin FROM VALUES ('2023-01-01T18:18:18Z'), ('2023-01-03T19:00:03Z') t(time); +---------------------+ | bin | +---------------------+ | 2023-01-01T03:00:00 | | 2023-01-03T03:00:00 | +---------------------+ 2 row(s) fetched. -- Bin the time into 15 minute intervals starting at 1 min > SELECT date_bin(interval '15 minutes', time, TIME '00:01:00') as bin FROM VALUES (TIME '02:18:18'), (TIME '19:00:03') t(time); +----------+ | bin | +----------+ | 02:16:00 | | 18:46:00 | +----------+ 2 row(s) fetched. ``` ### date_part Returns the specified part of the date as an integer. ``` date_part(part, expression) ``` **Arguments** - `part`: Part of the date to return. The following date parts are supported: - year - isoyear (ISO 8601 week-numbering year) - quarter (emits value in inclusive range [1, 4] based on which quartile of the year the date is in) - month - week (week of the year) - day (day of the month) - hour - minute - second - millisecond - microsecond - nanosecond - dow (day of the week where Sunday is 0) - doy (day of the year) - epoch (seconds since Unix epoch for timestamps/dates, total seconds for intervals) - isodow (ISO 8601 day of the week where Monday is 1 and Sunday is 7) - `expression`: Time expression to operate on. Can be a constant, column, or function. ```sql > SELECT date_part('year', '2024-05-01T00:00:00'); +-----------------------------------------------------+ | date_part(Utf8("year"),Utf8("2024-05-01T00:00:00")) | +-----------------------------------------------------+ | 2024 | +-----------------------------------------------------+ > SELECT extract(day FROM timestamp '2024-05-01T00:00:00'); +----------------------------------------------------+ | date_part(Utf8("DAY"),Utf8("2024-05-01T00:00:00")) | +----------------------------------------------------+ | 1 | +----------------------------------------------------+ ``` ### date_trunc Truncates a timestamp or time value to a specified precision. ``` date_trunc(precision, expression) ``` **Arguments** - `precision`: Time precision to truncate to. The following precisions are supported: For Timestamp types: - year / YEAR - quarter / QUARTER - month / MONTH - week / WEEK - day / DAY - hour / HOUR - minute / MINUTE - second / SECOND - millisecond / MILLISECOND - microsecond / MICROSECOND For Time types (hour, minute, second, millisecond, microsecond only): - hour / HOUR - minute / MINUTE - second / SECOND - millisecond / MILLISECOND - microsecond / MICROSECOND - `expression`: Timestamp or time expression to operate on. Can be a constant, column, or function. ```sql > SELECT date_trunc('month', '2024-05-15T10:30:00'); +-----------------------------------------------+ | date_trunc(Utf8("month"),Utf8("2024-05-15T10:30:00")) | +-----------------------------------------------+ | 2024-05-01T00:00:00 | +-----------------------------------------------+ > SELECT date_trunc('hour', '2024-05-15T10:30:00'); +----------------------------------------------+ | date_trunc(Utf8("hour"),Utf8("2024-05-15T10:30:00")) | +----------------------------------------------+ | 2024-05-15T10:00:00 | +----------------------------------------------+ ``` ### from_unixtime Converts an integer to RFC3339 timestamp format (`YYYY-MM-DDT00:00:00.000000000Z`). Integers and unsigned integers are interpreted as seconds since the unix epoch (`1970-01-01T00:00:00Z`) return the corresponding timestamp. ``` from_unixtime(expression[, timezone]) ``` **Arguments** - `timezone`: Optional timezone to use when converting the integer to a timestamp. If not provided, the default timezone is UTC. ```sql > select from_unixtime(1599572549, 'America/New_York'); +-----------------------------------------------------------+ | from_unixtime(Int64(1599572549),Utf8("America/New_York")) | +-----------------------------------------------------------+ | 2020-09-08T09:42:29-04:00 | +-----------------------------------------------------------+ ``` ### make_date Make a date from year/month/day component parts. ``` make_date(year, month, day) ``` **Arguments** - `year`: Year to use when making the date. Can be a constant, column or function, and any combination of arithmetic operators. - `month`: Month to use when making the date. Can be a constant, column or function, and any combination of arithmetic operators. - `day`: Day to use when making the date. Can be a constant, column or function, and any combination of arithmetic operators. ### make_time Make a time from hour/minute/second component parts. ``` make_time(hour, minute, second) ``` **Arguments** - `hour`: Hour to use when making the time. Can be a constant, column or function, and any combination of arithmetic operators. - `minute`: Minute to use when making the time. Can be a constant, column or function, and any combination of arithmetic operators. - `second`: Second to use when making the time. Can be a constant, column or function, and any combination of arithmetic operators. ### now Returns the current timestamp in the system configured timezone (None by default). The `now()` return value is determined at query time and will return the same timestamp, no matter when in the query plan the function executes. ``` now() ``` ### to_char Returns a string representation of a date, time, timestamp or duration based on a [Chrono format](https://docs.rs/chrono/latest/chrono/format/strftime/index.html). Unlike the PostgreSQL equivalent of this function numerical formatting is not supported. ``` to_char(expression, format) ``` **Arguments** - `expression`: Expression to operate on. Can be a constant, column, or function that results in a date, time, timestamp or duration. - `format`: A [Chrono format](https://docs.rs/chrono/latest/chrono/format/strftime/index.html) string to use to convert the expression. - `day`: Day to use when making the date. Can be a constant, column or function, and any combination of arithmetic operators. ### to_date Converts a value to a date (`YYYY-MM-DD`). Supports strings, numeric and timestamp types as input. Strings are parsed as YYYY-MM-DD (e.g. '2023-07-20') if no [Chrono format](https://docs.rs/chrono/latest/chrono/format/strftime/index.html)s are provided. Integers and doubles are interpreted as days since the unix epoch (`1970-01-01T00:00:00Z`). Returns the corresponding date. Note: `to_date` returns Date32, which represents its values as the number of days since unix epoch(`1970-01-01`) stored as signed 32 bit value. The largest supported date value is `9999-12-31`. ``` to_date('2017-05-31', '%Y-%m-%d') ``` **Arguments** - `expression`: String expression. - `format_n`: Optional [Chrono format](https://docs.rs/chrono/latest/chrono/format/strftime/index.html) strings to use to parse the expression. Formats will be tried in the order they appear with the first successful one being returned. If none of the formats successfully parse the expression an error will be returned. ### to_local_time Converts a timestamp with a timezone to a timestamp without a timezone (with no offset or timezone information). This function handles daylight saving time changes. ``` to_local_time(expression) ``` **Arguments** - `expression`: Time expression to operate on. Can be a constant, column, or function. ```sql > SELECT to_local_time('2024-04-01T00:00:20Z'::timestamp); +---------------------------------------------+ | to_local_time(Utf8("2024-04-01T00:00:20Z")) | +---------------------------------------------+ | 2024-04-01T00:00:20 | +---------------------------------------------+ > SELECT to_local_time('2024-04-01T00:00:20Z'::timestamp AT TIME ZONE 'Europe/Brussels'); +---------------------------------------------+ | to_local_time(Utf8("2024-04-01T00:00:20Z")) | +---------------------------------------------+ | 2024-04-01T00:00:20 | +---------------------------------------------+ > SELECT time, arrow_typeof(time) as type, to_local_time(time) as to_local_time, arrow_typeof(to_local_time(time)) as to_local_time_type FROM ( SELECT '2024-04-01T00:00:20Z'::timestamp AT TIME ZONE 'Europe/Brussels' AS time ); +---------------------------+----------------------------------+---------------------+--------------------+ | time | type | to_local_time | to_local_time_type | +---------------------------+----------------------------------+---------------------+--------------------+ | 2024-04-01T00:00:20+02:00 | Timestamp(ns, "Europe/Brussels") | 2024-04-01T00:00:20 | Timestamp(ns) | +---------------------------+----------------------------------+---------------------+--------------------+ # combine `to_local_time()` with `date_bin()` to bin on boundaries in the timezone rather # than UTC boundaries > SELECT date_bin(interval '1 day', to_local_time('2024-04-01T00:00:20Z'::timestamp AT TIME ZONE 'Europe/Brussels')) AS date_bin; +---------------------+ | date_bin | +---------------------+ | 2024-04-01T00:00:00 | +---------------------+ > SELECT date_bin(interval '1 day', to_local_time('2024-04-01T00:00:20Z'::timestamp AT TIME ZONE 'Europe/Brussels')) AT TIME ZONE 'Europe/Brussels' AS date_bin_with_timezone; +---------------------------+ | date_bin_with_timezone | +---------------------------+ | 2024-04-01T00:00:00+02:00 | +---------------------------+ ``` ### to_time Converts a value to a time (`HH:MM:SS.nnnnnnnnn`). Supports strings and timestamps as input. Strings are parsed as `HH:MM:SS`, `HH:MM:SS.nnnnnnnnn`, or `HH:MM` if no [Chrono format](https://docs.rs/chrono/latest/chrono/format/strftime/index.html)s are provided. Timestamps will have the time portion extracted. Returns the corresponding time. Note: `to_time` returns Time64(Nanosecond), which represents the time of day in nanoseconds since midnight. ``` to_time('12:30:45', '%H:%M:%S') ``` **Arguments** - `expression`: String or Timestamp expression. - `format_n`: Optional [Chrono format](https://docs.rs/chrono/latest/chrono/format/strftime/index.html) strings to use to parse the expression. Formats will be tried in the order they appear with the first successful one being returned. If none of the formats successfully parse the expression an error will be returned. ### to_timestamp Converts a value to a timestamp (`YYYY-MM-DDT00:00:00.000000<TZ>`) in the session time zone. Supports strings, integer, unsigned integer, and double types as input. Strings are parsed as RFC3339 (e.g. '2023-07-20T05:44:00') if no [Chrono formats](https://docs.rs/chrono/latest/chrono/format/strftime/index.html) are provided. Strings that parse without a time zone are treated as if they are in the session time zone, or UTC if no session time zone is set. Integers, unsigned integers, and doubles are interpreted as seconds since the unix epoch (`1970-01-01T00:00:00Z`). Note: `to_timestamp` returns `Timestamp(ns, TimeZone)` where the time zone is the session time zone. The supported range for integer input is between`-9223372037` and `9223372036`. Supported range for string input is between `1677-09-21T00:12:44.0` and `2262-04-11T23:47:16.0`. Please use `to_timestamp_seconds` for the input outside of supported bounds. The session time zone can be set using the statement `SET TIMEZONE = 'desired time zone'`. The time zone can be a value like +00:00, 'Europe/London' etc. ``` to_timestamp(expression[, ..., format_n]) ``` **Arguments** - `expression`: Expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. - `format_n`: Optional [Chrono format](https://docs.rs/chrono/latest/chrono/format/strftime/index.html) strings to use to parse the expression. Formats will be tried in the order they appear with the first successful one being returned. If none of the formats successfully parse the expression an error will be returned. Note: parsing of named timezones (e.g. 'America/New_York') using %Z is only supported at the end of the string preceded by a space. ### to_timestamp_micros Converts a value to a timestamp (`YYYY-MM-DDT00:00:00.000000<TZ>`) in the session time zone. Supports strings, integer, unsigned integer, and double types as input. Strings are parsed as RFC3339 (e.g. '2023-07-20T05:44:00') if no [Chrono formats](https://docs.rs/chrono/latest/chrono/format/strftime/index.html) are provided. Strings that parse without a time zone are treated as if they are in the session time zone, or UTC if no session time zone is set. Integers, unsigned integers, and doubles are interpreted as microseconds since the unix epoch (`1970-01-01T00:00:00Z`). The session time zone can be set using the statement `SET TIMEZONE = 'desired time zone'`. The time zone can be a value like +00:00, 'Europe/London' etc. ``` to_timestamp_micros(expression[, ..., format_n]) ``` **Arguments** - `expression`: Expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. - `format_n`: Optional [Chrono format](https://docs.rs/chrono/latest/chrono/format/strftime/index.html) strings to use to parse the expression. Formats will be tried in the order they appear with the first successful one being returned. If none of the formats successfully parse the expression an error will be returned. Note: parsing of named timezones (e.g. 'America/New_York') using %Z is only supported at the end of the string preceded by a space. ### to_timestamp_millis Converts a value to a timestamp (`YYYY-MM-DDT00:00:00.000<TZ>`) in the session time zone. Supports strings, integer, unsigned integer, and double types as input. Strings are parsed as RFC3339 (e.g. '2023-07-20T05:44:00') if no [Chrono formats](https://docs.rs/chrono/latest/chrono/format/strftime/index.html) are provided. Strings that parse without a time zone are treated as if they are in the session time zone, or UTC if no session time zone is set. Integers, unsigned integers, and doubles are interpreted as milliseconds since the unix epoch (`1970-01-01T00:00:00Z`). The session time zone can be set using the statement `SET TIMEZONE = 'desired time zone'`. The time zone can be a value like +00:00, 'Europe/London' etc. ``` to_timestamp_millis(expression[, ..., format_n]) ``` **Arguments** - `expression`: Expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. - `format_n`: Optional [Chrono format](https://docs.rs/chrono/latest/chrono/format/strftime/index.html) strings to use to parse the expression. Formats will be tried in the order they appear with the first successful one being returned. If none of the formats successfully parse the expression an error will be returned. Note: parsing of named timezones (e.g. 'America/New_York') using %Z is only supported at the end of the string preceded by a space. ### to_timestamp_nanos Converts a value to a timestamp (`YYYY-MM-DDT00:00:00.000000000<TZ>`) in the session time zone. Supports strings, integer, unsigned integer, and double types as input. Strings are parsed as RFC3339 (e.g. '2023-07-20T05:44:00') if no [Chrono formats](https://docs.rs/chrono/latest/chrono/format/strftime/index.html) are provided. Strings that parse without a time zone are treated as if they are in the session time zone. Integers, unsigned integers, and doubles are interpreted as nanoseconds since the unix epoch (`1970-01-01T00:00:00Z`). The session time zone can be set using the statement `SET TIMEZONE = 'desired time zone'`. The time zone can be a value like +00:00, 'Europe/London' etc. ``` to_timestamp_nanos(expression[, ..., format_n]) ``` **Arguments** - `expression`: Expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. - `format_n`: Optional [Chrono format](https://docs.rs/chrono/latest/chrono/format/strftime/index.html) strings to use to parse the expression. Formats will be tried in the order they appear with the first successful one being returned. If none of the formats successfully parse the expression an error will be returned. Note: parsing of named timezones (e.g. 'America/New_York') using %Z is only supported at the end of the string preceded by a space. ### to_timestamp_seconds Converts a value to a timestamp (`YYYY-MM-DDT00:00:00<TZ>`) in the session time zone. Supports strings, integer, unsigned integer, and double types as input. Strings are parsed as RFC3339 (e.g. '2023-07-20T05:44:00') if no [Chrono formats](https://docs.rs/chrono/latest/chrono/format/strftime/index.html) are provided. Strings that parse without a time zone are treated as if they are in the session time zone, or UTC if no session time zone is set. Integers, unsigned integers, and doubles are interpreted as seconds since the unix epoch (`1970-01-01T00:00:00Z`). The session time zone can be set using the statement `SET TIMEZONE = 'desired time zone'`. The time zone can be a value like +00:00, 'Europe/London' etc. ``` to_timestamp_seconds(expression[, ..., format_n]) ``` **Arguments** - `expression`: Expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. - `format_n`: Optional [Chrono format](https://docs.rs/chrono/latest/chrono/format/strftime/index.html) strings to use to parse the expression. Formats will be tried in the order they appear with the first successful one being returned. If none of the formats successfully parse the expression an error will be returned. Note: parsing of named timezones (e.g. 'America/New_York') using %Z is only supported at the end of the string preceded by a space. ### to_unixtime Converts a value to seconds since the unix epoch (`1970-01-01T00:00:00`). Supports strings, dates, timestamps, integer, unsigned integer, and float types as input. Strings are parsed as RFC3339 (e.g. '2023-07-20T05:44:00') if no [Chrono formats](https://docs.rs/chrono/latest/chrono/format/strftime/index.html) are provided. Integers, unsigned integers, and floats are interpreted as seconds since the unix epoch (`1970-01-01T00:00:00`). ``` to_unixtime(expression[, ..., format_n]) ``` **Arguments** - `expression`: Expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. - `format_n`: Optional [Chrono format](https://docs.rs/chrono/latest/chrono/format/strftime/index.html) strings to use to parse the expression. Formats will be tried in the order they appear with the first successful one being returned. If none of the formats successfully parse the expression an error will be returned. ```sql > select to_unixtime('2020-09-08T12:00:00+00:00'); +------------------------------------------------+ | to_unixtime(Utf8("2020-09-08T12:00:00+00:00")) | +------------------------------------------------+ | 1599566400 | +------------------------------------------------+ > select to_unixtime('01-14-2023 01:01:30+05:30', '%q', '%d-%m-%Y %H/%M/%S', '%+', '%m-%d-%Y %H:%M:%S%#z'); +-----------------------------------------------------------------------------------------------------------------------------+ | to_unixtime(Utf8("01-14-2023 01:01:30+05:30"),Utf8("%q"),Utf8("%d-%m-%Y %H/%M/%S"),Utf8("%+"),Utf8("%m-%d-%Y %H:%M:%S%#z")) | +-----------------------------------------------------------------------------------------------------------------------------+ | 1673638290 | +-----------------------------------------------------------------------------------------------------------------------------+ ``` --- # Conditional & utility functions Source: https://www.hotdata.dev/docs/sql-functions-conditional Site index: https://www.hotdata.dev/llms.txt Reference for conditional & utility functions in [HotSQL](/docs/sql). Names, signatures, and examples match the engine exactly. ### arrow_cast Casts a value to a specific Arrow data type. ``` arrow_cast(expression, datatype) ``` **Arguments** - `expression`: Expression to cast. The expression can be a constant, column, or function, and any combination of operators. - `datatype`: [Arrow data type](https://docs.rs/arrow/latest/arrow/datatypes/enum.DataType.html) name to cast to, as a string. The format is the same as that returned by [`arrow_typeof`] ```sql > select arrow_cast(-5, 'Int8') as a, arrow_cast('foo', 'Dictionary(Int32, Utf8)') as b, arrow_cast('bar', 'LargeUtf8') as c; +----+-----+-----+ | a | b | c | +----+-----+-----+ | -5 | foo | bar | +----+-----+-----+ > select arrow_cast('2023-01-02T12:53:02', 'Timestamp(µs, "+08:00")') as d, arrow_cast('2023-01-02T12:53:02', 'Timestamp(µs)') as e; +---------------------------+---------------------+ | d | e | +---------------------------+---------------------+ | 2023-01-02T12:53:02+08:00 | 2023-01-02T12:53:02 | +---------------------------+---------------------+ ``` ### arrow_field Returns a struct containing the Arrow field information of the expression, including name, data type, nullability, and metadata. ``` arrow_field(expression) ``` **Arguments** - `expression`: Expression to evaluate. The expression can be a constant, column, or function, and any combination of operators. ```sql > select arrow_field(1); +-------------------------------------------------------------+ | arrow_field(Int64(1)) | +-------------------------------------------------------------+ | {name: lit, data_type: Int64, nullable: false, metadata: {}} | +-------------------------------------------------------------+ > select arrow_field(1)['data_type']; +-----------------------------------+ | arrow_field(Int64(1))[data_type] | +-----------------------------------+ | Int64 | +-----------------------------------+ ``` ### arrow_metadata Returns the metadata of the input expression. If a key is provided, returns the value for that key. If no key is provided, returns a Map of all metadata. ``` arrow_metadata(expression[, key]) ``` **Arguments** - `expression`: The expression to retrieve metadata from. Can be a column or other expression. - `key`: Optional. The specific metadata key to retrieve. ```sql > select arrow_metadata(col) from table; +----------------------------+ | arrow_metadata(table.col) | +----------------------------+ | {k: v} | +----------------------------+ > select arrow_metadata(col, 'k') from table; +-------------------------------+ | arrow_metadata(table.col, 'k')| +-------------------------------+ | v | +-------------------------------+ ``` ### arrow_try_cast Casts a value to a specific Arrow data type, returning NULL if the cast fails. ``` arrow_try_cast(expression, datatype) ``` **Arguments** - `expression`: Expression to cast. The expression can be a constant, column, or function, and any combination of operators. - `datatype`: [Arrow data type](https://docs.rs/arrow/latest/arrow/datatypes/enum.DataType.html) name to cast to, as a string. The format is the same as that returned by [`arrow_typeof`] ```sql > select arrow_try_cast('123', 'Int64') as a, arrow_try_cast('not_a_number', 'Int64') as b; +-----+------+ | a | b | +-----+------+ | 123 | NULL | +-----+------+ ``` ### arrow_typeof Returns the name of the underlying [Arrow data type](https://docs.rs/arrow/latest/arrow/datatypes/enum.DataType.html) of the expression. ``` arrow_typeof(expression) ``` **Arguments** - `expression`: Expression to evaluate. The expression can be a constant, column, or function, and any combination of operators. ```sql > select arrow_typeof('foo'), arrow_typeof(1); +---------------------------+------------------------+ | arrow_typeof(Utf8("foo")) | arrow_typeof(Int64(1)) | +---------------------------+------------------------+ | Utf8 | Int64 | +---------------------------+------------------------+ ``` ### cast_to_type Casts the first argument to the data type of the second argument. Only the type of the second argument is used; its value is ignored. ``` cast_to_type(expression, reference) ``` **Arguments** - `expression`: The expression to cast. It can be a constant, column, or function, and any combination of operators. - `reference`: Reference expression whose data type determines the target cast type. The value is ignored. ```sql > select cast_to_type('42', NULL::INTEGER) as a; +----+ | a | +----+ | 42 | +----+ > select cast_to_type(1 + 2, NULL::DOUBLE) as b; +-----+ | b | +-----+ | 3.0 | +-----+ ``` ### coalesce Returns the first of its arguments that is not _null_. Returns _null_ if all arguments are _null_. This function is often used to substitute a default value for _null_ values. ``` coalesce(expression1[, ..., expression_n]) ``` **Arguments** - `expression1, expression_n`: Expression to use if previous expressions are _null_. Can be a constant, column, or function, and any combination of arithmetic operators. Pass as many expression arguments as necessary. ### get_field Returns a field within a map or a struct with the given key. Supports nested field access by providing multiple field names. Note: most users invoke `get_field` indirectly via field access syntax such as `my_struct_col['field_name']` which results in a call to `get_field(my_struct_col, 'field_name')`. Nested access like `my_struct['a']['b']` is optimized to a single call: `get_field(my_struct, 'a', 'b')`. ``` get_field(expression, field_name[, field_name2, ...]) ``` **Arguments** - `expression`: The map or struct to retrieve a field from. - `field_name`: The field name(s) to access, in order for nested access. Must evaluate to strings. ```sql > -- Access a field from a struct column > create table test( struct_col) as values ({name: 'Alice', age: 30}), ({name: 'Bob', age: 25}); > select struct_col from test; +-----------------------------+ | struct_col | +-----------------------------+ | {name: Alice, age: 30} | | {name: Bob, age: 25} | +-----------------------------+ > select struct_col['name'] as name from test; +-------+ | name | +-------+ | Alice | | Bob | +-------+ > -- Nested field access with multiple arguments > create table test(struct_col) as values ({outer: {inner_val: 42}}); > select struct_col['outer']['inner_val'] as result from test; +--------+ | result | +--------+ | 42 | +--------+ ``` ### greatest Returns the greatest value in a list of expressions. Returns _null_ if all expressions are _null_. ``` greatest(expression1[, ..., expression_n]) ``` **Arguments** - `expression1, expression_n`: Expressions to compare and return the greatest value.. Can be a constant, column, or function, and any combination of arithmetic operators. Pass as many expression arguments as necessary. ```sql > select greatest(4, 7, 5); +---------------------------+ | greatest(4,7,5) | +---------------------------+ | 7 | +---------------------------+ ``` ### least Returns the smallest value in a list of expressions. Returns _null_ if all expressions are _null_. ``` least(expression1[, ..., expression_n]) ``` **Arguments** - `expression1, expression_n`: Expressions to compare and return the smallest value. Can be a constant, column, or function, and any combination of arithmetic operators. Pass as many expression arguments as necessary. ```sql > select least(4, 7, 5); +---------------------------+ | least(4,7,5) | +---------------------------+ | 4 | +---------------------------+ ``` ### nullif Returns _null_ if _expression1_ equals _expression2_; otherwise it returns _expression1_. This can be used to perform the inverse operation of [`coalesce`](#coalesce). ``` nullif(expression1, expression2) ``` **Arguments** - `expression1`: Expression to compare and return if equal to expression2. Can be a constant, column, or function, and any combination of operators. - `expression2`: Expression to compare to expression1. Can be a constant, column, or function, and any combination of operators. ### nvl Returns _expression2_ if _expression1_ is NULL otherwise it returns _expression1_ and _expression2_ is not evaluated. This function can be used to substitute a default value for NULL values. ``` nvl(expression1, expression2) ``` **Arguments** - `expression1`: Expression to return if not null. Can be a constant, column, or function, and any combination of operators. - `expression2`: Expression to return if expr1 is null. Can be a constant, column, or function, and any combination of operators. ```sql > select nvl(null, 'a'); +---------------------+ | nvl(NULL,Utf8("a")) | +---------------------+ | a | +---------------------+\ > select nvl('b', 'a'); +--------------------------+ | nvl(Utf8("b"),Utf8("a")) | +--------------------------+ | b | +--------------------------+ ``` ### nvl2 Returns _expression2_ if _expression1_ is not NULL; otherwise it returns _expression3_. ``` nvl2(expression1, expression2, expression3) ``` **Arguments** - `expression1`: Expression to test for null. Can be a constant, column, or function, and any combination of operators. - `expression2`: Expression to return if expr1 is not null. Can be a constant, column, or function, and any combination of operators. - `expression3`: Expression to return if expr1 is null. Can be a constant, column, or function, and any combination of operators. ```sql > select nvl2(null, 'a', 'b'); +--------------------------------+ | nvl2(NULL,Utf8("a"),Utf8("b")) | +--------------------------------+ | b | +--------------------------------+ > select nvl2('data', 'a', 'b'); +----------------------------------------+ | nvl2(Utf8("data"),Utf8("a"),Utf8("b")) | +----------------------------------------+ | a | +----------------------------------------+ ``` ### try_cast_to_type Casts the first argument to the data type of the second argument, returning NULL if the cast fails. Only the type of the second argument is used; its value is ignored. ``` try_cast_to_type(expression, reference) ``` **Arguments** - `expression`: The expression to cast. It can be a constant, column, or function, and any combination of operators. - `reference`: Reference expression whose data type determines the target cast type. The value is ignored. ```sql > select try_cast_to_type('123', NULL::INTEGER) as a, try_cast_to_type('not_a_number', NULL::INTEGER) as b; +-----+------+ | a | b | +-----+------+ | 123 | NULL | +-----+------+ ``` ### version Returns the engine version string. ``` version() ``` ### with_metadata Attaches Arrow field metadata (key/value pairs) to the input expression. Keys must be non-empty constant strings and values must be constant strings (empty values are allowed). Existing metadata on the input field is preserved; new keys overwrite on collision. This is the inverse of `arrow_metadata`. ``` with_metadata(expression, key1, value1[, key2, value2, ...]) ``` **Arguments** - `expression`: The expression whose output Arrow field should be annotated. Values flow through unchanged. - `key`: Metadata key. Must be a non-empty constant string literal. - `value`: Metadata value. Must be a constant string literal (may be empty). ```sql > select arrow_metadata(with_metadata(column1, 'unit', 'ms'), 'unit') from (values (1)); +---------------------------------------------------------------+ | arrow_metadata(with_metadata(column1,Utf8("unit"),Utf8("ms")),Utf8("unit")) | +---------------------------------------------------------------+ | ms | +---------------------------------------------------------------+ > select arrow_metadata(with_metadata(column1, 'unit', 'ms', 'source', 'sensor')) from (values (1)); +--------------------------+ | {source: sensor, unit: ms} | +--------------------------+ ``` --- # Aggregate functions Source: https://www.hotdata.dev/docs/sql-functions-aggregate Site index: https://www.hotdata.dev/llms.txt Reference for aggregate functions in [HotSQL](/docs/sql). Names, signatures, and examples match the engine exactly. ### approx_distinct Returns the approximate number of distinct input values calculated using the HyperLogLog algorithm. ``` approx_distinct(expression) ``` ```sql > SELECT approx_distinct(column_name) FROM table_name; +-----------------------------------+ | approx_distinct(column_name) | +-----------------------------------+ | 42 | +-----------------------------------+ ``` ### approx_median Returns the approximate median (50th percentile) of input values. It is an alias of `approx_percentile_cont(0.5) WITHIN GROUP (ORDER BY x)`. ``` approx_median(expression) ``` ```sql > SELECT approx_median(column_name) FROM table_name; +-----------------------------------+ | approx_median(column_name) | +-----------------------------------+ | 23.5 | +-----------------------------------+ ``` ### approx_percentile_cont Returns the approximate percentile of input values using the t-digest algorithm. ``` approx_percentile_cont(percentile [, centroids]) WITHIN GROUP (ORDER BY expression) ``` **Arguments** - `percentile`: Percentile to compute. Must be a float value between 0 and 1 (inclusive). - `centroids`: Number of centroids to use in the t-digest algorithm. _Default is 100_. A higher number results in more accurate approximation but requires more memory. ```sql > SELECT approx_percentile_cont(0.75) WITHIN GROUP (ORDER BY column_name) FROM table_name; +------------------------------------------------------------------+ | approx_percentile_cont(0.75) WITHIN GROUP (ORDER BY column_name) | +------------------------------------------------------------------+ | 65.0 | +------------------------------------------------------------------+ > SELECT approx_percentile_cont(0.75, 100) WITHIN GROUP (ORDER BY column_name) FROM table_name; +-----------------------------------------------------------------------+ | approx_percentile_cont(0.75, 100) WITHIN GROUP (ORDER BY column_name) | +-----------------------------------------------------------------------+ | 65.0 | +-----------------------------------------------------------------------+ ``` An alternate syntax is also supported: ```sql > SELECT approx_percentile_cont(column_name, 0.75) FROM table_name; +-----------------------------------------------+ | approx_percentile_cont(column_name, 0.75) | +-----------------------------------------------+ | 65.0 | +-----------------------------------------------+ > SELECT approx_percentile_cont(column_name, 0.75, 100) FROM table_name; +----------------------------------------------------------+ | approx_percentile_cont(column_name, 0.75, 100) | +----------------------------------------------------------+ | 65.0 | +----------------------------------------------------------+ ``` ### approx_percentile_cont_with_weight Returns the weighted approximate percentile of input values using the t-digest algorithm. ``` approx_percentile_cont_with_weight(weight, percentile [, centroids]) WITHIN GROUP (ORDER BY expression) ``` **Arguments** - `expression`: The expression. - `weight`: Expression to use as weight. Can be a constant, column, or function, and any combination of arithmetic operators. - `percentile`: Percentile to compute. Must be a float value between 0 and 1 (inclusive). - `centroids`: Number of centroids to use in the t-digest algorithm. _Default is 100_. A higher number results in more accurate approximation but requires more memory. ```sql > SELECT approx_percentile_cont_with_weight(weight_column, 0.90) WITHIN GROUP (ORDER BY column_name) FROM table_name; +---------------------------------------------------------------------------------------------+ | approx_percentile_cont_with_weight(weight_column, 0.90) WITHIN GROUP (ORDER BY column_name) | +---------------------------------------------------------------------------------------------+ | 78.5 | +---------------------------------------------------------------------------------------------+ > SELECT approx_percentile_cont_with_weight(weight_column, 0.90, 100) WITHIN GROUP (ORDER BY column_name) FROM table_name; +--------------------------------------------------------------------------------------------------+ | approx_percentile_cont_with_weight(weight_column, 0.90, 100) WITHIN GROUP (ORDER BY column_name) | +--------------------------------------------------------------------------------------------------+ | 78.5 | +--------------------------------------------------------------------------------------------------+ ``` An alternative syntax is also supported: ```sql > SELECT approx_percentile_cont_with_weight(column_name, weight_column, 0.90) FROM table_name; +--------------------------------------------------+ | approx_percentile_cont_with_weight(column_name, weight_column, 0.90) | +--------------------------------------------------+ | 78.5 | +--------------------------------------------------+ ``` ### array_agg Returns an array created from the expression elements. If ordering is required, elements are inserted in the specified order. This aggregation function can only mix DISTINCT and ORDER BY if the ordering expression is exactly the same as the argument expression. ``` array_agg(expression [ORDER BY expression]) ``` ```sql > SELECT array_agg(column_name ORDER BY other_column) FROM table_name; +-----------------------------------------------+ | array_agg(column_name ORDER BY other_column) | +-----------------------------------------------+ | [element1, element2, element3] | +-----------------------------------------------+ > SELECT array_agg(DISTINCT column_name ORDER BY column_name) FROM table_name; +--------------------------------------------------------+ | array_agg(DISTINCT column_name ORDER BY column_name) | +--------------------------------------------------------+ | [element1, element2, element3] | +--------------------------------------------------------+ ``` ### avg Returns the average of numeric values in the specified column. ``` avg(expression) ``` ```sql > SELECT avg(column_name) FROM table_name; +---------------------------+ | avg(column_name) | +---------------------------+ | 42.75 | +---------------------------+ ``` ### bool_and Returns true if all non-null input values are true, otherwise false. ``` bool_and(expression) ``` **Arguments** - `expression`: The expression. ```sql > SELECT bool_and(column_name) FROM table_name; +----------------------------+ | bool_and(column_name) | +----------------------------+ | true | +----------------------------+ ``` ### corr Returns the coefficient of correlation between two numeric values. ``` corr(expression1, expression2) ``` **Arguments** - `expression1`: First expression. - `expression2`: Second expression. ```sql > SELECT corr(column1, column2) FROM table_name; +--------------------------------+ | corr(column1, column2) | +--------------------------------+ | 0.85 | +--------------------------------+ ``` ### count Returns the number of non-null values in the specified column. To include null values in the total count, use `count(*)`. ``` count(expression) ``` ```sql > SELECT count(column_name) FROM table_name; +-----------------------+ | count(column_name) | +-----------------------+ | 100 | +-----------------------+ > SELECT count(*) FROM table_name; +------------------+ | count(*) | +------------------+ | 120 | +------------------+ ``` ### covar_samp Returns the sample covariance of a set of number pairs. ``` covar_samp(expression1, expression2) ``` **Arguments** - `expression1`: First expression. - `expression2`: Second expression. ```sql > SELECT covar_samp(column1, column2) FROM table_name; +-----------------------------------+ | covar_samp(column1, column2) | +-----------------------------------+ | 8.25 | +-----------------------------------+ ``` ### first_value Returns the first element in an aggregation group according to the requested ordering. If no ordering is given, returns an arbitrary element from the group. ``` first_value(expression [ORDER BY expression]) ``` ```sql > SELECT first_value(column_name ORDER BY other_column) FROM table_name; +-----------------------------------------------+ | first_value(column_name ORDER BY other_column)| +-----------------------------------------------+ | first_element | +-----------------------------------------------+ ``` ### grouping Returns 1 if the data is aggregated across the specified column, or 0 if it is not aggregated in the result set. ``` grouping(expression) ``` **Arguments** - `expression`: Expression to evaluate whether data is aggregated across the specified column. Can be a constant, column, or function. ```sql > SELECT column_name, GROUPING(column_name) AS group_column FROM table_name GROUP BY GROUPING SETS ((column_name), ()); +-------------+-------------+ | column_name | group_column | +-------------+-------------+ | value1 | 0 | | value2 | 0 | | NULL | 1 | +-------------+-------------+ ``` ### last_value Returns the last element in an aggregation group according to the requested ordering. If no ordering is given, returns an arbitrary element from the group. ``` last_value(expression [ORDER BY expression]) ``` ```sql > SELECT last_value(column_name ORDER BY other_column) FROM table_name; +-----------------------------------------------+ | last_value(column_name ORDER BY other_column) | +-----------------------------------------------+ | last_element | +-----------------------------------------------+ ``` ### max Returns the maximum value in the specified column. ``` max(expression) ``` ```sql > SELECT max(column_name) FROM table_name; +----------------------+ | max(column_name) | +----------------------+ | 150 | +----------------------+ ``` ### median Returns the median value in the specified column. ``` median(expression) ``` **Arguments** - `expression`: The expression. ```sql > SELECT median(column_name) FROM table_name; +----------------------+ | median(column_name) | +----------------------+ | 45.5 | +----------------------+ ``` ### min Returns the minimum value in the specified column. ``` min(expression) ``` ```sql > SELECT min(column_name) FROM table_name; +----------------------+ | min(column_name) | +----------------------+ | 12 | +----------------------+ ``` ### nth_value Returns the nth value in a group of values. ``` nth_value(expression, n ORDER BY expression) ``` **Arguments** - `expression`: The column or expression to retrieve the nth value from. - `n`: The position (nth) of the value to retrieve, based on the ordering. ```sql > SELECT dept_id, salary, NTH_VALUE(salary, 2) OVER (PARTITION BY dept_id ORDER BY salary ASC) AS second_salary_by_dept FROM employee; +---------+--------+-------------------------+ | dept_id | salary | second_salary_by_dept | +---------+--------+-------------------------+ | 1 | 30000 | NULL | | 1 | 40000 | 40000 | | 1 | 50000 | 40000 | | 2 | 35000 | NULL | | 2 | 45000 | 45000 | +---------+--------+-------------------------+ ``` ### percentile_cont Returns the exact percentile of input values, interpolating between values if needed. ``` percentile_cont(percentile) WITHIN GROUP (ORDER BY expression) ``` **Arguments** - `expression`: The expression. - `percentile`: Percentile to compute. Must be a float value between 0 and 1 (inclusive). ```sql > SELECT percentile_cont(0.75) WITHIN GROUP (ORDER BY column_name) FROM table_name; +----------------------------------------------------------+ | percentile_cont(0.75) WITHIN GROUP (ORDER BY column_name) | +----------------------------------------------------------+ | 45.5 | +----------------------------------------------------------+ ``` An alternate syntax is also supported: ```sql > SELECT percentile_cont(column_name, 0.75) FROM table_name; +---------------------------------------+ | percentile_cont(column_name, 0.75) | +---------------------------------------+ | 45.5 | +---------------------------------------+ ``` ### stddev Returns the standard deviation of a set of numbers. ``` stddev(expression) ``` ```sql > SELECT stddev(column_name) FROM table_name; +----------------------+ | stddev(column_name) | +----------------------+ | 12.34 | +----------------------+ ``` ### stddev_pop Returns the population standard deviation of a set of numbers. ``` stddev_pop(expression) ``` ```sql > SELECT stddev_pop(column_name) FROM table_name; +--------------------------+ | stddev_pop(column_name) | +--------------------------+ | 10.56 | +--------------------------+ ``` ### string_agg Concatenates the values of string expressions and places separator values between them. If ordering is required, strings are concatenated in the specified order. This aggregation function can only mix DISTINCT and ORDER BY if the ordering expression is exactly the same as the first argument expression. ``` string_agg([DISTINCT] expression, delimiter [ORDER BY expression]) ``` **Arguments** - `expression`: The string expression to concatenate. Can be a column or any valid string expression. - `delimiter`: A literal string used as a separator between the concatenated values. ```sql > SELECT string_agg(name, ', ') AS names_list FROM employee; +--------------------------+ | names_list | +--------------------------+ | Alice, Bob, Bob, Charlie | +--------------------------+ > SELECT string_agg(name, ', ' ORDER BY name DESC) AS names_list FROM employee; +--------------------------+ | names_list | +--------------------------+ | Charlie, Bob, Bob, Alice | +--------------------------+ > SELECT string_agg(DISTINCT name, ', ' ORDER BY name DESC) AS names_list FROM employee; +--------------------------+ | names_list | +--------------------------+ | Charlie, Bob, Alice | +--------------------------+ ``` ### sum Returns the sum of all values in the specified column. ``` sum(expression) ``` ```sql > SELECT sum(column_name) FROM table_name; +-----------------------+ | sum(column_name) | +-----------------------+ | 12345 | +-----------------------+ ``` ### var Returns the statistical sample variance of a set of numbers. ``` var(expression) ``` **Arguments** - `expression`: Numeric expression. ### var_pop Returns the statistical population variance of a set of numbers. ``` var_pop(expression) ``` **Arguments** - `expression`: Numeric expression. --- # Window functions Source: https://www.hotdata.dev/docs/sql-functions-window Site index: https://www.hotdata.dev/llms.txt Reference for window functions in [HotSQL](/docs/sql). Names, signatures, and examples match the engine exactly. ### cume_dist Relative rank of the current row: (number of rows preceding or peer with the current row) / (total rows). ``` cume_dist() ``` ```sql -- Example usage of the cume_dist window function: SELECT salary, cume_dist() OVER (ORDER BY salary) AS cume_dist FROM employees; +--------+-----------+ | salary | cume_dist | +--------+-----------+ | 30000 | 0.33 | | 50000 | 0.67 | | 70000 | 1.00 | +--------+-----------+ ``` ### dense_rank Rank of the current row within its partition, without gaps. Rows with equal ORDER BY values share a rank and the next rank is consecutive. ``` dense_rank() ``` **Related:** `rank` ### first_value Value of `expression` from the first row of the window frame. ``` first_value(expression) ``` **Arguments** - `expression`: Expression to evaluate. **Related:** `last_value`, `nth_value` ### lag Value of `expression` from a row `offset` positions before the current row in the partition (offset defaults to 1). Returns `default` (or NULL) when out of range. ``` lag(expression [, offset [, default]]) ``` **Arguments** - `expression`: Expression to evaluate on the offset row. - `offset`: Number of rows back. Default 1. - `default`: Value returned when the offset is out of range. **Related:** `lead` ### last_value Value of `expression` from the last row of the window frame. ``` last_value(expression) ``` **Arguments** - `expression`: Expression to evaluate. **Related:** `first_value`, `nth_value` ### lead Value of `expression` from a row `offset` positions after the current row in the partition (offset defaults to 1). Returns `default` (or NULL) when out of range. ``` lead(expression [, offset [, default]]) ``` **Arguments** - `expression`: Expression to evaluate on the offset row. - `offset`: Number of rows forward. Default 1. - `default`: Value returned when the offset is out of range. **Related:** `lag` ### nth_value Value of `expression` from the nth row (1-based) of the window frame. ``` nth_value(expression, n) ``` **Arguments** - `expression`: Expression to evaluate. - `n`: 1-based row position within the frame. **Related:** `first_value`, `last_value` ### ntile Integer ranging from 1 to the argument value, dividing the partition as equally as possible ``` ntile(expression) ``` **Arguments** - `expression`: An integer describing the number groups the partition should be split into ```sql -- Example usage of the ntile window function: SELECT employee_id, salary, ntile(4) OVER (ORDER BY salary DESC) AS quartile FROM employees; +-------------+--------+----------+ | employee_id | salary | quartile | +-------------+--------+----------+ | 1 | 90000 | 1 | | 2 | 85000 | 1 | | 3 | 80000 | 2 | | 4 | 70000 | 2 | | 5 | 60000 | 3 | | 6 | 50000 | 3 | | 7 | 40000 | 4 | | 8 | 30000 | 4 | +-------------+--------+----------+ ``` ### percent_rank Relative rank of the current row: (rank - 1) / (total partition rows - 1). ``` percent_rank() ``` **Related:** `rank`, `cume_dist` ### rank Rank of the current row within its partition, with gaps. Rows with equal ORDER BY values share a rank; the next rank skips ahead. ``` rank() ``` **Related:** `dense_rank`, `percent_rank` ### row_number Number of the current row within its partition, counting from 1. ``` row_number() ``` ```sql -- Example usage of the row_number window function: SELECT department, salary, row_number() OVER (PARTITION BY department ORDER BY salary DESC) AS row_num FROM employees; +-------------+--------+---------+ | department | salary | row_num | +-------------+--------+---------+ | Sales | 70000 | 1 | | Sales | 50000 | 2 | | Sales | 50000 | 3 | | Sales | 30000 | 4 | | Engineering | 90000 | 1 | | Engineering | 80000 | 2 | +-------------+--------+---------+ ``` --- # Array, struct & map functions Source: https://www.hotdata.dev/docs/sql-functions-array Site index: https://www.hotdata.dev/llms.txt Reference for array, struct & map functions in [HotSQL](/docs/sql). Names, signatures, and examples match the engine exactly. ### any_match Returns whether any elements of an array match the given predicate. Returns true if one or more elements match, false if none match (including empty arrays), and null if the predicate returns null for some elements and false for all others. ``` any_match(array, predicate) ``` **Arguments** - `array`: Array expression. Can be a constant, column, or function, and any combination of array operators. - `predicate`: Lambda predicate that returns a boolean ```sql > select any_match([1, 2, 3], x -> x > 2); +----------------------------------+ | any_match([1, 2, 3], x -> x > 2) | +----------------------------------+ | true | +----------------------------------+ ``` ### array_any_value Returns the first non-null element in the array. ``` array_any_value(array) ``` **Arguments** - `array`: Array expression. Can be a constant, column, or function, and any combination of array operators. ```sql > select array_any_value([NULL, 1, 2, 3]); +-------------------------------+ | array_any_value(List([NULL,1,2,3])) | +-------------------------------------+ | 1 | +-------------------------------------+ ``` ### array_append Appends an element to the end of an array. ``` array_append(array, element) ``` **Arguments** - `array`: Array expression. Can be a constant, column, or function, and any combination of array operators. - `element`: Element to append to the array. ```sql > select array_append([1, 2, 3], 4); +--------------------------------------+ | array_append(List([1,2,3]),Int64(4)) | +--------------------------------------+ | [1, 2, 3, 4] | +--------------------------------------+ ``` ### array_compact Removes null values from the array. ``` array_compact(array) ``` **Arguments** - `array`: Array expression. Can be a constant, column, or function, and any combination of array operators. ```sql > select array_compact([1, NULL, 2, NULL, 3]) arr; +-----------+ | arr | +-----------+ | [1, 2, 3] | +-----------+ ``` ### array_concat Concatenates arrays. ``` array_concat(array[, ..., array_n]) ``` **Arguments** - `array`: Array expression. Can be a constant, column, or function, and any combination of array operators. - `array_n`: Subsequent array column or literal array to concatenate. ```sql > select array_concat([1, 2], [3, 4], [5, 6]); +---------------------------------------------------+ | array_concat(List([1,2]),List([3,4]),List([5,6])) | +---------------------------------------------------+ | [1, 2, 3, 4, 5, 6] | +---------------------------------------------------+ ``` ### array_dims Returns an array of the array's dimensions. ``` array_dims(array) ``` **Arguments** - `array`: Array expression. Can be a constant, column, or function, and any combination of array operators. ```sql > select array_dims([[1, 2, 3], [4, 5, 6]]); +---------------------------------+ | array_dims(List([1,2,3,4,5,6])) | +---------------------------------+ | [2, 3] | +---------------------------------+ ``` ### array_distance Returns the Euclidean distance between two input arrays of equal length. ``` array_distance(array1, array2) ``` **Arguments** - `array1`: Array expression. Can be a constant, column, or function, and any combination of array operators. - `array2`: Array expression. Can be a constant, column, or function, and any combination of array operators. ```sql > select array_distance([1, 2], [1, 4]); +------------------------------------+ | array_distance(List([1,2], [1,4])) | +------------------------------------+ | 2.0 | +------------------------------------+ ``` ### array_distinct Returns distinct values from the array after removing duplicates. ``` array_distinct(array) ``` **Arguments** - `array`: Array expression. Can be a constant, column, or function, and any combination of array operators. ```sql > select array_distinct([1, 3, 2, 3, 1, 2, 4]); +---------------------------------+ | array_distinct(List([1,2,3,4])) | +---------------------------------+ | [1, 2, 3, 4] | +---------------------------------+ ``` ### array_element Extracts the element with the index n from the array. ``` array_element(array, index) ``` **Arguments** - `array`: Array expression. Can be a constant, column, or function, and any combination of array operators. - `index`: Index to extract the element from the array. ```sql > select array_element([1, 2, 3, 4], 3); +-----------------------------------------+ | array_element(List([1,2,3,4]),Int64(3)) | +-----------------------------------------+ | 3 | +-----------------------------------------+ ``` ### array_except Returns an array of the elements that appear in the first array but not in the second. ``` array_except(array1, array2) ``` **Arguments** - `array1`: Array expression. Can be a constant, column, or function, and any combination of array operators. - `array2`: Array expression. Can be a constant, column, or function, and any combination of array operators. ```sql > select array_except([1, 2, 3, 4], [5, 6, 3, 4]); +----------------------------------------------------+ | array_except([1, 2, 3, 4], [5, 6, 3, 4]); | +----------------------------------------------------+ | [1, 2] | +----------------------------------------------------+ > select array_except([1, 2, 3, 4], [3, 4, 5, 6]); +----------------------------------------------------+ | array_except([1, 2, 3, 4], [3, 4, 5, 6]); | +----------------------------------------------------+ | [1, 2] | +----------------------------------------------------+ ``` ### array_filter filters the values of an array using a boolean lambda ``` array_filter(array, x -> x > 2) ``` **Arguments** - `array`: Array expression. Can be a constant, column, or function, and any combination of array operators. - `lambda`: Lambda that returns a boolean. Elements for which the lambda returns true are kept. ```sql > select array_filter([1, 2, 3, 4, 5], x -> x > 2); +--------------------------------------------+ | array_filter([1, 2, 3, 4, 5], x -> x > 2) | +--------------------------------------------+ | [3, 4, 5] | +--------------------------------------------+ ``` ### array_has Returns true if the array contains the element. ``` array_has(array, element) ``` **Arguments** - `array`: Array expression. Can be a constant, column, or function, and any combination of array operators. - `element`: Scalar or Array expression. Can be a constant, column, or function, and any combination of array operators. ```sql > select array_has([1, 2, 3], 2); +-----------------------------+ | array_has(List([1,2,3]), 2) | +-----------------------------+ | true | +-----------------------------+ ``` ### array_has_all Returns true if all elements of sub-array exist in array. ``` array_has_all(array, sub-array) ``` **Arguments** - `array`: Array expression. Can be a constant, column, or function, and any combination of array operators. - `sub-array`: Array expression. Can be a constant, column, or function, and any combination of array operators. ```sql > select array_has_all([1, 2, 3, 4], [2, 3]); +--------------------------------------------+ | array_has_all(List([1,2,3,4]), List([2,3])) | +--------------------------------------------+ | true | +--------------------------------------------+ ``` ### array_has_any Returns true if the arrays have any elements in common. ``` array_has_any(array1, array2) ``` **Arguments** - `array1`: Array expression. Can be a constant, column, or function, and any combination of array operators. - `array2`: Array expression. Can be a constant, column, or function, and any combination of array operators. ```sql > select array_has_any([1, 2, 3], [3, 4]); +------------------------------------------+ | array_has_any(List([1,2,3]), List([3,4])) | +------------------------------------------+ | true | +------------------------------------------+ ``` ### array_intersect Returns an array of elements in the intersection of array1 and array2. ``` array_intersect(array1, array2) ``` **Arguments** - `array1`: Array expression. Can be a constant, column, or function, and any combination of array operators. - `array2`: Array expression. Can be a constant, column, or function, and any combination of array operators. ```sql > select array_intersect([1, 2, 3, 4], [5, 6, 3, 4]); +----------------------------------------------------+ | array_intersect([1, 2, 3, 4], [5, 6, 3, 4]); | +----------------------------------------------------+ | [3, 4] | +----------------------------------------------------+ > select array_intersect([1, 2, 3, 4], [5, 6, 7, 8]); +----------------------------------------------------+ | array_intersect([1, 2, 3, 4], [5, 6, 7, 8]); | +----------------------------------------------------+ | [] | +----------------------------------------------------+ ``` ### array_length Returns the length of the array dimension. ``` array_length(array, dimension) ``` **Arguments** - `array`: Array expression. Can be a constant, column, or function, and any combination of array operators. - `dimension`: Array dimension. ```sql > select array_length([1, 2, 3, 4, 5], 1); +-------------------------------------------+ | array_length(List([1,2,3,4,5]), 1) | +-------------------------------------------+ | 5 | +-------------------------------------------+ ``` ### array_max Returns the maximum value in the array. ``` array_max(array) ``` **Arguments** - `array`: Array expression. Can be a constant, column, or function, and any combination of array operators. ```sql > select array_max([3,1,4,2]); +-----------------------------------------+ | array_max(List([3,1,4,2])) | +-----------------------------------------+ | 4 | +-----------------------------------------+ ``` ### array_min Returns the minimum value in the array. ``` array_min(array) ``` **Arguments** - `array`: Array expression. Can be a constant, column, or function, and any combination of array operators. ```sql > select array_min([3,1,4,2]); +-----------------------------------------+ | array_min(List([3,1,4,2])) | +-----------------------------------------+ | 1 | +-----------------------------------------+ ``` ### array_ndims Returns the number of dimensions of the array. ``` array_ndims(array, element) ``` **Arguments** - `array`: Array expression. Can be a constant, column, or function, and any combination of array operators. - `element`: Array element. ```sql > select array_ndims([[1, 2, 3], [4, 5, 6]]); +----------------------------------+ | array_ndims(List([1,2,3,4,5,6])) | +----------------------------------+ | 2 | +----------------------------------+ ``` ### array_normalize Returns the L2-normalized vector for the input numeric array, computed as `array[i] / sqrt(sum(array[i]^2))` per element. Returns NULL if the input is NULL, contains NULL elements, or has zero magnitude (all elements are zero). Returns an empty array for an empty input array. ``` array_normalize(array) ``` **Arguments** - `array`: Array expression. Can be a constant, column, or function, and any combination of array operators. ```sql > select array_normalize([3.0, 4.0]); +-----------------------------+ | array_normalize(List([3.0,4.0])) | +-----------------------------+ | [0.6, 0.8] | +-----------------------------+ ``` ### array_pop_back Returns the array without the last element. ``` array_pop_back(array) ``` **Arguments** - `array`: Array expression. Can be a constant, column, or function, and any combination of array operators. ```sql > select array_pop_back([1, 2, 3]); +-------------------------------+ | array_pop_back(List([1,2,3])) | +-------------------------------+ | [1, 2] | +-------------------------------+ ``` ### array_pop_front Returns the array without the first element. ``` array_pop_front(array) ``` **Arguments** - `array`: Array expression. Can be a constant, column, or function, and any combination of array operators. ```sql > select array_pop_front([1, 2, 3]); +-------------------------------+ | array_pop_front(List([1,2,3])) | +-------------------------------+ | [2, 3] | +-------------------------------+ ``` ### array_position Returns the position of the first occurrence of the specified element in the array, or NULL if not found. Comparisons are done using `IS DISTINCT FROM` semantics, so NULL is considered to match NULL. ``` array_position(array, element) array_position(array, element, index) ``` **Arguments** - `array`: Array expression. Can be a constant, column, or function, and any combination of array operators. - `element`: Element to search for in the array. - `index`: Index at which to start searching (1-indexed). ```sql > select array_position([1, 2, 2, 3, 1, 4], 2); +----------------------------------------------+ | array_position(List([1,2,2,3,1,4]),Int64(2)) | +----------------------------------------------+ | 2 | +----------------------------------------------+ > select array_position([1, 2, 2, 3, 1, 4], 2, 3); +----------------------------------------------------+ | array_position(List([1,2,2,3,1,4]),Int64(2), Int64(3)) | +----------------------------------------------------+ | 3 | +----------------------------------------------------+ ``` ### array_positions Searches for an element in the array, returns all occurrences. ``` array_positions(array, element) ``` **Arguments** - `array`: Array expression. Can be a constant, column, or function, and any combination of array operators. - `element`: Element to search for in the array. ```sql > select array_positions([1, 2, 2, 3, 1, 4], 2); +-----------------------------------------------+ | array_positions(List([1,2,2,3,1,4]),Int64(2)) | +-----------------------------------------------+ | [2, 3] | +-----------------------------------------------+ ``` ### array_prepend Prepends an element to the beginning of an array. ``` array_prepend(element, array) ``` **Arguments** - `array`: Array expression. Can be a constant, column, or function, and any combination of array operators. - `element`: Element to prepend to the array. ```sql > select array_prepend(1, [2, 3, 4]); +---------------------------------------+ | array_prepend(Int64(1),List([2,3,4])) | +---------------------------------------+ | [1, 2, 3, 4] | +---------------------------------------+ ``` ### array_remove Removes the first element from the array equal to the given value. NULL elements already in the array are preserved when removing a non-NULL value. If `element` evaluates to NULL, the result is NULL rather than removing NULL entries. ``` array_remove(array, element) ``` **Arguments** - `array`: Array expression. Can be a constant, column, or function, and any combination of array operators. - `element`: Element to be removed from the array. ```sql > select array_remove([1, 2, 2, 3, 2, 1, 4], 2); +----------------------------------------------+ | array_remove(List([1,2,2,3,2,1,4]),Int64(2)) | +----------------------------------------------+ | [1, 2, 3, 2, 1, 4] | +----------------------------------------------+ > select array_remove([1, 2, NULL, 2, 4], 2); +---------------------------------------------------+ | array_remove(List([1,2,NULL,2,4]),Int64(2)) | +---------------------------------------------------+ | [1, NULL, 2, 4] | +---------------------------------------------------+ ``` ### array_remove_all Removes all elements from the array equal to the given value. NULL elements already in the array are preserved when removing a non-NULL value. If `element` evaluates to NULL, the result is NULL rather than removing NULL entries. ``` array_remove_all(array, element) ``` **Arguments** - `array`: Array expression. Can be a constant, column, or function, and any combination of array operators. - `element`: Element to be removed from the array. ```sql > select array_remove_all([1, 2, 2, 3, 2, 1, 4], 2); +--------------------------------------------------+ | array_remove_all(List([1,2,2,3,2,1,4]),Int64(2)) | +--------------------------------------------------+ | [1, 3, 1, 4] | +--------------------------------------------------+ > select array_remove_all([1, 2, NULL, 2, 4], 2); +-----------------------------------------------------+ | array_remove_all(List([1,2,NULL,2,4]),Int64(2)) | +-----------------------------------------------------+ | [1, NULL, 4] | +-----------------------------------------------------+ ``` ### array_remove_n Removes the first `max` elements from the array equal to the given value. NULL elements already in the array are preserved when removing a non-NULL value. If `element` evaluates to NULL, the result is NULL rather than removing NULL entries. ``` array_remove_n(array, element, max) ``` **Arguments** - `array`: Array expression. Can be a constant, column, or function, and any combination of array operators. - `element`: Element to be removed from the array. - `max`: Number of first occurrences to remove. ```sql > select array_remove_n([1, 2, 2, 3, 2, 1, 4], 2, 2); +---------------------------------------------------------+ | array_remove_n(List([1,2,2,3,2,1,4]),Int64(2),Int64(2)) | +---------------------------------------------------------+ | [1, 3, 2, 1, 4] | +---------------------------------------------------------+ > select array_remove_n([1, 2, NULL, 2, 4], 2, 2); +----------------------------------------------------------+ | array_remove_n(List([1,2,NULL,2,4]),Int64(2),Int64(2)) | +----------------------------------------------------------+ | [1, NULL, 4] | +----------------------------------------------------------+ ``` ### array_repeat Returns an array containing element `count` times. ``` array_repeat(element, count) ``` **Arguments** - `element`: Element expression. Can be a constant, column, or function, and any combination of array operators. - `count`: Value of how many times to repeat the element. ```sql > select array_repeat(1, 3); +---------------------------------+ | array_repeat(Int64(1),Int64(3)) | +---------------------------------+ | [1, 1, 1] | +---------------------------------+ > select array_repeat([1, 2], 2); +------------------------------------+ | array_repeat(List([1,2]),Int64(2)) | +------------------------------------+ | [[1, 2], [1, 2]] | +------------------------------------+ ``` ### array_replace Replaces the first occurrence of the specified element with another specified element. ``` array_replace(array, from, to) ``` **Arguments** - `array`: Array expression. Can be a constant, column, or function, and any combination of array operators. - `from`: Initial element. - `to`: Final element. ```sql > select array_replace([1, 2, 2, 3, 2, 1, 4], 2, 5); +--------------------------------------------------------+ | array_replace(List([1,2,2,3,2,1,4]),Int64(2),Int64(5)) | +--------------------------------------------------------+ | [1, 5, 2, 3, 2, 1, 4] | +--------------------------------------------------------+ ``` ### array_replace_all Replaces all occurrences of the specified element with another specified element. ``` array_replace_all(array, from, to) ``` **Arguments** - `array`: Array expression. Can be a constant, column, or function, and any combination of array operators. - `from`: Initial element. - `to`: Final element. ```sql > select array_replace_all([1, 2, 2, 3, 2, 1, 4], 2, 5); +------------------------------------------------------------+ | array_replace_all(List([1,2,2,3,2,1,4]),Int64(2),Int64(5)) | +------------------------------------------------------------+ | [1, 5, 5, 3, 5, 1, 4] | +------------------------------------------------------------+ ``` ### array_replace_n Replaces the first `max` occurrences of the specified element with another specified element. ``` array_replace_n(array, from, to, max) ``` **Arguments** - `array`: Array expression. Can be a constant, column, or function, and any combination of array operators. - `from`: Initial element. - `to`: Final element. - `max`: Number of first occurrences to replace. ```sql > select array_replace_n([1, 2, 2, 3, 2, 1, 4], 2, 5, 2); +-------------------------------------------------------------------+ | array_replace_n(List([1,2,2,3,2,1,4]),Int64(2),Int64(5),Int64(2)) | +-------------------------------------------------------------------+ | [1, 5, 5, 3, 2, 1, 4] | +-------------------------------------------------------------------+ ``` ### array_resize Resizes the list to contain size elements. Initializes new elements with value or empty if value is not set. ``` array_resize(array, size, value) ``` **Arguments** - `array`: Array expression. Can be a constant, column, or function, and any combination of array operators. - `size`: New size of given array. - `value`: Defines new elements' value or empty if value is not set. ```sql > select array_resize([1, 2, 3], 5, 0); +-------------------------------------+ | array_resize(List([1,2,3],5,0)) | +-------------------------------------+ | [1, 2, 3, 0, 0] | +-------------------------------------+ ``` ### array_reverse Returns the array with the order of the elements reversed. ``` array_reverse(array) ``` **Arguments** - `array`: Array expression. Can be a constant, column, or function, and any combination of array operators. ```sql > select array_reverse([1, 2, 3, 4]); +------------------------------------------------------------+ | array_reverse(List([1, 2, 3, 4])) | +------------------------------------------------------------+ | [4, 3, 2, 1] | +------------------------------------------------------------+ ``` ### array_slice Returns a slice of the array based on 1-indexed start and end positions. ``` array_slice(array, begin, end) ``` **Arguments** - `array`: Array expression. Can be a constant, column, or function, and any combination of array operators. - `begin`: Index of the first element. If negative, it counts backward from the end of the array. - `end`: Index of the last element. If negative, it counts backward from the end of the array. - `stride`: Stride of the array slice. The default is 1. ```sql > select array_slice([1, 2, 3, 4, 5, 6, 7, 8], 3, 6); +--------------------------------------------------------+ | array_slice(List([1,2,3,4,5,6,7,8]),Int64(3),Int64(6)) | +--------------------------------------------------------+ | [3, 4, 5, 6] | +--------------------------------------------------------+ ``` ### array_sort Sort array. ``` array_sort(array, desc, nulls_first) ``` **Arguments** - `array`: Array expression. Can be a constant, column, or function, and any combination of array operators. - `desc`: Whether to sort in ascending (`ASC`) or descending (`DESC`) order. The default is `ASC`. - `nulls_first`: Whether to sort nulls first (`NULLS FIRST`) or last (`NULLS LAST`). The default is `NULLS FIRST`. ```sql > select array_sort([3, 1, 2]); +-----------------------------+ | array_sort(List([3,1,2])) | +-----------------------------+ | [1, 2, 3] | +-----------------------------+ ``` ### array_to_string Converts each element to its text representation. ``` array_to_string(array, delimiter[, null_string]) ``` **Arguments** - `array`: Array expression. Can be a constant, column, or function, and any combination of array operators. - `delimiter`: Array element separator. - `null_string`: Optional. String to use for null values in the output. If not provided, nulls will be omitted. ```sql > select array_to_string([[1, 2, 3, 4], [5, 6, 7, 8]], ','); +----------------------------------------------------+ | array_to_string(List([1,2,3,4,5,6,7,8]),Utf8(",")) | +----------------------------------------------------+ | 1,2,3,4,5,6,7,8 | +----------------------------------------------------+ ``` ### array_transform transforms the values of an array ``` array_transform(array, x -> x*2) ``` **Arguments** - `array`: Array expression. Can be a constant, column, or function, and any combination of array operators. - `lambda`: Lambda ```sql > select array_transform([1, 2, 3, 4, 5], x -> x*2); +-------------------------------------------+ | array_transform([1, 2, 3, 4, 5], x -> x*2) | +-------------------------------------------+ | [2, 4, 6, 8, 10] | +-------------------------------------------+ ``` ### array_union Returns an array of elements that are present in both arrays (all elements from both arrays) without duplicates. ``` array_union(array1, array2) ``` **Arguments** - `array1`: Array expression. Can be a constant, column, or function, and any combination of array operators. - `array2`: Array expression. Can be a constant, column, or function, and any combination of array operators. ```sql > select array_union([1, 2, 3, 4], [5, 6, 3, 4]); +----------------------------------------------------+ | array_union([1, 2, 3, 4], [5, 6, 3, 4]); | +----------------------------------------------------+ | [1, 2, 3, 4, 5, 6] | +----------------------------------------------------+ > select array_union([1, 2, 3, 4], [5, 6, 7, 8]); +----------------------------------------------------+ | array_union([1, 2, 3, 4], [5, 6, 7, 8]); | +----------------------------------------------------+ | [1, 2, 3, 4, 5, 6, 7, 8] | +----------------------------------------------------+ ``` ### arrays_zip Returns an array of structs created by combining the elements of each input array at the same index. If the arrays have different lengths, shorter arrays are padded with NULLs. ``` arrays_zip(array1[, ..., array_n]) ``` **Arguments** - `array1`: First array expression. - `array_n`: Optional additional array expressions. ```sql > select arrays_zip([1, 2, 3]); +---------------------------------------------------+ | arrays_zip([1, 2, 3]) | +---------------------------------------------------+ | [{1: 1}, {1: 2}, {1: 3}] | +---------------------------------------------------+ > select arrays_zip([1, 2], [3, 4, 5]); +---------------------------------------------------+ | arrays_zip([1, 2], [3, 4, 5]) | +---------------------------------------------------+ | [{1: 1, 2: 3}, {1: 2, 2: 4}, {1: NULL, 2: 5}] | +---------------------------------------------------+ ``` ### cardinality Returns the total number of elements in the array. ``` cardinality(array) ``` **Arguments** - `array`: Array expression. Can be a constant, column, or function, and any combination of array operators. ```sql > select cardinality([[1, 2, 3, 4], [5, 6, 7, 8]]); +--------------------------------------+ | cardinality(List([1,2,3,4,5,6,7,8])) | +--------------------------------------+ | 8 | +--------------------------------------+ ``` ### cosine_distance Returns the cosine distance between two input arrays of equal length. The cosine distance is defined as 1 - cosine_similarity, i.e. `1 - dot(a,b) / (||a|| * ||b||)`. Returns NULL if either array is NULL or contains only zeros. ``` cosine_distance(array1, array2) ``` **Arguments** - `array1`: Array expression. Can be a constant, column, or function, and any combination of array operators. - `array2`: Array expression. Can be a constant, column, or function, and any combination of array operators. ```sql > select cosine_distance([1.0, 0.0], [0.0, 1.0]); +-----------------------------------------------+ | cosine_distance(List([1.0,0.0]),List([0.0,1.0])) | +-----------------------------------------------+ | 1.0 | +-----------------------------------------------+ ``` ### empty Returns 1 for an empty array or 0 for a non-empty array. ``` empty(array) ``` **Arguments** - `array`: Array expression. Can be a constant, column, or function, and any combination of array operators. ```sql > select empty([1]); +------------------+ | empty(List([1])) | +------------------+ | 0 | +------------------+ ``` ### flatten Converts an array of arrays to a flat array. - Applies to any depth of nested arrays - Does not change arrays that are already flat The flattened array contains all the elements from all source arrays. ``` flatten(array) ``` **Arguments** - `array`: Array expression. Can be a constant, column, or function, and any combination of array operators. ```sql > select flatten([[1, 2], [3, 4]]); +------------------------------+ | flatten(List([1,2], [3,4])) | +------------------------------+ | [1, 2, 3, 4] | +------------------------------+ ``` ### generate_series Similar to the range function, but it includes the upper bound. ``` generate_series(stop) generate_series(start, stop[, step]) ``` **Arguments** - `start`: Start of the series. Ints, timestamps, dates or string types that can be coerced to Date32 are supported. - `end`: End of the series (included). Type must be the same as start. - `step`: Increase by step (can not be 0). Steps less than a day are supported only for timestamp ranges. ```sql > select generate_series(1,3); +------------------------------------+ | generate_series(Int64(1),Int64(3)) | +------------------------------------+ | [1, 2, 3] | +------------------------------------+ ``` ### inner_product Returns the inner product (dot product) of two input arrays of equal length, computed as `sum(array1[i] * array2[i])`. Returns NULL if either array is NULL or contains NULL elements. Returns 0.0 for two empty arrays. ``` inner_product(array1, array2) ``` **Arguments** - `array1`: Array expression. Can be a constant, column, or function, and any combination of array operators. - `array2`: Array expression. Can be a constant, column, or function, and any combination of array operators. ```sql > select inner_product([1.0, 2.0, 3.0], [4.0, 5.0, 6.0]); +-------------------------------------------------------+ | inner_product(List([1.0,2.0,3.0]),List([4.0,5.0,6.0])) | +-------------------------------------------------------+ | 32.0 | +-------------------------------------------------------+ ``` ### make_array Returns an array using the specified input expressions. ``` make_array(expression1[, ..., expression_n]) ``` **Arguments** - `expression_n`: Expression to include in the output array. Can be a constant, column, or function, and any combination of arithmetic or string operators. ```sql > select make_array(1, 2, 3, 4, 5); +----------------------------------------------------------+ | make_array(Int64(1),Int64(2),Int64(3),Int64(4),Int64(5)) | +----------------------------------------------------------+ | [1, 2, 3, 4, 5] | +----------------------------------------------------------+ ``` ### map Returns an Arrow map with the specified key-value pairs. The `make_map` function creates a map from two lists: one for keys and one for values. Each key must be unique and non-null. ``` map(key, value) map(key: value) make_map(['key1', 'key2'], ['value1', 'value2']) ``` **Arguments** - `key`: For `map`: Expression to be used for key. Can be a constant, column, function, or any combination of arithmetic or string operators. For `make_map`: The list of keys to be used in the map. Each key must be unique and non-null. - `value`: For `map`: Expression to be used for value. Can be a constant, column, function, or any combination of arithmetic or string operators. For `make_map`: The list of values to be mapped to the corresponding keys. ```sql -- Using map function SELECT MAP('type', 'test'); ---- {type: test} SELECT MAP(['POST', 'HEAD', 'PATCH'], [41, 33, null]); ---- {POST: 41, HEAD: 33, PATCH: NULL} SELECT MAP([[1,2], [3,4]], ['a', 'b']); ---- {[1, 2]: a, [3, 4]: b} SELECT MAP { 'a': 1, 'b': 2 }; ---- {a: 1, b: 2} -- Using make_map function SELECT MAKE_MAP(['POST', 'HEAD'], [41, 33]); ---- {POST: 41, HEAD: 33} SELECT MAKE_MAP(['key1', 'key2'], ['value1', null]); ---- {key1: value1, key2: } ``` ### map_entries Returns a list of all entries in the map. ``` map_entries(map) ``` **Arguments** - `map`: Map expression. Can be a constant, column, or function, and any combination of map operators. ```sql SELECT map_entries(MAP {'a': 1, 'b': NULL, 'c': 3}); ---- [{'key': a, 'value': 1}, {'key': b, 'value': NULL}, {'key': c, 'value': 3}] SELECT map_entries(map([100, 5], [42, 43])); ---- [{'key': 100, 'value': 42}, {'key': 5, 'value': 43}] ``` ### map_extract Returns a list containing the value for the given key or an empty list if the key is not present in the map. ``` map_extract(map, key) ``` **Arguments** - `map`: Map expression. Can be a constant, column, or function, and any combination of map operators. - `key`: Key to extract from the map. Can be a constant, column, or function, any combination of arithmetic or string operators, or a named expression of the previously listed. ```sql SELECT map_extract(MAP {'a': 1, 'b': NULL, 'c': 3}, 'a'); ---- [1] SELECT map_extract(MAP {1: 'one', 2: 'two'}, 2); ---- ['two'] SELECT map_extract(MAP {'x': 10, 'y': NULL, 'z': 30}, 'y'); ---- [NULL] -- non-existing key SELECT map_extract(MAP {'x': 10, 'y': NULL, 'z': 30}, 'a'); ---- [] ``` ### map_keys Returns a list of all keys in the map. ``` map_keys(map) ``` **Arguments** - `map`: Map expression. Can be a constant, column, or function, and any combination of map operators. ```sql SELECT map_keys(MAP {'a': 1, 'b': NULL, 'c': 3}); ---- [a, b, c] SELECT map_keys(map([100, 5], [42, 43])); ---- [100, 5] ``` ### map_values Returns a list of all values in the map. ``` map_values(map) ``` **Arguments** - `map`: Map expression. Can be a constant, column, or function, and any combination of map operators. ```sql SELECT map_values(MAP {'a': 1, 'b': NULL, 'c': 3}); ---- [1, , 3] SELECT map_values(map([100, 5], [42, 43])); ---- [42, 43] ``` ### range Returns an Arrow array between start and stop with step. The range start..end contains all values with start <= x < end. It is empty if start >= end. Step cannot be 0. ``` range(stop) range(start, stop[, step]) ``` **Arguments** - `start`: Start of the range. Ints, timestamps, dates or string types that can be coerced to Date32 are supported. - `end`: End of the range (not included). Type must be the same as start. - `step`: Increase by step (cannot be 0). Steps less than a day are supported only for timestamp ranges. ```sql > select range(2, 10, 3); +-----------------------------------+ | range(Int64(2),Int64(10),Int64(3))| +-----------------------------------+ | [2, 5, 8] | +-----------------------------------+ > select range(DATE '1992-09-01', DATE '1993-03-01', INTERVAL '1' MONTH); +--------------------------------------------------------------------------+ | range(DATE '1992-09-01', DATE '1993-03-01', INTERVAL '1' MONTH) | +--------------------------------------------------------------------------+ | [1992-09-01, 1992-10-01, 1992-11-01, 1992-12-01, 1993-01-01, 1993-02-01] | +--------------------------------------------------------------------------+ ``` ### string_to_array Splits a string into an array of substrings based on a delimiter. Any substrings matching the optional `null_str` argument are replaced with NULL. ``` string_to_array(str, delimiter[, null_str]) ``` **Arguments** - `str`: String expression to split. - `delimiter`: Delimiter string to split on. - `null_str`: Substring values to be replaced with `NULL`. ```sql > select string_to_array('abc##def', '##'); +-----------------------------------+ | string_to_array(Utf8('abc##def')) | +-----------------------------------+ | ['abc', 'def'] | +-----------------------------------+ > select string_to_array('abc def', ' ', 'def'); +---------------------------------------------+ | string_to_array(Utf8('abc def'), Utf8(' '), Utf8('def')) | +---------------------------------------------+ | ['abc', NULL] | +---------------------------------------------+ ``` --- # JSON functions Source: https://www.hotdata.dev/docs/sql-functions-json Site index: https://www.hotdata.dev/llms.txt Query JSON held in text columns. A path is one or more segments—string object keys or integer array indices. The `->` / `->>` / `?` operators are shorthands (`json_get` / `json_as_text` / `json_contains`); casting a `->` result, e.g. `(data -> 'age')::int`, rewrites to the typed getter. ### json_as_text Value at a path, rendered as text. Operator: `json ->> key`. ``` json_as_text(json, path...) ``` ### json_contains Whether a value exists at the given path (boolean). Operator: `json ? key`. ``` json_contains(json, path...) ``` ### json_get JSON value at a path. ``` json_get(json, path...) ``` ### json_get_array Value at a path as an array. ``` json_get_array(json, path...) ``` ### json_get_bool Value at a path as a boolean. ``` json_get_bool(json, path...) ``` ### json_get_float Value at a path as a float. ``` json_get_float(json, path...) ``` ### json_get_int Value at a path as an integer. ``` json_get_int(json, path...) ``` ### json_get_json Nested raw JSON string at a path. ``` json_get_json(json, path...) ``` ### json_get_str Value at a path as a string. ``` json_get_str(json, path...) ``` ### json_length Length of the array or object at a path. Alias: `json_len`. ``` json_length(json, path...) ``` ### json_object_keys Keys of the object at a path, as an array. Alias: `json_keys`. ``` json_object_keys(json, path...) ``` --- # Spatial functions Source: https://www.hotdata.dev/docs/sql-functions-spatial Site index: https://www.hotdata.dev/llms.txt Spatial functions use PostGIS-style `ST_` names and operate on planar (Cartesian) geometry—there is no geography type; coordinates are always interpreted as x = easting/longitude. Buffering and overlay operations (`ST_Buffer`, `ST_Union`, `ST_Intersection`, `ST_MakeValid`, ...) and coordinate reprojection (`ST_Transform(geom, 'EPSG:4326', 'EPSG:3857')`) are available. Geodesic measurements in metres use the `_Spheroid` family (`ST_Area_Spheroid`, `ST_Distance_Spheroid`, ...) or `ST_DistanceSphere` on lon/lat data. See the [SQL reference](/docs/sql#geospatial-functions) for worked examples. ### st_3dmakebox Creates a box3d defined by two 3D Point geometries. ``` ST_3DMakeBox(ST_MakePoint(-989502.1875, 528439.5625, 10), ST_MakePoint(-987121.375 ,529933.1875, 10)) ``` ### st_affine Applies a 2D affine transform: x' = a·x + b·y + xoff, y' = d·x + e·y + yoff. Z/M ordinates are dropped. ``` ST_Affine(geom, a, b, d, e, xoff, yoff) ``` ### st_area Returns the area of a polygonal geometry. ``` ST_Area(geom) ``` ### st_area_spheroid Returns the area of a (multi)polygon in square meters on the WGS84 ellipsoid. Coordinates are interpreted as lon/lat degrees. Non-areal geometries return 0. ``` ST_Area_Spheroid(geom) ``` ### st_asbinary Returns the OGC/ISO Well-Known Binary (WKB) representation of the geometry. ``` ST_AsBinary(geometry) ``` ### st_asgeojson Returns the GeoJSON geometry object for the geometry (2D). ``` ST_AsGeoJSON(geom) ``` ### st_ashexwkb Returns the uppercase hex-encoded little-endian WKB of the geometry (2D). ``` ST_AsHEXWKB(geom) ``` ### st_astext Returns the OGC Well-Known Text (WKT) representation of the geometry/geography. ``` ST_AsText(geometry) ``` ### st_azimuth Planar azimuth in radians (clockwise from north, in [0, 2π)) of the segment from the first point to the second. NULL if the points coincide or either is not a point. ``` ST_Azimuth(origin, target) ``` ### st_boundary Returns the topological boundary of a geometry. ``` ST_Boundary(geom) ``` ### st_box2dfromgeohash Return a BOX2D from a GeoHash string. ``` ST_Box2dFromGeoHash(geohash) ``` ### st_buffer Returns a geometry covering all points within the given distance of the input (8 quadrant segments). ``` ST_Buffer(geom, distance) ``` ### st_centroid Computes a point which is the geometric center of mass of a geometry. ``` ST_Centroid(geometry) ``` ### st_collectionextract Extracts sub-geometries of a single dimension into a multi-geometry: 1 = points → MultiPoint, 2 = lines → MultiLineString, 3 = polygons → MultiPolygon. Multi- and collection geometries are flattened. ``` ST_CollectionExtract(geom, type) ``` ### st_contains Returns TRUE if geometry A contains geometry B. A contains B if and only if all points of B lie inside (i.e. in the interior or boundary of) A (or equivalently, no points of B lie in the exterior of A), and the interiors of A and B have at least one point in common. ``` ST_Contains(geomA, geomB) ``` ### st_containsproperly Returns true if every point of the second geometry is in the interior of the first. ``` ST_ContainsProperly(geomA, geomB) ``` ### st_convexhull Computes the convex hull of a geometry. The convex hull is the smallest convex geometry that encloses all geometries in the input. ``` ST_ConvexHull(geometry) ``` ### st_coorddim Return the coordinate dimension of the ST_Geometry value. ``` ST_CoordDim(geometry) ``` ### st_coveredby Returns true if every point in Geometry/Geography A lies inside (i.e. intersects the interior or boundary of) Geometry/Geography B. Equivalently, tests that no point of A lies outside (in the exterior of) B. ``` ST_CoveredBy(geomA, geomB) ``` ### st_covers Returns true if every point in Geometry/Geography B lies inside (i.e. intersects the interior or boundary of) Geometry/Geography A. Equivalently, tests that no point of B lies outside (in the exterior of) A. ``` ST_Covers(geomA, geomB) ``` ### st_crosses Compares two geometry objects and returns true if their intersection "spatially crosses"; that is, the geometries have some, but not all interior points in common. The intersection of the interiors of the geometries must be non-empty and must have dimension less than the maximum dimension of the two input geometries, and the intersection of the two geometries must not equal either geometry. Otherwise, it returns false. The crosses relation is symmetric and irreflexive. ``` ST_Crosses(geomA, geomB) ``` ### st_difference Returns the part of the first geometry that does not intersect the second. ``` ST_Difference(geomA, geomB) ``` ### st_dimension Returns the topological dimension of a geometry: 0 for points, 1 for curves, 2 for surfaces. For a collection, returns the maximum dimension of its members (0 if empty). ST_Dimension(NULL) is NULL. ``` ST_Dimension(geom) ``` ### st_disjoint Returns true if two geometries are disjoint. Geometries are disjoint if they have no point in common. ``` ST_Disjoint(geomA, geomB) ``` ### st_distance For geometry types returns the minimum 2D Cartesian (planar) distance between two geometries, in projected units (spatial ref units). ``` ST_Distance(geomA, geomB) ``` ### st_distance_geos Returns the Cartesian distance between two geometries. ``` ST_Distance_GEOS(geomA, geomB) ``` ### st_distance_spheroid Geodesic (WGS84 ellipsoid) distance in meters between two points given as lon/lat degrees. Returns NULL if either argument is not a point. ``` ST_Distance_Spheroid(pointA, pointB) ``` ### st_distancesphere Great-circle (haversine) distance in meters between two points given as lon/lat degrees. Returns NULL if either argument is not a point. (Alias: `st_distance_sphere`.) ``` ST_DistanceSphere(pointA, pointB) ``` ### st_dump Decomposes a geometry into its atomic components (Point, LineString, Polygon). Multi-geometries and GeometryCollections are split (recursively) into their parts. Atomic geometries are returned unchanged. This includes Polygons, which are returned whole. Returns, per input row, a list of `(path, geom)` structs where `path` is the 1-based navigation path to the component (empty for an atomic input). Use `unnest` if you want to expand into a row per component instead of row per input geom. Empty inputs (recursively, per the same definition as ST_Empty) produce zero components. ``` ST_Dump(geom) ``` ### st_dwithin Returns true if the two geometries are within the given planar (Cartesian) distance of each other. ``` ST_DWithin(geomA, geomB, distance) ``` ### st_dwithin_geos Returns true if the two geometries are within the given Cartesian distance of each other. ``` ST_DWithin_GEOS(geomA, geomB, distance) ``` ### st_dwithin_spheroid Returns true if two points are within the given geodesic (WGS84 ellipsoid) distance in meters of each other. Coordinates are interpreted as lon/lat degrees (x = longitude), matching the other *_Spheroid functions. NULL if either argument is not a point. ``` ST_DWithin_Spheroid(pointA, pointB, distance) ``` ### st_endpoint Returns the last point of a LINESTRING geometry as a POINT. Returns NULL if the input is not a LINESTRING. ``` ST_EndPoint(line_string) ``` ### st_equals Returns true if the given geometries are "topologically equal". Use this for a 'better' answer than '='. Topological equality means that the geometries have the same dimension, and their point-sets occupy the same space. This means that the order of vertices may be different in topologically equal geometries. ``` ST_Equals(geomA, geomB) ``` ### st_exteriorring Returns the exterior ring of a polygon as a LineString. NULL for non-polygons. ``` ST_ExteriorRing(geom) ``` ### st_flipcoordinates Returns a version of the geometry with X and Y axes swapped. Useful for fixing lat/lon vs lon/lat ordering. ``` ST_FlipCoordinates(geom) ``` ### st_force2d Returns the geometry with any Z and M ordinates removed (forced to XY). ``` ST_Force2D(geom) ``` ### st_geohash Computes a GeoHash representation of a geometry. A GeoHash encodes a geographic Point into a text form that is sortable and searchable based on prefixing. A shorter GeoHash is a less precise representation of a point. It can be thought of as a box that contains the point. ``` ST_GeoHash(point) ``` ### st_geometrytype Returns the type of the geometry as a string. Eg: 'LINESTRING', 'POLYGON', 'MULTIPOINT', etc. ``` ST_GeometryType(geometry) ``` ### st_geomfromgeojson Parses a geometry from a GeoJSON geometry object. ``` ST_GeomFromGeoJSON(geojson) ``` ### st_geomfromhexwkb Parses a geometry from a hex-encoded WKB string. ``` ST_GeomFromHEXWKB(hex) ``` ### st_geomfromtext Constructs a geometry object from the OGC Well-Known text representation. (Aliases: `st_geometryfromtext`, `st_wkttosql`.) ``` ST_GeomFromText(text) ``` ### st_geomfromwkb Takes a well-known binary representation of a geometry and a Spatial Reference System ID (SRID) and creates an instance of the appropriate geometry type (Alias: `st_wkbtosql`.) ``` ST_GeomFromWKB(buffer) ``` ### st_hasm Returns true if the geometry has an M coordinate. ST_HasM(NULL) is NULL. ``` ST_HasM(geom) ``` ### st_hasz Returns true if the geometry has a Z coordinate. ST_HasZ(NULL) is NULL. ``` ST_HasZ(geom) ``` ### st_intersection Returns the geometric intersection of two geometries. ``` ST_Intersection(geomA, geomB) ``` ### st_intersects Returns true if two geometries intersect. Geometries intersect if they have any point in common. ``` ST_Intersects(geomA, geomB) ``` ### st_intersects_extent Returns true if the 2D bounding boxes (extents) of the two geometries intersect, including when they merely touch. This is a cheap extent-only test and does not consider the actual geometry shape. Returns NULL if either argument is NULL or has no extent. ``` ST_Intersects_Extent(geomA, geomB) ``` ### st_isclosed Tests if a LineStrings's start and end points are coincident. ``` ST_IsClosed(geom) ``` ### st_isempty Tests if a geometry is topologically empty. Multi-geometries and GeometryCollections where every leaf is empty (e.g. GEOMETRYCOLLECTION(POINT EMPTY, POLYGON EMPTY) are reported empty. ST_IsEmpty(NULL) is NULL. ``` ST_IsEmpty(geom) ``` ### st_isring Returns true if the linestring is closed and simple (a ring). ``` ST_IsRing(geom) ``` ### st_issimple Returns true if the geometry has no anomalous points such as self-intersections. ``` ST_IsSimple(geom) ``` ### st_isvalid Tests if an ST_Geometry value is well-formed and valid in 2D according to the OGC rules ``` ST_IsValid(geomA) ``` ### st_isvalidreason Returns text stating if a geometry is valid, or if invalid a reason why. ``` ST_IsValidReason(geomA) ``` ### st_length Returns the 2D Cartesian length of the geometry if it is a LineString or MultiLineString. For areal geometries 0 is returned; use ST_Perimeter instead. (Alias: `st_length2d`.) ``` ST_Length(geom) ``` ### st_length_spheroid Returns the length of a (multi)linestring in meters on the WGS84 ellipsoid. Coordinates are interpreted as lon/lat degrees. Non-lineal geometries return 0. ``` ST_Length_Spheroid(geom) ``` ### st_lineinterpolatepoint Returns the point at the given fraction (0..1) along a linestring. ``` ST_LineInterpolatePoint(line, fraction) ``` ### st_linelocatepoint Returns the fraction (0..1) along a linestring that is closest to the given point. NULL if the first argument is not a line or the second is not a point. ``` ST_LineLocatePoint(line, point) ``` ### st_linemerge Returns a (set of) LineString(s) formed by sewing together the constituent line work of a MultiLineString. Lines are joined at endpoints where exactly two lines meet; lines are not merged across intersections of three or more lines. When `directed` is true, lines are only merged when their directions agree. Non-linear inputs yield an empty GeometryCollection. This function strips the M dimension. ``` ST_LineMerge(geometry, directed) ``` ### st_linesubstring Returns the portion of a linestring between the start and end fractions (each 0..1). ``` ST_LineSubstring(line, start, end) ``` ### st_m Return the M coordinate of the point, or NULL if not available. Input must be a point. ``` ST_M(geometry) ``` ### st_makebox2d Creates a box2d defined by two Point geometries. This is useful for doing range queries. ``` ST_MakeBox2D(ST_Point(-989502.1875, 528439.5625), ST_Point(-987121.375, 529933.1875)) ``` ### st_makeenvelope Constructs a rectangular polygon from the bounds (xmin, ymin, xmax, ymax). ``` ST_MakeEnvelope(xmin, ymin, xmax, ymax) ``` ### st_makeline Creates a LineString from the concatenated vertices of two geometries (e.g. two points). ``` ST_MakeLine(geomA, geomB) ``` ### st_makepoint Creates a 2D XY or 3D XYZ or 4D XYZM Point geometry. Use ST_MakePointM to make points with XYM coordinates ``` ST_MakePoint(-71.104, 42.315) ``` ### st_makepointm Creates a point with X, Y and M (measure) ordinates. Use ST_MakePoint to make points with XY, XYZ, or XYZM coordinates. ``` ST_MakePointM(-71.104, 42.315, 10) ``` ### st_makepolygon Builds a Polygon whose exterior ring is the given closed LineString. Errors if the shell is not a closed LineString. ``` ST_MakePolygon(linestring) ``` ### st_makevalid Returns a valid version of the (possibly invalid) input geometry. ``` ST_MakeValid(geom) ``` ### st_minimumrotatedrectangle Returns the minimum-area rotated rectangle enclosing the geometry, as a Polygon. NULL when no rectangle can be formed (e.g. a single point). ``` ST_MinimumRotatedRectangle(geom) ``` ### st_mmax Returns the maximum M coordinate of a geometry, or NULL if it has no M ordinate. ST_MMax(NULL) is NULL. ``` ST_MMax(geom) ``` ### st_mmin Returns the minimum M coordinate of a geometry, or NULL if it has no M ordinate. ST_MMin(NULL) is NULL. ``` ST_MMin(geom) ``` ### st_multi Wraps a single Point/LineString/Polygon into the corresponding multi-geometry; multi-geometries and collections are returned unchanged. ``` ST_Multi(geom) ``` ### st_ndims Return the coordinate dimension of the geometry. ``` ST_NDims(geometry) ``` ### st_normalize Returns the geometry in canonical (normalized) form. ``` ST_Normalize(geom) ``` ### st_npoints Return the number of points in a geometry. Works for all geometries. (Alias: `st_numpoints`.) ``` ST_NPoints(geometry) ``` ### st_numgeometries Returns the number of sub-geometries: the member count for multi-geometries and geometry collections, otherwise 1. ST_NumGeometries(NULL) is NULL. (Alias: `st_ngeometries`.) ``` ST_NumGeometries(geom) ``` ### st_numinteriorrings Returns the number of interior rings (holes) of a Polygon. NULL for non-polygon inputs. (Alias: `st_numinteriorring`.) ``` ST_NumInteriorRings(polygon) ``` ### st_orientedenvelope Returns the minimum-area rotated rectangle enclosing a geometry. Note that more than one such rectangle may exist. May return a Point or LineString in the case of degenerate inputs. ``` ST_OrientedEnvelope(geometry) ``` ### st_overlaps Returns TRUE if geometry A and B "spatially overlap". Two geometries overlap if they have the same dimension, their interiors intersect in that dimension. and each has at least one point inside the other (or equivalently, neither one covers the other). The overlaps relation is symmetric and irreflexive. ``` ST_Overlaps(geomA, geomB) ``` ### st_perimeter Returns the planar (Cartesian) perimeter of a geometry: the total boundary length of its polygonal parts, including hole boundaries, in the input's own coordinate units. Non-areal geometries return 0. ``` ST_Perimeter(geom) ``` ### st_perimeter_spheroid Returns the perimeter of a (multi)polygon in meters on the WGS84 ellipsoid. Coordinates are interpreted as lon/lat degrees. Non-areal geometries return 0. ``` ST_Perimeter_Spheroid(geom) ``` ### st_point Returns a Point with the given X and Y coordinate values. ``` ST_Point(-71.104, 42.315) or ST_Point(-71.104, 42.315, 4326) ``` ### st_point2d Constructs a 2D point from X and Y coordinates. ``` ST_Point2D(x, y) ``` ### st_pointfromgeohash Return a point from a GeoHash string. The point represents the center point of the GeoHash. ``` ST_PointFromGeoHash(geohash) ``` ### st_pointm Returns an Point with the given X, Y and M coordinate values, and optionally an SRID number. ``` ST_PointM(-71.104, 42.315, 3.4) or ST_PointM(-71.104, 42.315, 3.4, 4326) ``` ### st_pointonsurface Returns a POINT which is guaranteed to lie in the interior of a surface. ``` ST_PointOnSurface(geometry) ``` ### st_points Collects every vertex of a geometry into a MultiPoint. ``` ST_Points(geom) ``` ### st_pointz Returns an Point with the given X, Y and Z coordinate values, and optionally an SRID number. ``` ST_Point(-71.104, 42.315) or ST_Point(-71.104, 42.315, 4326) ``` ### st_pointzm Returns an Point with the given X, Y, Z and M coordinate values, and optionally an SRID number. ``` ST_Point(-71.104, 42.315) or ST_Point(-71.104, 42.315, 4326) ``` ### st_quadkey Returns the Bing Maps quadkey string for a longitude/latitude at the given zoom level. ``` ST_QuadKey(longitude, latitude, level) ``` ### st_reduceprecision Snaps the geometry's coordinates to the given grid size. ``` ST_ReducePrecision(geom, gridsize) ``` ### st_removerepeatedpoints Removes consecutive duplicate vertices from a geometry (exact equality; the optional tolerance argument is not supported). ``` ST_RemoveRepeatedPoints(geom) ``` ### st_reverse Returns the geometry with the vertex order of each component reversed. ``` ST_Reverse(geom) ``` ### st_rotate Rotates the geometry counter-clockwise about the origin by the given angle in radians. (Alias: `st_rotatez`.) ``` ST_Rotate(geom, radians) ``` ### st_scale Scales the geometry about the origin by the given X and Y factors. ``` ST_Scale(geom, xfactor, yfactor) ``` ### st_shortestline Returns the shortest 2-point line between two geometries (nearest points). ``` ST_ShortestLine(geomA, geomB) ``` ### st_simplify Computes a simplified representation of a geometry using the Douglas-Peucker algorithm. The simplification tolerance is a distance value, in the units of the input SRS. Simplification removes vertices which are within the tolerance distance of the simplified linework. The result may not be valid even if the input is. ``` ST_Simplify(geometry, epsilon) ``` ### st_simplifypreservetopology Computes a simplified representation of a geometry using a variant of the Douglas-Peucker algorithm which limits simplification to ensure the result has the same topology as the input. The simplification tolerance is a distance value, in the units of the input SRS. Simplification removes vertices which are within the tolerance distance of the simplified linework, as long as topology is preserved. The result will be valid and simple if the input is. ``` ST_SimplifyPreserveTopology(geometry, epsilon) ``` ### st_simplifyvw Returns a simplified representation of a geometry using the Visvalingam-Whyatt algorithm. The simplification tolerance is an area value, in the units of the input SRS. Simplification removes vertices which form "corners" with area less than the tolerance. The result may not be valid even if the input is. ``` ST_SimplifyVW(geometry, epsilon) ``` ### st_startpoint Returns the first point of a LINESTRING geometry as a POINT. Returns NULL if the input is not a LINESTRING ``` ST_StartPoint(line_string) ``` ### st_tileenvelope Returns the Web Mercator (EPSG:3857) bounds of an XYZ map tile as a polygon. ``` ST_TileEnvelope(zoom, x, y) ``` ### st_touches Returns TRUE if A and B intersect, but their interiors do not intersect. Equivalently, A and B have at least one point in common, and the common points lie in at least one boundary. For Point/Point inputs the relationship is always FALSE, since points do not have a boundary. ``` ST_Touches(geomA, geomB) ``` ### st_transform Reprojects a geometry between coordinate reference systems, e.g. 'EPSG:4326' -> 'EPSG:3857'. Coordinates are always interpreted as x = easting/longitude (always-XY); the optional 4th argument is accepted for dialect compatibility but must be true. ``` ST_Transform(geom, source_crs, target_crs [, always_xy]) ``` ### st_translate Translates the geometry by the given X and Y offsets. ``` ST_Translate(geom, dx, dy) ``` ### st_transscale Translates the geometry by (dx, dy) then scales it by (xfactor, yfactor). ``` ST_TransScale(geom, dx, dy, xfactor, yfactor) ``` ### st_union Returns the geometric union of two geometries. ``` ST_Union(geomA, geomB) ``` ### st_within Returns TRUE if geometry A is within geometry B. A is within B if and only if all points of A lie inside (i.e. in the interior or boundary of) B (or equivalently, no points of A lie in the exterior of B), and the interiors of A and B have at least one point in common. ``` ST_Within(geomA, geomB) ``` ### st_x Return the X coordinate of the point, or NULL if not available. Input must be a point. ``` ST_X(geometry) ``` ### st_xmax Returns X maxima of a bounding box 2d or 3d or a geometry ``` ST_XMax(geometry) ``` ### st_xmin Returns X minima of a bounding box 2d or 3d or a geometry ``` ST_XMin(geometry) ``` ### st_y Return the Y coordinate of the point, or NULL if not available. Input must be a point. ``` ST_Y(geometry) ``` ### st_ymax Returns Y maxima of a bounding box 2d or 3d or a geometry ``` ST_YMax(geometry) ``` ### st_ymin Returns Y minima of a bounding box 2d or 3d or a geometry ``` ST_YMin(geometry) ``` ### st_z Return the Z coordinate of the point, or NULL if not available. Input must be a point. ``` ST_Z(geometry) ``` ### st_zmax Returns Z maxima of a bounding box 2d or 3d or a geometry ``` ST_ZMax(geometry) ``` ### st_zmflag Returns a flag describing the geometry's dimensionality: 0 = XY, 1 = XYM, 2 = XYZ, 3 = XYZM. ST_ZMFlag(NULL) is NULL. ``` ST_ZMFlag(geom) ``` ### st_zmin Returns the Z minima of a 2D or 3D bounding box or a geometry ``` ST_ZMin(geometry) ``` --- # MCP Reference Source: https://www.hotdata.dev/docs/mcp Site index: https://www.hotdata.dev/llms.txt # Hotdata MCP Model Context Protocol server – expose workspace tools to AI agents. **Spec:** [Model Context Protocol](https://modelcontextprotocol.io/) ## Endpoints - **GET** [https://mcp.hotdata.dev/mcp/sse](https://mcp.hotdata.dev/mcp/sse) – SSE endpoint - **POST** [https://mcp.hotdata.dev/mcp/message](https://mcp.hotdata.dev/mcp/message) – Message endpoint ## Authentication All MCP requests require: ``` Authorization: Bearer X-Workspace-Id: ``` ## Available tools The server exposes the tools below to AI agents. Each tool is described with its parameters as reported by the server's `tools/list`. ## Query execution Run SQL and check query status. --- ### run_query Execute a SQL query and get results. Returns `result_id`, `columns`, `rows`, and `execution_time_ms`. Reuse the returned `result_id` with the query-result tools to inspect, sample, or export the same result set without re-running. **Parameters:** | Param | Type | Required | Description | |-------|------|----------|-------------| | `sql` | string | Yes | SQL to execute (Postgres-compatible). | | `connection_id` | string | No | Connection public ID to run the query against. Optional when the workspace has a single connection. | --- ### get_query_status Get the status of a query result by id (e.g. `pending`, `ready`, `failed`). **Parameters:** | Param | Type | Required | Description | |-------|------|----------|-------------| | `result_id` | string | Yes | Query result id from `run_query`. | ## Query results Inspect and export result sets by `result_id` (from `run_query`). --- ### list_query_results List recent query result ids in the workspace. Use returned ids with `get_query_result_schema`, `get_query_result_sample`, or `export_query_results`. **Parameters:** | Param | Type | Required | Description | |-------|------|----------|-------------| | `limit` | integer | No | Maximum number of results to return. Default `10`. | | `offset` | integer | No | Offset for pagination. Default `0`. | --- ### get_query_result_schema Get column names and types for a result set by id. **Parameters:** | Param | Type | Required | Description | |-------|------|----------|-------------| | `result_id` | string | Yes | Query result id from `run_query`. | --- ### get_query_result_sample Get the first N rows of a result set by id. Use `limit`/`offset` for pagination. **Parameters:** | Param | Type | Required | Description | |-------|------|----------|-------------| | `result_id` | string | Yes | Query result id from `run_query`. | | `limit` | integer | No | Maximum number of rows to return. Default `10`. | | `offset` | integer | No | Row offset for pagination. Default `0`. | --- ### export_query_results Export a result set by id to CSV or JSON. **Parameters:** | Param | Type | Required | Description | |-------|------|----------|-------------| | `result_id` | string | Yes | Query result id from `run_query`. | | `format` | string | No | `csv` or `json`. Default `csv`. | ## Data sources & tables List and inspect connections and their tables. --- ### list_data_sources List data sources (connections) in the current workspace. Returns names; use `connection_id` from context when calling `list_tables`. --- ### list_tables List tables for a connection. `connection_id` is the connection's `public_id` (e.g. from `list_data_sources`). **Parameters:** - `connection_id` (string, required) — Connection public ID. --- ### describe_table Get table description: column metadata (name, type, nullable, default). `schema` is usually `public`. **Parameters:** - `connection_id` (string, required) — Connection public ID. - `schema` (string, required) — Schema name. - `table` (string, required) — Table name. --- ### get_tables_summary List tables for a connection; optionally include row count and size for each (up to `table_limit`). **Parameters:** | Param | Type | Required | Description | |-------|------|----------|-------------| | `connection_id` | string | Yes | Connection public ID. | | `include_row_count` | boolean | No | Include row count per table. Default `false`. | | `include_size` | boolean | No | Include size per table. Default `false`. | | `table_limit` | integer | No | Maximum number of tables to summarize. Default `20`. | --- ### get_table_row_count Get row count for a table. `approximate=true` uses an estimate when available (faster). **Parameters:** - `connection_id` (string, required) — Connection public ID. - `schema` (string, required) — Schema name. - `table` (string, required) — Table name. - `approximate` (boolean, optional) — Use an estimate when available. Default `true`. --- ### get_table_size Get table size (bytes/GB). Dialect-dependent (e.g. Postgres). **Parameters:** - `connection_id` (string, required) — Connection public ID. - `schema` (string, required) — Schema name. - `table` (string, required) — Table name. --- ### get_column_stats Get column statistics for a table: cardinality, nulls, min, max, and sample values. Use to guide filtering and joins. **Parameters:** - `connection_id` (string, required) — Connection public ID. - `schema` (string, required) — Schema name. - `table` (string, required) — Table name. --- ### list_datasets List datasets in the current workspace. **Parameters:** none ## Connections Inspect and manage existing connections. --- ### get_connection Get connection details by id: name, source_type, table_count, discovery_status. **Parameters:** - `connection_id` (string, required) — Connection public ID. --- ### delete_connection Delete a connection by id. **Parameters:** - `connection_id` (string, required) — Connection public ID. --- ### run_discovery Re-run schema discovery for a connection by id. Use after the source schema changes. **Parameters:** - `connection_id` (string, required) — Connection public ID. --- ### purge_connection_cache Purge cached schema and table data for a connection by id. Use to force a fresh read after upstream changes. **Parameters:** - `connection_id` (string, required) — Connection public ID. ## Metrics & queries Inspect query metrics, latency, storage, and saved queries. --- ### explain_workspace_metrics Explain current workspace metrics (connection count, total queries, p50/p99 latency). Includes trend if available. --- ### get_query_latency_trend Get query latency trend for the last 24 hours. Returns p99 min/max/latest. --- ### get_query_storage Get visibility into query result storage: object count and total bytes for the workspace. --- ### get_query_count Get total query count for the workspace. --- ### list_query_errors List recent query errors in the workspace (time, SQL preview, status_code). **Parameters:** - `limit` (integer, optional) — Maximum number of errors to return. Default `10`. --- ### list_recent_queries List recent queries in the workspace (time, SQL preview, latency_ms, execution_time_ms). **Parameters:** - `limit` (integer, optional) — Maximum number of queries to return. Default `10`. --- ### get_slow_queries List slowest queries in the workspace by latency. **Parameters:** - `limit` (integer, optional) — Maximum number of queries to return. Default `5`. --- ### list_saved_queries List saved (pinned) queries in the workspace. **Parameters:** - `limit` (integer, optional) — Maximum number of saved queries to return. Default `10`. ## Improvements summary The current MCP tool set is read- and analysis-oriented. To support full analyst and setup flows, the following additions would align it with the platform and API: | Area | Addition | Why | |------|----------|-----| | **Workspaces** | list_workspaces, create_workspace, get_workspace | Platform is workspace-centric; agents need to create task-scoped workspaces and check provision status. | | **Connections** | create_connection | Lifecycle is read-only today (get, delete, run_discovery, purge_connection_cache); agents cannot attach new data sources. | | **Managed databases** | create_database, load_managed_table, create_upload | Loading parquet into a managed database requires an upload id and a table load call; without these, file-to-table flows are unavailable over MCP. | | **Unified schema** | list_all_tables or get_workspace_schema (optional) | Single view of all connection tables + managed databases helps with writing JOINs across sources. | --- # Python SDK Source: https://www.hotdata.dev/docs/python-sdk Site index: https://www.hotdata.dev/llms.txt Official Python client for the Hotdata HTTP API — typed, Pydantic-validated, and generated from the OpenAPI spec. ## Install ```bash pip install hotdata ``` For Apache Arrow result support (faster, more memory-efficient for large result sets): ```bash pip install 'hotdata[arrow]' ``` ## Authentication ```python import hotdata configuration = hotdata.Configuration( api_key="YOUR_API_KEY", workspace_id="YOUR_WORKSPACE_ID", ) ``` `host` defaults to `https://api.hotdata.dev`. Override it if you target another environment. ## Quickstart ```python import hotdata configuration = hotdata.Configuration( api_key="YOUR_API_KEY", workspace_id="YOUR_WORKSPACE_ID", ) with hotdata.ApiClient(configuration) as api_client: query_api = hotdata.QueryApi(api_client) response = query_api.query( hotdata.QueryRequest(sql="SELECT 1 AS ok") ) print(response) ``` ## Execute SQL ```python with hotdata.ApiClient(configuration) as api_client: query_api = hotdata.QueryApi(api_client) # Synchronous query response = query_api.query( hotdata.QueryRequest(sql="SELECT * FROM orders LIMIT 10") ) # Async query — returns a query run ID for polling response = query_api.query( hotdata.QueryRequest( sql="SELECT * FROM large_table", var_async=True, ) ) run_id = response.query_run_id # Try sync first, fall back to async after 3 s response = query_api.query( hotdata.QueryRequest( sql="SELECT * FROM orders", var_async=True, async_after_ms=3000, ) ) ``` Scope a query to a specific managed database with `x_database_id`: ```python response = query_api.query( hotdata.QueryRequest(sql="SELECT * FROM default.public.orders LIMIT 5"), x_database_id="db_abc123", ) ``` ## Managed databases ```python with hotdata.ApiClient(configuration) as api_client: db_api = hotdata.DatabasesApi(api_client) # Create a database and declare tables created = db_api.create_database( hotdata.CreateDatabaseRequest( name="sales", expires_at="24h", schemas=[ hotdata.DatabaseDefaultSchemaDecl( name="public", tables=[ hotdata.DatabaseDefaultTableDecl(name="orders"), hotdata.DatabaseDefaultTableDecl(name="customers"), ], ) ], ) ) print(created.id) # e.g. "db_abc123" # List all databases listing = db_api.list_databases() for db in listing.databases: print(db.id, db.name) # Get a specific database detail = db_api.get_database("db_abc123") print(detail.default_connection_id) # Delete a database db_api.delete_database("db_abc123") ``` ## Load parquet into a managed table Upload a parquet file and load it into a declared table: ```python with hotdata.ApiClient(configuration) as api_client: uploads_api = hotdata.UploadsApi(api_client) connections_api = hotdata.ConnectionsApi(api_client) # Upload the file directly to storage: the SDK opens an upload session, # PUTs the bytes, and finalizes it in one call. upload = uploads_api.upload_file( "orders.parquet", content_type="application/parquet", ) # Load into the declared table # (the generated client names the schema parameter var_schema) result = connections_api.load_managed_table( connection_id=detail.default_connection_id, var_schema="public", table="orders", load_managed_table_request=hotdata.LoadManagedTableRequest( mode="replace", upload_id=upload.upload_id, ), ) print(result.row_count, result.table_name) ``` ## Apache Arrow results Fetch results as an Arrow table instead of JSON — faster and more memory-efficient for large result sets: ```python from hotdata import ApiClient, Configuration from hotdata.arrow import ResultsApi with ApiClient(configuration) as client: results = ResultsApi(client) # Buffered — returns a pyarrow.Table table = results.get_result_arrow(result_id, x_database_id=db_id) # Streaming — yields batches without materializing the full table with results.stream_result_arrow(result_id, x_database_id=db_id) as reader: for batch in reader: print(batch.to_pandas()) ``` Both methods take `x_database_id` — the database the result belongs to — and accept `offset` and `limit` for pagination. They raise `hotdata.arrow.ResultNotReadyError` if the result is still pending — poll `results.get_result(result_id, x_database_id=db_id)` until `status == "ready"` first. ## Workspaces ```python with hotdata.ApiClient(configuration) as api_client: workspaces_api = hotdata.WorkspacesApi(api_client) listing = workspaces_api.list_workspaces() for ws in listing.workspaces: print(ws.public_id, ws.name) ``` ## Query run history ```python with hotdata.ApiClient(configuration) as api_client: runs_api = hotdata.QueryRunsApi(api_client) results_api = hotdata.ResultsApi(api_client) # List recent runs — scoped to a database (X-Database-Id is required) listing = runs_api.list_query_runs(x_database_id=db_id, limit=20) for run in listing.query_runs: print(run.id, run.status, run.execution_time_ms) # Fetch stored result rows — results are scoped to the database # they were queried in result = results_api.get_result(run.result_id, x_database_id=db_id) ``` ## Error handling ```python from hotdata.rest import ApiException try: response = query_api.query(hotdata.QueryRequest(sql="SELECT * FROM missing_table")) except ApiException as e: print(f"API error {e.status}: {e.reason}") print(e.body) ``` ## API classes | Class | Description | |-------|-------------| | `QueryApi` | Execute SQL queries | | `DatabasesApi` | Create, list, and delete managed databases | | `ConnectionsApi` | Manage connections and load managed tables | | `WorkspacesApi` | List and create workspaces | | `InformationSchemaApi` | List tables and columns | | `QueryRunsApi` | Inspect query run history | | `ResultsApi` | Retrieve stored query results | | `UploadsApi` | Upload files for managed table loads | | `IndexesApi` | Create and list indexes (BM25, vector) | | `JobsApi` | Monitor background jobs | ## See also - [hotdata on PyPI](https://pypi.org/project/hotdata) - [sdk-python on GitHub](https://github.com/hotdata-dev/sdk-python) - [Rust SDK](/docs/rust-sdk) — the same API from Rust - [API Reference](/docs/api-reference) — Full HTTP API documentation - [Quick Start](/docs/quick-start) — CLI and workspace setup --- # Rust SDK Source: https://www.hotdata.dev/docs/rust-sdk Site index: https://www.hotdata.dev/llms.txt Official Rust client for the Hotdata HTTP API — async and strongly typed. ## Install The crate needs Rust 1.74+ and a [Tokio](https://tokio.rs/) runtime — every call is `async`. ```toml [dependencies] hotdata = "0.14" tokio = { version = "1", features = ["macros", "rt-multi-thread"] } ``` Apache Arrow result support (faster and more memory-efficient for large result sets) is behind an optional feature: ```toml [dependencies] hotdata = { version = "0.14", features = ["arrow"] } ``` The crate builds against `native-tls` by default. To use `rustls` instead: ```toml [dependencies] hotdata = { version = "0.14", default-features = false, features = ["rustls"] } ``` ## Authentication An API token (prefixed `hd_`) is the only credential. It is sent verbatim as `Authorization: Bearer ` on every request — there is nothing to exchange, refresh, or cache — alongside `X-Workspace-Id` on workspace-scoped calls. ```rust use hotdata::prelude::*; let client = Client::builder() .api_token("hd_your_api_token") .workspace_id("your_workspace_id") .build()?; ``` Both are optional on the builder: when omitted they fall back to the `HOTDATA_API_KEY` and `HOTDATA_WORKSPACE_ID` environment variables, and `build()` returns `ClientError::MissingApiToken` / `ClientError::MissingWorkspaceId` if neither is set. `base_url` defaults to `https://api.hotdata.dev` (or `HOTDATA_API_URL`). Override it with `.base_url(..)` to target another environment. ## Quickstart ```rust use hotdata::prelude::*; #[tokio::main] async fn main() -> Result<(), Box> { let client = Client::builder() .api_token("hd_your_api_token") .workspace_id("your_workspace_id") .build()?; // Queries, results, and query runs are scoped to a database via the // required X-Database-Id header, so pick one first. let database_id = "db_abc123"; let response = client .query_in(QueryRequest::new("SELECT 1 AS ok".to_string()), database_id) .await?; println!("{:?} {:?}", response.columns, response.rows); Ok(()) } ``` ## Managed databases ```rust use hotdata::prelude::*; // Create a database and declare schemas and tables on its default catalog let created = client .databases() .create(CreateDatabaseRequest { name: field::set("sales"), expires_at: field::set("24h"), schemas: Some(vec![DatabaseDefaultSchemaDecl { name: "public".to_string(), tables: Some(vec![ DatabaseDefaultTableDecl::new("orders".to_string()), DatabaseDefaultTableDecl::new("customers".to_string()), ]), }]), ..CreateDatabaseRequest::new() }) .await?; println!("{} {}", created.id, created.default_connection_id); // List databases, newest first — (limit, cursor, search, batch) let listing = client.databases().list(Some(20), None, None, None).await?; for db in &listing.databases { println!("{} {:?}", db.id, db.name); } // Fetch one, then delete it let detail = client.databases().get(&created.id).await?; println!("{}", detail.default_connection_id); client.databases().delete(&created.id).await?; ``` Tables declared at create time live on the database's auto-provisioned default catalog, addressed by `created.default_connection_id`. To add a schema or table to a database that already exists, reach for the generated operations: ```rust use hotdata::apis::connections_api; use hotdata::models; let config = client.configuration(); let connection_id = &created.default_connection_id; connections_api::add_managed_schema( config, connection_id, models::AddManagedSchemaRequest::new("staging".to_string()), ) .await?; connections_api::add_managed_table( config, connection_id, "staging", models::AddManagedTableRequest::new("orders_raw".to_string()), ) .await?; ``` ## Load a file into a managed table `upload_file` runs the whole presigned direct-to-storage flow — open a session, `PUT` the bytes straight to storage, finalize — and hands back an `upload_id` to load from. ```rust use hotdata::prelude::*; let upload = client .upload_file( "orders.parquet", UploadOptions { content_type: Some("application/parquet".to_string()), ..UploadOptions::default() }, ) .await?; let loaded = client .connections() .load_managed_table( &created.default_connection_id, "public", "orders", LoadManagedTableRequest { upload_id: field::set(upload.upload_id.clone()), ..LoadManagedTableRequest::new("replace".to_string()) }, ) .await?; println!("{} rows into {}", loaded.row_count, loaded.table_name); ``` `mode` is `"replace"` to overwrite the table's contents or `"append"` to add to them; `"delete"`, `"update"`, and `"upsert"` match rows by the table's key columns. A large load should set `r#async: Some(true)` and poll the returned job with `client.jobs()`. ## Send rows inline For a small table, skip the upload entirely and put CSV text — header row included, up to 2 MiB — in the request: ```rust let loaded = client .connections() .load_managed_table( &created.default_connection_id, "public", "customers", LoadManagedTableRequest { data: field::set("id,name\n1,Ada\n2,Grace\n"), idempotency_key: field::set("customers-batch-1"), ..LoadManagedTableRequest::new("append".to_string()) }, ) .await?; ``` Column types are detected from the data unless `columns` declares them. `idempotency_key` — valid only with inline `data` — makes the load safe to retry: send the same key again and the rows land at most once. ## Execute SQL ```rust use hotdata::prelude::*; // Scoped to a database, with 429 retry and truncated-result auto-follow let response = client .query_in( QueryRequest::new("SELECT * FROM orders LIMIT 10".to_string()), database_id, ) .await?; // Just the bounded inline preview — no auto-follow of a truncated result let preview = client .query_preview(QueryRequest::new("SELECT * FROM large_table".to_string())) .await?; if preview.truncated { println!("preview only; page the full set via result_id"); } // Asynchronous submission — returns an acknowledgement to poll let outcome = client .submit_query( QueryRequest { r#async: Some(true), async_after_ms: field::set(3000), ..QueryRequest::new("SELECT * FROM large_table".to_string()) }, Some(database_id), ) .await?; // QueryOutcome is #[non_exhaustive], so the match needs a wildcard arm match outcome { QueryOutcome::Inline(response) => println!("{} rows inline", response.row_count), QueryOutcome::Submitted(ack) => println!("poll query run {}", ack.query_run_id), _ => println!("unrecognized outcome"), } ``` `query_in` transparently retries HTTP 429 (`OVERLOADED`) and, when the server truncates a large result, pages the full row set into `response.rows` — bounded by the client's `QueryConfig` (1M rows / 64 MiB by default), so a runaway result is an error rather than an OOM. Clone the config to tune it per call: ```rust let config = client.query_config().clone().with_auto_follow(false); let response = client .query_with( QueryRequest::new("SELECT * FROM orders".to_string()), Some(database_id), &config, ) .await?; ``` ## Persisted results A query returns rows inline *and* a `result_id` that persists asynchronously. `await_result` polls it to `ready` without a hand-rolled loop: ```rust use hotdata::prelude::*; // result_id is Option>: absent, or explicitly null when the // result could not be persisted (see response.warning). if let Some(result_id) = response.result_id.flatten() { let ready = client .await_result(&result_id, database_id, PollConfig::default()) .await?; if ready.result_status().is_ready() { println!("{:?}", ready.rows); } } ``` `PollConfig::default()` is a 120-second timeout polled every second. `result_status()` and `run_status()` read the wire's plain-string statuses as the typed `ResultStatus` / `QueryRunStatus` enums, each carrying an `Other(String)` variant so a status added later round-trips instead of breaking deserialization. ## Apache Arrow results With the `arrow` feature on, fetch a result as an Arrow IPC stream instead of JSON: ```rust use hotdata::prelude::*; // Buffered — decodes every batch into a Vec let arrow = client .get_result_arrow(&result_id, database_id, None, None) .await?; println!("{:?} / {:?} rows", arrow.schema, arrow.total_row_count); for batch in &arrow.batches { println!("{} rows", batch.num_rows()); } // Streaming — yields batches lazily, without holding them all at once let stream = client .stream_result_arrow(&result_id, database_id, None, None) .await?; for batch in stream { let batch = batch?; println!("{} rows", batch.num_rows()); } ``` Both take `offset` and `limit` for pagination and return `ArrowError::NotReady` while the result is still pending — poll `client.get_result(&result_id, database_id)` until its status is `ready` first, or use `await_result`. To run a query and decode its result as Arrow in a single call — submit, await `ready`, decode: ```rust let arrow = client .query_to_arrow( QueryRequest::new("SELECT * FROM big_table".to_string()), database_id, PollConfig::default(), None, // offset None, // limit ) .await?; ``` ## Upload options `upload_file` picks its strategy from the file's size: a single `PUT` for a small file, and for a large one a multipart upload that mints each part URL just before uploading that part, so a presigned URL cannot expire mid-transfer. `UploadOptions` tunes the rest, and every field is optional: ```rust use std::sync::Arc; use hotdata::prelude::*; let upload = client .upload_file( "events.csv.gz", UploadOptions { content_type: Some("text/csv".to_string()), content_encoding: Some("gzip".to_string()), filename: Some("events.csv.gz".to_string()), part_size: Some(16 * 1024 * 1024), max_concurrency: Some(8), progress: Some(Arc::new(|done, total| { println!("{done}/{total} bytes"); })), }, ) .await?; ``` `content_type`, `content_encoding`, and `filename` are advisory metadata recorded with the upload. `part_size` is a hint the server clamps to its own range and ignores for single-`PUT` uploads; left unset, the SDK scales one itself — 8 MiB, larger only for very large files, to keep the part count bounded. `max_concurrency` caps in-flight part `PUT`s, and the effective count is the lower of it and a peak-memory budget derived from the server's actual part size, so memory stays bounded whatever you pass. A multi-gigabyte upload legitimately takes minutes, and storage `PUT`s reuse the configured reqwest client, so supply one with no request timeout via `ClientBuilder::reqwest_client` when uploading large files. ## Error handling Every error type in the SDK implements `std::error::Error`, so `{err}` and the `source()` chain are always meaningful. The SDK's own enums are `#[non_exhaustive]` — match them with a wildcard arm. ```rust use hotdata::prelude::*; use hotdata::Error; match client .query_in( QueryRequest::new("SELECT * FROM missing_table".to_string()), database_id, ) .await { Ok(response) => println!("{} rows", response.row_count), Err(QueryError::Overloaded { attempts, .. }) => { println!("server shedding load; gave up after {attempts} attempt(s)") } Err(QueryError::Submit(Error::ResponseError(response))) => { println!("API error {}: {}", response.status, response.content) } Err(other) => println!("query failed: {other}"), } ``` | Type | Returned by | |------|-------------| | `Error` | Every generated operation — `Reqwest`, `Serde`, `Io`, or `ResponseError` carrying the status, body, and typed error entity | | `ClientError` | `ClientBuilder::build` — a missing API token or workspace id | | `QueryError` | The `query` family — `Overloaded`, `Submit`, `AsyncRequested`, `Async`, `Poll`, and `Result(ResultError)` | | `ResultError` | Result-lifecycle failures during auto-follow — `Failed`, `Timeout`, `TooLarge`, `Incomplete`, `Unavailable` | | `AwaitResultError` | `await_result` — `Api`, `Failed`, `Timeout` | | `UploadError` | `upload_file` — `Io`, `CreateSession`, `Storage`, `StorageStatus`, `MissingETag`, `MalformedSession`, `SizeOverflow`, `Finalize`, `MintParts` | | `ArrowError` | The Arrow fetches — `NotReady`, `Failed`, `NotFound`, `InvalidParams`, `Http` | | `QueryToArrowError` | `query_to_arrow` — `Query`, `NoResultId`, `Timeout`, `Arrow` | ## Resource handles The generator emits free functions; the client groups them into workspace-scoped handles so you never pass a `Configuration` around. ```rust let connections = client.connections().list().await?; let secrets = client.secrets().list().await?; // Query runs are database-scoped — (database_id, limit, cursor, status, saved_query_id) let runs = client .query_runs() .list(database_id, Some(50), None, None, None) .await?; ``` | Handle | Description | |--------|-------------| | `queries()` | Execute SQL — plus the `query`, `query_in`, `query_preview`, `query_with`, and `submit_query` shortcuts on `Client` | | `databases()` | Create, fork, list, count, and delete managed databases; attach catalogs | | `connections()` | Inspect connections and load managed tables | | `database_context()` | Read and write a database's stored context | | `information_schema()` | List tables and columns | | `query_runs()` | Inspect query run history | | `results()` | Retrieve stored query results | | `uploads()` | Upload files for managed table loads | | `saved_queries()` | Create, execute, and version saved queries | | `indexes()` | Create and list indexes (BM25, vector) | | `embedding_providers()` | Manage embedding providers | | `secrets()` | Manage workspace secrets | | `jobs()` | Monitor background jobs | | `refresh()` | Refresh source tables | | `workspaces()` | List, create, and delete workspaces | | `connection_types()` | Describe the available connector types | Anything not yet wrapped is one call away through the full generated surface: ```rust use hotdata::apis::workspaces_api; let workspaces = workspaces_api::list_workspaces(client.configuration(), None).await?; ``` Request and response types live under `hotdata::models`, and `hotdata::prelude` re-exports the client, the handles, and every model. Several requests model a field that is both optional and nullable as `Option>`; the `field` helpers name the three intents — `field::set(value)` to set it, `field::clear()` to send `null`, and `None` to omit it. ## Debug logging Every HTTP call emits `log::debug!` records on the `hotdata::http` target — the request line, headers, and body, then the response status and body, with bearer tokens and sensitive fields masked. The SDK installs no logger, so wire up any [`log`](https://docs.rs/log) backend to see them: ```rust // RUST_LOG=hotdata::http=debug cargo run env_logger::init(); ``` ## See also - [hotdata on crates.io](https://crates.io/crates/hotdata) - [API documentation on docs.rs](https://docs.rs/hotdata) - [sdk-rust on GitHub](https://github.com/hotdata-dev/sdk-rust) - [Python SDK](/docs/python-sdk) — the same API from Python - [API Reference](/docs/api-reference) — full HTTP API documentation - [Quick Start](/docs/quick-start) — CLI and workspace setup --- # Agent Skills Source: https://www.hotdata.dev/docs/agent-skills Site index: https://www.hotdata.dev/llms.txt Agent skills are machine-readable instructions that teach coding assistants — Claude Code, Cursor, and others — how to use the Hotdata CLI on your behalf. Once installed, your agent can list workspaces, run queries, create managed databases, and manage ingest sources without you copying commands by hand. Claude Code integration ([YouTube](https://youtu.be/rKrwUg6rGoE)): ## Install Skills ship inside [hotdata-cli](https://github.com/hotdata-dev/hotdata-cli). Install or refresh them with: ```bash hotdata manage skills install ``` This writes skill files into the directories that Claude Code, Cursor, and other supported agents watch for instructions. Check which skills are present: ```bash hotdata manage skills status ``` ## How it works Skills are structured Markdown files (under `skills/hotdata` in the CLI repo) that describe: - **What the CLI can do** — commands, flags, and expected output - **When to use each command** — context and preconditions - **How to chain commands** — multi-step workflows like create → load → query The agent reads these files at startup and uses them to decide which `hotdata` commands to run in response to your natural-language requests. You stay in control: the CLI and API rules still apply. The skill teaches the agent *how* to call them — it does not grant any new permissions. ## Example: ask your agent to query data After installation, open Claude Code (or Cursor) in a project and ask: ``` List my workspaces, then run SELECT COUNT(*) FROM default.public.orders using the mydb managed database. ``` The agent will: 1. Run `hotdata workspaces list` to find the active workspace 2. Run `hotdata databases list` to resolve `mydb` to its database id 3. Run `hotdata query "SELECT COUNT(*) FROM default.public.orders" --database ` 4. Return the result inline in the chat ## Supported agents | Agent | Supported | |-------|-----------| | Claude Code | yes | | Cursor | yes | | Other agents that read skill files | yes | ## Configuration Skills use your existing login and workspace config at `~/.hotdata/config.yml`. No additional setup is needed after `hotdata auth login` and `hotdata workspaces use`. To re-install after updating the CLI: ```bash hotdata manage upgrade hotdata manage skills install ``` ## See also - [CLI Reference](/docs/cli-reference) — Full command documentation - [Quick Start](/docs/quick-start) — Install and authenticate the CLI - [hotdata-cli on GitHub](https://github.com/hotdata-dev/hotdata-cli) --- # Ibis Source: https://www.hotdata.dev/docs/ibis Site index: https://www.hotdata.dev/llms.txt Use [Ibis](https://ibis-project.org/) to create on-demand databases, upload data, and query with Python expressions — get pandas or Arrow results back without writing SQL. ## Install ```bash pip install hotdata-ibis ``` ## Connect ```python import ibis con = ibis.hotdata.connect( api_url="https://api.hotdata.dev", token="YOUR_API_KEY", workspace_id="", ) ``` URL-style also works: ```python con = ibis.connect("hotdata://api.hotdata.dev/?token=...&workspace_id=") ``` ## Quickstart: create a database and query it ```python import time import pandas as pd import ibis con = ibis.hotdata.connect( api_url="https://api.hotdata.dev", token="YOUR_API_KEY", workspace_id="", ) # 1. Create a database and declare the tables you'll load. # Hotdata database names are not unique — create_database returns the id # you'll use for every subsequent operation on this database. db_id = con.create_database("sales", schema="public", tables=["orders"]) # 2. Upload a pandas DataFrame (or PyArrow table) df = pd.DataFrame({ "order_id": [1, 2, 3], "amount": [9.99, 49.99, 5.00], "region": ["west", "east", "west"], }) con.create_table("orders", df, database=(db_id, "public"), overwrite=True) # 3. Uploads are async — wait briefly before querying time.sleep(2) # 4. Query with Ibis expressions # Managed tables are always accessed with catalog "default" t = con.table("orders", database=("default", "public")) result = ( t.group_by("region") .agg(total=t.amount.sum()) .order_by(ibis.desc("total")) .execute() # returns a pandas DataFrame ) # 5. Clean up con.drop_table("orders", database=(db_id, "public")) con.drop_database(db_id) ``` ## Managed databases Managed databases are the primary way to bring data into Hotdata with Ibis. You declare a database and its tables, upload data as parquet, and query immediately. ### Create and load ```python # Declare the database and all table names up front. # create_database returns the database id — pass that id, not the name, # to create_table, drop_table, and drop_database. db_id = con.create_database("analytics", schema="public", tables=["events", "users"]) # Upload from a pandas DataFrame con.create_table("events", events_df, database=(db_id, "public"), overwrite=True) con.create_table("users", users_df, database=(db_id, "public"), overwrite=True) # PyArrow tables also work import pyarrow as pa table = pa.table({"id": [1, 2], "name": ["alice", "bob"]}) con.create_table("users", table, database=(db_id, "public"), overwrite=True) ``` Table names must be declared when the database is created — you cannot upload to a table name that was not listed in `tables=`. ### Query When querying, use `"default"` as the catalog — that is always the SQL prefix for managed tables: ```python t = con.table("events", database=("default", "public")) # Ibis expression result = ( t.filter(t.event_type == "click") .group_by("user_id") .agg(n=t.count()) .execute() ) # Or raw SQL result = con.sql( 'SELECT user_id, COUNT(*) AS n ' 'FROM "default"."public"."events" ' 'WHERE event_type = \'click\' ' 'GROUP BY user_id' ).execute() ``` ### Delete ```python con.drop_table("events", database=(db_id, "public")) con.drop_database(db_id) ``` ### Addressing summary | Operation | `database=` argument | |-----------|----------------------| | `create_table` / `drop_table` | `(database_id, schema)` — the id returned by `create_database` | | `con.table(...)` / `con.sql(...)` when querying | `("default", schema)` | ## Query with Ibis expressions `.execute()` returns a pandas DataFrame. Use `.to_pyarrow()` for an Arrow table or `.to_pyarrow_batches()` to stream batches: ```python t = con.table("orders", database=("default", "public")) # Aggregate and sort summary = ( t.filter(t.amount > 10) .group_by("region") .agg(total=t.amount.sum(), n=t.count()) .order_by(ibis.desc("total")) .execute() ) # Arrow output arrow_table = t.limit(1000).to_pyarrow() # Streaming batches with t.to_pyarrow_batches() as reader: for batch in reader: process(batch) ``` ## Raw SQL Use `con.sql(...)` when you need Hotdata-specific syntax that Ibis doesn't model. You can chain Ibis expressions on the result: ```python base = con.sql( 'SELECT * FROM "default"."public"."orders"', dialect="postgres", ) result = base.filter(base.amount > 10).execute() ``` ## Querying data from existing sources To work with data from your existing databases or warehouses (Postgres, Snowflake, BigQuery, etc.), pull it into a managed database first with an [ingest source](/docs/data-sources). Once ingested, the data is an ordinary managed table — query it through Ibis exactly like the managed-database examples above, using catalog `"default"`: ```python t = con.table("orders", database=("default", "public")) result = t.filter(t.amount > 10).execute() ``` Discover what's available: ```python con.list_catalogs() # catalogs con.list_databases(catalog="default") # schemas con.list_tables(database=("default", "public")) # tables con.get_schema("orders", catalog="default", database="public") ``` ## What's supported | Feature | | |---------|---| | `create_database` / `drop_database` | yes | | `create_table` / `drop_table` (DataFrame or Arrow upload) | yes | | `con.table(...)` with full schema metadata | yes | | Filter, select, join, group\_by, agg, order\_by, limit | yes | | `con.sql(...)` raw SQL | yes | | `.execute()` → pandas, `.to_pyarrow()`, `.to_pyarrow_batches()` | yes | | `list_catalogs`, `list_databases`, `list_tables` | yes | | Temporary tables | no | | Python UDFs | no | | INSERT / UPDATE / DELETE | no | SQL compilation uses Ibis's Postgres dialect. Use `con.sql(...)` as a fallback for expressions that don't compile cleanly. ## See also - [hotdata-ibis on GitHub](https://github.com/hotdata-dev/hotdata-ibis) - [Ibis documentation](https://ibis-project.org) - [Python SDK](/docs/python-sdk) — lower-level `hotdata` API client --- # LangChain Source: https://www.hotdata.dev/docs/langchain Site index: https://www.hotdata.dev/llms.txt LangChain tools for Hotdata — give your chains and agents SQL execution and managed database access. ## Install ```bash pip install hotdata-langchain ``` ## Authentication Set `HOTDATA_API_KEY` in your environment. Optionally set `HOTDATA_WORKSPACE` to pin a specific workspace. ## Quickstart ```python from hotdata_framework import from_env from hotdata_langchain import make_hotdata_tools from langchain.agents import create_agent client = from_env() tools = make_hotdata_tools(client, database_id="dbid...") agent = create_agent( model="openai:gpt-4o", tools=tools, system_prompt="You are a data analyst.", ) result = agent.invoke( {"messages": [{"role": "user", "content": "How many orders are in the database?"}]} ) print(result["messages"][-1].content) ``` Queries run against a database scope, so pass `database_id=` (a managed database id). `from_env().list_managed_databases()` shows what is available in the workspace, with the id of each. ## Tools `make_hotdata_tools` returns a list of `StructuredTool` objects ready to pass to any LangChain agent. By default it returns five tools — `hotdata_describe_tables` is registered unless you pass `describe_tables=False` — plus `hotdata_search_text` when you configure full-text search: | Tool | Description | |------|-------------| | `hotdata_execute_sql` | Run SQL and return JSON rows | | `hotdata_list_managed_databases` | List Hotdata-managed databases, with the id of each | | `hotdata_create_managed_database` | Create a database and declare tables | | `hotdata_load_managed_table` | Load a local parquet file into a table | | `hotdata_describe_tables` | List tables, or one table's columns and types | | `hotdata_search_text` | Full-text search an indexed column, ranked by relevance (opt-in) | ```python tools = make_hotdata_tools( client, max_rows=50, # rows returned to the agent per query (default 100) database_id="dbid...", # scope queries to a specific database id (optional) ) ``` ## Run SQL directly ```python from hotdata_langchain import execute_sql_json, result_rows_for_llm # Returns a JSON string — useful for custom tool wrappers json_str = execute_sql_json(client, "SELECT * FROM orders LIMIT 5") # Returns list[dict] from a QueryResult, trimmed to max_rows result = client.execute_sql("SELECT * FROM orders LIMIT 100") rows = result_rows_for_llm(result, max_rows=20) ``` ## Managed databases ```python from hotdata_langchain.databases import ( create_managed_database, list_managed_databases_json, load_managed_table, ) # Create a database and declare tables db = create_managed_database( client, name="sales", schema="public", tables=["orders", "customers"], ) # Load a local parquet file into a declared table loaded = load_managed_table( client, database_id=db.id, table="orders", file="orders.parquet", ) print(f"Loaded {loaded.row_count} rows → {loaded.full_name}") ``` ## See also - [hotdata-langchain on GitHub](https://github.com/hotdata-dev/hotdata-langchain) - [LangChain](https://python.langchain.com) - [Python SDK](/docs/python-sdk) — low-level `hotdata` HTTP API client --- # dlt Source: https://www.hotdata.dev/docs/dlthub Site index: https://www.hotdata.dev/llms.txt Use [dlt](https://dlthub.com) to build pipelines that load data from any source into Hotdata managed databases — with automatic schema inference, incremental loading, and Parquet-based delivery. The Hotdata destination is a native dlt destination published as a standalone package, [`hotdata-dlt-destination`](https://github.com/hotdata-dev/hotdata-dlt-destination). ## Install ```bash pip install hotdata-dlt-destination ``` For the live ibis backend (`pipeline.dataset().ibis()`), add the `ibis` extra: ```bash pip install "hotdata-dlt-destination[ibis]" ``` ## Authentication The API key is a secret, so it's read from the environment (or a dlt secrets provider): ```bash export HOTDATA_API_KEY="your_api_key" ``` The workspace ID is a routing value, not a secret — pass it as the `workspace_id=` parameter on the destination (there is no environment variable for it). You can also supply the key explicitly with `credentials={"api_key": "..."}` instead of the env var. ## Quickstart ```python import dlt from hotdata_dlt_destination import hotdata @dlt.resource(name="customers", write_disposition="append") def customers(): yield [ {"id": 1, "name": "Alice", "amount": 99.99}, {"id": 2, "name": "Bob", "amount": 49.50}, ] pipeline = dlt.pipeline( pipeline_name="my_pipeline", destination=hotdata( workspace_id="your_workspace_id", database_name="sales", declared_tables=["customers"], ), ) info = pipeline.run(customers()) print(info) ``` dlt infers the schema from your data, creates a managed database labelled `sales` on the first run, and loads the records as Parquet. Nested/child tables and dlt's internal columns (`_dlt_id`, `_dlt_load_id`) are preserved. ## Reuse a database with `database_id` Managed databases are addressed by **id**, not name — Hotdata database names are not unique, so a name can't identify one. On first run the destination prints the new database's id: ``` hotdata: created managed database db_abc123 (name='sales'). Pin it for future runs by setting database_id=db_abc123. ``` To keep loading into the **same** database on later runs, pin that id — via `hotdata(database_id="db_abc123")`, the `HOTDATA_DATABASE_ID` environment variable, or `[destination.hotdata] database_id` in `.dlt/config.toml`. **Without a pinned id, every run creates a new database.** `database_name` is only a display label used when creating a database; it never looks one up. ## Configure the destination ```python from hotdata_dlt_destination import hotdata destination = hotdata( workspace_id="your_workspace_id", # required (no env var) database_id="db_abc123", # reuse an existing database (recommended) database_name="sales", # label when creating a new database (default: "dlt") schema="public", # schema within the database (default: "public") write_disposition="append", # default disposition (default: "append") declared_tables=["customers"], # all table names the pipeline writes create_database_if_missing=True, # auto-create the database (default: True) ) ``` The API key is the exception — being a secret it comes from `HOTDATA_API_KEY` (or `credentials={"api_key": "..."}`), not a keyword above. Every other parameter can also be set via environment variable or `.dlt/config.toml`: | Parameter | Env variable | Default | |-----------|--------------|---------| | `workspace_id` | — (param only) | required | | `database_id` | `HOTDATA_DATABASE_ID` | — | | `database_name` | `HOTDATA_DATABASE` | `dlt` | | `schema` | `HOTDATA_SCHEMA` | `public` | | `write_disposition` | `HOTDATA_WRITE_DISPOSITION` | `append` | | `declared_tables` | `HOTDATA_DECLARED_TABLES` | — | | `create_database_if_missing` | `HOTDATA_CREATE_DATABASE_IF_MISSING` | `True` | | `api_base_url` | `HOTDATA_API_BASE_URL` | `https://api.hotdata.dev` | When a pipeline writes more than one table, pass every table name via `declared_tables`. If you add a new table later, include it on the next run — it's added to the existing database in place, without recreating it or moving data. ## Load from a source Use any dlt-verified source or a custom generator: ```python import dlt from dlt.sources.sql_database import sql_database from hotdata_dlt_destination import hotdata source = sql_database( credentials="postgresql://user:pass@host/db", schema="public", table_names=["orders", "customers"], ) pipeline = dlt.pipeline( pipeline_name="postgres_to_hotdata", destination=hotdata( workspace_id="your_workspace_id", database_name="sales", declared_tables=["orders", "customers"], ), ) info = pipeline.run(source) print(f"Loaded {info.loads_ids} into Hotdata") ``` ## Incremental loading dlt tracks state between runs — pipeline state is persisted in the managed database, so only new or updated rows are loaded on subsequent executions. Pin `database_id` so the state is found on the next run: ```python import dlt from hotdata_dlt_destination import hotdata @dlt.resource(primary_key="id", write_disposition="merge") def events( updated_at=dlt.sources.incremental("updated_at") ): # fetch rows newer than updated_at.last_value yield fetch_events(since=updated_at.last_value) pipeline = dlt.pipeline( pipeline_name="events_pipeline", destination=hotdata( workspace_id="your_workspace_id", database_id="db_abc123", declared_tables=["events"], ), ) pipeline.run(events()) ``` Write dispositions: | Disposition | Behaviour | |-------------|-----------| | `append` | Add new rows to the table | | `replace` | Replace the full table on each run | | `merge` | Upsert rows matched by `primary_key` (updates matches, inserts the rest); without a `primary_key` it falls back to a client-side combine | A table's key is declared the first time it's created — changing a resource's `primary_key` on a later run does not update the server-side key. ## Partition and sort keys A managed table's partition and sort keys are fixed when the table is first created, so declare them before the first load. Simple cases use per-column hints: ```python @dlt.resource(columns={ "event_date": {"partition": True}, "event_time": {"sort": True}, }) def events(): ... ``` For key order, partition transforms (`year` / `month` / `day` / `hour`), or sort direction, use the adapter: ```python from hotdata_dlt_destination import hotdata_adapter hotdata_adapter( events, partition_by=[("event_date", "identity")], sorted_by=["event_time", ("tag_mac", "asc", "last")], ) ``` ## Read your data back The same dlt [dataset interface](https://dlthub.com/docs/general-usage/dataset-access/dataset) reads loaded tables back — queries run server-side on Hotdata's Apache DataFusion engine. Point the reading pipeline at the `database_id` you pinned: ```python pipeline = dlt.pipeline( pipeline_name="my_pipeline", destination=hotdata( workspace_id="your_workspace_id", database_id="db_abc123", declared_tables=["customers"], ), ) ds = pipeline.dataset() ds.table("customers").df() # whole table -> pandas.DataFrame ds.table("customers").arrow() # -> pyarrow.Table # raw SQL ds("SELECT name, sum(amount) AS spend FROM customers GROUP BY name").df() # fluent ds.table("customers").select("id", "amount").where("amount > 50").limit(10).df() ``` With the `[ibis]` extra, `ds.ibis()` returns a live [`ibis.hotdata` backend](/docs/ibis) for authoring queries as ibis expressions. ## Verify a load Use the [Hotdata CLI](/docs/quick-start) to confirm the data landed. Address the database by the id printed on first-run create — names aren't unique: ```bash # List managed databases (shows each id) hotdata databases list # Query the loaded data hotdata query "SELECT name, amount FROM public.customers ORDER BY amount DESC" --database db_abc123 ``` ## See also - [hotdata-dlt-destination on GitHub](https://github.com/hotdata-dev/hotdata-dlt-destination) - [dlt documentation](https://dlthub.com/docs) - [Python SDK](/docs/python-sdk) — low-level `hotdata` HTTP API client - [Quick Start](/docs/quick-start) — CLI and workspace setup --- # API Reference Source: https://www.hotdata.dev/docs/api-reference Site index: https://www.hotdata.dev/llms.txt Powerful data platform API for managed databases, queries, and analytics. Hotdata exposes a `/v1/*` HTTP API at [api.hotdata.dev](https://api.hotdata.dev). [OpenAPI 3.1 specification](/openapi.yaml) ## Authentication Most `/v1/*` endpoints require these headers: ```http Authorization: Bearer X-Workspace-Id: ``` - `Authorization` — Org-scoped API token obtained via CLI login or the dashboard. - `X-Workspace-Id` — Public ID of the target workspace. Operations that need a narrower scope take an extra header — those are listed on the endpoint itself. ## All endpoints ### Workspaces | Method | Path | Operation | | ------ | ---- | --------- | | `GET` | `/v1/workspaces` | [List workspaces](/docs/api-reference/workspaces#list-workspaces) | | `POST` | `/v1/workspaces` | [Create a workspace](/docs/api-reference/workspaces#create-a-workspace) | | `DELETE` | `/v1/workspaces/{public_id}` | [Delete a workspace](/docs/api-reference/workspaces#delete-a-workspace) | ### Query | Method | Path | Operation | | ------ | ---- | --------- | | `POST` | `/v1/query` | [Execute SQL query](/docs/api-reference/query#execute-sql-query) | ### Information Schema | Method | Path | Operation | | ------ | ---- | --------- | | `GET` | `/v1/information_schema` | [List tables](/docs/api-reference/information-schema#list-tables) | ### Results | Method | Path | Operation | | ------ | ---- | --------- | | `GET` | `/v1/results` | [List results](/docs/api-reference/results#list-results) | | `GET` | `/v1/results/{id}` | [Get result](/docs/api-reference/results#get-result) | ### Query Runs | Method | Path | Operation | | ------ | ---- | --------- | | `GET` | `/v1/query-runs` | [List query runs](/docs/api-reference/query-runs#list-query-runs) | | `GET` | `/v1/query-runs/{id}` | [Get query run](/docs/api-reference/query-runs#get-query-run) | ### Uploads | Method | Path | Operation | | ------ | ---- | --------- | | `POST` | `/v1/uploads` | [Create upload session](/docs/api-reference/uploads#create-upload-session) | | `POST` | `/v1/uploads/batch` | [Create upload sessions in bulk](/docs/api-reference/uploads#create-upload-sessions-in-bulk) | | `POST` | `/v1/uploads/{upload_id}/finalize` | [Finalize upload](/docs/api-reference/uploads#finalize-upload) | | `POST` | `/v1/uploads/{upload_id}/parts` | [Mint upload part URLs](/docs/api-reference/uploads#mint-upload-part-urls) | ### Saved Queries | Method | Path | Operation | | ------ | ---- | --------- | | `GET` | `/v1/queries` | [List saved queries](/docs/api-reference/saved-queries#list-saved-queries) | | `POST` | `/v1/queries` | [Create saved query](/docs/api-reference/saved-queries#create-saved-query) | | `GET` | `/v1/queries/{id}` | [Get saved query](/docs/api-reference/saved-queries#get-saved-query) | | `PUT` | `/v1/queries/{id}` | [Update saved query](/docs/api-reference/saved-queries#update-saved-query) | | `DELETE` | `/v1/queries/{id}` | [Delete saved query](/docs/api-reference/saved-queries#delete-saved-query) | | `POST` | `/v1/queries/{id}/execute` | [Execute saved query](/docs/api-reference/saved-queries#execute-saved-query) | | `GET` | `/v1/queries/{id}/versions` | [List saved query versions](/docs/api-reference/saved-queries#list-saved-query-versions) | ### Indexes | Method | Path | Operation | | ------ | ---- | --------- | | `GET` | `/v1/indexes` | [List indexes across tables in a database](/docs/api-reference/indexes#list-indexes-across-tables-in-a-database) | ### Embedding Providers | Method | Path | Operation | | ------ | ---- | --------- | | `GET` | `/v1/embedding-providers` | [List embedding providers](/docs/api-reference/embedding-providers#list-embedding-providers) | | `POST` | `/v1/embedding-providers` | [Create embedding provider](/docs/api-reference/embedding-providers#create-embedding-provider) | | `GET` | `/v1/embedding-providers/{id}` | [Get embedding provider](/docs/api-reference/embedding-providers#get-embedding-provider) | | `PUT` | `/v1/embedding-providers/{id}` | [Update embedding provider](/docs/api-reference/embedding-providers#update-embedding-provider) | | `DELETE` | `/v1/embedding-providers/{id}` | [Delete embedding provider](/docs/api-reference/embedding-providers#delete-embedding-provider) | ### Jobs | Method | Path | Operation | | ------ | ---- | --------- | | `GET` | `/v1/jobs` | [List jobs](/docs/api-reference/jobs#list-jobs) | | `GET` | `/v1/jobs/{id}` | [Get job status](/docs/api-reference/jobs#get-job-status) | ### Database context | Method | Path | Operation | | ------ | ---- | --------- | | `GET` | `/v1/databases/{database_id}/context` | [List database contexts](/docs/api-reference/database-context#list-database-contexts) | | `POST` | `/v1/databases/{database_id}/context` | [Create or update database context](/docs/api-reference/database-context#create-or-update-database-context) | | `GET` | `/v1/databases/{database_id}/context/{name}` | [Get one database context](/docs/api-reference/database-context#get-one-database-context) | | `DELETE` | `/v1/databases/{database_id}/context/{name}` | [Delete database context](/docs/api-reference/database-context#delete-database-context) | ### Databases | Method | Path | Operation | | ------ | ---- | --------- | | `GET` | `/v1/databases` | [List databases](/docs/api-reference/databases#list-databases) | | `POST` | `/v1/databases` | [Create database](/docs/api-reference/databases#create-database) | | `POST` | `/v1/databases/bulk` | [Create many databases at once](/docs/api-reference/databases#create-many-databases-at-once) | | `GET` | `/v1/databases/bulk/{batch_id}` | [Get a database batch](/docs/api-reference/databases#get-a-database-batch) | | `DELETE` | `/v1/databases/bulk/{batch_id}` | [Delete a database batch](/docs/api-reference/databases#delete-a-database-batch) | | `GET` | `/v1/databases/count` | [Count databases](/docs/api-reference/databases#count-databases) | | `GET` | `/v1/databases/{database_id}` | [Get database](/docs/api-reference/databases#get-database) | | `DELETE` | `/v1/databases/{database_id}` | [Delete database](/docs/api-reference/databases#delete-database) | | `POST` | `/v1/databases/{database_id}/catalogs` | [Attach catalog to database](/docs/api-reference/databases#attach-catalog-to-database) | | `DELETE` | `/v1/databases/{database_id}/catalogs/{connection_id}` | [Detach catalog from database](/docs/api-reference/databases#detach-catalog-from-database) | | `POST` | `/v1/databases/{database_id}/fork` | [Fork database](/docs/api-reference/databases#fork-database) | | `POST` | `/v1/databases/{database_id}/schemas` | [Add schema to database default catalog](/docs/api-reference/databases#add-schema-to-database-default-catalog) | | `POST` | `/v1/databases/{database_id}/schemas/{schema}/tables` | [Add table to database default catalog](/docs/api-reference/databases#add-table-to-database-default-catalog) | | `POST` | `/v1/databases/{database_id}/schemas/{schema}/tables/{table}/loads` | [Load database table from inline data, upload, or query result](/docs/api-reference/databases#load-database-table-from-inline-data-upload-or-query-result) | ### Usage | Method | Path | Operation | | ------ | ---- | --------- | | `GET` | `/v1/usage` | [Get workspace usage snapshot](/docs/api-reference/usage#get-workspace-usage-snapshot) | ## Error responses Failures carry a non-2xx HTTP status and a JSON body. Each endpoint lists the statuses it can return; the bodies use these shapes: ### ApiErrorResponse Standard error response body. Used by 73 responses. - `error` `ApiErrorDetail` — **required**. Error detail within an API error response - `code` `string` — **required** - `message` `string` — **required** ```json { "error": { "code": "string", "message": "string" } } ``` ### Error Used by 11 responses. - `error` `string` — **required**. Machine-readable error code. ```json { "error": "missing_authorization" } ``` ## Rate limiting API requests are rate limited. When a limit is exceeded the API returns `429 Too Many Requests` with a `Retry-After` header giving the seconds to wait before retrying. --- # Workspaces Source: https://www.hotdata.dev/docs/api-reference/workspaces Site index: https://www.hotdata.dev/llms.txt Workspace management ## List workspaces `GET /v1/workspaces` Lists all workspaces in the user's organization. **Query parameters** - `organization_public_id` `string` — Filter by organization. Defaults to the user's current organization. **Response** `200` — Successful response - `ok` `boolean` — **required** - `workspaces` `WorkspaceListItem`[] — **required** - `public_id` `string` — **required** - `name` `string` — **required** - `active` `boolean` — **required** - `favorite` `boolean` — **required** - `provision_status` `string` — **required** ```json { "ok": true, "workspaces": [ { "public_id": "workm4lz2mp899l2i7h9lk9u84azg3", "name": "production-analytics", "active": true, "favorite": true, "provision_status": "provisioned" } ] } ``` **Errors** | Status | Description | | ------ | ----------- | | `401` | Missing or invalid authorization | | `403` | Forbidden — not a member of the organization or workspace token used | | `404` | Organization not found | ## Create a workspace `POST /v1/workspaces` Creates a new workspace in the specified organization. **Request body** - `name` `string` — **required**. Name for the new workspace. - `organization_public_id` `string` — Target organization. Defaults to the user's current organization. ```json { "name": "production-analytics" } ``` **Response** `201` — Workspace created - `ok` `boolean` — **required** - `workspace` `WorkspaceDetail` — **required** - `public_id` `string` — **required** - `name` `string` — **required** - `provision_status` `string` — **required** ```json { "ok": true, "workspace": { "public_id": "workm4lz2mp899l2i7h9lk9u84azg3", "name": "production-analytics", "provision_status": "pending" } } ``` **Errors** | Status | Description | | ------ | ----------- | | `400` | Invalid JSON body | | `401` | Missing or invalid authorization | | `403` | Forbidden | | `404` | Organization not found | | `422` | Validation error (e.g. name required) | ## Delete a workspace `DELETE /v1/workspaces/{public_id}` Hard-deletes the workspace. Namespace, storage, and catalog deprovisioning runs asynchronously after the row is removed. **Response** `204` — Workspace deleted **Errors** | Status | Description | | ------ | ----------- | | `401` | Missing or invalid authorization | | `403` | Workspace-scoped tokens are not allowed | | `404` | Workspace not found, or caller is not a member of its organization | --- # Query Source: https://www.hotdata.dev/docs/api-reference/query Site index: https://www.hotdata.dev/llms.txt Execute SQL queries against connected data sources. Use standard Postgres-compatible SQL syntax to query and join across multiple connections. Results are returned inline and also persisted asynchronously for later retrieval. ## Execute SQL query `POST /v1/query` Execute a SQL query scoped to a database. A database is the only window into catalogs: the query sees only that database's auto `default` catalog plus any catalogs explicitly attached to it. Select the database with EITHER the `X-Database-Id` header OR the `database_id` body field (exactly one must be given; if both are sent and disagree, that's a 400). Use standard Postgres-compatible SQL; reference the default catalog as `default..
` (or just `.
` / `
`) and attached catalogs by their alias. Results are returned inline and a `result_id` is provided for later retrieval via the Results API. Set `async: true` to execute asynchronously — returns a query run ID for polling. Optionally set `async_after_ms` to attempt synchronous execution first, falling back to async if the query exceeds the timeout. **Headers** - `X-Database-Id` `string,null` — Database id to scope the query to. Required unless the `database_id` body field is set; if both are present they must match. Only that database's catalogs are visible during planning. A malformed value is a 400; an unknown database id is a 404. **Request body** - `async` `boolean` — When true, execute the query asynchronously and return a query run ID for polling via GET /query-runs/{id}. The query results can be retrieved via GET /results/{id} once the query run status is "succeeded". Default: `false` - `async_after_ms` `integer,null` — If set (requires `async` = true), first attempt the query synchronously and wait up to this many milliseconds: if it finishes in time the full result is returned, otherwise an async response (a run id to poll) is returned. Must be at least 1000 and at most the server's configured maximum; a value out of that range, or set without `async` = true, is rejected with 400. Min: `1000` - `database_id` `string,null` — Database to scope the query to (its id). Alternative to the `X-Database-Id` header — exactly one source must be provided. If both this field and the header are set and they disagree, the request is rejected with a 400. - `default_catalog` `string,null` — Catalog that unqualified table references resolve against within the query's database scope. Must name a catalog visible in the database (`default`, an attached catalog alias, or a system catalog). Defaults to `default` when omitted. - `default_schema` `string,null` — Schema that unqualified table references resolve against within the query's database scope. Defaults to `main` when omitted. Existence is not validated up front — an unknown schema surfaces as a "table not found" error at planning time. - `dialect` `string,null` — SQL dialect the `sql` field is written in. One of `hotsql` (the default), `duckdb`, `postgres`, or `snowflake`. When set to anything other than `hotsql`, the query is translated to HotSQL before it runs, so you can use idioms from that dialect (for example Snowflake `IFF(...)` or Postgres `MOD(a, b)`). Only read-only queries are accepted. An unrecognized value is rejected with a 400. - `sql` `string` — **required** ```json { "async": false, "database_id": "dbid6lguax1dxn9y1xj5gxnameyywl", "default_catalog": "default", "default_schema": "main", "dialect": "hotsql", "sql": "SELECT customer_id, sum(amount) AS total FROM orders GROUP BY customer_id ORDER BY total DESC LIMIT 10" } ``` **Response** `200` — Query executed successfully - `columns` `string`[] — **required** - `execution_time_ms` `integer` — **required**. Min: `0` - `nullable` `boolean`[] — **required**. Nullable flags for each column (parallel to columns vec). True if the column allows NULL values, false if NOT NULL. - `preview_row_count` `integer` — **required**. Number of rows in *this* response body. Always present. For a large result this is a bounded preview, not the grand total — see `total_row_count` and `truncated`. - `query_run_id` `string` — **required**. Unique identifier for the query run record (qrun...). - `result_id` `string,null` — Unique identifier for retrieving this result via GET /results/{id}. When non-null, the result is being persisted asynchronously. Null only when the result fit entirely in this response (`truncated: false`) but could not be persisted for later retrieval — see the `warning` field. A `truncated: true` response ALWAYS carries a non-null, resolvable `result_id`: a truncated result that cannot be persisted fails the request with a retryable HTTP 503 (`PERSISTENCE_UNAVAILABLE`, with a `Retry-After` header) rather than returning a partial body with a dead ticket. - `row_count` `integer` — **required**. **Deprecated** — use `preview_row_count` (rows in this body) and `total_row_count` (grand total) instead. Retained as a back-compat alias and always equal to `preview_row_count`; for a truncated result it is the preview count, *not* the grand total — read `total_row_count` for that. Will be removed in a future release once clients migrate. Min: `0` - `rows` `any`[][] — **required**. Array of rows, where each row is an array of column values. Values can be strings, numbers, booleans, or null. - `total_row_count` `integer,null` — Grand total rows in the full result. Present (and equal to `preview_row_count`) when the whole result fit in this response; `null` while a truncated result is still being persisted. When `null`, read the authoritative total from `GET /v1/query-runs/{id}` (`row_count`) or the `X-Total-Row-Count` header on `GET /v1/results/{id}`. - `truncated` `boolean` — **required**. True when `rows` is a bounded preview of a larger result. Fetch the full result via `result_id`. - `warning` `string,null` — Warning message if result persistence could not be initiated. Present only when the full result is returned inline (`truncated: false`) but could not be persisted: `result_id` is then null and the result cannot be re-fetched later, though every row is in this response. A truncated result never carries a warning — if it cannot be persisted the request fails with a retryable HTTP 503 (`PERSISTENCE_UNAVAILABLE`, with a `Retry-After` header) instead. ```json { "columns": [ "string" ], "execution_time_ms": 0, "nullable": [ true ], "preview_row_count": 0, "query_run_id": "string", "result_id": "string", "row_count": 0, "rows": [ [ null ] ], "total_row_count": 0, "truncated": true, "warning": "string" } ``` **Response** `202` — Query submitted asynchronously - `query_run_id` `string` — **required**. Unique identifier for the query run. - `reason` `string,null` — Human-readable reason why the query went async (e.g., caching tables for the first time). - `status` `string` — **required**. Current status of the query run. - `status_url` `string` — **required**. URL to poll for query run status. Requires the same `X-Database-Id` header used to submit the query. ```json { "query_run_id": "string", "reason": "string", "status": "string", "status_url": "string" } ``` **Errors** | Status | Description | | ------ | ----------- | | `400` | Invalid request (no database specified, or header/body database_id conflict) | | `404` | Database not found | | `429` | The engine was too busy to run this query right now — too many concurrent queries, or not enough memory available (often because of other queries running at the same time). Retry after the Retry-After delay; if it persists, narrowing the query (add a filter or LIMIT) may help. | | `500` | Internal server error | | `503` | Result store temporarily unavailable (a truncated result could not be persisted); retry after the Retry-After delay | --- # Information Schema Source: https://www.hotdata.dev/docs/api-reference/information-schema Site index: https://www.hotdata.dev/llms.txt Inspect table and column metadata across all connections. Returns schema information including column names, data types, and sync status for every discovered table. ## List tables `GET /v1/information_schema` List discovered tables with optional filtering and pagination. Supports wildcard patterns (SQL %) for schema and table name filters. Set include_columns=true to include column definitions (omitted by default). Every table carries its declared storage layout — `partition_by` and `sorted_by` — which is fixed when the table is created and cannot be changed afterwards. Both are always present; an empty array means none was declared. Only tables in a hotdata-managed database declare a layout here, so a table discovered from an external connection always reports empty arrays. **Query parameters** - `connection_id` `string` — Filter by connection ID - `schema` `string` — Filter by schema name (supports % wildcards) - `table` `string` — Filter by table name (supports % wildcards) - `include_columns` `boolean` — Include column definitions (default: false) - `limit` `integer` — Maximum number of tables per page - `cursor` `string` — Pagination cursor from a previous response **Response** `200` — Table metadata - `count` `integer` — **required**. Min: `0` - `has_more` `boolean` — **required** - `limit` `integer` — **required**. Min: `0` - `next_cursor` `string,null` - `tables` `TableInfo`[] — **required** - `columns` `ColumnInfo`[] | `null` - `data_type` `string` — **required** - `name` `string` — **required** - `nullable` `boolean` — **required** - `connection` `string` — **required** - `last_sync` `string,null` - `partition_by` `TablePartitionKey`[] — **required**. The table's partition keys, in the order they were declared when the table was created. Empty when the table is not partitioned. A table's storage layout is fixed when the table is created and cannot be changed afterwards, so this is how to confirm a table really was created with the layout that was asked for. The field is always present: an empty array means "no partitioning declared", which is not the same as a response that omits the field entirely. Reported for tables in a hotdata-managed database, which are the only ones whose layout is declared here. A table discovered from an external connection always reports an empty array — its layout belongs to the upstream system, so an empty array there means "not known from here", not "confirmed unpartitioned". - `column` `string` — **required**. Column the key reads. - `transform` `string` — **required**. How the value is derived from the column. One of `identity` (the column value itself), `year`, `month`, `day`, or `hour`. - `schema` `string` — **required** - `sorted_by` `TableSortKey`[] — **required**. The table's sort keys, in the order they were declared when the table was created. Empty when no sort order was declared. Always present, and limited to tables in a hotdata-managed database, for the same reasons as `partition_by`. - `column` `string` — **required** - `direction` `string,null` — `asc` (the default) or `desc`. Null when the table was declared without an explicit direction for this key. - `nulls` `string,null` — Where nulls are placed: `first` or `last`. Defaults to the SQL default for the chosen direction. Null when the table was declared without an explicit placement for this key. - `synced` `boolean` — **required** - `table` `string` — **required** ```json { "count": 0, "has_more": true, "limit": 0, "next_cursor": "string", "tables": [ { "columns": [ {} ], "connection": "string", "last_sync": "string", "partition_by": [ {} ], "schema": "string", "sorted_by": [ {} ], "synced": true, "table": "string" } ] } ``` **Errors** | Status | Description | | ------ | ----------- | | `404` | Connection not found | --- # Results Source: https://www.hotdata.dev/docs/api-reference/results Site index: https://www.hotdata.dev/llms.txt Retrieve persisted query results. Every query execution persists its results asynchronously. Results transition through statuses: processing → ready (or failed). Once ready, the full result data can be retrieved by ID. ## List results `GET /v1/results` List stored results for the database named by the required X-Database-Id header. **Query parameters** - `limit` `integer` — Maximum number of results (default: 100, max: 1000) - `offset` `integer` — Pagination offset (default: 0) **Headers** - `X-Database-Id` `string` — **required**. Database to scope the results to (required) **Response** `200` — List of results - `count` `integer` — **required**. Number of results returned in this response. Min: `0` - `has_more` `boolean` — **required**. Whether there are more results available after this page - `limit` `integer` — **required**. Limit used for this request. Min: `0` - `offset` `integer` — **required**. Pagination offset used for this request. Min: `0` - `results` `ResultInfo`[] — **required** - `created_at` `string` — **required** - `error_message` `string,null` - `id` `string` — **required** - `status` `string` — **required** ```json { "count": 0, "has_more": true, "limit": 0, "offset": 0, "results": [ { "created_at": "2026-01-01T00:00:00Z", "error_message": "string", "id": "string", "status": "string" } ] } ``` **Errors** | Status | Description | | ------ | ----------- | | `400` | Missing or malformed X-Database-Id header | | `404` | Database not found | ## Get result `GET /v1/results/{id}` Retrieve a persisted query result by ID. The response format for the `ready` state is selected by `Accept` header or `?format=` query param; non-ready states use the same status codes and JSON body shape regardless of format. | Result status | Status × body | |-----------------------|------------------------------------------------------------------------------| | `ready` + JSON | 200 `application/json` — `GetResultResponse` with `columns`, `rows`, etc. | | `ready` + Arrow | 200 `application/vnd.apache.arrow.stream` — schema, RecordBatches, EOS | | `ready` + CSV | 200 `text/csv; charset=utf-8` — single header row, streamed batch-by-batch | | `ready` + Markdown | 200 `text/markdown; charset=utf-8` — GitHub-flavored pipe table, streamed | | `ready` + Parquet | 200 `application/vnd.apache.parquet` — raw parquet bytes (no conversion) | | `pending`/`processing`| 202 `application/json` `{status, result_id}` + `Retry-After` | | `failed` | 409 `application/json` `{status, result_id, error_message}` | | not found | 404 `application/json` (`ApiErrorResponse`) | `?format=` accepts `arrow`, `json`, `csv`, `md`, `parquet` and takes precedence over `Accept`. `markdown` is accepted as a runtime alias for `md`. Use `?offset=N&limit=M` to slice the result; `offset` defaults to 0 and `limit` is unbounded by default. Both must be non-negative; invalid values return 400. When a finite `limit` doesn't reach the end of the result, a `Link` header with `rel="next"` points at the following page. `?offset`/`?limit` are ignored for `format=parquet` since that path returns the underlying file unchanged. Ready responses (Arrow, CSV, Markdown, JSON) carry `X-Total-Row-Count` (the full result row count, independent of offset/limit). Responses are streamed end-to-end, so a client can disconnect at any time and the server stops reading. IEEE special floats (`±Inf`, `NaN`) have no canonical JSON representation. For cross-format consistency the JSON, CSV, and Markdown paths emit them as `null` / empty cells, and JSON `nullable[]` is widened to match. The Arrow IPC and Parquet bodies are binary round-trip formats and preserve the raw IEEE values; callers cross-checking a result across CSV and Parquet should not byte-compare those slots. **Path parameters** - `id` `string` — **required**. Result ID **Query parameters** - `offset` `integer` — Rows to skip (default: 0) - `limit` `integer` — Maximum rows to return (default: unbounded) - `format` `ResultsFormatQuery` — `arrow`, `json`, `csv`, `md`, or `parquet` — overrides the `Accept` header. `markdown` is also accepted at runtime as an alias for `md`. **Headers** - `X-Database-Id` `string` — **required**. Database the result belongs to (required) **Response** `200` — Result data. The body depends on the negotiated format: JSON callers receive `GetResultResponse`; Arrow callers receive an Arrow IPC stream; CSV callers receive comma-separated text (LF-terminated, double-quote escaped, RFC 4180-style quoting but not RFC 4180-strict on line endings); Markdown callers receive a single GitHub-flavored pipe table; Parquet callers receive the raw parquet bytes, served as-is. Non-finite floats (`±Inf`, `NaN`) render as `null` (JSON) or empty cells (CSV, Markdown) for cross-format consistency. `Accept` is treated as a flat content-type list — `q=` quality values are ignored; use `?format=` to disambiguate. - `columns` `string`[] | `null` - `error_message` `string,null` - `nullable` `boolean`[] | `null` - `result_id` `string` — **required** - `row_count` `integer,null` - `rows` `any`[][] | `null` — Array of rows, where each row is an array of column values. - `status` `string` — **required** ```json { "columns": [ "string" ], "error_message": "string", "nullable": [ true ], "result_id": "string", "row_count": 0, "rows": [ [ null ] ], "status": "string" } ``` **Response** `202` — Result is still being computed (`pending` or `processing`). Poll the same URL. - `columns` `string`[] | `null` - `error_message` `string,null` - `nullable` `boolean`[] | `null` - `result_id` `string` — **required** - `row_count` `integer,null` - `rows` `any`[][] | `null` — Array of rows, where each row is an array of column values. - `status` `string` — **required** ```json { "columns": [ "string" ], "error_message": "string", "nullable": [ true ], "result_id": "string", "row_count": 0, "rows": [ [ null ] ], "status": "string" } ``` **Errors** | Status | Description | | ------ | ----------- | | `400` | Invalid offset, limit, or format. | | `404` | Result not found. | | `409` | Result computation failed. Body carries `error_message` describing the failure. | --- # Query Runs Source: https://www.hotdata.dev/docs/api-reference/query-runs Site index: https://www.hotdata.dev/llms.txt View the history of executed queries. Each query run records the SQL text, execution time, result reference, and status. Useful for auditing and debugging query performance. ## List query runs `GET /v1/query-runs` List query runs for the database named by the required X-Database-Id header. **Query parameters** - `limit` `integer` — Maximum number of results - `cursor` `string` — Pagination cursor - `status` `string` — Filter by status (comma-separated, e.g. status=running,failed) - `saved_query_id` `string` — Filter by saved query ID **Headers** - `X-Database-Id` `string` — **required**. Database to scope the query runs to (required) **Response** `200` — List of query runs - `count` `integer` — **required**. Min: `0` - `has_more` `boolean` — **required** - `limit` `integer` — **required**. Min: `0` - `next_cursor` `string,null` - `query_runs` `QueryRunInfo`[] — **required** - `bytes_scanned` `integer,null` — Total bytes of table data read from storage to run this query. `null` when the query touches no table at all (for example a constant expression like `SELECT 1`). May be `0` when the query reads a table but not its row data — for example a row count served from table statistics. - `completed_at` `string,null` - `created_at` `string` — **required** - `error_message` `string,null` - `execution_time_ms` `integer,null` - `id` `string` — **required** - `result_id` `string,null` - `row_count` `integer,null` - `rows_scanned` `integer,null` — Total rows read from storage to run this query, before any filtering or aggregation. Distinct from `row_count`, which is how many rows the query returned. `null` when the query reads no table data from storage. - `saved_query_id` `string,null` - `saved_query_version` `integer,null` - `server_processing_ms` `integer,null` — Total server-side processing time for this query (milliseconds). Measured from query start to result ready. Includes SQL execution, task spawning, and result preparation. Does not include network transit. Populated for all completed query runs (sync and async). - `snapshot_id` `string` — **required** - `sql_hash` `string` — **required** - `sql_text` `string` — **required** - `status` `string` — **required** - `trace_id` `string,null` - `user_public_id` `string,null` — Who ran this query: the account id from the access token the request was made with. Use it to group a caller's query history. Requests made with a credential that identifies no account instead record an opaque `user_`-prefixed identifier, which is stable for that credential but cannot be resolved to an account. - `warning_message` `string,null` ```json { "count": 0, "has_more": true, "limit": 0, "next_cursor": "string", "query_runs": [ { "bytes_scanned": 0, "completed_at": "2026-01-01T00:00:00Z", "created_at": "2026-01-01T00:00:00Z", "error_message": "string", "execution_time_ms": 0, "id": "string", "result_id": "string", "row_count": 0, "rows_scanned": 0, "saved_query_id": "string", "saved_query_version": 0, "server_processing_ms": 0, "snapshot_id": "string", "sql_hash": "string", "sql_text": "string", "status": "string", "trace_id": "string", "user_public_id": "string", "warning_message": "string" } ] } ``` **Errors** | Status | Description | | ------ | ----------- | | `400` | Missing or malformed X-Database-Id header | | `404` | Database not found | ## Get query run `GET /v1/query-runs/{id}` Get the status and details of a specific query run by ID, scoped to the database named by the required X-Database-Id header. **Path parameters** - `id` `string` — **required**. Query run ID **Headers** - `X-Database-Id` `string` — **required**. Database the query run belongs to (required) **Response** `200` — Query run details - `bytes_scanned` `integer,null` — Total bytes of table data read from storage to run this query. `null` when the query touches no table at all (for example a constant expression like `SELECT 1`). May be `0` when the query reads a table but not its row data — for example a row count served from table statistics. - `completed_at` `string,null` - `created_at` `string` — **required** - `error_message` `string,null` - `execution_time_ms` `integer,null` - `id` `string` — **required** - `result_id` `string,null` - `row_count` `integer,null` - `rows_scanned` `integer,null` — Total rows read from storage to run this query, before any filtering or aggregation. Distinct from `row_count`, which is how many rows the query returned. `null` when the query reads no table data from storage. - `saved_query_id` `string,null` - `saved_query_version` `integer,null` - `server_processing_ms` `integer,null` — Total server-side processing time for this query (milliseconds). Measured from query start to result ready. Includes SQL execution, task spawning, and result preparation. Does not include network transit. Populated for all completed query runs (sync and async). - `snapshot_id` `string` — **required** - `sql_hash` `string` — **required** - `sql_text` `string` — **required** - `status` `string` — **required** - `trace_id` `string,null` - `user_public_id` `string,null` — Who ran this query: the account id from the access token the request was made with. Use it to group a caller's query history. Requests made with a credential that identifies no account instead record an opaque `user_`-prefixed identifier, which is stable for that credential but cannot be resolved to an account. - `warning_message` `string,null` ```json { "bytes_scanned": 0, "completed_at": "2026-01-01T00:00:00Z", "created_at": "2026-01-01T00:00:00Z", "error_message": "string", "execution_time_ms": 0, "id": "string", "result_id": "string", "row_count": 0, "rows_scanned": 0, "saved_query_id": "string", "saved_query_version": 0, "server_processing_ms": 0, "snapshot_id": "string", "sql_hash": "string", "sql_text": "string", "status": "string", "trace_id": "string", "user_public_id": "string", "warning_message": "string" } ``` **Errors** | Status | Description | | ------ | ----------- | | `400` | Missing or malformed X-Database-Id header | | `404` | Query run or database not found | --- # Uploads Source: https://www.hotdata.dev/docs/api-reference/uploads Site index: https://www.hotdata.dev/llms.txt Upload files, then reference them by ID when loading a managed table. You create an upload session, then `PUT` the bytes straight to the URL it returns (so the file never passes through this API), and finalize. ### Upload flow 1. `POST /v1/uploads` — returns a one-time `finalize_token` plus one of three shapes: • a single `url` (`mode: single`) — `PUT` the whole file to it; • a list of `part_urls` (`mode: multipart`) when you declared the size — `PUT` each part to its entry; • when you omit `declared_size_bytes` (streaming), `mode: multipart` with a `part_size` but no `part_urls` — call `POST /v1/uploads/{upload_id}/parts` with a batch of `part_numbers` to get a `PUT` URL for each part you're about to upload. 2. `PUT` the bytes: the whole file to `url`, or each part to its URL (keeping each part's `ETag`). For a streaming upload, slice the file into `part_size` chunks and mint part URLs as you go. 3. `POST /v1/uploads/{upload_id}/finalize` with the token (and, for multipart, the part `ETag`s) to confirm the upload and make it usable. ### Which approach to use - **One file, or a handful** — create a session, `PUT`, then finalize. - **Many small files** — `POST /v1/uploads/batch` creates up to 100 sessions in one call; upload and finalize each one independently, at your own pace. - **A large file** — small files are returned as a single `PUT`; larger files are returned as a multi-part upload automatically, which you can upload in parallel and resume part by part. ## Create upload session `POST /v1/uploads` Create an upload session for a file you will upload directly to the URL the response carries, without sending it through this API. The response is one of three shapes. For a small file (`mode: single`) it contains a short-lived `url` to `PUT` the whole file to. For a large file with a known size (`mode: multipart`) it contains `part_urls` and `part_size`: split the file into `part_size`-byte chunks (the last is the remainder) and `PUT` chunk *i* (1-based) to `part_urls[i - 1]`, keeping each response's `ETag`. Slice by `part_size`, not by an even division across the number of part URLs (which can make a non-final part too small). For a file whose size you do not know up front, omit `declared_size_bytes`: the response is `mode: multipart` with a `part_size` but NO `part_urls`. As you stream, call `POST /v1/uploads/{upload_id}/parts` with a batch of `part_numbers` to mint per-part `PUT` URLs, `PUT` each part and keep its `ETag`, then finalize as for any multipart upload. In all cases the response also includes a one-time `finalize_token`. After uploading, call the finalize endpoint with the token (and, for multipart, the `{part_number, e_tag}` list) to make the upload usable as managed-table contents. The returned upload ID can then be passed to the managed-table load endpoint. You may hint a preferred part size with `part_size`; the service clamps it to the allowed range and ignores it for single-`PUT` uploads. A `501` with error code `PRESIGN_UNSUPPORTED` means this deployment cannot issue upload URLs; send the data inline on the load endpoint instead. **Request body** - `checksum_algo` `string,null` — Integrity checksum algorithm you are volunteering for this file. Currently only `sha256` is accepted. Optional; pair with `checksum_value`. - `checksum_value` `string,null` — Integrity checksum value, paired with `checksum_algo`. Optional. - `content_encoding` `string,null` — Content encoding to record for the uploaded file (for example `gzip`). Optional. - `content_type` `string,null` — Content type to record for the uploaded file (for example the Parquet, CSV, or JSON MIME type). Optional. - `declared_size_bytes` `integer,null` — The exact size, in bytes, of the file you will upload. Optional. When provided, it is validated at create time against the maximum allowed size, and again at finalize against the bytes actually uploaded — a mismatch fails the finalize. Omit it to create a streaming (unknown-size) upload: the session is always multi-part and returns no part URLs up front; instead you mint part URLs on demand from `POST /v1/uploads/{upload_id}/parts` as you upload, and finalize validates only that the file is non-empty. Min: `0` - `filename` `string,null` — Original file name, recorded with the upload for your own bookkeeping. Optional and advisory — it does not affect how the file is uploaded or loaded. - `part_size` `integer,null` — Preferred size, in bytes, of each part for a large (multi-part) upload. Optional hint — the service clamps it to the allowed part-size range and to the maximum number of parts, and ignores it for small files uploaded with a single `PUT`. Omit to let the service choose. Min: `0` ```json { "checksum_algo": "sha256", "checksum_value": "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08", "content_type": "application/vnd.apache.parquet", "declared_size_bytes": 10485760, "filename": "orders.parquet", "part_size": 8388608 } ``` **Response** `201` — Upload session created - `finalize_token` `string` — **required**. One-time token that authorizes finalizing this upload. Returned exactly once at create time — store it; it cannot be retrieved again. - `headers` `object` — **required**. Headers you must send verbatim with each `PUT`. Currently always empty; present so a future mode can require signed headers without changing the response shape. - `mode` `string` — **required**. Upload mode: `single` (upload the whole file with one `PUT` to `url`) or `multipart` (upload each part with one `PUT` to the matching entry in `part_urls`). Modeled as a string so additional modes can be added later without breaking clients. - `part_size` `integer,null` — For a `multipart` upload (both known-size and streaming), the size in bytes to split the file into: send bytes `[(i-1) * part_size, i * part_size)` as part *i*, with the last part carrying the remainder. Slice by this value — do **not** divide the file evenly by the number of parts, which can make a non-final part smaller than the 5 MiB minimum for a non-final part (the upload then fails at finalize). Absent for `single` uploads. Min: `0` - `part_urls` `string`[] | `null` — For a known-size `multipart` upload, the per-part URLs in ascending part order: `PUT` your file's part *i* (1-based) to `part_urls[i - 1]` and keep each response's `ETag`, then pass the `{part_number, e_tag}` list to finalize. Absent for `single` uploads, and also absent for a streaming (unknown-size) `multipart` upload — there, mint part URLs on demand via `POST /v1/uploads/{upload_id}/parts`. - `upload_id` `string` — **required**. Identifier for this upload. Pass it to the finalize endpoint and to the managed-table load endpoint once finalized. - `url` `string,null` — The URL to `PUT` the raw file bytes to, for a `single` upload. Short-lived — upload promptly and finalize. Absent for `multipart` uploads (use `part_urls`). ```json { "finalize_token": "string", "headers": {}, "mode": "string", "part_size": 0, "part_urls": [ "string" ], "upload_id": "string", "url": "string" } ``` **Errors** | Status | Description | | ------ | ----------- | | `400` | Invalid request (e.g. file too large, unsupported checksum algorithm) | | `501` | This deployment cannot issue upload URLs | ## Create upload sessions in bulk `POST /v1/uploads/batch` Create upload sessions for several files in one request. Each file is planned independently and the response returns one session per requested file, in the same order. Each session is finalized separately via the finalize endpoint, so you can upload and finalize files at your own pace. A `501` with error code `PRESIGN_UNSUPPORTED` means this deployment cannot issue upload URLs; send the data inline on the load endpoint instead. **Request body** - `uploads` `CreateUploadRequest`[] — **required** - `checksum_algo` `string,null` — Integrity checksum algorithm you are volunteering for this file. Currently only `sha256` is accepted. Optional; pair with `checksum_value`. - `checksum_value` `string,null` — Integrity checksum value, paired with `checksum_algo`. Optional. - `content_encoding` `string,null` — Content encoding to record for the uploaded file (for example `gzip`). Optional. - `content_type` `string,null` — Content type to record for the uploaded file (for example the Parquet, CSV, or JSON MIME type). Optional. - `declared_size_bytes` `integer,null` — The exact size, in bytes, of the file you will upload. Optional. When provided, it is validated at create time against the maximum allowed size, and again at finalize against the bytes actually uploaded — a mismatch fails the finalize. Omit it to create a streaming (unknown-size) upload: the session is always multi-part and returns no part URLs up front; instead you mint part URLs on demand from `POST /v1/uploads/{upload_id}/parts` as you upload, and finalize validates only that the file is non-empty. Min: `0` - `filename` `string,null` — Original file name, recorded with the upload for your own bookkeeping. Optional and advisory — it does not affect how the file is uploaded or loaded. - `part_size` `integer,null` — Preferred size, in bytes, of each part for a large (multi-part) upload. Optional hint — the service clamps it to the allowed part-size range and to the maximum number of parts, and ignores it for small files uploaded with a single `PUT`. Omit to let the service choose. Min: `0` ```json { "uploads": [ { "content_type": "text/csv", "declared_size_bytes": 4096, "filename": "orders.csv" }, { "content_type": "text/csv", "declared_size_bytes": 2048, "filename": "customers.csv" } ] } ``` **Response** `201` — Upload sessions created - `uploads` `UploadSessionResponse`[] — **required** - `finalize_token` `string` — **required**. One-time token that authorizes finalizing this upload. Returned exactly once at create time — store it; it cannot be retrieved again. - `headers` `object` — **required**. Headers you must send verbatim with each `PUT`. Currently always empty; present so a future mode can require signed headers without changing the response shape. - `mode` `string` — **required**. Upload mode: `single` (upload the whole file with one `PUT` to `url`) or `multipart` (upload each part with one `PUT` to the matching entry in `part_urls`). Modeled as a string so additional modes can be added later without breaking clients. - `part_size` `integer,null` — For a `multipart` upload (both known-size and streaming), the size in bytes to split the file into: send bytes `[(i-1) * part_size, i * part_size)` as part *i*, with the last part carrying the remainder. Slice by this value — do **not** divide the file evenly by the number of parts, which can make a non-final part smaller than the 5 MiB minimum for a non-final part (the upload then fails at finalize). Absent for `single` uploads. Min: `0` - `part_urls` `string`[] | `null` — For a known-size `multipart` upload, the per-part URLs in ascending part order: `PUT` your file's part *i* (1-based) to `part_urls[i - 1]` and keep each response's `ETag`, then pass the `{part_number, e_tag}` list to finalize. Absent for `single` uploads, and also absent for a streaming (unknown-size) `multipart` upload — there, mint part URLs on demand via `POST /v1/uploads/{upload_id}/parts`. - `upload_id` `string` — **required**. Identifier for this upload. Pass it to the finalize endpoint and to the managed-table load endpoint once finalized. - `url` `string,null` — The URL to `PUT` the raw file bytes to, for a `single` upload. Short-lived — upload promptly and finalize. Absent for `multipart` uploads (use `part_urls`). ```json { "uploads": [ { "finalize_token": "string", "headers": {}, "mode": "string", "part_size": 0, "part_urls": [ "string" ], "upload_id": "string", "url": "string" } ] } ``` **Errors** | Status | Description | | ------ | ----------- | | `400` | Invalid request (e.g. a file too large, unsupported checksum algorithm) | | `501` | This deployment cannot issue upload URLs | ## Finalize upload `POST /v1/uploads/{upload_id}/finalize` Confirm that a file has been uploaded and make it usable as managed-table contents. Supply the `finalize_token` returned when the session was created, in the `X-Upload-Finalize-Token` header. When you declared a size at create time, the uploaded file's size is validated against it and a mismatch is rejected. An upload created without a declared size is finalized from its uploaded parts; it must be non-empty and is rejected if it exceeds the server's maximum upload size. Finalize is exactly-once: a second finalize of the same upload is rejected. **Path parameters** - `upload_id` `string` — **required**. Upload session ID returned at create time **Headers** - `X-Upload-Finalize-Token` `string` — **required**. One-time finalize token returned when the session was created **Request body** - `parts` `FinalizeUploadPart`[] | `null` — Parts to assemble, for a multi-part upload. Omit for single-`PUT` uploads (the common case). - `e_tag` `string` — **required**. The `ETag` response header returned by that part's `PUT`. - `part_number` `integer` — **required**. The 1-based part number you uploaded this part as. ```json { "parts": [ { "e_tag": "\"9f8c1e5b7a2d4f60b3c8e1a9d7f4b206\"", "part_number": 1 } ] } ``` **Response** `200` — Upload finalized - `content_type` `string,null` - `created_at` `string` — **required** - `size_bytes` `integer` — **required**. The validated size of the uploaded file in bytes. - `status` `string` — **required** - `upload_id` `string` — **required** ```json { "content_type": "string", "created_at": "2026-01-01T00:00:00Z", "size_bytes": 0, "status": "string", "upload_id": "string" } ``` **Errors** | Status | Description | | ------ | ----------- | | `400` | Invalid finalize token, uploaded size mismatch, missing file, or upload not finalizable | | `404` | Upload session not found | ## Mint upload part URLs `POST /v1/uploads/{upload_id}/parts` Get short-lived upload URLs for specific parts of a multi-part upload. This is required for a streaming (unknown-size) upload — created by omitting the declared size — which mints no part URLs up front. It also works for a known-size multi-part upload: use it to re-mint a part whose URL expired before you uploaded that part. Supply the `finalize_token` returned when the session was created, in the `X-Upload-Finalize-Token` header, and the 1-based `part_numbers` you want URLs for. `PUT` each part's bytes to its URL, keep each response's `ETag`, then pass the `{part_number, e_tag}` list to finalize. You may mint parts in batches as you upload, and re-mint a part number whose URL expired before you finished uploading it. **Path parameters** - `upload_id` `string` — **required**. Upload session ID returned at create time **Headers** - `X-Upload-Finalize-Token` `string` — **required**. One-time finalize token returned when the session was created **Request body** - `part_numbers` `integer`[] — **required**. The 1-based part numbers to mint URLs for. Must be non-empty; each number must be between 1 and the maximum number of parts allowed. ```json { "part_numbers": [ 1, 2, 3 ] } ``` **Response** `200` — Minted part URLs - `parts` `MintedUploadPartResponse`[] — **required**. The minted part URLs, in ascending part-number order. `PUT` each part's bytes to its URL and keep the response's `ETag` to pass to finalize. - `part_number` `integer` — **required**. The 1-based part number this URL is for. - `url` `string` — **required**. Short-lived URL to `PUT` this part's bytes to. Keep the response's `ETag` and pass the `{part_number, e_tag}` pair to finalize. ```json { "parts": [ { "part_number": 0, "url": "string" } ] } ``` **Errors** | Status | Description | | ------ | ----------- | | `400` | Invalid finalize token, invalid part numbers, batch too large, or the upload is not a multi-part upload | | `404` | Upload session not found | | `501` | This deployment cannot issue upload URLs | --- # Saved Queries Source: https://www.hotdata.dev/docs/api-reference/saved-queries Site index: https://www.hotdata.dev/llms.txt Save, version, and execute named SQL queries. Each update creates a new version, preserving the full history. Saved queries are automatically classified by category (e.g., aggregation, join, filtered scan) and can be executed by ID. ## List saved queries `GET /v1/queries` **Query parameters** - `limit` `integer` — Maximum number of results - `offset` `integer` — Pagination offset **Response** `200` — List of saved queries - `count` `integer` — **required**. Min: `0` - `has_more` `boolean` — **required** - `limit` `integer` — **required**. Min: `0` - `offset` `integer` — **required**. Min: `0` - `queries` `SavedQuerySummary`[] — **required** - `created_at` `string` — **required** - `description` `string` — **required** - `id` `string` — **required** - `latest_version` `integer` — **required** - `name` `string` — **required** - `tags` `string`[] — **required** - `updated_at` `string` — **required** ```json { "count": 0, "has_more": true, "limit": 0, "offset": 0, "queries": [ { "created_at": "2026-01-01T00:00:00Z", "description": "string", "id": "string", "latest_version": 0, "name": "string", "tags": [ "string" ], "updated_at": "2026-01-01T00:00:00Z" } ] } ``` ## Create saved query `POST /v1/queries` Save a named SQL query. The SQL is stored as version 1 and automatically analyzed for classification metadata (category, table count, predicate/join/aggregation flags). **Request body** - `description` `string,null` - `name` `string` — **required** - `sql` `string` — **required** - `tags` `string`[] | `null` ```json { "description": "Ten highest-spending customers by order total", "name": "top-customers", "sql": "SELECT customer_id, sum(amount) AS total FROM orders GROUP BY customer_id ORDER BY total DESC LIMIT 10", "tags": [ "sales", "weekly" ] } ``` **Response** `201` — Saved query created - `category` `string,null` - `created_at` `string` — **required** - `description` `string` — **required** - `has_aggregation` `boolean,null` - `has_group_by` `boolean,null` - `has_join` `boolean,null` - `has_limit` `boolean,null` - `has_order_by` `boolean,null` - `has_predicate` `boolean,null` - `id` `string` — **required** - `latest_version` `integer` — **required** - `name` `string` — **required** - `num_tables` `integer,null` - `sql` `string` — **required** - `sql_hash` `string` — **required** - `table_size` `string,null` - `tags` `string`[] — **required** - `updated_at` `string` — **required** ```json { "category": "string", "created_at": "2026-01-01T00:00:00Z", "description": "string", "has_aggregation": true, "has_group_by": true, "has_join": true, "has_limit": true, "has_order_by": true, "has_predicate": true, "id": "string", "latest_version": 0, "name": "string", "num_tables": 0, "sql": "string", "sql_hash": "string", "table_size": "string", "tags": [ "string" ], "updated_at": "2026-01-01T00:00:00Z" } ``` **Errors** | Status | Description | | ------ | ----------- | | `400` | Invalid request | ## Get saved query `GET /v1/queries/{id}` **Path parameters** - `id` `string` — **required**. Saved query ID **Response** `200` — Saved query details - `category` `string,null` - `created_at` `string` — **required** - `description` `string` — **required** - `has_aggregation` `boolean,null` - `has_group_by` `boolean,null` - `has_join` `boolean,null` - `has_limit` `boolean,null` - `has_order_by` `boolean,null` - `has_predicate` `boolean,null` - `id` `string` — **required** - `latest_version` `integer` — **required** - `name` `string` — **required** - `num_tables` `integer,null` - `sql` `string` — **required** - `sql_hash` `string` — **required** - `table_size` `string,null` - `tags` `string`[] — **required** - `updated_at` `string` — **required** ```json { "category": "string", "created_at": "2026-01-01T00:00:00Z", "description": "string", "has_aggregation": true, "has_group_by": true, "has_join": true, "has_limit": true, "has_order_by": true, "has_predicate": true, "id": "string", "latest_version": 0, "name": "string", "num_tables": 0, "sql": "string", "sql_hash": "string", "table_size": "string", "tags": [ "string" ], "updated_at": "2026-01-01T00:00:00Z" } ``` **Errors** | Status | Description | | ------ | ----------- | | `404` | Saved query not found | ## Update saved query `PUT /v1/queries/{id}` Update a saved query. If the SQL changes, a new version is created (previous versions are preserved). Name, tags, description, and classification overrides can also be updated. **Path parameters** - `id` `string` — **required**. Saved query ID **Request body** - `category_override` `string,null` — Override the auto-detected category. Send `null` to clear (revert to auto). - `description` `string,null` - `name` `string,null` — Optional new name. When omitted the existing name is preserved. - `sql` `string,null` — Optional new SQL. When omitted the existing SQL is preserved. - `table_size_override` `string,null` — User annotation for table size. Send `null` to clear. - `tags` `string`[] | `null` All fields are optional. Send only the ones you want to set. **Response** `200` — Saved query updated - `category` `string,null` - `created_at` `string` — **required** - `description` `string` — **required** - `has_aggregation` `boolean,null` - `has_group_by` `boolean,null` - `has_join` `boolean,null` - `has_limit` `boolean,null` - `has_order_by` `boolean,null` - `has_predicate` `boolean,null` - `id` `string` — **required** - `latest_version` `integer` — **required** - `name` `string` — **required** - `num_tables` `integer,null` - `sql` `string` — **required** - `sql_hash` `string` — **required** - `table_size` `string,null` - `tags` `string`[] — **required** - `updated_at` `string` — **required** ```json { "category": "string", "created_at": "2026-01-01T00:00:00Z", "description": "string", "has_aggregation": true, "has_group_by": true, "has_join": true, "has_limit": true, "has_order_by": true, "has_predicate": true, "id": "string", "latest_version": 0, "name": "string", "num_tables": 0, "sql": "string", "sql_hash": "string", "table_size": "string", "tags": [ "string" ], "updated_at": "2026-01-01T00:00:00Z" } ``` **Errors** | Status | Description | | ------ | ----------- | | `400` | Invalid request | | `404` | Saved query not found | ## Delete saved query `DELETE /v1/queries/{id}` **Path parameters** - `id` `string` — **required**. Saved query ID **Response** `204` — Saved query deleted **Errors** | Status | Description | | ------ | ----------- | | `404` | Saved query not found | ## Execute saved query `POST /v1/queries/{id}/execute` Execute a saved query, scoped to a database (required `X-Database-Id` header). By default runs the latest version. Optionally specify a version number to execute a previous version. The SQL runs inside the given database scope, the same way POST /v1/query does. Returns the same response format as POST /v1/query. **Path parameters** - `id` `string` — **required**. Saved query ID **Headers** - `X-Database-Id` `string` — **required**. Required. Scope execution to this database (its id). A missing or malformed value is a 400; an unknown database id is a 404. **Request body** - `version` `integer,null` All fields are optional. Send only the ones you want to set. **Response** `200` — Query executed - `columns` `string`[] — **required** - `execution_time_ms` `integer` — **required**. Min: `0` - `nullable` `boolean`[] — **required**. Nullable flags for each column (parallel to columns vec). True if the column allows NULL values, false if NOT NULL. - `preview_row_count` `integer` — **required**. Number of rows in *this* response body. Always present. For a large result this is a bounded preview, not the grand total — see `total_row_count` and `truncated`. - `query_run_id` `string` — **required**. Unique identifier for the query run record (qrun...). - `result_id` `string,null` — Unique identifier for retrieving this result via GET /results/{id}. When non-null, the result is being persisted asynchronously. Null only when the result fit entirely in this response (`truncated: false`) but could not be persisted for later retrieval — see the `warning` field. A `truncated: true` response ALWAYS carries a non-null, resolvable `result_id`: a truncated result that cannot be persisted fails the request with a retryable HTTP 503 (`PERSISTENCE_UNAVAILABLE`, with a `Retry-After` header) rather than returning a partial body with a dead ticket. - `row_count` `integer` — **required**. **Deprecated** — use `preview_row_count` (rows in this body) and `total_row_count` (grand total) instead. Retained as a back-compat alias and always equal to `preview_row_count`; for a truncated result it is the preview count, *not* the grand total — read `total_row_count` for that. Will be removed in a future release once clients migrate. Min: `0` - `rows` `any`[][] — **required**. Array of rows, where each row is an array of column values. Values can be strings, numbers, booleans, or null. - `total_row_count` `integer,null` — Grand total rows in the full result. Present (and equal to `preview_row_count`) when the whole result fit in this response; `null` while a truncated result is still being persisted. When `null`, read the authoritative total from `GET /v1/query-runs/{id}` (`row_count`) or the `X-Total-Row-Count` header on `GET /v1/results/{id}`. - `truncated` `boolean` — **required**. True when `rows` is a bounded preview of a larger result. Fetch the full result via `result_id`. - `warning` `string,null` — Warning message if result persistence could not be initiated. Present only when the full result is returned inline (`truncated: false`) but could not be persisted: `result_id` is then null and the result cannot be re-fetched later, though every row is in this response. A truncated result never carries a warning — if it cannot be persisted the request fails with a retryable HTTP 503 (`PERSISTENCE_UNAVAILABLE`, with a `Retry-After` header) instead. ```json { "columns": [ "string" ], "execution_time_ms": 0, "nullable": [ true ], "preview_row_count": 0, "query_run_id": "string", "result_id": "string", "row_count": 0, "rows": [ [ null ] ], "total_row_count": 0, "truncated": true, "warning": "string" } ``` **Errors** | Status | Description | | ------ | ----------- | | `400` | Invalid request (including a missing X-Database-Id header) | | `404` | Saved query or database not found | ## List saved query versions `GET /v1/queries/{id}/versions` **Path parameters** - `id` `string` — **required**. Saved query ID **Query parameters** - `limit` `integer` — Maximum number of versions - `offset` `integer` — Pagination offset **Response** `200` — List of versions - `count` `integer` — **required**. Min: `0` - `has_more` `boolean` — **required** - `limit` `integer` — **required**. Min: `0` - `offset` `integer` — **required**. Min: `0` - `saved_query_id` `string` — **required** - `versions` `SavedQueryVersionInfo`[] — **required** - `category` `string,null` - `created_at` `string` — **required** - `has_aggregation` `boolean,null` - `has_group_by` `boolean,null` - `has_join` `boolean,null` - `has_limit` `boolean,null` - `has_order_by` `boolean,null` - `has_predicate` `boolean,null` - `num_tables` `integer,null` - `sql` `string` — **required** - `sql_hash` `string` — **required** - `table_size` `string,null` - `version` `integer` — **required** ```json { "count": 0, "has_more": true, "limit": 0, "offset": 0, "saved_query_id": "string", "versions": [ { "category": "string", "created_at": "2026-01-01T00:00:00Z", "has_aggregation": true, "has_group_by": true, "has_join": true, "has_limit": true, "has_order_by": true, "has_predicate": true, "num_tables": 0, "sql": "string", "sql_hash": "string", "table_size": "string", "version": 0 } ] } ``` **Errors** | Status | Description | | ------ | ----------- | | `404` | Saved query not found | --- # Indexes Source: https://www.hotdata.dev/docs/api-reference/indexes Site index: https://www.hotdata.dev/llms.txt Create, list, and delete indexes on cached tables. Supports sorted indexes for range queries and BM25 full-text indexes for keyword search. ## List indexes across tables in a database `GET /v1/indexes` List all indexes in the database identified by the required X-Database-Id header, paginated. Optional filters narrow by connection, schema, table, or index type. **Query parameters** - `connection_id` `string` — Filter to one connection - `schema` `string` — Filter by schema name - `table` `string` — Filter by table name - `index_type` `string` — Filter by index type - `limit` `integer` — Max indexes per page - `cursor` `string` — Pagination cursor **Headers** - `X-Database-Id` `string` — **required**. Database to scope to (required) **Response** `200` — Indexes listed - `count` `integer` — **required**. Min: `0` - `has_more` `boolean` — **required** - `indexes` `IndexEntryResponse`[] — **required** - `limit` `integer` — **required**. Min: `0` - `next_cursor` `string,null` ```json { "count": 0, "has_more": true, "indexes": [ { "columns": [ "string" ], "created_at": "2026-01-01T00:00:00Z", "index_name": "string", "index_type": "string", "metric": "string", "source_column": "string", "status": "ready", "updated_at": "2026-01-01T00:00:00Z", "connection_id": "string", "schema_name": "string", "table_name": "string" } ], "limit": 0, "next_cursor": "string" } ``` **Errors** | Status | Description | | ------ | ----------- | | `400` | Missing X-Database-Id or bad cursor | | `404` | Database not found | | `500` | Internal server error | --- # Embedding Providers Source: https://www.hotdata.dev/docs/api-reference/embedding-providers Site index: https://www.hotdata.dev/llms.txt Manage embedding providers that generate vector embeddings for text columns. Providers can be service-based (e.g., OpenAI) or local. Register a provider, then reference it when creating vector indexes on text columns. ## List embedding providers `GET /v1/embedding-providers` List all registered embedding providers. **Response** `200` — List of embedding providers - `embedding_providers` `EmbeddingProviderResponse`[] — **required** - `config` `any` — **required** - `created_at` `string` — **required** - `has_secret` `boolean` — **required** - `id` `string` — **required** - `name` `string` — **required** - `provider_type` `string` — **required** - `source` `string` — **required**. Provider source: "system" (from config) or "user" (created via API). - `updated_at` `string` — **required** ```json { "embedding_providers": [ { "config": null, "created_at": "2026-01-01T00:00:00Z", "has_secret": true, "id": "string", "name": "string", "provider_type": "string", "source": "string", "updated_at": "2026-01-01T00:00:00Z" } ] } ``` ## Create embedding provider `POST /v1/embedding-providers` Register a new embedding provider that can be used to generate vector embeddings for text columns. Providers can be service-based (e.g., OpenAI) or local. **Request body** - `api_key` `string,null` — Inline API key. If provided, a secret is auto-created and referenced. Cannot be used together with `secret_name`. - `config` `object` — Provider-specific configuration (model name, base URL, dimensions, etc.) - `name` `string` — **required** - `provider_type` `string` — **required**. Provider type: "local" or "service" - `secret_name` `string,null` — Reference an existing secret by name (for service providers). ```json { "config": { "base_url": "https://api.openai.com/v1", "dimensions": 1536, "model": "text-embedding-3-small" }, "name": "openai-text-embedding-3-small", "provider_type": "service", "secret_name": "openai-api-key" } ``` **Response** `201` — Embedding provider created - `config` `any` — **required** - `created_at` `string` — **required** - `id` `string` — **required** - `name` `string` — **required** - `provider_type` `string` — **required** ```json { "config": null, "created_at": "2026-01-01T00:00:00Z", "id": "string", "name": "string", "provider_type": "string" } ``` **Errors** | Status | Description | | ------ | ----------- | | `400` | Invalid request | | `409` | Provider with this name already exists | ## Get embedding provider `GET /v1/embedding-providers/{id}` **Path parameters** - `id` `string` — **required**. Embedding provider ID **Response** `200` — Embedding provider details - `config` `any` — **required** - `created_at` `string` — **required** - `has_secret` `boolean` — **required** - `id` `string` — **required** - `name` `string` — **required** - `provider_type` `string` — **required** - `source` `string` — **required**. Provider source: "system" (from config) or "user" (created via API). - `updated_at` `string` — **required** ```json { "config": null, "created_at": "2026-01-01T00:00:00Z", "has_secret": true, "id": "string", "name": "string", "provider_type": "string", "source": "string", "updated_at": "2026-01-01T00:00:00Z" } ``` **Errors** | Status | Description | | ------ | ----------- | | `404` | Provider not found | ## Update embedding provider `PUT /v1/embedding-providers/{id}` **Path parameters** - `id` `string` — **required**. Embedding provider ID **Request body** - `api_key` `string,null` — Inline API key. If provided, updates (or creates) the auto-managed secret. - `config` `any` - `name` `string,null` - `secret_name` `string,null` — Secret name containing the API key. Pass null to clear. All fields are optional. Send only the ones you want to set. **Response** `200` — Embedding provider updated - `id` `string` — **required** - `name` `string` — **required** - `updated_at` `string` — **required** ```json { "id": "string", "name": "string", "updated_at": "2026-01-01T00:00:00Z" } ``` **Errors** | Status | Description | | ------ | ----------- | | `404` | Provider not found | ## Delete embedding provider `DELETE /v1/embedding-providers/{id}` **Path parameters** - `id` `string` — **required**. Embedding provider ID **Response** `204` — Embedding provider deleted **Errors** | Status | Description | | ------ | ----------- | | `404` | Provider not found | --- # Jobs Source: https://www.hotdata.dev/docs/api-reference/jobs Site index: https://www.hotdata.dev/llms.txt Track background jobs. Jobs are submitted internally by other APIs when async execution is requested. Poll job status by ID or list all jobs. ## List jobs `GET /v1/jobs` List background jobs with optional filters by type and status. **Query parameters** - `job_type` `JobType` — Filter by job type - `status` `string` — Filter by status (comma-separated, e.g. status=pending,running) - `limit` `integer` — Max results (default 50) - `offset` `integer` — Offset for pagination **Response** `200` — List of jobs - `jobs` `JobStatusResponse`[] — **required** - `attempts` `integer` — **required**. Number of execution attempts (including the current one). - `completed_at` `string,null` - `created_at` `string` — **required** - `error_message` `string,null` — Error or warning message. Set when status is `failed` or `partially_succeeded`. - `id` `string` — **required** - `job_type` `JobType` — **required**. Background job types returned by the API. - `result` `null` | `JobResult` - `status` `JobStatus` — **required**. Current status of a background job. ```json { "jobs": [ { "attempts": 0, "completed_at": "2026-01-01T00:00:00Z", "created_at": "2026-01-01T00:00:00Z", "error_message": "string", "id": "string", "job_type": "noop", "result": null, "status": "pending" } ] } ``` ## Get job status `GET /v1/jobs/{id}` Get the current status of a background job. Poll this endpoint to track job progress. **Path parameters** - `id` `string` — **required**. Job ID **Response** `200` — Job status - `attempts` `integer` — **required**. Number of execution attempts (including the current one). - `completed_at` `string,null` - `created_at` `string` — **required** - `error_message` `string,null` — Error or warning message. Set when status is `failed` or `partially_succeeded`. - `id` `string` — **required** - `job_type` `JobType` — **required**. Background job types returned by the API. - `result` `null` | `JobResult` - `status` `JobStatus` — **required**. Current status of a background job. ```json { "attempts": 0, "completed_at": "2026-01-01T00:00:00Z", "created_at": "2026-01-01T00:00:00Z", "error_message": "string", "id": "string", "job_type": "noop", "result": null, "status": "pending" } ``` **Errors** | Status | Description | | ------ | ----------- | | `404` | Job not found | --- # Database context Source: https://www.hotdata.dev/docs/api-reference/database-context Site index: https://www.hotdata.dev/llms.txt Store and retrieve named text or Markdown documents scoped to a specific database. ## List database contexts `GET /v1/databases/{database_id}/context` **Path parameters** - `database_id` `string` — **required**. Database ID **Response** `200` — Contexts - `contexts` `DatabaseContextEntry`[] — **required** - `content` `string` — **required** - `name` `string` — **required** - `updated_at` `string` — **required** ```json { "contexts": [ { "content": "string", "name": "string", "updated_at": "2026-01-01T00:00:00Z" } ] } ``` **Errors** | Status | Description | | ------ | ----------- | | `404` | Database not found | ## Create or update database context `POST /v1/databases/{database_id}/context` Stores a named document (for example Markdown) scoped to a database. Reuses the same name to replace content. **Path parameters** - `database_id` `string` — **required**. Database ID **Request body** - `content` `string` — **required** - `name` `string` — **required**. Upsert key in the catalog. Validated with table-name rules (preserves case): ASCII letter or `_` first; then alphanumeric or `_` only; 1–128 chars; not a SQL reserved word. ```json { "content": "The orders table holds one row per completed purchase. `amount` is in USD cents.", "name": "orders_overview" } ``` **Response** `200` — Context saved - `context` `DatabaseContextEntry` — **required**. One context entry returned by the API. - `content` `string` — **required** - `name` `string` — **required** - `updated_at` `string` — **required** ```json { "context": { "content": "string", "name": "string", "updated_at": "2026-01-01T00:00:00Z" } } ``` **Errors** | Status | Description | | ------ | ----------- | | `400` | Invalid request | | `404` | Database not found | ## Get one database context `GET /v1/databases/{database_id}/context/{name}` **Path parameters** - `database_id` `string` — **required**. Database ID - `name` `string` — **required**. Context key: same character rules as a table name **Response** `200` — Context found - `context` `DatabaseContextEntry` — **required**. One context entry returned by the API. - `content` `string` — **required** - `name` `string` — **required** - `updated_at` `string` — **required** ```json { "context": { "content": "string", "name": "string", "updated_at": "2026-01-01T00:00:00Z" } } ``` **Errors** | Status | Description | | ------ | ----------- | | `400` | Invalid request | | `404` | Database or context not found | ## Delete database context `DELETE /v1/databases/{database_id}/context/{name}` Removes a named context document from a database. **Path parameters** - `database_id` `string` — **required**. Database ID - `name` `string` — **required**. Context key: same character rules as a table name **Response** `204` — Context deleted **Errors** | Status | Description | | ------ | ----------- | | `400` | Invalid request | | `404` | Database or context not found | --- # Databases Source: https://www.hotdata.dev/docs/api-reference/databases Site index: https://www.hotdata.dev/llms.txt Group catalogs into id-addressable databases. Each database auto-creates a `default` catalog and can attach existing connections under an optional alias. A query that carries the `X-Database-Id` header is planned only against that database's catalogs. ## List databases `GET /v1/databases` List databases in the workspace, newest first, one page at a time. When no `limit` is given a default page size is applied, so a single call returns at most one page rather than every database. If the response's `has_more` is true, pass its `next_cursor` value back as the `cursor` query parameter to fetch the next page. Pass `search` to return only databases whose name contains that text (case-insensitive). Pass `batch` with the `batch_id` returned by a bulk-creation call to list only that batch's databases. **Query parameters** - `limit` `integer` — Maximum number of databases to return in this page (1–100). Values outside the range are clamped. - `cursor` `string` — Opaque pagination cursor from a previous response's `next_cursor`. - `search` `string` — Case-insensitive substring filter on the database name. When set, only databases whose name contains this text are returned; paging and newest-first ordering are unchanged. - `batch` `string` — List only the databases belonging to one bulk-creation batch, identified by the `batch_id` that call returned. Bulk-created databases also appear in the unfiltered listing alongside every other database; this narrows the listing to one batch. Paging works the same way, but results are ordered by database id rather than newest-first, because every database in a batch is created at once. **Response** `200` — One page of databases - `count` `integer,null` — Number of databases returned in this page. Min: `0` - `databases` `DatabaseSummary`[] — **required** - `created_at` `string,null` — When the database was created. - `default_catalog` `string` — **required**. Name the database's default catalog answers to inside its query scope. - `default_schema` `string` — **required**. Schema that unqualified table names resolve to inside this database's query scope. `main` unless the database declares a single schema or a `default_schema` was set at create time. - `expires_at` `string,null` - `id` `string` — **required** - `name` `string,null` - `has_more` `boolean,null` — Whether more databases exist beyond this page. - `limit` `integer,null` — Page size applied to this response (after clamping to the maximum). Min: `0` - `next_cursor` `string,null` — Opaque cursor for the next page; present only when `has_more` is true. ```json { "count": 0, "databases": [ { "created_at": "2026-01-01T00:00:00Z", "default_catalog": "string", "default_schema": "string", "expires_at": "2026-01-01T00:00:00Z", "id": "string", "name": "string" } ], "has_more": true, "limit": 0, "next_cursor": "string" } ``` ## Create database `POST /v1/databases` Create a new database (a metadata-only grouping). A managed default catalog is auto-created and addressable inside the database as `default` (or the optional `default_catalog` name), with a `main` schema pre-declared so `default.main.
` works out of the box. The optional `name` is a free-form display label and is not required to be unique. Optional `default_catalog` overrides the name the default catalog answers to; it must be a valid SQL identifier and may not collide with the reserved catalog names `hotdata` or `information_schema`. Optional `schemas` declares additional schemas/tables on the default catalog at create time; declared tables can be loaded via the standard managed-tables-load endpoint targeting `default_connection_id`. Optional `expires_at` sets when the database expires — accepts either an RFC 3339 timestamp or a relative duration suffixed with `h` (hours), `m` (minutes), or `d` (days), e.g. `24h`, `48h`, `90m`, `7d`. When omitted, the database never expires. Expiry is best-effort: the database will not be deleted before `expires_at`, but cleanup may run later than the exact timestamp. **Request body** - `default_catalog` `string,null` — Optional name the database's auto-created default catalog answers to inside its query scope. Must be a valid SQL identifier (`[a-z0-9_]`, not starting with a digit) and may not collide with the reserved catalog names `hotdata` or `information_schema`. Defaults to `default` when omitted, so `default.main.
` keeps working. - `default_schema` `string,null` — Optional schema that unqualified table names resolve to inside this database's query scope. Must be a valid SQL identifier (`[a-z0-9_]`, not starting with a digit). When omitted, a database that declares exactly one schema adopts that schema as its default; otherwise unqualified names resolve to `main`. Fully-qualified names (`..
`) are unaffected, and a per-query `default_schema` still takes precedence. - `expires_at` `string,null` — When this database expires. Accepts either an RFC 3339 timestamp (e.g. `"2026-06-01T00:00:00Z"`) or a relative duration suffixed with `h` (hours), `m` (minutes), or `d` (days) — for example `"24h"`, `"48h"`, or `"7d"`. Omitted (or empty) means the database never expires. Expiry is best-effort: the database will not be deleted before `expires_at`, but cleanup may run later than the exact timestamp. - `name` `string,null` — Optional free-form display label (for UIs/CLIs). Not unique. Not an identifier — databases are always addressed by `id`. Accepts the legacy `description` key as an alias so clients that predate the rename keep populating this field. - `schemas` `DatabaseDefaultSchemaDecl`[] — Optional schemas/tables to declare on the database's auto-created default catalog. Tables declared here can be loaded via the standard managed-table load endpoint targeting `default_connection_id`. Omitted or empty means the default catalog starts empty. - `name` `string` — **required** - `tables` `DatabaseDefaultTableDecl`[] - `key` `string`[] — Columns that uniquely identify a row, enabling the key-based load modes (`delete`, `update`, `upsert`) on this table: those loads match rows by these columns' values. Omit (the default) to declare no key; the table can still be loaded with `replace` and `append`, but key-based modes are then rejected. - `name` `string` — **required** - `partition_by` `TablePartitionKey`[] — Partition keys for this table, applied in order. Omit for no partitioning. Declared when the table is created and fixed thereafter. - `sorted_by` `TableSortKey`[] — Sort keys for this table, applied in order. Omit for no sort order. Declared when the table is created and fixed thereafter. ```json { "schemas": [ { "name": "sales" } ] } ``` **Response** `201` — Database created - `default_catalog` `string` — **required**. Name the database's default catalog answers to inside its query scope (`default` unless overridden at create time). - `default_connection_id` `string` — **required**. Internal id of the connection that backs this database's `default` catalog. Workspace-level connection endpoints (list, get, health, delete, cache purge) refuse to act on this id — it is exposed only for the managed-tables load endpoint (`POST /v1/connections/{id}/schemas/{s}/tables/{t}/loads`) so callers can load data into tables declared at database-create time. Addressing it directly in SQL is not the recommended path — use `default` inside an `X-Database-Id` scope instead. - `default_schema` `string` — **required**. Schema that unqualified table names resolve to inside this database's query scope. `main` unless the database declares a single schema or a `default_schema` was set at create time. - `expires_at` `string,null` — When this database expires. - `id` `string` — **required** - `name` `string,null` ```json { "default_catalog": "string", "default_connection_id": "string", "default_schema": "string", "expires_at": "2026-01-01T00:00:00Z", "id": "string", "name": "string" } ``` **Errors** | Status | Description | | ------ | ----------- | | `400` | Invalid request | | `500` | Internal server error | ## Create many databases at once `POST /v1/databases/bulk` Create many databases from one template in a single request. The databases are created in the background: the response returns immediately with a batch and a job to poll. The databases are not returned inline. List the ones a batch created with `GET /databases?batch=`; they also appear in the normal database listing alongside every other database. Each database gets a default catalog and schema. Declare tables on all of them by passing `schemas`, in the same shape a single create accepts — a batch of 10,000 declaring one table yields 10,000 databases that each hold that table and are ready to load, with no follow-up call per database. Omit `schemas` and the databases are created empty. Either way, load data into them exactly as you would a database created individually. **Request body** - `count` `integer` — **required**. How many databases to create. Min: `1`. Max: `10000` - `default_catalog` `string,null` — Name the default catalog answers to inside each database, as on a single create. Defaults to `default`. - `default_schema` `string,null` — Schema that unqualified table names resolve to inside each database. - `expires_at` `string,null` — When the created databases expire. Accepts an RFC 3339 timestamp or a relative duration such as `24h`, `90m`, or `7d`. - `idempotency_key` `string,null` — Repeat this value to retry a request safely. A retry carrying a key that was already used returns the original batch — the same `batch_id` and the same databases — instead of creating a second set. The key identifies the request, not its contents: reusing a key with a different `count` or template returns the original batch unchanged rather than reporting a mismatch. Use a fresh key per distinct request. - `name_template` `string,null` — Optional display-label pattern for each database. `{index}` is replaced with the database's zero-based position — for example `tenant-{index}` produces `tenant-0`, `tenant-1`, and so on. Labels are not identifiers and are not required to be unique. - `schemas` `DatabaseDefaultSchemaDecl`[] — Schemas and tables to declare on every database in the batch, in the same shape a single create accepts. The declaration applies identically to each database, so a batch of 10,000 declaring one table yields 10,000 databases that each hold that table and are ready to load — with no follow-up call per database. Omitted or empty means each database starts with no tables. - `name` `string` — **required** - `tables` `DatabaseDefaultTableDecl`[] - `key` `string`[] — Columns that uniquely identify a row, enabling the key-based load modes (`delete`, `update`, `upsert`) on this table: those loads match rows by these columns' values. Omit (the default) to declare no key; the table can still be loaded with `replace` and `append`, but key-based modes are then rejected. - `name` `string` — **required** - `partition_by` `TablePartitionKey`[] — Partition keys for this table, applied in order. Omit for no partitioning. Declared when the table is created and fixed thereafter. - `sorted_by` `TableSortKey`[] — Sort keys for this table, applied in order. Omit for no sort order. Declared when the table is created and fixed thereafter. ```json { "count": 100, "default_catalog": "default", "default_schema": "main", "expires_at": "7d", "idempotency_key": "3f6b1c1e-9a2b-4a5f-9a1e-2f0d6a7c8b91", "name_template": "tenant-{index}", "schemas": [ { "name": "sales" } ] } ``` **Response** `202` — Batch accepted and filling in the background - `batch_id` `string` — **required** - `cancel_requested` `boolean` — **required**. True once stopping has been requested. Databases already created are kept; only further creation stops. - `count` `integer` — **required**. How many databases the batch was asked to create. - `created_count` `integer` — **required**. How many exist so far. Advances as the batch fills. - `expires_at` `string,null` - `job_id` `string,null` — Job filling this batch. Poll it for status. - `status_url` `string,null` ```json { "batch_id": "string", "cancel_requested": true, "count": 0, "created_count": 0, "expires_at": "2026-01-01T00:00:00Z", "job_id": "string", "status_url": "string" } ``` **Errors** | Status | Description | | ------ | ----------- | | `400` | Invalid count, template, or expiry | | `409` | A batch with this idempotency key is already running | ## Get a database batch `GET /v1/databases/bulk/{batch_id}` Fetch a batch by id: how many databases were requested and how many exist so far. Poll this to follow progress. **Path parameters** - `batch_id` `string` — **required**. Batch ID **Response** `200` — The batch - `batch_id` `string` — **required** - `cancel_requested` `boolean` — **required**. True once stopping has been requested. Databases already created are kept; only further creation stops. - `count` `integer` — **required**. How many databases the batch was asked to create. - `created_count` `integer` — **required**. How many exist so far. Advances as the batch fills. - `expires_at` `string,null` - `job_id` `string,null` — Job filling this batch. Poll it for status. - `status_url` `string,null` ```json { "batch_id": "string", "cancel_requested": true, "count": 0, "created_count": 0, "expires_at": "2026-01-01T00:00:00Z", "job_id": "string", "status_url": "string" } ``` **Errors** | Status | Description | | ------ | ----------- | | `404` | Batch not found | ## Delete a database batch `DELETE /v1/databases/bulk/{batch_id}` Stop a batch that is still filling and delete the databases it created, then the batch itself. Only batches whose databases hold no data can be removed this way. Tables that were declared but never loaded do not prevent it, so a batch created with `schemas` stays deletable. If any database in the batch has had data loaded into it, the request is rejected and those databases must be deleted one at a time — removing a database that holds data is per-database work that cannot be batched. **Path parameters** - `batch_id` `string` — **required**. Batch ID **Response** `200` — Batch deleted - `batch_id` `string` — **required** - `deleted_count` `integer` — **required**. How many databases were removed. ```json { "batch_id": "string", "deleted_count": 0 } ``` **Errors** | Status | Description | | ------ | ----------- | | `404` | Batch not found | | `409` | Some of the batch's databases hold loaded data | ## Count databases `GET /v1/databases/count` Return the total number of databases in the workspace. This is the whole-workspace total, not a page size: the `count` field on the listing reports how many rows that one page returned, so totalling a workspace from `GET /v1/databases` means walking every page. Pass `search` to count only databases whose name contains that text (case-insensitive), or `batch` with the `batch_id` returned by a bulk-creation call to count only that batch's databases. The filters mean exactly what they mean on the listing, so a count and a listing given the same filters describe the same set. **Query parameters** - `search` `string` — Case-insensitive substring filter on the database name. When set, only databases whose name contains this text are counted. - `batch` `string` — Count only the databases belonging to one bulk-creation batch, identified by the `batch_id` that call returned. **Response** `200` — Total databases in the workspace - `total` `integer` — **required**. Total databases matching the filters, across all pages. ```json { "total": 0 } ``` ## Get database `GET /v1/databases/{database_id}` Fetch a database by id. The `name` field is a display label only; it is not accepted as an identifier here. **Path parameters** - `database_id` `string` — **required**. Database ID **Response** `200` — Database details - `attachments` `DatabaseAttachmentInfo`[] — **required** - `alias` `string,null` — Alias under which this catalog is reachable inside the database. When `None`, the catalog is reachable by its original connection name. - `connection_id` `string` — **required** - `created_at` `string,null` — When the database was created. - `default_catalog` `string` — **required**. Name the database's default catalog answers to inside its query scope (`default` unless overridden at create time). - `default_connection_id` `string` — **required** - `default_schema` `string` — **required**. Schema that unqualified table names resolve to inside this database's query scope. `main` unless the database declares a single schema or a `default_schema` was set at create time. - `expires_at` `string,null` — When this database expires. - `id` `string` — **required** - `name` `string,null` ```json { "attachments": [ { "alias": "string", "connection_id": "string" } ], "created_at": "2026-01-01T00:00:00Z", "default_catalog": "string", "default_connection_id": "string", "default_schema": "string", "expires_at": "2026-01-01T00:00:00Z", "id": "string", "name": "string" } ``` **Errors** | Status | Description | | ------ | ----------- | | `404` | Database not found | ## Delete database `DELETE /v1/databases/{database_id}` Delete a database and its auto-created default catalog. Attached catalogs are detached (their underlying connections are not deleted). **Path parameters** - `database_id` `string` — **required**. Database ID **Response** `204` — Database deleted **Errors** | Status | Description | | ------ | ----------- | | `404` | Database not found | ## Attach catalog to database `POST /v1/databases/{database_id}/catalogs` Attach an existing connection (catalog) to a database with an optional alias. Inside the database the catalog is reachable as the alias (when set) or its original name. **Path parameters** - `database_id` `string` — **required**. Database ID **Request body** - `alias` `string,null` — Optional alias under which this catalog is reachable inside the database. When omitted, it is reachable by the connection's name. - `connection_id` `string` — **required** ```json { "alias": "warehouse", "connection_id": "connk9p34y6n3wd25rq4f5zr37e3p3" } ``` **Response** `204` — Catalog attached **Errors** | Status | Description | | ------ | ----------- | | `400` | Invalid request | | `404` | Database or connection not found | | `409` | Catalog already attached or alias collides with an existing attachment | ## Detach catalog from database `DELETE /v1/databases/{database_id}/catalogs/{connection_id}` **Path parameters** - `database_id` `string` — **required**. Database ID - `connection_id` `string` — **required**. Connection ID **Response** `204` — Catalog detached **Errors** | Status | Description | | ------ | ----------- | | `400` | Cannot detach a database's own default catalog | | `404` | Database or attachment not found | ## Fork database `POST /v1/databases/{database_id}/fork` Create a new database that is an independent fork of an existing one. The fork has its own default catalog and contains the same schemas, tables, and data as the source; the source is left unchanged. External catalogs attached to the source are re-attached to the fork. Optional `name` sets the fork's display label (defaults to the source's). Optional `expires_at` sets when the fork expires — accepts an RFC 3339 timestamp or a relative duration suffixed with `h` (hours), `m` (minutes), or `d` (days), e.g. `24h`, `90m`, `7d`. When omitted, a still-future expiry on the source is carried over; otherwise the fork never expires. Any indexes on the source's tables are not carried over. **Path parameters** - `database_id` `string` — **required**. Source database ID **Request body** - `expires_at` `string,null` — When the fork expires. Accepts either an RFC 3339 timestamp (e.g. `"2026-06-01T00:00:00Z"`) or a relative duration suffixed with `h` (hours), `m` (minutes), or `d` (days) — for example `"24h"` or `"7d"`. When omitted, a still-future expiry on the source is carried over; otherwise the fork never expires. - `name` `string,null` — Optional display label for the fork. When omitted, the source database's name (if any) is carried over. All fields are optional. Send only the ones you want to set. **Response** `201` — Database forked - `default_catalog` `string` — **required**. Name the database's default catalog answers to inside its query scope (`default` unless overridden at create time). - `default_connection_id` `string` — **required**. Internal id of the connection that backs this database's `default` catalog. Workspace-level connection endpoints (list, get, health, delete, cache purge) refuse to act on this id — it is exposed only for the managed-tables load endpoint (`POST /v1/connections/{id}/schemas/{s}/tables/{t}/loads`) so callers can load data into tables declared at database-create time. Addressing it directly in SQL is not the recommended path — use `default` inside an `X-Database-Id` scope instead. - `default_schema` `string` — **required**. Schema that unqualified table names resolve to inside this database's query scope. `main` unless the database declares a single schema or a `default_schema` was set at create time. - `expires_at` `string,null` — When this database expires. - `id` `string` — **required** - `name` `string,null` ```json { "default_catalog": "string", "default_connection_id": "string", "default_schema": "string", "expires_at": "2026-01-01T00:00:00Z", "id": "string", "name": "string" } ``` **Errors** | Status | Description | | ------ | ----------- | | `400` | The source database can't be forked as-is (for example, one of its tables has rows that were individually deleted or updated) | | `404` | Source database not found | ## Add schema to database default catalog `POST /v1/databases/{database_id}/schemas` Declare a new schema (and optionally its tables) on the database's auto-created default catalog after creation. The schema becomes reachable inside the database scope (e.g. `default..
` and `information_schema.schemata`) without the caller naming the database's default connection. Identifiers are normalized to lowercase. **Path parameters** - `database_id` `string` — **required**. Database ID **Request body** - `name` `string` — **required** - `tables` `AddManagedTableDecl`[] - `key` `string`[] — Columns that uniquely identify a row, enabling the key-based load modes (`delete`, `update`, `upsert`) on this table: those loads match rows by these columns' values. Omit (the default) to declare no key; the table can still be loaded with `replace` and `append`, but key-based modes are then rejected. - `name` `string` — **required** - `partition_by` `TablePartitionKey`[] — Partition keys for this table, applied in order. Omit for no partitioning. Declared when the table is created and fixed thereafter. - `column` `string` — **required**. Column the key reads. - `transform` `string` — **required**. How the value is derived from the column. One of `identity` (the column value itself), `year`, `month`, `day`, or `hour`. - `sorted_by` `TableSortKey`[] — Sort keys for this table, applied in order. Omit for no sort order. Declared when the table is created and fixed thereafter. - `column` `string` — **required** - `direction` `string,null` — `asc` (the default) or `desc`. Null when the table was declared without an explicit direction for this key. - `nulls` `string,null` — Where nulls are placed: `first` or `last`. Defaults to the SQL default for the chosen direction. Null when the table was declared without an explicit placement for this key. ```json { "name": "sales", "tables": [ { "key": [ "order_id" ], "name": "orders" } ] } ``` **Response** `201` — Schema added - `connection_id` `string` — **required**. Connection backing the catalog the schema was added to. For a database default catalog this is the database's `default_connection_id`. - `schema` `string` — **required** - `tables` `string`[] — **required** ```json { "connection_id": "string", "schema": "string", "tables": [ "string" ] } ``` **Errors** | Status | Description | | ------ | ----------- | | `400` | Invalid identifier | | `404` | Database not found | | `409` | Schema already exists | ## Add table to database default catalog `POST /v1/databases/{database_id}/schemas/{schema}/tables` Declare a new table on an existing schema of the database's default catalog after creation. The table is added empty (declared-but-unloaded) and can be populated via the managed-table load endpoint targeting the default connection. Identifiers are normalized to lowercase. **Path parameters** - `database_id` `string` — **required**. Database ID - `schema` `string` — **required**. Schema name **Request body** - `key` `string`[] — Columns that uniquely identify a row, enabling the key-based load modes (`delete`, `update`, `upsert`) on this table: those loads match rows by these columns' values. Omit (the default) to declare no key; the table can still be loaded with `replace` and `append`, but key-based modes are then rejected. - `name` `string` — **required** - `partition_by` `TablePartitionKey`[] — Partition keys for this table, applied in order. Omit for no partitioning. Declared when the table is created and fixed thereafter. - `column` `string` — **required**. Column the key reads. - `transform` `string` — **required**. How the value is derived from the column. One of `identity` (the column value itself), `year`, `month`, `day`, or `hour`. - `sorted_by` `TableSortKey`[] — Sort keys for this table, applied in order. Omit for no sort order. Declared when the table is created and fixed thereafter. - `column` `string` — **required** - `direction` `string,null` — `asc` (the default) or `desc`. Null when the table was declared without an explicit direction for this key. - `nulls` `string,null` — Where nulls are placed: `first` or `last`. Defaults to the SQL default for the chosen direction. Null when the table was declared without an explicit placement for this key. ```json { "key": [ "order_id" ], "name": "orders", "partition_by": [ { "column": "created_at", "transform": "day" } ], "sorted_by": [ { "column": "created_at", "direction": "asc", "nulls": "last" } ] } ``` **Response** `201` — Table added - `connection_id` `string` — **required** - `schema` `string` — **required** - `table` `string` — **required** ```json { "connection_id": "string", "schema": "string", "table": "string" } ``` **Errors** | Status | Description | | ------ | ----------- | | `400` | Invalid identifier | | `404` | Database or schema not found | | `409` | Table already exists | ## Load database table from inline data, upload, or query result `POST /v1/databases/{database_id}/schemas/{schema}/tables/{table}/loads` Publish data as the new contents of a table on the database's default catalog, from one of three sources — provide exactly one. The database-scoped equivalent of the connection-scoped managed-table load — addressed by `database_id`, so no `default_connection_id` is needed. With `data`, CSV text is sent inline in this request, up to 2 MiB; column types are detected from the data unless `columns` declares them, and a larger payload is rejected with 413 and the error code `INLINE_DATA_TOO_LARGE`, at which point the data should be uploaded and loaded by `upload_id` instead. With `upload_id`, a previously-uploaded file is published: CSV, JSON, and Parquet are supported; the format is auto-detected or set via `format`. With `result_id`, a persisted query result is copied into the table, so the table keeps its data even after the result expires. If the target table (or its schema) has not been declared yet, it is created automatically as part of the load — declaring tables up front is optional. `mode` selects how the data is applied: `replace` overwrites the table's contents, `append` inserts the new rows on top of the existing data. Concurrent loads against the same upload return 409. For an upload or inline data, set `async` to run the load in the background and get back a job ID to poll; add `async_after_ms` to wait briefly for it to finish before falling back to a job ID. A `result_id` load runs synchronously. **Path parameters** - `database_id` `string` — **required**. Database ID - `schema` `string` — **required**. Schema name - `table` `string` — **required**. Table name **Request body** - `async` `boolean` — When true, run the load as a background job and return a job ID to poll instead of blocking until it finishes. Recommended for large uploads, which can take longer than an HTTP request should stay open. Default: `false` - `async_after_ms` `integer,null` — If set (requires `async` = true), wait up to this many milliseconds for the load to finish: if it completes in time the full result is returned (200), otherwise a 202 with a job ID to poll. Must be between 1000 and the server maximum; a value out of that range, or set without `async` = true, is rejected with 400. Min: `1000` - `columns` `object,null` — Column types for inline `data`, keyed by column name. Optional — types are detected from the data when omitted. Each value is either a type name (`"VARCHAR"`, `"BIGINT"`, `"DECIMAL(10,2)"`) or an object carrying explicit parameters (`{"type": "DECIMAL", "precision": 10, "scale": 2}`). Supported types: `VARCHAR`, `TEXT`, `STRING`, `CHAR`, `BOOLEAN`, `TINYINT`, `SMALLINT`, `INTEGER`, `BIGINT`, `UTINYINT`, `USMALLINT`, `UINTEGER`, `UBIGINT`, `REAL`, `FLOAT`, `DOUBLE`, `DECIMAL`, `NUMERIC`, `DATE`, `TIME`, `TIMESTAMP`, `TIMESTAMPTZ`, `BINARY`, `BLOB`, `UUID`, and `JSON`. When given, it must name every column in the CSV header and no others. Only valid together with `data`. - `data` `string,null` — The data to load, sent inline in this request instead of being uploaded first — the quickest way to get a small table in. CSV text with a header row, up to 2 MiB. Larger payloads are rejected with `413` and the error code `INLINE_DATA_TOO_LARGE`; upload the file (see `POST /v1/uploads`) and load it by `upload_id` instead. Column types are detected from the data unless `columns` declares them. Provide exactly one of this, `upload_id`, or `result_id`. - `format` `string,null` — File format of the upload: `"csv"`, `"json"`, or `"parquet"`. Optional — when omitted, the format is auto-detected from the upload's `Content-Type` and, failing that, from the file contents. Provide it explicitly to override detection or when the contents are ambiguous. `"json"` expects newline-delimited JSON (one object per line), not a JSON array. With inline `data` the only accepted value is `"csv"` (the default); upload the file and load it by `upload_id` for any other format. Not valid with `result_id` — query results are always parquet. - `idempotency_key` `string,null` — A key of your own that makes this load safe to retry. Send the same key again — after a timeout, a dropped connection, or any answer you did not receive — and the load runs at most once: the retry returns the original result instead of loading the rows a second time. Generate the key before the first attempt (a UUID is a good choice) and reuse it for every retry of that same data. Use a new key for the next batch: sending different data under a key already used returns `409`, and loads nothing. A retry sent while the first attempt is still running also returns `409`, because the table is busy with it; wait and send the same key again. A `409` never means the data was loaded twice — under one key it is loaded once or not at all. Only valid with inline `data`. A load from `upload_id` is already safe to retry — re-send the same `upload_id`. Keys are at most 255 characters. - `key` `string`[] | `null` — Key columns identifying rows for `"delete"`, `"update"`, and `"upsert"` loads — the columns whose values decide which existing row an incoming row removes, updates, or replaces. Omit to use the key the table was created with. Keep the key consistent across loads of the same table: changing it re-targets which rows are matched. Ignored for `"replace"` and `"append"`. - `mode` `string` — **required**. How the data is applied: `"replace"` overwrites the table's contents, `"append"` inserts the new rows on top of the existing data. - `result_id` `string,null` — ID of a persisted query result (see `GET /v1/results/{result_id}`) to publish as the table's contents. The result is copied into the table, so the table keeps its data even after the result expires. A result can be loaded into any number of tables. Provide exactly one of this, `upload_id`, or `data`. - `upload_id` `string,null` — ID of a previously-staged upload (see `POST /v1/uploads`). The upload is claimed atomically; concurrent loads against the same `upload_id` return 409. Provide exactly one of this, `result_id`, or `data`. ```json { "data": "order_id,customer_id,amount\n1001,42,1999\n1002,7,4550\n", "mode": "replace" } ``` **Response** `200` — Table loaded - `arrow_schema_json` `string` — **required**. Schema of the loaded table, as JSON. - `connection_id` `string` — **required** - `row_count` `integer` — **required**. Total number of rows in the table after the load. Min: `0` - `schema_name` `string` — **required** - `table_name` `string` — **required** ```json { "arrow_schema_json": "string", "connection_id": "string", "row_count": 0, "schema_name": "string", "table_name": "string" } ``` **Response** `202` — Load accepted and running in the background; poll the returned job for status and result - `id` `string` — **required**. Job ID for status polling. - `status` `JobStatus` — **required**. Current status of a background job. - `status_url` `string` — **required**. URL to poll for job status. ```json { "id": "string", "status": "pending", "status_url": "string" } ``` **Errors** | Status | Description | | ------ | ----------- | | `400` | Invalid request (bad mode, none or several of `upload_id`/`result_id`/`data`, `format` combined with `result_id`, `columns` without `data`, a non-CSV inline `format`, unparseable inline data, invalid identifier, bad parquet, or the result failed to compute) | | `404` | Database, upload, or result not found, or the table was deleted | | `409` | Upload already consumed or in flight, the result is still being computed, or the incoming data changes a column's type incompatibly (only widening to a larger compatible type can be applied automatically); the existing data is unchanged and remains queryable | | `413` | Inline `data` is over the 2 MiB limit (error code `INLINE_DATA_TOO_LARGE`); upload the data and load it by `upload_id` instead | --- # Usage Source: https://www.hotdata.dev/docs/api-reference/usage Site index: https://www.hotdata.dev/llms.txt ## Get workspace usage snapshot `GET /v1/usage` Return aggregated bytes scanned and current storage size for a billing period. Pass `since` as the start of your billing period so the totals line up with your invoice rather than the calendar month. **Query parameters** - `since` `string` — Billing period start (ISO-8601). Defaults to the start of the current UTC calendar month when omitted. **Response** `200` — Workspace usage snapshot - `bytes_scanned` `integer` — **required**. Sum of `bytes_scanned` across all completed/failed query runs since `since`. Null bytes (queries that touched no row data) contribute 0. - `query_count` `integer` — **required**. Number of query runs (succeeded + failed) since `since`. - `since` `string` — **required**. The period start used for this response (echoed back for the caller to verify). - `storage_bytes` `integer` — **required**. The workspace's current stored-data footprint in bytes, measured at request time: managed-database data, plus un-consumed uploads, connection caches, and search-index artifacts. - `storage_captured_at` `string,null` — When `storage_bytes` was measured (the time this response was produced). ```json { "bytes_scanned": 0, "query_count": 0, "since": "2026-01-01T00:00:00Z", "storage_bytes": 0, "storage_captured_at": "2026-01-01T00:00:00Z" } ``` --- # Query every backend through one SQL surface Source: https://www.hotdata.dev/use-cases/unified-sql Site index: https://www.hotdata.dev/llms.txt Most teams don't want a different tool for every backend. Register your sources once and query databases, SaaS apps, and uploads from the same SQL surface without bouncing between consoles or dialects. ## How it works ### Step 1: Query Postgres through `hotdata query` Here `postgres.public.orders`. GitHub-backed tables use the same `connection.schema.table` pattern (confirm names with `hotdata databases tables list`). **Claude** ``` Our app database and GitHub are already linked here. Give me every open order with just order id and dollar total, up to 100 rows. ``` **CLI** ```bash hotdata query "SELECT order_id, total FROM postgres.public.orders WHERE status = 'open' LIMIT 100" --workspace-id ``` **Python SDK** ```python import hotdata query_api = hotdata.QueryApi(api_client) query_api.query( hotdata.QueryRequest( sql=( "SELECT order_id, total FROM postgres.public.orders " "WHERE status = 'open' LIMIT 100" ), ), ) ``` Heavy or routed queries sometimes return async metadata instead. The engine would rather give you a run id than hold the connection until everything finishes. ### Step 2: Query managed-database tables like connection tables Managed-database tables sit in the same SQL namespace as `postgres` and `github` (for example `default.public.*`). **Claude** ``` Pull the first 50 rows from my partner_accounts table with every column included. ``` **CLI** ```bash hotdata query "SELECT * FROM default.public.partner_accounts LIMIT 50" ``` **Python SDK** ```python import hotdata query_api = hotdata.QueryApi(api_client) query_api.query( hotdata.QueryRequest(sql="SELECT * FROM default.public.partner_accounts LIMIT 50"), ) ``` ### Step 3: Mirror everything in the CLI for debugging **Claude** ``` Run the open-orders check again, then show the 50 most recently updated GitHub issues (number, title, state). After that, list recent query activity so support can match what was tried. ``` **CLI** ```bash hotdata query "SELECT order_id, total FROM postgres.public.orders WHERE status = 'open' LIMIT 100" --workspace-id hotdata query "SELECT number, title, state FROM github.github.issues ORDER BY updated_at DESC LIMIT 50" --workspace-id hotdata databases queries list ``` **Python SDK** ```python import hotdata query_api = hotdata.QueryApi(api_client) query_api.query( hotdata.QueryRequest( sql=( "SELECT order_id, total FROM postgres.public.orders " "WHERE status = 'open' LIMIT 100" ), ), ) query_api.query( hotdata.QueryRequest( sql=( "SELECT number, title, state FROM github.github.issues " "ORDER BY updated_at DESC LIMIT 50" ), ), ) runs = hotdata.QueryRunsApi(api_client) runs.list_query_runs(limit=20) ``` ### Step 4: Guardrails with database context Optional shared docs (like **`DATAMODEL`**) spell out joins and naming beyond what the raw catalog shows. See [Database context](/docs/api-reference/database-context). ## Who uses this - Teams shipping one SQL interface to multiple apps or dashboards. - Products that run agent tools against the same SQL surface across several backends. - Internal platforms that want one workspace and one governed path for queries. --- # Load parquet into managed databases and query it Source: https://www.hotdata.dev/use-cases/dataset-uploads Site index: https://www.hotdata.dev/llms.txt Parquet dumps and other one-offs usually need to live next to warehouse tables. Load them into a managed database, inspect the shape, then join them with everything else without standing up another pipeline. ## How it works ### Step 1: Upload from a file **Claude** ``` Create a managed database with catalog q2, declare a targets table, and load targets.parquet into it. What do I type in SQL to read it? ``` **CLI** ```bash hotdata databases create --catalog q2 --table targets hotdata databases load --catalog q2 --table targets --file ./targets.parquet ``` **Python SDK** ```python import hotdata db_api = hotdata.DatabasesApi(api_client) db = db_api.create_database( hotdata.CreateDatabaseRequest( name="Q2 targets", schemas=[hotdata.DatabaseDefaultSchemaDecl( name="public", tables=[hotdata.DatabaseDefaultTableDecl(name="targets")] )] ) ) uploads = hotdata.UploadsApi(api_client) with open("targets.parquet", "rb") as f: up = uploads.upload_file(f.read()) conn_api = hotdata.ConnectionsApi(api_client) conn_api.load_managed_table( db.default_connection_id, "public", "targets", hotdata.LoadManagedTableRequest(mode="replace", upload_id=up.upload_id), ) # Query as: SELECT * FROM default.public.targets ``` ### Step 2: Load more files into the same database **Claude** ``` Load accounts.parquet as an accounts table into the same q2 database. ``` **CLI** ```bash hotdata databases load --catalog q2 --table accounts --file ./accounts.parquet ``` **Python SDK** ```python import hotdata uploads = hotdata.UploadsApi(api_client) with open("accounts.parquet", "rb") as f: up = uploads.upload_file(f.read()) conn_api = hotdata.ConnectionsApi(api_client) conn_api.load_managed_table( "", "public", "accounts", hotdata.LoadManagedTableRequest(mode="replace", upload_id=up.upload_id), ) ``` ### Step 3: Inspect before you join **Claude** ``` Show my managed databases and list the tables inside q2. ``` **CLI** ```bash hotdata databases list hotdata databases tables list ``` **Python SDK** ```python import hotdata db_api = hotdata.DatabasesApi(api_client) db_api.list_databases() db_api.get_database("") ``` ### Step 4: Query like any other table **Claude** ``` Match my Q2 targets to sales accounts on account id. Give me account_id and plan for up to 50 pairs. ``` **CLI** ```bash hotdata query "SELECT t.account_id, s.plan FROM q2.public.targets t JOIN sales.public.accounts s ON s.id = t.account_id LIMIT 50" ``` **Python SDK** ```python import hotdata query_api = hotdata.QueryApi(api_client) query_api.query( hotdata.QueryRequest( sql=( "SELECT t.account_id, s.plan FROM default.public.targets t " "JOIN sales.public.accounts s ON s.id = t.account_id LIMIT 50" ), ), ) ``` ## Who uses this - Ops bringing in recurring CSV extracts without touching warehouse DDL. - Analytics freezing cohort definitions as stable managed-database tables. - Automation that reads uploaded tables by name instead of parsing attachments ad hoc. --- # Spin up ephemeral databases for scratch work Source: https://www.hotdata.dev/use-cases/sql-sandboxes Site index: https://www.hotdata.dev/llms.txt Exploratory SQL is easier when you're not leaning on shared prod or a crowded staging schema. Create a throwaway managed database with an expiry, load and query in isolation, then let it disappear — no cleanup, no stepping on teammates or shared quotas. ## How it works ### Step 1: Create an ephemeral database **Claude** ``` Spin up a throwaway database called scratch that auto-expires in 24 hours, so my experiments don’t linger. ``` **CLI** ```bash hotdata databases create --catalog scratch --name "Staging validation" --expires-at 24h ``` **Python SDK** ```python import hotdata db_api = hotdata.DatabasesApi(api_client) db_api.create_database( hotdata.CreateDatabaseRequest( name="Staging validation", default_catalog="scratch", expires_at="24h", ) ) ``` ### Step 2: Explore and query **Claude** ``` Show the columns on our product analytics data source, then give me event counts by week for the last 24 weeks from the events table. ``` **CLI** ```bash hotdata databases tables list hotdata query "SELECT date_trunc('week', created_at), COUNT(*) FROM analytics.public.events GROUP BY 1 LIMIT 24" ``` **Python SDK** ```python import hotdata query_api = hotdata.QueryApi(api_client) query_api.query( hotdata.QueryRequest( sql=( "SELECT date_trunc('week', created_at), COUNT(*) " "FROM analytics.public.events GROUP BY 1 LIMIT 24" ), ), ) ``` ### Step 3: Load a one-off file into it **Claude** ``` Load segment.parquet as a high_value_users table in my scratch database. What table name do I query afterward? ``` **CLI** ```bash hotdata databases load --catalog scratch --table high_value_users --file ./segment.parquet ``` **Python SDK** ```python import hotdata uploads = hotdata.UploadsApi(api_client) with open("segment.parquet", "rb") as f: up = uploads.upload_file(f.read()) conn_api = hotdata.ConnectionsApi(api_client) conn_api.load_managed_table( "", "public", "high_value_users", hotdata.LoadManagedTableRequest(mode="replace", upload_id=up.upload_id), ) # Query as: SELECT * FROM scratch.public.high_value_users ``` ### Step 4: Point the rest of your session at the sandbox `databases use` makes a database the default target for later commands, so one-off exploration lands in the throwaway instead of a shared database. It takes the database id that `create` prints. Combined with `--expires-at` at create time, the sandbox cleans itself up. **Claude** ``` Create a throwaway database for a one-off exploration, make it my default so my next queries land there, and let it auto-expire. ``` **CLI** ```bash hotdata databases create --catalog exploration --name "API exploration" --expires-at 24h # databases use takes the id printed by create hotdata databases use hotdata query "SELECT 1" ``` **Python SDK** ```python import hotdata db_api = hotdata.DatabasesApi(api_client) db = db_api.create_database( hotdata.CreateDatabaseRequest( name="API exploration", default_catalog="exploration", expires_at="24h", ) ) query_api = hotdata.QueryApi(api_client) query_api.query(hotdata.QueryRequest(sql="SELECT 1"), x_database_id=db.id) ``` ### Step 5: Isolate messy uploads in their own throwaway database **Claude** ``` I’m loading a rough partner extract for a one-off QA join. Keep it in a throwaway database that expires so it doesn’t linger after we’re done. ``` **CLI** ```bash hotdata databases create --catalog qa --expires-at 24h hotdata databases load --catalog qa --table partner_extract --file ./partner_extract.parquet ``` **Python SDK** ```python import hotdata db_api = hotdata.DatabasesApi(api_client) db = db_api.create_database( hotdata.CreateDatabaseRequest( name="Partner extract QA", default_catalog="qa", expires_at="24h", ) ) uploads = hotdata.UploadsApi(api_client) with open("partner_extract.parquet", "rb") as f: up = uploads.upload_file(f.read()) conn_api = hotdata.ConnectionsApi(api_client) conn_api.load_managed_table( db.default_connection_id, "public", "partner_extract", hotdata.LoadManagedTableRequest(mode="replace", upload_id=up.upload_id), ) ``` ### Step 6: Tear it down when you're done Delete early, or just let `expires_at` clean it up for you. **Claude** ``` I’m done with the scratch database — tear it down now instead of waiting for it to expire. ``` **CLI** ```bash hotdata databases remove scratch ``` **Python SDK** ```python import hotdata db_api = hotdata.DatabasesApi(api_client) db_api.delete_database("") ``` ## Who uses this - Analysts prototyping joins before promoting logic to scheduled jobs. - Agents that need real compute on temporary data, kept in a database that expires after the task. - Platform folks who don't want half the company experimenting on one shared warehouse role. --- # Run semantic search on embeddings from SQL Source: https://www.hotdata.dev/use-cases/vector-search Site index: https://www.hotdata.dev/llms.txt Semantic search doesn't have to mean another database. Keep embeddings in the workspace, build vector indexes, and run nearest-neighbor queries beside ordinary SQL. Similarity scores and relational filters stay in one place. ## How it works ### Step 1: Pick a table and text or vector column **Claude** ``` Open the help articles table in our support KB and show the columns. I need to pick which field to use for meaning-based search. ``` **CLI** ```bash hotdata databases tables show support.public.help_articles ``` **Python SDK** ```python import hotdata api = hotdata.InformationSchemaApi(api_client) api.information_schema( connection_id="support", var_schema="public", table="help_articles", include_columns=True, ) ``` ### Step 2: Create a vector index **Claude** ``` Index article bodies for semantic (meaning-based) search, using cosine to compare how close two pieces of text are. ``` **Python SDK** ```python import hotdata indexes = hotdata.IndexesApi(api_client) indexes.create_index( "support", "public", "help_articles", hotdata.CreateIndexRequest( index_name="help_body_semantic", index_type="vector", columns=["body"], metric="cosine", ), ) ``` With **`--async`**, the build runs as a background **job**. Embedding and index materialization can take a while, so poll **`hotdata jobs`** until it's done. ### Step 3: Search from the CLI with an embedding model **Claude** ``` Ask in plain language how refunds work and return the ~10 help articles that best match the question by meaning. ``` **CLI** ```bash hotdata search "how do I get a refund on my subscription" \ --index help_body_semantic \ --select id,title,body \ --limit 10 ``` **Python SDK** ```python import hotdata query_api = hotdata.QueryApi(api_client) query_api.query( hotdata.QueryRequest( sql=( "SELECT id, title, body, _distance FROM vector_search(" "'support.public.help_articles', 'body', " "'how do I get a refund on my subscription', 50) LIMIT 10" ), ), ) ``` ### Step 4: Same retrieval in SQL **Claude** ``` Do that kind of search in SQL: closest matches first, with id, title, and how far each row is from the question. Limit to 10 rows. ``` **CLI** ```bash hotdata query "SELECT id, title, body, _distance FROM vector_search('support.public.help_articles', 'body', 'onboarding and first invoice', 50) ORDER BY _distance ASC LIMIT 10" ``` **Python SDK** ```python import hotdata query_api = hotdata.QueryApi(api_client) query_api.query( hotdata.QueryRequest( sql=( "SELECT id, title, body, _distance FROM vector_search(" "'support.public.help_articles', 'body', " "'onboarding and first invoice', 50) ORDER BY _distance ASC LIMIT 10" ), ), ) ``` When your **`ORDER BY … LIMIT`** lines up with HNSW and the index is ready, the planner can skip scanning the whole table. If not, distances still compute. They are just slower without the shortcut. ## Who uses this - Support and docs search ranked by meaning, not just keywords. - Product catalogs mixing similarity with inventory, pricing, or eligibility filters. - Agents that need nearest-neighbor rows with distance metadata on the same path as other SQL. --- # Upload and filter GeoParquet using spatial SQL Source: https://www.hotdata.dev/use-cases/spatial-parquet Site index: https://www.hotdata.dev/llms.txt GeoParquet usually shows up next to everything else you're querying. Load the file into a managed database and run spatial filters in the same engine without a sidecar GIS stack. ## How it works ### Step 1: Load Parquet into a managed database **Claude** ``` Create a managed database with catalog geo, declare a store_locations table, and load stores.parquet into it. What name do I use when I query it? ``` **CLI** ```bash hotdata databases create --catalog geo --table store_locations hotdata databases load --catalog geo --table store_locations --file ./stores.parquet ``` **Python SDK** ```python import hotdata db_api = hotdata.DatabasesApi(api_client) db = db_api.create_database( hotdata.CreateDatabaseRequest( name="Store locations", schemas=[hotdata.DatabaseDefaultSchemaDecl( name="public", tables=[hotdata.DatabaseDefaultTableDecl(name="store_locations")] )] ) ) uploads = hotdata.UploadsApi(api_client) with open("stores.parquet", "rb") as f: up = uploads.upload_file(f.read()) conn_api = hotdata.ConnectionsApi(api_client) conn_api.load_managed_table( db.default_connection_id, "public", "store_locations", hotdata.LoadManagedTableRequest(mode="replace", upload_id=up.upload_id), ) # Query as: SELECT * FROM default.public.store_locations ``` ### Step 2: Confirm lon/lat (or WKT) columns **Claude** ``` Show my managed databases and list the tables inside q2. ``` **CLI** ```bash hotdata databases list hotdata databases tables list ``` **Python SDK** ```python import hotdata db_api = hotdata.DatabasesApi(api_client) db_api.list_databases() db_api.get_database("") ``` ### Step 3: Radius filter from a reference point **Claude** ``` Using my store locations upload, which stores fall roughly near downtown San Francisco? Give id and name, up to 50. ``` **CLI** ```bash hotdata query "SELECT id, name FROM geo.public.store_locations WHERE ST_Distance(ST_MakePoint(lon, lat), ST_GeomFromText('POINT(-122.4194 37.7749)')) < 5000 LIMIT 50" ``` **Python SDK** ```python import hotdata query_api = hotdata.QueryApi(api_client) query_api.query( hotdata.QueryRequest( sql=( "SELECT id, name FROM default.public.store_locations WHERE " "ST_Distance(ST_MakePoint(lon, lat), " "ST_GeomFromText('POINT(-122.4194 37.7749)')) < 5000 LIMIT 50" ), ), ) ``` See [SQL Reference: Geospatial](/docs/sql#geospatial-functions) for units and other predicates (`ST_Within`, `ST_Intersects`). ## Who uses this - Field ops blending telemetry files with fixed reference points. - Growth or marketing scoring leads from uploaded geographic lists. - Anyone joining warehouse tables to geo extracts without switching dialects. --- # Control spend and long-running hybrid queries Source: https://www.hotdata.dev/use-cases/query-spend Site index: https://www.hotdata.dev/llms.txt Some queries finish in milliseconds; others chew through huge tables. Async execution, saved results, background refreshes, and a single place to watch jobs mean you're not guessing what's still running or re-paying for the same read twice. ## How it works ### Step 1: Prefer async when integrations might lag **Claude** ``` How many events did we get today in the huge events table? If the answer isn’t ready yet, show me how to check when the count is done. ``` **CLI** ```bash hotdata query "SELECT COUNT(*) FROM huge.public.events WHERE dt = CURRENT_DATE" # If the CLI prints query_run_id → poll: hotdata query status qr_abc123 ``` **Python SDK** ```python import hotdata query_api = hotdata.QueryApi(api_client) runs = hotdata.QueryRunsApi(api_client) submitted = query_api.query( hotdata.QueryRequest( sql="SELECT COUNT(*) FROM huge.public.events WHERE dt = CURRENT_DATE", var_async=True, ), ) runs.get_query_run(submitted.query_run_id) ``` When waiting synchronously isn't worth it, the CLI hands back a **`query_run_id`**. Poll status instead of blocking on the terminal. ### Step 2: Reuse stored results instead of re-running **Claude** ``` Show my last 5 query runs, then let me read the stored rows from one of them without re-running the same SQL. ``` **CLI** ```bash hotdata databases queries list --limit 5 hotdata databases results get rslt_xyz789 ``` **Python SDK** ```python import hotdata runs = hotdata.QueryRunsApi(api_client) runs.list_query_runs(limit=5) results = hotdata.ResultsApi(api_client) results.get_result("rslt_xyz789") ``` ### Step 3: Tune connection refresh, not blind rescans **Claude** ``` Our shared finance Snowflake is out of date. Refresh it so the table list here matches the warehouse again. ``` **Python SDK** ```python import hotdata refresh = hotdata.RefreshApi(api_client) refresh.refresh(hotdata.RefreshRequest(connection_id="snowflake")) ``` A schema refresh reconciles the catalog in the background. Give it a moment before you trust **`hotdata databases tables list`** for brand-new upstream DDL. ### Step 4: Watch jobs that balloon latency **Claude** ``` What’s running in the background right now? Then show me full detail for one of those runs. ``` **CLI** ```bash hotdata jobs list --status running hotdata jobs job_123 ``` **Python SDK** ```python import hotdata jobs = hotdata.JobsApi(api_client) jobs.list_jobs(status="running") jobs.get_job("job_123") ``` **`jobs list`** shows what's still running: long index builds, refreshes, whatever's in flight. **`jobs `** tells you when it's finished. ## Who uses this - Finance or ops tying spend and latency back to query-run metadata. - ML or eval pipelines firing lots of short reads against shared warehouses. Reuse results instead of repeating work. - Anyone who'd rather poll or reuse than block on long-running hybrid queries. --- # Find slow SQL in history and add indexes for your filters Source: https://www.hotdata.dev/use-cases/index-tuning Site index: https://www.hotdata.dev/llms.txt Dashboards and tools usually hammer the same patterns. Pull execution time and SQL from run history, look at the tables involved, and add indexes that match how people actually query, not how you guessed they would. ## How it works ### Step 1: Mine query runs for slow or frequent SQL **Claude** ``` Show my last 25 queries and how long each took. I’m trying to spot the slow ones. ``` **CLI** ```bash hotdata databases queries list --limit 25 ``` **Python SDK** ```python import hotdata runs = hotdata.QueryRunsApi(api_client) runs.list_query_runs(limit=25) ``` Each run stores **`execution_time_ms`** and **`sql_text`** so you can sort by what's actually slow. ### Step 2: Align with the physical model **Claude** ``` Our slow reports filter orders by status and when they were created. Show the structure of the orders table on our commerce database. ``` **CLI** ```bash hotdata databases tables show commerce.public.orders ``` **Python SDK** ```python import hotdata api = hotdata.InformationSchemaApi(api_client) api.information_schema( connection_id="commerce", var_schema="public", table="orders", include_columns=True, ) ``` ### Step 3: Create indexes that match common access paths **Claude** ``` Speed up those filters: add an index on status and created_at for orders. If the build can happen in the background, use that. ``` **Python SDK** ```python import hotdata indexes = hotdata.IndexesApi(api_client) indexes.create_index( "commerce", "public", "orders", hotdata.CreateIndexRequest( index_name="orders_status_created_idx", index_type="sorted", columns=["status", "created_at"], ), ) ``` With **`--async`**, index creation returns a **job**. Poll **`hotdata jobs`** until it's done before you expect the faster path. ## Who uses this - Platform engineers connecting production query patterns to concrete index changes. - Analytics owners tuning tables that BI and agents hit hard. - Teams whose agent-generated SQL volume keeps climbing and needs guardrails. --- # Publish database context for agents Source: https://www.hotdata.dev/use-cases/agent-context Site index: https://www.hotdata.dev/llms.txt Agents need structure, but dumping full schemas into every prompt doesn't scale. Database context stores things like your data model once; queries pick it up automatically. List what's there, push when the join graph changes, cross-check against live tables, and keep throwaway exploration in local files instead of promoting half-baked ideas to shared docs. ## How it works ### Step 1: See what already exists **Claude** ``` What shared docs are stored for this database? If there’s a data model, show the whole thing. ``` **CLI** ```bash hotdata databases context list hotdata databases context show DATAMODEL ``` **Python SDK** ```python import hotdata ctx = hotdata.DatabaseContextApi(api_client) ctx.list_database_contexts("") ctx.get_database_context("", "DATAMODEL") ``` ### Step 2: Edit locally, push when the join graph changes **Claude** ``` Pull the data model down so I can edit it on disk, then publish my edits back to the database. ``` **CLI** ```bash hotdata databases context pull DATAMODEL # optional: creates ./DATAMODEL.md # edit ./DATAMODEL.md (entities, keys, naming) hotdata databases context push DATAMODEL ``` **Python SDK** ```python import hotdata ctx = hotdata.DatabaseContextApi(api_client) body = open("DATAMODEL.md", encoding="utf-8").read() ctx.upsert_database_context( "", hotdata.UpsertDatabaseContextRequest(name="DATAMODEL", content=body), ) ``` ### Step 3: Pair with live schema **`hotdata databases tables`** is ground truth for columns. Context explains relationships and intent (`context:DATAMODEL`). Context names follow the same rules as SQL identifiers. **Claude** ``` I want to compare our written data model to what’s live in the catalog. Show a slice of current tables and then show the DATAMODEL context so I can see definitions next to real columns. ``` **CLI** ```bash hotdata databases tables list --limit 30 hotdata databases context show DATAMODEL ``` **Python SDK** ```python import hotdata api = hotdata.InformationSchemaApi(api_client) api.information_schema(limit=30) ctx = hotdata.DatabaseContextApi(api_client) ctx.get_database_context("", "DATAMODEL") ``` ### Step 4: Keep scratch local, publish what's stable Keep throwaway exploration in **local files** on disk. When something should guide the whole team, not just today's thread, publish it into **DATAMODEL** (or another stem) with `context push`. See [Database context](/docs/api-reference/database-context). **Claude** ``` Keep my quick exploration note in a local file, then publish my updated DATAMODEL from disk so scratch stays local and the team gets the stable definitions. ``` **CLI** ```bash # Scratch stays on disk — not pushed to the database printf '%s\n' '## Exploration' '- Hypothesis: orders join to customers on customer_id' > NOTES.md hotdata databases context push DATAMODEL ``` **Python SDK** ```python import hotdata # Scratch stays on disk — not published to the database with open("NOTES.md", "w", encoding="utf-8") as f: f.write("## Exploration\n- Hypothesis: orders join to customers on customer_id\n") ctx = hotdata.DatabaseContextApi(api_client) body = open("DATAMODEL.md", encoding="utf-8").read() ctx.upsert_database_context( "", hotdata.UpsertDatabaseContextRequest(name="DATAMODEL", content=body), ) ``` ## Who uses this - Teams shipping agents against one shared semantic description. - Engineers maintaining join docs next to what catalog discovery turns up. - Anyone keeping glossary or policy text beside technical mappings. --- # Build shared context with your team Source: https://www.hotdata.dev/use-cases/shared-context Site index: https://www.hotdata.dev/llms.txt Useful context usually splits between Git and hallway conversations. Publish structured docs to a managed database (relationships, ownership, naming) and update them as the team's picture of the data sharpens. ## How it works ### Step 1: Publish the shared map **Claude** ``` Generate the shared data model file, then publish it so the whole team shares one map of tables and joins. ``` **CLI** ```bash hotdata databases context push DATAMODEL ``` **Python SDK** ```python import hotdata ctx = hotdata.DatabaseContextApi(api_client) body = open("DATAMODEL.md", encoding="utf-8").read() ctx.upsert_database_context( "", hotdata.UpsertDatabaseContextRequest(name="DATAMODEL", content=body), ) ``` ### Step 2: Add workspace-specific context for the team **Claude** ``` We keep internal notes on who owns metrics and naming rules. Publish those next to the shared data model so teammates see both. ``` **CLI** ```bash hotdata databases context push TEAM_CONTEXT ``` **Python SDK** ```python import hotdata ctx = hotdata.DatabaseContextApi(api_client) body = open("TEAM_CONTEXT.md", encoding="utf-8").read() ctx.upsert_database_context( "", hotdata.UpsertDatabaseContextRequest(name="TEAM_CONTEXT", content=body), ) ``` ### Step 3: Confirm both documents are live **Claude** ``` List what’s published, then show the data model and the team notes so I can confirm both are there. ``` **CLI** ```bash hotdata databases context list hotdata databases context show DATAMODEL hotdata databases context show TEAM_CONTEXT ``` **Python SDK** ```python import hotdata ctx = hotdata.DatabaseContextApi(api_client) ctx.list_database_contexts("") ctx.get_database_context("", "DATAMODEL") ctx.get_database_context("", "TEAM_CONTEXT") ``` ### Step 4: Create, update, or remove a shared document **Push** creates or overwrites a stem from `./.md`. **Pull → edit → push** updates in place. **Delete** drops that stem from the database when a doc is retired (use the Python client or `DELETE /v1/databases/{database_id}/context/{name}`; there isn’t a `context delete` subcommand in the CLI yet). **Claude** ``` Publish a RUNBOOK context for the team, revise it after our naming workshop, then delete RUNBOOK when we fold it into DATAMODEL. ``` **CLI** ```bash # Create: write ./RUNBOOK.md then push (creates or replaces that stem) printf '%s\n' '# Runbook' '- Revenue metrics owned by Finance Analytics.' > RUNBOOK.md hotdata databases context push RUNBOOK # Update: pull, edit on disk, push again hotdata databases context pull RUNBOOK # … edit ./RUNBOOK.md … hotdata databases context push RUNBOOK # Delete: no context delete subcommand yet; call the API (same auth as the CLI) curl -sS -X DELETE "\${HOTDATA_API_URL}/v1/databases/\${DATABASE_ID}/context/RUNBOOK" \ -H "Authorization: Bearer \${HOTDATA_API_KEY}" \ -H "X-Workspace-Id: \${HOTDATA_WORKSPACE}" ``` **Python SDK** ```python import hotdata ctx = hotdata.DatabaseContextApi(api_client) ctx.upsert_database_context( "", hotdata.UpsertDatabaseContextRequest( name="RUNBOOK", content="""# Runbook - Revenue metrics owned by Finance Analytics. """, ), ) body = open("RUNBOOK.md", encoding="utf-8").read() ctx.upsert_database_context( "", hotdata.UpsertDatabaseContextRequest(name="RUNBOOK", content=body), ) ctx.delete_database_context("", "RUNBOOK") ``` ## Who uses this - Platform leads owning a central model while individual workspaces layer local rules. - Product analytics documenting metric ownership at workspace scope. - Teams piping both schema detail and policy context into agents or codegen. --- # Create full-text indexes and run keyword search from SQL Source: https://www.hotdata.dev/use-cases/text-search Site index: https://www.hotdata.dev/llms.txt Keyword search usually means another index and another service. Built-in full-text indexing keeps ranked keyword queries in the workspace (the same engine as the rest of your SQL). ## How it works ### Step 1: Pick a table and text column **Claude** ``` For our support knowledge base, open the help articles table and list its columns. I need to know which one holds the article text. ``` **CLI** ```bash hotdata databases tables show support.public.help_articles ``` **Python SDK** ```python import hotdata api = hotdata.InformationSchemaApi(api_client) api.information_schema( connection_id="support", var_schema="public", table="help_articles", include_columns=True, ) ``` ### Step 2: Create a full-text index **Claude** ``` Turn on keyword search for the body field of those help articles so we can match phrases quickly. ``` **Python SDK** ```python import hotdata indexes = hotdata.IndexesApi(api_client) indexes.create_index( "support", "public", "help_articles", hotdata.CreateIndexRequest( index_name="help_body_bm25", index_type="bm25", columns=["body"], ), ) ``` With **`--async`**, BM25 index creation runs as a background **job**. Poll until it's finished before you count on that index for **`search`**. ### Step 3: Keyword search from the CLI **Claude** ``` Find help articles that talk about billing refunds (about 10 results), and include id, title, and full body text. ``` **CLI** ```bash hotdata search "billing refund" \ --index help_body_bm25 \ --select id,title,body \ --limit 10 ``` **Python SDK** ```python import hotdata query_api = hotdata.QueryApi(api_client) query_api.query( hotdata.QueryRequest( sql=( "SELECT id, title, body FROM bm25_search(" "'support.public.help_articles', 'body', 'billing refund', 50) " "LIMIT 10" ), ), ) ``` ### Step 4: Same search in SQL (for apps and agents) **Claude** ``` Run the same kind of search in SQL: strongest matches first, with id, title, body, and a relevance score. Return 10 rows. ``` **CLI** ```bash hotdata query "SELECT id, title, body, score FROM bm25_search('support.public.help_articles', 'body', 'billing refund', 50) ORDER BY score DESC LIMIT 10" ``` **Python SDK** ```python import hotdata query_api = hotdata.QueryApi(api_client) query_api.query( hotdata.QueryRequest( sql=( "SELECT id, title, body, score FROM bm25_search(" "'support.public.help_articles', 'body', 'billing refund', 50) " "ORDER BY score DESC LIMIT 10" ), ), ) ``` ## Who uses this - Support knowledge bases with huge article corpora. - Internal apps that used to rely on `LIKE` and crossed fingers. - Agents that want ranked rows and relevance metadata on the same path as other SQL. --- # Explore tables and columns before you write SQL Source: https://www.hotdata.dev/use-cases/table-catalog Site index: https://www.hotdata.dev/llms.txt Different databases usually mean different tools. Here you list connections, inspect tables and columns, and refresh metadata after upstream changes. The workflow stays the same whether you're on Postgres, Snowflake, or something else. ## How it works ### Step 1: Pick the connection **Claude** ``` What data sources are connected in this workspace, and what short name should I use for each when I run commands? ``` **CLI** ```bash hotdata ingest sources list ``` **Python SDK** ```python import hotdata connections = hotdata.ConnectionsApi(api_client) connections.list_connections() ``` ### Step 2: Inspect tables and columns **Claude** ``` On our finance database, list tables whose names start with “revenue.” For each one I need every column: its name, type, and whether blank values are allowed. ``` **CLI** ```bash hotdata databases tables list --table revenue% --schema public ``` **Python SDK** ```python import hotdata api = hotdata.InformationSchemaApi(api_client) api.information_schema( connection_id="finance", table="revenue%", include_columns=True, ) ``` ### Step 3: After the database schema changes **Claude** ``` Finance changed their database structure. Refresh my catalog here so what I see matches what’s actually there now. ``` **Python SDK** ```python import hotdata refresh = hotdata.RefreshApi(api_client) refresh.refresh(hotdata.RefreshRequest(connection_id="finance")) ``` A schema refresh reconciles the catalog in the background. Let it settle before **`hotdata databases tables list`** reflects brand-new DDL. ### Step 4: Page through a wide catalog **Claude** ``` Give me a browseable snapshot: up to 50 tables from finance with full column details so I can skim what exists. ``` **CLI** ```bash hotdata databases tables list --limit 50 ``` **Python SDK** ```python import hotdata api = hotdata.InformationSchemaApi(api_client) api.information_schema( connection_id="finance", include_columns=True, limit=50, ) ``` ## Who uses this - Anyone writing SQL or filters who wants names and types confirmed first. - Engineers building table pickers on top of workspace catalog output. - Teams checking renames before dashboards ship. - Agent tooling that should only touch columns the catalog actually lists. --- # Benchmarking Concurrency on Hotdata, Snowflake and BigQuery Source: https://www.hotdata.dev/blog/benchmarking-concurrency-on-hotdata-snowflake-and-bigquery Published: Jul 16, 2026 Author: Divya Ranganathan Site index: https://www.hotdata.dev/llms.txt For AI applications, concurrency is one of the most important performance metrics. Agents tend to make many requests at the same time, and the slowest requests impact the entire workflow. We are benchmarking Hotdata across several workloads, including TPC-H and ClickBench, with particular attention to how these systems perform as the number of concurrent requests increases. Each point represents the sum of every query's best-of-three median execution time. Hotdata remains relatively flat across the tested range, Snowflake's response times increase once concurrency passes 10 streams while BigQuery scales just as flat as Hotdata but slower per-query overall. ## Hotdata and Snowflake Sum of every query's best-of-3 median execution time, at each concurrency level. | Concurrent streams | Hotdata | Snowflake | | --- | --- | --- | | 1 | 2.61s | 2.88s | | 5 | 2.59s | 2.90s | | 10 | 2.90s | 3.19s | | 20 | 2.79s | 4.31s | | 30 | 2.90s | 7.25s | | 50 | 2.73s | 9.26s | The individual TPC-H queries show the same overall pattern: nearly every query remains stable on Hotdata as concurrency increases, while Snowflake shows increasing latency. The appendix below includes the complete per-query breakdown. ## Hotdata and BigQuery BigQuery shows a third pattern. Its execution time is also flat across concurrency, but the flat line sits about 4 times higher than Hotdata's. Summed across the same six concurrency levels, Hotdata holds 2.59–2.90s while BigQuery holds 11.27–12.96s, with no growth trend in either direction as concurrency increases. The same 19-query clean set against BigQuery, summed per concurrency level. | Concurrent streams | Hotdata | BigQuery | BigQuery ÷ Hotdata | | --- | --- | --- | --- | | 1 | 2.61s | 12.63s | 4.84× | | 5 | 2.59s | 11.33s | 4.37× | | 10 | 2.90s | 11.27s | 3.89× | | 20 | 2.79s | 12.96s | 4.65× | | 30 | 2.90s | 11.36s | 3.92× | | 50 | 2.73s | 11.40s | 4.18× | Any increase in query latency can turn a short interaction into a prolonged multi-step wait. ## Sizing and cost implications The benchmark also shows the difference between Hotdata's base account and Snowflake's warehouse-based scaling model. Hotdata sustained a 50-fold increase in concurrent streams on the same base account, while Snowflake delivered similar throughput through 10 streams before it plateaued. Snowflake supports additional concurrency by increasing warehouse capacity and hourly cost, while Hotdata remained on the base account at every concurrency level tested. For agent workloads, the system must maintain predictable latency and throughput as many independent requests arrive in parallel. The same 19-query set for Snowflake at five warehouse sizes, against Hotdata on its base account. | Concurrent streams | Hotdata | Snowflake X-Small | Snowflake Small | Snowflake Medium | Snowflake Large | Snowflake X-Large | | --- | --- | --- | --- | --- | --- | --- | | 1 | 2.61s | 2.88s | 2.88s | 2.69s | 2.91s | 3.05s | | 5 | 2.59s | 2.90s | 3.20s | 2.75s | 2.83s | 2.95s | | 10 | 2.90s | 3.19s | 3.47s | 2.92s | 2.92s | 3.21s | | 20 | 2.79s | 4.31s | 5.44s | 3.56s | 3.37s | 3.49s | | 30 | 2.90s | 7.25s | 7.55s | 4.46s | 3.95s | 3.44s | | 50 | 2.73s | 9.26s | 8.04s | 5.58s | 4.55s | 4.14s | ## Conclusion Agents explore, retrieve, validate, and reason through thousands of data interactions in parallel. As more agents run concurrently, databases need to keep response times fast and predictable without requiring compute to scale linearly. Agent workloads also shift the performance bottleneck from individual query speed to sustained concurrency. As more agents run simultaneously, the ability to maintain predictable, low-latency responses under load becomes just as important as raw query performance. ### Appendix: methodology notes > Snowflake reports compilation, queueing, and execution separately, while Hotdata's execution path performs planning in-process. The Snowflake results use `EXECUTION_TIME` and exclude compilation and queueing. Result caching was disabled for both systems so repeated executions measured query processing rather than cached responses. ### Appendix: every query's own trend The aggregate results above hide substantial variation between queries. Each chart below shows one TPC-H query across the same six concurrency levels using an independent y-axis, since query durations range from tens of milliseconds to several seconds. The 21 queries plotted, at 1, 5, 10, 20, 30, 50 concurrent streams. Only the shape of each is published, not the durations behind it. - Q1: Pricing summary report - Q2: Minimum cost supplier - Q3: Shipping priority - Q4: Order priority checking - Q5: Local supplier volume - Q6: Forecasting revenue change - Q7: Volume shipping - Q8: National market share - Q9: Product type profit measure - Q10: Returned item reporting - Q11: Important stock identification - Q12: Shipping modes & order priority - Q13: Customer distribution - Q14: Promotion effect - Q15: Top supplier - Q16: Parts/supplier relationship - Q17: Small-quantity-order revenue - Q19: Discounted revenue - Q20: Potential part promotion - Q21: Suppliers who kept orders waiting - Q22: Global sales opportunity --- # ClickBench 10M: Hotdata vs Neon Source: https://www.hotdata.dev/blog/clickbench-10m-hotdata-vs-neon-no-indexes Published: Jun 30, 2026 Author: Divya Ranganathan Site index: https://www.hotdata.dev/llms.txt We ran the [ClickBench](https://benchmark.clickhouse.com/) benchmark across 43 queries and 10 million rows (1.5 GB), comparing Hotdata to Neon. I know it may not seem like a fair comparison to benchmark a transactional database against a columnar database. However, we've found that developers are increasingly using PostgreSQL as the primary data layer for agents because of its flexibility and ecosystem. While Postgres is great as the primary application database, when the workload shifts toward high-concurrency, low-latency search and analytical queries, a purpose-built engine like Hotdata is a better fit. This benchmark is intended to evaluate that increasingly common deployment pattern rather than compare the databases in their traditional use cases. - **Methodology:** Best of 3 server-side execution times - **Hotdata:** `execution_time_ms` from JSON API response - **Neon:** `EXPLAIN (ANALYZE, TIMING)` Execution Time via direct psql ClickBench on a 10M-row hits table, no indexes on either system. Best of 3 runs. | Q | Query pattern | Hotdata | Neon (PostgreSQL) | Speedup | | --- | --- | --- | --- | --- | | Q01 | COUNT(*) full scan | 40ms | 1.20s | 30.0× | | Q02 | COUNT WHERE filter | 58ms | 1.22s | 21.0× | | Q03 | SUM + COUNT + AVG | 154ms | 1.28s | 8.3× | | Q04 | AVG(UserID) | 555ms | 1.10s | 2.0× | | Q05 | COUNT DISTINCT users | 589ms | 7.26s | 12.3× | | Q06 | COUNT DISTINCT phrases | 394ms | 4.14s | 10.5× | | Q07 | MIN / MAX date | 49ms | 1.19s | 24.2× | | Q08 | GROUP BY adv engine | 60ms | 1.26s | 21.1× | | Q09 | Top regions by users | 615ms | 4.01s | 6.5× | | Q10 | Regions × multi-agg | 128ms | 4.59s | 35.9× | | Q11 | Top phone models | 111ms | 1.11s | 10.0× | | Q12 | Phone + model combos | 114ms | 1.29s | 11.3× | | Q13 | Top search phrases | 536ms | 2.63s | 4.9× | | Q14 | Phrases by distinct users | 669ms | 2.42s | 3.6× | | Q15 | Engine × phrase combos | 495ms | 2.47s | 5.0× | | Q16 | Top users by hit count | 79ms | 3.62s | 45.8× | | Q17 | Users × phrases ordered | 644ms | 3.69s | 5.7× | | Q18 | Users × phrases unordered | 427ms | 1.83s | 4.3× | | Q19 | Per-minute user × phrase | 819ms | 6.69s | 8.2× | | Q20 | Point lookup by UserID | 49ms | 798ms | 16.3× | | Q21 | URL LIKE %google% | 334ms | 1.09s | 3.3× | | Q22 | Google URLs + phrases | 398ms | 1.25s | 3.2× | | Q23 | Title LIKE Google | 776ms | 1.97s | 2.5× | | Q24 | SELECT * google URLs | 416ms | 1.11s | 2.7× | | Q25 | Search phrases by time | 124ms | 1.06s | 8.5× | | Q26 | Phrases alphabetical | 100ms | 1.20s | 12.0× | | Q27 | Phrases × time multi-sort | 136ms | 1.09s | 8.0× | | Q28 | Avg URL length by counter | 309ms | 1.80s | 5.8× | | Q29 | Regex domain extraction | 1.26s | 15.6s | 12.4× | | Q30 | 90-column SUM (wide) | 94ms | 3.31s | 35.2× | | Q31 | Engine × IP filtered agg | 150ms | 7.64s | 50.9× | | Q32 | WatchID × IP filtered agg | 169ms | 8.54s | 50.5× | | Q33 | WatchID × IP full-table agg | 184ms | 27.9s | 151.7× | | Q34 | Top URLs by count | 307ms | 8.32s | 27.1× | | Q35 | URL + literal GROUP BY | 305ms | 8.99s | 29.5× | | Q36 | IP arithmetic GROUP BY | 90ms | 2.68s | 29.8× | | Q37 | CounterID 62 — July views | 109ms | 1.74s | 16.0× | | Q38 | CounterID 62 — titles | 71ms | 1.61s | 22.7× | | Q39 | CounterID 62 — link clicks | 89ms | 1.22s | 13.7× | | Q40 | Traffic source breakdown | 158ms | 1.73s | 11.0× | | Q41 | URL hash × referer filter | 68ms | 1.62s | 23.9× | | Q42 | Window size heatmap | 59ms | 1.63s | 27.6× | | Q43 | Per-minute pageviews | 57ms | 1.32s | 23.1× | Totals: Hotdata 12.4s, Neon 158.3s. Geomean speedup 12.7×. --- # Bringing DuckLake to Apache DataFusion Source: https://www.hotdata.dev/blog/bringing-ducklake-to-apache-datafusion Published: Jun 23, 2026 Author: Eddie A Tejeda Site index: https://www.hotdata.dev/llms.txt We've just released a new version of [DuckLake + DataFusion](https://github.com/datafusion-contrib/datafusion-ducklake) and have donated it to the [datafusion-contrib](https://github.com/datafusion-contrib/) repository. We’re delighted to bring a new lakehouse catalog format to the DataFusion ecosystem. But let’s back up and give context on what Apache DataFusion is, what DuckLake is, and why we think it’s a useful combination for building low-latency data systems. ## What is Apache DataFusion? [Apache DataFusion](https://datafusion.apache.org/) is a query engine written in Rust that uses Apache Arrow as its in-memory format. It provides SQL and DataFrame APIs, a query planner, query optimization, vectorized processing, and Parquet support. It is also embeddable, which means you can put it directly inside your application without running a separate service. It provides the core machinery of a database while leaving room for developers to extend as they see it. If you use DuckDB, the idea of a pure query engine may not be clear. To give some context: while DuckDB and DataFusion are both embeddable query engines, DuckDB also includes the surrounding database product: a command-line tool, a storage format, a catalog, transaction semantics, readers, writers, and a polished end-user experience. DuckDB is a complete database that you can configure and embed. DataFusion operates at a lower level. The library gives you planning, optimization, and execution, but it does not prescribe the rest of the architecture. To build a full database on top of DataFusion, you need to define several important pieces: storage format, catalog format, table representation, indexing, caching, and how tables are represented on disk. So that’s DataFusion in a nutshell. It is a powerful foundation for building a database, but it purposely leaves the higher-level product decisions to the builder. A tradeoff that drew us to using DataFusion for building Hotdata. We wanted control over the internals of the system and wanted to define how data is stored, how it is indexed, how it is cached, how requests are planned, and how the execution layer fits into the rest of the platform. This is where it’s useful to know about lakehouse architectures. ## What is a Lakehouse? Unlike a traditional database, where your data files and database are on the same machine, lakehouses and other distributed data systems store their data files in object storage, and those files have to be ingested into the execution engine before results get back to the user. At first glance, this may seem slow since disk access is much faster than going out to the network. But object storage has gotten so fast that it's possible to build large systems that ingest data from remote objects in milliseconds. And that's why lakehouses are so popular. At a simplified level, a lakehouse maps table names to objects that live in object storage, usually as Parquet files in a bucket such as S3. A request comes in with details like the organization, dataset, table, and version. The system resolves that metadata, finds the relevant files, and passes those files into an execution engine such as DataFusion or DuckDB. ![How a lakehouse works: a request keyed by (org, dataset, version) hits a metadata lookup table, which resolves Parquet and DuckDB files in S3 for execution.](/blog/bringing-ducklake-to-apache-datafusion/image-1.svg) When we previously built custom lakehouse metadata systems, this is the approach we took. It is a barebones system that works well when you have clear product requirements and can focus on reducing overhead. But difficulty rears its head over time, and extending this basic architecture becomes extremely difficult. As products evolve, so do the requirements. For example, adding table versions, snapshots, dealing with schema evolution, reassigning data from one customer to another, and auditability all require custom bookkeeping. That is why Apache Iceberg was exciting when it was released. Iceberg defines a table format with metadata, snapshots, schemas, manifests, and object-store-backed files. It gives structure to a problem that otherwise tends to become a collection of custom conventions. In Iceberg, a request comes in, the system resolves the table, and then walks through metadata files stored in object storage. It then reads snapshot metadata, partition specs, schema details, manifest lists, file manifests, per-file statistics, and information about deleted files. After traversing that metadata graph, the engine can identify the relevant Parquet files and begin execution. ![Iceberg-style metadata path: the query engine asks a REST catalog for the current snapshot, walks immutable metadata files in S3 (snapshot, manifest list, manifest files, delete files) to prune partitions, then opens the candidate Parquet files for row-group and page pruning.](/blog/bringing-ducklake-to-apache-datafusion/image-2.svg) That model is powerful, especially when a broad data ecosystem and throughput matters. For a low-latency query system, though, that metadata path can involve many steps. Multiple object-store requests before opening the actual data files can become expensive when the goal is to serve small, fast queries. That is why DuckLake caught our attention. ## What is DuckLake? DuckLake defines a lakehouse format where the metadata lives in a database such as Postgres, DuckDB, or MySQL, while the data files live in object storage. Instead of walking through many metadata files in S3, the system can query a metadata database and retrieve the information it needs about tables, snapshots, schemas, and Parquet files. For our use case, that is perfect. With one metadata query, we can resolve the table, understand the relevant files, get the paths to the Parquet files, and begin execution in DataFusion. ![DuckLake-style relational metadata catalog: the query engine issues one SQL call to a metadata database (Postgres, SQLite, or DuckDB) that returns the candidate Parquet files and row-group stats directly — no manifest walk, no extra object-store hops — before opening the files for row-group and page pruning.](/blog/bringing-ducklake-to-apache-datafusion/image-4.svg) The important point is that DuckLake is a specification. It is not only an application or a feature inside DuckDB. It defines how the metadata is represented and how the data files are organized. That means other systems can implement it. That led us to building DataFusion+DuckLake. Our goal is to combine the flexibility of Apache DataFusion with the structure of DuckLake. DataFusion gives us the execution engine. DuckLake gives us a concrete table and catalog model that works well with object storage and low-latency metadata access. So far, we have implemented key parts of the DuckLake architecture. We support reads and writes, multiple catalog backends including DuckDB, Postgres, and MySQL, encrypted Parquet files, hints for optimized I/O, filter pushdown for row-group pruning, and page-level filtering. We are also working closely with the DuckLake team to help advance the standard. ## Why DuckLake Fits DataFusion This replaces a large amount of custom database infrastructure that we would otherwise have to build ourselves. Instead of inventing our own table format, catalog model, versioning system, and metadata layout, we can implement a shared format and focus our energy on execution, performance, caching, indexing, and the developer experience around the system. ![Where time goes when you execute a query: Iceberg-style planning spends it on metadata traversal, object-store latency, and planning before the Parquet scan, while DuckLake-style planning collapses that to a single metadata DB lookup plus I/O.](/blog/bringing-ducklake-to-apache-datafusion/image-3.svg) We do not think DuckLake will replace Iceberg for every use case. If you are building a lakehouse that needs deep interoperability, Iceberg is the right choice. DuckLake is great for a different class of systems: applications, embedded query engines, low-latency serving layers, and systems that want the economics of object storage with the responsiveness of a database-backed catalog. That is the gap we care about for Hotdata. ## Conclusion The project is now available in the Apache DataFusion Contrib GitHub repository, and we would like more people to help shape it. If you are building with DataFusion, DuckLake, Parquet, object storage, or embedded query engines, contributions are welcome. Issues, bug reports, tests, documentation improvements, catalog backend work, and performance benchmarks are all useful at this stage. --- # How Hotdata accelerates Neon and Supabase Source: https://www.hotdata.dev/blog/how-hotdata-accelerates-neon-and-supabase Published: Apr 22, 2026 Author: Divya Ranganathan Site index: https://www.hotdata.dev/llms.txt Agents rarely connect to a single database. Transactional databases like Postgres generally store core application data, but agents often need to query multiple systems and require fast response times. Given today’s stack, what are good options for building an application that spans multiple databases? In this piece we compare: - **Neon:** serverless PostgreSQL optimized for cloud workloads - **Supabase:** a full backend platform built around PostgreSQL - **Hotdata:** a low-latency, PostgreSQL-compatible multi-modal query engine We’ll look at performance and explore where each tool fits in a project. ## A Review of the Platforms ### Neon: Serverless Postgres Neon provides serverless PostgreSQL that scales to zero when idle. Its architecture separates compute from storage and supports fast branching for preview environments. It’s great as a standalone Postgres database for applications. Key characteristics: - You pay for compute hours used - Compute scales to zero when idle - Instant branching (Git-like database copies) - Well-suited for variable workloads and many small databases ### Supabase: Postgres Platform At first glance Supabase looks similar to Neon, but it goes further than just Postgres. Supabase provides auth, real-time updates, storage, edge functions, and APIs. It’s an excellent all-in-one backend for developers who want fewer moving parts when building applications. Key traits: - Predictable base pricing plus usage-based costs - Includes auth, storage, real-time updates, and an edge runtime ### Hotdata: PostgreSQL-Compatible Multi-Modal Query Engine Hotdata is a read-only cache that sits in front of your databases, applications, or APIs. It can make multiple databases feel like one, allowing joins across multiple sources. Key characteristics: - Optimized for fast query execution - Data is synced in the background - Supports structured data, vectors, and full-text search in a single query ## Performance Snapshot Neon and Supabase are common choices for agent databases because developers like Postgres for its flexibility and ecosystem. As agent workloads grow, however, both can hit performance walls. Hotdata addresses these gaps without replacing the primary database. To illustrate the difference, we ran the ClickBench hits dataset (~100M rows, ~14GB Parquet) against each system using analytical queries involving filters, group-bys, and aggregations — the kind agents run when analyzing user data or building context. Hotdata was consistently orders of magnitude faster than both Neon and Supabase on these workloads. ![](/blog/how-hotdata-accelerates-neon-and-supabase/cover.png) > **We Use Neon and Supabase to Build Hotdata** Behind the scenes, we rely on Supabase and Neon for specific use cases. We use Neon, in particular, to manage customer metadata and both serve as backing stores with well-defined transactional qualities. But when we need to aggregate and search that data, we treat them as sources that Hotdata queries directly. This lets us cache and accelerate our heaviest queries. ## When Should You Use Hotdata? - **Your AI/LLM applications need fast data access:** agent queries, tool calling, function execution, and context retrieval under ~50ms - **Your workloads combine multiple access patterns in a single loop:** structured queries, vector search, and full-text, without stitching systems together - **You want to decouple compute from your primary database:** offloading read-heavy traffic without impacting production - **You care about both latency and cost at scale**: millisecond queries without paying warehouse-level costs per query --- # Building Vector Search Into a SQL Query Engine Source: https://www.hotdata.dev/blog/building-vector-search-into-a-sql-query-engine Published: Apr 16, 2026 Author: Eddie A Tejeda Site index: https://www.hotdata.dev/llms.txt > We added native [ANN](https://www.google.com/search?q=Approximate+Nearest+Neighbor+%28ANN%29+search&oq=what+is+ANN+search&gs_lcrp=EgZjaHJvbWUqBwgAEAAYgAQyBwgAEAAYgAQyCAgBEAAYFhgeMg0IAhAAGIYDGIAEGIoFMg0IAxAAGIYDGIAEGIoFMg0IBBAAGIYDGIAEGIoFMgcIBRAAGO8FMgoIBhAAGIAEGKIEMgcIBxAAGO8FMgoICBAAGIAEGKIEMgcICRAAGO8F0gEIMzc2NmowajeoAgCwAgA&sourceid=chrome&ie=UTF-8&ved=2ahUKEwj17ZWI7umTAxU7PzQIHcyUOuwQgK4QegYIAQgAEAU) search to our DataFusion-based engine using [USearch](https://github.com/unum-cloud/usearch), Parquet, and SQLite — with adaptive filtering and got impressive results. Today, if you want to add vector search to an application, you typically query a separate vector database that lives alongside your primary data store. Inspired by pgvector, we decided to bring vector search directly into our query engine, built on Apache DataFusion, and make vector search a first-class SQL operator. In this post, we walk through the architecture, key design decisions, especially around filtered search, and benchmark results against LanceDB Cloud, who we consider the industry leader, on a 1.2 million row dataset. ## What We Were Optimizing For > Vector search should be a SQL operator, not a separate service you query around. Before diving in, it’s useful to understand our goals and design principles. Hotdata is primarily focused on latency rather than throughput, so we don’t have the same requirements as a many other databases. We are primarily optimized for queries that process millions of records at a time, not full warehouse-scale scans. In practice, datasets are roughly 1–10 million vectors at 500–2000 dimensions, and we store data entirely in a single node. Since we’re not doing distributed vector search, managing what we keep in memory is our primary challenge. We do not assume all vectors fit into RAM, so we fall back to NVMe when necessary. Finally, we target sub-100ms queries, even with filters applied. So we want to mix vector operations with standard filters like WHERE category=’nlp’ ORDER BY l2_distance() LIMIT 10 without any special syntax. This is important, and I’ll explain why below. ## How We Store the Data > [Hierarchical Navigable Small World](https://www.google.com/search?q=Hierarchical+Navigable+Small+World&oq=HNSW&gs_lcrp=EgZjaHJvbWUyBggAEEUYOdIBBzE5NGowajeoAgCwAgA&sourceid=chrome&ie=UTF-8&ved=2ahUKEwiyzta44PCTAxUkDjQIHb8lJ1kQgK4QegYIAAgAEAQ) (HNSW) is a state-of-the-art, graph-based algorithm for [Approximate Nearest Neighbor (ANN)](https://www.google.com/search?q=Approximate+Nearest+Neighbor+%28ANN%29+search&oq=what+is+ANN+search&gs_lcrp=EgZjaHJvbWUqBwgAEAAYgAQyBwgAEAAYgAQyCAgBEAAYFhgeMg0IAhAAGIYDGIAEGIoFMg0IAxAAGIYDGIAEGIoFMg0IBBAAGIYDGIAEGIoFMgcIBRAAGO8FMgoIBhAAGIAEGKIEMgcIBxAAGO8FMgoICBAAGIAEGKIEMgcICRAAGO8F0gEIMzc2NmowajeoAgCwAgA&sourceid=chrome&ie=UTF-8&ved=2ahUKEwj17ZWI7umTAxU7PzQIHcyUOuwQgK4QegYIAQgAEAU) search used in vector databases for high-dimensional data retrieval. When we ingest data, we build different indexes optimized for different access patterns. All three indexes share the same keys so we can easily toggle between stores. For example, key 0 in each store refers to the same record. This means that we are storage-heavy, but that’s a trade we’re willing to make. The stores we have are: [**USearch**](https://github.com/unum-cloud/usearch) **index:** We use this to store the HNSW graph and raw vectors. This handles ANN search and lets us retrieve vectors directly by key. This is optimized for fast nearest neighbor search, but it doesn’t understand SQL predicates, which is why we use other layers for filtering. **SQLite:** We store all scalar columns here. After we identify keys, we use SQLite to fetch the corresponding rows. Here we can get sub-millisecond lookups for fields like id, title, and metadata through a simple B-tree lookup. This works great with small datasets. **Parquet:** We use Parquet to store the full dataset, including vectors, and to evaluate filters. It’s a columnar layout, and supports row group statistics, bloom filters, and page indexes. This lets us push down predicates and skip large portions of data. This works well for scanning and filtering, but not for point lookups. So those are the building blocks. Now, let’s step through the different kinds of searches. ## Unfiltered vector search Lets analyze a simple vector search without any conditions or filters: ``` SELECT id, title, l2_distance(embedding, ARRAY[0.1, 0.2, ...]) AS dist FROM my_table ORDER BY dist ASC LIMIT 10 ``` When we execute it, the query goes through the following steps: ``` SQL query │ ▼ Optimizer rewrites ORDER BY distance_fn(...) LIMIT k │ ▼ USearch HNSW search(query_vector, k) │ returns: top-k keys + distances ▼ SQLite fetch_by_keys(keys) │ returns: scalar columns for those k rows ▼ Attach _distance column → result ``` We let USearch traverse the HNSW graph and return the k closest keys with their distances. On a million-row index, this usually runs in single-digit milliseconds. We then use SQLite to fetch the corresponding row data, such as id, title, and metadata, via primary key lookups. Since there’s no filtering, we don’t touch Parquet at all. ## Filtered vector search Things get interesting when you introduce a WHERE clause. Consider this query: ``` SELECT id, l2_distance(embedding, ARRAY[...]) AS dist FROM my_table WHERE category = 'nlp' ORDER BY dist ASC LIMIT 10 ``` The complicating part of this query is that the HNSW index is built over the entire dataset, but we only want results from the category = 'nlp' subset. The naive approach would have us first filter, then search, but that would require that we build a separate index per filter value. Instead, we use a two-phase strategy that decides the best execution path at runtime based on how selective the filter is. ### Phase 1: Which rows match the filter? Before we can search for nearest vectors, we need to know which rows match the WHERE clause. We do this with a lightweight Parquet scan that reads the key column and the columns referenced by the filter. The filter predicate is pushed down to the Parquet reader, which can skip entire row groups and pages using Parquet’s built-in statistics, bloom filters, and page indexes. ``` 1.2M rows in Parquet │ ▼ scan _key + category only, push predicate down │ ┌─────┴──────┐ │ valid_keys │ → { 42, 197, 1053, 8841, ... } │ (71K keys) │ └─────┬──────┘ │ ▼ selectivity = 71K / 1.2M = 5.8% ``` This produces a set of keys that we need for the next phase and tells us both *which rows* to consider and *how many.* ### Phase 2: Choosing between two paths Once we know the total number of valid keys, we calculate the percentage of the keys we have on hand relative to the size of the table. When we have high selectivity (> 5% of rows match), we pass the keys as a predicate callback to USearch’s filtered_search() which traverses the HNSW graph. It then returns top-k results and we fetch the result rows from SQLite. ``` selectivity? / \ > 5% ≤ 5% │ │ ▼ ▼ HNSW filtered_search index.get(key) for each valid_key with valid_keys compute distances as predicate maintain top-k heap │ │ ▼ ▼ SQLite fetch(k) SQLite fetch(k) │ │ ▼ ▼ result result ``` When we have low selectivity (≤ 5% of rows match), we use the USearch index as a key-value store: retrieve the vector for each valid key via index.get(), compute exact distances, maintain a top-k heap, and fetch result rows from SQLite. This means we can avoid graph traversal completely and only do lookups. Which is very fast! ## Benchmarks To see how our approach compares, we benchmarked against LanceDB Cloud, a purpose-built vector database with HNSW support backed by the Lance columnar format. This is not an apples-to-apples comparison of equivalent architectures, but it’s good comparison point to understand how our approach stacks up. **Dataset**: 1.2 million rows from the Sphere dataset. 768-dimensional embeddings (float64) plus scalar columns: id, url, title, sha, raw, filename. **Hotdata config**: HNSW, M=16, ef_construction=128, ef_search=64, L2, F32 precision. **LanceDB config**: IVF_HNSW_SQ, M=16, ef_construction=128, L2 (SQ = scalar quantization; the highest-recall HNSW variant available on their cloud). **Methodology**: 5 random query vectors from the dataset. 1 cold run discarded, 3 warm runs averaged. All times are server-side execution time — LanceDB’s analyze_plan and our internal execution_time_ms . ## Results In our first comparison, Hotdata’s adaptive filtered search beats LanceDB across different filter types: ![All queries return k=10.](/blog/building-vector-search-into-a-sql-query-engine/image-1.png) *All queries return k=10.* Since Hotdata is focused on serving data, when comparing plain search, we see that performance scales well from k=10 to 500. ![Hotdata’s latency is nearly flat from K=10 to K=500.](/blog/building-vector-search-into-a-sql-query-engine/image-2.png) *Hotdata’s latency is nearly flat from K=10 to K=500.* We were excited to see that this approach is proving to be quite fast and for our use-case we think we can refine it further. We’ll continue to post our findings. ## What We Learned The main takeaway is architectural: vector search and SQL dot not need to be fundamentally different systems. If you use a query engine that can treat ANN search as a natural extension of SQL, you can collapse layers of architecture. From a developer experience standpoint, this is a major win. --- # Why Agents Need Version Control for Data Source: https://www.hotdata.dev/blog/why-agents-need-version-control-for-data Published: Apr 16, 2026 Author: Eddie A Tejeda Site index: https://www.hotdata.dev/llms.txt In the early days, developers could keep knowledge of the systems they worked on in their heads. You changed a file, ran a test, and moved on. That worked when codebases were relatively small. ![](/blog/why-agents-need-version-control-for-data/cover.png) But as systems grew, that model broke. Once projects reach a certain size, you often need to understand how the system arrived at its current state. For example, to answer basic questions like “when was this bug introduced?”, you need a way to track changes over time. That’s why version control is so important. Version control creates a durable record of every change you make. This allows you to see how the codebase looked at any point in time, branch off at any snapshot, and then rejoin the main codebase whenever you are ready. Git pushed this idea further by encouraging even smaller steps, making it easier to work across branches. Git itself came out of necessity, as Linus Torvalds struggled to coordinate one of the largest globally distributed teams on earth. This basic idea has allowed us to build more ambitious systems and allowed hyper-collaborative tools like GitHub to exist. Agents are now running into a similar situation as developers, but through the lens of data. Agents’ context windows act as working memory. An agent pulls in a few tables, runs some queries, forms a local understanding of the problem, answers questions, and when the task ends, that context disappears. This makes it hard for agents to build on earlier work. When an agent wants to explore a different path, it often repeats the same queries and recomputes the same results because there is no durable record of what it already did. The work looks iterative, but much of it is being recomputed every time. Now imagine managing this when the work is exploratory. The number of combination explodes! This is where the version control analogy becomes useful. Version control is not only about collaboration. It is also about maintaining an audit trail of state. We have this concept for code, and even for Docker containers capturing the state of a system, but we are still early on the data side. Today, agents assemble tables and queries and work within limited scopes, but it’s all ephemeral. Unless it is a specific project requirement, teams are not building agents with durable, re-playable data, at the level of individual sessions. As agents take on more open-ended tasks, managing context windows will be a challenge, and properly managing the evolution of context will matter as much as the models we use. --- # Agents Are Only as Good as the Data They Can JOIN Source: https://www.hotdata.dev/blog/agents-are-only-as-good-as-the-data-they-can-join Published: Apr 15, 2026 Author: Divya Ranganathan Site index: https://www.hotdata.dev/llms.txt Now that everyone is building agents on diverse data sources and agents have better reasoning tools, **how do you quickly assemble data from many different sources, in the right shape, so the model can actually reason well with it?** A version of this problem has always existed and DBAs have wrestled with this for a long time. It’s the join problem. > *“An agent’s reasoning is only as rich as the context you can assemble for it. More sources means more opportunity and more complexity in stitching them together.”* This post is about that assembly challenge that agents need to work, specifically, joins. Not just the SQL JOIN keyword (though we’ll get there), but the broader problem of reconciling data spread across your CRM, your warehouse, your SaaS APIs, and your vector stores. We’ll walk through how teams are solving this today and where each approach runs into limits. ## Why Joins Matter Lets walk through an example: when a user asks an agent “Which enterprise accounts are at risk this quarter?”, the agent needs renewal dates from Salesforce, product usage from the warehouse, support ticket volume from Zendesk, and payment history from Stripe, then it needs to reconcile all of that on a shared key and handle missing values gracefully. In a traditional single-database environment, this is done in SQL query. In an agent environment spanning multiple live systems every source has its own latency profile and error handling. A join across three SaaS APIs and a warehouse is essentially a small distributed systems problem. And unlike a query you run once and fine tune, agents run these queries in an **ad hoc manner**, at runtime, against whatever the user asks. ![](/blog/agents-are-only-as-good-as-the-data-they-can-join/cover.png) ## How Teams Are Solving It Here are the five main patterns teams use to handle joins in agent systems. Lets go over them. - **PATTERN 1 — WAREHOUSE TABLES** > Generate SQL, run it in Snowflake/BigQuery/Databricks. Joins happen inside the database, results come back clean. This is the default approach for any question that’s fundamentally analytical. If your agent can express the question as SQL and the data is already in the warehouse, let the database do the join. It will be faster, cheaper, and more accurate than anything you build in application code. The problem comes in expectations. Pushdown assumes your data is already stored in the warehouse. The moment the answer requires a live SaaS API call, you’re out of luck. You’ll need to build pipelines to transform your data sources into the correct formats. - **PATTERN 2 — PRE-JOINED VIEWS** > Materialize wide tables via dbt or Fivetran. Agent queries a single denormalized source. Fast, predictable, simple. Next, prejoined views, also known as materialized views. This has taken off in the past decade by technologies like dbt. The good news is that you get predictable schemas, relatively fast queries, and you dramatically reduce the surface area for the LLM to make SQL mistakes. But the trade-off is freshness and flexibility: you’ll end up maintaining a long-tail of increasingly specific wide tables as your agent’s query patterns expand. This pattern is when you are interacting with your most common, high-value queries, and static list of trusted datasets. That is both it’s biggest strength and limitation - **PATTERN 3 — RAG-STYLE RETRIEVAL** > Fetch embeddings from multiple stores, merge in-context. Works for unstructured reasoning, breaks on aggregations. This is the pattern most agent builders reach for first, and it’s frequently misapplied. Semantic retrieval is excellent when the question is about meaning and relevance — “find support tickets similar to this complaint” but it’s dangerous when the question requires precise aggregation or exact matching. “Which customers had more than 5 support tickets last quarter?” is not necessarily a retrieval problem, it’s mostly a counting problem. So treating it like a retrieval problem will give you plausible-sounding answers, but when you dig a little further you’ll have incorrect answers. So the biggest takeaway is to make sure you understand that boundary before you ship. - **PATTERN 4 — FEDERATED QUERY LAYER** > A query engine (Trino, Presto, or custom) joins across SaaS + warehouse in real time, transparent to the agent. This is where we think agent infrastructure is heading. Federated queries became popular in the early 2010s with Trino/Presto, but the power of scalability of data warehouses meant they quickly fell out of favor. A federated query engine sits between your agent and your data sources, accepting queries and handling the cross-source join logic internally and pushing down filters where it can and merging results on-demand. It interacts with the data as a source be it an API or warehouse, and queries it like any other table. But there are some serious engineering challenges: query planning, predicate pushdown, caching, and identity resolution across systems all have to be solved. For agents answering cross-system questions in production, this is showing much more promise. - **PATTERN 5 — APPLICATION-LAYER JOINS** > Agent fetches data step-by-step, merges in Python. Simple to write, but latency compounds, schemas drift, and it breaks at scale. Application-layer joins are surprisingly popular, and we see teams using this approach for quick prototypes, but then find themselves locked into an architecture that does not scale. Fetching data from three APIs sequentially in Python, then merging it in a dictionary comprehension, will work fine in a demo and fall apart in production for three reasons: 1. **Latency compounds.** Three 200ms API calls in series is 600ms before the model even starts reasoning. 2. **Schema drift breaks silently.** When the CRM adds a new account ID format, your join condition returns empty and no one notices. 3. **It doesn’t scale.** What works for 50 records in a test doesn’t work for 50,000 records in a real query. Use it to prototype. Replace it before you ship to production. As you build or scale your agent infrastructure, it’s worth auditing where your joins actually live. How many cross-source questions can your agent answer reliably today? Where does it silently degrade — returning partial results, slow responses, or stale data because the assembly layer isn’t keeping up with the reasoning layer? --- # Query engines for Agents Source: https://www.hotdata.dev/blog/query-engines-for-agents Published: Apr 15, 2026 Author: Eddie A Tejeda Site index: https://www.hotdata.dev/llms.txt We’re building a query engine, so I’ve been thinking about the difference between query engines built for humans and those specially made for agents. Regardless of the differences, I think two things will be the same: speed will be important, and it will, of course, use SQL. So what’s the difference? A big part is how queries are generated. Humans and agents operate at very different speeds and under different constraints. Humans are slow. They have a question, perform a query, look at the results, tweak it, and maybe come back to it later. Once they have an answer, they can carry that information forward without having to go back to the results again. A lot of tooling is built around that rhythm. On the other hand, agents are fast and repetitive. They issue lots of small queries, chain them together, and reuse intermediate results. As a result, queries are tightly coupled with the reasoning process. ![](/blog/query-engines-for-agents/cover.png) ### Execution structure Agent queries are rarely independent. They form a dependency graph, where the output of one step feeds directly into the next, and intermediate results are reused across branches of a larger plan. Humans can hold context in their heads, while agents need systems to help manage that context. Humans can analyze by skimming data and building an intuitive sense of what they are looking at. Agents need to replay what they’ve done and compare results. I think version control presents an internal mental model for how agents can keep track of change. ### Durability and context loss What this means is that if the results are not durable, context is lost. When intermediate results disappear or change across runs, the agent loses its place. The reasoning process becomes non-replayable, and small inconsistencies can cascade across an entire workflow. I wrote about that [here](https://medium.com/@eddietejeda/why-agents-need-version-control-for-data-66872ef1cdfc). ### Latency amplification Agents are also more sensitive to latency. A human can tolerate a few seconds of delay, but an agent may have dozens of parallel requests going, and having an answer block can interrupt a much larger workflow. Latency compounds in agent workflows. A single slow query can stall dozens of dependent steps, turning tail latency into a dominant failure mode rather than a minor inconvenience. The question that looms large over this topic is “will OpenAI solve this?” and the answer is: yes, but it doesn’t matter. There will be better models, but there will also be more data, and those new models will likely be interacting with more data sources, not fewer. What does that interface look like? --- # Why we are building Hotdata Source: https://www.hotdata.dev/blog/why-we-are-building-hotdata Published: Apr 6, 2026 Author: Eddie A Tejeda Site index: https://www.hotdata.dev/llms.txt ## The Problem Over the past decade, data infrastructure prioritized throughput and scale over latency and interactivity, cementing large distributed warehouses as the default. Today, data platform teams pass queries through layers of services, schedulers, pipelines, and caches to get data ready for serving. Each layer adds overhead that mask underlying problems. That model worked fine for the “big data” era, but agents are changing how we think about data access. Fortunately, many of the techniques created to handle petabyte-scale systems work incredibly well when optimized for smaller datasets (<100GB). Processing tens of millions of rows is now possible in under 10ms. As hardware has improved, a single machine can process more than before. Vectorized execution, columnar formats, SIMD instructions, and fast NVMe storage make it possible to run substantial workloads on a single machine, with improvements in cost and latency. If you follow the academic literature, you can see an explosion in database research that is still making its way into industry. At the same time, the interface to databases is changing. Databricks reported that 80% of databases are created by AI agents, and agents inherently work with data differently. They create many datasets, generate large amounts of small queries, and connect working results into larger flows. The problem is that traditional databases execute queries without awareness of the context. So coordination of all this data moves into the application layer and introduces additional overhead. To take full advantage of these performance gains, we need to rethink the interface to databases. ![](/blog/why-we-are-building-hotdata/cover.png) ## Why We’re Building Hotdata Hotdata comes from understanding that agents have different needs and priorities when accessing data. Agents and AI apps operate on small, dynamic slices of data where latency, context, and composability matter more than raw throughput. Our goal is have agents access any data source, including databases, lakes, or SaaS systems, through a unified interface that prioritizes latency. Agents should work with data without having to worry about data movement. Every result should be materialized and easily re-accessible. Intermediate results should be reused as part of the workflow instead of being rebuilt each time. Isolation and replayability should live within the data layer itself, making it easy to manage and reason about complex flows. Hotdata is built around these key principles to build a fundamentally new kind of query engine for agents. --- # About Us Source: https://www.hotdata.dev/about Site index: https://www.hotdata.dev/llms.txt Hotdata is the execution layer for AI agents. AI agents are data workloads. They create datasets, fan out across many small queries, join results from multiple systems, and materialize intermediate state as they work. Traditional data platforms were not designed for this model. Shared warehouses create contention, while separate systems introduce data copies, delay, and operational overhead. Hotdata gives each agent workload an isolated, high-concurrency environment to search, query, join, analyze, and persist results across the data systems already in place without adding load to systems of record. It is the data substrate agents use to execute multi-step work, retain state, and reuse results. We’re a team of data-infrastructure engineers who have spent decades building, optimizing, and operating data platforms at scale. ![Hotdata founders](/assets/about/founders.jpg) *Founders, left to right: Eddie A Tejeda (CTO), Divya Ranganathan (CEO), Zac Farrell (Principal Engineer).* **Advisors:** - [Andrew Lamb](https://andrew.nerdnetworks.org/), PMC Chair of Apache DataFusion, Apache Arrow, Apache Parquet, and a Member of the Apache Software Foundation. - [Kathryn Vandiver](https://www.linkedin.com/in/kathrynvandiver/), Technology Executive, SheTO Board Member and Advisor to Berkeley SkyDeck --- # Careers Source: https://www.hotdata.dev/careers Site index: https://www.hotdata.dev/llms.txt Join us building the hybrid query engine for AI agents. ## Open roles - [Research Intern – Database Internals - Remote](https://jobs.gusto.com/postings/hotdata-inc-research-intern-database-internals-2c704725-9afc-42ed-bcc9-e47ffe47696f) - [Sr. Full Stack Agent Engineer - San Francisco Bay Area](https://jobs.gusto.com/postings/hotdata-inc-sr-full-stack-agent-engineer-3d7c008f-2fb7-46e1-859a-c009b0f7d2f4) - [Sr. Backend Engineer - San Francisco Bay Area](https://jobs.gusto.com/postings/hotdata-inc-sr-backend-engineer-9973f5a9-396c-40f7-a5fe-e114b2276da5) - [Sr. Full Stack Agent Engineer - Remote, Bengaluru, India](https://jobs.gusto.com/postings/hotdata-inc-sr-full-stack-agent-engineer-5e887b93-a67f-4baa-a61c-d20ba802bdf7) --- # Privacy Policy Source: https://www.hotdata.dev/privacy-policy Effective: February 28, 2026 Site index: https://www.hotdata.dev/llms.txt Hotdata, Inc. ("Hotdata", "we", "us", or "our") respects your privacy and is committed to protecting your personal data. This Privacy Policy explains how we collect, use, disclose, store, transfer, and protect personal data when you use our website, products, services, and applications (collectively, the "Services"). By accessing or using our Services, you consent to the practices described in this Privacy Policy. ## 1. Information We Collect We collect personal data from and about you when you interact with our Services. The categories of personal data we collect may include: ### a. Personal Information You Provide - Name, email address, phone number - Company or organization, job title - Account credentials - Communication content (e.g., support requests) ### b. Automatically Collected Data - IP address, device type, browser type - Usage data (pages visited, interactions, timestamps) - Cookies and similar technologies ### c. Third-Party Data We may receive data from third-party services you choose to connect with Hotdata (e.g., analytics providers, identity providers). ## 2. How We Use Your Information We use personal data for the following business and operational purposes: - To provide, operate, maintain, and improve the Services - To create and manage user accounts - To communicate with you (including service updates and support) - To personalize content and features - For analytics, diagnostics, and internal research - To detect, prevent, and respond to security risks and fraud - To enforce legal rights and comply with laws We will not use collected personal data for materially different purposes without giving you notice. ## 3. Legal Bases for Processing (Where Applicable) Where required by law (e.g., GDPR), we rely on one or more lawful bases to process personal data, including: - Consent - Performance of a contract with you - Compliance with legal obligations - Legitimate interests (such as improving our Services) ## 4. How We Share Your Data ### Service Providers & Partners We share data with vendors and third parties who perform services on our behalf, such as: - Hosting and infrastructure providers - Analytics and telemetry partners - Customer support platforms - Identity and authentication services ### Legal Obligations We may disclose personal data to comply with legal obligations, enforce agreements, or respond to lawful requests by public authorities. ### Business Transfers Your personal data may be transferred if Hotdata undergoes a merger, acquisition, reorganization, or sale of assets. ## 5. Cookies and Tracking Technologies We and our partners use cookies, pixels, and similar technologies to collect information about your interactions with the Services. This data helps with: - Session management - Performance analytics - Personalization You can control cookie preferences through your browser settings. ## 6. International Data Transfers Hotdata's operations are primarily located in the United States. Consequently, this means that your Personal Data will be transferred to the United States. ## 7. Data Retention We retain personal data as long as necessary to provide the Services, comply with legal obligations, resolve disputes, and enforce agreements. Retention periods vary based on the type of data and applicable law. ## 8. Your Privacy Rights Depending on your jurisdiction, you may have rights related to your personal data, such as: - Access - Correction - Deletion - Restriction of processing - Objection to processing - Data portability You can exercise these rights by contacting us at the address below. ## 9. Children's Privacy As noted in the Terms of Use, we do not knowingly collect or solicit Personal Data from children under 18 years of age; if you are a child under the age of 18, please do not attempt to register for or otherwise use the Site or send us any Personal Data. If we learn we have collected Personal Data from a child under 18 years of age, we will delete that information as quickly as possible. If you believe that a child under 18 years of age may have provided Personal Data to us, please contact us at contact@hotdata.dev. ## 10. Security We implement appropriate technical and organizational security measures designed to protect personal data from loss, misuse, and unauthorized access, disclosure, alteration, or destruction. ## 11. Changes to this Policy We may update this Privacy Policy from time to time. We will notify you of significant changes by posting the updated policy on our website and updating the "Last Updated" date. ## 12. Contact Us If you have questions, requests, or concerns about this Privacy Policy or our privacy practices, please contact: **Privacy Team** Hotdata, Inc. Email: contact@hotdata.dev Address: 95 3rd St, 2nd Floor, San Francisco, CA 94103, United States --- # Terms of Service Source: https://www.hotdata.dev/terms-of-service Effective: March 31, 2026 Site index: https://www.hotdata.dev/llms.txt Welcome to Hotdata. Please read these Terms of Service before using the Services. These Terms of Service ("Terms") are an agreement between Hotdata and you, or the organization you represent ("Customer"). They govern Customer's use of the Hotdata API and any other offerings that reference these Terms, including related tools, documentation, and services (the "Services"). These Terms take effect on the earlier of Customer's acceptance or first use of the Services ("Effective Date"). You may only accept these Terms on behalf of an entity if you have authority to bind that entity. ## A. Services ### 1. Overview Subject to these Terms, Hotdata grants Customer a limited right to use the Services, including to support products and services Customer provides to its users ("Users"). ### 2. Third-Party Features Customer may use third-party features made available through the Services ("Third Party Features"). These are not part of the Services, and Hotdata is not responsible for them. ### 3. Feedback If Customer provides feedback, Hotdata may use it without restriction or obligation. ## B. Customer Content As between the parties, Customer retains all right, title, and interest in and to Customer Content. To the extent Outputs are generated from Customer Content through authorized use of the Services, Customer owns those Outputs. Customer grants Hotdata a non-exclusive, worldwide, limited license to host, store, copy, transmit, display, modify, transform, index, cache, back up, and otherwise process Customer Content solely as necessary to provide, maintain, secure, support, bill for, and comply with law in connection with the Services. Hotdata may generate and use metadata, logs, performance data, and other operational information derived from Customer Content solely to operate, secure, and improve the Services, provided such information does not identify Customer Content. "Inputs" means data, content, and other materials submitted to the Services by Customer or its Users. "Outputs" means results, responses, and other materials generated or returned by the Services in response to Inputs. Inputs and Outputs together are "Customer Content." ## C. Trust, Safety, and Restrictions ### 1. Compliance Each party will comply with applicable laws, including data privacy laws. ### 2. Policies Customer and Users must comply with these Terms and the Usage Policy. Customer must cooperate with reasonable requests related to compliance. ### 3. Outputs Customer is responsible for evaluating Outputs before use. Hotdata does not guarantee Outputs are complete, accurate, up-to-date, or error-free. Customer must inform Users accordingly where appropriate. ### 4. Restrictions Customer may not: (a) use the Services to build or train competing products or services; (b) use Outputs to train or improve competing systems; (c) reverse engineer or replicate the Services; or (d) assist others in doing so. ### 5. Account Responsibility Customer is responsible for all activity under its account and must promptly notify Hotdata of any compromise or misuse. ## D. Confidentiality ### 1. Confidential Information Confidential Information includes information identified as confidential or reasonably understood to be confidential. Customer Content is Customer’s Confidential Information. ### 2. Use and Protection The receiving party may use Confidential Information only to exercise its rights and perform its obligations under these Terms, including processing permitted under Section B (Customer Content). Confidential Information may be shared only with personnel who need access and are bound by confidentiality obligations. Each party will protect Confidential Information with reasonable care. ### 3. Exclusions Confidential Information excludes information that is publicly available, lawfully obtained from a third party, or independently developed. Disclosure required by law is permitted with notice where allowed. ### 4. Destruction Confidential Information must be destroyed upon request, except where retention is legally required or part of backup systems. ## E. Intellectual Property Except as stated in these Terms, neither party grants the other rights to its intellectual property. ## F. Publicity Hotdata may use Customer’s name and logo to identify Customer as a user of the Services. Customer may opt out via contact@hotdata.dev. Hotdata will not disclose Customer Content in publicity materials. Customer will reasonably consider requests for quotes or co-marketing participation. ## G. Fees ### 1. Payment Customer must pay applicable fees as listed on the Pricing Page unless otherwise agreed. Fees may be prepaid through credits. Pricing may change with 30 days notice. ### 2. Taxes Customer is responsible for all applicable taxes. If withholding applies, Customer will ensure Hotdata receives the full intended amount. ### 3. Billing Failure to pay may result in suspension or termination. Hotdata may pursue collection. ## H. Termination and Suspension ### 1. Term These Terms continue until terminated. ### 2. Termination Either party may terminate for convenience (Hotdata with 30 days notice) or for material breach (after a 30-day cure period). Hotdata may terminate immediately if required by law. ### 3. Suspension Hotdata may suspend access to the Services if necessary to address security risks, violations of these Terms, legal requirements, excessive or abusive resource usage, or third-party service disruptions. Hotdata will provide notice where reasonably possible and restore access when conditions are resolved. ### 4. Effect Upon termination, Customer may no longer access the Services. Provisions that by their nature should survive will remain in effect. ## I. Disputes ### 1. Informal Resolution The parties will attempt to resolve disputes informally for 45 days following notice. ### 2. Arbitration Unresolved disputes will be resolved by binding arbitration in English in Oakland, CA under JAMS rules. Jury trials and class actions are waived. ### 3. Equitable Relief Either party may seek equitable relief. ## J. Indemnification ### 1. By Hotdata Hotdata will defend and indemnify Customer against third-party claims that Customer’s authorized use of the Services, in unmodified form and in accordance with these Terms, infringes intellectual property rights. ### 2. By Customer Customer will defend and indemnify Hotdata against claims arising from Customer Content or misuse of the Services. ### 3. Exclusions Indemnification does not apply to claims arising from misconduct, unauthorized modifications, improper use, or combinations with third-party materials. ### 4. Process The indemnifying party controls the defense and settlement. The indemnified party must cooperate. ### 5. Sole Remedy Indemnification is the exclusive remedy for covered third-party claims. ## K. Warranties and Liability ### 1. Warranties Each party confirms it has authority to enter into these Terms. Customer confirms it has rights to its Inputs. ### 2. Disclaimer The Services are provided "as is" and "as available." Hotdata disclaims all implied warranties, including merchantability, non-infringement, and fitness for a particular purpose. Hotdata does not guarantee uninterrupted operation or that the Services will be error-free. ### 3. Liability Limits Liability excludes indirect, incidental, and consequential damages. Each party’s total liability is limited to fees paid in the previous 12 months. These limits do not apply to indemnification obligations. ## L. Miscellaneous ### 1. Notices Notices must be in writing and may be delivered electronically. ### 2. Electronic Communications Customer agrees to receive communications electronically. ### 3. Changes Hotdata may update these Terms with 30 days notice. Changes required by law take effect immediately. ### 4. Assignment Assignment requires consent, except in connection with corporate transactions. ### 5. Severability Invalid provisions do not affect the rest of the Terms. ### 6. Interpretation Headings are for convenience only. Terms are interpreted neutrally. ### 7. Governing Law These Terms are governed by Delaware law. Non-arbitrated disputes will be handled in Delaware courts. ### 8. Export Controls Customer must comply with export and sanctions laws. ### 9. Entire Agreement These Terms represent the full agreement between the parties. ### 10. Force Majeure Neither party is liable for delays caused by events beyond reasonable control. ## M. Third-Party Provider Terms The Services may include technology from third-party providers ("Upstream Providers"). Customer and Users must comply with their applicable terms.