Querying Hotdata with Ibis

Ibis is a Python dataframe library that turns your expressions into SQL and runs them on a database. hotdata-ibis adds Hotdata as one of those databases. You write Ibis code in Python, Hotdata does the work, and you get the results back as pandas or Arrow.

What you can do

You can handle the whole life of an instant database from Ibis: create it, upload a pandas DataFrame or Arrow table, query it, and drop it when you're finished.

Queries can filter, select, join, group, aggregate, sort, and limit. .execute() gives you a pandas DataFrame, and .to_pyarrow() or .to_pyarrow_batches() give you Arrow. If you need SQL that Ibis doesn't cover, con.sql(...) runs it directly, and you can keep chaining Ibis expressions on top of the result.

There are vector search helpers too. semantic_search writes its query in the shape Hotdata needs to use its HNSW vector index, so searches don't fall back to scanning the whole table.

And if your data is in another database or warehouse today, you can ingest it into an instant database first and then query it through Ibis like any other table.

Creating, loading, and querying

First, install the package:

pip install hotdata-ibis

Then connect, create a database, and upload a DataFrame:

import time
import ibis
import pandas as pd

con = ibis.hotdata.connect(
    api_url="https://api.hotdata.dev",
    token="YOUR_API_KEY",
    workspace_id="<workspace_id>",
)

db_id = con.create_database("sales", schema="public", tables=["orders"])

df = pd.DataFrame({
    "order_id": [1, 2, 3],
    "amount": [9.99, 49.99, 5.00],
    "region": ["west", "east", "west"],
})
con.create_table("orders", df, database=(db_id, "public"), overwrite=True)

time.sleep(2)  # uploads are async

Now you can query it with an Ibis expression:

t = con.table("orders", database=("default", "public"))

result = (
    t.group_by("region")
    .agg(total=t.amount.sum())
    .order_by(ibis.desc("total"))
    .execute()
)

create_database returns the database id, which you'll use to load or drop tables. When querying, the catalog is always "default".

Try it out

An easy way to start is with a DataFrame you already have. Upload it, then rewrite one of your pandas transformations in Ibis and let Hotdata run it.

For more, see the Ibis guide for connection options, raw SQL, and querying existing sources, and the repository for vector search and examples.