Collecting Web Data into Hotdata with Bright Data

A lot of useful questions depend on data from the web, like product prices, reviews, or job listings. Bright Data is one of the most widely used ways to collect that data, with a catalog of more than 1,200 ready-made scrapers for sites like Amazon and LinkedIn. The community-built brightdata-hotdata-bridge loads what those scrapers collect straight into Hotdata tables, where agents and applications can query it with SQL.

How it works

You give the bridge a scraper's dataset id and a CSV of inputs, such as URLs, search keywords, or usernames. It starts the collection on Bright Data, waits for it to finish, checks the records, and loads them into a Hotdata table. It works with any scraper in the catalog, whether it collects specific URLs or discovers results by keyword or category.

It's careful along the way. Your inputs are checked against the scraper's requirements before a paid collection starts, so a bad CSV doesn't cost you anything. Records are checked against the scraper's output schema before anything is written, and records Bright Data flags as errors are set aside instead of loaded.

Large collections are streamed to disk and uploaded to Hotdata in parts. Every step is saved locally, so if something fails, like a timeout or a schema problem, bdh replay picks up where it left off, and a run is never loaded twice.

Getting started

Install the bridge and set your keys:

pip install git+https://github.com/adomhamza/brightdata-hotdata-bridge
export BRIGHTDATA_API_KEY=...
export HOTDATA_API_KEY=...
export HOTDATA_WORKSPACE_ID=...

Find a scraper and see which inputs it needs:

bdh scrapers list --search amazon
bdh scrapers show gd_l7q7dkf244hwjntr0

Then collect and load in one step. Here, urls.csv has a url column:

bdh push --dataset-id gd_l7q7dkf244hwjntr0 --input-file urls.csv --table amazon_products

By default, each push replaces the table's contents. To keep earlier data, use --mode append, or --mode upsert --key <column> to keep one up-to-date row per record.

Try it out

A simple way to start is with one scraper and a short list of URLs. Push the results into Hotdata and query the table with the Hotdata CLI or hand it to an agent.

For the full set of commands and options, see the bridge repository.