Benchmarking Concurrency on Hotdata, Snowflake and BigQuery
For AI applications, concurrency is one of the most important performance metrics. Agents tend to make many requests at the same time, and the slowest requests impact the entire workflow. We are benchmarking Hotdata across several workloads, including TPC-H and ClickBench, with particular attention to how these systems perform as the number of concurrent requests increases.
Each point represents the sum of every query's best-of-three median execution time. Hotdata remains relatively flat across the tested range, Snowflake's response times increase once concurrency passes 10 streams while BigQuery scales just as flat as Hotdata but slower per-query overall.
Hotdata and Snowflake
The individual TPC-H queries show the same overall pattern: nearly every query remains stable on Hotdata as concurrency increases, while Snowflake shows increasing latency. The appendix below includes the complete per-query breakdown.
Hotdata and BigQuery
BigQuery shows a third pattern. Its execution time is also flat across concurrency, but the flat line sits about 4 times higher than Hotdata's. Summed across the same six concurrency levels, Hotdata holds 2.59–2.90s while BigQuery holds 11.27–12.96s, with no growth trend in either direction as concurrency increases.
Speedup (BigQuery ÷ Hotdata): 4.84× (n=1) → 4.37× (n=5) → 3.89× (n=10) → 4.65× (n=20) → 3.92× (n=30) → 4.18× (n=50). The ratio holds in a narrow band with no trend up or down as concurrency increases — BigQuery’s per-query overhead looks roughly fixed rather than a function of concurrent load.
Any increase in query latency can turn a short interaction into a prolonged multi-step wait.
Sizing and cost implications
The benchmark also shows the difference between Hotdata's base account and Snowflake's warehouse-based scaling model. Hotdata sustained a 50-fold increase in concurrent streams on the same base account, while Snowflake delivered similar throughput through 10 streams before it plateaued.
Snowflake supports additional concurrency by increasing warehouse capacity and hourly cost, while Hotdata remained on the base account at every concurrency level tested. For agent workloads, the system must maintain predictable latency and throughput as many independent requests arrive in parallel.
At low concurrency (1–10 streams), all five Snowflake sizes and Hotdata sit within a narrow ~2.6–3.5s band — warehouse size barely matters when there’s little contention. The lines fan out sharply from n=20 onward, and every larger size visibly pulls its curve down relative to the smaller ones by n=50, where concurrency pressure is highest (the ordering isn’t perfectly monotonic at n=30, where Small briefly runs slower than X-Small). Even X-Large, at 16× the credit cost of X-Small, still runs slower than Hotdata’s fixed node at every concurrency level tested.
Conclusion
Agents explore, retrieve, validate, and reason through thousands of data interactions in parallel. As more agents run concurrently, databases need to keep response times fast and predictable without requiring compute to scale linearly.
Agent workloads also shift the performance bottleneck from individual query speed to sustained concurrency. As more agents run simultaneously, the ability to maintain predictable, low-latency responses under load becomes just as important as raw query performance.
Appendix: methodology notes
Snowflake reports compilation, queueing, and execution separately, while Hotdata's execution path performs planning in-process. The Snowflake results use
EXECUTION_TIMEand exclude compilation and queueing. Result caching was disabled for both systems so repeated executions measured query processing rather than cached responses.
Appendix: every query's own trend
The aggregate results above hide substantial variation between queries. Each chart below shows one TPC-H query across the same six concurrency levels using an independent y-axis, since query durations range from tens of milliseconds to several seconds.