getting-started.md

Getting Started in 5 Minutes

Follow these steps to install BenchBox, verify your environment, and run a reproducible benchmark. Everything below works on macOS, Linux, and Windows with Python 3.11+.

Step 0 - Prerequisites

1. Install uv (recommended) or make sure you have pip available.
2. Ensure DuckDB can create temporary files in your working directory. No other services are required for the quick start.

mkdir benchbox-demo && cd benchbox-demo

Creating an isolated project directory is optional.

Step 1 - Install BenchBox with DuckDB

Install the duckdb extra. DuckDB is an optional dependency - a plain uv add benchbox installs the core package with SQLite only, and every command in this guide uses DuckDB.

uv add benchbox --extra duckdb

Prefer pip? python -m pip install "benchbox[duckdb]" (quote the brackets so your shell does not expand them).

Verify the CLI is on your PATH:

uv run -- benchbox --version

For other platforms, extras, and dependency checks, see the dedicated installation guide.

Step 2 - Profile Your Environment

The profile command confirms CPU, memory, and available adapters. BenchBox uses this information to suggest scale factors and concurrency levels.

uv run -- benchbox profile

Look for the Available Databases table. duckdb should be Ready because Step 1 installed the duckdb extra; if it is not, re-run the Step 1 install command. If you plan to use cloud platforms later, run uv run -- benchbox check-deps --platform <name>, or use the guided benchbox setup --platform <name> wizard once you have credentials handy.

Step 3 - Run Your First Benchmark

Run a minimal TPC-H benchmark to generate data, load it into DuckDB, and execute the standard power test.

uv run -- benchbox run \
  --platform duckdb \
  --benchmark tpch \
  --scale 0.01

Unattended execution? Add --non-interactive to skip prompts. BenchBox stores outputs under benchmark_runs/ by default.

Step 4 - Inspect the Results

Summarize the most recent run:

uv run -- benchbox results --limit 1

The results display shows timing, validation status, and per-query metrics from your benchmark execution.

Step 5 - Export Results (Optional)

Share your results in different formats without re-running the benchmark:

uv run -- benchbox export --last --format csv

uv run -- benchbox export --last --format html --output-dir ./reports/

uv run -- benchbox export --last --format json --format csv --format html

The commands export to CSV for spreadsheet analysis, generate an HTML report for sharing with your team, and export to all formats.

The export command is useful for:

• Creating shareable HTML reports for stakeholders
• Analyzing query performance in spreadsheets (Excel, Google Sheets)
• Archiving results in multiple formats
• Converting between formats without re-running expensive benchmarks

Optional - DataFrame Platforms

BenchBox also supports benchmarking DataFrame libraries using their native APIs. This enables direct comparison between SQL and DataFrame execution paradigms.

Quick Start with DataFrames

uv add benchbox --extra polars
uv run -- benchbox run --platform polars-df --benchmark tpch --scale 0.01

uv add benchbox --extra pandas
uv run -- benchbox run --platform pandas-df --benchmark tpch --scale 0.01

Polars and pandas each require an extra (polars and pandas); neither ships in the base install.

Compare SQL vs DataFrame

uv run -- benchbox run --platform duckdb --benchmark tpch --scale 0.1
uv run -- benchbox run --platform polars-df --benchmark tpch --scale 0.1

These commands run the same benchmark with different paradigms: DuckDB runs SQL, and Polars runs DataFrame operations.

For more details, see the DataFrame Platforms Guide.

Optional - Python API in 15 Lines

import duckdb
from benchbox import TPCH

conn = duckdb.connect(":memory:")
benchmark = TPCH(scale_factor=0.01, output_dir="./tpch_data")

benchmark.generate_data()
conn.execute(benchmark.get_create_tables_sql())

for table, path in benchmark.tables.items():
    conn.execute(
        f"COPY {table} FROM '{path}' (DELIMITER '|' NULL '' HEADER FALSE);"
    )

rows = conn.execute(benchmark.get_query(1)).fetchall()
print(f"Query 1 returned {len(rows)} rows")

Next Steps

Essential Guides

• CLI Quick Reference, Complete command reference with examples
• Configuration Handbook, CLI flags, config files, and advanced options
• Examples Guide, Code snippets and automation patterns

Understanding BenchBox

• Architecture Overview - How BenchBox components work together
• Workflow Patterns - Common benchmarking workflows
• Data Model - Understanding result schemas and analysis
• Glossary - Benchmark terminology reference

Expanding Your Testing

• Platform Selection Guide - Choosing the right database
• Platform Quick Reference - Setup for each platform
• DataFrame Platforms - Native DataFrame API benchmarking
• Benchmark Catalog - Available benchmarks beyond TPC-H
• Data Generation Guide - Advanced data generation options

Troubleshooting

• Troubleshooting Guide - Common issues and solutions
• Dry Run Mode - Preview queries before execution