Skip to content

configuration.md

Latest commit

 

History

History
124 lines (83 loc) · 4.61 KB

configuration.md

File metadata and controls

124 lines (83 loc) · 4.61 KB
Raw

Configuration Handbook

BenchBox reads configuration from three layers, highest precedence first:

  1. CLI options - flags supplied to benchbox run, benchbox export, etc.
  2. Environment variables - primarily tuning overrides (see below).
  3. Configuration files - benchbox.yaml in the working directory or ~/.benchbox/config.yaml.

If no file is present, BenchBox uses sensible defaults. You can generate a tuning template with uv run -- benchbox tuning init --platform duckdb.

Minimal Example

Save the following as benchbox.yaml:

benchmarks:
  default_scale: 0.1
  continue_on_error: false

database:
  preferred: duckdb
  connection_timeout: 30

output:
  directory: ./benchmark_runs/results
  formats: [json, console]

execution:
  power_run:
    iterations: 3
    warm_up_iterations: 1
  concurrent_queries:
    enabled: false

Save the file next to your project and run:

uv run -- benchbox run --benchmark tpch --platform duckdb

CLI flags still win. For example --scale 1 overrides benchmarks.default_scale for that invocation only.

File Locations & Schema

BenchBox searches in this order:

  1. ./benchbox.yaml
  2. ~/.benchbox/config.yaml

Files are parsed as YAML and merged with built-in defaults defined in benchbox.cli.config.ConfigManager. Unknown keys are preserved so you can store project metadata alongside official sections (system, database, benchmarks, output, execution, tuning).

Use these helpers when you need to inspect or persist configuration programmatically:

from benchbox.cli.config import ConfigManager

config = ConfigManager()
scale = config.get("benchmarks.default_scale")
config.set("execution.power_run.iterations", 5)
config.save_config()

Execution Profiles

benchbox.utils.ExecutionConfigHelper wraps common tuning operations. It works with or without an existing config file.

from benchbox.cli.config import ConfigManager
from benchbox.utils import ExecutionConfigHelper

config = ConfigManager()
helper = ExecutionConfigHelper(config_manager=config)

helper.enable_power_run_iterations(iterations=3, warm_up_iterations=1)

helper.enable_concurrent_queries(max_concurrent=4)
helper.optimize_for_system(cpu_cores=16, memory_gb=64)
config.save_config()

enable_power_run_iterations enables a quick power run profile. enable_concurrent_queries turns on concurrent streams, and optimize_for_system optimizes for the hardware you give it.

The helper updates the configuration object it captured at construction. This example passes the same ConfigManager that saves the changes. In standalone code without a registered provider, ExecutionConfigHelper() captures a fresh in-memory provider; its changes stay local to that helper and are not persisted. Later provider registrations do not replace an existing helper's captured provider. Pass a ConfigManager explicitly when you intend to save the settings.

Environment Overrides

Some tuning settings can be toggled at deploy time without editing files. The default mapping is stored under tuning.environment_overrides:

Environment variable Maps to Typical values
BENCHBOX_TUNING_CONFIG tuning.default_config_file Path to a YAML file

Point BENCHBOX_TUNING_CONFIG at a configuration checked into your repo to make it the default whenever --tuning tuned is used (see Tuning Commands for the full --tuning resolution order). To actually enable tuned runs in CI, pass --tuning tuned (or --tuning auto) on the benchbox run command line; there is no environment variable that turns tuning on by itself.

Validating Configuration

Before running large jobs, dry-run the plan and validate dependencies:

uv run -- benchbox run --dry-run ./plan --platform duckdb --benchmark tpch

uv run -- benchbox check-deps --matrix

The first command renders the execution plan without running anything. The second checks the platform requirements declared in the config.

benchbox run respects values from benchbox.yaml, so you can set project defaults once and execute repeatable runs with only a few flags.

Related Commands

  • uv run -- benchbox tuning init - scaffold a tuning YAML file for a specific platform.
  • uv run -- benchbox validate --config benchbox.yaml - ensure configuration syntax and schema are valid.
  • uv run -- benchbox platforms setup - interactively enable adapters defined in your config.

See the CLI reference for detailed command usage and the examples library for advanced automation patterns.