Apache Doris Platform

Apache Doris is a high-performance real-time analytical database based on MPP (Massively Parallel Processing) architecture. Originally developed at Baidu as Palo, it graduated as an Apache Top-Level Project in 2022. BenchBox connects via the MySQL protocol using PyMySQL (port 9030) and supports Stream Load via the FE HTTP API (port 8030) for high-throughput data ingestion. SQLGlot provides official doris dialect support for SQL translation.

Apache Doris is used in production by Baidu, Xiaomi, ByteDance, JD.com, Meituan, and thousands of other enterprises globally. It delivers competitive performance against ClickHouse, StarRocks, and Trino on analytical workloads, with particularly strong support for real-time analytics and high-concurrency point queries.

Features

MySQL protocol connectivity - Standard MySQL wire protocol via PyMySQL (port 9030)
Stream Load HTTP API - High-throughput data ingestion via FE HTTP port 8030
Full TPC-H support - All 22 queries with row count validation
Full TPC-DS support - All 99 queries with row count validation
Official SQLGlot dialect - Native doris dialect for accurate SQL translation
Vectorized execution engine - High-performance columnar processing (Doris 2.0+)
Multiple table models - Duplicate, Aggregate, Unique, and Primary Key models
Distributed hash partitioning - Automatic data distribution across Backend nodes
Tuning support - Partitioning, sorting, and distribution tuning at table creation time

Quick Start

uv add pymysql

uv add benchbox --extra doris

export DORIS_HOST=localhost
export DORIS_PORT=9030

benchbox run --platform doris --benchmark tpch --scale 0.01

The first two commands are alternatives: install the PyMySQL dependency directly, or install it through the Doris extra. Doris must be running before you set the connection variables and run the benchmark.

Docker Quick Start

docker compose -f docker/doris/docker-compose.yml up --wait

export DORIS_HOST=localhost
export DORIS_PORT=19031
export DORIS_HTTP_PORT=18030

mysql -h 127.0.0.1 -P 19031 -u root -e "SELECT 1"

benchbox run --platform doris --benchmark tpch --scale 1.0

The docker compose command starts the repo-managed Apache Doris 4.0.3 stack and waits for FE and BE readiness. The environment variables point BenchBox at the mapped Doris ports, and the mysql command verifies connectivity before the benchmark runs.

The checked-in compose file uses the official apache/doris:4.0.3-all-slim image and applies a small amount of startup configuration for local Docker Desktop environments, including Doris' vm.max_map_count, ulimit -n, and Java requirements.

By default the stack runs without privileged mode. Set DORIS_PRIVILEGED=true only if your Docker VM still reports vm.max_map_count < 2000000 and you want the container to raise it at startup; preconfiguring the Docker VM once is the narrower alternative.

The compose file still defaults SKIP_CHECK_ULIMIT=true because Docker Desktop cannot satisfy Doris' swap-disabled preflight check. BenchBox's wrapper re-enforces the ulimit -n >= 60000 and vm.max_map_count >= 2000000 requirements before Doris starts, and emits a warning if swap remains enabled.

BenchBox records the Doris engine version from @@version_comment, not SELECT version(), because Doris exposes MySQL compatibility version 5.7.99 on the latter even when the engine itself is 4.x.

Known Limits In Low-Memory Docker Setups

The bundled all-in-one Docker configuration targets an 8 GB backend memory limit so it can run on typical developer laptops. That is enough for smoke tests and many SF=1 workloads, but it is not enough for every benchmark:

h2odb at SF=1 can exceed the BE limit while loading trips
tpch_skew at SF=1 can exceed the BE limit while loading orders

These are environment limits, not adapter correctness issues. Increase Docker memory and the Doris mem_limit if you need those datasets to load reliably at SF=1.

Validation Caveat

TPC-H and TPC-DS validation on Doris checks row counts only, so it can miss wrong result values.

Configuration Options

CLI options are passed as --platform-option KEY=VALUE.

Option	CLI	Environment Variable	Default	Description
`host`	yes	`DORIS_HOST`	`localhost`	Doris FE node hostname
`port`	yes	`DORIS_PORT`	`9030`	MySQL protocol port (FE)
`username`	yes	`DORIS_USER` / `DORIS_USERNAME`	`root`	Database username
`password`	yes	`DORIS_PASSWORD`	`""`	Database password
`database`	yes	`DORIS_DATABASE`	auto-generated	Target database name
`http_port`	yes	`DORIS_HTTP_PORT`	`8030`	FE HTTP port for Stream Load
`be_http_port`	yes	`DORIS_BE_HTTP_PORT`	`8040`	BE HTTP port for Stream Load redirects
`use_tls`	no	-	`false`	Use HTTPS for Stream Load API
`stream_load_chunk_size`	no	-	`10485760`	Stream Load chunk size in bytes for CSV/TPC inputs
`stream_load_max_filter_ratio`	no	-	`0`	Allowed fraction of filtered rows; default rejects silent partial loads

Options marked "no" are adapter constructor parameters; the benchbox run CLI does not accept them.

Data Loading

BenchBox loads data into Apache Doris using the Stream Load HTTP API when the requests library is available. When requests is not installed, it falls back to batch INSERT statements via the MySQL protocol. The adapter handles both TPC pipe-delimited (.tbl) and standard CSV formats automatically.

Loading Process

Schema creation - Tables are created with Doris-optimized DDL (Duplicate Key model, hash distribution)
Type conversion - DuckDB/standard SQL types are translated via the SQLGlot doris dialect
Stream Load (primary) - Data files are sent via HTTP PUT to the FE Stream Load endpoint on port 8030
INSERT fallback - If requests is not installed, data is loaded in batches of 1,000 rows using parameterized INSERT INTO ... VALUES statements
Constraint handling - Foreign keys are removed (Doris does not enforce them); primary keys map to Doris key models

Stream Load (HTTP API)

Apache Doris provides a high-throughput Stream Load API on the FE HTTP port (default 8030). BenchBox uses this as the primary data loading method:

Endpoint: http://<fe_host>:8030/api/<database>/<table>/_stream_load
Protocol: HTTP PUT with CSV payload and 100-continue header
Authentication: HTTP Basic Auth using Doris credentials
TPC format handling: Trailing delimiters are automatically stripped before loading
TLS support: Set the adapter use_tls parameter to use HTTPS for encrypted Stream Load transfers
Throughput: Significantly higher than row-by-row INSERT for large datasets

INSERT Fallback

When the requests library is not available, the adapter falls back to batch INSERT:

Rows are loaded in batches of 1,000 using executemany()
TPC pipe-delimited format is handled automatically
Suitable for small to medium datasets (up to SF 1)
For larger datasets, install requests to enable Stream Load

Table Models

Apache Doris supports four table models, each optimized for different workloads. BenchBox uses the Duplicate Key model by default for benchmark tables, as it preserves all rows without deduplication.

Model	Use Case	Key Behavior	BenchBox Usage
Duplicate Key	Analytics, logs	All rows preserved, no deduplication	Default for benchmarks
Aggregate	Pre-aggregation	Rows with same key are merged by aggregate functions	Not used
Unique Key	Dimension tables	Last write wins for same key	Not used
Primary Key	Real-time updates	Last write wins with merge-on-read	Not used

The Duplicate Key model is optimal for TPC-H and TPC-DS workloads because:

No deduplication overhead during data loading
All original rows are preserved for accurate query results
Sort keys can be specified for scan optimization

Usage Examples

Basic Benchmarks

benchbox run --platform doris --benchmark tpch --scale 1.0

benchbox run --platform doris --benchmark tpcds --scale 10.0

benchbox run --platform doris --benchmark tpch --queries Q1,Q6,Q17

Environment Variable Configuration

export DORIS_HOST=doris-fe.example.com
export DORIS_PORT=9030
export DORIS_USER=benchbox
export DORIS_PASSWORD=secret
export DORIS_DATABASE=benchmark_db
export DORIS_HTTP_PORT=8030

benchbox run --platform doris --benchmark tpch --scale 10.0

CLI Option Configuration

benchbox run --platform doris --benchmark tpch --scale 1.0 \
    --platform-option host=doris-fe.example.com \
    --platform-option port=9030 \
    --platform-option username=benchbox \
    --platform-option password=secret \
    --platform-option database=my_benchmarks

Dry Run (Preview)

benchbox run --platform doris --benchmark tpch --scale 1.0 --dry-run ./preview

TLS-Encrypted Stream Load

HTTPS for the Stream Load API (e.g. managed cloud deployments) is set with the adapter use_tls parameter. It has no benchbox run option.

Architecture

Adapter Structure

The Doris adapter is a single-file implementation:

Module	Class	Responsibility
`doris.py`	`DorisAdapter`	Connection management, schema creation, data loading, query execution, tuning

Connection Model

BenchBox CLI
    |
    v
DorisAdapter
    |
    +-- MySQL Protocol (PyMySQL) --> FE (port 9030)
    |       - Schema DDL
    |       - Query execution
    |       - Batch INSERT fallback
    |
    +-- HTTP API (Stream Load) --> FE (port 8030)
            - High-throughput CSV data ingestion
            - HTTP PUT with Basic Auth

Doris Cluster Ports

Port	Service	Protocol	Purpose
9030	FE MySQL	TCP	SQL queries, DDL, DML via MySQL protocol
8030	FE HTTP	HTTP/HTTPS	Stream Load API, web UI, REST API
8040	BE HTTP	HTTP	BE web server, internal data transfer
9010	FE Edit Log	TCP	FE metadata replication (internal)

Platform Information

At runtime, BenchBox captures platform metadata:

{
    "platform_type": "doris",
    "platform_name": "Apache Doris",
    "configuration": {
        "host": "localhost",
        "port": 9030,
        "database": "benchbox_tpch_sf1",
        "http_port": 8030,
        "stream_load_available": True
    },
    "platform_version": "4.0.x",
    "client_library_version": "1.x.x"
}

Tuning and Optimization

Automatic Benchmark Configuration

The adapter automatically applies session-level optimizations when running benchmarks:

SQL cache: Disabled (enable_sql_cache = false) for accurate timing
Parallel execution: Set to 8 fragment instances (parallel_fragment_exec_instance_num = 8)
Memory limit: Set to 8 GB for OLAP workloads (exec_mem_limit = 8589934592)

Supported Tuning Types

Tuning Type	Support	Notes
Partitioning	Yes	`PARTITION BY RANGE` clause at table creation
Sorting	Yes	Sort keys via Duplicate Key model key ordering
Distribution	Yes	`DISTRIBUTED BY HASH` clause at table creation
Clustering	No	No CLUSTER command
Primary Keys	Yes	Unique/Primary Key table models
Foreign Keys	No	No FK enforcement in Doris

Distribution Keys

Choosing effective distribution keys is critical for Doris query performance:

CREATE TABLE lineitem (
    l_orderkey BIGINT,
    l_partkey BIGINT,
    ...
) DUPLICATE KEY(l_orderkey)
DISTRIBUTED BY HASH(l_orderkey) BUCKETS 16;

Guidelines:

Use high-cardinality columns for even data distribution
Align distribution keys with common join predicates
Start with 8-16 buckets and adjust based on data volume

Bloom Filter and Bitmap Indexes

Doris supports secondary indexes for accelerating point queries and filter predicates:

ALTER TABLE lineitem SET ("bloom_filter_columns" = "l_orderkey, l_partkey");

CREATE INDEX idx_shipmode ON lineitem (l_shipmode) USING BITMAP;

Use a Bloom filter index for high-cardinality columns and a bitmap index for low-cardinality columns.

Colocate Join Groups

For frequently joined tables, colocate groups ensure data locality:

CREATE TABLE orders (
    o_orderkey BIGINT,
    ...
) DUPLICATE KEY(o_orderkey)
DISTRIBUTED BY HASH(o_orderkey) BUCKETS 16
PROPERTIES ("colocate_with" = "tpch_group");

CREATE TABLE lineitem (
    l_orderkey BIGINT,
    ...
) DUPLICATE KEY(l_orderkey)
DISTRIBUTED BY HASH(l_orderkey) BUCKETS 16
PROPERTIES ("colocate_with" = "tpch_group");

Colocate joins eliminate data shuffle across BE nodes for join operations on aligned distribution keys.

Managed Cloud Options

Several vendors offer managed Apache Doris cloud services:

Provider	Service	Description
VeloDB Cloud	velodb.io	Fully managed Doris by core contributors
SelectDB Cloud	selectdb.com	Enterprise managed Doris with compute-storage separation
ApsaraDB for SelectDB	Alibaba Cloud	Managed Doris on Alibaba Cloud infrastructure

When using managed cloud services:

The host, port, and credentials are provided by the service
Stream Load endpoints may use HTTPS (set the adapter use_tls parameter)
HTTP port may differ from the default 8030

Troubleshooting

Connection Refused

Error: Failed to connect to Doris

Solutions:

Verify Doris FE is running and accessible on the configured host and port
Check that port 9030 (MySQL protocol) is open and not blocked by a firewall
For Docker deployments, ensure port mapping is correct (-p 9030:9030)
Test connectivity directly: mysql -h <host> -P 9030 -u root

Missing PyMySQL Dependency

Error: Missing dependencies for doris platform: pymysql

Solutions:

Install PyMySQL: uv add pymysql
Or install the Doris extra: uv add benchbox --extra doris

Stream Load Failures

Error: Stream Load failed with status 503

Solutions:

Verify the FE HTTP port (default 8030) is accessible from the BenchBox host
Check that BE nodes are alive: SHOW BACKENDS; via MySQL client
Ensure the target table exists before loading data
For large files (>1 GB), consider increasing the timeout or using smaller batch files
Install the requests library: uv add requests

Schema Creation Failures

Error: critical CREATE TABLE statement(s) failed

Solutions:

Check that the Doris user has CREATE TABLE and CREATE DATABASE permissions
Verify the target database exists or the user has CREATE DATABASE privilege
Review Doris FE logs (fe.log) for detailed error messages
Ensure sufficient disk space and memory on BE nodes
Check for incompatible column types in the translated DDL

Slow Data Loading

Solutions:

Install requests to enable Stream Load (the recommended bulk-ingest path; INSERT fallback is row-by-row and typically 10x or more slower for large datasets)
Ensure the FE HTTP port (8030) is accessible for Stream Load
Increase bucket count for better parallelism during loading
Check network latency between BenchBox host and Doris cluster
Monitor BE resource utilization during loading

Query Timeout

Error: Query execution exceeded timeout

Solutions:

Check Doris resource utilization (CPU, memory, disk I/O) via SHOW PROC '/backends';
Verify data distribution is balanced across BE nodes
Run ANALYZE TABLE to update column statistics for the query optimizer
Review the query plan with EXPLAIN for performance bottlenecks
Consider increasing exec_mem_limit for memory-intensive queries

FilesExpand file tree

doris.md

Latest commit

History

doris.md

File metadata and controls

Apache Doris Platform

Features

Quick Start

Docker Quick Start

Known Limits In Low-Memory Docker Setups

Validation Caveat

Configuration Options

Data Loading

Loading Process

Stream Load (HTTP API)

INSERT Fallback

Table Models

Usage Examples

Basic Benchmarks

Environment Variable Configuration

CLI Option Configuration

Dry Run (Preview)

TLS-Encrypted Stream Load

Architecture

Adapter Structure

Connection Model

Doris Cluster Ports

Platform Information

Tuning and Optimization

Automatic Benchmark Configuration

Supported Tuning Types

Distribution Keys

Bloom Filter and Bitmap Indexes

Colocate Join Groups

Managed Cloud Options

Troubleshooting

Connection Refused

Missing PyMySQL Dependency

Stream Load Failures

Schema Creation Failures

Slow Data Loading

Query Timeout

See Also