FlagSparse

GPU sparse operations package (SpMV, SpMM, SpGEMM, SDDMM, gather, scatter, sparse formats).

Bringing this up on a non-NVIDIA accelerator? Start at prompt.md, then read docs/<BACKEND>.md (how to run there) and modified/<BACKEND>.md (what previous work changed, and why). This README documents the CUDA reference path.

Install

pip install . --no-deps --no-build-isolation

Use --no-build-isolation to avoid downloading build deps when offline.

Runtime dependencies (install when needed):

pip install torch triton cupy-cuda12x

Reproducing the delivery run (40 variants, 30 matrices)

This is the run whose summary.json is the deliverable: accuracy and performance for the 40 registered delivery variants in conf/operators.yaml (delivery_variants; the delivery list has 42 -- sddmm_csr c32/c64 wait for a complex kernel), performance measured on a directory of 30 MatrixMarket matrices with 5 warmup and 20 timed iterations. Every backend runs the same command; the table below lists what changes per backend, and each docs/<BACKEND>.md has the full recipe in its "交付复现" section.

export PYTHONPATH=$PWD/src
python3 -c "import flagsparse; print(flagsparse.__file__)"   # must be <repo>/src/flagsparse/__init__.py

timeout -s KILL 21600 python3 run_flagsparse_pytest.py \
  --phase both --mode normal --delivery-only --gpus 0 --timeout 3600 \
  --benchmark-input <dir with the 30 .mtx> --benchmark-warmup 5 --benchmark-iters 20 \
  --results-dir pytest_results_<backend>_delivery

python3 tools/delivery_table.py pytest_results_<backend>_delivery      # 40-row table, exit 1 if any is missing

Flag	Why it is not optional
--delivery-only	Runs exactly the 11 parents behind the 40 variants, and narrows each benchmark sweep to the delivery axes (int32 indices, non op). Without it the runner reads the 18-operator superset and the full int64/trans/conj grid
--mode normal	quick keeps one shape per class, drops ~40% of the cases and skips the two historically failing ones
--timeout 3600	Per operator per phase (per matrix for per-matrix routes). The default 0 never gives up on a hung kernel
outer timeout -s KILL	A deadlocked triangular solve blocks in the driver; Ctrl-C does not reach it

Backend	Set before the command	Differs from the command above	Speedup is against	Accuracy reference	Recipe
CUDA	-- (pip install cupy-cuda12x)	--	cuSPARSE (CuPy)	cuSPARSE + torch	this section
DCU / ROCm	pip install hip-python	SpSV/SpSM may deadlock (read as Timeout)	hipSPARSE	hipSPARSE + torch	docs/DCU.md §0.5
MetaX C550	FLAGSPARSE_BACKEND=metax FLAGSPARSE_MACA_VENDOR=none	--op-benchmark-args='sddmm_csr=--no-cusparse'; SDDMM K sweep needs --timeout 4500	PyTorch	SciPy (CPU)	docs/MACA.md §0.5
Moore Threads	FLAGSPARSE_BACKEND=mthreads	use run_flagsparse_split_delivery.py (performance from the C API)	muSPARSE (C API)	SciPy (CPU)	docs/MUSA.md §0.5
Ascend 910B	CANN set_env.sh; FLAGSPARSE_BACKEND=ascend FLAGSPARSE_ASCEND_VENDOR=torch	--gpus 6,7 only	PyTorch-NPU (5 ops; others probe only)	SciPy (CPU)	docs/ASCEND.md "交付复现"
Kunlunxin XPU	FLAGSPARSE_BACKEND=xpu FLAGTREE_BACKEND=xpu TRITON_BACKEND=xpu	--	PyTorch-XPU (5 ops; others probe only)	SciPy (CPU)	docs/XPU.md §1.5

Results produced before 86a09cd (2026-09-18) are not comparable: the old projection averaged int64 and trans/conj rows into each ..._int_non variant and counted rows without a speedup as 0. Rerun with the current runner; see prompt.md section 2.

What the statuses mean in summary.json: Passed / Failed as measured; Timeout the operator ran and hit --timeout; NotFound no row for that variant (not run, or not configured on this backend); NoBaseline (C API) the kernel ran and passed but the vendor library has no such dtype/operator to compare against. On CUDA (RTX 5090, 2026-09-17, before sweep narrowing) the full run took 79 minutes, SpSV and SpGEMM being the longest (~20 min each).

Backends (8 registered)

FlagSparse dispatches its vendor reference and baseline paths on the detected runtime; the Triton kernels themselves are unchanged across backends.

Runtime	Detected by	Vendor sparse library	Python binding
NVIDIA CUDA	torch.version.hip is None	cuSPARSE	CuPy (cupy-cuda12x)
DCU / ROCm	torch.version.hip is not None	hipSPARSE	hip-python
MetaX / MACA (C550)	see _detect_maca_runtime()	CuPy when it is really installed, else torch -- probed, not assumed: the C550 this was brought up on has none	CuPy
Moore Threads / MUSA	torch.musa available	none by default -- torch.sparse builds CSR/COO tensors there but registers no sparse matmul, measured on MTT S5000	torch_musa
Ascend / CANN (910B)	torch.npu available	torch -- ops-sparse stays one env var away for a vendor A/B	torch_npu
Kunlunxin XPU	vendor plugin importable (torch_xmlir / torch_xpu)	none -- XDNN is a fixed operator set, not a descriptor API, so there is nothing to bind	torch_xmlir
Enflame GCU	torch_gcu importable	none yet	torch_gcu
Cambricon MLU	env-routed only, never auto-detected: it is the generic reserve slot, so a vendor with no entry of its own can be driven through it	none yet	torch_mlu

The last three carry no per-operator kernels of their own: backends/{xpu,gcu,mlu}/ hold only __init__.py, so they resolve to the shared implementation. The XPU namespace check is not excess caution -- upstream PyTorch ships a torch.xpu namespace for Intel GPUs, so accepting the namespace alone would claim an Intel card as Kunlunxin silicon.

⚠️ Moore Threads and Ascend are not CUDA-compatible: torch exposes them as separate device types (musa / npu), so torch.cuda.* and Tensor.is_cuda do not apply. The device abstraction migration is done: no operator module calls torch.cuda.* any more -- they go through _ACCEL.* and _is_accel_tensor(). The only direct uses left are the runtime probes inside _common.py itself (11, all device-name detection or the OOM fallback). On CUDA, ROCm and MACA _ACCEL is torch.cuda, so those three backends are byte-for-byte equivalent and no kernel was touched. The standalone benchmark scripts in tests/ take the same pair of names from tests/benchmark_utils.py.

MACA is CUDA-source-compatible, so torch.version.cuda is set and torch.version.hip is None there — the ROCm probe cannot tell MetaX from NVIDIA. Detection walks, in order: FLAGSPARSE_BACKEND, a MetaX-specific torch.version attribute, the MACA SDK environment (MACA_PATH / MACA_HOME), then the device name (vendor tokens, then the c550 / c500 model strings). On a real machine, pin it explicitly until the automatic probe is confirmed:

export FLAGSPARSE_BACKEND=metax     # cuda | rocm | metax | mthreads | ascend | xpu | gcu | mlu
export FLAGSPARSE_MACA_MODEL=c550   # overrides model detection
export FLAGSPARSE_MACA_VENDOR=none  # skip the vendor baseline if CuPy is unusable

# Baseline choice for Moore Threads / Ascend
export FLAGSPARSE_MTHREADS_VENDOR=none       # none (default) | torch | musparse
export FLAGSPARSE_ASCEND_VENDOR=torch        # torch (default) | ops_sparse | none
export FLAGSPARSE_XPU_VENDOR=torch           # torch (default) | none

# Correctness reference. CUDA and ROCm compare against their vendor library plus
# torch; every other backend compares against SciPy on CPU, because torch.sparse
# there is another thing under test rather than a reference. Forcing "scipy" on a
# CUDA box is how that path gets exercised before it ships.
export FLAGSPARSE_ACCURACY_REFERENCE=auto    # auto (default) | scipy | torch

Tuning is per model, in _MACA_SPMV_PROFILES (spmv_csr.py) and _MACA_SPSV_PROFILES (spsv.py). C550 — FlagTree's reference metax part — is the only profile so far and is seeded from the CUDA path; an unrecognised model falls back to it. Each knob carries the ROCm value measured on gfx936 as a comment, deliberately not inherited.

On a DCU/ROCm host, install the hip-python bindings matching your ROCm release:

pip install hip-python

Both bindings are optional. When neither is importable the benchmarks fall back to the portable torch.sparse reference and report the reason in the *_reason / backend_status fields rather than failing.

Selection happens in _*_sparse_ref_backend() helpers, which return ("hipsparse" | "cupy_cusparse" | None, reason):

• flagsparse.sparse_operations._common - SpMV CSR/COO
• .spmm_csr / .spmm_coo / .spgemm_csr / .gather_scatter - the remaining operators

Running the tests on DCU

Prefix every command with PYTHONPATH=src, and first confirm you are not running a stale installed copy — the most common DCU pitfall:

python -c "import flagsparse; print(flagsparse.__file__)"   # must be <repo>/src/flagsparse/__init__.py

1. Diagnose before benchmarking. A hipSPARSE misuse hangs instead of raising, so probe phase by phase rather than starting with --op all:

python tests/diagnose_hipsparse_ref.py --op env        # environment probe, touches no operator
python tests/diagnose_hipsparse_ref.py --timing-only   # HIP event timing chain
python tests/diagnose_hipsparse_ref.py --op spmv-csr   # then one operator at a time
python tests/diagnose_hipsparse_ref.py --op all        # only once every single probe passes

2. Correctness suite.

PYTHONPATH=src python -m pytest tests/pytest -q

SpSV and SpSM currently deadlock in the GPU kernel on DCU (see the known-limits section of docs/DCU.md), so exclude them for a run that terminates:

PYTHONPATH=src python -m pytest tests/pytest -q \
  --ignore=tests/pytest/test_spsv_csr_accuracy.py \
  --ignore=tests/pytest/test_spsv_coo_accuracy.py \
  --ignore=tests/pytest/test_spsv_sell_accuracy.py \
  --ignore=tests/pytest/test_spsm_accuracy.py

DCU baseline: 984 passed / 1 failed in ~60 s, with 851 SpSV/SpSM tests excluded. The single failure is spmv_coo / spmv_csc tolerance jitter — a different dtype parameter each run, and it passes when run alone. Only a failure whose parameter stays fixed is a real one. CUDA baseline for the full suite: 1613 passed / 3 failed.

3. Policy/contract tests — no GPU needed, runs in seconds:

python -m pytest tests/ci -q     # criterion: 0 failed (the pass count grows as
                                 # tests are added: 102 passed / 3 skipped on 2026-09-19)

4. Per-operator benchmarks:

M=matrix   # any directory of .mtx files
python tests/test_spmv.py     $M --warmup 2 --iters 5
python tests/test_spmm.py     $M --warmup 2 --iters 5
python tests/test_spgemm.py   $M --warmup 2 --iters 5
python tests/test_spmm_coo.py $M --warmup 2 --iters 5

Make sure no other job is competing for the GPU before trusting the timings.

5. Unified runner. For the delivery run use the command in Reproducing the delivery run. Its --timeout 3600 is what keeps a deadlocked SpSV/SpSM from stalling the sweep: the operator is recorded as Timeout and the run moves on. Note that --gpus 0,1 does not help on its own: it splits the operators into two queues, and whichever queue holds SpSV/SpSM still blocks until the timeout fires.

For the full DCU bring-up procedure — environment checks, the stale-install trap, how to confirm hipSPARSE was actually selected, known limits, and a troubleshooting table — see docs/DCU.md.

Running the tests on MetaX / MACA C550

For the delivery run (40 variants) use the command in Reproducing the delivery run with the MetaX settings from its table; docs/MACA.md §0.5 has the exact command.

The command below is a different thing: a full sweep of the other operators as well (CSC, BSR, ...), with the PyTorch baseline, over 30 MatrixMarket inputs. SpSV and SpSM are excluded because their current kernels can hang on this platform, and SpMM BELL is excluded because a single matrix can take much longer than the rest of the sweep put together:

PYTHONPATH=src python -u run_flagsparse_pytest.py --phase both --mode quick --gpus 0 \
  --ops gather,scatter,spmv_csr,spmv_coo,spmv_csc,spmv_bsr,spmm_csr,spmm_coo,spmm_bsr,spmm_csc,spgemm_csr,sddmm_csr \
  --benchmark-input /root/gcx/matrix --benchmark-warmup 5 --benchmark-iters 20 \
  --op-benchmark-args='sddmm_csr=--no-cusparse' --op-benchmark-args='spmv_bsr=--resume' \
  --timeout 7200 --results-dir pytest_results_metax_runner_both_w5_i20

--benchmark-args is split and appended to every selected performance script. Use the repeatable --op-benchmark-args=OP=ARGS for an argument only one script supports; the command above sends --resume to BSR alone. Quote ARGS when it contains spaces. For a BSR-only resume, reuse the same result directory with:

PYTHONPATH=src python -u run_flagsparse_pytest.py --phase performance --gpus 0 \
  --ops spmv_bsr --benchmark-input /root/gcx/matrix \
  --benchmark-warmup 5 --benchmark-iters 20 \
  --benchmark-args="--no-cusparse --resume" --timeout 7200 \
  --results-dir pytest_results_metax_spmv_bsr

The BSR script retains completed PASS/FAIL cases, retries prior ERROR cases, and reports bsr_speedup_vs_pytorch only for accuracy PASS rows.

The result directory contains per-operator accuracy and performance files plus the combined summary. For backend checks, baseline details, and known C550 limitations, see docs/MACA.md.

For MetaX SDDMM, run only the currently supported float32,float64 value dtypes. The MACA build's torch.sparse.sampled_addmm returns incorrect sampled-dot values, so the SDDMM runner uses --no-cusparse and benchmarks the independent PyTorch sampled-dot reference (sum(X[row] * Y[col])), recorded as pytorch_ms.

That reference plays two roles, split by dtype on every backend: it is timed at the operator's own dtype, so pytorch_ms is a like-for-like baseline, while fp32 correctness is still checked against an fp64 evaluation run outside the timed window -- an oracle at the same precision as the code under test cannot separate a real accumulation bug from rounding both sides share. triton_speedup_vs_pytorch is filled on MetaX only; on CUDA and ROCm it is left empty so the vendor metric stays the reported one (this reference materialises nnz x K temporaries and is not an optimised SDDMM). A dedicated full C550 SDDMM run is:

PYTHONPATH=src python -u run_flagsparse_pytest.py --phase both --mode normal --gpus 0 \
  --ops sddmm_csr --benchmark-input /root/gcx/matrix \
  --benchmark-warmup 5 --benchmark-iters 20 --op-benchmark-args='sddmm_csr=--no-cusparse' \
  --timeout 7200 --results-dir pytest_results_metax_sddmm_csr_pytorch_full_w5_i20

On MetaX, test_spmm_csc.py uses a direct PyTorch CSC baseline: torch.sparse_csc_tensor followed by torch.sparse.mm. CSC format construction is outside the timed window; the CSV records the baseline as pytorch_ms and reports triton_speedup_vs_pytorch. The --no-cusparse option disables only the optional vendor baseline, not this PyTorch CSC baseline. The COO path remains a correctness reference.

Running the tests on Ascend 910B

Only NPU cards 6 and 7 may be used. The delivery run is the command in Reproducing the delivery run with the Ascend settings from its table; docs/ASCEND.md has the exact command, the five operators that are benchmarked against PyTorch-NPU (the rest are capability probes), environment checks and known limits.

Layout

Path	What it is
src/flagsparse/	Core package. sparse_operations/ holds one shared implementation per operator; sparse_operations/backends/<backend>/ is an override layer that is empty by default, so only a file that genuinely diverged shows up there
conf/operators.yaml	The operator inventory, and the single source of truth for the 40 registered delivery variants (the delivery list has 42; sddmm_csr c32/c64 wait for a complex kernel) (gather_f32_int, spmv_csr_f32_int_non, ...). Both front ends read it through tools/delivery_variants.py
run_flagsparse_pytest.py	The unified runner: accuracy and performance per operator, writes summary.json / summary.csv / result.html keyed by delivery variant
tests/pytest/	Accuracy suites (shared across backends; the golden reference stays on CPU)
tests/ci/	Policy and contract tests. No GPU needed
tests/*.py	Standalone per-operator benchmark CLIs, also driven by the runner's performance phase
benchmark/	Backend probes and vendor-baseline benchmarks
capi/	The cuSPARSE-compatible C API wrapper, its CTest suite and per-backend baselines
docs/	One document per backend: how to run on that machine. docs/README.md is the index
modified/	One change ledger per backend: what was changed on that machine, where, and why -- so backends exchange descriptions instead of whole files
prompt.md	Start here when bringing up a new backend

Tests

Run from project root, or cd tests then run scripts (paths like ../matrix for .mtx dir).

The commands below are the repository's documented invocation standard. CPU-only install, build, help-text, and smoke paths are checked in CI; GPU-specific examples are documented but not executed there unless you opt into the triton smoke job locally.

Operator test runners - YAML-driven accuracy/performance runs by operator:

python run_flagsparse_accuracy.py --list-ops
python run_flagsparse_accuracy.py --mode quick --gpus 0
python run_flagsparse_performance.py --ops spmv_csr,spmm_csr --benchmark-input matrix --benchmark-warmup 5 --benchmark-iters 20
python run_flagsparse_pytest.py --phase both --mode quick --gpus 0,1 --benchmark-input matrix --results-dir pytest_results

By default, run_flagsparse_accuracy.py and run_flagsparse_performance.py read operator ids from conf/operators.yaml, filter by --stages, and distribute operators across --gpus. run_flagsparse_pytest.py --phase both remains available when one command should run both phases. --ops and --op-list override the YAML selection. The default sweep excludes manual-test entries alpha_spmm_alg1 and spmv_coo_tocsr; include them explicitly with --ops or --op-list when needed. Helper APIs such as spsv_descriptor_api and sparse_format_constructors are not operator test entries.

The accuracy phase launches pytest tests/pytest -m <operator marker> --mode quick|normal --record json --output <op>/accuracy_result.json and uses synthetic CUDA data. The performance phase launches the configured tests/test_*.py benchmark command for each operator; MatrixMarket-backed commands receive --benchmark-input (default tests/data, or pass matrix for the local matrix directory), and the CSV output is also normalized into a FlagGems-style <op>/performance_result.json. Results are written under pytest_results_<timestamp>/ unless --results-dir is provided. Each operator directory contains accuracy_stdout.log, accuracy_stderr.log, accuracy_result.json, accuracy_detail.json, performance_stdout.log, performance_stderr.log, performance.csv, performance_result.json, and performance_detail.json when those phases run. The root summary.json uses the FlagGems timestamp / env / result structure. FlagSparse-only fields such as GPU id, commands, logs, totals, parsed pytest cases, and normalized benchmark records are kept in summary_flat.json and the per-operator *_detail.json files. summary.csv and optional summary.xlsx provide table-friendly views, and result.html is generated automatically for browser inspection. The generated result.html is rendered from summary_flat.json; summary.json remains the compact FlagGems-compatible summary for external tools.

Direct pytest accuracy suite - development-oriented accuracy checks, selectable by marker:

pytest tests/pytest --mode quick
pytest tests/pytest --mode normal -m "spmv_csr or spmm_csr"
pytest tests/pytest --mode quick -m "spmv_coo_tocsr"

Backend-owned test suites - each accelerator has a separate profile under tests/backends/; the profile owns its runtime selector and C API status while the operator oracle remains shared:

python tools/run_backend_tests.py --backend cuda --phase accuracy --mode quick
python tools/run_backend_tests.py --backend rocm --phase both --ops spmv_csr,spmm_csr

The six profiles are cuda, rocm, maca, musa, ascend, and xpu. Their corresponding C API profiles live in capi/ctest/backends/; only CUDA and MUSA currently have C API adaptors, so the other four remain Python-only.

Delivery report variants - Python and C API summaries use the same 40-entry delivery_variants registry in conf/operators.yaml. The summary always has every registry key. A selected parent operator contributes only its own dtype's measured cases; a missing run is reported as NotFound. Add future reportable variants to this registry first, then wire their test and benchmark producer without changing either summary schema.

When adding or changing an operator test entry, keep the implementation/API registration, conf/operators.yaml entry, pytest marker in pytest.ini, accuracy test, performance command, and public replacement/export registration in sync.

test_spmv.py - CSR SpMV (SuiteSparse .mtx, synthetic, or CSR CSV export):

python tests/test_spmv.py <dir_or_file.mtx>              # batch run, default float32
python tests/test_spmv.py <dir/> --dtype float64         # optional: --index-dtype int32|int64, --warmup, --iters, --no-cusparse
python tests/test_spmv.py --synthetic                    # synthetic benchmark
python tests/test_spmv.py <dir/> --csv-csr results.csv   # all value×index dtypes -> one CSV (per-matrix lines while running)

test_spmv_coo.py - COO SpMV (requires --synthetic or --csv-coo; no standalone .mtx batch):

python tests/test_spmv_coo.py --synthetic
python tests/test_spmv_coo.py <dir/> --csv-coo out.csv

test_spmv_opt.py - SpMV baseline vs optimised A/B (float32 / float64 only):

python tests/test_spmv_opt.py <dir_or_file.mtx> [...]
python tests/test_spmv_opt.py <dir/> --csv out.csv

test_spmv_bsr.py - native BSR SpMV with padded block-grid output:

python tests/test_spmv_bsr.py --synthetic --ops non,trans,conj
python tests/test_spmv_bsr.py <dir/> --csv-bsr out.csv --block-dims 2,4 --ops non,trans,conj --alg compare
# correctness uses BSR-expanded COO as the exact reference; PyTorch BSR is a baseline only.
# --alg blockrow_reduce runs the non-only block-row tile reduction path; compare keeps trans/conj on base.
# --resume retains completed PASS/FAIL CSV cases and retries prior ERROR cases.

test_spmm.py - CSR SpMM (.mtx batch, synthetic, or --csv):

python tests/test_spmm.py <dir_or_file.mtx>
python tests/test_spmm.py --synthetic                    # optional: --ops non,trans,conj
python tests/test_spmm.py <dir/> --csv results.csv      # float32/float64/complex64/complex128 + int32/int64 + ops grid
# common options: --dtype, --index-dtype, --ops, --dense-cols, --block-n, --block-nnz, --max-segments, --warmup, --iters, --no-cusparse
# CSR SpMM supports op="non" (A @ B), op="trans" (A.T @ B), and op="conj" (A.conj().T @ B).

test_spmm_opt.py - CSR SpMM baseline vs optimised A/B:

python tests/test_spmm_opt.py <dir_or_file.mtx> --dense-cols 32
python tests/test_spmm_opt.py <dir/> --csv spmm_opt.csv  # optional: --dtype float32|float64, --dense-cols
# common options: --dtype, --dense-cols, --warmup, --iters

test_spmm_coo.py - native COO SpMM:

python tests/test_spmm_coo.py <dir_or_file.mtx>
python tests/test_spmm_coo.py --synthetic                # optional: --op non|trans|conj|all, --route rowrun|atomic|compare
python tests/test_spmm_coo.py <dir/> --csv out.csv      # only --route rowrun or atomic (not compare); optional: --op all
# same tuning flags as CSR SpMM where applicable: --op, --dense-cols, --block-n, --block-nnz, --warmup, --iters, --no-cusparse

test_spmm_bsr.py - native BSR SpMM with padded block-grid output:

python tests/test_spmm_bsr.py --synthetic --block-dims 2 --ops non
python tests/test_spmm_bsr.py <dir/> --csv-bsr out.csv --block-dims 2 --ops non --dense-cols 32
# correctness uses the same BSR arrays expanded to COO as Ref=torch_spmm_coo; PyTorch/CuPy BSR are same-format baselines only when available.

test_sddmm.py - CSR SDDMM (.mtx batch or --csv):

python tests/test_sddmm.py <dir_or_file.mtx> --k 64
python tests/test_sddmm.py <dir/> --csv out.csv --no-cusparse  # dtype defaults: float32,float64
# common options: --dtype float32|float64, --index-dtype, --acc_mode f32|f64, --k,
# --alpha, --beta, --warmup, --iters, --no-cusparse, --skip-api-checks

test_spgemm.py - CSR SpGEMM (.mtx batch or --csv):

python tests/test_spgemm.py <dir_or_file.mtx> --input-mode auto
python tests/test_spgemm.py <dir/> --csv results.csv     # optional: --dtype float32|float64, --input-mode auto|a_equals_b|a_at, --compare-device cpu|gpu
# common options: --dtype, --index-dtype, --warmup, --iters, --input-mode, --adaptive-loops, --no-cusparse, --ref-blocked-retry, --ref-isolated-retry, --ref-block-rows, --compare-device, --run-api-checks

test_spsv.py - SpSV (triangular solve; square matrices only). CSR and COO share this script; there is no test_spsv_coo.py.

test_spsv_sell.py - lower, UNIT/NON_UNIT, real/complex, native column-major SELL SpSV with NON/TRANS/CONJ operation modes. Its CSV and terminal fields follow the CSR SpSV output. FlagSparse_ms and cuSPARSE_ms both cover every per-call preparation/analysis plus solve; static descriptors and SELL conversion are outside the timed interval. The direct flagsparse_spsv_sell API defaults to ALG1; use --alg_num 2 or the explicit flagsparse_spsv_analysis_sell + flagsparse_spsv_solve_sell lifecycle for the slice-cooperative ALG2 path. TRANS/CONJ use a dedicated reverse-dependency kernel and do not accept --alg_num or --alg2-workers.

python tests/test_spsv.py --synthetic
python tests/test_spsv.py <dir/> --csv-csr spsv.csv
python tests/test_spsv.py <dir/> --csv-coo out.csv      # same CSV columns as CSR
pytest -q -s tests/test_spsv_sell.py
python tests/test_spsv_sell.py <dir_or_file.mtx> --csv sell_alg1.csv --slice-size 32 --alg_num 1
python tests/test_spsv_sell.py <dir_or_file.mtx> --csv sell_alg2.csv --slice-size 32 --alg_num 2
python tests/test_spsv_sell.py <dir_or_file.mtx> --csv sell_unit.csv --unit-diagonal
python tests/test_spsv_sell.py --csv sell_trans.csv --dtype float32 --slice-size 32 --ops TRANS <dir_or_file.mtx>
python tests/test_spsv_sell.py --csv sell_conj.csv --dtype complex64 --slice-size 32 --ops CONJ <dir_or_file.mtx>
python tests/test_spsv_sell.py <dir_or_file.mtx> --csv sell_complex.csv --dtype complex
# Optional ALG2 tuning: append --alg2-workers 32|64|128|256|512

test_spsm.py - SpSM (triangular matrix-matrix solve; square matrices only):

python tests/test_spsm.py --synthetic --n 512 --rhs 1024
python tests/test_spsm.py <dir/> --csv-csr spsm_csr.csv --rhs 1024
python tests/test_spsm.py <dir/> --csv-coo spsm_coo.csv --rhs 1024

test_gather.py / test_scatter.py - gather/scatter benchmarks (pytest or python tests/test_gather.py).

Accuracy suites should use tests/pytest/accuracy_utils.py for FlagGems-style golden reference and tolerance policy. Numeric compute operators compare against CPU-FP64 golden references cast back to the dtype under test, while exact/logical outputs compare against CPU int32 references.

CI/CD

• .github/workflows/ci.yml is CPU-only and runs compile, format checks, lint, source-critical static checks, build, install, and smoke tests on GitHub-hosted runners.
• The smoke set now covers installed-wheel validation, packaging metadata, public API surface, operator registry consistency, shared runtime policy helpers, CLI --help, and README command snippets.
• conf/operators.yaml is the FlagGems-style operator interface registry for public FlagSparse sparse operators used by the unified test runner.
• .github/workflows/nightly-cpu.yml is a main-branch-only nightly CPU check that repeats the package, lint, and shared-runtime smoke tests.
• .github/workflows/release.yml builds source and wheel artifacts, then attaches them to GitHub Releases on v* tags.
• .github/workflows/triton-smoke.yml is a manual opt-in job for triton-dependent smoke checks.
• .github/workflows/gpu-ci.yml is a manual GPU accuracy smoke workflow that runs on the test-flagsparse Actions Runner Controller scale set.
• .github/workflows/gpu-benchmark.yml adds an Actions button for synthetic GPU benchmark runs on the test-flagsparse Actions Runner Controller scale set.
• .github/workflows/release-drafter.yml keeps draft release notes current from merged PRs.
• make help lists the local entry points.
• make ci / make check run the same CPU-only pipeline used by CI.
• make format-check, make lint, and make lint-src are the non-GPU quality gates for CI formatting, CI helper lint, and critical package-source static checks.
• make smoke is the CPU smoke stage alias.
• make release-check / make release build, validate, and checksum release artifacts.
• make triton-smoke and make triton-deps are opt-in local targets for the triton-dependent runtime checks.
• make gpu-env-check validates CUDA visibility through tools/ci/check_gpu_environment.py on a GPU runner.
• make gpu-benchmark runs the quick synthetic benchmark suite on a CUDA machine.
• python tools/ci/run_gpu_benchmark.py --suite quick mirrors the manual GPU benchmark workflow locally on a CUDA machine.
• python tools/ci/run_gpu_benchmark.py --suite full --matrix-dir tests/data runs the full benchmark matrix, including .mtx-backed SpGEMM and SDDMM suites against the repository test matrices.
• tools/ci/requirements-ci.lock.txt and tools/ci/requirements-triton-smoke.lock.txt are the pinned local dependency bundles behind those make targets.
• .github/dependabot.yml keeps GitHub Actions and Python dependency updates visible.
• .github/ISSUE_TEMPLATE/ keeps issue entry points structured for bugs and feature requests.
• The CI dependency bundle now stays on packaging and test tooling only; triton-dependent smoke is opt-in through FLAGSPARSE_TRITON_SMOKE=1.
• Release artifacts now ship with a generated SHA256SUMS manifest and a matching checksum verification step in CI.
• PR quality gates are implemented through the default CPU CI workflow; configure branch protection in GitHub to require the CI / Build and smoke test check before merge.
• GPU accuracy and benchmark scripts still require CUDA hardware; the GPU workflows are manual and only run on a self-hosted GPU runner.

Performance

• benchmark/performance_utils.py defines the pytest-style performance base class, default metrics (latency_base, latency, speedup), median timing, warmup/iteration controls, CUDA synchronization, CSV record helpers, and the two-level average speedup rule.
• benchmark/attri_util.py and benchmark/core_shapes.yaml keep default and special shape grids centralized.
• benchmark/summary_for_plot.py reads recorded benchmark CSV files and reports the two-level speedup summary.
• benchmark/test_sparse_perf.py is an opt-in pytest entry point; real GPU runs remain manual or self-hosted because GitHub-hosted runners do not provide CUDA GPUs.
• tests/data/*.mtx can be used as the default MatrixMarket smoke dataset for mtx-backed GPU benchmark suites.

License

This project is licensed under the Apache (Version 2.0) license.