This document enumerates every symbol that is part of FlashRT's public stability contract. Symbols listed here will not be removed or have their signatures changed without a major version bump.
Symbols not listed here (internal modules, private functions, class internals) may change between minor releases.
import flash_rt
flash_rt.__version__ # str, e.g. "2.2.0"
flash_rt.load_model # → VLAModel
flash_rt.VLAModel # inference wrapperText-only Flint is an opt-in, model-specific entry point:
flash_rt.frontends.torch.flint_rtx.FlintTorchFrontendRtx. It supports torch on
SM120 with the official NVFP4 GGUF, batch 1 and greedy decoding. It is outside
the VLA load_model / predict surface; config="flint" raises with a pointer
to the direct frontend. See Flint usage. Existing stable
signatures and routing defaults are unchanged.
def load_model(
checkpoint: str,
framework: str = "torch", # "torch" | "jax"
num_views: int = 2, # 1, 2, or 3
autotune: int = 3, # 0=off, 3=default, 5+=thorough
recalibrate: bool = False,
weight_cache: bool = True, # JAX only
config: str = "pi05", # "pi05" | "pi0" | "groot" | "groot_n15" | "groot_n17" | "pi0fast" | "motus" | "wan22_ti2v_5b" | "cosmos3_video" | "cosmos3_edge" | "chameleon" | "hyvla" | "xvla" | "lawam" | "molmoact2" | "dm05" | "vla_jepa"
device=None, # reserved
# Pi0-FAST-specific:
decode_cuda_graph: bool = False,
decode_graph_steps: int = 80,
max_decode_steps: int = 256,
hardware: str = "auto", # "auto" | "thor" | "rtx_sm120" | "rtx_sm89" | "rtx_sm87" | "amd_cdna4" | "amd_rdna3" | "amd_rdna35" | "amd_rdna4" | "apple_mps"
# GROOT-specific:
embodiment_tag: str | None = None,
action_horizon: int | None = None,
# Pi0.5-specific:
use_fp4: bool = False,
fp4_layers: tuple[int, ...] | None = None,
use_awq: bool | None = None,
awq_alpha: float | None = None, # None -> per-stage production value
use_p1_split_gu: bool | None = None,
use_fp4_decoder: bool = False,
use_fp4_encoder_attn: bool | None = None, # None -> preset
use_fp4_siglip_ffn: bool | None = None, # None -> preset
use_fa4: bool = False,
encoder_p1_combiner: str | None = None, # None -> preset
encoder_down_variant: int = 7,
decoder_gate_up_variant: int = 10,
num_steps: int | None = None,
vision_pool_factor: int | None = None,
vision_num_layers: int | None = None,
cache_frames: int | None = None,
state_prompt_mode: str = "exact",
state_prompt_fixed_max_len: int | None = None,
# Frontends with an FP8/BF16 switch:
use_fp8: bool = True,
# Pi0.5 torch RTX SM120/SM89 opt-in:
use_fp16: bool = False,
precision: str = "auto", # Backend-specific: "auto" | "bf16" | "int8" | "fp8"
# X-VLA torch RTX only:
xvla_precision: str = "nvfp4", # "nvfp4" | "fp8" | "bf16"
# Robot output schema (Apple/RDNA Pi0.5 / Jetson Pi):
action_dim: int | None = None,
# Keyword-only, GR00T N1.7 RDNA input and graph controls:
fuse_image_embeds: bool | None = None,
use_backbone_graph: bool | None = None,
) -> VLAModelReturns a VLAModel wrapping the appropriate frontend for the detected
(or explicitly specified) GPU architecture.
-
action_dimdeclares the robot output width for Apple/RDNA Pi0.5 (1..32). If omitted, this backend requiresoutput_action_dimin checkpointconfig.json; it never infers padding from constant normalization statistics. Constant final action channels are retained. Other existing hardware routes keep their output conventions. -
decode_cuda_graph,decode_graph_steps,max_decode_stepsapply to Pi0-FAST. -
embodiment_tagandaction_horizonapply to GROOT. -
use_fp4,fp4_layers,use_awq,awq_alpha, anduse_p1_split_guapply to the Pi0.5 torch and JAX NVFP4 encoder path on Thor. The JAX path loads Orbax checkpoints; the torch path loads safetensors checkpoints. -
use_fp4_decoder,use_fp4_encoder_attn,use_fp4_siglip_ffn,encoder_p1_combiner,encoder_down_variantanddecoder_gate_up_variantextend that Pi0.5 Thor NVFP4 path. They requireuse_fp4=Trueand Pi0.5 torch on Thor; aTruevalue on any other config or hardware raises rather than being silently dropped. -
use_fp4_decoder=Trueselects the complete Thor NVFP4 tier that the published latency table is measured with, so the sub-flags left atNoneresolve to it:use_fp4_encoder_attn=True,use_fp4_siglip_ffn=True,encoder_p1_combiner="epilogue_hw"andawq_alpha=0.8. Withuse_fp4=Truealone they resolve to the encoder-only preset (False,False,"lut_native",0.5). Any of them passed explicitly overrides the preset. The measured configuration is therefore exactly:model = load_model(checkpoint, config="pi05", framework="torch", hardware="thor", num_views=3, use_fp4=True, use_fp4_decoder=True, use_fa4=True)
-
use_fa4selects the FA4 attention backend for SigLIP and encoder attention on Pi0.5 torch/Thor. It is not part of the NVFP4 tier and does not requireuse_fp4: both the FP8 frontend and its NVFP4 subclass accept it, and the published NVFP4 speedups are measured against an FP8 baseline that also runs FA4. It raises on any other config, framework or hardware. -
use_fp4=Truewithconfig="groot_n17"on Thor selects the GROOT N1.7 NVFP4 tier (NVFP4 DiT action head + cross-KV, vectorized backbone kernels, FA4 attention when available). It takes no Pi0.5 sub-flags and falls back to the FP8 tier when theflash_rt_fp4extension is missing. -
num_steps,vision_pool_factor,vision_num_layers, andcache_framesapply only to frontends that expose those constructor parameters today. The Pi0.5 torch RTX/Orin frontend validatesvision_pool_factor in {1, 2, 4},vision_num_layers in [1, 27], andcache_frames >= 1. -
state_prompt_modeapplies to Pi0.5 RTX/Thor state-in-prompt execution."exact"tracks the exact token length (RTX caches recurring lengths; Thor reuses same-length updates)."fixed"captures one max-length graph and masks padded state-prompt tokens with a device-side valid length; use it when live robot state changes make token lengths drift. The default remains"exact", so existing calls keep the exact-length path and the 50-step Pi0.5 action graph is unchanged. -
state_prompt_fixed_max_lenapplies only to Pi0.5 Thor fixed mode.Nonekeeps the default 200-token state-prompt cap; serving code can lower it when it knows the live state-prompt bound. The cap must cover the actual token length. On Thor, a close cap such as 120 for a 117-token prompt measured roughly a 1 ms normal overhead versus a warmed exact graph, while larger caps pay for the extra padded tokens. It can also be set withFLASHRT_PI05_STATE_PROMPT_FIXED_MAX_LEN. -
hardware="amd_cdna4"(auto-detected on ROCm builds for gfx950 / MI350-series) routesconfig="pi05",framework="torch"to the self-contained AMD backend (flash_rt/amd/, HIP moduleflash_rt_amd_kernels). Single-stream surface only:set_prompt/calibrate/inferwith bothstate_prompt_modegraphs; the RL and CFG-batched extensions raiseNotImplementedError. Build and usage:docs/deployment_amd.md. -
hardware="apple_mps"(auto-detected on Apple Silicon when MPS is available and CUDA/Ascend are unavailable) supports onlyconfig="pi05",framework="torch". Execution is BF16 eager withprecision="auto"orprecision="bf16"; the historicaluse_fp8=Truedefault emits a warning and selects BF16. Explicit FP8/INT8 precision, FP4, FP16 and FA4 requests raise. This route requires no CUDA/HIP extension, calibration or graph capture;autotunehas no effect andcache_framesmust be 1 (orNone). The eager route uses exact prompt lengths, so fixed state-prompt graph options raise. The separately enabled experimental native Metal plan can reuse bounded prompt storage across length changes; see the deployment guide. Images must be two or three RGB uint8 arrays of shape(224, 224, 3); state supplied topredict()/set_prompt()must already be normalized with the checkpoint's state statistics. Setaction_dimor declareoutput_action_dimin checkpointconfig.json. Setup, real-input smoke results and the limitations of the CPU comparison are documented in Apple deployment. -
use_fp8=Falsedisables FP8 where the selected frontend exposes a BF16 fallback; unsupported frontends ignore it. GROOT N1.7 on RTX/Thor is stricter: the default route is FP8, anduse_fp8=Falsealone raises because there is no separate BF16-only path. -
use_fp16=Trueselects the opt-in reference path. It requiresuse_fp8=Falseand is currently valid for:config="pi05",framework="torch",hardware in {"rtx_sm120", "rtx_sm89"}config="groot",framework="torch",hardware in {"thor", "rtx_sm120"}config="groot_n17",framework="torch",hardware in {"thor", "rtx_sm120", "rtx_sm89"}
-
config="motus"is a beta RTX SM120 frontend. It expects a Motus checkpoint plus Wan and VLM checkpoint paths supplied to the Motus quickstart/frontend; seedocs/motus_usage_beta.md. -
config="wan22_ti2v_5b"is an RTX SM120 official-pipeline Wan2.2 baseline. It exposesset_prompt(prompt, negative_prompt=...)andinfer(mode="t2v"|"i2v", width=..., height=..., frames=..., steps=..., shift=..., guide_scale=..., seed=..., teacache=False, teacache_threshold=..., teacache_start_step=..., teacache_end_step=..., teacache_cache_device=...);predict()is not part of this video-generation API. Seedocs/wan22_usage.md. -
config="cosmos3_video"is an RTX SM120 Cosmos3-Nano text2video FP8 denoise model (non-VLA). It exposesset_prompt(ref=<reference dump>)for conditioning andinfer(teacache_skip=..., shift=..., compare_ref=..., return_metadata=...), returning the denoised vision latent;predict()is not part of this API. Precision is selected withload_model(..., use_fp8=True|False). Seedocs/cosmos3_video_usage.md. -
config="cosmos3_edge"is the Thor integration for NVIDIA Cosmos Framework. It exposesset_prompt(input_json=...)orset_prompt(sample=...), theninfer(output_dir=..., vae_path=..., cosmos_root=...). The official backend establishes the end-to-end reference; the fixture-backed FlashRT backend validates and benchmarks the optimized denoise boundary. Seedocs/cosmos3_edge_thor.md. -
config="groot_n15"is registered forframework="torch"onhardware="rtx_sm89"only, as FlashRT kernels in one CUDA graph per prompt length: FP8 by default, BF16 (the reference's algorithms) withuse_fp8=False. It needsflash_rt_kernels,flash_rt_fa2andflash_rt_groot_n15(-DFLASHRT_ENABLE_GROOT_N15=ON) and the upstream Isaac-GR00Tgr00tpackage for the observation / action transforms (set$GROOT_N15_ROOTand$GROOT_N15_DATA_CONFIG, or passgroot_root=/data_config=to the frontend);predict(...)returns the GR00T per-key action dict. Seedocs/groot_n15_rtx_sm89.md. -
config="groot_n17"is registered forframework="torch"onhardware in {"thor", "rtx_sm120", "rtx_sm89", "rtx_sm87", "amd_cdna3", "amd_cdna4", "amd_rdna3", "amd_rdna35", "amd_rdna4", "npu"}. On RTX,rtx_sm120resolves through the historical shared RTX registration andload_model()refines that default route to the FP8 production frontend;rtx_sm89resolves directly to its dedicated SM89 frontend.use_fp16=True, use_fp8=Falserequests the explicit RTX reference frontend for the selected hardware. Onnputhe tier is BF16 andprecision="int8"is refused rather than downgraded; that backend's native kernels are built per model, soFLASHRT_ENABLE_NPU_GROOT_N17=ONis required before this route can load. Seedocs/deployment_npu_groot_n17.md. On every hardware this model takes its observation bundle from the caller, so it is driven throughmodel.pipeline—set_prompt(aux=...),normalize_state,infer,denormalize_action— rather than throughVLAModel.predict. RDNA (gfx1100,gfx1151,gfx1201) runs BF16. As with other RDNA models,load_modelconvertsuse_fp8=True(including the loader default) to BF16 with a warning; passuse_fp8=Falseto select it explicitly. FP4/FP16 requests fail before backend loading. Direct RDNA frontend construction rejects FP8. Its defaultfuse_image_embeds=Falseaccepts the existing projected observation bundle (pixel_features,llm_input_embeds,grid_thw,visual_pos_masks,rope_cos,rope_sin). Explicitfuse_image_embeds=Trueselects Orin's raw-patch/token-ID contract and enablesinfer(state_normalized, frames=...). Extra dictionary fields never select a different mode. Projected inputs include model computation performed by the caller and must match the checkpoint and observation.load_model(fuse_image_embeds=None, use_backbone_graph=None)preserves frontend defaults. On RDNA, fusion defaults off and backbone graphs default on. Explicit values must be bools and are supported only for N1.7 torch on RDNA; other routes retain their existing construction interfaces.use_dit_graphremains an independent inference argument. RDNA provides an optional processor-onlyprepare_observationhelper andset_hf_processorinjection for offline use.framesaccepts ordered RGB uint8 arrays/tensors or a sequence of images, including every history frame in time-then-camera order. It cannot accompanyaux['pixel_values']and does not replace prompt setup or state normalization.flash_rt.amd.frontends.torch.groot_n17_rdna.frames_from_observation(observation, camera_order)packs batch-one video arrays in this order; use the checkpoint's video modality keys forcamera_order. RDNA uses the checkpoint's official host image transforms, including non-square images; device frames incur a host transfer. Processing stays outside the HIP graph. Input/output semantics remain the normalized-stateinferand explicit physical action decoding workflow. See AMD GR00T deployment. -
config="chameleon"is a chat-style VLM and is not served throughload_model's VLA wrapper. Callingload_model(config="chameleon")raisesNotImplementedErrorwith direct-construction instructions. Construct the frontend explicitly:ChameleonTorchFrontendRtxSm87(Jetson Orin SM87) orChameleonTorchFrontendThor(Jetson Thor SM110). Seedocs/chameleon_usage.md. -
config="hyvla"(Hy-Embodied-0.5-VLA) is registered forframework="torch"onhardware in {"thor", "rtx_sm87", "rtx_sm120"}. Thor and Orin SM87 use runtime dynamic FP8 with fused megakernels and optional NVFP4 FFN (use_fp4=True). RTX SM120 runs SM120 block-128 FP8 and defaults to an NVFP4 ViT + VLM-prefill tier with the expert denoise tower on FP8 (use_fp4=True, use_fp4_expert=False);use_fp4_expert=Truepromotes the expert tower to NVFP4 (opt-in, below the 0.999 precision gate). Seedocs/hyvla05_thor_sm110.md,docs/hyvla05_orin_sm87.md, anddocs/hyvla05_rtx_sm120.md. -
config="xvla"(lerobot/xvla-libero) is registered forframework="torch"onhardware="rtx_sm120"(the detected arch for SM120 and SM121).xvla_precisionselects"nvfp4"(default),"fp8", or"bf16"for the action transformer and encoder; the DaViT tower runs NVFP4 W4A4.predict(images, prompt, state)is supported;num_views=2(or 3 when the caller includes the configured empty slot). Seedocs/xvla_rtx_sm120.md. -
config="molmoact2"(allenai MolmoAct2) is registered forframework="torch"onhardware="rtx_sm120"(SM120/SM121). It is not served through thepredict(images, prompt)wrapper: every call carries the 14-dim robot state, and the graph must be calibrated on real observations before capture. Callingload_model(config="molmoact2")raisesNotImplementedErrorwith direct-construction instructions (MolmoAct2TorchFrontendRtx;set_prompt,calibrate,record_infer_graph,infer). Requires-DFLASHRT_ENABLE_MOLMOACT2=ON. Seedocs/molmoact2_usage.md.
Jetson AGX Thor (SM110) has an opt-in BF16/FP32 mode for the OpenPI Pi0.5
LIBERO PyTorch checkpoint. The mode accepts one 224 × 224 RGB uint8 image,
a text-only prompt, and returns a 10 × 7 action chunk. Build with
-DFLASHRT_ENABLE_PI05_THOR_BF16=ON and pass precision="bf16", num_views=1,
and use_fp8=False. See the single-view Thor guide.
For the optimized torch/Thor full NVFP4 path, build with
-DFLASHRT_ENABLE_PI05_THOR=ON and call:
flash_rt.load_model(checkpoint, config="pi05", framework="torch", hardware="thor",
num_views=2, use_fa4=True, use_fp4=True, use_fp4_decoder=True,
awq_alpha=0.5, encoder_p1_combiner="epilogue_hw_nod",
encoder_down_variant=8)The optimized decoder uses the Pi0.5-owned flash_rt_pi05_thor module.
An absent module raises an error naming the required build flag. Encoder-only,
JAX and other hardware routes retain their existing frontend. See the
Thor guide for calibration and precision checks.
Pi0.5 state prompts zero-pad shorter normalized states to 32 dimensions before discretization, matching OpenPI's tokenizer input. Text-only prompts are unchanged.
class VLAModel:
def set_prompt(self, *args, **kwargs): ...
def infer(self, *args, **kwargs): ...
def predict(self, images, prompt=None, state=None) -> np.ndarray: ...
def calibrate(
self,
observations,
*,
percentile: float = 99.9,
max_samples: int | None = None,
verbose: bool = False,
) -> None: ...
def warm_state_prompt_buckets(self, images, prompt, states) -> list[int]: ...
def recalibrate(self) -> None: ...
@property
def framework(self) -> str: ...
@property
def prompt(self) -> str | None: ...
@property
def pipeline(self): ...-
set_prompt(*args, **kwargs)— delegate prompt setup to the selected frontend. GROOT N1.7 currently usesset_prompt(aux=..., prompt=...), whereauxcontains the captured Qwen3-VL setup tensors consumed by the N1.7 calibration path. -
infer(*args, **kwargs)— delegate inference to the selected frontend. GROOT N1.7 currently usesinfer(state_normalized, aux=..., initial_noise=..., use_dit_graph=...)and returns normalized actions. For the RTX FP8 frontend, optionalauxsupplies fresh observation/model inputs and lazily captures, then runs, the backbone graph before the action graph. Fresh inputs must retain thegrid_thw, visual mask, RoPE tables, and tensor shapes established byset_prompt(). Omittingauxreuses the eager backbone features without capturing the optional graph.RDNA accepts the same
num_inference_timesteps,action_horizon,num_timestep_buckets,initial_noise, anduse_dit_graphkeywords. Omitted schedule values use checkpoint/construction defaults; explicit positive values reconfigure the schedule and invalidate affected graphs.action_horizoncannot exceed the checkpoint horizon. Omittingauxreuses the most recently supplied observation (initially the setup observation). Supply fresh aux for every new camera observation.use_dit_graph=Falsedisables DiT capture; the backbone may still use its graph. The RDNA frontend'suse_backbone_graphconstructor flag controls that stage;use_graph=Falseoninferselects fully eager execution. The default fresh observation path captures both stages in one graph. Frontendpredictis an alias of normalized-stateinfer, as on CUDA/CDNA. -
predict(images, prompt, state)— run one inference step.images: list of(224,224,3)uint8 numpy arrays, or a dict with"image"/"wrist_image"/"wrist_image_right"keys.prompt: required on first call, reused on subsequent calls ifNone.state: optional robot state array.predict()attaches it to the observation as"state"when the caller passes image lists, while preserving an explicit"state"already present in a dict observation. For frontends whoseset_prompt()acceptsstate,predict()refreshes the prompt prefix when either the prompt text or that prompt-state value changes. Returnsnp.ndarrayof shape(action_horizon, action_dim).stateis part of the VLA observation schema; each model encodes it through its own reference contract:- Pi0 uses a continuous state token in the action-expert suffix.
- Pi0.5 encodes state as openpi-compatible discretized prompt tokens:
Task: <prompt>, State: <openpi state bins>;\nAction:. The bins use OpenPI'snp.digitize(state, np.linspace(-1, 1, 257)[:-1]) - 1convention: normalized in-range values usually become 0..255, while values below -1 become -1. RTX/Thor torch frontends and the JAX Thor Pi0.5 frontend acceptstateinset_prompt()and throughpredict(). Same-length state prompt updates reuse the captured graph. RTX exact mode reuses cached recurring prompt lengths; RTX/Thor fixed mode uses one max-length graph for drifting state-token lengths. - Pi0-FAST encodes state in the FAST token prefix.
- GROOT N1.6 consumes proprioceptive state from
obs["state"]; if omitted, the backend uses zeros. - GROOT N1.7 currently uses the lower-level
normalize_state(...)+infer(state_normalized, initial_noise=...)contract rather than the image-listpredict()contract.
-
calibrate(observations, *, percentile=99.9, max_samples=None, verbose=False)— run the selected frontend's public calibration path.observationsmay be a single observation dict or an iterable of samples in the frontend's calibration format.max_samplescaps the consumed sample list before the frontend chooses its N=1 or N>=2 path. Frontends that document N>=2 support run dataset calibration with percentile-clipped amax reduction; unsupported frontends raise a clearNotImplementedError. GROOT N1.7 uses captured aux dict samples rather than raw image observation dicts. -
warm_state_prompt_buckets(images, prompt, states)— Pi0.5 RTX torch helper for realtime loops that pass changing state through the prompt. It calls the selected frontend's bucket warmup hook, usingimagesas the calibration/capture observation and each item instatesas a representative robot state. The return value is the sorted list of warmed prompt lengths. The method preserves the OpenPI state prompt text format; it does not zero-pad or otherwise rewrite state tokens. A recurring prompt token length reuses its cached runtime pipeline, but a previously unseen state-token length still pays one first-time bucket build/capture cost. Serving code should keep state serialization stable (fixed state dimension and numeric rounding/precision policy) and prewarm representative states from the deployment range, for example reset, mid-rollout, near-goal, and a few recorded rollout observations whose discretized values may tokenize to different lengths. -
recalibrate()— clear FP8 calibration cache and force re-calibration on the nextpredict()call. -
pipeline— the wrapped frontend instance.predict()is the supported inference entry point; this accessor exists for benchmarking and validation harnesses that need the frontend's own surface (for example the raw pre-decode noise buffer) while still constructing the model throughload_model(), so that a measured configuration is by construction the one the public API produces.
from flash_rt.hardware import detect_arch, resolve_pipeline_classReturns "thor", "rtx_sm120", "rtx_sm89", or "rtx_sm87" based on
the current CUDA device's compute capability. SM87 is currently supported
only for config="pi05", framework="torch".
Lazily imports and returns the concrete frontend class for the given
(config, framework, arch) triple. Used internally by load_model.
Motus beta is registered for (config="motus", framework="torch", arch="rtx_sm120").
GROOT N1.7 is registered for (config="groot_n17", framework="torch", arch in {"thor", "rtx_sm120", "rtx_sm89", "rtx_sm87", "amd_cdna3", "amd_cdna4", "amd_rdna3", "amd_rdna35", "amd_rdna4", "npu"}). On RTX, rtx_sm120
keeps the shared-base registration and load_model() refines that
resolved class to the FP8 default or the explicit RTX reference frontend
based on use_fp8 / use_fp16; rtx_sm89 resolves directly to the
dedicated SM89 frontend class.
Wan2.2 TI2V-5B is registered for (config="wan22_ti2v_5b", framework="torch", arch="rtx_sm120").
Chameleon-7B is registered for (config="chameleon", framework="torch", arch in {"rtx_sm87", "thor"}).
Hy-Embodied-0.5-VLA is registered for (config="hyvla", framework="torch", arch in {"thor", "rtx_sm87", "rtx_sm120"}).
VLA-JEPA is registered for (config="vla_jepa", framework="torch", arch="rtx_sm120") for discovery; load_model raises an explicit redirect
to VlaJepaTorchFrontendRtx (per-call state and real-data calibration).
flash_rt.hardware._PIPELINE_MAP: dict[tuple[str, str, str], tuple[str, str]]The dispatch table mapping (config, framework, arch) to
(module_path, class_name). External plugins may mutate this dict
at import time to register new models — see
Plugin Model Template.
from flash_rt.hardware.backend import (
AttentionBackend, # Protocol class
AttentionBackendBase, # Optional base with accessor defaults
AttentionSpec, # Full model attention specification
SiteSpec, # One attention site descriptor
)@dataclass
class SiteSpec:
num_layers: int
num_q_heads: int
num_kv_heads: int
head_dim: int
max_q_seq: int
max_kv_seq: int | None = None # None → self-attention
batch_axis: int = 1
sliding_window: int | None = None # reserved for SWA models
causal: bool = False
extra: dict = field(default_factory=dict)@dataclass
class AttentionSpec:
sites: dict[str, SiteSpec]
def add_site(self, name: str, **kwargs) -> AttentionSpec: ...
def site(self, name: str) -> SiteSpec: ...class AttentionBackend(Protocol):
def sites(self) -> tuple[str, ...]: ...
def get_slot_ptrs(self, site: str, layer_idx: int) -> dict[str, int]: ...
def run(self, site: str, layer_idx: int, q_seq: int, *,
kv_seq: int | None = None, stream: int = 0) -> int: ...
def head_dim(self, site: str) -> int: ...
def num_q_heads(self, site: str) -> int: ...
def num_kv_heads(self, site: str) -> int: ...These concrete backends are available for reuse by plugins. Their existence is stable, but their internal signatures may change between minor releases. Plugins that subclass or call into these should pin a minor version.
from flash_rt.hardware.rtx.attn_backend import RtxFlashAttnBackend
from flash_rt.hardware.rtx.attn_backend_groot import RtxFlashAttnBackendGrootBoth classes are framework-neutral — used by torch and jax frontends
alike. The old names TorchFlashAttnBackend /
TorchFlashAttnBackendGroot are kept as deprecated module-level
aliases and will be removed in the next major version.
from flash_rt.core.cuda_buffer import CudaBuffer
from flash_rt.core.cuda_graph import CUDAGraph
from flash_rt.core.quant.calibrator import load_calibration, save_calibrationThese are used by both pipelines and frontends. Their public API is stable; internal helper functions are not.
FlashRT ships two pybind11 Python extension modules:
from flash_rt import flash_rt_kernels # always present
from flash_rt import flash_rt_fa2 # RTX (SM80/86/89/120) onlyThe main kernel module — hand-written CUDA code plus cuBLASLt/
CUTLASS wrappers for memory-bound ops (norm, activation, fusion,
FP8 quant, residual, gate-geglu, true-silu, etc.) and Thor-specific attention
(fvk.attention_qkv_fp16). Binary name pattern:
flash_rt_kernels.cpython-<abi>.so, ~3 MB.
All fvk.<symbol>(...) calls seen in pipeline code live here.
Signatures are internal — plug-ins should go through the
AttentionBackend protocol, not call fvk.* directly.
Motus beta SM120 kernels are built into flash_rt_kernels when CMake is
configured with -DFLASHRT_ENABLE_MOTUS=ON (currently the default).
They do not require a separate runtime kernel library directory.
from flash_rt.runtime.rtc import (
ActionChunkAdapter,
CallablePolicyAdapter,
AsyncChunkRunner,
RTCStats,
)
from flash_rt.runtime import (
AsyncTemporalFusionRunner,
FusedChunk,
ObservationSnapshotter,
PredictionTicket,
TemporalFusionBuffer,
TemporalFusionConfig,
TemporalFusionStats,
TimedActionChunk,
)
from flash_rt.runtime.vlash import (
AsyncVLAShRunner,
VLAShChunkResult,
VLAShConfig,
VLAShStats,
)The legacy async chunk runner is a beta inference scheduling utility for action-chunk
policies. It does not change model numerics or calibration; it only
serves action chunks at a fixed controller rate while a background worker
prepares the next chunk. See docs/rtc_lite_design.md.
The temporal-fusion runner is an opt-in scheduling policy that retains raw
predicted chunks, aligns them on the controller-step timeline, and fuses up to
TemporalFusionConfig.max_chunks overlapping predictions with exponential
position-difference weights. It supports latency- and state-based chunk
switching without changing model kernels or calibration. See
docs/rtc_temporal_fusion.md for the adapter contract, configuration, deadline
behavior, and real Pi0.5 checkpoint gate.
AsyncVLAShRunner is an optional host-side runtime for projected-state
action-chunk scheduling. It estimates a future robot state from the active
chunk, injects that state into the next observation, and activates the completed
chunk from index zero. See docs/vlash.md.
Vendored Flash-Attention 2 v2.7.4.post1 (forward only, fp16 + bf16,
SM80-family SASS). Binary name pattern:
flash_rt_fa2.cpython-<abi>.so, ~135 MB. Only built when
GPU_ARCH ∈ {80, 86, 87, 89, 120}. Exposes:
flash_rt_fa2.fwd_fp16(
Q, K, V, O, softmax_lse,
softmax_lse_accum=0, o_accum=0, # splitkv scratch ptrs; 0 disables splitkv
*,
batch, seqlen_q, seqlen_k,
num_heads_q, num_heads_kv, head_dim,
q_strides, k_strides, v_strides, o_strides, # 3-tuples (batch, row, head) in elements
softmax_scale=1.0,
num_sms=0, # required for splitkv heuristic
stream=0,
)
flash_rt_fa2.fwd_bf16(...) # same signature, bfloat16 dtypeAll pointer args are int device pointers (tensor.data_ptr()).
Pipeline code should go through RtxFlashAttnBackend (which calls
this module internally) rather than invoking it directly — direct
use is unstable and may change without notice. RtxFlashAttnBackend
selects between this module and the pip flash-attn wheel via the
FVK_RTX_FA2 env var (default "1" = use vendored FA2; "0" =
fallback to flash_attn.flash_attn_func). The backend name reflects
the hardware family (RTX), not the frontend framework — the same
backend instance serves both torch and jax frontends.
Thor (SM110) builds do not produce this module — attention on
Thor uses flash_rt_kernels.attention_qkv_fp16 (cuBLAS-decomposed)
because FA2's Ampere tile shapes aren't tuned for Thor's unified
LPDDR memory model. Code importing flash_rt_fa2 must therefore
guard for ImportError on Thor deployments, or stay inside the
AttentionBackend protocol which handles the dispatch transparently.
flash_rt/
├── __init__.py # load_model, VLAModel
├── api.py # load_model implementation
├── core/ # shared utilities (CudaBuffer, CUDAGraph, calibrator)
├── hardware/
│ ├── __init__.py # detect_arch, _PIPELINE_MAP, resolve_pipeline_class
│ ├── backend.py # AttentionBackend protocol
│ ├── rtx/ # rtx attention backends (hardware primitives only)
│ └── thor/
│ ├── attn_backend.py # Thor FMHA wrapper
│ ├── attn_backend_groot.py # GROOT-specific Thor attention
│ └── shared_primitives.py # CLOSED SET: model-agnostic Thor helpers only
│ # (_gpu_*, _measure_scale_gpu,
│ # siglip_forward, encoder_forward,
│ # encoder_forward_calibrate)
├── models/
│ ├── pi05/
│ │ ├── pipeline_thor.py # Pi0.5 Thor compute (postln_project, decoder_*)
│ │ └── pipeline_rtx.py # Pi0.5 RTX Pi05Pipeline class
│ ├── pi0/
│ │ ├── pipeline_thor.py # Pi0 Thor decoder fns
│ │ └── pipeline_rtx.py # Pi0 RTX Pi0Pipeline class
│ ├── pi0fast/
│ │ └── pipeline.py # DEPRECATED PATTERN: Thor+SM120 runtime fork
│ │ # do NOT copy this style for new models
│ └── groot/
│ ├── pipeline_thor.py # GROOT Thor pipeline
│ ├── pipeline_rtx.py # GROOT RTX pipeline
│ └── embodiments.py # per-embodiment MLP slots
└── frontends/
├── torch/
│ ├── pi05_thor.py (Pi05TorchFrontendThor)
│ ├── pi05_rtx.py (Pi05TorchFrontendRtx)
│ ├── pi0_thor.py (Pi0TorchFrontendThor)
│ ├── pi0_rtx.py (Pi0TorchFrontendRtx)
│ ├── pi0fast.py (Pi0FastTorchFrontend) DEPRECATED — Thor+RTX hybrid
│ ├── groot_thor.py (GrootTorchFrontendThor)
│ └── groot_rtx.py (GrootTorchFrontendRtx)
└── jax/
├── pi05_thor.py (Pi05JaxFrontendThor)
├── pi05_rtx.py (Pi05JaxFrontendRtx)
├── pi0_thor.py (Pi0JaxFrontendThor)
├── pi0_rtx.py (Pi0JaxFrontendRtx)
└── pi0fast.py (Pi0FastJaxFrontend) DEPRECATED — Thor+RTX hybrid
Naming convention (established 2026-04, stage 8 unified-pipeline-layout refactor):
- Every (model, hardware) compute path is its own file:
models/<m>/pipeline_<hw>.pywhere<hw>∈ {thor,rtx}. Nopipeline.py(no-suffix default entry) is allowed. - Every (model, framework, hardware) IO path is its own frontend file:
frontends/<fw>/<m>_<hw>.pywith class<Model><Fw>Frontend<Hw>(e.g.Pi05TorchFrontendThor,Pi05TorchFrontendRtx). - No runtime hardware forks (
if self._has_sm100,hasattr(fvk, ...)): if a model needs different code on Thor vs RTX, those are separate files. hardware/<hw>/shared_primitives.pyis a closed set of model-agnostic helpers. Model-specific forwards/decoders go intomodels/<m>/pipeline_<hw>.py, never intoshared_primitives.py._PIPELINE_MAPis one-to-one: each(model, framework, hw)tuple routes to exactly one frontend file/class.
Known historical exception (do NOT copy for new models):
pi0fastships as a singlepipeline.pywith 14+if self._has_sm100branches and a single multi-hw frontend file. This in-file SM-fork pattern is retained for the existing Pi0-FAST implementation only; new models should follow the standard(model, framework, hw)split.
Thor frontends' per-layer weight-loading loops are expressed as
ModelWeightSpec objects in private spec modules next to the frontend.
The public surface is three things:
from flash_rt.executors.weight_loader import (
Item, LayerBlock, ModelWeightSpec, WeightLoader,
)
from flash_rt.executors.torch_weights import ( # torch side
SafetensorsSource, DictSource,
Cat, FusedQKV, FusedGateUp,
ToFp16, ToFp32, T, tT, InterleaveQK, Quant, Mul,
Attr, TensorList, FlatCat,
)
from flash_rt.executors.jax_weights import ( # jax side
OrbaxDictSource,
Transpose, Astype, Contiguous, JaxQuant,
NumpyAttr, NumpyList, CudaBufferAttr, CudaBufferFlat,
)Stability contract: the classes above and their constructor signatures are public. Adding new sink/transform/composite classes is backwards-compatible; existing ones will not be removed or renamed without a major version bump.
Spec file naming: flash_rt/frontends/{torch,jax}/_<model>_thor_spec.py,
each exporting a build_spec() -> ModelWeightSpec. Shared block
builders live in _thor_spec_common.py per framework.
Convention for scale lists: spec items that quant set
scale_into="_<group>_scales"; frontends wrap these into device
tensors after loader.run():
self._enc_w_dev = torch.tensor(self._enc_w_scales,
dtype=torch.float32, device='cuda')See docs/adding_new_model.md for the end-to-end
model adaptation walkthrough, and
docs/plugin_model_template.md for
registering an external-plugin model via _PIPELINE_MAP.
The Pi0.5 RTX frontend's infer and infer_batch accept optional
noise or generator arguments. They are mutually exclusive for the
initial noise draw. The default result does not contain a noise host copy.
Pass return_noise=True to export the BF16-rounded initial noise under
"noise", after inference completes. This also applies to batched CFG.
The construction-time denoise_trace and prefix_features exports remain
opt-in; batch prefix pooling excludes each slot's padded prompt positions.
When adding a new model or kernel, read these in order:
docs/adding_new_model.md— end-to-end walkthrough for wiring a new VLA model into FlashRT on Thor (AttentionSpec → WEIGHT_SPEC → pipeline forward → frontend → calibration → graph capture → registration → tests).docs/calibration.md— FP8 weight/activation scale mechanics,alpha = act_scale × weight_scaleinvariants, calibration cache format, and the four historical bugs every new model's_calibrateshould guard against.docs/kernel_fusion.md— the 93 publicfvkkernels grouped by purpose, current production fusion patterns, what does and does not fuse, and the catalog of failed optimizations (OPT-3 / OPT-5 / v1.5-B2 / v1.5-B4) to avoid.
load_model(checkpoint, config="dm05", framework="torch", hardware="rtx_sm120", use_fp4=True, num_views=2) selects the DM05 frontend and its mixed NVFP4/FP8/INT8
tier. Existing model defaults are unchanged. DM05 requires its external model
distribution for loading and IO transforms, and FLASHRT_ENABLE_DM05=ON for kernels.
Before predict(images, prompt=..., state=...), load calibrated scales through
model.pipeline.load_calibration_state(state_or_path), or call
model.calibrate(observations, percentile=100.0) with representative real data.
The frontend consumes the model package; it does not replace runtime methods.
LIBERO outputs are (10, 7). Other hardware/frameworks and unimplemented precision
requests fail explicitly. Full-action numerical and gripper-decision validation
remains required; see DM05 validation.
load_model(checkpoint, config="lawam", framework="torch", hardware="rtx_sm120")
selects LawamTorchFrontendRtx for the LeRobot LaWAM checkpoint; it needs
FLASHRT_ENABLE_LAWAM=ON for flash_rt_lawam. predict([primary, wrist], prompt=task) takes two uint8 256x256 RGB views and returns the unnormalized
(action_horizon, 7) chunk; infer(frames, noise=...) accepts the initial flow
sample for reproducible comparisons. Other hardware/frameworks fail explicitly.
Precision tiers and validation: LaWAM on RTX SM120.
load_model(checkpoint, config="vla_jepa", framework="torch", hardware="rtx_sm120")
raises an explicit redirect: the policy takes the 8-dim robot state on every
call and its NVFP4 block/layer selection is calibrated on real observations
before capture, so it is constructed directly as
VlaJepaTorchFrontendRtx. It is registered in _PIPELINE_MAP for discovery.
The model-specific preprocessing kernels require -DFLASHRT_ENABLE_VLA_JEPA=ON.
Other hardware, frameworks and precision tiers outside the documented RTX
SM120/SM121 path fail explicitly. See VLA-JEPA usage.
Alpamayo produces trajectories and reasoning from camera frames and ego history, not VLAModel.predict(images). load_model(config="alpamayo15") and load_model(config="alpamayo15_flashdrive") raise an explicit redirect to their dedicated Thor frontends. Existing model routes and predict signatures are unchanged. See the consolidated guide for imports, checkpoint selection, reset behavior and four-arm reproduction.
Higgs uses its TTS-specific frontend rather than the VLA predict() contract:
from flash_rt.models.higgs_audio_v3 import create_frontend
fe = create_frontend("/path/to/higgs-tts-3-4b", fp8=False)
codes = fe.predict("Hello.") # CPU int64 [frames, 8]
waveform = fe.synthesize(codes) # CPU float32 mono, 24 kHz
waveform = fe.generate("Hello.", temperature=1.0, seed=1234)
for packet in fe.generate_stream("Hello."):
consume(packet)hardware="auto" preserves the CUDA constructor and selects the RDNA frontend
on ROCm. Explicit RDNA targets are amd_rdna3, amd_rdna35 and amd_rdna4.
RDNA constructor defaults include device="cuda:0", max_seq=2048,
max_new_frames=1024, max_prefill_tokens=512, alloc_own_forward_buffers=True
and fp8=None. Auto precision selects native FP8 W8A8 on supported RDNA4 and
BF16 otherwise; unsupported explicit FP8 warns and falls back to BF16.
Both frontends preserve temperature=0.0 (greedy) and optional seed on
predict, decode_stream, generate and generate_stream. A
temperature > 1e-5 selects native full-vocabulary categorical sampling. The same
seed is repeatable within a fixed precision and algorithm selection, not a
promise of cross-device identical generation.
Streaming defaults to first_chunk=8, chunk=25, ctx=8, holdback=8. It waits for
right context, decodes overlapping codec windows and trims output to committed
frames. Native generation retains delayed-codebook EOC winddown and the
repeated-code guard. max_new_frames bounds delayed rows; RDNA also stops before
exceeding max_seq. RDNA's last_finish_reason reports eoc, repeated_code,
max_new_frames or max_seq. Serialize requests to each frontend; validated
resident prefix KV can be reused with prefill(..., cached_tokens=...) or
generate_stream(..., system=...).
See RDNA setup and usage.