Skip to content

Latest commit

 

History

History
916 lines (796 loc) · 43.4 KB

stable_api.md

File metadata and controls

916 lines (796 loc) · 43.4 KB

FlashRT — Stable Public API

This document enumerates every symbol that is part of FlashRT's public stability contract. Symbols listed here will not be removed or have their signatures changed without a major version bump.

Symbols not listed here (internal modules, private functions, class internals) may change between minor releases.


Top-level (flash_rt)

import flash_rt

flash_rt.__version__   # str, e.g. "2.2.0"
flash_rt.load_model    # → VLAModel
flash_rt.VLAModel      # inference wrapper

Text-only Flint is an opt-in, model-specific entry point: flash_rt.frontends.torch.flint_rtx.FlintTorchFrontendRtx. It supports torch on SM120 with the official NVFP4 GGUF, batch 1 and greedy decoding. It is outside the VLA load_model / predict surface; config="flint" raises with a pointer to the direct frontend. See Flint usage. Existing stable signatures and routing defaults are unchanged.

flash_rt.load_model(...)

def load_model(
    checkpoint: str,
    framework: str = "torch",       # "torch" | "jax"
    num_views: int = 2,             # 1, 2, or 3
    autotune: int = 3,              # 0=off, 3=default, 5+=thorough
    recalibrate: bool = False,
    weight_cache: bool = True,      # JAX only
    config: str = "pi05",           # "pi05" | "pi0" | "groot" | "groot_n15" | "groot_n17" | "pi0fast" | "motus" | "wan22_ti2v_5b" | "cosmos3_video" | "cosmos3_edge" | "chameleon" | "hyvla" | "xvla" | "lawam" | "molmoact2" | "dm05" | "vla_jepa"
    device=None,                    # reserved
    # Pi0-FAST-specific:
    decode_cuda_graph: bool = False,
    decode_graph_steps: int = 80,
    max_decode_steps: int = 256,
    hardware: str = "auto",         # "auto" | "thor" | "rtx_sm120" | "rtx_sm89" | "rtx_sm87" | "amd_cdna4" | "amd_rdna3" | "amd_rdna35" | "amd_rdna4" | "apple_mps"
    # GROOT-specific:
    embodiment_tag: str | None = None,
    action_horizon: int | None = None,
    # Pi0.5-specific:
    use_fp4: bool = False,
    fp4_layers: tuple[int, ...] | None = None,
    use_awq: bool | None = None,
    awq_alpha: float | None = None,   # None -> per-stage production value
    use_p1_split_gu: bool | None = None,
    use_fp4_decoder: bool = False,
    use_fp4_encoder_attn: bool | None = None,   # None -> preset
    use_fp4_siglip_ffn: bool | None = None,     # None -> preset
    use_fa4: bool = False,
    encoder_p1_combiner: str | None = None,     # None -> preset
    encoder_down_variant: int = 7,
    decoder_gate_up_variant: int = 10,
    num_steps: int | None = None,
    vision_pool_factor: int | None = None,
    vision_num_layers: int | None = None,
    cache_frames: int | None = None,
    state_prompt_mode: str = "exact",
    state_prompt_fixed_max_len: int | None = None,
    # Frontends with an FP8/BF16 switch:
    use_fp8: bool = True,
    # Pi0.5 torch RTX SM120/SM89 opt-in:
    use_fp16: bool = False,
    precision: str = "auto",        # Backend-specific: "auto" | "bf16" | "int8" | "fp8"
    # X-VLA torch RTX only:
    xvla_precision: str = "nvfp4",  # "nvfp4" | "fp8" | "bf16"
    # Robot output schema (Apple/RDNA Pi0.5 / Jetson Pi):
    action_dim: int | None = None,
    # Keyword-only, GR00T N1.7 RDNA input and graph controls:
    fuse_image_embeds: bool | None = None,
    use_backbone_graph: bool | None = None,
) -> VLAModel

Returns a VLAModel wrapping the appropriate frontend for the detected (or explicitly specified) GPU architecture.

  • action_dim declares the robot output width for Apple/RDNA Pi0.5 (1..32). If omitted, this backend requires output_action_dim in checkpoint config.json; it never infers padding from constant normalization statistics. Constant final action channels are retained. Other existing hardware routes keep their output conventions.

  • decode_cuda_graph, decode_graph_steps, max_decode_steps apply to Pi0-FAST.

  • embodiment_tag and action_horizon apply to GROOT.

  • use_fp4, fp4_layers, use_awq, awq_alpha, and use_p1_split_gu apply to the Pi0.5 torch and JAX NVFP4 encoder path on Thor. The JAX path loads Orbax checkpoints; the torch path loads safetensors checkpoints.

  • use_fp4_decoder, use_fp4_encoder_attn, use_fp4_siglip_ffn, encoder_p1_combiner, encoder_down_variant and decoder_gate_up_variant extend that Pi0.5 Thor NVFP4 path. They require use_fp4=True and Pi0.5 torch on Thor; a True value on any other config or hardware raises rather than being silently dropped.

  • use_fp4_decoder=True selects the complete Thor NVFP4 tier that the published latency table is measured with, so the sub-flags left at None resolve to it: use_fp4_encoder_attn=True, use_fp4_siglip_ffn=True, encoder_p1_combiner="epilogue_hw" and awq_alpha=0.8. With use_fp4=True alone they resolve to the encoder-only preset (False, False, "lut_native", 0.5). Any of them passed explicitly overrides the preset. The measured configuration is therefore exactly:

    model = load_model(checkpoint, config="pi05", framework="torch",
                       hardware="thor", num_views=3,
                       use_fp4=True, use_fp4_decoder=True, use_fa4=True)
  • use_fa4 selects the FA4 attention backend for SigLIP and encoder attention on Pi0.5 torch/Thor. It is not part of the NVFP4 tier and does not require use_fp4: both the FP8 frontend and its NVFP4 subclass accept it, and the published NVFP4 speedups are measured against an FP8 baseline that also runs FA4. It raises on any other config, framework or hardware.

  • use_fp4=True with config="groot_n17" on Thor selects the GROOT N1.7 NVFP4 tier (NVFP4 DiT action head + cross-KV, vectorized backbone kernels, FA4 attention when available). It takes no Pi0.5 sub-flags and falls back to the FP8 tier when the flash_rt_fp4 extension is missing.

  • num_steps, vision_pool_factor, vision_num_layers, and cache_frames apply only to frontends that expose those constructor parameters today. The Pi0.5 torch RTX/Orin frontend validates vision_pool_factor in {1, 2, 4}, vision_num_layers in [1, 27], and cache_frames >= 1.

  • state_prompt_mode applies to Pi0.5 RTX/Thor state-in-prompt execution. "exact" tracks the exact token length (RTX caches recurring lengths; Thor reuses same-length updates). "fixed" captures one max-length graph and masks padded state-prompt tokens with a device-side valid length; use it when live robot state changes make token lengths drift. The default remains "exact", so existing calls keep the exact-length path and the 50-step Pi0.5 action graph is unchanged.

  • state_prompt_fixed_max_len applies only to Pi0.5 Thor fixed mode. None keeps the default 200-token state-prompt cap; serving code can lower it when it knows the live state-prompt bound. The cap must cover the actual token length. On Thor, a close cap such as 120 for a 117-token prompt measured roughly a 1 ms normal overhead versus a warmed exact graph, while larger caps pay for the extra padded tokens. It can also be set with FLASHRT_PI05_STATE_PROMPT_FIXED_MAX_LEN.

  • hardware="amd_cdna4" (auto-detected on ROCm builds for gfx950 / MI350-series) routes config="pi05", framework="torch" to the self-contained AMD backend (flash_rt/amd/, HIP module flash_rt_amd_kernels). Single-stream surface only: set_prompt / calibrate / infer with both state_prompt_mode graphs; the RL and CFG-batched extensions raise NotImplementedError. Build and usage: docs/deployment_amd.md.

  • hardware="apple_mps" (auto-detected on Apple Silicon when MPS is available and CUDA/Ascend are unavailable) supports only config="pi05", framework="torch". Execution is BF16 eager with precision="auto" or precision="bf16"; the historical use_fp8=True default emits a warning and selects BF16. Explicit FP8/INT8 precision, FP4, FP16 and FA4 requests raise. This route requires no CUDA/HIP extension, calibration or graph capture; autotune has no effect and cache_frames must be 1 (or None). The eager route uses exact prompt lengths, so fixed state-prompt graph options raise. The separately enabled experimental native Metal plan can reuse bounded prompt storage across length changes; see the deployment guide. Images must be two or three RGB uint8 arrays of shape (224, 224, 3); state supplied to predict()/set_prompt() must already be normalized with the checkpoint's state statistics. Set action_dim or declare output_action_dim in checkpoint config.json. Setup, real-input smoke results and the limitations of the CPU comparison are documented in Apple deployment.

  • use_fp8=False disables FP8 where the selected frontend exposes a BF16 fallback; unsupported frontends ignore it. GROOT N1.7 on RTX/Thor is stricter: the default route is FP8, and use_fp8=False alone raises because there is no separate BF16-only path.

  • use_fp16=True selects the opt-in reference path. It requires use_fp8=False and is currently valid for:

    • config="pi05", framework="torch", hardware in {"rtx_sm120", "rtx_sm89"}
    • config="groot", framework="torch", hardware in {"thor", "rtx_sm120"}
    • config="groot_n17", framework="torch", hardware in {"thor", "rtx_sm120", "rtx_sm89"}
  • config="motus" is a beta RTX SM120 frontend. It expects a Motus checkpoint plus Wan and VLM checkpoint paths supplied to the Motus quickstart/frontend; see docs/motus_usage_beta.md.

  • config="wan22_ti2v_5b" is an RTX SM120 official-pipeline Wan2.2 baseline. It exposes set_prompt(prompt, negative_prompt=...) and infer(mode="t2v"|"i2v", width=..., height=..., frames=..., steps=..., shift=..., guide_scale=..., seed=..., teacache=False, teacache_threshold=..., teacache_start_step=..., teacache_end_step=..., teacache_cache_device=...); predict() is not part of this video-generation API. See docs/wan22_usage.md.

  • config="cosmos3_video" is an RTX SM120 Cosmos3-Nano text2video FP8 denoise model (non-VLA). It exposes set_prompt(ref=<reference dump>) for conditioning and infer(teacache_skip=..., shift=..., compare_ref=..., return_metadata=...), returning the denoised vision latent; predict() is not part of this API. Precision is selected with load_model(..., use_fp8=True|False). See docs/cosmos3_video_usage.md.

  • config="cosmos3_edge" is the Thor integration for NVIDIA Cosmos Framework. It exposes set_prompt(input_json=...) or set_prompt(sample=...), then infer(output_dir=..., vae_path=..., cosmos_root=...). The official backend establishes the end-to-end reference; the fixture-backed FlashRT backend validates and benchmarks the optimized denoise boundary. See docs/cosmos3_edge_thor.md.

  • config="groot_n15" is registered for framework="torch" on hardware="rtx_sm89" only, as FlashRT kernels in one CUDA graph per prompt length: FP8 by default, BF16 (the reference's algorithms) with use_fp8=False. It needs flash_rt_kernels, flash_rt_fa2 and flash_rt_groot_n15 (-DFLASHRT_ENABLE_GROOT_N15=ON) and the upstream Isaac-GR00T gr00t package for the observation / action transforms (set $GROOT_N15_ROOT and $GROOT_N15_DATA_CONFIG, or pass groot_root= / data_config= to the frontend); predict(...) returns the GR00T per-key action dict. See docs/groot_n15_rtx_sm89.md.

  • config="groot_n17" is registered for framework="torch" on hardware in {"thor", "rtx_sm120", "rtx_sm89", "rtx_sm87", "amd_cdna3", "amd_cdna4", "amd_rdna3", "amd_rdna35", "amd_rdna4", "npu"}. On RTX, rtx_sm120 resolves through the historical shared RTX registration and load_model() refines that default route to the FP8 production frontend; rtx_sm89 resolves directly to its dedicated SM89 frontend. use_fp16=True, use_fp8=False requests the explicit RTX reference frontend for the selected hardware. On npu the tier is BF16 and precision="int8" is refused rather than downgraded; that backend's native kernels are built per model, so FLASHRT_ENABLE_NPU_GROOT_N17=ON is required before this route can load. See docs/deployment_npu_groot_n17.md. On every hardware this model takes its observation bundle from the caller, so it is driven through model.pipeline — set_prompt(aux=...), normalize_state, infer, denormalize_action — rather than through VLAModel.predict. RDNA (gfx1100, gfx1151, gfx1201) runs BF16. As with other RDNA models, load_model converts use_fp8=True (including the loader default) to BF16 with a warning; pass use_fp8=False to select it explicitly. FP4/FP16 requests fail before backend loading. Direct RDNA frontend construction rejects FP8. Its default fuse_image_embeds=False accepts the existing projected observation bundle (pixel_features, llm_input_embeds, grid_thw, visual_pos_masks, rope_cos, rope_sin). Explicit fuse_image_embeds=True selects Orin's raw-patch/token-ID contract and enables infer(state_normalized, frames=...). Extra dictionary fields never select a different mode. Projected inputs include model computation performed by the caller and must match the checkpoint and observation. load_model(fuse_image_embeds=None, use_backbone_graph=None) preserves frontend defaults. On RDNA, fusion defaults off and backbone graphs default on. Explicit values must be bools and are supported only for N1.7 torch on RDNA; other routes retain their existing construction interfaces. use_dit_graph remains an independent inference argument. RDNA provides an optional processor-only prepare_observation helper and set_hf_processor injection for offline use. frames accepts ordered RGB uint8 arrays/tensors or a sequence of images, including every history frame in time-then-camera order. It cannot accompany aux['pixel_values'] and does not replace prompt setup or state normalization. flash_rt.amd.frontends.torch.groot_n17_rdna.frames_from_observation(observation, camera_order) packs batch-one video arrays in this order; use the checkpoint's video modality keys for camera_order. RDNA uses the checkpoint's official host image transforms, including non-square images; device frames incur a host transfer. Processing stays outside the HIP graph. Input/output semantics remain the normalized-state infer and explicit physical action decoding workflow. See AMD GR00T deployment.

  • config="chameleon" is a chat-style VLM and is not served through load_model's VLA wrapper. Calling load_model(config="chameleon") raises NotImplementedError with direct-construction instructions. Construct the frontend explicitly: ChameleonTorchFrontendRtxSm87 (Jetson Orin SM87) or ChameleonTorchFrontendThor (Jetson Thor SM110). See docs/chameleon_usage.md.

  • config="hyvla" (Hy-Embodied-0.5-VLA) is registered for framework="torch" on hardware in {"thor", "rtx_sm87", "rtx_sm120"}. Thor and Orin SM87 use runtime dynamic FP8 with fused megakernels and optional NVFP4 FFN (use_fp4=True). RTX SM120 runs SM120 block-128 FP8 and defaults to an NVFP4 ViT + VLM-prefill tier with the expert denoise tower on FP8 (use_fp4=True, use_fp4_expert=False); use_fp4_expert=True promotes the expert tower to NVFP4 (opt-in, below the 0.999 precision gate). See docs/hyvla05_thor_sm110.md, docs/hyvla05_orin_sm87.md, and docs/hyvla05_rtx_sm120.md.

  • config="xvla" (lerobot/xvla-libero) is registered for framework="torch" on hardware="rtx_sm120" (the detected arch for SM120 and SM121). xvla_precision selects "nvfp4" (default), "fp8", or "bf16" for the action transformer and encoder; the DaViT tower runs NVFP4 W4A4. predict(images, prompt, state) is supported; num_views=2 (or 3 when the caller includes the configured empty slot). See docs/xvla_rtx_sm120.md.

  • config="molmoact2" (allenai MolmoAct2) is registered for framework="torch" on hardware="rtx_sm120" (SM120/SM121). It is not served through the predict(images, prompt) wrapper: every call carries the 14-dim robot state, and the graph must be calibrated on real observations before capture. Calling load_model(config="molmoact2") raises NotImplementedError with direct-construction instructions (MolmoAct2TorchFrontendRtx; set_prompt, calibrate, record_infer_graph, infer). Requires -DFLASHRT_ENABLE_MOLMOACT2=ON. See docs/molmoact2_usage.md.

Pi0.5 Thor BF16

Jetson AGX Thor (SM110) has an opt-in BF16/FP32 mode for the OpenPI Pi0.5 LIBERO PyTorch checkpoint. The mode accepts one 224 × 224 RGB uint8 image, a text-only prompt, and returns a 10 × 7 action chunk. Build with -DFLASHRT_ENABLE_PI05_THOR_BF16=ON and pass precision="bf16", num_views=1, and use_fp8=False. See the single-view Thor guide.

Pi0.5 optimized Thor decoder

For the optimized torch/Thor full NVFP4 path, build with -DFLASHRT_ENABLE_PI05_THOR=ON and call:

flash_rt.load_model(checkpoint, config="pi05", framework="torch", hardware="thor",
                    num_views=2, use_fa4=True, use_fp4=True, use_fp4_decoder=True,
                    awq_alpha=0.5, encoder_p1_combiner="epilogue_hw_nod",
                    encoder_down_variant=8)

The optimized decoder uses the Pi0.5-owned flash_rt_pi05_thor module. An absent module raises an error naming the required build flag. Encoder-only, JAX and other hardware routes retain their existing frontend. See the Thor guide for calibration and precision checks.

Pi0.5 state prompts zero-pad shorter normalized states to 32 dimensions before discretization, matching OpenPI's tokenizer input. Text-only prompts are unchanged.

flash_rt.VLAModel

class VLAModel:
    def set_prompt(self, *args, **kwargs): ...
    def infer(self, *args, **kwargs): ...
    def predict(self, images, prompt=None, state=None) -> np.ndarray: ...
    def calibrate(
        self,
        observations,
        *,
        percentile: float = 99.9,
        max_samples: int | None = None,
        verbose: bool = False,
    ) -> None: ...
    def warm_state_prompt_buckets(self, images, prompt, states) -> list[int]: ...
    def recalibrate(self) -> None: ...

    @property
    def framework(self) -> str: ...
    @property
    def prompt(self) -> str | None: ...
    @property
    def pipeline(self): ...
  • set_prompt(*args, **kwargs) — delegate prompt setup to the selected frontend. GROOT N1.7 currently uses set_prompt(aux=..., prompt=...), where aux contains the captured Qwen3-VL setup tensors consumed by the N1.7 calibration path.

  • infer(*args, **kwargs) — delegate inference to the selected frontend. GROOT N1.7 currently uses infer(state_normalized, aux=..., initial_noise=..., use_dit_graph=...) and returns normalized actions. For the RTX FP8 frontend, optional aux supplies fresh observation/model inputs and lazily captures, then runs, the backbone graph before the action graph. Fresh inputs must retain the grid_thw, visual mask, RoPE tables, and tensor shapes established by set_prompt(). Omitting aux reuses the eager backbone features without capturing the optional graph.

    RDNA accepts the same num_inference_timesteps, action_horizon, num_timestep_buckets, initial_noise, and use_dit_graph keywords. Omitted schedule values use checkpoint/construction defaults; explicit positive values reconfigure the schedule and invalidate affected graphs. action_horizon cannot exceed the checkpoint horizon. Omitting aux reuses the most recently supplied observation (initially the setup observation). Supply fresh aux for every new camera observation. use_dit_graph=False disables DiT capture; the backbone may still use its graph. The RDNA frontend's use_backbone_graph constructor flag controls that stage; use_graph=False on infer selects fully eager execution. The default fresh observation path captures both stages in one graph. Frontend predict is an alias of normalized-state infer, as on CUDA/CDNA.

  • predict(images, prompt, state) — run one inference step. images: list of (224,224,3) uint8 numpy arrays, or a dict with "image" / "wrist_image" / "wrist_image_right" keys. prompt: required on first call, reused on subsequent calls if None. state: optional robot state array. predict() attaches it to the observation as "state" when the caller passes image lists, while preserving an explicit "state" already present in a dict observation. For frontends whose set_prompt() accepts state, predict() refreshes the prompt prefix when either the prompt text or that prompt-state value changes. Returns np.ndarray of shape (action_horizon, action_dim).

    state is part of the VLA observation schema; each model encodes it through its own reference contract:

    • Pi0 uses a continuous state token in the action-expert suffix.
    • Pi0.5 encodes state as openpi-compatible discretized prompt tokens: Task: <prompt>, State: <openpi state bins>;\nAction: . The bins use OpenPI's np.digitize(state, np.linspace(-1, 1, 257)[:-1]) - 1 convention: normalized in-range values usually become 0..255, while values below -1 become -1. RTX/Thor torch frontends and the JAX Thor Pi0.5 frontend accept state in set_prompt() and through predict(). Same-length state prompt updates reuse the captured graph. RTX exact mode reuses cached recurring prompt lengths; RTX/Thor fixed mode uses one max-length graph for drifting state-token lengths.
    • Pi0-FAST encodes state in the FAST token prefix.
    • GROOT N1.6 consumes proprioceptive state from obs["state"]; if omitted, the backend uses zeros.
    • GROOT N1.7 currently uses the lower-level normalize_state(...) + infer(state_normalized, initial_noise=...) contract rather than the image-list predict() contract.
  • calibrate(observations, *, percentile=99.9, max_samples=None, verbose=False) — run the selected frontend's public calibration path. observations may be a single observation dict or an iterable of samples in the frontend's calibration format. max_samples caps the consumed sample list before the frontend chooses its N=1 or N>=2 path. Frontends that document N>=2 support run dataset calibration with percentile-clipped amax reduction; unsupported frontends raise a clear NotImplementedError. GROOT N1.7 uses captured aux dict samples rather than raw image observation dicts.

  • warm_state_prompt_buckets(images, prompt, states) — Pi0.5 RTX torch helper for realtime loops that pass changing state through the prompt. It calls the selected frontend's bucket warmup hook, using images as the calibration/capture observation and each item in states as a representative robot state. The return value is the sorted list of warmed prompt lengths. The method preserves the OpenPI state prompt text format; it does not zero-pad or otherwise rewrite state tokens. A recurring prompt token length reuses its cached runtime pipeline, but a previously unseen state-token length still pays one first-time bucket build/capture cost. Serving code should keep state serialization stable (fixed state dimension and numeric rounding/precision policy) and prewarm representative states from the deployment range, for example reset, mid-rollout, near-goal, and a few recorded rollout observations whose discretized values may tokenize to different lengths.

  • recalibrate() — clear FP8 calibration cache and force re-calibration on the next predict() call.

  • pipeline — the wrapped frontend instance. predict() is the supported inference entry point; this accessor exists for benchmarking and validation harnesses that need the frontend's own surface (for example the raw pre-decode noise buffer) while still constructing the model through load_model(), so that a measured configuration is by construction the one the public API produces.


Hardware dispatch (flash_rt.hardware)

from flash_rt.hardware import detect_arch, resolve_pipeline_class

detect_arch() -> str

Returns "thor", "rtx_sm120", "rtx_sm89", or "rtx_sm87" based on the current CUDA device's compute capability. SM87 is currently supported only for config="pi05", framework="torch".

resolve_pipeline_class(config, framework, arch)

Lazily imports and returns the concrete frontend class for the given (config, framework, arch) triple. Used internally by load_model. Motus beta is registered for (config="motus", framework="torch", arch="rtx_sm120"). GROOT N1.7 is registered for (config="groot_n17", framework="torch", arch in {"thor", "rtx_sm120", "rtx_sm89", "rtx_sm87", "amd_cdna3", "amd_cdna4", "amd_rdna3", "amd_rdna35", "amd_rdna4", "npu"}). On RTX, rtx_sm120 keeps the shared-base registration and load_model() refines that resolved class to the FP8 default or the explicit RTX reference frontend based on use_fp8 / use_fp16; rtx_sm89 resolves directly to the dedicated SM89 frontend class. Wan2.2 TI2V-5B is registered for (config="wan22_ti2v_5b", framework="torch", arch="rtx_sm120"). Chameleon-7B is registered for (config="chameleon", framework="torch", arch in {"rtx_sm87", "thor"}). Hy-Embodied-0.5-VLA is registered for (config="hyvla", framework="torch", arch in {"thor", "rtx_sm87", "rtx_sm120"}). VLA-JEPA is registered for (config="vla_jepa", framework="torch", arch="rtx_sm120") for discovery; load_model raises an explicit redirect to VlaJepaTorchFrontendRtx (per-call state and real-data calibration).

_PIPELINE_MAP

flash_rt.hardware._PIPELINE_MAP: dict[tuple[str, str, str], tuple[str, str]]

The dispatch table mapping (config, framework, arch) to (module_path, class_name). External plugins may mutate this dict at import time to register new models — see Plugin Model Template.


AttentionBackend protocol (flash_rt.hardware.backend)

from flash_rt.hardware.backend import (
    AttentionBackend,    # Protocol class
    AttentionBackendBase,  # Optional base with accessor defaults
    AttentionSpec,       # Full model attention specification
    SiteSpec,            # One attention site descriptor
)

SiteSpec

@dataclass
class SiteSpec:
    num_layers: int
    num_q_heads: int
    num_kv_heads: int
    head_dim: int
    max_q_seq: int
    max_kv_seq: int | None = None     # None → self-attention
    batch_axis: int = 1
    sliding_window: int | None = None  # reserved for SWA models
    causal: bool = False
    extra: dict = field(default_factory=dict)

AttentionSpec

@dataclass
class AttentionSpec:
    sites: dict[str, SiteSpec]
    def add_site(self, name: str, **kwargs) -> AttentionSpec: ...
    def site(self, name: str) -> SiteSpec: ...

AttentionBackend (Protocol)

class AttentionBackend(Protocol):
    def sites(self) -> tuple[str, ...]: ...
    def get_slot_ptrs(self, site: str, layer_idx: int) -> dict[str, int]: ...
    def run(self, site: str, layer_idx: int, q_seq: int, *,
            kv_seq: int | None = None, stream: int = 0) -> int: ...
    def head_dim(self, site: str) -> int: ...
    def num_q_heads(self, site: str) -> int: ...
    def num_kv_heads(self, site: str) -> int: ...

Reference implementations (reusable, not stable)

These concrete backends are available for reuse by plugins. Their existence is stable, but their internal signatures may change between minor releases. Plugins that subclass or call into these should pin a minor version.

from flash_rt.hardware.rtx.attn_backend       import RtxFlashAttnBackend
from flash_rt.hardware.rtx.attn_backend_groot import RtxFlashAttnBackendGroot

Both classes are framework-neutral — used by torch and jax frontends alike. The old names TorchFlashAttnBackend / TorchFlashAttnBackendGroot are kept as deprecated module-level aliases and will be removed in the next major version.


Core utilities (flash_rt.core)

from flash_rt.core.cuda_buffer import CudaBuffer
from flash_rt.core.cuda_graph import CUDAGraph
from flash_rt.core.quant.calibrator import load_calibration, save_calibration

These are used by both pipelines and frontends. Their public API is stable; internal helper functions are not.


Native extension modules

FlashRT ships two pybind11 Python extension modules:

from flash_rt import flash_rt_kernels   # always present
from flash_rt import flash_rt_fa2       # RTX (SM80/86/89/120) only

flash_rt.flash_rt_kernels

The main kernel module — hand-written CUDA code plus cuBLASLt/ CUTLASS wrappers for memory-bound ops (norm, activation, fusion, FP8 quant, residual, gate-geglu, true-silu, etc.) and Thor-specific attention (fvk.attention_qkv_fp16). Binary name pattern: flash_rt_kernels.cpython-<abi>.so, ~3 MB.

All fvk.<symbol>(...) calls seen in pipeline code live here. Signatures are internal — plug-ins should go through the AttentionBackend protocol, not call fvk.* directly.

Motus beta SM120 kernels are built into flash_rt_kernels when CMake is configured with -DFLASHRT_ENABLE_MOTUS=ON (currently the default). They do not require a separate runtime kernel library directory.


Runtime utilities

from flash_rt.runtime.rtc import (
    ActionChunkAdapter,
    CallablePolicyAdapter,
    AsyncChunkRunner,
    RTCStats,
)

from flash_rt.runtime import (
    AsyncTemporalFusionRunner,
    FusedChunk,
    ObservationSnapshotter,
    PredictionTicket,
    TemporalFusionBuffer,
    TemporalFusionConfig,
    TemporalFusionStats,
    TimedActionChunk,
)

from flash_rt.runtime.vlash import (
    AsyncVLAShRunner,
    VLAShChunkResult,
    VLAShConfig,
    VLAShStats,
)

The legacy async chunk runner is a beta inference scheduling utility for action-chunk policies. It does not change model numerics or calibration; it only serves action chunks at a fixed controller rate while a background worker prepares the next chunk. See docs/rtc_lite_design.md.

The temporal-fusion runner is an opt-in scheduling policy that retains raw predicted chunks, aligns them on the controller-step timeline, and fuses up to TemporalFusionConfig.max_chunks overlapping predictions with exponential position-difference weights. It supports latency- and state-based chunk switching without changing model kernels or calibration. See docs/rtc_temporal_fusion.md for the adapter contract, configuration, deadline behavior, and real Pi0.5 checkpoint gate.

AsyncVLAShRunner is an optional host-side runtime for projected-state action-chunk scheduling. It estimates a future robot state from the active chunk, injects that state into the next observation, and activates the completed chunk from index zero. See docs/vlash.md.

flash_rt.flash_rt_fa2

Vendored Flash-Attention 2 v2.7.4.post1 (forward only, fp16 + bf16, SM80-family SASS). Binary name pattern: flash_rt_fa2.cpython-<abi>.so, ~135 MB. Only built when GPU_ARCH ∈ {80, 86, 87, 89, 120}. Exposes:

flash_rt_fa2.fwd_fp16(
    Q, K, V, O, softmax_lse,
    softmax_lse_accum=0, o_accum=0,   # splitkv scratch ptrs; 0 disables splitkv
    *,
    batch, seqlen_q, seqlen_k,
    num_heads_q, num_heads_kv, head_dim,
    q_strides, k_strides, v_strides, o_strides,   # 3-tuples (batch, row, head) in elements
    softmax_scale=1.0,
    num_sms=0,                         # required for splitkv heuristic
    stream=0,
)
flash_rt_fa2.fwd_bf16(...)            # same signature, bfloat16 dtype

All pointer args are int device pointers (tensor.data_ptr()). Pipeline code should go through RtxFlashAttnBackend (which calls this module internally) rather than invoking it directly — direct use is unstable and may change without notice. RtxFlashAttnBackend selects between this module and the pip flash-attn wheel via the FVK_RTX_FA2 env var (default "1" = use vendored FA2; "0" = fallback to flash_attn.flash_attn_func). The backend name reflects the hardware family (RTX), not the frontend framework — the same backend instance serves both torch and jax frontends.

Thor (SM110) builds do not produce this module — attention on Thor uses flash_rt_kernels.attention_qkv_fp16 (cuBLAS-decomposed) because FA2's Ampere tile shapes aren't tuned for Thor's unified LPDDR memory model. Code importing flash_rt_fa2 must therefore guard for ImportError on Thor deployments, or stay inside the AttentionBackend protocol which handles the dispatch transparently.


Directory structure (post-refactor)

flash_rt/
├── __init__.py              # load_model, VLAModel
├── api.py                   # load_model implementation
├── core/                    # shared utilities (CudaBuffer, CUDAGraph, calibrator)
├── hardware/
│   ├── __init__.py          # detect_arch, _PIPELINE_MAP, resolve_pipeline_class
│   ├── backend.py           # AttentionBackend protocol
│   ├── rtx/                 # rtx attention backends (hardware primitives only)
│   └── thor/
│       ├── attn_backend.py        # Thor FMHA wrapper
│       ├── attn_backend_groot.py  # GROOT-specific Thor attention
│       └── shared_primitives.py   # CLOSED SET: model-agnostic Thor helpers only
│                                  #   (_gpu_*, _measure_scale_gpu,
│                                  #    siglip_forward, encoder_forward,
│                                  #    encoder_forward_calibrate)
├── models/
│   ├── pi05/
│   │   ├── pipeline_thor.py       # Pi0.5 Thor compute (postln_project, decoder_*)
│   │   └── pipeline_rtx.py        # Pi0.5 RTX Pi05Pipeline class
│   ├── pi0/
│   │   ├── pipeline_thor.py       # Pi0 Thor decoder fns
│   │   └── pipeline_rtx.py        # Pi0 RTX Pi0Pipeline class
│   ├── pi0fast/
│   │   └── pipeline.py            # DEPRECATED PATTERN: Thor+SM120 runtime fork
│   │                              # do NOT copy this style for new models
│   └── groot/
│       ├── pipeline_thor.py       # GROOT Thor pipeline
│       ├── pipeline_rtx.py        # GROOT RTX pipeline
│       └── embodiments.py         # per-embodiment MLP slots
└── frontends/
    ├── torch/
    │   ├── pi05_thor.py    (Pi05TorchFrontendThor)
    │   ├── pi05_rtx.py     (Pi05TorchFrontendRtx)
    │   ├── pi0_thor.py     (Pi0TorchFrontendThor)
    │   ├── pi0_rtx.py      (Pi0TorchFrontendRtx)
    │   ├── pi0fast.py      (Pi0FastTorchFrontend)    DEPRECATED — Thor+RTX hybrid
    │   ├── groot_thor.py   (GrootTorchFrontendThor)
    │   └── groot_rtx.py    (GrootTorchFrontendRtx)
    └── jax/
        ├── pi05_thor.py    (Pi05JaxFrontendThor)
        ├── pi05_rtx.py     (Pi05JaxFrontendRtx)
        ├── pi0_thor.py     (Pi0JaxFrontendThor)
        ├── pi0_rtx.py      (Pi0JaxFrontendRtx)
        └── pi0fast.py      (Pi0FastJaxFrontend)      DEPRECATED — Thor+RTX hybrid

Naming convention (established 2026-04, stage 8 unified-pipeline-layout refactor):

  • Every (model, hardware) compute path is its own file: models/<m>/pipeline_<hw>.py where <hw> ∈ {thor, rtx}. No pipeline.py (no-suffix default entry) is allowed.
  • Every (model, framework, hardware) IO path is its own frontend file: frontends/<fw>/<m>_<hw>.py with class <Model><Fw>Frontend<Hw> (e.g. Pi05TorchFrontendThor, Pi05TorchFrontendRtx).
  • No runtime hardware forks (if self._has_sm100, hasattr(fvk, ...)): if a model needs different code on Thor vs RTX, those are separate files.
  • hardware/<hw>/shared_primitives.py is a closed set of model-agnostic helpers. Model-specific forwards/decoders go into models/<m>/pipeline_<hw>.py, never into shared_primitives.py.
  • _PIPELINE_MAP is one-to-one: each (model, framework, hw) tuple routes to exactly one frontend file/class.

Known historical exception (do NOT copy for new models):

  • pi0fast ships as a single pipeline.py with 14+ if self._has_sm100 branches and a single multi-hw frontend file. This in-file SM-fork pattern is retained for the existing Pi0-FAST implementation only; new models should follow the standard (model, framework, hw) split.

Declarative weight loading (stage 7)

Thor frontends' per-layer weight-loading loops are expressed as ModelWeightSpec objects in private spec modules next to the frontend. The public surface is three things:

from flash_rt.executors.weight_loader import (
    Item, LayerBlock, ModelWeightSpec, WeightLoader,
)
from flash_rt.executors.torch_weights import (  # torch side
    SafetensorsSource, DictSource,
    Cat, FusedQKV, FusedGateUp,
    ToFp16, ToFp32, T, tT, InterleaveQK, Quant, Mul,
    Attr, TensorList, FlatCat,
)
from flash_rt.executors.jax_weights import (  # jax side
    OrbaxDictSource,
    Transpose, Astype, Contiguous, JaxQuant,
    NumpyAttr, NumpyList, CudaBufferAttr, CudaBufferFlat,
)

Stability contract: the classes above and their constructor signatures are public. Adding new sink/transform/composite classes is backwards-compatible; existing ones will not be removed or renamed without a major version bump.

Spec file naming: flash_rt/frontends/{torch,jax}/_<model>_thor_spec.py, each exporting a build_spec() -> ModelWeightSpec. Shared block builders live in _thor_spec_common.py per framework.

Convention for scale lists: spec items that quant set scale_into="_<group>_scales"; frontends wrap these into device tensors after loader.run():

self._enc_w_dev = torch.tensor(self._enc_w_scales,
                               dtype=torch.float32, device='cuda')

See docs/adding_new_model.md for the end-to-end model adaptation walkthrough, and docs/plugin_model_template.md for registering an external-plugin model via _PIPELINE_MAP.


Adaptation / extension guides

Pi0.5 RTX Sampling Exports

The Pi0.5 RTX frontend's infer and infer_batch accept optional noise or generator arguments. They are mutually exclusive for the initial noise draw. The default result does not contain a noise host copy. Pass return_noise=True to export the BF16-rounded initial noise under "noise", after inference completes. This also applies to batched CFG. The construction-time denoise_trace and prefix_features exports remain opt-in; batch prefix pooling excludes each slot's padded prompt positions.

When adding a new model or kernel, read these in order:

  1. docs/adding_new_model.md — end-to-end walkthrough for wiring a new VLA model into FlashRT on Thor (AttentionSpec → WEIGHT_SPEC → pipeline forward → frontend → calibration → graph capture → registration → tests).
  2. docs/calibration.md — FP8 weight/activation scale mechanics, alpha = act_scale × weight_scale invariants, calibration cache format, and the four historical bugs every new model's _calibrate should guard against.
  3. docs/kernel_fusion.md — the 93 public fvk kernels grouped by purpose, current production fusion patterns, what does and does not fuse, and the catalog of failed optimizations (OPT-3 / OPT-5 / v1.5-B2 / v1.5-B4) to avoid.

DM05-LIBERO on RTX SM120

load_model(checkpoint, config="dm05", framework="torch", hardware="rtx_sm120", use_fp4=True, num_views=2) selects the DM05 frontend and its mixed NVFP4/FP8/INT8 tier. Existing model defaults are unchanged. DM05 requires its external model distribution for loading and IO transforms, and FLASHRT_ENABLE_DM05=ON for kernels.

Before predict(images, prompt=..., state=...), load calibrated scales through model.pipeline.load_calibration_state(state_or_path), or call model.calibrate(observations, percentile=100.0) with representative real data. The frontend consumes the model package; it does not replace runtime methods. LIBERO outputs are (10, 7). Other hardware/frameworks and unimplemented precision requests fail explicitly. Full-action numerical and gripper-decision validation remains required; see DM05 validation.

LaWAM on RTX SM120

load_model(checkpoint, config="lawam", framework="torch", hardware="rtx_sm120") selects LawamTorchFrontendRtx for the LeRobot LaWAM checkpoint; it needs FLASHRT_ENABLE_LAWAM=ON for flash_rt_lawam. predict([primary, wrist], prompt=task) takes two uint8 256x256 RGB views and returns the unnormalized (action_horizon, 7) chunk; infer(frames, noise=...) accepts the initial flow sample for reproducible comparisons. Other hardware/frameworks fail explicitly. Precision tiers and validation: LaWAM on RTX SM120.

VLA-JEPA (LeRobot) on RTX SM120

load_model(checkpoint, config="vla_jepa", framework="torch", hardware="rtx_sm120") raises an explicit redirect: the policy takes the 8-dim robot state on every call and its NVFP4 block/layer selection is calibrated on real observations before capture, so it is constructed directly as VlaJepaTorchFrontendRtx. It is registered in _PIPELINE_MAP for discovery. The model-specific preprocessing kernels require -DFLASHRT_ENABLE_VLA_JEPA=ON. Other hardware, frameworks and precision tiers outside the documented RTX SM120/SM121 path fail explicitly. See VLA-JEPA usage.

Alpamayo trajectory frontends

Alpamayo produces trajectories and reasoning from camera frames and ego history, not VLAModel.predict(images). load_model(config="alpamayo15") and load_model(config="alpamayo15_flashdrive") raise an explicit redirect to their dedicated Thor frontends. Existing model routes and predict signatures are unchanged. See the consolidated guide for imports, checkpoint selection, reset behavior and four-arm reproduction.

Higgs Audio v3 TTS

Higgs uses its TTS-specific frontend rather than the VLA predict() contract:

from flash_rt.models.higgs_audio_v3 import create_frontend

fe = create_frontend("/path/to/higgs-tts-3-4b", fp8=False)
codes = fe.predict("Hello.")                  # CPU int64 [frames, 8]
waveform = fe.synthesize(codes)                # CPU float32 mono, 24 kHz
waveform = fe.generate("Hello.", temperature=1.0, seed=1234)
for packet in fe.generate_stream("Hello."):
    consume(packet)

hardware="auto" preserves the CUDA constructor and selects the RDNA frontend on ROCm. Explicit RDNA targets are amd_rdna3, amd_rdna35 and amd_rdna4. RDNA constructor defaults include device="cuda:0", max_seq=2048, max_new_frames=1024, max_prefill_tokens=512, alloc_own_forward_buffers=True and fp8=None. Auto precision selects native FP8 W8A8 on supported RDNA4 and BF16 otherwise; unsupported explicit FP8 warns and falls back to BF16.

Both frontends preserve temperature=0.0 (greedy) and optional seed on predict, decode_stream, generate and generate_stream. A temperature > 1e-5 selects native full-vocabulary categorical sampling. The same seed is repeatable within a fixed precision and algorithm selection, not a promise of cross-device identical generation.

Streaming defaults to first_chunk=8, chunk=25, ctx=8, holdback=8. It waits for right context, decodes overlapping codec windows and trims output to committed frames. Native generation retains delayed-codebook EOC winddown and the repeated-code guard. max_new_frames bounds delayed rows; RDNA also stops before exceeding max_seq. RDNA's last_finish_reason reports eoc, repeated_code, max_new_frames or max_seq. Serialize requests to each frontend; validated resident prefix KV can be reused with prefill(..., cached_tokens=...) or generate_stream(..., system=...).

See RDNA setup and usage.