Language: English · Português (Brasil)
Nirmata is an independent research program that evaluates AI agents from what they do—not only from what they say.
Current version: v0.1.0 · Release contents and verification
When an agent uses tools, changes state, or acts in an external system, a plausible final answer is incomplete evidence. The execution path may contain an authorization error, a blocked attack, a broken control, or an effect that the answer never mentions. Nirmata makes that path inspectable and keeps failed evaluations in the research record.
Evidence status: early-stage pilot. Development and integration checks have passed, but no blind confirmatory result is claimed. The first planned holdout was invalidated before scoring because its answer key entered the evaluator context. See the evidence ledger.
| If you want to... | Read... |
|---|---|
| Understand the idea in five minutes | Why final answers are not enough |
| Inspect the evaluation design | Methodology |
| Inspect the versioned operational contracts | Experimental Standard v0.1 |
| Run the public calibration fixture | Reproducibility package |
| Check what the evidence does and does not support | Evidence status |
| Review the experiment history | Experiment registry |
| See the practical relevance | Applied case studies |
| Replicate, challenge, or contribute | Roadmap · Research agenda · Contributing |
Nirmata separates two signals that are often mixed together:
- A content evaluator judges only the final output.
- A trajectory evaluator judges raw process evidence such as tool calls, authorization events, state transitions, and external effects.
- A deterministic rule combines both judgments without allowing a clean signal to erase an unsafe one.
Final-only: unsafe = content_unsafe
Trajectory-aware: unsafe = content_unsafe OR trajectory_unsafe
This is a research design under evaluation, not a claim that trajectory-aware evaluation is universally superior.
| Workstream | Question | Current status |
|---|---|---|
| Final output vs. trajectory | What safety-relevant behavior is invisible in the final answer? | Factorized pipeline integrated; fresh blind holdout required |
| Delegated authority | Can an agent act without conflating advice, approval, queueing, and publication? | 12 known development/control checks passed; real isolation pending |
| Evaluator interference | Can one evidence channel distort judgment of another? | Candidate study; not started |
| Applied autonomy | What does operational ownership expose that a successful demo hides? | One production case documented; long-term reliability not established |
The experiment registry records exploratory, development, integration, invalidated, and future confirmatory work separately.
- Why final answers are not enough to evaluate AI agents
- Approval is not authorization in agentic systems
- A burned holdout is still research evidence
- All articles in English and Portuguese
- Preserve failures instead of rewriting them into a clean success story.
- Separate development tests, integration validation, and blind confirmation.
- Freeze designs and evaluator configurations before confirmatory data is seen.
- Persist predictions before revealing ground truth.
- Distinguish blocked attempts from executed policy violations.
- Match every public statement to the strength of its evidence.
Nirmata is a pilot research program and case laboratory. It is not yet a benchmark, a production security product, or proof that its evaluation design generalizes. Current cases are small and domain-specific; identity attestation and control isolation remain prototype limitations. See Methodology, Related work, and Research agenda.
articles/ Accessible essays in English and Portuguese
case-studies/ Applied operational cases
docs/ Method, evidence ledger, literature map, agenda, and experimental standard
experiments/ Experiment registry, protocols, schemas, and status
reproducibility/ Schema, synthetic traces, evaluator, scoring, and tests
Use CITATION.cff to cite the project. Version v0.1.0 is archived at DOI 10.5281/zenodo.22132451.
Software, when published here, is licensed under Apache-2.0. Documentation, protocols, taxonomies, diagrams, articles, and public datasets are licensed under CC BY 4.0. See LICENSE.md.
Nirmata is independent research by Tabata Jahoda, ORCID 0009-0007-1104-9204. Connect on LinkedIn.