R4: Build Dhyanesh orchestration, simulation and release layer

.github/workflows/ci.yml

@@ -0,0 +1,24 @@
+name: CI
+on:
+  push:
+  pull_request:
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.10", "3.12"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - run: python -m pip install .
+      - run: python -m unittest discover -s tests -v
+      - name: Simulated evaluation
+        run: |
+          if [ -f tinyagent/parser.py ] && [ -f tinyagent/executor.py ] && [ -f tinyagent/coercion.py ]; then
+            python -m eval.run_eval --mode sim
+          else
+            echo "Simulated evaluation deferred: parser, executor, or coercion module pending"
+          fi

.gitignore

@@ -0,0 +1,7 @@
+prd/
+__pycache__/
+*.py[cod]
+*.egg-info/
+.venv/
+build/
+dist/

README.md

@@ -0,0 +1,48 @@
+# TinyAgent
+
+TinyAgent routes one query to one registered Python tool through a local language
+model, with at most `1 + max_repairs` model calls. Its architecture and tradeoffs
+are documented in [ARCHITECTURE.md](ARCHITECTURE.md).
+
+Aisha's reflection, registry, prompt, and frozen contract modules are integrated.
+The transport, parser, and executor modules are owned by the other engineers
+and have not yet landed. The CLI and full evaluation require the parser and
+executor. The evaluation uses the PRD-04 fixture signatures in
+`examples/demo_tools.py`; Aisha's `tests/fixtures_tools.py` has different
+`search` and `set_alarm` signatures.
+
+## Quickstart
+
+```python
+from examples.demo_tools import build_registry
+from tinyagent import TinyAgent
+from tinyagent.backends.simulator import SimulatedBackend
+
+backend = SimulatedBackend(['ACTION: add(a="2", b="3")'])
+agent = TinyAgent(backend, build_registry())
+result = agent.run("What is 2+3?")
+print(f"{result.text} (provenance: {result.provenance})")
+```
+
+After the other modules land, run `python -m tinyagent --backend sim`.
+The CLI labels this mode `[SIMULATED]` and prints provenance next to every
+answer. A `FINAL:` answer has `provenance=model`; it is not verified.
+
+## Supported runtimes
+
+The planned live backends are Ollama and llama.cpp server. Tested runtime
+versions will be added after Deepak's transport handoff and a live run.
+Use `python -m tinyagent --backend ollama --model llama3:8b "What is 2+3?"`
+or `--backend llamacpp` when those modules are available.
+
+## Evaluation
+
+`python -m eval.run_eval --mode sim` runs the seeded fault corpus without a GPU
+once the parser and executor are available. The latest
+[evaluation report](eval/report.md) currently marks both simulated and live
+runs as not executed. No live accuracy is claimed until a live run is recorded.
+
+Known limits: one tool per query; no conversation memory, tool chaining, or
+complex argument types. A valid but wrong tool choice cannot be detected by the
+parser. Registered tools must enforce their own authorization and timeouts;
+the executor cannot kill a tool thread after timeout.

eval/__init__.py

@@ -0,0 +1 @@
+"""Offline and live TinyAgent evaluation."""

eval/faults.py

@@ -0,0 +1,66 @@
+"""Forward-generated faults from PRD-03 Section 7.
+
+The verdict table requires co-review by the parser owner before release.
+"""
+
+from __future__ import annotations
+
+import json
+import re
+from dataclasses import dataclass
+from typing import Callable
+
+from eval.scenarios import Scenario
+
+
+@dataclass(frozen=True)
+class Fault:
+    id: str
+    verdict: str
+    expected_code: str | None
+    corrupt: Callable[[Scenario], str]
+
+
+def _replace_first_arg(s: Scenario, name: str, value: str | None = None) -> str:
+    first = next(iter(s.args))
+    old = f'{first}="{s.args[first]}"'
+    if value is None:
+        return s.canonical_output.replace(old, f'{name}="{s.args[first]}"', 1)
+    return s.canonical_output.replace(old, f'{name}="{value}"', 1)
+
+
+def _positional(s: Scenario) -> str:
+    values = ", ".join(f'"{value}"' for value in s.args.values())
+    return f"ACTION: {s.tool}({values})"
+
+
+def _drop_required(s: Scenario) -> str:
+    first = next(iter(s.args))
+    parts = [f'{key}="{value}"' for key, value in s.args.items() if key != first]
+    return f'ACTION: {s.tool}({", ".join(parts)})'
+
+
+def _bad_int(s: Scenario) -> str:
+    name = next(key for key in s.args if key in {"a", "b", "limit", "hour", "minute"})
+    return s.canonical_output.replace(f'{name}="{s.args[name]}"', f'{name}="two"', 1)
+
+
+FAULTS = (
+    Fault("F01", "recover", None, lambda s: f"```\n{s.canonical_output}\n```"),
+    Fault("F02", "recover", None, lambda s: f"Here is the answer:\n{s.canonical_output}"),
+    Fault("F03", "recover", None, lambda s: f"{s.canonical_output}\nThat is all."),
+    Fault("F04", "recover", None, lambda s: s.canonical_output.replace("ACTION:", "action:", 1)),
+    Fault("F05", "recover", None, lambda s: s.canonical_output.replace('"', "'")),
+    Fault("F06", "recover", None, lambda s: re.sub(r'="([+-]?\d+)"', r'=\1', s.canonical_output)),
+    Fault("F07", "recover", None, _positional),
+    Fault("F08", "recover", None, lambda s: s.canonical_output[:-1] + ",)"),
+    Fault("F09", "repair", "E_NO_PREFIX", lambda s: json.dumps({"tool": s.tool, "args": s.args})),
+    Fault("F10", "repair", "E_BAD_SYNTAX", lambda s: s.canonical_output[:-1]),
+    Fault("F11", "repair", "E_UNKNOWN_TOOL", lambda s: s.canonical_output.replace(
+        f"ACTION: {s.tool}(", f"ACTION: {s.tool}_numbers(", 1)),
+    Fault("F12", "repair", "E_UNEXPECTED_ARG", lambda s: _replace_first_arg(s, "x")),
+    Fault("F13", "repair", "E_MISSING_ARG", _drop_required),
+    Fault("F14", "repair", "E_TYPE_COERCION", _bad_int),
+    Fault("F15", "repair", "E_EMPTY", lambda s: ""),
+    Fault("F16", "reject", "E_TOOL_EXCEPTION", lambda s: "ACTION: boom()"),
+)

eval/fixtures/README.md

@@ -0,0 +1,5 @@
+# Replay fixtures
+
+Recordings contain full prompts and model outputs. Commit only recordings
+generated from `eval/live_queries.jsonl`; keep user-data recordings outside
+this directory and out of version control.

eval/live_queries.jsonl

@@ -0,0 +1,50 @@
+{"query":"Add 2 and 3","tool":"add","args":{"a":2,"b":3}}
+{"query":"Add 0 and 0","tool":"add","args":{"a":0,"b":0}}
+{"query":"Add 4 and 9","tool":"add","args":{"a":4,"b":9}}
+{"query":"Add 17 and 25","tool":"add","args":{"a":17,"b":25}}
+{"query":"Add 100 and 1","tool":"add","args":{"a":100,"b":1}}
+{"query":"Add 8 and 13","tool":"add","args":{"a":8,"b":13}}
+{"query":"Add 12 and 5","tool":"add","args":{"a":12,"b":5}}
+{"query":"Add 42 and 58","tool":"add","args":{"a":42,"b":58}}
+{"query":"Add 1 and 999","tool":"add","args":{"a":1,"b":999}}
+{"query":"Add 123 and 456","tool":"add","args":{"a":123,"b":456}}
+{"query":"What is 6 plus 7?","tool":"add","args":{"a":6,"b":7}}
+{"query":"What is 30 plus 40?","tool":"add","args":{"a":30,"b":40}}
+{"query":"Sum 5 and 10","tool":"add","args":{"a":5,"b":10}}
+{"query":"Sum 99 and 1","tool":"add","args":{"a":99,"b":1}}
+{"query":"Compute 11 + 22","tool":"add","args":{"a":11,"b":22}}
+{"query":"Get weather in Pune in c","tool":"get_weather","args":{"city":"Pune","unit":"c"}}
+{"query":"Get weather in Delhi in c","tool":"get_weather","args":{"city":"Delhi","unit":"c"}}
+{"query":"Get weather in Mumbai in c","tool":"get_weather","args":{"city":"Mumbai","unit":"c"}}
+{"query":"Get weather in Chennai in c","tool":"get_weather","args":{"city":"Chennai","unit":"c"}}
+{"query":"Get weather in Kolkata in c","tool":"get_weather","args":{"city":"Kolkata","unit":"c"}}
+{"query":"Weather for Jaipur, c units","tool":"get_weather","args":{"city":"Jaipur","unit":"c"}}
+{"query":"Weather for Hyderabad, c units","tool":"get_weather","args":{"city":"Hyderabad","unit":"c"}}
+{"query":"Weather for Bengaluru, c units","tool":"get_weather","args":{"city":"Bengaluru","unit":"c"}}
+{"query":"Weather for Nagpur, c units","tool":"get_weather","args":{"city":"Nagpur","unit":"c"}}
+{"query":"Weather for Surat, c units","tool":"get_weather","args":{"city":"Surat","unit":"c"}}
+{"query":"Search Python, limit 1","tool":"search","args":{"q":"Python","limit":1}}
+{"query":"Search agents, limit 2","tool":"search","args":{"q":"agents","limit":2}}
+{"query":"Search Pune, limit 3","tool":"search","args":{"q":"Pune","limit":3}}
+{"query":"Search solar, limit 4","tool":"search","args":{"q":"solar","limit":4}}
+{"query":"Search maps, limit 5","tool":"search","args":{"q":"maps","limit":5}}
+{"query":"Search books, limit 6","tool":"search","args":{"q":"books","limit":6}}
+{"query":"Search trains, limit 7","tool":"search","args":{"q":"trains","limit":7}}
+{"query":"Search music, limit 8","tool":"search","args":{"q":"music","limit":8}}
+{"query":"Search science, limit 9","tool":"search","args":{"q":"science","limit":9}}
+{"query":"Search recipes, limit 10","tool":"search","args":{"q":"recipes","limit":10}}
+{"query":"Set alarm for 1:11 labelled event1","tool":"set_alarm","args":{"hour":1,"minute":11,"label":"event1"}}
+{"query":"Set alarm for 2:12 labelled event2","tool":"set_alarm","args":{"hour":2,"minute":12,"label":"event2"}}
+{"query":"Set alarm for 3:13 labelled event3","tool":"set_alarm","args":{"hour":3,"minute":13,"label":"event3"}}
+{"query":"Set alarm for 4:14 labelled event4","tool":"set_alarm","args":{"hour":4,"minute":14,"label":"event4"}}
+{"query":"Set alarm for 5:15 labelled event5","tool":"set_alarm","args":{"hour":5,"minute":15,"label":"event5"}}
+{"query":"Set alarm for 6:16 labelled event6","tool":"set_alarm","args":{"hour":6,"minute":16,"label":"event6"}}
+{"query":"Set alarm for 7:17 labelled event7","tool":"set_alarm","args":{"hour":7,"minute":17,"label":"event7"}}
+{"query":"Set alarm for 8:18 labelled event8","tool":"set_alarm","args":{"hour":8,"minute":18,"label":"event8"}}
+{"query":"Set alarm for 9:19 labelled event9","tool":"set_alarm","args":{"hour":9,"minute":19,"label":"event9"}}
+{"query":"Set alarm for 10:20 labelled event10","tool":"set_alarm","args":{"hour":10,"minute":20,"label":"event10"}}
+{"query":"Set alarm for 11:21 labelled event11","tool":"set_alarm","args":{"hour":11,"minute":21,"label":"event11"}}
+{"query":"Set alarm for 12:22 labelled event12","tool":"set_alarm","args":{"hour":12,"minute":22,"label":"event12"}}
+{"query":"Set alarm for 13:23 labelled event13","tool":"set_alarm","args":{"hour":13,"minute":23,"label":"event13"}}
+{"query":"Set alarm for 14:24 labelled event14","tool":"set_alarm","args":{"hour":14,"minute":24,"label":"event14"}}
+{"query":"Set alarm for 15:25 labelled event15","tool":"set_alarm","args":{"hour":15,"minute":25,"label":"event15"}}

eval/report.json

@@ -0,0 +1,5 @@
+{
+  "simulated": "not executed",
+  "simulated_reason": "collaborator modules pending",
+  "live": "not executed"
+}

eval/report.md

@@ -0,0 +1,6 @@
+# TinyAgent evaluation
+
+| Metric | Value |
+|---|---|
+| Simulated evaluation | not executed; collaborator modules pending |
+| Live evaluation | not executed |