Skip to content
View gangjig's full-sized avatar

Block or report gangjig

Block user

Prevent this user from interacting with your repositories and sending you notifications. Learn more about blocking users.

You must be logged in to block users.

Content in all repositories owned by your account will be closed.
Maximum 250 characters. Please don’t include any personal information such as legal names or email addresses. This note will only be visible to you.
Report abuse

Contact GitHub support about this user’s behavior. Learn more about reporting abuse.

Report abuse
gangjig/README.md

Hi, I'm Gang Ji

Software engineer working on AI agent systems — harnesses, evaluation, and the tooling that makes agents reliable in production. My open-source projects are small, tested, single-purpose tools built around ideas from current agent research, each with a design note explaining the trade-offs.

Featured projects

  • ace-in-a-box — A self-evolving playbook framework for agents (ACE-style): structured context bullets with helpful/harmful feedback, curation, and budget-aware rendering. 192 tests
  • stateledger — Explicit-state memory for long-running agents: append-only evidence kept separate from conclusions, with full provenance, plus a gap ledger that keeps unanswered questions together with the evidence tried. 125 tests
  • decidekit — Evaluate decision models (yes-no / choice / score) on accuracy, calibration (Brier, ECE), and cost per correct decision. 81 tests
  • cost-per-task — Rank LLMs by cost per successful task, not price per token — retries and failures included. 80 tests
  • agent-hooks — Pre-commit hooks that enforce your rules on AI coding agents; rejection output is a corrective instruction written for the agent. Now also a runtime hook that reminds the agent of the specific danger after each tool call returns. 129 tests
  • llm-decide — Plugin for Simon Willison's llm CLI: typed decision commands returning structured JSON instead of prose. 37 tests
  • eval-leakcheck — Pre-publish leakage scanner for eval/training task repos: finds solutions hiding in leftover Git objects, history, side branches, and the working tree. 89 tests
  • agent-replay-diff — Find where repeated runs of the same agent task diverge: volatile tool outputs (timestamps, IDs, latencies) are normalized away, then traces are diffed step by step to the first real fork. 138 tests
  • verifier-forge — Generate a verifier for an agent task from a spec (evidence head + domain rules), then self-test it against constructed candidates with known verdicts and grade the agreement — false passes are named as verifier holes and cap the grade. 49 tests
  • mcp-faultcheck — Fault-injection harness for MCP servers: a stdio proxy injects tool failures (timeouts, empty/malformed results, JSON-RPC errors, wrong-typed fields) and reports how much task success each fault type costs; a goals mode hands the agent unachievable tasks and checks whether it concedes honestly instead of spinning or claiming false success. 113 tests
  • llm-spend — Plugin for Simon Willison's llm CLI: price what your llm history actually cost, from the token usage already logged in its database. 11 tests
  • provider-parity — Check whether the same model name served through different providers is really the same model: one fixed suite of probes against OpenAI-compatible endpoints, compared with evidence. 39 tests
  • llm-cassette — VCR for LLM calls: record an agent's model calls once, replay them in tests forever — offline, deterministic, free. 40 tests
  • tracecheck — Unit tests for agent behavior: deterministic assertions over the trace your agent actually left behind, with every failure naming the exact step, plus perturbation checks that deliberately tamper with a trace to expose blind spots in the assertions. 30 tests
  • contextpack — Assemble LLM context under a hard token budget, with the drop decisions explicit and a manifest to prove them. 19 tests
  • compaction-bench — Compare context-compaction strategies (full history, single-action, end-of-conversation, sliding window, periodic summary) on the same traces: success rate, peak context, and token cost in one table. 44 tests
  • cred-proxy — Local egress proxy that swaps real API credentials in at request time: the agent sends a placeholder, only declared routes are forwarded, and secrets are redacted out of responses. 52 tests
  • trace-waste — Audit an agent's run traces for wasted tokens: repeated tool calls, redundant reasoning, and tool results or context that nothing later used — with the waste share and estimated savings broken down by category. 45 tests
  • bypass-battery — Adversarial test battery for URL validators: a fixed corpus of hostile inputs with browser ground truth (backslash authority truncation, numeric IP encodings, Unicode dots, redirect chains) catches the gaps between what a validator checks and what the consumer of the value actually does — locally, with no Docker and no LLM. 28 tests
  • poppy-probe — Probe a website's Poppy discovery file: fetch /.well-known/poppy.json under the spec's fetch rules and report what the site allows personal agents to do, what sign-in it requires, and what limits it sets. 136 tests
  • trojan-neuron-scan — Audit a model checkpoint's last MLP layer for dormant-neuron trojans: neurons that never fire on benign traces, checked for the Lazarus edit signature (aligned gate/up rows, a down column promoting one token, an outsized write), with synthetic clean/trojan fixtures for end-to-end testing. 129 tests
  • sandbox-probe — Probe whether an agent sandbox actually contains the agent: escape probes (read outside the mount, env secrets, network bypass, write to read-only dirs) with contained/escaped/error verdicts. 70 tests
  • exec-judge — Compile a scoring rubric into a readable Python judge program with one LLM call, then batch-score documents on CPU with zero model calls, reporting agreement against reference labels and cost saved. 83 tests
  • spec-first-agent — A harness that makes an agent write a machine-checkable spec (properties and invariants) before any implementation, with a sanity check that rejects vacuous specs and a done-gate that only passes when every property holds. 31 tests

What I care about

  • Agent harness design: context engineering, explicit state, self-improvement loops
  • Evaluation that matches production: calibration and cost per successful task, not leaderboard scores or per-token prices
  • Guardrails enforced by tooling, not by hoping the model follows instructions

All projects are MIT-licensed, tested, and built to be read — start with the README and the design notes.

Popular repositories Loading

  1. agent-hooks agent-hooks Public

    Pre-commit hooks that enforce your rules on AI coding agents — rejection output is a corrective instruction written for the agent.

    Python 1

  2. llm-decide llm-decide Public

    Plugin for Simon Willison's llm CLI: typed decision commands (yes-no / choice / score) returning structured JSON instead of prose.

    Python 1

  3. ace-in-a-box ace-in-a-box Public

    ACE in a box: a self-evolving playbook framework for agents — structured context bullets with helpful/harmful feedback, curation, and budget-aware rendering.

    Python 1

  4. decidekit decidekit Public

    Evaluate decision models like Jev: typed yes-no/choice/score decisions scored for accuracy, calibration (Brier, ECE), and cost per correct decision.

    Python 1

  5. cost-per-task cost-per-task Public

    Rank LLMs by cost per successful task, not price per token — retries and failures included.

    Python 1

  6. stateledger stateledger Public

    Explicit-state memory for long-running agents: append-only evidence kept separate from conclusions with full provenance.

    Python 1