Repository navigation
Rebuild BackIntel analyst and benchmark with a Sol analysis specialist - #4
Conversation
Add the specialist runtime prerequisites, isolated analysis workspace, and report artifacts. Resolve the specialist model from the CLI, then BACKINTEL_ANALYSIS_MODEL, then ChatGPT-plan Sol with medium reasoning; preserve explicit overrides and subscription limits. Validation: staged snapshot passed 126 tests with 10 optional skips, Ruff checks, and a redacted secret scan.
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
Security findingsAdvisory findings (2)
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: dfe4c6450f
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
🛡️ Codex Security Review · Automatically triggered
Here are some automated security review suggestions for this pull request.
Reviewed commit: dfe4c6450f
ℹ️ About Codex security reviews in GitHub
This is an experimental Codex feature. Security reviews are triggered when:
- You comment "@codex security review"
- A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review
Once complete, Codex will leave suggestions, or a comment if no findings are found.
Separate invalid evaluations from capability coverage, classify subscription outages as infrastructure, reject incomplete acceptance campaigns, and reserve unique local experiments. Decouple ad-hoc questions from evaluator references. Require Docker for model-generated Python, remap local sources outside workspace roots, and isolate assigned endpoints on disposable internal networks. Keep trusted host execution exclusively in repository-owned test fixtures. Validation: 146 tests passed with 11 optional skips; Docker acceptance 8 passed with the large-data test skipped. Ruff, Vulture, and Gitleaks passed.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0a9ff10722
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| | olist-files-gpt-6-luna-L123-20261110-0127-240c2aa3 | 1 | 8 | 100% | 100% | 100% | 0.727 | 100% | 0 | 11.9 | 0.0076 | 6.5 | | ||
| | olist-files-gpt-6-luna-L123-20261110-0127-240c2aa3 | 2 | 8 | 100% | 100% | 100% | 0.686 | 100% | 0 | 18.4 | 0.0126 | 8.125 | | ||
| | olist-files-gpt-6-luna-L123-20261110-0127-240c2aa3 | 3 | 8 | 100% | 100% | 100% | 0.748 | 100% | 0 | 15.6 | 0.0091 | 7.375 | | ||
| | olist-files-gpt-6-luna-L123-20261110-0213-b3df2153 | 1 | 1 | 0% | 0% | 0% | 0.0 | 0% | 1 | 0.6 | 0.0000 | 0.0 | |
There was a problem hiding this comment.
Fresh evidence after the aggregation fix is that the committed scorecard was not regenerated: this provider-credit failure is still published as n=1 with 0% capability, while the current aggregate() would report zero valid evaluations plus attempted/invalid counts. The machine-readable scorecard has the same stale schema and values, so users and downstream consumers still interpret an infrastructure outage as an analyst regression; rebuild both scorecard artifacts from the corrected records before publishing them.
AGENTS.md reference: AGENTS.md:L61-L63
Useful? React with 👍 / 👎.
| "--network", | ||
| network or "none", |
There was a problem hiding this comment.
When run_python is used against any network-backed store while another testbed service is running, this attaches model-generated code directly to the shared backintel-net, which also contains PostgreSQL, ClickHouse, Garage, FerretDB, and its DocumentDB backend; Docker documents that containers on the same user-defined network can communicate by container name or IP (Docker CLI reference). The fresh evidence beyond the earlier analysis-sandbox issue is that source-specific isolation was added only to AnalysisSandbox, while the benchmark's PythonSandbox still exposes every peer and can let an experiment inspect data outside its bound store. Give each invocation a disposable network containing only the selected endpoint and required backend peers.
AGENTS.md reference: AGENTS.md:L54-L58
Useful? React with 👍 / 👎.
|
|
||
| import base64 | ||
| import contextlib | ||
| import fcntl |
There was a problem hiding this comment.
On the Windows desktop designated for scale factors above 1, importing backintel.bench.run unconditionally imports this module for PlanUsageStopped, but Python on Windows has no fcntl module. Consequently backintel bench run fails with ModuleNotFoundError before opening any TPC-H scenario, even when the selected provider is not chatgpt-plan; use a portable file lock or conditionally load the platform-specific implementation.
AGENTS.md reference: AGENTS.md:L78-L79
Useful? React with 👍 / 👎.
| judge=not args.no_judge, | ||
| ) | ||
| try: | ||
| scenarios = select(harness.scenarios.values(), levels=levels, ids=ids, limit=args.limit) |
There was a problem hiding this comment.
When --ids contains a typo or --levels matches no scenarios, this proceeds with an empty list; in the --no-langsmith path, run_local() then creates an empty experiment file and the command exits successfully after rebuilding the scorecard. Unlike the acceptance command's selection checks, this lets automation record a completed benchmark that evaluated nothing, so validate the requested IDs and levels and fail before starting the run.
AGENTS.md reference: AGENTS.md:L61-L63
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
🛡️ Codex Security Review · Automatically triggered
Here are some automated security review suggestions for this pull request.
Reviewed commit: 0a9ff10722
ℹ️ About Codex security reviews in GitHub
This is an experimental Codex feature. Security reviews are triggered when:
- You comment "@codex security review"
- A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review
Once complete, Codex will leave suggestions, or a comment if no findings are found.
| NETWORK = "backintel-net" | ||
| IMAGE = "postgres:17" | ||
| HOST_PORT = 55432 | ||
| ADMIN_PASSWORD = "backintel-local-admin" # local container bound to 127.0.0.1 only |
There was a problem hiding this comment.
🛡️ Codex Security Review · Automatically triggered
When a Docker-backed run uses postgres:*, a malicious or compromised model can make run_python log in to backintel-postgres with this published postgres password and issue writes. The store is automatically attached to the sandbox network, so the read-only analyst DSN and SQL filter do not apply; mutations persist in backintel-postgres-data. This build is local and pre-production, which caps severity at P2, but the model still crosses from read-only analysis to superuser access. Generate a private loader credential and make privileged authentication unreachable or disable it before model execution.
Useful? React with 👍 / 👎.
BackIntel needs reproducible analysis and evaluation before broader automation. This branch replaces the earlier demo implementation with a Deep Agents analyst, an evidence-checked LangSmith benchmark, and an opt-in sandboxed report specialist. Running
backintel analyzewithout a model now selectschatgpt-plan/gpt-6.1-solwith medium reasoning.Scope
This PR contains the full branch since
main, including the existing rebuild and supervised-improvement work, not just the final default change .--model, thenBACKINTEL_ANALYSIS_MODEL, then Sol; preserve explicit model and reasoning overrides.Validation
Review fixes
Commit
0a9ff10addresses the eight code/security findings:/sources. Each analysis run gets a disposable internal network with only assigned endpoints; existing source/backend connections remain intact.Current local validation: 146 passed, 11 optional skips. Docker acceptance: 8 passed, 1 large-data opt-in skip, covering network isolation, source access, resource limits and cleanup. Ruff, Vulture, Gitleaks and diff checks passed. No model-provider calls were used for these fixes.
Review boundaries
This is an experimental implementation, not a production release. Durable background scheduling and hosted delivery remain future work. Human calibration was waived by the owner; method-judge scores remain uncalibrated.
Pending local version-reset edits, expanded specialist benchmark scripts/tests, and additional sandbox/comparison documentation are excluded from this PR. The historical branch and README “v2” wording does not establish a stable product version.
Specialist documentation: docs/specialist-default.md. Related development tracking: INT-454.