Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions evaluations/conversation_logs/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# Conversation Logs

This directory preserves the Codex conversation around the data harmonization benchmark work so reviewers can inspect the user's instructions and Codex responses.

- `rollout-2026-06-30T13-49-41-019f1a4b-79ea-7070-bee5-609ceccdba08.jsonl`: complete raw Codex session JSONL copied from `/h/jmc/.codex/sessions/2026/06/30/` at artifact creation time. It includes tool calls and tool outputs as recorded by Codex.
- `conversation_019f1a4b-79ea-7070-bee5-609ceccdba08.md`: readable Markdown extraction of user and assistant messages from the raw JSONL.

The raw transcript was scanned for obvious credential patterns before check-in. Token-shaped values were redacted in the committed copy. It includes Codex system/developer context because that is part of the raw session record.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

19 changes: 19 additions & 0 deletions evaluations/post_report_results/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Post-Report Completed Results

Created: 2026-08-11T17:43:41+00:00

This directory captures benchmark-related outputs that were found after the overall evaluation report branch had been checked in.

## Contents

- `parallel_batch_20260701/`: local `.runs/fold-*` result artifacts for the parallel leave-one-out batch described in GitHub issue #42. These include manifests, instructions, action logs, generated harmonizer code, mapping JSON, curator bundles, and notes.
- `parallel_batch_20260701/large_generated_csv_manifest.csv`: checksums, sizes, and line counts for generated harmonized CSVs that were too large to commit directly to GitHub.
- `github_actions/run_28483156122_eval_holdout_7/`: downloaded artifact from the successful `Run Harmonization Eval` GitHub Actions run for holdout 7.

## Trace Caveat

The `.runs` folders do not contain complete raw sub-agent tool-call transcripts. They contain `AGENT_ACTION_LOG.md` files and final outputs. Future benchmark harness runs should persist the complete raw sub-agent JSONL/tool-call transcript for each sub-agent, not only action logs and output files.

## Large CSV Policy

Generated harmonized CSVs were not committed because several are too large for normal GitHub storage, including files over 100 MB. They are represented by size, line count, and SHA-256 digest in `parallel_batch_20260701/large_generated_csv_manifest.csv`.
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# Run environment — leave-one-cluster-out

Harmonize the held-out dataset(s) below using ONLY:
- the skills in `skills/`,
- the exemplars in `data/processed/.../sm_data_harmonization_mapping.json` and the code patterns in `data/gold/expert_code/harmonize_sm/` (both have the held-out cluster removed),
- the shared raw inputs under `~/ess-dive_wfsfa_soil_datasets/` and the cached metadata under `data/external/ess-dive_meta/`.

Do NOT look up the held-out dataset's existing harmonized output, expert code, or mapping entry from any other location. The held-out datasets are:

- `ess-dive-38e901ec3d7bd24-20230504T211548257225`
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
{
"name": "holdout-7",
"holdout_indices": [
7
],
"holdout_identifiers": [
"ess-dive-38e901ec3d7bd24-20230504T211548257225"
],
"exemplar_indices": [
1,
2,
3,
4,
5,
6,
8,
9,
10,
15,
16,
17,
18,
23,
24,
25,
26,
27
],
"n_exemplars": 27,
"sources": {
"package": "data/gold/expert_code/harmonize_sm",
"mapping": "data/gold/sm_data_harmonization_mapping.json",
"skills": "skills"
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
import pandas as pd, numpy as np
D="/home/runner/ess-dive_wfsfa_soil_datasets/ess-dive-38e901ec3d7bd24-20230504T211548257225/"
df=pd.read_csv(D+"BM_Merged_T_VWC_0616_1018.csv")
print("VWC file shape",df.shape)
print("cols",df.columns.tolist())
print("depths(cm)",sorted(df['Depth (cm)'].dropna().unique()))
print("first/last",df['date.time'].iloc[0],"/",df['date.time'].iloc[-1])
print("vwc min/max",df['Volumetric Water Content'].min(),df['Volumetric Water Content'].max())
print(df.groupby('Depth (cm)').size())
# interval check at one depth
g=df[df['Depth (cm)']==10].copy()
g['dt']=pd.to_datetime(g['date.time'],format="%m/%d/%y %H:%M",errors='coerce')
print("median interval min (depth10)", g['dt'].diff().dt.total_seconds().median()/60)
print("nat count", g['dt'].isna().sum())
print("\n--- CO2/loc file ---")
co=pd.read_csv(D+"BM_EGM_Well_CO2.csv")
print("shape",co.shape,"cols",co.columns.tolist())
print("Locations",co['Location'].unique())
print("lat/lon",co.iloc[0,1],co.iloc[0,2])
Loading
Loading