feat(skills): add the talk-to-data loop (build-view, assess-gap, update-to-atlan) by shubhambatlan · Pull Request #246 · atlanhq/agent-toolkit

shubhambatlan · 2026-08-25T19:48:31Z

Summary

A self-contained talk-to-data loop: build a governed Snowflake Cortex semantic view from Atlan context, then close the eval loop by diagnosing failures and writing the proven fix back to Atlan. No LLM authoring in the build; the LLM only diagnoses.

Three skills, each self-contained with its script co-located:

build-view — build the model from Atlan context via the governed /semantic-model/build endpoint (build_model.py). No LLM authoring.
assess-gap — diagnose why the view fails its evals against a fixed gap taxonomy, reading only the safe eval projection (never the golden answers); one grounded, typed fix per gap.
update-to-atlan — persist an approved, typed fix (description / glossary_term / filter / relationship / popular_query) via atlan_writeback.py; approval-gated, read-back verified.

What ships

skills/build-view/{SKILL.md, build_model.py}
skills/assess-gap/SKILL.md
skills/update-to-atlan/{SKILL.md, atlan_writeback.py}

Config: $ATLAN_BUILD_ENDPOINT (your tenant's /semantic-model/build) and $ATLAN_API_KEY. Skills reference no MCP tools, so the skill-tool validator passes.

Note

build-view overlaps build-semantic-view (#241) on the build step. Happy to reconcile to one build skill; the net-new here is the eval to fix to write-back loop on top of the build.

Claude Code only for now (skills are auto-discovered from skills/); the Cursor and Codex plugins remain MCP-only.

Generated with Claude Code

…te-to-atlan) Build a governed Snowflake Cortex semantic view from Atlan context, then close the eval loop: assess-gap diagnoses failures (reading only the safe projection, never goldens) and update-to-atlan persists the approved typed fix back to Atlan via atlan_writeback.py. Each skill is self-contained with its script co-located. Generated with Claude Code

shivanshpahwa24

Set this up locally against eops1 and ran the build end to end three times (14, 12 and 21 tables). The build path works, and because build_model.py reaches the same build_only.build_semantic_model the repository flow uses, it is consistent with wisdom by construction. That part of the design does what it says.

Three things bit me that I think would bite anyone working from the docs alone. All three are in build-view and all three are small. Details inline.

Scope note on what I did not cover: assess-gap I could not exercise, since it needs a safe trace projection and that only exists after a deploy plus eval run. update-to-atlan authenticates against eops1 and resolves all three SqlInsight* typedefs, but I stopped short of a production write.

One more I am less sure about, so leaving it out of the inline set: on the 21-table build, filters fell from 42 to 27 on tables that were already present when I added 9 more, and dropped reported no filter drops. That may be wisdom-side rather than anything here.

…in-set, surface dropped reasons From Shivansh's review (eops1, 3 builds): - tables[] docs: state it's the Atlan qualifiedName WITH connection prefix (not DB/SCHEMA/TABLE, which 404s), with a concrete example; fix the build_model.py docstring example (was A,B,C). - add a "close the join set" note: relationships render only when both join ends are in tables[]; density/importance selection drops the referenced dimension tables and silently yields a joinless model. - build_model.py: dropped was reduced to a length. Print a per-section count and write the full structured list to <out>.dropped.json so the caller has the reasons (static-SQL placeholders, dup synonyms, non-unique keys) to act on at the handoff. Generated with Claude Code

shubhambatlan · 2026-08-26T15:18:24Z

Thanks for the eops1 pass, all three inline items are fixed in 86a0dc4 (notes on each thread).

Scope: assess-gap and update-to-atlan we've exercised separately end to end, assess-gap on a safe projection from a real deploy+eval run, and update-to-atlan create then read-back ACTIVE for filter / description / glossary_term (with revert). Happy to share those traces.

On the 42 to 27 filter drop when you added 9 tables: I suspect wisdom-side too. The new <out>.dropped.json should make it diagnosable next run, if there are no filter-drop entries there, the loss is upstream of this script.

… dataset names) The qualifiedName example used real internal/demo dataset names (GTM_OPERATIONS_PROD/MARTS/DIM_ACCOUNTS) in a public repo. Replace with fully-neutral placeholders <DATABASE>/<SCHEMA>/<TABLE> in both the SKILL.md bullet and the build_model.py docstring; the teaching point (connection prefix vs short form) is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

ankitjaggi · 2026-08-31T05:30:29Z

@shubhambatlan @shivanshpahwa24 These will be individual skills that will be applicable for all users across all harnesses? Will the underlying tools be enabled for all users on Day 1 or will it be a controlled release?

shivanshpahwa24

Re-checked this against wisdom beta at a660b6db5, where the build route now lives. The contract lines up: right path, right request shape (tables / engine / name), right auth. All three comments from last round landed too, thanks -- the qualifiedName format with the 404 warning, the close-the-join-set note, and the per-section dropped breakdown with the full list written next to --out.

Five things left, all on the response side. The first one is the only one I'd call blocking; the rest are small.

Context that is not a change request for this PR: beta has the route, but eops1 has not picked up the image yet -- a probe there still returns Path was not found. So this cannot be exercised against that tenant until the deploy catches up, regardless of what merges here.

shivanshpahwa24 · 2026-09-01T12:05:41Z

+    except Exception as e:
+        sys.exit(f"ERROR: build call failed: {type(e).__name__}: {e}")
+
+    if not resp.get("success") or not resp.get("content"):


Partial builds get discarded here, and I think this is the one worth fixing before merge.

Beta returns success: false with content populated whenever any single table fails, plus tables_failed[] naming which table and why. build_only.py is explicit that this is deliberate:

success is False when ANY requested table failed, even if others rendered. [...] the tables that DID build are still returned - a caller debugging one bad table should not have to re-run the other sixteen.

This line throws all of that away. Concretely: a 17-table build where one table fails costs the user five minutes, reports build did not return a model (which is not true, a model came back), names no table, and discards a perfectly usable 16-table model.

Suggest: when success is false but content is present, write the model, print each tables_failed entry with its reason, and exit non-zero so it still reads as a partial rather than a clean success.

The same change covers warnings[], which the response also carries and the script never reads (grep for tables_failed|warnings in this file returns 0).

shivanshpahwa24 · 2026-09-01T12:05:41Z

+    try:
+        with urllib.request.urlopen(req, context=ctx, timeout=a.timeout) as r:
+            resp = json.loads(r.read().decode())
+    except urllib.error.HTTPError as e:


Beta caps tables at max_length=50 in the request schema, so anything larger fails Pydantic validation before the build starts.

This handler prints HTTP 422 (body withheld), so a caller who passes 60 tables gets an unexplained failure with no way to learn the limit exists. Nothing in the skill mentions it either.

Suggest special-casing 422 to name the cap, and adding it to Phase 1 next to the qualifiedName format.

shivanshpahwa24 · 2026-09-01T12:05:41Z

+    p.add_argument("--out", required=True, help="path to write the returned model YAML")
+    p.add_argument("--endpoint", default=None)
+    p.add_argument(
+        "--timeout", type=int, default=400, help="seconds; real build takes 4-5 min"


The 400s default is under the real range once the table list grows.

Measured on eops1: 4m44s for 5 tables, roughly 9 min for 14. So past about 10 tables the script times out on its own default, and the server permits up to 50.

The help text here and the "real builds take 4-5 min" line in the skill are both accurate only for small sets. Worth either raising the default or scaling the guidance to table count.

shivanshpahwa24 · 2026-09-01T12:05:41Z

+It POSTs `{tables, engine, name}` to the governed `/semantic-model/build`
+(store-nothing) endpoint and writes the returned model YAML to `--out`. Endpoint
+resolution: `--endpoint` › `$ATLAN_BUILD_ENDPOINT` (override, e.g. the local mock)
+› the script's baked-in hosted default. Auth is automatic: it sends


This still promises "the script's baked-in hosted default" as the third tier of the fallback, but build_model.py line 34 is DEFAULT_ENDPOINT = "".

So there is no third tier: $ATLAN_BUILD_ENDPOINT (or --endpoint) is mandatory, not the override this paragraph calls it. Worth rewording so the required setting reads as required.

shivanshpahwa24

One more against beta a660b6db5, on the engine list.

shivanshpahwa24 · 2026-09-01T12:21:31Z

+        help="comma list or @file.json (list or {tables:[...]})",
+    )
+    p.add_argument(
+        "--engine", default="cortex", choices=["cortex", "genie", "databricks"]


dbt is missing here.

It is a first-class engine on the build endpoint: mapped in ENGINE_ALIASES, with its own render branch in build_only, and it validates offline against dbt-semantic-interfaces -- the same suite dbt-core and MetricFlow run -- reporting an errors list of its own. So it is a real deployable output, not a stub.

Right now there is no way to ask this script for a dbt semantic model. Suggest adding it to choices.

While that line is being touched: the skill says to ask the user which engine, but nothing tells them what distinguishes the options. Adding dbt takes that from three unexplained choices to four, so a short gloss on each would be worth the two lines.

… engines Five threads from the 2026-09-01 review. Partial builds were discarded (#246 thread on build_model.py:103). The gate was `success or content`, but only the endpoint's final return path populates `content`, so `content` is the model's existence flag and `success` is its completeness flag. Gate on `content`. `success` is `not failed and not rejected`, so a false `success` with content means one of two different things, and only one of them is a partial build: - PARTIAL — `tables_failed` non-empty; the model covers the rest and deploys. - REJECTED — `validation.status == "invalid"`; `tables_failed` can be EMPTY. The model is whole and will NOT deploy. Both now write the model to disk and exit non-zero, reported separately, so a 17-table build losing one table no longer discards a usable 16-table model. Also reads `tables_failed`, `warnings` and `message`, none of which the script had ever read. Table cap: the endpoint caps `tables` at 50 (Pydantic `max_length`), so 51+ is refused before any build starts. A bare "HTTP 422 (body withheld)" gave no way to learn the limit exists; 422 now names the cap and the count that was sent. Timeout: 400s was under the real range past ~10 tables (measured 4m44s for 5, ~9 min for 14, and the endpoint allows 50). Default raised to 1200s and the guidance scaled to table count rather than a flat "4-5 min". Engines: `dbt` is first-class on the endpoint (own render branch, offline validation against dbt-semantic-interfaces) and was unreachable. Added, along with `atlan` — the endpoint's own default engine, also unreachable. `databricks` and `genie` are NOT the same output and the skill now says so, with a one-line gloss per engine. Endpoint resolution: the skill promised a "baked-in hosted default" third tier that does not exist (`DEFAULT_ENDPOINT = ""`), so `$ATLAN_BUILD_ENDPOINT` or `--endpoint` now reads as required rather than an override. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

… engine The --timeout help text carried build times measured on an internal tenant (~5 min for 5 tables, ~9 min for 14). Those are not a customer's numbers and they read as a promise, so the help and the skill now say only that build time grows with the table count. The 1200s default is unchanged. Dropped `atlan` from --engine: it returns Atlan's own canonical model, which is not a deployable target for this skill's callers. `databricks` and `genie` are kept separate, because the endpoint renders them separately: genie -> databricks/genie_config.json via execute_build_genie_config_artifact, databricks -> databricks/metric_view.yaml via execute_build_databricks_metric_view_yaml. Different file, format and render path, and genie emits no validation verdict where databricks does. The Genie config only references a metric view by name, so a Genie space needs both artifacts and collapsing the names would make the metric view unreachable. The skill now explains that relationship instead of listing them as rivals. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…he 422, not the help text Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…d ordering The previous wording implied the Genie render references a metric view, which reads as "build databricks first". It does not. Genie's config is stamped `_partial: true` unconditionally — a literal in the builder — meaning its deploy-time fields (space title, table identifiers, warehouse, metric_view_fqn) are for the DEPLOY to fill. The render was verified byte-identical with sections in memory, sections from storage, and no sections at all, so nothing about a prior databricks build changes its output. Two things a caller needs instead: a near-empty genie config is normal (it holds only cross-table extras and defers table/column identity to deploy), and the `_partial` warning is a note, not an error. Both are now stated. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…internal dataset name Two things the SKILL.md pass fixed but the module docstring still carried. Thread 7 (the phantom third tier) was corrected in SKILL.md only. The docstring still listed `DEFAULT_ENDPOINT constant below - the hosted governed endpoint (if baked)` as a resolution tier and called $ATLAN_BUILD_ENDPOINT an override. DEFAULT_ENDPOINT is empty, so one of --endpoint / $ATLAN_BUILD_ENDPOINT is required; the docstring now says that. Both usage examples passed `--name gtm`, naming an internal dataset in a customer-facing example. 0ac62fa neutralised the tables[] example but --name survived it. Now <use_case>. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

… metric view The previous wording led with "build order does not matter", which is true of the builds and misleading about everything else. A Genie space cannot stand alone. genie_config.py is explicit that `metric_view_fqn` "needs catalog/schema/view from form" — it is filled at deploy time from a metric view that ALREADY EXISTS in Databricks. So end to end: build the metric view, deploy it, build the genie config, deploy the space. If the target is Genie, both engines are required. What stays true, and is now stated as the narrower claim it is: the two BUILDS are independent, and `_partial: true` is unconditional, so it must not be read as "the metric view is missing". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The deploy dependency was documented in the engine list but nothing acted on it, so a caller asking for a Genie space got one artifact and a space that cannot deploy. Phase 2 now instructs two runs over the same tables (--engine databricks and --engine genie) whenever the target is a Genie space, and Phase 3 returns both paths with the deploy order stated. One build call returns one engine's artifact; Genie needs two. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

… miner build_relationship sorted the FORMATTED "SOURCE=JOINED" strings. The miner orders by the source column alone — sql_intelligence/sql.py:912: STRING_AGG(cp.source || '=' || cp.joined, ',' ORDER BY cp.source) and pyatlan's SqlInsightJoin.generate_qualified_name does the same. Sorting the formatted strings diverges whenever one source column is a prefix of another followed by a digit (ORDER_ID / ORDER_ID2), because '=' is 0x3D while digits are 0x30-0x39. Different sortedPairs -> different md5 -> a qualifiedName the miner would never produce, so the write DUPLICATES the miner's row instead of converging on it. That convergence is the reason the canonical identity exists. Only composite-key joins were affected; a single-pair join sorts one element and was always correct. Verified: the two composite cases above now reproduce the miner's string and their qualifiedNames changed, while an ordinary two-column join (ACCOUNT_ID, REGION) is byte-identical to before. The sort is stable, so pairs sharing a source column keep the caller's order, which is what pyatlan does. The miner's ORDER BY is unspecified for those ties. Note for whoever owns SQL Intelligence: atlan-frontend's sqlInsightIdentity.ts has this same formatted-string sort, so UI-authored composite joins can duplicate mined rows. Not fixed here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

shivanshpahwa24 · 2026-09-03T21:45:08Z

Context on where this ended up, since it feeds directly into three new PRs.

This branch became the correctness baseline. Because build_model.py and atlan_writeback.py here were tested by hand and trusted, the tool version was built to reproduce them rather than to replace them on its own terms, and then checked against them row by row. dc62d81 is pinned as the reference.

The result, same five tables through this script path and through the tools:

engine	this branch	the tools
cortex	24,740 bytes	byte-identical
databricks	19,032 bytes	byte-identical
genie	1,845 bytes	byte-identical
dbt	refuses, no dbt-materialised tables	same refusal, same message

Write-back was run against a dev tenant through both paths and the stored entities compared: same typeName, same status, same qualifiedName shape, 11 of 11 filter attributes and 6 of 6 question attributes populated identically. The join identity formula here was also checked against the adversarial case its own comment warns about, two source columns where one is a prefix of the other followed by a digit, and it holds.

Why it moved to tools. Only Claude Code can run a skill's scripts. The cursor-plugin and codex-plugin manifests in this repository carry an MCP URL and nothing else, and most people using Atlan through MCP have no plugin at all. The build is now an MCP tool (agent-toolkit-internal #560) over an async wisdom route (wisdom #2119), with thin skills over it in #248.

Five small defects found while porting were fixed rather than carried across: --engine snowflake documented but rejected by argparse; the filter payload table omitting the required operator; delete --hard --guid crashing on argument order; exit 0 on an HTTP failure; and raw tracebacks where the SKILL.md promises a clean message.

Left open rather than closed, pending the merge order.

shubhambatlan requested review from Hk669 and firecast as code owners August 25, 2026 19:48

shubhambatlan requested review from Hk669, firecast and shivanshpahwa24 August 25, 2026 19:48

shubhambatlan force-pushed the feat/ttd-loop-skills branch from 5ee4798 to 35e5b19 Compare August 25, 2026 19:53

shivanshpahwa24 reviewed Aug 26, 2026

View reviewed changes

Comment thread skills/build-view/SKILL.md Outdated

Comment thread skills/build-view/SKILL.md

Comment thread skills/build-view/build_model.py Outdated

shivanshpahwa24 reviewed Sep 1, 2026

View reviewed changes

shubhambatlan and others added 8 commits September 1, 2026 14:15

chore(build-view): correct MAX_TABLES comment — the cap is named in t…

93868ec

…he 422, not the help text Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Navigation Menu

feat(skills): add the talk-to-data loop (build-view, assess-gap, update-to-atlan) - #246

feat(skills): add the talk-to-data loop (build-view, assess-gap, update-to-atlan)#246
shubhambatlan wants to merge 11 commits into
mainfrom
feat/ttd-loop-skills

shubhambatlan commented Aug 25, 2026

Uh oh!

shivanshpahwa24 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shubhambatlan commented Aug 26, 2026

Uh oh!

ankitjaggi commented Aug 31, 2026

Uh oh!

shivanshpahwa24 left a comment

Uh oh!

shivanshpahwa24 Sep 1, 2026

Uh oh!

shivanshpahwa24 Sep 1, 2026

Uh oh!

shivanshpahwa24 Sep 1, 2026

Uh oh!

shivanshpahwa24 Sep 1, 2026

Uh oh!

shivanshpahwa24 left a comment

Uh oh!

shivanshpahwa24 Sep 1, 2026

Uh oh!

shivanshpahwa24 commented Sep 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

shubhambatlan commented Aug 25, 2026

Summary

What ships

Note

Uh oh!

shivanshpahwa24 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shubhambatlan commented Aug 26, 2026

Uh oh!

ankitjaggi commented Aug 31, 2026

Uh oh!

shivanshpahwa24 left a comment

Choose a reason for hiding this comment

Uh oh!

shivanshpahwa24 Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

shivanshpahwa24 Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

shivanshpahwa24 Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

shivanshpahwa24 Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

shivanshpahwa24 left a comment

Choose a reason for hiding this comment

Uh oh!

shivanshpahwa24 Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

shivanshpahwa24 commented Sep 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants