Land panel-sim's wire layer as ebus_sdk.wire: declarative profiles, catalog hydration, device-graph building

[dcj]

Status: proposal, with a working spike
Relates to: #46 (independent), and internal tracking SDK-6do / SDK-6do.4 / SDK-03o

Why this doc exists

An earlier internal plan argued the generalization case from one body of evidence: the SPAN Gen2 adapter's spancommon.PropertyDefinition, 135 definitions across 15 lists, and what it would take to migrate it onto PropertySpec.

This is a second, independent body of evidence for the same question, arrived at from the opposite direction. ebus-panel-sim grew a wire/ layer that solves an overlapping problem in a different way, it has been in production across four PyPI releases, and it has now been extracted and proved portable. Neither analysis knew about the other's evidence: the July plan contains no mention of panel-sim, profiles, or the wire layer.

Two independent implementations converging on the same missing SDK capability is a stronger signal than either alone.

What was extracted, and what the spike proved

Eight modules, 1089 lines, lifted from ebus-panel-sim into a scratch package:

profile_loader   catalog hydration + per-phase pattern expansion
mapping_loader   placement descriptors (root-device vs child-of-parent, topic rendering)
graph_builder    manifest + mappings + profiles -> SDK Device tree
property_bag     PropertyBag + PropertyDiffer
publisher        per-tick diff-only publish loop
set_router       setter registry, coverage checking, per-datatype /set coercion
manifest         DeviceInstance / DeviceManifest
exceptions       one root + three leaves

Two measurements, both reproducible from the spike branches:

• It stands alone. The extracted package builds a real device tree (7 profiles hydrated, 44 properties bound) with a sys.meta_path hook installed that raises on any ebus_panel_sim import. There is no hidden coupling.
• Its origin still passes. ebus-panel-sim consuming the extracted package instead of owning the code: 165 passed, unchanged, plus ruff, ruff-format and mypy --strict clean, with no test file edited. src/ went 166 insertions, 899 deletions across 8 files, which became re-export shims.

The spike is local and unpublished; the distribution-enclosure-simulator side is on an unmerged branch that points at it by path. Both are reference material, not proposed changes.

How this maps onto work the SDK has already planned

This is the part that matters, and the reason this should not land as a parallel submodule dropped in beside the existing plan.

Extracted module	Status in the SDK today
property_bag + publisher	SDK-03o already proposes this, and names these exact files. Should be fulfilled as publish-on-change, not added alongside.
graph_builder	Overlaps SDK-6do.4 (DeviceSpec + tree-aware incremental builder). Two independent solutions to multi-device materialization; reconcile rather than ship both.
manifest (DeviceInstance/DeviceManifest)	Relates to the DeviceSpec shape SDK-6do.4 proposes. Same problem, different vocabulary.
set_router	Partial overlap. The SDK owns /set subscription and payload decode; this adds a registry, per-datatype coercion, and coverage checking that fails at construction when a settable property has no handler.
profile_loader	No SDK equivalent. See below.
mapping_loader	No SDK equivalent.
exceptions	The SDK has no exception hierarchy at all today. This would introduce the first.

profile_loader is the genuinely new capability

declaration.PropertySpec is the hand-written path: a caller states capability, prop_id, datatype, unit in Python, per property. profile_loader is the data-driven path: a profile selects capability properties by name, and the vendored spec capability catalogs supply datatype, unit, $format and settability.

The difference is not convenience. A selection that carries no datatype cannot drift from the specification by hand-copying one. That is the same class of guarantee .ebus-spec.json and the catalog-drift test provide at the repository level, pushed down to the property.

It also solves a gap the SDK has already conceded in writing. ha/customize.py:50:

# Power factor is unitless, so inference has nothing to go on. The table has
# no pattern support, so the per-phase forms are spelled out.
"power-factor":   {...},
"power-factor-a": {...},
"power-factor-b": {...},
"power-factor-c": {...},

profile_loader._expand_pattern reads property_patterns straight from the catalogs (voltage-{a,b,c}, current-{a,b,c,n}, and eight more in meter.json alone) and expands them. Demonstrated in the spike: a profile selecting voltage-a by name alone had datatype=float, unit=V hydrated from the catalog with nothing hand-written.

This capability is also the defining shape of the next simulator. A utility meter's whole model is per-phase suffixes; see utility-meter-simulator.

Proposed shape

ebus_sdk.wire, delivered as the extra ebus-sdk[wire].

This is not a new pattern for the SDK, it is the established one. ha/ is already a 1751-line non-normative integration submodule (larger than this at 1089), reachable only by explicit import and absent from ebus_sdk/__init__.py. mdns and validation are already optional extras, the latter documented as optional precisely "so a constrained build (e.g. Yocto) can omit it without a hard dependency".

Requirements that follow from those precedents:

1. Not imported from ebus_sdk/__init__.py. import ebus_sdk must stay free of it, matching ha/. Consumers write from ebus_sdk.wire import build_graph.
2. pyyaml guarded at its use site in mapping_loader, the way homie.py:71 guards jsonschema. Unlike jsonschema, it should raise with an install hint rather than degrade: without YAML there is no mapping table at all, so silent degradation would surface far downstream as "exactly one root-device descriptor", which is what an empty table looks like. That failure was observed during the spike.
3. wire = ["pyyaml>=6.0"] in optional-dependencies, with the same rationale comment the other two carry.
4. No module-relative default data directories. load_profiles and load_mapping_table must require their directories. Inside the SDK a module-relative default resolves into the SDK's own install, so a consumer that omitted it would load nothing. Each consumer vendors its own catalogs, which is what keeps their spec-reconciliation cadences independent.
5. No vendor name in a type signature. overlay: str | None, not Literal["span", "reference"]. A shared layer must not export one panel vendor's name to every consumer that imports it.

The SPAN panel is unaffected

It installs plain ebus-sdk, never imports ebus_sdk.wire, and gains no dependency. The code ships in the wheel the way ha/ already does.

Three things the spike found that analysis had asserted wrongly

Worth recording, because each was believed the other way round before the code was actually moved:

• make_property is not a no-op wrapper over Node.add_property_from_dict. Its conditional key omission is the profile-to-SDK adapter: an absent unit must be missing from the spec dict rather than present as None, or the device declares an attribute it does not have. Inlined into graph_builder rather than dropped.
• MissingSetterError carries a structured missing payload (the offending (entity_class, property_path) pairs) that a naive rewrite silently dropped. mypy --strict caught it.
• mapping_loader had the same module-relative-default hazard as profile_loader, and it bit immediately when panel-sim was wired up: 47 failures reading "mapping table must have exactly one root-device descriptor". This is requirement 4 above, learned rather than reasoned.

Open questions

1. Reconcile or replace? graph_builder and SDK-6do.4's DeviceSpec builder solve the same problem from different directions: one walks a declarative manifest, the other materialises an externally-owned model incrementally with late-bound ids. The SPAN adapter needs the latter's incremental/late-binding behaviour, which graph_builder does not have. Likely answer: SDK-6do.4's builder is the general one, and graph_builder becomes a thin declarative front end onto it. That needs a real design pass, not a guess.
2. Does PropertyDiffer survive SDK-03o? If publish-on-change lands on Property.set_value, the differ largely dissolves and publisher shrinks to a loop. Doing SDK-03o first may delete work rather than move it.
3. Exception hierarchy. The SDK has none. Options: scope WireError to the submodule (minimal, matches ha/ self-containment), or introduce an SDK-wide root now and hang wire errors off it. The second is a larger public-surface decision that should not be made incidentally.
4. Where do the wire tests live? They must move with the code. panel-sim's tests/wire/ plus test_wire_units.py test wire behaviour; one reaches into a private _to_sdk_unit, which the spike re-exported as an expedient.
5. Does manifest belong at all, or is it subsumed by DeviceSpec? Two vocabularies for the same idea should not both ship.

Sequencing

1. SDK-03o first. Cheapest, self-contained, and may delete property_bag/publisher before they are moved.
2. Design pass on graph_builder vs SDK-6do.4. Do not land two multi-device builders. This is the real work.
3. profile_loader + mapping_loader + set_router. These have no SDK counterpart and can land as ebus_sdk.wire largely as-is, with the five requirements above.
4. panel-sim migrates, deleting 899 lines. The spike shows the diff and proves the suite stays at 165.
5. utility-meter-simulator consumes it from the start, which is the second consumer that validates the API against something other than panel-sim.

python-sdk#46 (teardown) is independent and blocks none of this: the wire layer deliberately contains no lifecycle code.

[dcj]

Correction to the proposed shape: this is not a new submodule, it is the declarative layer

The original write-up proposed ebus_sdk.wire. That was wrong, and the reason is worth recording because it changes the plan rather than just the spelling.

The SDK already has a name for this concept. declaration.py's own docstring:

It is the declarative "schema" layer of the proxy pattern.

and the tracking epic behind #6do-family work is titled "Generalize declarative/build layer". So:

	panel-sim calls it	the SDK calls it
declare properties in data	wire/profile_loader	declaration.PropertySpec
materialise a Device from them	wire/graph_builder	build_from_declarations
the layer as a whole	"the wire layer"	"the declarative layer"

Two vocabularies for one idea. Landing ebus_sdk.wire beside ebus_sdk.declaration would put schema-declaration next to schema-declaration under two names, which is the confusion this issue was opened to remove, merely relocated inside the same package.

What "wire" actually means, and why it does not travel

For the record, since it is a good name where it lives. "Wire" is standard protocol jargon (the serialized representation, as against the in-memory one), and panel-sim uses it rigorously — _circuit_wire_active_power and test_circuit_active_power_wire_sign_is_inverse_of_internal_model both mean the published value's reference frame rather than the internal model. Inside panel-sim it earns its keep by contrasting wire/ with the domain model beside it (snapshot.py, relay_resolver.py, panel_meter.py).

Extraction removes that contrast. There is no domain model next to a shared package, so the word stops distinguishing anything. It also undersells the contents: only publisher.py is genuinely wire-shaped.

module	what it actually is
profile_loader	schema — which properties exist, their datatypes and units
mapping_loader	topology — root vs child, id and topic rendering
graph_builder	topology — builds the device tree
set_router	inbound routing
property_bag / publisher	publishing — the only wire-shaped part

Schema plus topology is most of it, and "wire" names neither.

Consequence for the plan

The reconciliation with the tree-aware builder moves from open question to prerequisite. The earlier write-up listed "reconcile or replace?" as something to decide later and treated the submodule as landable meanwhile. It is not: shipping the code under a second name is the thing that would have to be undone.

Revised shape:

• profile_loader extends the declarative layer rather than sitting beside it. PropertySpec is the hand-written declaration; catalog hydration is the data-driven one. Two front ends onto one builder, not two builders.
• graph_builder reconciles with the tree-aware builder before anything lands, not after.
• The code lands in declaration.py or a declarative/ package alongside it, and "wire" retires with panel-sim's internals, where it remains correct and should not be renamed. panel-sim 0.4.0 is released; churning its internals for vocabulary would be cost without benefit.
• The ebus-sdk[wire] extra becomes ebus-sdk[...] named for whatever the layer ends up called. The pyyaml-behind-an-extra mechanism is unaffected, and so is the Yocto argument.

Everything else in the original write-up stands: the extraction is proved portable, panel-sim passes 185 unchanged against it (now including 0.4.0's bring-your-own-transport tests), and the five implementation requirements still hold.

[dcj]

Step 1 of the sequencing is done: publish-on-change shipped in 0.20.0

This issue's plan opens with "publish-on-change first. Cheapest, self-contained, and may delete property_bag/publisher before they are moved." That landed in #51 and is released.

One detail changes what it means for the extraction. The gate went into Property.publish_value(), not into set_value() as an opt-in only_if_changed=. So it applies to every producer with no caller change at all, rather than to producers that opt in.

Concretely, for wire/:

• PropertyDiffer is now redundant as a dedup mechanism. Anything it suppresses, the SDK suppresses one layer down, and it suppresses strictly more: the comparison is on the final wire payload, after rounding, datatype coercion and empty-string encoding, so it also collapses two distinct model values that serialize identically. A round_to property fed 0.14494210481643677 then 0.14501120000000001 publishes once; a differ comparing model values cannot see that.
• publisher should shrink to a tick loop that just calls set_value on everything.
• So this is code to delete during the extraction, not move. Open question 2 ("does PropertyDiffer survive?") resolves to: not for dedup. If it carries anything else (ordering, batching, a queryable snapshot), that part should be extracted deliberately and named for what it actually does.

Two carve-outs worth knowing before you lean on the SDK gate:

1. Non-retained (event) properties are never gated. The broker stores nothing for them, so an identical consecutive payload is a second real event. If any wire-layer property is retained: false, it still publishes every tick, by design.
2. Every whole-tree republish forces past the gate (refresh_tree(), and the reconnect that calls it), so a broker restarted with an empty retained store is still fully repopulated.

Property.get_last_published_value() now returns the memoized wire payload if the layer wants to introspect what is actually on the broker, and Property.invalidate_publish_cache() exists for anything that deletes a retained value topic behind the SDK's back.

Nothing else in the plan changes. The graph_builder reconciliation is still the prerequisite, and still the real work.