Tell the judge which adaptors exist - #672
Open
elias-ba wants to merge 1 commit into
Open
Conversation
2 tasks
The workflow expert was asked to flag an adaptor that "doesn't correspond to a real system", and told two rules below that it has no reliable knowledge of adaptors. It followed the first using the memory the second warns about, and failed a spec because @openfn/language-redis looked made up. It is real, and the service offers it to the model in the prompt. The judge now gets that same list, the way build_adaptor_docs already grounds claims about function signatures, and only for workflow_chat specs, since the block tells the judge the workflow service offered it. validate_adaptors skips when the list comes back empty, which means the lookup failed rather than that no adaptor exists. Without that, a container that cannot reach the adaptor list calls every adaptor in every workflow invented. It also stripped the version before the prefix, so only a fully qualified name with a version ever matched.
elias-ba
force-pushed
the
fix/judge-adaptor-names
branch
from
September 4, 2026 23:40
74a7794 to
18e6e39
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Short Description
Gives the acceptance-test judge the real list of adaptor packages, so it stops flagging genuine adaptors as invented.
Fixes #671
Implementation Details
The workflow expert failed a spec because it decided
@openfn/language-rediswas made up. It is published, and the service offers it to the model in the prompt, so the model chose an adaptor it was given.The judge contradicted itself: one rule asked it to flag an adaptor that "doesn't correspond to a real system", another two rules below said it has no reliable knowledge of adaptors and must not claim one does not exist.
build_adaptor_namesfollowsbuild_adaptor_docs, which already grounds claims about function signatures: same fail-open contract, same size cap. It only goes toworkflow_chatspecs, since the block tells the judge that service was offered the list.validate_adaptorshad two faults underneath. It treated an empty list as "no adaptor exists" rather than "the lookup failed", so a container that can't reach the adaptor list would call every adaptor in every workflow invented. And it stripped the version before the prefix, so only a fully qualified name with a version ever matched; the other three shapes reported as invented. Both would have turned into Sentry noise now that it reports rather than only logs.poetry run pytest services/*/tests/unit— 129 passed, eight of them new here, and each fails when the thing it pins is removed.services/testing/testswas missing fromtestpaths, though CI'sservices/*/tests/unitglob would have picked it up. It's there now so a barepytestcovers it too.AI Usage
Please disclose whether you've used AI in this work (it's cool, we just want to know!):
You can read more details in our Responsible AI Policy