Skip to content

feat: Add provider clients and multi-turn replay coverage - #96

Draft
jiahuei wants to merge 17 commits into
mainfrom
feat/provider-compatibility-evidence
Draft

jiahuei wants to merge 17 commits into
mainfrom
feat/provider-compatibility-evidence

Conversation

@jiahuei

@jiahuei jiahuei commented Nov 3, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Added explicit provider/protocol clients: Responses and Chat Completions for Bedrock, Together AI, and Fireworks; Responses for OpenRouter; Chat Completions for Gemini. Each class implements one protocol, reusing the shared implementation where sufficient. The registry selects clients without automatic protocol switching. Provider classes own destinations and credential requirements; Bedrock requires an explicit regional URL and accepts BEDROCK_API_KEY alongside the standard AWS credential name.
  • Added OpenRouter Responses handling for explicitly labeled raw reasoning and signed summaries, plus Gemini thought-signature translation and unambiguous streamed tool-index repair. Gemini preserves each Google signature in ordered IR thinking parts, projects it as Responses encrypted_content or Messages signature, and restores its text/tool attachment on continuation. Unsupported reasoning or opaque fields fail explicitly. Shared Chat streaming closes tool parts before subsequent content and rejects resumed noncontiguous calls.
  • Extended tests/upstream/test_multi_turn.py through its CLIENTS inventory, covering independent SDK parity, actual-output history, normalized continuation, and public translation or rejection. Added 87 conventionally named provider cassettes under upstream_ir/. Forty Chat Completions exchanges cover two-turn text and three-turn tools in both native modes for Bedrock Luna, Together Qwen3.5-9B, Fireworks Nemotron, and Gemini 3.1 Flash-Lite. Responses coverage includes those providers and OpenRouter's free Nemotron model. Captures retain their original bytes; tests use documented input-copy adjustments where SDK output contains fields outside request schemas.
  • Fixed recording-proxy staging, retry policy, credential masking, non-JSON error-byte fidelity, and partial-stream evidence. Removed duplicate conversation suites, the PR WIP: Add more providers #76 reference corpus, manifests, source snapshots, and standalone recording drivers. Updated architecture, conventions, and the provider support matrix.

Related to #78. This remains a draft: recordings establish selected models and scenarios, not complete catalogs. Gemini native, normalized, and Responses/Messages paths preserve the recorded text/tool thought signatures. Parallel-call signature association also has synthetic regression coverage. Other models, images, and some parallel/reasoning conversations remain unverified. Fireworks unsigned Responses summaries reject translation; OpenRouter raw output projects to Messages, but unsigned Messages reasoning input still rejects. Bedrock coverage uses a global inference profile through the Singapore runtime endpoint, which does not establish the actual inference region.

Verification

  • Merged main at 70ad86c without conflicts. A clean core-only environment passed import molto with PyYAML absent, matching the new CI import check.

  • .venv/bin/python -m pytest -q -o addopts='' --tb=short: 2,633 passed, Python 3.14.7, offline replay only.

  • .venv/bin/python -m pytest tests/upstream/test_multi_turn.py -q -o addopts='' --tb=short: 89 passed, including four Gemini native conversations, two normalized signature continuations, and eight Responses/Messages conversations across streaming and non-streaming output.

  • uv run --extra lint ruff check --output-format github --config pyproject.toml .: passed.

  • uv run --extra lint ruff format --diff --config pyproject.toml .: passed.

  • npx prettier@3.8 --check .: passed.

  • .venv/bin/python -m pytest tests/test_providers.py tests/upstream/test_openai.py tests/test_hosted_providers.py tests/core/test_provider_history_routes.py tests/upstream/test_multi_turn.py -q -o addopts='' --tb=short: 480 passed. After strengthening the synthetic streaming helper, the 30-test provider-history suite also passed.

  • git diff --check: passed. All 40 Chat Completions cassettes are byte-identical to their reviewed sources, with complete responses and no configured credentials. Existing cassette content was unchanged. Recording used native SDKs outside pytest, zero retries, and bounded request deadlines.

  • Bedrock Chat Completions recording: ten successful exchanges with global.openai.gpt-5.6-luna, reasoning disabled. Ministral 3B attempts returned runtime HTTP 400 and a 75-second Mantle timeout; unsuccessful captures remain outside the committed corpus and recorder processes exited.

  • Together Chat Completions recording: ten successful exchanges with Qwen/Qwen3.5-9B, reasoning disabled. Caller history restores the single assistant role after SDK accumulation of repeated role deltas. Fireworks: ten successful exchanges with accounts/fireworks/models/nemotron-lightning-3p5-30b-a3b, reasoning disabled and null output-only history fields omitted.

  • Retained OpenRouter recordings use an explicit free model, disabled fallbacks, zero price caps, and no plugins.

  • Gemini recording: run 42160882628 completed ten exchanges using the existing Actions secret, native OpenAI SDK calls, minimal reasoning, zero retries, and bounded request deadlines. Artifact YAML was imported unchanged and passed credential/completion checks. Caller history restores repeated assistant roles and omits top-level null fields while preserving nested Google state. Independent SDK references repair only the recorded single-call missing index; actual Molto output is not repaired by tests. Temporary recording tooling remains on a separate branch; no credential upload or secret changes were made.

  • gh pr checks 96 --repo EmbeddedLLM/molto --watch --interval 15: all five checks passed at 56c7a22: tests on Python 3.12, 3.13, and 3.14, Ruff, and Prettier.

@jiahuei jiahuei changed the title feat: Add provider compatibility and recording evidence feat: Add provider-specific protocol clients and compatibility evidence Nov 5, 2026
@jiahuei jiahuei changed the title feat: Add provider-specific protocol clients and compatibility evidence feat: Add provider clients and multi-turn replay coverage Nov 5, 2026
@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Coverage

Coverage Report (Python 3.14) •
FileStmtsMissCoverMissing
configs.py41490%45, 49, 57, 61
core.py146398%119, 132, 186
providers.py120100% 
adapters
   anthropic.py4614690%147, 157, 195, 223, 260, 311, 316, 320, 327, 331, 335, 341, 347, 351–353, 356–358, 361, 369, 371, 376, 382, 389, 413–414, 521, 583, 609, 633, 635, 637, 699, 706, 729, 755, 761, 828, 848, 873, 877, 882, 888, 896, 950
   openai.py6566490%112, 192–194, 196, 205, 362, 367, 379, 420, 422, 459, 468, 483, 494, 519, 534, 559, 591, 595, 732, 743, 747, 778, 840, 843, 848, 909, 923, 959–962, 971, 975, 982, 986, 1042, 1117, 1187, 1227, 1263, 1269, 1280, 1290, 1304, 1313, 1325, 1398, 1400, 1458, 1460, 1469, 1475, 1528, 1535, 1541, 1548, 1552, 1566, 1577, 1584, 1590, 1594
types
   openai_chat_completions.py1190100% 
   openai_responses.py1450100% 
upstream
   base.py61493%113, 118, 132, 137
   bedrock.py70100% 
   fireworks.py70100% 
   gemini.py81298%67, 74
   openai.py8093496%156, 158, 191, 203, 213, 225, 289–290, 299, 439, 468, 626, 634, 636, 639, 673–674, 860, 950, 1029, 1050, 1218, 1250, 1286, 1288, 1292–1293, 1477, 1523, 1527, 1540–1541, 1558, 1560
   openrouter.py190100% 
   registry.py110100% 
   together.py70100% 
   vllm.py50100% 
utils
   test.py36416355%196, 245–248, 258–263, 273–277, 308, 313, 354–356, 366–369, 379–380, 411, 440–443, 459, 469–470, 472–474, 485–486, 495–499, 508–510, 517, 527, 570, 579–581, 583–586, 594–595, 606–607, 609–613, 615–617, 630–632, 651, 660, 691–692, 707–709, 715, 730–731, 738–740, 742, 750–751, 753, 755–756, 777–784, 798, 803, 811, 819–826, 828–829, 836, 838, 845, 847, 852–853, 900, 905, 909–911, 917–919, 925–926, 928–931, 933–934, 940–941, 945–948, 953–956, 961, 963–967, 986–987, 989–994, 1022, 1043–1045
TOTAL414547489% 

Tests Skipped Failures Errors Time
2633 0 💤 0 ❌ 0 🔥 31.005s ⏱️

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Coverage

Coverage Report (Python 3.12) •
FileStmtsMissCoverMissing
configs.py41490%45, 49, 57, 61
core.py146398%119, 132, 186
providers.py130100% 
adapters
   anthropic.py4614690%147, 157, 195, 223, 260, 311, 316, 320, 327, 331, 335, 341, 347, 351–353, 356–358, 361, 369, 371, 376, 382, 389, 413–414, 521, 583, 609, 633, 635, 637, 699, 706, 729, 755, 761, 828, 848, 873, 877, 882, 888, 896, 950
   openai.py6566490%112, 192–194, 196, 205, 362, 367, 379, 420, 422, 459, 468, 483, 494, 519, 534, 559, 591, 595, 732, 743, 747, 778, 840, 843, 848, 909, 923, 959–962, 971, 975, 982, 986, 1042, 1117, 1187, 1227, 1263, 1269, 1280, 1290, 1304, 1313, 1325, 1398, 1400, 1458, 1460, 1469, 1475, 1528, 1535, 1541, 1548, 1552, 1566, 1577, 1584, 1590, 1594
types
   openai_chat_completions.py1190100% 
   openai_responses.py1450100% 
upstream
   base.py62494%113, 118, 132, 137
   bedrock.py70100% 
   fireworks.py70100% 
   gemini.py81298%67, 74
   openai.py8093496%156, 158, 191, 203, 213, 225, 289–290, 299, 439, 468, 626, 634, 636, 639, 673–674, 860, 950, 1029, 1050, 1218, 1250, 1286, 1288, 1292–1293, 1477, 1523, 1527, 1540–1541, 1558, 1560
   openrouter.py190100% 
   registry.py110100% 
   together.py70100% 
   vllm.py50100% 
utils
   test.py36416355%196, 245–248, 258–263, 273–277, 308, 313, 354–356, 366–369, 379–380, 411, 440–443, 459, 469–470, 472–474, 485–486, 495–499, 508–510, 517, 527, 570, 579–581, 583–586, 594–595, 606–607, 609–613, 615–617, 630–632, 651, 660, 691–692, 707–709, 715, 730–731, 738–740, 742, 750–751, 753, 755–756, 777–784, 798, 803, 811, 819–826, 828–829, 836, 838, 845, 847, 852–853, 900, 905, 909–911, 917–919, 925–926, 928–931, 933–934, 940–941, 945–948, 953–956, 961, 963–967, 986–987, 989–994, 1022, 1043–1045
TOTAL419047489% 

Tests Skipped Failures Errors Time
2633 0 💤 0 ❌ 0 🔥 59.703s ⏱️

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Coverage

Coverage Report (Python 3.13) •
FileStmtsMissCoverMissing
configs.py41490%45, 49, 57, 61
core.py146398%119, 132, 186
providers.py130100% 
adapters
   anthropic.py4614690%147, 157, 195, 223, 260, 311, 316, 320, 327, 331, 335, 341, 347, 351–353, 356–358, 361, 369, 371, 376, 382, 389, 413–414, 521, 583, 609, 633, 635, 637, 699, 706, 729, 755, 761, 828, 848, 873, 877, 882, 888, 896, 950
   openai.py6566490%112, 192–194, 196, 205, 362, 367, 379, 420, 422, 459, 468, 483, 494, 519, 534, 559, 591, 595, 732, 743, 747, 778, 840, 843, 848, 909, 923, 959–962, 971, 975, 982, 986, 1042, 1117, 1187, 1227, 1263, 1269, 1280, 1290, 1304, 1313, 1325, 1398, 1400, 1458, 1460, 1469, 1475, 1528, 1535, 1541, 1548, 1552, 1566, 1577, 1584, 1590, 1594
types
   openai_chat_completions.py1190100% 
   openai_responses.py1450100% 
upstream
   base.py62494%113, 118, 132, 137
   bedrock.py70100% 
   fireworks.py70100% 
   gemini.py81298%67, 74
   openai.py8093496%156, 158, 191, 203, 213, 225, 289–290, 299, 439, 468, 626, 634, 636, 639, 673–674, 860, 950, 1029, 1050, 1218, 1250, 1286, 1288, 1292–1293, 1477, 1523, 1527, 1540–1541, 1558, 1560
   openrouter.py190100% 
   registry.py110100% 
   together.py70100% 
   vllm.py50100% 
utils
   test.py36416355%196, 245–248, 258–263, 273–277, 308, 313, 354–356, 366–369, 379–380, 411, 440–443, 459, 469–470, 472–474, 485–486, 495–499, 508–510, 517, 527, 570, 579–581, 583–586, 594–595, 606–607, 609–613, 615–617, 630–632, 651, 660, 691–692, 707–709, 715, 730–731, 738–740, 742, 750–751, 753, 755–756, 777–784, 798, 803, 811, 819–826, 828–829, 836, 838, 845, 847, 852–853, 900, 905, 909–911, 917–919, 925–926, 928–931, 933–934, 940–941, 945–948, 953–956, 961, 963–967, 986–987, 989–994, 1022, 1043–1045
TOTAL419047489% 

Tests Skipped Failures Errors Time
2633 0 💤 0 ❌ 0 🔥 1m 24s ⏱️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant