fix: Add automatic citation buffering and preserve point citations

docs/protocols/anthropic-messages.md

@@ -360,7 +360,7 @@ Recognized errors use the Anthropic SDK namespace and the [shared error contract
 
 Declare `{"type":"web_search_20250305","name":"web_search"}` in `tools`. Translation to Responses supports allowed domains, approximate location, and search budgets; explicit blocked domains are rejected.
 
-With Responses upstream, Messages receives server-tool blocks. `Molto.messages(..., citation_policy="buffer")` also preserves URL citations with reduced source metadata by withholding every text part until completion, including uncited parts. The default `"drop"` streams text without citations. Both modes preserve request-history attribution; native Messages passthrough is unaffected. With Chat upstream, the policy is a no-op: text streams immediately with unchanged chunk boundaries in both modes. See [Hosted web search](../web-search.md) for mappings, placeholders, and continuation limits.
+With Responses upstream, Messages receives server-tool blocks. `Molto.messages(..., citation_policy="buffer")` also preserves URL citations with reduced source metadata by withholding every text part until completion, including uncited parts. `"auto"` buffers only when the normalized request declares hosted search or carries search or citation history; otherwise it streams text without citations. Opaque `extra_body` extensions do not select automatic buffering. The default `"drop"` always streams text without citations. All policies preserve request-history attribution; native Messages passthrough is unaffected. With Chat upstream, the policy is a no-op: text streams immediately with unchanged chunk boundaries. See [Hosted web search](../web-search.md) for mappings, placeholders, and continuation limits.
 
 When Messages is the translated upstream and the provider is Anthropic, missing, empty, or `"N/A"` encrypted search-source or citation state raises `UnsupportedInput` before transport. Preserve the original state through the Responses metadata extension or explicitly choose text-only history. Native Messages continues to forward history for provider validation; other provider profiles retain reduced-history support.
 

docs/web-search.md

@@ -10,7 +10,7 @@ Molto preserves provider-hosted web search on native routes and translates basic
 
 Translated Chat cannot preserve reasoning signatures or encrypted reasoning. The OpenAI Responses streaming search recording contains encrypted reasoning, and Anthropic Messages search turns can contain signed thinking; these turns raise `ValueError("Chat Completions cannot preserve reasoning signatures.")`. Use Responses or Messages downstream to retain that output. Search without reasoning or with unsigned raw reasoning remains representable. The `vllm` profile supports unsigned raw reasoning; provider support for hosted search is separate. Rejection depends on the returned reasoning data, not whether search was requested. Native Chat passthrough is unaffected.
 
-Messages callers receive translated URL citations only with `Molto.messages(..., citation_policy="buffer")`. The default `"drop"` streams text without citations; native Messages passthrough preserves citations in either mode.
+Messages callers receive translated URL citations with `Molto.messages(..., citation_policy="buffer")`, or with `citation_policy="auto"` when the normalized request declares hosted search or carries search or citation history. The default `"drop"` streams text without citations; native Messages passthrough preserves citations under every policy.
 
 Both streaming and non-streaming calls use this contract. Native routes preserve provider fields and events. Translated non-streaming results collect the same projected events as streaming calls.
 
@@ -45,7 +45,7 @@ Budget exhaustion remains provider-specific. A Messages `max_uses_exceeded` resu
 
 Responses URL citations identify a range within a text part. Translated Messages splits that part at citation boundaries and attaches each citation to the corresponding blocks. Concatenating those blocks preserves the answer exactly, including whitespace. Overlapping ranges can associate one source with several blocks. Messages citations map back to ranges spanning their associated text blocks, with offsets local to each projected Responses part.
 
-Messages rejects empty and whitespace-only text blocks in conversation history. Citation projection folds blank fragments into the preceding block, or into the first following block for leading whitespace. Merging can widen a citation's associated block, while its original excerpt remains unchanged. Zero-length annotations remain representable in the normalized output and attach to the block containing their offset; a zero-length annotation at the end of the answer attaches to the final block. Their fallback `cited_text` remains empty because they cover no answer characters. An entirely blank text part has no valid Messages block that can preserve its content and attribution, so projection raises `ValueError`; assistant-history requests reject it with `UnsupportedInput` before transport.
+Messages rejects empty and whitespace-only text blocks in conversation history. Citation projection folds blank fragments into the preceding block, or into the first following block for leading whitespace. Merging can widen a citation's associated block, while its original excerpt remains unchanged. Zero-length annotations do not split the text and attach to the block containing their offset; a zero-length annotation at the end of the answer attaches to the final block. Their fallback `cited_text` uses that block's text because they cover no answer characters. An original provider-supplied `cited_text` remains unchanged. An entirely blank text part has no valid Messages block that can preserve its content and attribution, so projection raises `ValueError`; assistant-history requests reject it with `UnsupportedInput` before transport.
 
 For example, this upstream Responses text part contains answer `"Alpha. Beta."` with a citation on `[7, 12)`:
 
@@ -86,11 +86,17 @@ It becomes these illustrative Messages content blocks:
 ]
 ```
 
-`cited_text` on this translated route contains the associated **answer range**, which may paraphrase sources. It is not a verified source quotation. `encrypted_index: "N/A"` means citation state is unavailable; it cannot restore provider continuation state. The Anthropic Messages-to-Responses route retains the original excerpt and encrypted reference in the [provider metadata extension](#preserving-anthropic-search-history). Native Messages preserves both directly. Missing or null titles become `""` when the translated target requires a string.
+`cited_text` on this translated route contains the associated **answer range**, or the containing block for a zero-length annotation, which may paraphrase sources. It is not a verified source quotation. `encrypted_index: "N/A"` means citation state is unavailable; it cannot restore provider continuation state. The Anthropic Messages-to-Responses route retains the original excerpt and encrypted reference in the [provider metadata extension](#preserving-anthropic-search-history). Native Messages preserves both directly. Missing or null titles become `""` when the translated target requires a string.
 
 Offsets use Python string characters without normalization. Recordings verify ASCII ranges; synthetic Unicode tests do not establish the provider's Unicode indexing convention.
 
-With Responses upstream, `citation_policy="buffer"` buffers every Messages text part until completion, including uncited text, then splits it at citation boundaries while preserving part order. This increases text latency and changes chunk boundaries. The default `"drop"` streams text and discards both citation deltas and snapshot citations. Both modes retain server-tool calls and results, whose buffering is needed to obtain complete actions and queries, and preserve input citations during request construction. The policy is a no-op with a Chat upstream: its normalization drops annotations, and both modes preserve immediate text streaming and chunk boundaries. It does not apply to Responses or Chat downstream. A future `"auto"` mode may consult only the normalized request; it is not implemented.
+With Responses upstream, `citation_policy` controls translated Messages text:
+
+- `"drop"` (default) streams text and discards both citation deltas and snapshot citations.
+- `"buffer"` waits for every text part to complete, including uncited text, then splits it at citation boundaries while preserving part order.
+- `"auto"` behaves like `"buffer"` when the normalized request declares a hosted-search tool or carries search calls, search results, or text with citations in its history. Otherwise it behaves like `"drop"`, including when unexpected citations arrive later. It does not inspect `extra_body`.
+
+Buffering increases text latency and changes chunk boundaries. All policies retain server-tool calls and results, whose buffering is needed to obtain complete actions and queries, and preserve input citations during request construction. The policy is a no-op with a Chat upstream: its normalization drops annotations, and all policies preserve immediate text streaming and chunk boundaries. It does not apply to Responses or Chat downstream.
 
 Each buffered Messages text block starts with empty text and its citations, then emits the full block text in one `text_delta` before stopping. This preserves compatibility with the Anthropic SDK's `text_stream` iterator.
 
@@ -190,7 +196,7 @@ An [independent continuation recording](../tests/cassettes/README.md#anthropic-h
 
 ## Follow-ups and recovery
 
-Keep the upstream and provider fixed across a conversation. Native routes preserve search history. The Anthropic Responses route preserves opaque search state when clients retain its metadata extension; other translated Responses/Messages history follows the reductions above. Translated Chat history retains only visible text and supported function calls. History can carry citations without redeclaring search. Citation buffering is never inferred from history or tools. A Messages caller who wants attribution selects `citation_policy="buffer"`, including when search context is supplied only through `extra_body.input` or `extra_body.tools`.
+Keep the upstream and provider fixed across a conversation. Native routes preserve search history. The Anthropic Responses route preserves opaque search state when clients retain its metadata extension; other translated Responses/Messages history follows the reductions above. Translated Chat history retains only visible text and supported function calls. History can carry citations without redeclaring search, and `citation_policy="auto"` uses that normalized history to select buffering. A Messages caller who wants attribution regardless of normalized tools or history selects `citation_policy="buffer"`, including when search context is supplied only through `extra_body.input` or `extra_body.tools`. Opaque transport extensions cannot trigger automatic buffering.
 
 When translating history to Messages with `provider="anthropic"` (the endpoint default), Molto rejects missing, empty, or `"N/A"` source `encrypted_content` and citation `encrypted_index` with `UnsupportedInput` before transport. Anthropic requires the original encrypted state for continuation; a display placeholder cannot restore it. Preserve the metadata extension or explicitly choose text-only history. This check applies even without new search tools. Empty results, unknown results, and search errors do not require source ciphertext. Other provider profiles retain their existing reduced-history behavior, and native Messages passthrough leaves validation to the provider. Molto does not inspect other opaque values or validate their issuer; transport overrides remain opaque.
 

python/ARCHITECTURE.md

@@ -350,7 +350,7 @@ The normalized termination reason maps shared semantics explicitly: normal compl
 
 Chat projection rejects reasoning signatures and encrypted reasoning based on output data, including signature-only parts. Search declarations and transport overrides do not change that rule; turns without reasoning or with unsigned raw reasoning remain representable.
 
-`TextOutput.citations` holds `WebCitation` ranges local to each text part. `CitationDelta` adds attribution, and the final snapshot is authoritative. Citation buffering belongs to the Messages adapter, exposed by the `Molto.messages()` argument `citation_policy`, which accepts `"buffer"` or `"drop"`. With Responses upstream, the default `"drop"` streams text and discards attribution; `"buffer"` waits for every text part's completed snapshot, including uncited text, before splitting it at citation boundaries. The coordinator selects `"drop"` for Chat upstream regardless of the requested policy, preserving immediate text streaming and chunk boundaries because Chat normalization discards annotations. The policy never changes request construction or native passthrough. The Responses normalizer and Messages projector buffer unresolved search parts in start order, allowing ordinary parts to stream when they reach the front. No upstream client predicts citation buffering, and opaque extensions cannot influence it. Any future `"auto"` policy must inspect only the normalized request; callers supplying search context only through `extra_body` must explicitly select `"buffer"` for attribution. Projection assigns output indices because parts can expand or disappear.
+`TextOutput.citations` holds `WebCitation` ranges local to each text part. `CitationDelta` adds attribution, and the final snapshot is authoritative. Citation buffering belongs to the Messages adapter, exposed by the `Molto.messages()` argument `citation_policy`, which accepts `"buffer"`, `"auto"`, or `"drop"`. With Responses upstream, the default `"drop"` streams text and discards attribution; `"buffer"` waits for every text part's completed snapshot, including uncited text, before splitting it at citation boundaries. `"auto"` buffers when the normalized request declares hosted search or carries search calls, search results, or text with citations in its history; otherwise text streams and unexpected citations are dropped. The coordinator selects `"drop"` for Chat upstream regardless of the requested policy, preserving immediate text streaming and chunk boundaries because Chat normalization discards annotations. The policy never changes request construction or native passthrough. The Responses normalizer and Messages projector buffer unresolved search parts in start order, allowing ordinary parts to stream when they reach the front. No upstream client predicts citation buffering, and opaque extensions cannot influence it. The `"auto"` policy inspects only the normalized request; callers supplying search context only through `extra_body` must explicitly select `"buffer"` for attribution. Projection assigns output indices because parts can expand or disappear.
 
 The [hosted-search contract](../docs/web-search.md) defines supported routes, reductions, failures, usage, and continuation.
 

python/molto/adapters/anthropic.py

@@ -9,7 +9,7 @@
 from collections import deque
 from collections.abc import AsyncIterator
 from contextlib import aclosing
-from typing import Any, Literal, NoReturn
+from typing import Any, Literal, NoReturn, TypeAlias
 
 from anthropic import APIError as AnthropicAPIError
 from anthropic import APIStatusError as AnthropicAPIStatusError
@@ -90,6 +90,8 @@
     "unknown": "end_turn",
 }
 
+CitationPolicy: TypeAlias = Literal["auto", "buffer", "drop"]
+
 _EXTENDED_THINKING_EFFORTS: tuple[tuple[int, ReasoningEffort], ...] = (
     (1024, "minimal"),
     (2048, "low"),
@@ -157,9 +159,24 @@ def _citations(citations: list[dict[str, Any]] | None, text: str) -> list[WebCit
     return result
 
 
+def _citations_are_possible(request: NormalizedRequest) -> bool:
+    """Report whether normalized tools or history signal possible URL citations."""
+    if any(isinstance(tool, WebSearchTool) for tool in request.tools or []):
+        return True
+    for item in request.input or []:
+        parts = item.content if isinstance(item, AssistantInput) else [item]
+        for part in parts:
+            if isinstance(part, (SearchCall, SearchResult)):
+                return True
+            if isinstance(part, TextOutput) and part.citations:
+                return True
+    return False
+
+
 async def _ordered_parts(
     events: AsyncIterator[ModelResponseStreamEvent],
-    citation_policy: Literal["buffer", "drop"],
+    *,
+    buffer_text: bool,
 ) -> AsyncIterator[ModelResponseStreamEvent]:
     """Stream parts in start order, yielding only the end for snapshot-based parts."""
     # Responses may stream "Alpha. Beta." before citing only "Beta.". Messages
@@ -168,8 +185,8 @@ async def _ordered_parts(
     # us to expand a query batch into searches or omit browsing.
     # If part A expands into two blocks, part B must follow both, so later parts
     # wait to preserve order and receive the correct consecutive block indices.
-    # The explicit citation policy determines whether text waits for its final
-    # snapshot; upstream request extensions never select that behavior.
+    # The adapter resolves text buffering from the policy and normalized request;
+    # upstream transport extensions never select that behavior.
     pending: dict[int, deque[ModelResponseStreamEvent]] = {}
     snapshots: set[int] = set()
     async for event in events:
@@ -182,7 +199,7 @@ async def _ordered_parts(
             pending[event.index] = deque()
             part = event.part
             if isinstance(part, (SearchCall, SearchResult)) or (
-                isinstance(part, TextOutput) and citation_policy == "buffer"
+                isinstance(part, TextOutput) and buffer_text
             ):
                 snapshots.add(event.index)
         if isinstance(event, PartEndEvent):
@@ -219,10 +236,10 @@ def __init__(
         self,
         capabilities: ProviderCapabilities,
         *,
-        citation_policy: Literal["buffer", "drop"] = "drop",
+        citation_policy: CitationPolicy = "drop",
     ) -> None:
         super().__init__(capabilities)
-        if citation_policy not in {"buffer", "drop"}:
+        if citation_policy not in {"auto", "buffer", "drop"}:
             raise ValueError(f"Unsupported citation policy: {citation_policy!r}.")
         self._citation_policy = citation_policy
 
@@ -641,7 +658,10 @@ def event(data: dict[str, Any]) -> AnthropicMessagesStreamEvent:
         thinking_parts: set[int] = set()
         thinking_signatures: dict[int, str] = {}
 
-        async with aclosing(_ordered_parts(events, self._citation_policy)) as ordered_events:
+        buffer_text = self._citation_policy == "buffer" or (
+            self._citation_policy == "auto" and _citations_are_possible(request)
+        )
+        async with aclosing(_ordered_parts(events, buffer_text=buffer_text)) as ordered_events:
             async for normalized_event in ordered_events:
                 if isinstance(normalized_event, ResponseStartEvent):
                     yield event(