Public-web research by role: Exa and Firecrawl beside Parallel

CHANGELOG.md

@@ -32,6 +32,43 @@ Newest first. `Unreleased` is what is on `main` and not yet tagged.
   `code_required` (below). A server and a computer must be upgraded together: an older computer has
   no `/sign-in/code`.
 
+### A reply's tool calls fold into one line
+
+A Bot that searched, read two pages and checked Drive before answering drew four lines above its
+answer. Adjacent connector calls and browser steps now fold into one line that says what was done,
+such as "Searched the web and read 2 pages", with an icon for each kind of step. It names the step in
+progress while the Bot works, and counts issues when a step was blocked or failed. Opened, it lists
+each step with its query, page or connector, and each step opens to what it was given and what came
+back. A single connector call keeps its own line, and approvals, connection requests and anything the
+app draws itself stay outside the fold.
+
+### Public-web research works with Exa and Firecrawl, not only Parallel
+
+Four new catalogue entries give a Bot public-web search and page reading without its browser:
+**Exa Search** (free, rate-limited), **Exa Search (API key)**, **Firecrawl** (free) and **Firecrawl
+(API key)**. They sit beside the two Parallel entries, and nothing is granted to any Bot until an administrator does it.
+See [docs/web-research.md](docs/web-research.md).
+
+- **The research guidance follows whichever provider a Bot holds.** It used to switch on only for
+  Parallel's two tools. Now a Bot holding any provider's search and fetch tools is told how to use
+  them, and a Bot holding more than one uses the one the person asks for.
+- **A Bot holding a research provider searches without being asked** when an answer depends on facts
+  that change, such as current figures, rankings or recent events, and stops after a search or two.
+  Before, every Bot was told not to go looking, so one holding a search tool still answered from
+  memory. A Bot with only its browser now says it can open pages but cannot search.
+- **A conversation doesn't say which provider searched.** Research calls show as "Search the web"
+  and "Read web pages" rather than "Web search exa", and the Bot doesn't name the provider unless
+  asked. The Bot's access page still shows which provider it holds.
+- **The shipped `research-public-web` skill declares roles, not Parallel.** Its tools are now
+  `@research/search` and `@research/fetch`, which match the search and fetch tools of every research
+  provider the Bot was granted. A skill written by a person can declare the same roles, and the
+  Skills page offers them under "Public-web research". A role grants nothing.
+- **Exa and Firecrawl list their reads.** Every other tool they advertise counts as a write, so a
+  Firecrawl monitor deletion or an Exa agent run goes through the same policy as any write.
+- **An API key can go in the vendor's own header.** Exa reads `x-api-key`; the deployment key for
+  that entry is sent there. The admin page now says the key is "Sent with every call to this vendor"
+  rather than calling it a bearer token.
+
 ### Bot turns hold their conversation more carefully
 
 Routines, responsibilities, background research, wake-ups, group turns and Bot-to-Bot handoffs keep a

README.md

@@ -226,7 +226,7 @@ Leave `EMBEDDED_POSTGRES` off and set `DATABASE_URL` to point at a database you
 - **Secrets never enter the transcript**: the trail records that a secret was requested and how long it was, not what it said.
 - **Bring your own agent**: any AG-UI endpoint is a Bot, on a framework or hand-written. Endpoints are validated with the same target checks used for browser navigation, and an auth header is stored write-only.
 - **Components instead of prose**: compiled React components live in `app/src/components/gallery/`, sandboxed ones are authored in `/admin/playground` and published with no deployment. Every call asks the server whether the component exists, is published, and is not withheld from that Bot. Data functions are granted per component.
-- **Governed MCP**: Google Drive, Notion and [Parallel Search](docs/parallel-research.md) ship in the catalogue, and Composio brokers a few hundred more apps behind one account, each reached as the person asking. The catalogue carries only vendors this deployment stands behind, so adding one is a review of that vendor. Custom servers must pass URL checks; unknown tools and custom-server tools are treated as writes, and a catalogue tool the server advertises but does not name as a write classifies as a read. A Bot is told which connectors exist here and which it holds, so it says it has not been granted one rather than browsing to the vendor's website.
+- **Governed MCP**: Google Drive, Notion and the [public-web research providers](docs/web-research.md) Parallel, Exa and Firecrawl ship in the catalogue, and Composio brokers a few hundred more apps behind one account, each reached as the person asking. The catalogue carries only vendors this deployment stands behind, so adding one is a review of that vendor. Custom servers must pass URL checks; unknown tools and custom-server tools are treated as writes, and a catalogue tool the server advertises but does not name as a write classifies as a read. A Bot is told which connectors exist here and which it holds, so it says it has not been granted one rather than browsing to the vendor's website.
 - **Skills are instructions, not capabilities**: personal skills attach only to Bots their author owns, deployment skills are admin-owned, and both are invoked with `/` in the composer. A Bot granted the shipped `skill-creator` skill can write one with you in the conversation, and saves it only when you press the button on the card.
 - **Sign in with what your company already has**: Google, Microsoft or Okta from the environment, or a company's own SAML or OpenID Connect provider registered while the deployment runs and routed by email domain. Any one turns sign-in on; several may be configured at once.
 - **Decide who gets in**: `/admin/people` lists everybody who has signed in, promotes and demotes them, and removes access, which ends the session they are using and stops the next sign-in. Every change is on the audit trail.
@@ -445,7 +445,7 @@ Use `bash scripts/start.sh` for the whole stack and `bash scripts/stop.sh` to ta
 - [docs/deployment.md](docs/deployment.md)
 - [charts/openbot/README.md](charts/openbot/README.md): the Helm chart
 - [docs/releasing.md](docs/releasing.md)
-- [docs/parallel-research.md](docs/parallel-research.md): public-web search and extraction through Parallel
+- [docs/web-research.md](docs/web-research.md): public-web search and extraction through Parallel, Exa or Firecrawl
 
 ## Contributing
 

app/src/components/channels/browser-activity.tsx

@@ -1,14 +1,6 @@
-import { PendingApprovals } from "@/components/approvals/inline-approval";
 import type { VisibleChatItem } from "./chat-messages";
-import { ToolLine } from "./tool-line";
 
 type BrowserStep = Extract<VisibleChatItem, { kind: "tool" }>;
-export type BrowserGroup = {
-  kind: "browser";
-  id: string;
-  steps: BrowserStep[];
-};
-export type TranscriptItem = VisibleChatItem | BrowserGroup;
 
 const labels: Record<string, string> = {
   computer_navigate: "Opened page",
@@ -20,24 +12,9 @@ const labels: Record<string, string> = {
   computer_scroll: "Scrolled page",
 };
 
-/** Group adjacent browsing steps without moving messages or hiding requests for human help. */
-export function groupBrowserSteps(
-  items: readonly VisibleChatItem[],
-): TranscriptItem[] {
-  const grouped: TranscriptItem[] = [];
-  for (const item of items) {
-    if (
-      item.kind !== "tool" ||
-      !Object.hasOwn(labels, item.toolCall.function.name)
-    ) {
-      grouped.push(item);
-      continue;
-    }
-    const last = grouped.at(-1);
-    if (last?.kind === "browser") last.steps.push(item);
-    else grouped.push({ kind: "browser", id: item.id, steps: [item] });
-  }
-  return grouped;
+/** Whether a tool call is one of the browser steps a tool group draws with its page link. */
+export function isBrowserStep(item: BrowserStep): boolean {
+  return Object.hasOwn(labels, item.toolCall.function.name);
 }
 
 function objectOf(value: string | undefined): Record<string, unknown> {
@@ -83,72 +60,3 @@ export function browserStepDetails(step: BrowserStep) {
     pending: step.result === undefined,
   };
 }
-
-export function BrowserActivity({
-  group,
-  active,
-}: {
-  group: BrowserGroup;
-  active: boolean;
-}) {
-  const steps = group.steps.map((step) => ({
-    id: step.id,
-    ...browserStepDetails(step),
-  }));
-  const pages = steps.filter((step) => step.visited).length;
-  const issues = steps.filter((step) => step.failed).length;
-  const running = active && steps.some((step) => step.pending);
-  // The folded line draws its own steps, so a waiting approval gets its card beside it, not inside.
-  return (
-    <>
-      <ToolLine
-        label={
-          running
-            ? "Browsing…"
-            : pages
-              ? `Browsed ${pages} ${pages === 1 ? "page" : "pages"}`
-              : "Browser activity"
-        }
-        detail={
-          issues ? `${issues} ${issues === 1 ? "issue" : "issues"}` : undefined
-        }
-        running={running}
-      >
-        <ol className="space-y-2 py-1">
-          {steps.map((step) => (
-            <li key={step.id} className="min-w-0">
-              <div className="flex min-w-0 items-baseline gap-2">
-                <span className="shrink-0 text-muted-foreground">
-                  {step.label}
-                </span>
-                {step.href ? (
-                  <a
-                    href={step.href}
-                    target="_blank"
-                    rel="noopener noreferrer"
-                    className="truncate underline decoration-muted-foreground/40 underline-offset-2 hover:decoration-current"
-                  >
-                    {step.title ?? step.href}
-                  </a>
-                ) : null}
-                {step.pending && (
-                  <span className="text-muted-foreground">
-                    {active ? "In progress" : "Not completed"}
-                  </span>
-                )}
-              </div>
-              {step.failed && (
-                <p className="mt-0.5 text-destructive">
-                  {step.reason ?? "This step could not be completed."}
-                </p>
-              )}
-            </li>
-          ))}
-        </ol>
-      </ToolLine>
-      <PendingApprovals
-        toolCallIds={group.steps.map((step) => step.toolCall.id)}
-      />
-    </>
-  );
-}

app/src/components/channels/chat-transcript.tsx

@@ -62,10 +62,10 @@ import { readTeamBotConsent } from "@/lib/team-bots";
 import { cn } from "@/lib/utils";
 import { VOICE_CHAT_ACTIVITY, type VoiceChatEntry } from "@/lib/voice/archive";
 import {
-  BrowserActivity,
-  groupBrowserSteps,
+  groupToolSteps,
+  ToolActivity,
   type TranscriptItem,
-} from "./browser-activity";
+} from "./tool-activity";
 import {
   attachmentModality,
   type SentAttachment,
@@ -1525,7 +1525,7 @@ export function ChatTranscript({
    * cost was markdown parsing and chart SVGs, and those are skipped by the memoised children below,
    * which is where the 25x came from. This runs per render and is not worth guarding.
    */
-  const items = groupBrowserSteps(toVisibleChatItems(messages));
+  const items = groupToolSteps(toVisibleChatItems(messages));
 
   /*
    * ONLY WHILE THERE IS NOTHING ELSE TO LOOK AT. Once a reply starts streaming, or a tool line
@@ -1668,12 +1668,12 @@ export function ChatTranscript({
             </div>
             {items.length === 0 && restoring ? <RestoringTranscript /> : null}
             {items.map((item, index) =>
-              item.kind === "browser" ? (
+              item.kind === "tools" ? (
                 <MessageScrollerItem key={item.id} messageId={item.id}>
                   <Arriving
                     delay={delays.delayFor(item.id, index, items.length)}
                   >
-                    <BrowserActivity
+                    <ToolActivity
                       group={item}
                       active={busy && index === items.length - 1}
                     />