Evaluating agents that help without being asked.
Built by Deepnote. Set inside PageMantle, a synthetic company.
Quickstart · Models · Results · Dataset
An agent joins PageMantle, a synthetic software company. Starting from recent Slack conversations, it uses tools to investigate across connected services and decide whether to help. It must choose what to say, who to contact and where to send it. If the problem is already handled, the right action is silence.
In Slack
Usage of this feature is taking off. Let's give it a more prominent spot.
In the data
- PostHog: almost all the extra activity comes from one customer.
- BigQuery: the same action keeps failing and being retried.
- Slack: that customer has an unresolved support thread.
The agent's job: connect the spike to the customer's problem before repeated failed attempts are mistaken for broad adoption.
Conversations, documents, customer records and analytics are linked across eleven services, each with its own tools:
Slack · Linear · Notion · HubSpot · Stripe · Google Calendar · PostHog · Google Analytics · Google Search Console · BigQuery · Snowflake
The tools run against a fixed local dataset. No live service accounts are needed.
PageMantle on Hugging Face · Dataset setup.
Judge models assess the decision to intervene, evidence, message quality and audience. Data, tasks and prompts are pinned. Tool calls, messages and both verdicts stay available in the CLI.
This repository includes eight runnable examples, one per task type.
Explore the eight task types
| Type | What it tests |
|---|---|
| Find | Find a fact, document, person in charge, past decision or current status. |
| Explain | Explain how something works or what a term, rule or process means. |
| Analyze | Combine and compare data to reveal patterns, trends or changes. |
| Verify | Check whether a claim, document or number matches the source evidence. |
| Diagnose | Find why something is failing and what could fix it. |
| Decide | Compare options, set priorities and recommend a choice with clear reasons. |
| Coordinate | Work out who needs to act, what is blocking progress and how related work fits together. |
| Guard | Check who may access or share information and whether a proposed action follows company rules. |
We evaluated 12 models on 50 tasks across all eight task types. See the full results.
To reference Agent Initiative Benchmark in your work, use:
@misc{deepnote2026agentinitiative,
title = {{Agent Initiative Benchmark}},
author = {{Deepnote}},
year = {2026},
version = {1.0.0},
url = {https://github.com/deepnote/agent-initiative-benchmark}
}You can also use GitHub's Cite this repository button.
Code, documentation and tasks: Apache 2.0. PageMantle dataset and its bundled manifest: CC BY 4.0. See NOTICE and third-party licenses.





