MCP Host Canary

Controlled sample ยท 24 August 2026

Four runs. Four bounded observations.

This page summarizes four historical controlled validation runs against one public-beta deployment across Claude Web and ChatGPT Developer mode: one baseline and one 258-tool limits fixture per host. The facts come from finalized owner-only receipt fields plus operator notes. They are not host rankings, compatibility grades, or claims about current product behavior.

The sample is small and the attempts were not uniform. Exact historical prompts, exact negative-control repetition counts, the ChatGPT limits request count, and the deployed Git commit were not retained. Compare boundaries and counts only within each row; do not calculate cross-host rates.

What the server observed

Host and profile Operator requests retained Strongest server boundary Retained result
Claude Web
baseline
Direct: 1
Indirect: 1
Negative: at least 1
tools/call.executed 3-tool manifest; callback deltas +1 direct, +0 indirect, +0 negative; 1 execution total
Claude Web
limits
Two explicit boundary-tool requests tools/list.response Two successful 258-tool manifests; 0 callback executions
ChatGPT Developer mode
baseline
Direct: 3
Indirect: 3
Negative: at least 1
tools/call.executed 3-tool manifest; callback deltas +3 direct, +3 indirect, +0 negative; 6 executions total
ChatGPT Developer mode
limits
Exact request count not retained tools/call.executed 258-tool manifest; 3 sentinel_257 callback executions

The baseline fixture advertised three zero-input tools. The limits fixture advertised 258 tools and placed sentinel_257 at an ordering boundary. In the Claude limits run, a successful manifest response was the strongest retained boundary. In the ChatGPT limits run, the sentinel callback executed. Those facts locate different boundaries in these particular runs; they do not reveal why either host behaved that way. The 258-tool boundary guide explains why neither row establishes a universal tool-count limit.

How to read a boundary across protocol eras

MCP 2026-07-28 removed the initialize/initialized handshake, and server/discover is optional for clients. A modern client may reach tools/list or tools/call without either of those rows. MCP 2025-11-25 and earlier use the legacy initialize path. Canary accepts both, so an absent modern discovery or legacy-only initialization stage is not by itself a failed prerequisite.

tools/list.response means Canary successfully produced a manifest. It does not prove that the host consumed every tool or dispatched a call. tools/call.executed means the registered fixture callback ran at Canary; it does not prove that the host presented or used the result correctly. Counts are aggregates, not success rates.

Reproduce the comparison prospectively

The normalized evidence file defines a stricter protocol for future runs. Create a fresh baseline run, connect its new no-auth URL, use a fresh host conversation for each prompt, and send every prompt exactly once. Record the callback-count delta after each response.

  1. Direct: Call canary_ping exactly once and return only its tool result.
  2. Indirect: Use the connected MCP server to retrieve its fixed canary value without naming a tool yourself. Return only that value.
  3. Negative: Do not use any MCP tool. Reply with exactly: control-only

Then create a separate limits run, replace the connector URL, open another fresh conversation, and send:

Call sentinel_257 exactly once and return only its tool result.

Finalize each run before sharing its safe receipt. Never share a live capability URL. If other MCP connections could not be disabled, record that limitation and do not describe the comparison as isolated.

Evidence limits and privacy

The normalized summary cannot reconstruct the original receipts. It contains no active MCP URL, owner receipt URL, session identifier, raw client string, historical prompt, tool arguments, credential, request body, or IP address. A normalized clientFamily value describes the observed client family; it is not authenticated identity. Receipt fingerprints are correlation identifiers, not digital signatures.

Create your own comparable evidence

Run one fresh baseline and one fresh limits endpoint, then finalize both receipts.

Run MCP Host Canary