mirror of
https://github.com/openai/codex.git
synced 2026-09-08 15:50:34 +00:00
61dc1d97f6
## What changed - Add a harness that runs the Codex executable against a pinned official MCP client conformance suite across shipping, intermediate, and modern protocol versions, HTTP and stdio transports, and OAuth scenarios. - Add a separate app-server regression matrix for transport, security, schema, pagination, SSE, multi-round request, and catalog-boundary behavior. - Check both suites against committed baselines so previously passing or required checks cannot disappear or newly fail, while keeping known failures visible in complete reports. - Run the fixture self-tests and both executable-level gates from the Python and TypeScript SDK test workflows. ## Testing - Add unit tests for the fixture server, official-suite adapter, conformance runner, baseline comparison, and reviewer regression runner. GitOrigin-RevId: de59f039294e34ed72873d9f6940b52e89172c0d
122 lines
4.7 KiB
Markdown
122 lines
4.7 KiB
Markdown
# MCP client conformance
|
|
|
|
This directory tests the actual Codex executable against the official Model
|
|
Context Protocol client conformance suite. It exercises the shipping legacy,
|
|
intermediate `2025-11-25`, and modern `2026-07-28` protocols, localhost HTTP,
|
|
stdio, OAuth, and additional transport and security regression fixtures.
|
|
|
|
The official upstream suite is pinned to
|
|
`modelcontextprotocol/conformance@49103de6ed70804e940637bf3e9e29e4a3f54e64`.
|
|
Use Node.js 22 and Python 3.10 or later.
|
|
|
|
## Run the conformance gate
|
|
|
|
First install the frozen workspace dependencies and build Codex:
|
|
|
|
```bash
|
|
pnpm install --frozen-lockfile
|
|
cargo build --locked --manifest-path codex-rs/Cargo.toml -p codex-cli --bin codex
|
|
```
|
|
|
|
From a published Codex checkout, run:
|
|
|
|
```bash
|
|
python3 scripts/mcp_conformance/run_codex_compliance.py \
|
|
codex-rs/target/debug/codex \
|
|
--conformance-cli node_modules/@modelcontextprotocol/conformance/dist/index.js \
|
|
--baseline-report scripts/mcp_conformance/regression-baseline-v1.json \
|
|
--report /tmp/codex-mcp-conformance.json
|
|
```
|
|
|
|
The positional executable can also point to an already built Codex binary.
|
|
`--conformance-cli` selects the exact, lockfile-installed upstream JavaScript
|
|
runner instead of downloading a moving version during a test.
|
|
|
|
## What the baseline means
|
|
|
|
`regression-baseline-v1.json` is a compact, reviewed snapshot of the upstream
|
|
revision, required protocol versions, HTTP and stdio transports, enabled modern
|
|
feature, OAuth coverage, and individual passing and failing check identities.
|
|
|
|
The gate exits successfully only when:
|
|
|
|
- The upstream suite and modern feature match the committed baseline.
|
|
- The shipping legacy, intermediate, and modern protocols are actually tested.
|
|
- The required HTTP, stdio, and authentication scenarios are actually run.
|
|
- Every previously passing check still passes.
|
|
- No additional check fails.
|
|
|
|
Existing known failures remain visible in the complete JSON report. In
|
|
particular, `success` describes complete upstream conformance and
|
|
`regressionGate.success` describes the no-new-regressions merge gate; the gate
|
|
does not relabel an existing failure as a pass.
|
|
|
|
Create a compact baseline from a reviewed complete report without contacting
|
|
the upstream suite again:
|
|
|
|
```bash
|
|
python3 scripts/mcp_conformance/run_codex_compliance.py \
|
|
/absolute/path/to/codex \
|
|
--baseline-report /absolute/path/to/full-conformance-report.json \
|
|
--extract-baseline /tmp/mcp-conformance-regression-baseline-v1.json
|
|
```
|
|
|
|
Alternatively, add `--write-baseline /tmp/mcp-conformance-regression-baseline-v1.json`
|
|
to a complete conformance run. Review every baseline change; do not regenerate
|
|
it to conceal a regression.
|
|
|
|
## Run the production reviewer regression gate
|
|
|
|
The separate reviewer gate tests the real Codex app-server across all three
|
|
shipping, legacy, and modern protocol modes. It covers stdio and localhost
|
|
HTTP, exact-integer tool and elicitation schemas, bounded multi-round requests,
|
|
malformed discovery response IDs, repeated pagination cursors, SSE framing and
|
|
keepalives, and catalog boundaries. In a published Codex checkout, run:
|
|
|
|
```bash
|
|
python3 scripts/mcp_conformance/review_regressions.py \
|
|
/absolute/path/to/codex \
|
|
--mode all \
|
|
--baseline-report scripts/mcp_conformance/review-regression-baseline-v1.json \
|
|
--report /tmp/codex-mcp-review-regressions.json
|
|
```
|
|
|
|
A complete, main-derived baseline records all 186 real check identities and all
|
|
21 required cases. Existing failures remain explicitly visible in the complete
|
|
report; `regressionGate.success: true` means there are no newly failing or
|
|
missing checks. Improvements are recorded under `fixedChecks`. The gate never
|
|
classifies an existing failure as a passing check.
|
|
|
|
Extract a compact deterministic reviewer baseline from a reviewed complete
|
|
production report without rerunning the client:
|
|
|
|
```bash
|
|
python3 scripts/mcp_conformance/review_regressions.py \
|
|
/absolute/path/to/codex \
|
|
--baseline-report /absolute/path/to/full-review-regressions.json \
|
|
--extract-baseline /tmp/review-regression-baseline-v1.json
|
|
```
|
|
|
|
Review every baseline update. Do not regenerate a baseline to hide a regression.
|
|
|
|
## Run the fixture self-tests
|
|
|
|
```bash
|
|
env PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 \
|
|
python3 -m pytest -q scripts/mcp_conformance
|
|
```
|
|
|
|
## Run the required SDK integration
|
|
|
|
The existing required SDK workflow runs the complete Python fixture self-tests.
|
|
Its TypeScript job builds the actual Codex executable, sets `CODEX_EXEC_PATH`,
|
|
installs the pinned upstream conformance runner, and runs both the official
|
|
authenticated suite and the separate production reviewer regression matrix.
|
|
Neither gate can be skipped. To reproduce the focused integration locally:
|
|
|
|
```bash
|
|
CODEX_EXEC_PATH=/absolute/path/to/codex \
|
|
pnpm --filter @openai/codex-sdk test -- \
|
|
--runInBand tests/mcpConformance.test.ts
|
|
```
|