feat: Add llms.txt catalog and execute backstop with hard read/write/delete gating

[egorpavlikhin]

Summary

Closes AIF-359 and AIF-377. Two phases shipped together as one progressive change.

Phase 1 — catalog resource + grep tool

• octopus://api/llms.txt resource with a 5-min in-memory TTL cache, shared with the grep tool so a search doesn't trigger a re-fetch.
• octopus://api/capabilities resource: server version, enabled toolsets, available tools, feature flags.
• grep_llms_txt tool — same GNU-grep parameter shape as grep_task_log. The catalog body is ~360 KB; reading it directly is what we're avoiding.
• Lifted grepLines into src/helpers/grepLines.ts so both grep tools share one implementation. Old test migrated alongside.

Phase 2 — execute backstop with hard read/write/delete tiering

• HTTP method is the authoritative classifier — never an isWrite flag the LLM sets. classifyMethod(method) maps GET → read, POST/PUT/PATCH → write, DELETE → delete.
• New --allow-deletes CLI flag + ToolsetConfig.allowDeletes. DELETE requires both --no-read-only AND --allow-deletes — a deliberate two-flag opt-in for irreversible operations.
• Six gates inside execute.ts, in order: sensitive denylist → tier mode gate → path allowlist by enabled toolset → elicitation (DELETE gets a stronger "IRREVERSIBLE" message) → dispatch → audit-on-stderr.
• Sensitive denylist covers API-key endpoints and catastrophic deletes (DELETE on /api/users/{id} and /api/spaces/{id}); enforced even with both flags on.
• Glob engine in pathGlob.ts is shared between allowlist and denylist. Only * and ** are wildcards; every other character is escaped, so denylist patterns can't be turned into regex injection.

Refactors landed in the same change

• Shared compilePathGlob between allow/deny (was duplicated).
• activeToolsetConfig lives under src/helpers/ (was misplaced under resources/catalog/ since execute also reads it).
• grepLines test migrated to src/helpers/__tests__/; grepTaskLog.ts no longer needs to re-export it.
• Audit emission in execute.ts uses a closure that captures the start time, so each gate's emit is one line (audit(\"blocked\", \"readOnlyMode\")) instead of six.

Out of scope (intentionally — Phase 3)

AIF-360 (JSON schema resources for write payloads) and AIF-367 (describe tool) are deferred. Once this lands, agents have grep_llms_txt to discover endpoint shapes — adequate as a stopgap.

Test plan

• npm run build — clean
• npm run lint — clean
• npx vitest run --exclude '**/*.integration.test.ts' — 137 unit tests pass (60+ new)
• Live smoke test against https://main.testoctopus.app: read octopus://api/capabilities, run grep_llms_txt for 'POST /releases', exercise the read tier of execute with GET /api/spaces, verify the readOnlyMode gate blocks a POST, verify the deletesNotAllowed gate blocks a DELETE under --no-read-only only.
• Pre-existing integration test failures (11) for findInterruptions/listTenants are unchanged — they fail on a missing 'Octopus Server' test space, unrelated to this work.

Coordination

The Linear tickets AIF-359 and AIF-377 describe the work as written before the three-tier DELETE design was confirmed. Worth updating the ticket bodies post-merge to reflect:

• AIF-359: scope grew to include grep_llms_txt and the grepLines helper extraction (~250 LOC, was ~150).
• AIF-377: three-tier read/write/delete classification with the new --allow-deletes flag, plus catastrophic-delete entries on the sensitive denylist.

🤖 Generated with Claude Code

[akirayamamoto]

PR Review Notes

Nice direction overall. The split between read, write, and delete modes makes
sense, and I like that DELETE has a separate opt-in rather than being bundled
into general write access.

A couple of UX suggestions:

Consider renaming --no-read-only

--no-read-only works, but it asks users to reason through a double negative.
Since the flag enables write operations across the MCP server, I think
--allow-writes would be easier to understand at a glance.

You could keep --no-read-only as an alias for compatibility, but document
--allow-writes as the preferred flag:

--allow-writes
--allow-deletes

That reads more naturally: writes are off by default, and deletes require an
additional opt-in.

Make --allow-deletes clearer when read-only mode is still active

Since --allow-deletes only takes effect when write mode is also enabled, I’d
consider either rejecting this combination at startup:

--allow-deletes

without:

--no-read-only

or printing a clear startup warning.

Otherwise a user may think they have enabled deletes when read-only mode still
blocks them. That is safe behavior, but it could be confusing during setup or
debugging.

Something like this would help:

--allow-deletes was provided, but read-only mode is still enabled. DELETE
requests remain blocked unless write mode is enabled.

Not a blocker from my side, but I think these tweaks would make the safety model
easier to understand.

Additional Codex Findings

Codex found a few issues worth looking at before merge:

1. Path checks should run on a canonical path

execute appears to run the allowlist and denylist checks against the raw path
argument, then passes that path to the API client. Codex found that paths with
dot segments, such as:

/api/spaces/Spaces-1/../../users/me/apikeys

can pass the /api/spaces/** allowlist check while resolving to a different API
path. I’d suggest normalizing or rejecting paths with .., encoded slashes,
backslashes, query strings, or fragments before any safety checks run.

2. /api/spaces/** makes the toolset allowlist too broad

Codex also found that core includes /api/spaces/**. Since core is always
enabled and checked first, this appears to allow all space-scoped endpoints
through execute, even when the owning toolset is disabled.

That makes the documented “toolset filtering as a kill switch” model weaker than
it looks. I’d suggest narrowing core to space discovery paths only, then adding
explicit /api/spaces/{space}/... patterns to each owning toolset.

3. octopus://api/capabilities may describe execute as read-only

Codex found that execute is registered as readOnly: true so it remains
available in read-only mode for GET requests. That makes sense internally, but
the capabilities resource may lead agents to think execute is fully read-only,
even when write mode allows POST/PUT/PATCH.

It may be clearer to expose execute as method-gated, or include effective
tiers such as read, write, and delete.

4. Minor docs issue: audit toolset

Codex noticed a couple of references to an audit toolset in comments/docs, but
that does not seem to be a current toolset. This is minor, but it could confuse
users reading the safety model.