Skip to content

fix: escalating circuit breaker for doom loops in headless mode #658

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
anandgupta42 wants to merge 4 commits into
base: main
Choose a base branch
from
+78 −4
Open

fix: escalating circuit breaker for doom loops in headless mode #658

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
7671cf9
fix: escalating circuit breaker for doom loops in headless mode
anandgupta42 Apr 6, 2026
7d9b09d
Merge branch 'main' into fix/doom-loop-escalation
anandgupta42 Apr 6, 2026
558b9fa
fix: address code review findings
github-actions[bot] Apr 6, 2026
ac01755
fix: address code review findings
github-actions[bot] Apr 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions packages/opencode/src/altimate/telemetry/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -243,6 +243,9 @@ export namespace Telemetry {
session_id: string
tool_name: string
repeat_count: number
// altimate_change start — escalation level for distinguishing ask/warn/stop in analytics
escalation_level?: number
// altimate_change end
}
| {
type: "environment_census"
Expand Down
79 changes: 75 additions & 4 deletions packages/opencode/src/session/processor.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,15 @@ export namespace SessionProcessor {
// 30 catches pathological patterns while avoiding false positives for power users.
const TOOL_REPEAT_THRESHOLD = 30
// altimate_change end
// altimate_change start — escalating circuit breaker for doom loops
// When the repeat threshold is hit and auto-accepted (headless, config allow), the
// counter resets and the loop continues indefinitely. Escalation levels:
// 1st hit (30 calls): ask permission (existing behavior)
// 2nd hit (60 calls): ask + inject synthetic warning telling model to change approach
// 3rd hit (90 calls): force-stop the session — the model is stuck
const DOOM_LOOP_WARN_ESCALATION = 2 // hits before injecting warning
const DOOM_LOOP_STOP_ESCALATION = 3 // hits before force-stopping
// altimate_change end
const log = Log.create({ service: "session.processor" })

export type Info = Awaited<ReturnType<typeof create>>
Expand All @@ -42,6 +51,9 @@ export namespace SessionProcessor {
// altimate_change start — per-tool call counter for varied-input loop detection
const toolCallCounts: Record<string, number> = {}
// altimate_change end
// altimate_change start — escalation counter: how many times each tool has hit TOOL_REPEAT_THRESHOLD
const toolLoopHits: Record<string, number> = {}
// altimate_change end
let snapshot: string | undefined
let blocked = false
let attempt = 0
Expand Down Expand Up @@ -201,20 +213,77 @@ export namespace SessionProcessor {
})
}

// altimate_change start — per-tool repeat counter (catches varied-input loops like todowrite 2,080x)
// altimate_change start — per-tool repeat counter with escalating circuit breaker
// Counter is scoped to the processor lifetime (create() call), so it accumulates
// across multiple process() invocations within a session. This is intentional:
// cross-turn accumulation catches slow-burn loops that stay under the threshold
// per-turn but add up over the session.
toolCallCounts[value.toolName] = (toolCallCounts[value.toolName] ?? 0) + 1
if (toolCallCounts[value.toolName] >= TOOL_REPEAT_THRESHOLD) {
toolLoopHits[value.toolName] = (toolLoopHits[value.toolName] ?? 0) + 1
const hits = toolLoopHits[value.toolName]
const totalCalls = hits * TOOL_REPEAT_THRESHOLD

Telemetry.track({
type: "doom_loop_detected",
timestamp: Date.now(),
session_id: input.sessionID,
tool_name: value.toolName,
repeat_count: toolCallCounts[value.toolName],
repeat_count: totalCalls,
escalation_level: hits,
})

// Escalation level 3+: force-stop — the model is irretrievably stuck
if (hits >= DOOM_LOOP_STOP_ESCALATION) {
log.warn("doom loop circuit breaker: force-stopping session", {
tool: value.toolName,
totalCalls,
hits,
sessionID: input.sessionID,
})
await Session.updatePart({
id: PartID.ascending(),
messageID: input.assistantMessage.id,
sessionID: input.assistantMessage.sessionID,
type: "text",
synthetic: true,
text:
`⚠️ altimate-code: session stopped — \`${value.toolName}\` was called ${totalCalls} times, ` +
`indicating the agent is stuck in a loop. Please start a new session with a revised prompt.`,
time: { start: Date.now(), end: Date.now() },
})
blocked = true
toolCallCounts[value.toolName] = 0
Copy link
Copy Markdown

@dev-punia-altimate dev-punia-altimate Apr 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 finding (warning)

At escalation level 3 (force-stop), toolCallCounts[value.toolName] is correctly reset to 0 at line 256, but toolLoopHits[value.toolName] is left at 3. In current code this is harmless because blocked = true causes the session to return "stop" and the processor is not reused. However, the two maps are now permanently out of sync: if any future code path ever calls process() again on the same processor instance after a force-stop (e.g., an error-recovery refactor), the very next threshold hit would read hits=4, immediately triggering another force-stop instead of cycling through the warn phase first. The force-stop branch should also include toolLoopHits[value.toolName] = 0 to keep both maps consistent.

toolLoopHits[value.toolName] = 0
break
}

// Escalation level 2: warn the model via synthetic message
if (hits >= DOOM_LOOP_WARN_ESCALATION) {
log.warn("doom loop escalation: injecting warning", {
tool: value.toolName,
totalCalls,
hits,
sessionID: input.sessionID,
})
await Session.updatePart({
Copy link
Copy Markdown

@dev-punia-altimate dev-punia-altimate Apr 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 finding (warning)

At escalation level 2, a warning text part is written to input.assistantMessage.id with synthetic: false. In message-v2.ts toModelMessages(), all type === 'text' parts on assistant messages are included in the LLM context WITHOUT filtering for synthetic (unlike user messages which are filtered at prompt.ts line 648). This means the warning string is permanently stored in the assistant message record and will be re-sent verbatim to the LLM on every future turn for the life of the session, not just the looping turn where it was injected. If the model course-corrects and the session continues for many more turns, every subsequent LLM request includes this warning in the conversation history. The force-stop message at level 3 uses synthetic: true which does not help here since toModelMessages does not filter synthetic on assistant messages. Both messages are affected, but the level-3 message is moot since the session ends. The level-2 case is the real concern.

id: PartID.ascending(),
messageID: input.assistantMessage.id,
sessionID: input.assistantMessage.sessionID,
type: "text",
synthetic: true,
Copy link
Copy Markdown

@cubic-dev-ai cubic-dev-ai bot Apr 6, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: The level 2 escalation warning should use synthetic: false so the LLM actually sees the "stop looping" instruction. With synthetic: true, prompt.ts filters this part out before building the LLM prompt (lines 648, 795), so the model never sees the warning and cannot change its behavior. This contradicts the PR's stated intent and makes level 2 functionally identical to level 1.

Prompt for AI agents 
Check if this issue is valid — if so, understand the root cause and fix it. At packages/opencode/src/session/processor.ts, line 274:

<comment>The level 2 escalation warning should use `synthetic: false` so the LLM actually sees the "stop looping" instruction. With `synthetic: true`, `prompt.ts` filters this part out before building the LLM prompt (lines 648, 795), so the model never sees the warning and cannot change its behavior. This contradicts the PR's stated intent and makes level 2 functionally identical to level 1.</comment>

<file context>
@@ -270,7 +271,7 @@ export namespace SessionProcessor {
                           sessionID: input.assistantMessage.sessionID,
                           type: "text",
-                          synthetic: false,
+                          synthetic: true,
                           text:
                             `⚠️ altimate-code: \`${value.toolName}\` has been called ${totalCalls} times this session. ` +
</file context>
Suggested change
synthetic: true,
synthetic: false,
Fix with Cubic

text:
`⚠️ altimate-code: \`${value.toolName}\` has been called ${totalCalls} times this session. ` +
`You appear to be stuck in a loop. Stop repeating the same approach. ` +
`Either try a fundamentally different strategy or explain to the user what is blocking you. ` +
`The session will be force-stopped if this continues.`,
time: { start: Date.now(), end: Date.now() },
Comment on lines +269 to +280
Copy link
Copy Markdown

@coderabbitai coderabbitai bot Apr 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Make the 60-call warning model-visible.

Line 274 marks this warning as synthetic: true, but Lines 433-435 in the same file describe synthetic text as TUI-only and excluded from replay to the LLM. That makes the warn tier ineffective in the exact headless/auto-accept path this circuit breaker is trying to correct.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@packages/opencode/src/session/processor.ts` around lines 269 - 280, The
warning inserted via Session.updatePart (using PartID.ascending(),
input.assistantMessage.id, input.assistantMessage.sessionID, value.toolName and
totalCalls) is marked synthetic:true which makes it TUI-only and excluded from
LLM replay; change the part creation so the warning is not synthetic (remove or
set synthetic to false) so the message is visible to the model/auto-accept path,
keeping the same text, type:"text" and timestamps.

})
}

// Escalation level 1: ask permission (existing behavior)
// Reset before ask so denial/exception doesn't leave count at threshold
toolCallCounts[value.toolName] = 0
const agent = await Agent.get(input.assistantMessage.agent)
await PermissionNext.ask({
permission: "doom_loop",
Expand All @@ -223,12 +292,11 @@ export namespace SessionProcessor {
metadata: {
tool: value.toolName,
input: value.input,
repeat_count: toolCallCounts[value.toolName],
repeat_count: totalCalls,
},
always: [value.toolName],
ruleset: agent.permission,
})
toolCallCounts[value.toolName] = 0
}
// altimate_change end
}
Expand Down Expand Up @@ -478,6 +546,9 @@ export namespace SessionProcessor {
continue
}
if (needsCompaction) break
// altimate_change start — exit stream loop immediately on doom loop force-stop
if (blocked) break
Comment on lines +549 to +550
Copy link
Copy Markdown

@coderabbitai coderabbitai bot Apr 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Scope the early stream break to the new hard-stop path only.

blocked is also set on Lines 345-350 for normal permission/question rejections. With this unconditional break, those pre-existing denial paths now short-circuit the rest of the stream too, which can skip the finish-step bookkeeping on Lines 372-460. A dedicated forceStopped flag would preserve the old rejection flow while still halting doom-loop stops immediately.

💡 Suggested change 
-    let blocked = false
+    let blocked = false
+    let forceStopped = false
...
-                        blocked = true
+                        blocked = true
+                        forceStopped = true
...
-              if (blocked) break
+              if (forceStopped) break
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@packages/opencode/src/session/processor.ts` around lines 549 - 550, The early
stream break currently uses the shared variable "blocked" which also represents
normal permission/question rejections; introduce a new boolean "forceStopped"
scoped alongside "blocked" and set it only in the doom-loop hard-stop path
(where the code currently sets "blocked" for force-stop), then change the
immediate break condition to test "forceStopped" (if (forceStopped) break) so
normal rejection flows still run the finish-step bookkeeping in the rest of the
stream; ensure "forceStopped" is initialized in the same scope as "blocked" and
is not used elsewhere.

// altimate_change end
}
} catch (e: any) {
log.error("process", {
Expand Down
Loading