Skip to content

OCPBUGS-77867: Fix perspective switcher E2E test flake that causes 6-hour CI timeout#16114

Merged
openshift-merge-bot[bot] merged 2 commits intoopenshift:mainfrom
sg00dwin:OCPBUGS-77867-ci-test-perspective-switcher
Mar 9, 2026
Merged

OCPBUGS-77867: Fix perspective switcher E2E test flake that causes 6-hour CI timeout#16114
openshift-merge-bot[bot] merged 2 commits intoopenshift:mainfrom
sg00dwin:OCPBUGS-77867-ci-test-perspective-switcher

Conversation

@sg00dwin
Copy link
Copy Markdown
Member

@sg00dwin sg00dwin commented Mar 5, 2026

Summary

  • Add post-reload verification in checkDeveloperPerspective() to confirm the Developer perspective is actually available (with a 30s timeout) before returning, preventing an unbounded retry loop that hangs CI for 6 hours

Root cause

After patching console.operator.openshift.io/cluster to enable the Developer perspective, checkDeveloperPerspective() returned without verifying the perspective was actually available. The config pipeline (oc patch → operator reconcile → console rollout → SERVER_FLAGS.perspectives update) can have delays, and when the cluster was slow to reconcile, changePerspectiveTo('Developer') would call checkDeveloperPerspective() again, creating an infinite loop with no exit condition.

This accounted for ~13% of e2e-gcp-console failures over the last 14 days.

Test plan

  • CI: e2e-gcp-console passes without the "perspective switcher menu refreshed" hang

Assisted by: Claude Code

@openshift-ci-robot openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Mar 5, 2026
@openshift-ci-robot
Copy link
Copy Markdown
Contributor

@sg00dwin: This pull request references Jira Issue OCPBUGS-77867, which is invalid:

  • expected the bug to target the "4.22.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Fix perspective switcher E2E test hang in e2e-gcp-console

Summary

  • Add post-reload verification in checkDeveloperPerspective() to confirm the Developer perspective is actually
    available before returning, preventing an unbounded retry loop that hangs CI for 6 hours
  • Add recursion guard in changePerspectiveTo('Developer') to fail fast with a clear error instead of looping
    indefinitely
  • Remove unnecessary cy.initAdmin() from the "Test perspective query parameters" test, which was a legacy
    timing workaround that now contributes to the hang when Developer perspective isn't enabled by default

Root cause

After patching console.operator.openshift.io/cluster to enable the Developer perspective,
checkDeveloperPerspective() returned without verifying the perspective was actually available. When the
cluster was slow to reconcile, changePerspectiveTo('Developer') would call checkDeveloperPerspective()
again, creating an infinite loop with no exit condition.

This accounted for ~13% of e2e-gcp-console failures over the last 14 days.

Test plan

  • yarn lint passes on all modified files
  • No TypeScript errors introduced
  • Cypress test passes locally on happy path (Developer perspective already enabled)
  • CI: e2e-gcp-console passes without the "perspective switcher menu refreshed" hang

Assisted by: Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci Bot added component/dev-console Related to dev-console kind/cypress Related to Cypress e2e integration testing labels Mar 5, 2026
@sg00dwin
Copy link
Copy Markdown
Member Author

sg00dwin commented Mar 5, 2026

/jira refresh

@openshift-ci-robot openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Mar 5, 2026
@openshift-ci-robot
Copy link
Copy Markdown
Contributor

@sg00dwin: This pull request references Jira Issue OCPBUGS-77867, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.22.0) matches configured target version for branch (4.22.0)
  • bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @yapei

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci Bot requested a review from yapei March 5, 2026 18:29
@openshift-ci-robot
Copy link
Copy Markdown
Contributor

@sg00dwin: This pull request references Jira Issue OCPBUGS-77867, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.22.0) matches configured target version for branch (4.22.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @yapei

Details

In response to this:

Fix perspective switcher E2E test hang in e2e-gcp-console

Summary

  • Add post-reload verification in checkDeveloperPerspective() to confirm the Developer perspective is actually
    available before returning, preventing an unbounded retry loop that hangs CI for 6 hours
  • Add recursion guard in changePerspectiveTo('Developer') to fail fast with a clear error instead of looping
    indefinitely
  • Remove unnecessary cy.initAdmin() from the "Test perspective query parameters" test, which was a legacy
    timing workaround that now contributes to the hang when Developer perspective isn't enabled by default

Root cause

After patching console.operator.openshift.io/cluster to enable the Developer perspective,
checkDeveloperPerspective() returned without verifying the perspective was actually available. When the cluster was slow to reconcile, changePerspectiveTo('Developer') would call checkDeveloperPerspective() again, creating an infinite loop with no exit condition.

This accounted for ~13% of e2e-gcp-console failures over the last 14 days.

Test plan

  • yarn lint passes on all modified files
  • No TypeScript errors introduced
  • Cypress test passes locally on happy path (Developer perspective already enabled)
  • CI: e2e-gcp-console passes without the "perspective switcher menu refreshed" hang

Assisted by: Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Mar 5, 2026

📝 Walkthrough

Walkthrough

This pull request enhances the robustness of Developer perspective verification in OpenShift Console's integration tests. Changes include adding runtime verification steps to checkDeveloperPerspective() that wait up to 30 seconds for the Developer perspective option to appear after a reload, introducing a module-scoped flag in the navigation module to prevent overlapping perspective checks and throw an error if a concurrent check is detected, and removing the cy.initAdmin() initialization step from a test suite that no longer requires it. These modifications aim to improve test reliability and guard against cluster reconciliation delays.

🚥 Pre-merge checks | ✅ 3
✅ Passed checks (3 passed)
Check name Status Explanation
Title check ✅ Passed The title directly and specifically addresses the main change: fixing a perspective switcher E2E test flake that causes CI timeouts. It clearly reflects the root cause (infinite retry loop) and primary objective (adding verification and guards to prevent hangs).
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
frontend/packages/integration-tests-cypress/views/nav.ts (1)

6-7: Module-level state persists across tests—consider documenting or exporting a reset.

Module-scoped variables in Cypress persist across all tests in the same spec file (and potentially across specs if the module is cached). While the flag prevents recursion, it also creates implicit coupling between tests.

If you keep this pattern, consider:

  1. Exporting a resetDeveloperPerspectiveFlag() helper for specs that need explicit control
  2. Documenting this behavior so future test authors understand the single-attempt semantics
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@frontend/packages/integration-tests-cypress/views/nav.ts` around lines 6 - 7,
The module-level flag developerPerspectiveCheckAttempted creates persistent
state across tests; add a small exported helper like
resetDeveloperPerspectiveFlag() and call it from specs that need a fresh state,
or at minimum export and document the flag's single-attempt semantics so test
authors can reset or understand behavior; locate the flag declaration
(developerPerspectiveCheckAttempted) in nav.ts and implement/export the reset
function and update module documentation/comments accordingly.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@frontend/packages/integration-tests-cypress/views/nav.ts`:
- Around line 65-80: The guard flag developerPerspectiveCheckAttempted can
remain true if checkDeveloperPerspective() or the subsequent Cypress commands
fail, causing cascading test failures; wrap the call chain (the
checkDeveloperPerspective() invocation and the cy.byLegacyTestID(...).click()
chain) inside a cy.wrap().then(...) and ensure
developerPerspectiveCheckAttempted is set back to false in both the success and
failure handlers (i.e., reset the flag in the then fulfillment and in the
rejection/error handler so it always runs), or alternatively provide/reset
developerPerspectiveCheckAttempted in a beforeEach by exporting a reset helper —
update the code around developerPerspectiveCheckAttempted and the
checkDeveloperPerspective()/perspective-switcher actions to implement this.

---

Nitpick comments:
In `@frontend/packages/integration-tests-cypress/views/nav.ts`:
- Around line 6-7: The module-level flag developerPerspectiveCheckAttempted
creates persistent state across tests; add a small exported helper like
resetDeveloperPerspectiveFlag() and call it from specs that need a fresh state,
or at minimum export and document the flag's single-attempt semantics so test
authors can reset or understand behavior; locate the flag declaration
(developerPerspectiveCheckAttempted) in nav.ts and implement/export the reset
function and update module documentation/comments accordingly.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: d4b58444-d008-4720-9c22-d3055d912e4f

📥 Commits

Reviewing files that changed from the base of the PR and between 7a5908e and 7ccf35e.

📒 Files selected for processing (3)
  • frontend/packages/dev-console/integration-tests/support/pages/functions/checkDeveloperPerspective.ts
  • frontend/packages/integration-tests-cypress/tests/crud/other-routes.cy.ts
  • frontend/packages/integration-tests-cypress/views/nav.ts
💤 Files with no reviewable changes (1)
  • frontend/packages/integration-tests-cypress/tests/crud/other-routes.cy.ts

Comment thread frontend/packages/integration-tests-cypress/views/nav.ts Outdated
Revert nav.ts recursion guard and other-routes.cy.ts initAdmin removal
per review feedback. The essential fix is the post-reload verification
in checkDeveloperPerspective that confirms the Developer perspective is
actually available after the oc patch + rollout + reload sequence.

Assisted by: Claude Code
@rhamilto
Copy link
Copy Markdown
Member

rhamilto commented Mar 6, 2026

/retest

3 similar comments
@rhamilto
Copy link
Copy Markdown
Member

rhamilto commented Mar 7, 2026

/retest

@rhamilto
Copy link
Copy Markdown
Member

rhamilto commented Mar 7, 2026

/retest

@rhamilto
Copy link
Copy Markdown
Member

rhamilto commented Mar 8, 2026

/retest

@rhamilto
Copy link
Copy Markdown
Member

rhamilto commented Mar 8, 2026

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Mar 8, 2026
@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented Mar 8, 2026

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rhamilto, sg00dwin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 8, 2026
@sg00dwin
Copy link
Copy Markdown
Member Author

sg00dwin commented Mar 9, 2026

/verified by @sg00dwin

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Mar 9, 2026
@openshift-ci-robot
Copy link
Copy Markdown
Contributor

@sg00dwin: This PR has been marked as verified by @sg00dwin.

Details

In response to this:

/verified by @sg00dwin

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented Mar 9, 2026

@sg00dwin: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@openshift-merge-bot openshift-merge-bot Bot merged commit 343ecab into openshift:main Mar 9, 2026
8 checks passed
@openshift-ci-robot
Copy link
Copy Markdown
Contributor

@sg00dwin: Jira Issue Verification Checks: Jira Issue OCPBUGS-77867
✔️ This pull request was pre-merge verified.
✔️ All associated pull requests have merged.
✔️ All associated, merged pull requests were pre-merge verified.

Jira Issue OCPBUGS-77867 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓

Details

In response to this:

Summary

  • Add post-reload verification in checkDeveloperPerspective() to confirm the Developer perspective is actually available (with a 30s timeout) before returning, preventing an unbounded retry loop that hangs CI for 6 hours

Root cause

After patching console.operator.openshift.io/cluster to enable the Developer perspective, checkDeveloperPerspective() returned without verifying the perspective was actually available. The config pipeline (oc patch → operator reconcile → console rollout → SERVER_FLAGS.perspectives update) can have delays, and when the cluster was slow to reconcile, changePerspectiveTo('Developer') would call checkDeveloperPerspective() again, creating an infinite loop with no exit condition.

This accounted for ~13% of e2e-gcp-console failures over the last 14 days.

Test plan

  • CI: e2e-gcp-console passes without the "perspective switcher menu refreshed" hang

Assisted by: Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-merge-robot
Copy link
Copy Markdown
Contributor

Fix included in accepted release 4.22.0-0.nightly-2026-03-10-100251

@sg00dwin sg00dwin deleted the OCPBUGS-77867-ci-test-perspective-switcher branch March 16, 2026 15:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. component/dev-console Related to dev-console jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. kind/cypress Related to Cypress e2e integration testing lgtm Indicates that a PR is ready to be merged. verified Signifies that the PR passed pre-merge verification criteria

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants