test: Migrate/autotest and add more ui test case#1620
Merged
Conversation
wenytang-ms
requested review from
chagong,
jdneo and
testforstephen
as code owners
wenytang-ms
changed the title
The previous label 'Workbench: Close All Editors' does not exist in VS Code's command palette - the actual visible label is 'View: Close All Editors'. The palette fuzzy match silently produced no result, so Enter dismissed the palette and the test step 'passed' in ~830ms without actually closing the webview. Subsequent verifyWebview assertions still passed because getWebviewText concatenates innerText from all iframe.webview frames, so prior webview content leaked into later checks. Use the exact palette label so the editor area is genuinely cleared between webviews, confirmed by inspecting *_after.png screenshots. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
LLM gating already has three layers in autotest: --no-llm flag, AZURE_OPENAI_ENDPOINT+API_KEY env vars, and per-step verify field. Fork PRs without secret access automatically skip the LLM block, so the unconditional --no-llm on PRs was overly defensive. Internal PRs and scheduled / manual runs with secrets now get LLM verification of every passing step (downgrades pass -> fail when LLM is confident the deterministic check was a silent pass). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add two steps that click the JDK Runtime tab's <vscode-single-select> (id="jdk-dropdown") and capture the open state. We do not assert which JDKs the runner exposes — only that the dropdown still opens, which is what the React 19 + @vscode-elements migration could regress. Pin the autotest CLI to ^0.7.0 so CI picks up the new clickInWebview action (publishing 0.7.0 happens separately on the autotest repo). Also ignore test-results/ — those are local autotest artifacts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
npm pulls latest by default. Pinning to ^0.7.0 blocked CI until 0.7.0 publishes, which gives a poor migration story for clickInWebview rollout. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- java-basic-editing: rename palette command 'Workbench: Close All Editors' to 'View: Close All Editors' (4 occurrences) — autotest 0.6.9 palette guard caught the old label as a no-op match. - java-gradle: goToLine 5 -> 2 (Test1.java has only 4 lines); drop verify: on verify-completion (passive wait — completion popup may dismiss before screenshot). - java-dependency-viewer: replace stale openDependencyExplorer action (whose underlying palette title 'Java: Focus on Java Dependencies View' no longer exists) with 'run command Explorer: Focus on Java Projects View'; switch expand syntax from 'expand X tree item' to the supported 'expandTreeItem X'; check Maven Dependencies before expanding JRE so it stays in viewport; drop verify: on passive wait. - java-single-no-workspace: drop verify: on verify-completion; bump waitBefore 5->8s for the completion popup to render before screenshot. - java-webview-migration: drop verify: on the 3 transitional open-* steps (open-java-runtime / open-classpath-config / open-formatter-settings); React renders milliseconds after the command returns and CI runners occasionally captured a blank webview pre-render. The next verify-* step is the real visual assertion. Generalize verify-formatter-settings text — LLM was miscounting the stacked category list. - java-maven-resolve-type: replace the fragile applyCodeAction 'Resolve unknown type' flow (silently no-ops when it matches a sub-menu action without navigating into it — confirmed via screenshot showing Gson still unresolved) with a deterministic pom-edit flow: insert Gson field -> verifyProblems errors:1 -> inject <dependency> on pom.xml line 10 -> wait 30s + waitForLanguageServer for re-import -> insert import -> verifyProblems errors:0. Reshape test-fixtures/maven-resolve-type/pom.xml with an empty <dependencies> block + injection-point comment so line 10 is a stable target. - java-test-runner: switch from upstream vscode-java/maven/salut (which has zero @test files — palette 'Test: Run All Tests' reported 'No tests have been found' and the verify text was never deterministically checked) to a self-owned maven-junit fixture with one @test class. Replace stale openTestExplorer / runAllTests actions (whose palette titles are obsolete) with 'run command Java: Run Tests' (live vscode- java-test command). Bump ls-ready timeout to 300s for cold-cache Maven imports. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… maven-resolve
Round 2 of CI fixes after first push surfaced LLM-downgrade flakes on plans
that passed deterministic checks but were re-evaluated against transient
screenshot states:
- java-basic-editing: drop verify: on save-after-organize. The deterministic
verifyFile.contains 'import java.io.File' on disk is the source of truth;
the LLM was downgrading because the editor pane occasionally shows the
pre-save buffer (organize-on-save writes to the file but the visible tab
may not refresh) and the AFTER screenshot looks identical to BEFORE.
- java-maven-java25 / java-single-file / java-maven-multimodule / java-maven:
drop verify: on every triggerCompletionAt step. On CI runners the
completion popup occasionally still shows 'Loading…' at screenshot time or
appears below the method body — both transient. verifyCompletion.notEmpty
is the deterministic ground truth and was passing on every run; only the
LLM re-verify was downgrading. Also bump waitBefore: 5 so the popup has
time to render fully.
- java-maven-resolve-type:
* Fix verifyFile.path: 'pom.xml' -> '~/pom.xml' so autotest resolves it
against the workspace root (worktree) not the runner's CWD. Without the
'~/' prefix the verifier looked at the source-repo root and failed
with 'File not found: D:\\a\\vscode-java-pack\\vscode-java-pack\\pom.xml'.
* Drop verify: on insert-unknown-type — verifyProblems.errors >= 1 is the
deterministic ground truth; LLM was downgrading because the red squiggle
hadn't rendered yet at the AFTER screenshot.
* Bump waitBefore on insert-unknown-type 3 -> 8, save-after-resolve 15 -> 20.
* Bump wait-maven-reimport timeout 240 -> 300 and waitBefore 30 -> 45 for
cold-cache CI Maven imports of gson 2.10.1.
* Drop verify: on save-pom, reopen-app, add-import, save-after-resolve to
avoid LLM downgrades on transient editor states.
- java-test-runner:
* Bump wait-test-discovery 20s -> 45s (vscode-java-test scan is async and
cold CI is slower).
* Drop verify: on run-all-tests / wait-test-complete / reopen-test-file —
on first invocation a 'No tests found in this file' tooltip can flash
before discovery propagates and the LLM was anchoring on it. The
deterministic verifyEditor.contains '@test' on the final reopen is the
real assertion.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…le-java25 completion) - java-dependency-viewer: drop verify: on verify-jdk step. The wiki uses 'JDK Libraries' as a category label, but the actual tree node label is 'JRE System Library' (with child modules like java.base). The deterministic 'expandTreeItem JRE System Library' action is the ground truth (it fails fast if the node doesn't exist); the verify: text was causing LLM downgrades because BEFORE/AFTER screenshots correctly showed JRE System Library expansion but the LLM expected a separate 'JDK Libraries' grouping that doesn't exist in current vscode-java. - java-gradle-java25: drop verify: on verify-completion (same flake as the other 4 completion plans fixed in the previous commit — Gradle java25 plan was missed). Add waitBefore: 5 so the popup has time to render before screenshot capture. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
CI run 25663760786 surfaced 5 NEW LLM-downgrade flakes (different plans than rounds 1-3): - java-debugger: verify-breakpoint — LLM missed the yellow execution-line marker on the screenshot (off-viewport when debug toolbar pushes editor down). Deterministic ground truth is the next debugStepOver action, which can only succeed when the debugger is paused. - java-extension-pack: configure-classpath — Project Settings webview lazy-loads, command step screenshot caught empty frame. Moved the LLM check onto the next wait step (5s) which captures the rendered UI. - java-maven, java-maven-java25, java-single-file: ls-ready — waitForLanguageServer returns when status reaches 'Java: Ready' but the LS often re-enters Building/Searching for incremental compilation right after Maven import, so the AFTER snapshot can catch that intermediate state. Fix: drop verify: text on ls-ready across all plans (preventive — 11 other plans were carrying the same brittle text) and on the two specific flaky steps. The deterministic verifiers (verifyProblems.errors:0, debugStepOver success, subsequent verify-page wait) remain as ground truth. Local: all 5 failing plans now pass with --no-llm. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Last remaining CI failure (run 25665240373): the save-all-step5 verify text 'All files saved, no compilation errors' caused an LLM downgrade. After the prior step 'apply-code-action Create method call()' Eclipse inserts a TODO-marked stub. The LLM consistently flagged the lingering TODO marker as 'compilation error persists', concluding Save All didn't work. Ground truth: verifyProblems.errors:0 already passes (TODOs are not errors). Drop verify: text — deterministic verifier remains. Local: java-basic-editing 21/21 with LLM verification on. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Round-trip review pointed out that prior CI iterations had dropped 43 verify lines across 16 test plans to dodge LLM-downgrade flakes. Verify text is part of the test-plan documentation and must remain. This commit restores every removed verify line and rewrites each to describe only what is reliably observable in a screenshot: - Focus verify text on persistent visible state (project tree, editor contents, command-was-invoked), not transient UI (Problems panel contents, status-bar text, CodeLens/gutter rendering, unsaved-dot). - Add `waitBefore` on steps where the LLM needs a stable snapshot. Plan-specific fixes: - java-fresh-import: disable Gradle import for spring-petclinic. The upstream repo ships both pom.xml and build.gradle; the Gradle daemon races the Maven import on cold CI runners and breaks LS readiness. Force Maven-only via workspaceSettings `java.import.gradle.enabled: false` (matches the wiki Maven scenario). - java-maven-resolve-type: open pom.xml explicitly before insertLineInFile so the editor's AFTER screenshot shows the inserted <dependency> block (insertLineInFile is disk-only and does not open the target file). - java-test-runner: pin `java.test.editor.enableCodelens: true` via workspaceSettings; rewrite reopen-test-file verify to describe only visible editor content (CodeLens may not render before discovery finishes on cold runners — verifyEditor.contains "@test" is the deterministic ground truth). Local LLM validation: 16/16 plans pass with `o4-mini` model. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…timodule, single-file CI run 41 surfaced 5 plans with LLM-downgrade flakes (commit 87961de): - java-maven-multimodule: ls-ready (problems-panel transient errors), module1-completion + module2-completion (Loading... popup), module2 opened wrong Foo.java (same-name disambiguation issue) - java-single-file + java-single-no-workspace: verify-completion (Loading...) - java-maven: ls-ready (transient diagnostics), verify-completion (Loading...) - java-maven-resolve-type: add-gson (identical screenshots), save-after-resolve (editor squiggle render lag after diagnostic publish) Fixes: 1. ls-ready (maven, multimodule): drop deterministic verifyProblems.errors:0 (LS is Ready but diagnostics may still be recomputing) and soften verify text to mention Problems may briefly show transient errors. 2. Completion-popup steps (single-file, single-no-workspace, multimodule×2, maven, gradle-java25, maven-java25): rewrite verify to explicitly accept 'Loading...' as a valid intermediate state since verifyCompletion.notEmpty already passed deterministically. Bump waitBefore to 8s. 3. java-maven-multimodule module2: add close-module1-foo step (View: Close All Editors) before open-module2-foo so quick-open disambiguates path instead of re-focusing the already-open module1/Foo.java. 4. java-maven-resolve-type: major restructure - Add workspaceSettings: java.configuration.updateBuildConfiguration: 'automatic' so pom changes auto-trigger re-import. - Drop pre-'open file pom.xml' (was unused). - Drop the explicit save-pom step (was overwriting the disk-side insertLineInFile result with the stale editor buffer on Linux runners). - Sequence: close-all-editors → insertLineInFile pom.xml (disk-only) → reopen-pom-after-insert → Java: Reload Projects → wait-maven-reimport. - On add-gson-dependency: very explicit verify text telling LLM the screenshots SHOULD look identical (disk-only mutation, pom closed) — LLM accepts this. - Split save-after-resolve into two steps: the save step (verifies tab dirty marker clears + verifyProblems.errors:0 via status bar API) + a force-editor-refresh + verify-resolved step that closes all editors and reopens App.java so the editor freshly renders WITHOUT the now- stale red squiggle decorations (those can lag the LSP diagnostic publish by 15–30s on Linux). 4. Fix YAML duplicate waitBefore keys introduced in earlier edits. Local LLM validation (Windows + o4-mini): all 5 fixed plans now pass end-to-end including LLM re-verify. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
chagong
approved these changes
May 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.