Skip to content

chore(ci): fix flakey integration tests #7958

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
eduardoboucas merged 11 commits into from
Feb 21, 2026
Merged

chore(ci): fix flakey integration tests #7958

Changes from 2 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
f0fb392
chore: add debug logs to deploy tests
eduardoboucas Feb 20, 2026
49f0934
chore: skip deleting site
eduardoboucas Feb 20, 2026
3e1052d
chore: add cleanup CI step
eduardoboucas Feb 20, 2026
932672a
chore: add more logs
eduardoboucas Feb 20, 2026
24f03db
chore: create site once
eduardoboucas Feb 20, 2026
3d7750a
chore: improve flow
eduardoboucas Feb 20, 2026
70ac7ad
chore: fix CI issues
eduardoboucas Feb 20, 2026
516de52
chore: use right site name
eduardoboucas Feb 20, 2026
7e66582
chore: replace PR title action
eduardoboucas Feb 20, 2026
5c987b4
chore: stop printing response body
eduardoboucas Feb 20, 2026
76d00cc
chore: fix E2E flake
eduardoboucas Feb 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
183 changes: 72 additions & 111 deletions tests/integration/commands/deploy/deploy.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,18 +29,16 @@
pathname?: string | undefined
siteUrl: string
}) => {
const response = await fetch(`${siteUrl}${pathname}`, { headers })
const url = `${siteUrl}${pathname}`
const response = await fetch(url, { headers })
const body = await response.text()
const requestId = response.headers.get('x-nf-request-id') ?? ''
if (content === undefined) {
expect(response.status).toBe(404)
return
}
expect(response.status, `status should be 200. request id: ${response.headers.get('x-nf-request-id') ?? ''}`).toBe(
200,
)
expect(body, `body should be as expected. request id: ${response.headers.get('x-nf-request-id') ?? ''}`).toEqual(
content,
)
expect(response.status, `status should be 200. url: ${url} request id: ${requestId}`).toBe(200)
expect(body, `body should be as expected. url: ${url} request id: ${requestId}`).toEqual(content)
}

type Deploy = {
Expand All @@ -57,6 +55,15 @@
logs: string
function_logs: string
edge_function_logs: string
source_zip_filename?: string
}

const parseDeploy = (output: string): Deploy => {
try {
return JSON.parse(output)
} catch {
throw new Error(`Failed to parse deploy output as JSON. Raw output:\n${output}`)
}
}

const validateDeploy = async ({
Expand Down Expand Up @@ -93,7 +100,7 @@
vi.setConfig({ maxConcurrency: 3 })

describe.skipIf(disableLiveTests).concurrent('commands/deploy', { timeout: 300_000 }, () => {
beforeAll(async () => {

Check failure on line 103 in tests/integration/commands/deploy/deploy.test.ts

View workflow job for this annotation

GitHub Actions / Integration (macOS-latest, 22, 4/4)

tests/integration/commands/deploy/deploy.test.ts > commands/deploy 

Error: Hook timed out in 90000ms.
If this is a long-running hook, pass a timeout value as the last argument or configure it globally with "hookTimeout".
 ❯ tests/integration/commands/deploy/deploy.test.ts:103:3
const { account, siteId } = await createLiveTestSite(SITE_NAME)
context.siteId = siteId
context.account = account
Expand All @@ -101,8 +108,9 @@

afterAll(async () => {
const { siteId } = context
console.log(`deleting test site "${SITE_NAME}". ${siteId}`)
await callCli(['sites:delete', siteId, '--force'])
// TODO: temporarily disabled to debug deploy test failures — re-enable after investigation
console.log(`skipping deletion of test site "${SITE_NAME}". ${siteId}`)
// await callCli(['sites:delete', siteId, '--force'])
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot Feb 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Don’t skip site deletion by default — it will leak resources in CI.

Leaving test sites undeleted can accumulate and hit quotas or clutter shared accounts. Gate the skip behind an explicit env flag and keep deletion as the default behavior.

🧹 Suggested safeguard 
   afterAll(async () => {
     const { siteId } = context
-    // TODO: temporarily disabled to debug deploy test failures — re-enable after investigation
-    console.log(`skipping deletion of test site "${SITE_NAME}". ${siteId}`)
-    // await callCli(['sites:delete', siteId, '--force'])
+    if (process.env.NETLIFY_TEST_SKIP_SITE_DELETE === 'true') {
+      console.log(`skipping deletion of test site "${SITE_NAME}". ${siteId}`)
+      return
+    }
+    await callCli(['sites:delete', siteId, '--force'])
   })
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
afterAll(async () => {
const { siteId } = context
console.log(`deleting test site "${SITE_NAME}". ${siteId}`)
await callCli(['sites:delete', siteId, '--force'])
// TODO: temporarily disabled to debug deploy test failures — re-enable after investigation
console.log(`skipping deletion of test site "${SITE_NAME}". ${siteId}`)
// await callCli(['sites:delete', siteId, '--force'])
afterAll(async () => {
const { siteId } = context
if (process.env.NETLIFY_TEST_SKIP_SITE_DELETE === 'true') {
console.log(`skipping deletion of test site "${SITE_NAME}". ${siteId}`)
return
}
await callCli(['sites:delete', siteId, '--force'])
})
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@tests/integration/commands/deploy/deploy.test.ts` around lines 109 - 113, The
afterAll cleanup currently logs and skips deleting the test site; restore
deletion as the default by re-enabling the await callCli(['sites:delete',
siteId, '--force']) inside afterAll and gate skipping only when an explicit env
var is set (e.g., process.env.SKIP_SITE_DELETION or SKIP_CLEANUP). Locate the
afterAll block that references context, SITE_NAME and callCli, change it so it
checks the env flag and if not set runs callCli(['sites:delete', siteId,
'--force']); wrap the deletion call in a try/catch to log any errors (using the
existing test logger or console) so cleanup failures don’t crash the test run
but are visible.

})

test('should deploy project when dir flag is passed', async (t) => {
Expand All @@ -118,7 +126,7 @@
const deploy = await callCli(['deploy', '--json', '--no-build', '--dir', 'public'], {
cwd: builder.directory,
env: { NETLIFY_SITE_ID: context.siteId },
}).then((output: string) => JSON.parse(output))
}).then(parseDeploy)

await validateDeploy({ deploy, siteName: SITE_NAME, content })
})
Expand All @@ -142,7 +150,7 @@

const deploy = await callCli(['deploy', '--json', '--no-build', '--site', SITE_NAME], {
cwd: builder.directory,
}).then((output: string) => JSON.parse(output))
}).then(parseDeploy)

await validateDeploy({ deploy, siteName: SITE_NAME, content })
})
Expand All @@ -167,7 +175,7 @@
const deploy = await callCli(['deploy', '--json', '--no-build'], {
cwd: builder.directory,
env: { NETLIFY_SITE_ID: context.siteId },
}).then((output: string) => JSON.parse(output))
}).then(parseDeploy)

await validateDeploy({ deploy, siteName: SITE_NAME, content })
})
Expand Down Expand Up @@ -214,9 +222,7 @@
if (shouldRunBuildBeforeDeploy) {
await callCli(['build'], options)
}
const deploy = await callCli(['deploy', '--json', '--no-build'], options).then((output: string) =>
JSON.parse(output),
)
const deploy = await callCli(['deploy', '--json', '--no-build'], options).then(parseDeploy)

// give edge functions manifest a couple ticks to propagate
await pause(500)
Expand Down Expand Up @@ -264,9 +270,7 @@
if (shouldRunBuildBeforeDeploy) {
await callCli(['build', '--cwd', pathPrefix], options)
}
const deploy = await callCli(['deploy', '--json', '--no-build', '--cwd', pathPrefix], options).then(
(output: string) => JSON.parse(output),
)
const deploy = await callCli(['deploy', '--json', '--no-build', '--cwd', pathPrefix], options).then(parseDeploy)

// give edge functions manifest a couple ticks to propagate
await pause(500)
Expand Down Expand Up @@ -312,9 +316,7 @@
if (shouldRunBuildBeforeDeploy) {
await callCli(['build'], options)
}
const deploy = await callCli(['deploy', '--json', '--no-build'], options).then((output: string) =>
JSON.parse(output),
)
const deploy = await callCli(['deploy', '--json', '--no-build'], options).then(parseDeploy)

// give edge functions manifest a couple ticks to propagate
await pause(500)
Expand Down Expand Up @@ -359,9 +361,7 @@
}

// skipping running build here, because it cleans up frameworks API directories
const deploy = await callCli(['deploy', '--json', '--no-build'], options).then((output: string) =>
JSON.parse(output),
)
const deploy = await callCli(['deploy', '--json', '--no-build'], options).then(parseDeploy)

// give edge functions manifest a couple ticks to propagate
await pause(500)
Expand Down Expand Up @@ -567,7 +567,7 @@
const deploy = await callCli(['deploy', '--json', '--no-build', '--dir', 'public'], {
cwd: builder.directory,
env: { NETLIFY_SITE_ID: context.siteId },
}).then((output: string) => JSON.parse(output))
}).then(parseDeploy)

await validateDeploy({ deploy, siteName: SITE_NAME, content })
expect(deploy).toHaveProperty('logs', `https://app.netlify.com/projects/${SITE_NAME}/deploys/${deploy.deploy_id}`)
Expand All @@ -594,7 +594,7 @@
const deploy = await callCli(['deploy', '--json', '--no-build', '--dir', 'public', '--prod'], {
cwd: builder.directory,
env: { NETLIFY_SITE_ID: context.siteId },
}).then((output: string) => JSON.parse(output))
}).then(parseDeploy)

await validateDeploy({ deploy, siteName: SITE_NAME, content })
expect(deploy).toHaveProperty('logs', `https://app.netlify.com/projects/${SITE_NAME}/deploys/${deploy.deploy_id}`)
Expand Down Expand Up @@ -720,7 +720,7 @@
const deploy = await callCli(['deploy', '--json', '--no-build'], {
cwd: builder.directory,
env: { NETLIFY_SITE_ID: context.siteId },
}).then((output: string) => JSON.parse(output))
}).then(parseDeploy)

await validateDeploy({ deploy, siteName: SITE_NAME, content: 'index' })
await validateContent({
Expand Down Expand Up @@ -765,7 +765,7 @@
const deploy = await callCli(['deploy', '--json', '--no-build'], {
cwd: builder.directory,
env: { NETLIFY_SITE_ID: context.siteId },
}).then((output: string) => JSON.parse(output))
}).then(parseDeploy)

await validateDeploy({ deploy, siteName: SITE_NAME, content: 'index' })
await validateContent({
Expand Down Expand Up @@ -800,7 +800,7 @@
const deploy = await callCli(['deploy', '--json', '--no-build'], {
cwd: builder.directory,
env: { NETLIFY_SITE_ID: context.siteId },
}).then((output: string) => JSON.parse(output))
}).then(parseDeploy)

await validateDeploy({ deploy, siteName: SITE_NAME, content: 'index' })
await validateContent({
Expand Down Expand Up @@ -860,14 +860,10 @@
})
.build()

const { deploy_url: deployUrl } = (await callCli(
['deploy', '--json'],
{
cwd: builder.directory,
env: { NETLIFY_SITE_ID: context.siteId },
},
true,
)) as unknown as Deploy
const { deploy_url: deployUrl } = await callCli(['deploy', '--json'], {
cwd: builder.directory,
env: { NETLIFY_SITE_ID: context.siteId },
}).then(parseDeploy)

const response = await fetch(`${deployUrl}/.netlify/functions/hello`)
t.expect(await response.text()).toEqual('Hello')
Expand Down Expand Up @@ -970,14 +966,10 @@
})
.build()

const { deploy_url: deployUrl } = (await callCli(
['deploy', '--json'],
{
cwd: builder.directory,
env: { NETLIFY_SITE_ID: context.siteId },
},
true,
)) as unknown as Deploy
const { deploy_url: deployUrl } = await callCli(['deploy', '--json'], {
cwd: builder.directory,
env: { NETLIFY_SITE_ID: context.siteId },
}).then(parseDeploy)

// Add retry logic for fetching deployed functions
const fetchWithRetry = async (url: string, maxRetries = 5) => {
Expand Down Expand Up @@ -1031,14 +1023,10 @@
})
.build()

const { deploy_url: deployUrl } = (await callCli(
['deploy', '--json'],
{
cwd: builder.directory,
env: { NETLIFY_SITE_ID: context.siteId },
},
true,
)) as unknown as Deploy
const { deploy_url: deployUrl } = await callCli(['deploy', '--json'], {
cwd: builder.directory,
env: { NETLIFY_SITE_ID: context.siteId },
}).then(parseDeploy)
const response = await fetch(`${deployUrl}/.netlify/functions/func-1`).then((res) => res.text())

t.expect(response).toEqual('Internal')
Expand Down Expand Up @@ -1089,23 +1077,18 @@
})
.build()

const deploy = (await callCli(
['deploy', '--json'],
{
cwd: builder.directory,
env: { NETLIFY_SITE_ID: context.siteId },
},
true,
)) as unknown as Deploy
const deploy = await callCli(['deploy', '--json'], {
cwd: builder.directory,
env: { NETLIFY_SITE_ID: context.siteId },
}).then(parseDeploy)

const fullDeploy = (await callCli(
const fullDeploy = await callCli(
['api', 'getDeploy', '--data', JSON.stringify({ deploy_id: deploy.deploy_id })],
{
cwd: builder.directory,
env: { NETLIFY_SITE_ID: context.siteId },
},
true,
)) as unknown as Deploy
).then(parseDeploy)
Comment on lines +1106 to +1112
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot Feb 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

rg -n "getDeploy" --type ts -C 3

Repository: netlify/cli

Length of output: 6885

🏁 Script executed:

rg -n "parseDeploy" --type ts -C 5

Repository: netlify/cli

Length of output: 27567

🏁 Script executed:

rg -n "getDeploy" --type ts -A 10 src/commands/api

Repository: netlify/cli

Length of output: 37

🏁 Script executed:

rg -n "getDeploy|api.*command" --type ts src/commands | head -50

Repository: netlify/cli

Length of output: 3804

🏁 Script executed:

cat -n src/commands/api/api.ts | head -100

Repository: netlify/cli

Length of output: 2224

🏁 Script executed:

cat -n tests/integration/commands/deploy/deploy.test.ts | head -100

Repository: netlify/cli

Length of output: 3564

🏁 Script executed:

rg -n "logJson|--json" src/commands/deploy/deploy.ts -B 3 -A 3 | head -50

Repository: netlify/cli

Length of output: 950

🏁 Script executed:

rg -n "jsonData\s*=" src/commands/deploy/deploy.ts -B 5 -A 15 | head -100

Repository: netlify/cli

Length of output: 37

🏁 Script executed:

sed -n '800,850p' src/commands/deploy/deploy.ts

Repository: netlify/cli

Length of output: 1965

🏁 Script executed:

rg -n "type JsonData|interface JsonData" src/commands/deploy/deploy.ts -B 2 -A 15

Repository: netlify/cli

Length of output: 419

🏁 Script executed:

sed -n '1094,1115p' tests/integration/commands/deploy/deploy.test.ts

Repository: netlify/cli

Length of output: 1079

🏁 Script executed:

rg -n "summary" tests/integration/commands/deploy/deploy.test.ts | head -20

Repository: netlify/cli

Length of output: 194

🏁 Script executed:

rg -n "const deploy = await callCli\(\['deploy'" tests/integration/commands/deploy/deploy.test.ts -A 3 | head -40

Repository: netlify/cli

Length of output: 1735

🏁 Script executed:

rg -n "@netlify/api" src/commands/api/api.ts

Repository: netlify/cli

Length of output: 112

🏁 Script executed:

sed -n '44,59p' tests/integration/commands/deploy/deploy.test.ts

Repository: netlify/cli

Length of output: 337

The Deploy type and parseDeploy are incompatible with the getDeploy API response structure.

The test's Deploy type includes a summary field (lines 45-49), and the test expects it in the API response (line 1102). However, the deploy command's JSON output (JsonData type in deploy.ts) doesn't include a summary field. The getDeploy API response from the Netlify API has a different structure than the deploy command's output, making the shared parseDeploy parser and type definition insufficient for both use cases.

Consider defining a separate type for the API response or transforming the API response to match the expected structure.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@tests/integration/commands/deploy/deploy.test.ts` around lines 1094 - 1100,
The test fails because the shared Deploy type and parseDeploy function expect a
summary field but the getDeploy API JSON (JsonData in deploy.ts) doesn't include
it; update the test to either (a) add a new API-specific type (e.g., ApiDeploy
or GetDeployResponse) and a small transformer that maps the Netlify API response
to the existing Deploy shape before calling parseDeploy, or (b) create a
separate parser (e.g., parseApiDeploy) that consumes the getDeploy output shape;
change references in the test from parseDeploy to the new transformer/parser and
ensure parseDeploy remains used only for the CLI deploy output shape.


const redirectsMessage = fullDeploy.summary.messages.find(({ title }) => title === '3 redirect rules processed')
t.expect(redirectsMessage).toBeDefined()
Expand Down Expand Up @@ -1174,14 +1157,10 @@
})
.build()

const { deploy_url: deployUrl } = (await callCli(
['deploy', '--json', '--no-build'],
{
cwd: builder.directory,
env: { NETLIFY_SITE_ID: context.siteId },
},
true,
)) as unknown as Deploy
const { deploy_url: deployUrl } = await callCli(['deploy', '--json', '--no-build'], {
cwd: builder.directory,
env: { NETLIFY_SITE_ID: context.siteId },
}).then(parseDeploy)
const response = await fetch(`${deployUrl}/.netlify/functions/bundled-function-1`).then((res) => res.text())
expect(response).toEqual('Pre-bundled')
})
Expand Down Expand Up @@ -1233,14 +1212,10 @@
})
.build()

const { deploy_url: deployUrl } = (await callCli(
['deploy', '--json', '--no-build', '--skip-functions-cache'],
{
cwd: builder.directory,
env: { NETLIFY_SITE_ID: context.siteId },
},
true,
)) as unknown as Deploy
const { deploy_url: deployUrl } = await callCli(['deploy', '--json', '--no-build', '--skip-functions-cache'], {
cwd: builder.directory,
env: { NETLIFY_SITE_ID: context.siteId },
}).then(parseDeploy)

const response = await fetch(`${deployUrl}/.netlify/functions/bundled-function-1`).then((res) => res.text())
t.expect(response).toEqual('Bundled at deployment')
Expand Down Expand Up @@ -1294,14 +1269,10 @@
})
.build()

const { deploy_url: deployUrl } = (await callCli(
['deploy', '--json', '--no-build'],
{
cwd: builder.directory,
env: { NETLIFY_SITE_ID: context.siteId },
},
true,
)) as unknown as { deploy_url: string }
const { deploy_url: deployUrl } = await callCli(['deploy', '--json', '--no-build'], {
cwd: builder.directory,
env: { NETLIFY_SITE_ID: context.siteId },
}).then(parseDeploy)

const response = await fetch(`${deployUrl}/.netlify/functions/bundled-function-1`).then((res) => res.text())
t.expect(response).toEqual('Bundled at deployment')
Expand Down Expand Up @@ -1353,14 +1324,10 @@
.build()

await execa.command('npm install', { cwd: builder.directory })
const { deploy_url: deployUrl } = (await callCli(
['deploy', '--json', '--no-build'],
{
cwd: builder.directory,
env: { NETLIFY_SITE_ID: context.siteId },
},
true,
)) as unknown as { deploy_url: string }
const { deploy_url: deployUrl } = await callCli(['deploy', '--json', '--no-build'], {
cwd: builder.directory,
env: { NETLIFY_SITE_ID: context.siteId },
}).then(parseDeploy)

const response = await fetch(`${deployUrl}/read-blob`).then((res) => res.text())
t.expect(response).toEqual('hello from the blob')
Expand All @@ -1374,14 +1341,10 @@
timeout: 300_000,
},
async ({ fixture }) => {
const { deploy_url: deployUrl } = (await callCli(
['deploy', '--json'],
{
cwd: fixture.directory,
env: { NETLIFY_SITE_ID: context.siteId },
},
true,
)) as unknown as { deploy_url: string }
const { deploy_url: deployUrl } = await callCli(['deploy', '--json'], {
cwd: fixture.directory,
env: { NETLIFY_SITE_ID: context.siteId },
}).then(parseDeploy)

const html = await fetch(deployUrl).then((res) => res.text())
const $ = load(html)
Expand Down Expand Up @@ -1422,7 +1385,7 @@
const deploy = await callCli(['deploy', '--json', '--no-build', '--dir', 'public', '--draft'], {
cwd: builder.directory,
env: { NETLIFY_SITE_ID: context.siteId },
}).then((output: string) => JSON.parse(output))
}).then(parseDeploy)

await validateDeploy({ deploy, siteName: SITE_NAME, content })
expect(deploy).toHaveProperty(
Expand Down Expand Up @@ -1486,7 +1449,7 @@
cwd: builder.directory,
env: { NETLIFY_SITE_ID: context.siteId },
},
).then((output: string) => JSON.parse(output))
).then(parseDeploy)

await validateDeploy({ deploy, siteName: SITE_NAME, content })
expect(deploy).toHaveProperty(
Expand Down Expand Up @@ -1515,7 +1478,7 @@
const deploy = await callCli(['deploy', '--json', '--no-build', '--dir', 'public', '--upload-source-zip'], {
cwd: builder.directory,
env: { NETLIFY_SITE_ID: context.siteId },
}).then((output: string) => JSON.parse(output))
}).then(parseDeploy)

await validateDeploy({ deploy, siteName: SITE_NAME, content })
expect(deploy).toHaveProperty('source_zip_filename')
Expand Down Expand Up @@ -1550,7 +1513,7 @@
const deploy = await callCli(['deploy', '--json', '--dir', 'public', '--upload-source-zip'], {
cwd: builder.directory,
env: { NETLIFY_SITE_ID: context.siteId },
}).then((output: string) => JSON.parse(output))
}).then(parseDeploy)

await validateDeploy({ deploy, siteName: SITE_NAME, content })
expect(deploy).toHaveProperty('source_zip_filename')
Expand Down Expand Up @@ -1614,9 +1577,7 @@
}

await callCli(['build'], options)
const deploy = (await callCli(['deploy', '--json', '--no-build'], options).then((output: string) =>
JSON.parse(output),
)) as Deploy
const deploy = await callCli(['deploy', '--json', '--no-build'], options).then(parseDeploy)

await pause(500)

Expand Down
Loading