Windows runners are consistently extremely slow compared to Linux and macOS

[myitcv]

Description

In the CUE project we are seeing incredibly slow run times on Windows 2022 runners.

For a recent example see https://github.com/cue-lang/cue-trybot/actions/runs/4477594562/.

Roughly speaking, here are the numbers we are seeing averaged out across 20-30 builds per day.

OS	actions/checkout	go test (cache hit)
ubuntu-22.04	2-3 secs	8 secs
macOS-11	5 secs	9 secs
windows-2022	15-20 secs	120+ secs

Notice that actions/checkout is consistently slower on Windows. Yes, there is a network access element to this, but it is minimal.

The go test comparison is the cleanest comparison. As indicated in the column heading, these are the timings for a full cache hit. i.e. no network access required, no rebuilds required, no tests actually running (because they hit the test cache). So the go test command is purely a function of disk access and CPU. For this command, Windows is consistently 15 times slower than Linux. macOS is, pleasingly, comparable with Linux.

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 18.04
• Ubuntu 20.04
• Ubuntu 22.04
• macOS 11
• macOS 12
• Windows Server 2019
• Windows Server 2022

Image version and build link

For windows-2019:

Current runner version: '2.303.0'
Operating System
  Microsoft Windows Server [2](https://github.com/myitcvscratch/slow-windows-actions/actions/runs/4477569194/jobs/7869257887#step:1:2)019
  10.0.1776[3](https://github.com/myitcvscratch/slow-windows-actions/actions/runs/4477569194/jobs/7869257887#step:1:3)
  Datacenter
Runner Image
  Image: windows-2019
  Version: 2023031[4](https://github.com/myitcvscratch/slow-windows-actions/actions/runs/4477569194/jobs/7869257887#step:1:4).1
  Included Software: https://github.com/actions/runner-images/blob/win19/20230314.1/images/win/Windows2019-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/win19%2F20230314.1
Runner Image Provisioner
  2.0.12[7](https://github.com/myitcvscratch/slow-windows-actions/actions/runs/4477569194/jobs/7869257887#step:1:8).1

For windows-2022:

Current runner version: '2.303.0'
Operating System
  Microsoft Windows Server [2](https://github.com/myitcvscratch/slow-windows-actions/actions/runs/4477569194/jobs/7869257680#step:1:2)022
  10.0.20[3](https://github.com/myitcvscratch/slow-windows-actions/actions/runs/4477569194/jobs/7869257680#step:1:3)[4](https://github.com/myitcvscratch/slow-windows-actions/actions/runs/4477569194/jobs/7869257680#step:1:4)8
  Datacenter
Runner Image
  Image: windows-2022
  Version: 20230314.1
  Included Software: https://github.com/actions/runner-images/blob/win22/20230314.1/images/win/Windows2022-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/win22%2F20230314.1
Runner Image Provisioner
  2.0.12[7](https://github.com/myitcvscratch/slow-windows-actions/actions/runs/4477569194/jobs/7869257680#step:1:8).1

Is it regression?

Unclear

Expected behavior

Windows runners to be comparable in terms of speed to Linux and macOS for actions/checkout and go test steps.

Actual behavior

Windows runners consistently taking 15 times as long as Linux and macOS for CPU and disk-intensive commands.

Repro steps

The CUE repo itself is quite involved. So as a proxy for something that is relatively CPU and disk intensive we have created a slimmed down repo using actions/checkout.

https://github.com/myitcvscratch/slow-windows-actions

See the most recent run for results:

https://github.com/myitcvscratch/slow-windows-actions/actions/runs/4477601338

Looking at averages of this setup across a number of runs we see similar figures for actions/checkout to those seen in our CUE setup:

Runner	actions/checkout
ubuntu-20.04	3 secs
ubuntu-22.04	3 secs
macos-11	5 secs
macos-12	5 secs
windows-2019	20 secs
windows-2022	20 secs

So whilst this doesn't include the go test step (because getting a warm cache is a tricky step to reproduce) the use of actions/checkout is a sufficiently good proxy to show the problem.

[ilia-shipitsin]

I suspect it might be Windows Defender in action.
can you try the following (in order to check whether Windows Defender disabling can help) ?

    - run: Set-MpPreference -DisableRealtimeMonitoring $true
      shell: powershell

[mvdan]

Here are many other people similarly seeing that actions/checkout is very slow on Windows: actions/checkout#1150

[myitcv]

@ilia-shipitsin

I suspect it might be Windows Defender in action.
can you try the following (in order to check whether Windows Defender disabling can help) ?

See commit myitcvscratch/slow-windows-actions@8b71de3 which resulted in run https://github.com/myitcvscratch/slow-windows-actions/actions/runs/4480846656/jobs/7876717612. It has basically no effect.

I also tried making this change in the CUE project in https://review.gerrithub.io/c/cue-lang/cue/+/551317. That resulted in https://github.com/cue-lang/cue-trybot/actions/runs/4476128503 which again showed no effect.

So assuming my testing is valid, turning off Windows Defender does not appear to have any effect in our case.

[myitcv]

Here are many other people similarly seeing that actions/checkout is very slow on Windows: actions/checkout#1150

Thanks, @mvdan.

Just to emphasise however that the use of actions/checkout in my example above is a proxy for the much bigger problem we are seeing in the go test step in the CUE project. It's very likely not the best proxy, but hopefully good enough.

The reason I flag this is that ultimately we will consider this issue "fixed" when the go test step is "fast" and not just an improvement in the actions/checkout step.

[ilia-shipitsin]

I've also tried to disable Windows Defender, no significant difference so far

https://github.com/ilia-shipitsin/slow-windows-actions/actions/runs/4483502669/jobs/7882868552

[ilia-shipitsin]

@myitcv , as for go test task performance degradation, do you see appropriate performance on standalone (not related to GH actions) Win 2019/2022 server ?

I mean, is degradation runner specific or platform specific

[myitcv]

@myitcv , as for go test task performance degradation, do you see appropriate performance on standalone (not related to GH actions) Win 2019/2022 server ?

I mean, is degradation runner specific or platform specific

We only have the numbers from GitHub actions workflow runs.

[ilia-shipitsin]

we need to narrow it, whether standalone Windows server behaves the same slow or not

[mikhailkoliada]

Duplicate of #5166