Reaper not terminating pods in ImagePullBackOff state

### Jenkins and plugins versions report

<details>
  <summary>Environment</summary>

  
  ```text
Jenkins: 2.528.2
OS: Linux - 5.4.0-121-generic
Java: 21.0.9 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)
---
active-directory:2.41
ansicolor:1.0.6
antisamy-markup-formatter:173.v680e3a_b_69ff3
apache-httpcomponents-client-4-api:4.5.14-269.vfa_2321039a_83
apache-httpcomponents-client-5-api:5.5-170.v023de017ccd7
asm-api:9.9-185.va_6c6b_3348b_c3
atlassian-jira-software-cloud:2.0.15
audit-trail:436.vc0d1e79fc5a_3
authentication-tokens:1.144.v5ff4a_5ec5c33
aws-credentials:254.v978a_5e206a_d7
aws-java-sdk-ec2:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-minimal:1.12.780-480.v4a_0819121a_9e
aws-java-sdk2-core:2.33.4-62.vc1a_8df64b_4c9
aws-java-sdk2-ec2:2.33.4-62.vc1a_8df64b_4c9
blueocean:1.27.23
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.27.23
blueocean-commons:1.27.23
blueocean-config:1.27.23
blueocean-core-js:1.27.23
blueocean-dashboard:1.27.23
blueocean-display-url:2.4.4
blueocean-events:1.27.23
blueocean-git-pipeline:1.27.23
blueocean-github-pipeline:1.27.23
blueocean-i18n:1.27.23
blueocean-jira:1.27.23
blueocean-jwt:1.27.23
blueocean-personalization:1.27.23
blueocean-pipeline-api-impl:1.27.23
blueocean-pipeline-editor:1.27.23
blueocean-pipeline-scm-api:1.27.23
blueocean-rest:1.27.23
blueocean-rest-impl:1.27.23
blueocean-web:1.27.23
bootstrap5-api:5.3.8-895.v4d0d8e47fea_d
bouncycastle-api:2.30.1.82-277.v70ca_0b_877184
branch-api:2.1259.v45c101731c76
build-failure-analyzer:2.6.1
build-name-setter:2.5.1
build-symlink:1.1
build-timeout:1.38
build-token-root:151.va_e52fe3215fc
build-user-vars-plugin:195.v8c35f9d5c3dc
build-with-parameters:76.v9382db_f78962
caffeine-api:3.2.3-194.v31a_b_f7a_b_5a_81
checks-api:373.vfe7645102093
cloud-stats:377.vd8a_6c953e98e
cloudbees-bitbucket-branch-source:937.2.1
cloudbees-folder:6.1073.va_7888eb_dd514
command-launcher:123.v37cfdc92ef67
commons-collections4-api:4.5.0-8.va_d5448ef9011
commons-compress-api:1.28.0-1
commons-lang3-api:3.19.0-104.v12125f33a_255
commons-text-api:1.14.0-194.v804a_dc3a_1b_d8
conditional-buildstep:1.5.0
configuration-as-code:2006.v001a_2ca_6b_574
copyartifact:770.va_6c69e063442
credentials:1447.v4cb_b_539b_5321
credentials-binding:702.vfe613e537e88
customizable-header:232.v6459b_dd4f8a_9
dark-theme:574.va_19f05d54df5
dashboard-view:2.543.vca_9da_3cb_9c60
discard-old-build:1.08
display-url-api:2.217.va_6b_de84cc74b_
downstream-ext:73.vdda_16e6eb_0da
durable-task:635.v3733cef34b_5e
echarts-api:6.0.0-1165.vd1283a_3e37d4
eddsa-api:0.3.0.1-19.vc432d923e5ee
envinject:2.926.v69c9b_3896a_96
envinject-api:1.235.va_14c74f8f487
external-monitor-job:223.vb_fddcf42c9b_3
favorite:2.253.v9b_413168133b_
flatpickr-api:4.6.13-18.vcf5f6a_5b_8468
font-awesome-api:7.1.0-882.v1dfb_771e3278
git:5.8.0
git-client:6.4.0
github:1.45.0
github-api:1.330-492.v3941a_032db_2a_
github-branch-source:1917.v9ee8a_39b_3d0d
global-slack-notifier:1.5
gson-api:2.13.2-173.va_a_092315913c
handy-uri-templates-2-api:2.1.8-36.v85e4cb_234a_13
hidden-parameter:504.v610a_b_b_c05003
htmlpublisher:427
instance-identity:203.v15e81a_1b_7a_38
ionicons-api:94.vcc3065403257
jackson2-api:2.20.1-423.v13951f6b_6532
jakarta-activation-api:2.1.3-2
jakarta-mail-api:2.1.3-3
jakarta-xml-bind-api:4.0.5-3.v3d5b_a_73965b_9
javax-activation-api:1.2.0-8
javax-mail-api:1.6.2-11
jaxb:2.3.9-133.vb_ec76a_73f706
jdk-tool:83.v417146707a_3d
jenkins-design-language:1.27.23
jersey2-api:2.47-165.ve7809a_3e87e0
jira:3.19
jira-steps:2.0.180.vccfe35b_5910d
jjwt-api:0.11.5-120.v0268cf544b_89
job-dsl:1.93
jobConfigHistory:1356.ve360da_6c523a_
joda-time-api:2.14.0-149.v1c3ce991d1b_9
join:1.21
jqs-monitoring:37.vf50a_82a_0b_f32
jquery3-api:3.7.1-619.vdb_10e002501a_
jsch:0.2.16-95.v3eecb_55fa_b_78
json-api:20250517-173.v596efb_962a_31
json-path-api:2.10.0-202.va_9cc16c1e476
junit:1369.v15da_00283f06
kubernetes:4392.v19cea_fdb_5913
kubernetes-client-api:7.3.1-256.v788a_0b_787114
kubernetes-credentials:206.vde31a_b_0f71a_c
kubernetes-credentials-provider:1.299.v610fa_e76761a_
log-parser:2.5.0
logstash:2.5.0218.v0a_ff8fefc12b_
mailer:522.va_995fa_cfb_8b_d
mask-passwords:212.v4967a_a_73b_506
material-theme:0.5.2-rc100.6121925fe229
matrix-project:870.v9db_fcfc2f45b_
metrics:4.2.37-489.vb_6db_69b_ce753
mina-sshd-api-common:2.16.0-167.va_269f38cc024
mina-sshd-api-core:2.16.0-167.va_269f38cc024
monitoring:2.5.0
naginator:1.530.vb_6d120f250b_1
next-build-number:66.v4b_4762172d53
oic-auth:4.609.v9de140f63d01
okhttp-api:4.12.0-195.vc02552c04ffd
parameterized-trigger:873.v8b_e37dd8418f
pipeline-build-step:571.v08a_fffd4b_0ce
pipeline-github-lib:65.v203688e7727e
pipeline-graph-analysis:245.v88f03631a_b_21
pipeline-graph-view:661.v6003f4542123
pipeline-groovy-lib:776.vfee5327b_b_a_5b_
pipeline-input-step:540.v14b_100d754dd
pipeline-milestone-step:138.v78ca_76831a_43
pipeline-model-api:2.2277.v00573e73ddf1
pipeline-model-definition:2.2277.v00573e73ddf1
pipeline-model-extensions:2.2277.v00573e73ddf1
pipeline-rest-api:2.38
pipeline-stage-step:322.vecffa_99f371c
pipeline-stage-tags-metadata:2.2277.v00573e73ddf1
pipeline-stage-view:2.38
pipeline-utility-steps:2.20.0
plain-credentials:199.v9f8e1f741799
plugin-util-api:6.1192.v30fe6e2837ff
postbuildscript:3.4.1-695.vf6b_0b_8053979
prism-api:1.30.0-630.va_e19d17f83b_0
prometheus:819.v50953a_c560dd
pubsub-light:1.19
rebuild:338.va_0a_b_50e29397
role-strategy:840.v206ff7f7312e
run-condition:243.v3c3f94e46a_8b_
saml:4.590.v6170b_1013da_f
scm-api:712.v8846fdd68c88
script-security:1385.v7d2d9ec4d909
simple-theme-plugin:211.v5424a_5510e47
slack:795.v4b_9705b_e6d47
snakeyaml-api:2.5-143.v93b_c004f89de
solarized-theme:28.vfe25223f14fe
sse-gateway:1.28
ssh-agent:386.v36cc0c7582f0
ssh-credentials:361.vb_f6760818e8c
sshd:3.374.v19b_d59ce6610
startup-trigger-plugin:2.9.4
structs:353.v261ea_40a_80fb_
theme-manager:327.v780d7096ec29
timestamper:1.30
token-macro:477.vd4f0dc3cb_cf1
trilead-api:2.209.v0e69b_c43c245
uno-choice:2.8.8
validating-string-parameter:255.v31e2e6d0d273
variant:70.va_d9f17f859e0
view-job-filters:405.v9a_5fddc9230d
workflow-aggregator:608.v67378e9d3db_1
workflow-api:1384.vdc05a_48f535f
workflow-basic-steps:1098.v808b_fd7f8cf4
workflow-cps:4218.vff679a_5c0f3a_
workflow-cps-global-lib-http:2.54.0
workflow-durable-task-step:1464.v2d3f5c68f84c
workflow-job:1559.va_a_533730b_ea_d
workflow-multibranch:821.vc3b_4ea_780798
workflow-scm-step:466.va_d69e602552b_
workflow-step-api:710.v3e456cc85233
workflow-support:1004.veee3a_d67cdb_9
  ```

</details>


### What Operating System are you using (both controller, and any agents involved in the problem)?

Controller
`Linux jenkins-sandbox-0 5.4.0-121-generic #137-Ubuntu SMP Wed Jun 15 13:33:07 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux`

Agent
`Linux one-button-ansible-debugging-5333-wkxnl-brsm5-4cpcn 5.4.0-121-generic #137-Ubuntu SMP Wed Jun 15 13:33:07 UTC 2022 x86_64 Linux`

### Reproduction steps

1. Configure Jenkins Kubernetes plugin:
   - Set pod retention policy to `Never`
   - Set system property: `Reaper.backoffEventsLimit=1` (via `JENKINS_JAVA_OPTIONS=-DReaper.backoffEventsLimit=1`)

2. Enable Reaper debug logging:
   - Go to: Manage Jenkins → System Log → Add new log recorder
   - Logger: `org.csanchez.jenkins.plugins.kubernetes.pod.retention.Reaper`
   - Level: `FINE`

3. Create a pipeline job that uses `podTemplate` with a non-existent container image (e.g., `nonexistent-image:tag`)

4. Run the job

5. Observe:
   - Pod enters `ImagePullBackOff` state
   - No Reaper logs appear in the log recorder
   - Pod is not terminated by Reaper
   - Jenkins times out after 120 seconds (`slaveConnectTimeout`)
   - New pod is created automatically
   - Cycle repeats indefinitely

### Expected Results

When a pod enters `ImagePullBackOff` state, the Reaper should:
1. Detect the `ImagePullBackOff` condition
2. Terminate the pod after the configured number of backoff events (1 in our case)
3. Prevent infinite looping by cleaning up the failed pod
4. Log Reaper activity in debug logs``

### Actual Results

1. Pod enters `ImagePullBackOff` state
2. No Reaper logs appear (Reaper does not detect/act on the condition)
3. Pod is not terminated by Reaper
4. Jenkins times out after 120 seconds (`slaveConnectTimeout`)
5. Jenkins automatically creates a new pod
6. Cycle repeats indefinitely, creating multiple pods in a loop


**Build Logs:**
```
[Pipeline] echo
[Pipeline] podTemplate
[Pipeline] {
[Pipeline] node
Created Pod: c5-rancher jenkins-sandbox/one-button-ansible-debugging-5331-wfkxd-hqvbh-5bq1p
Still waiting to schedule task
'one-button-ansible-debugging-5331-wfkxd-hqvbh-5bq1p' is offline
ERROR: Failed to launch one-button-ansible-debugging-5331-wfkxd-hqvbh-5bq1p
io.fabric8.kubernetes.client.KubernetesClientTimeoutException: Timed out waiting for [120000] milliseconds for [Pod] with name:[one-button-ansible-debugging-5331-wfkxd-hqvbh-5bq1p] in namespace [jenkins-sandbox].
Created Pod: c5-rancher jenkins-sandbox/one-button-ansible-debugging-5331-wfkxd-hqvbh-3nz7l
ERROR: Failed to launch one-button-ansible-debugging-5331-wfkxd-hqvbh-3nz7l
io.fabric8.kubernetes.client.KubernetesClientTimeoutException: Timed out waiting for [120000] milliseconds for [Pod] with name:[one-button-ansible-debugging-5331-wfkxd-hqvbh-3nz7l] in namespace [jenkins-sandbox].
Created Pod: c5-rancher jenkins-sandbox/one-button-ansible-debugging-5331-wfkxd-hqvbh-h8jvr
... (repeats)
```


**Pod State:**
```
Container [ansible-config] waiting [ImagePullBackOff] Back-off pulling image "artifactory-prod.company.io/docker/selfservice-example-image:25.91.2000"
Pod [Pending][ContainersNotReady] containers with unready status: [ansible-config]
```


**Jenkins Logs (showing termination from timeout, not Reaper):**
```
Dec 07, 2025 3:14:46 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave terminate
Terminating Kubernetes instance for agent one-button-ansible-debugging-5331-wfkxd-hqvbh-3nz7l
Dec 07, 2025 3:14:46 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave deleteSlavePod
Terminated Kubernetes instance for agent jenkins-sandbox/one-button-ansible-debugging-5331-wfkxd-hqvbh-3nz7l
```


**Reaper Debug Logs:**
No logs appear in the Reaper log recorder, confirming Reaper is not detecting or acting on the `ImagePullBackOff` condition.

### Anything else?

_No response_

### Are you interested in contributing a fix?

_No response_

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reaper not terminating pods in ImagePullBackOff state #2772

Jenkins and plugins versions report

What Operating System are you using (both controller, and any agents involved in the problem)?

Reproduction steps

Expected Results

Actual Results

Anything else?

Are you interested in contributing a fix?

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Reaper not terminating pods in ImagePullBackOff state #2772

Description

Jenkins and plugins versions report

What Operating System are you using (both controller, and any agents involved in the problem)?

Reproduction steps

Expected Results

Actual Results

Anything else?

Are you interested in contributing a fix?

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions