Skip to content

Pin test-fts to FTS prod EL9 versions and add service-versioned build workflow#495

Merged
bari12 merged 3 commits intorucio:masterfrom
0xquark:feature/pin-test-fts-38LTS
Apr 9, 2026
Merged

Pin test-fts to FTS prod EL9 versions and add service-versioned build workflow#495
bari12 merged 3 commits intorucio:masterfrom
0xquark:feature/pin-test-fts-38LTS

Conversation

@0xquark
Copy link
Copy Markdown
Member

@0xquark 0xquark commented Mar 11, 2026

Fixes #494

Right now the test-fts container installs latest versions from the FTS repos without pinning versions, and our tags are mostly tied to containers/Rucio releases or latest. That makes it hard to reproduce issues (especially for LTS branches) and it hides which FTS version is actually inside a given image.
This change does two things:

  • In test-fts/Dockerfile, I now install FTS from the CERN EL9 repos using explicit version-release values, controlled by build args:
    FTS_SERVER_VERSION (default 3.14.4-2.el9)
    FTS_REST_VERSION (default 3.14.2-1.el9)
    FTS_MONITORING_VERSION (default 3.14.2-1.el9)

The install step uses these args to pin fts-server, fts-mysql, fts-rest-client, fts-rest-server, fts-monitoring and their selinux packages. This means a given build is fully determined by those three RPM versions, and the defaults are aligned with the latest prod entries from the EL9 repo (https://fts-repo.web.cern.ch/fts-repo/el9/x86_64/).

  • I added a new workflow build-test-fts-by-service-version.yml which is meant for LTS / controlled builds. It’s a workflow_dispatch job that takes the three FTS version-release strings as inputs, derives a tag of the form rucio/test-fts:fts3-prod-<server-version>-el9, checks Docker Hub to see if that tag already exists, and only builds/pushes the image if it doesn’t. The idea is that for each FTS prod bundle we want to support, we run this once, get a stable service-versioned tag, and then Rucio LTS branches can pin to that tag instead of latest.

We only ask for and pin the three core versions (fts-server, fts-rest, fts-monitoring) because that’s how FTS itself structures its release matrix: the production bundle is defined by those three versions, and the other RPMs like fts-mysql, fts-server-selinux, fts-rest-client, etc. track those core versions rather than introducing independent version axes. The official FTS version table (https://fts3-docs.web.cern.ch/fts3-docs/versions/) is essentially a matrix of these three components per channel (epel, prod, rc, devel). By taking fts_server_version, fts_rest_version and fts_monitoring_version as inputs and deriving all the other FTS packages from them, we match how the prod bundle is defined while avoiding unnecessary complexity and the risk of mixing incompatible RPM versions.

I intentionally did not touch the existing docker-auto-build.yml so the current release flow (tags based on containers repo releases) keeps working as before. The new workflow and pinned Dockerfile are an extra lane we can use when we care about strict FTS versioning and reproducibility as we do it for the LTS releases.

…d build workflow

Right now the test-fts container installs latest versions from the FTS repos without pinning versions, and our tags are mostly tied to containers/Rucio releases or latest. That makes it hard to reproduce issues (especially for LTS branches) and it hides which FTS version is actually inside a given image.
This change does two things:

- In test-fts/Dockerfile I now install FTS from the CERN EL9 repos using explicit version-release values, controlled by build args:
FTS_SERVER_VERSION (default 3.14.4-2.el9)
FTS_REST_VERSION (default 3.14.2-1.el9)
FTS_MONITORING_VERSION (default 3.14.2-1.el9)
The install step uses these args to pin fts-server, fts-mysql, fts-rest-client, fts-rest-server, fts-monitoring and their selinux packages. This means a given build is fully determined by those three RPM versions, and the defaults are aligned with the latest prod entries from the EL9 repo (https://fts-repo.web.cern.ch/fts-repo/el9/x86_64/).
- I added a new workflow build-test-fts-by-service-version.yml which is meant for LTS / controlled builds. It’s a workflow_dispatch job that takes the three FTS version-release strings as inputs, derives a tag of the form rucio/test-fts:fts3-prod-<server-version>-el9, checks Docker Hub to see if that tag already exists, and only builds/pushes the image if it doesn’t. The idea is that for each FTS prod bundle we want to support, we run this once, get a stable service-versioned tag, and then Rucio LTS branches can pin to that tag instead of latest.

We only ask for and pin the three core versions (fts-server, fts-rest, fts-monitoring) because that’s how FTS itself structures its release matrix: the production bundle is defined by those three versions, and the other RPMs like fts-mysql, fts-server-selinux, fts-rest-client, etc. track those core versions rather than introducing independent version axes. The official FTS version table (https://fts3-docs.web.cern.ch/fts3-docs/versions/) is essentially a matrix of these three components per channel (epel, prod, rc, devel). By taking fts_server_version, fts_rest_version and fts_monitoring_version as inputs and deriving all the other FTS packages from them, we match how the prod bundle is defined while avoiding unnecessary complexity and the risk of mixing incompatible RPM versions.

I intentionally did not touch the existing docker-auto-build.yml so the current release flow (tags based on containers repo releases) keeps working as before. The new workflow and pinned Dockerfile are an extra lane we can use when we care about strict FTS versioning and reproducibility.
@0xquark 0xquark requested a review from bari12 March 11, 2026 16:33
…e to EPEL 9 removal.

The CI run at https://github.com/rucio/containers/actions/runs/22910878721/job/66489786550 was failing with:

`nothing provides python3-django needed by fts-monitoring-3.14.2-1.el9.noarch from fts3-el9`

Why this happened: python3-django3 (Django 3.2 LTS) was removed from EPEL 9 after Django 3.2 reached end-of-life in April 2024. Since fts-monitoring has a hard dependency on python3-django3. it was written against Django 3.2 APIs and can't simply be upgraded to Django 4/5 without porting the monitoring webapp itself. The package was no longer resolvable from any configured repo.

What we considered: The FTS dependency repo has a quarantine directory at https://fts-repo.web.cern.ch/fts-repo/fts-depend/quarantine/ which contains two Django RPMs. One is python3-django-1.11.27-1.el9, which is Django 1.11 (EOL since 2020 with multiple known CVEs). The other is python36-django3-3.2.11-1.el7.cern.noarch, which is an el7 CERN build not compatible with our AlmaLinux 9 base at all. Neither was a viable option.

What we went with: We install python3-django3-3.2.25 directly from Fedora Koji (kojipkgs.fedoraproject.org), which is the official Fedora build infrastructure the same binaries that were previously served through EPEL. Version 3.2.25 is the latest 3.2 LTS patch release, includes all security backports for that branch, and is the exact version that was present in the environment when the CI was previously passing (https://github.com/rucio/containers/actions/runs/22587212634/job/65450431533). It's a noarch el9 package so it fits perfectly.

The build is succeeding now with this fix in place.
@0xquark
Copy link
Copy Markdown
Member Author

0xquark commented Mar 12, 2026

Update on failing test-fts tests on docker auto-build workflow:

The CI run at https://github.com/rucio/containers/actions/runs/22910878721/job/66489786550 was failing with: nothing provides python3-django needed by fts-monitoring-3.14.2-1.el9.noarch from fts3-el9

Why? python3-django3 (Django 3.2 LTS) was removed from EPEL 9 after Django 3.2 reached end-of-life in April 2024. Since fts-monitoring has a hard dependency on python3-django3. it was written against Django 3.2 APIs and can't simply be upgraded to Django 4/5 without porting the monitoring itself. The package was no longer resolvable from any configured repo.

Issue? Options?: The FTS dependency repo has a quarantine directory at https://fts-repo.web.cern.ch/fts-repo/fts-depend/quarantine/ which contains two Django RPMs.
One is python3-django-1.11.27-1.el9, which is Django 1.11 (EOL since 2020 with multiple known CVEs).
The other is python36-django3-3.2.11-1.el7.cern.noarch, which is an el7 CERN build not compatible with our AlmaLinux 9 base at all. Neither was a viable option.

Solution: We install python3-django3-3.2.25 directly from Fedora Koji (kojipkgs.fedoraproject.org), which is the official Fedora build infrastructure with the same binaries that were previously served through EPEL. Version 3.2.25 is the latest 3.2 LTS patch release, includes all security backports for that branch, and is the exact version that was present in the environment when the CI was previously passing (https://github.com/rucio/containers/actions/runs/22587212634/job/65450431533). It's a noarch el9 package so it fits perfectly.

The build is now succeeding now with this fix in place.

The test-fts image was previously hard-pinned to specific FTS_*_VERSION defaults, which causes the autobuild workflow to always build specific versioned fts releases.

This change switches to empty default version args and conditional installs: pinned packages when versions are provided, and unpinned/latest packages otherwise.

So the docker-auto-build.yml workflow will install and push the latest tagged images while the service versioned workflow will build and push the pinned version of fts.
Copy link
Copy Markdown
Member

@bari12 bari12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merging, should be transparent to current containers

@bari12 bari12 merged commit d72799a into rucio:master Apr 9, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add FTS-production based tags for rucio/test-fts (for LTS pinning)

2 participants