Add registration admission control for httpd#495
Draft
pablomh wants to merge 4 commits intotheforeman:masterfrom
Draft
Add registration admission control for httpd#495pablomh wants to merge 4 commits intotheforeman:masterfrom
pablomh wants to merge 4 commits intotheforeman:masterfrom
Conversation
Deploy event.conf template for the Apache MPM event module with configurable parameters via role defaults. This establishes foremanctl management of ServerLimit, MaxRequestWorkers, ListenBacklog, and related parameters. Defaults: ServerLimit=25, ThreadsPerChild=16. MaxRequestWorkers is only rendered when explicitly set, allowing Apache to derive it as ServerLimit * ThreadsPerChild by default. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add httpd ServerLimit and MaxRequestWorkers overrides to each tuning profile, matching the values from foreman-installer custom-hiera tuning sizes. All tuning profiles (medium, large, extra-large, extra-extra-large): ServerLimit=64, MaxRequestWorkers=1024 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Verify that event.conf is deployed with the expected directives and that httpd config syntax is valid after deployment. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Limit concurrent registration connections to Puma for /rhsm and /register endpoints via an Apache balancer pool. When httpd_registration_admission_max > 0, requests beyond that limit are queued by Apache instead of overwhelming Puma during bursts. Disabled by default (httpd_registration_admission_max: 0). Tuning profiles set values based on puma_workers * threads * 5: medium=300, large=600, extra-large=1200, extra-extra-large=2400. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add registration admission control for
/rhsmand/registerendpoints via an Apachemod_proxy_balancerpool. Whenhttpd_registration_admission_max > 0, requests beyond that limit are queued by Apache instead of overwhelming Puma during registration bursts.Builds on top of #483 (MPM event module configuration).
Problem
Under high-concurrency registration (e.g., 912+ simultaneous hosts), all requests arrive at Puma simultaneously, creating a deep backlog where tail hosts exceed subscription-manager's 180s timeout. Without admission control, pass rates drop sharply above 760 concurrent.
Solution
A
<Proxy balancer://foreman-registration>block with amax=limit on theBalancerMemberdirective limits how many registration requests reach Puma concurrently. Excess requests are held in Apache's event loop (non-blocking) until a slot opens.The balancer block is injected conditionally in both
foreman-ssl-vhost.conf.j2andforeman-vhost.conf.j2before the catch-allProxyPass /, so/rhsmand/registermatch first while all other traffic (Web UI, API, Pulp) is unaffected.Configuration
Disabled by default (
httpd_registration_admission_max: 0). Tuning profiles set values based onpuma_workers * threads * 5:Results (16-CPU large Satellite)
Note
A similar mechanism could be implemented for foreman-installer deployments via
puppet-foremanusing the existingforeman::config::apache::fragmentmechanism.How to test
foremanctl deploy --tuning largegrep foreman-registration /etc/httpd/conf.d/foreman-ssl.confhttpd -t