fix: pre-set ECR Lambda pull policy to prevent concurrent SetRepositoryPolicy race condition (#8190)#8945
Open
abhishektang wants to merge 3 commits intoaws:developfrom
Open
Conversation
Implements issue aws#3888 to auto-create ECR repositories during packaging, matching sam deploy behavior. Enables package-once, deploy-many CI/CD workflows with managed ECR repos. - Add --resolve-image-repos CLI option to sam package - Call sync_ecr_stack() to auto-create managed ECR repositories - Add validation requiring --s3-bucket when flag is used - Add conflict detection with --image-repositories - Add unit tests for validation logic Closes aws#3888
…add integration test
…ryPolicy race condition (aws#8190) - Add _ensure_ecr_lambda_pull_policy() called before changeset creation to pre-set a stable SAMCliLambdaECRAccess SID on all referenced ECR repos. - Add _upsert_ecr_lambda_policy() to idempotently set/merge the policy, handling AccessDeniedException gracefully and retrying on concurrent SetRepositoryPolicy conflicts (ResourceInUseException). - Add ECRPolicySetError exception for unrecoverable policy failures. - Add 21 unit tests in test_ecr_policy_helpers.py covering all branches. - Patch _ensure_ecr_lambda_pull_policy in TestSamDeployCommand to isolate deploy-flow tests from ECR side-effects. - Fix test_updates_imageuri_when_pointing_to_local_archive: replace fragile CWD-relative file creation with pathlib.Path.is_file mock.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Fixes #8190.
When deploying a SAM application with multiple Lambda functions referencing the same or different ECR repositories, CloudFormation calls
ecr:SetRepositoryPolicyconcurrently — once per Lambda. Each call overwrites the existing policy rather than merging it, so whichever write lands last wins and earlier Lambdas lose access, resulting in a 403 on image pull.Solution
Before creating the changeset, SAM CLI now pre-sets a stable
SAMCliLambdaECRAccesspolicy SID on every ECR repository referenced by the deployment (viaImageRepository/ImageRepositories). Because the SID is deterministic, repeated calls are idempotent and safe.Three private helpers are added to
samcli/commands/deploy/deploy_context.py:_extract_ecr_repo_name_ensure_ecr_lambda_pull_policy_upsert_ecr_lambda_policyfor each_upsert_ecr_lambda_policySAMCliLambdaECRAccessstatement; retries on concurrentSetRepositoryPolicyconflicts; skips gracefully onAccessDeniedException_ensure_ecr_lambda_pull_policyis called fromDeployContext.run()immediately beforecreate_and_wait_for_changeset.Changes
samcli/commands/deploy/deploy_context.py— ECR policy helpers + call sitesamcli/commands/deploy/exceptions.py— newECRPolicySetError(UserException)tests/unit/commands/deploy/test_ecr_policy_helpers.py— 21 new unit tests covering all branches of the three helperstests/unit/commands/deploy/test_deploy_context.py— patch_ensure_ecr_lambda_pull_policyat class level to isolate existing deploy tests from ECR side-effectstests/unit/commands/_utils/test_template.py— fixtest_updates_imageuri_when_pointing_to_local_archive: replace fragile CWD-relative file creation (which caused aPermissionErroron macOS) with apathlib.Path.is_filemockTesting
Ruff and mypy also pass (mypy pre-existing errors are unrelated to this change).