pulling caches from ECR sometimes take very long

[flokain]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug, and:

• The documentation does not mention anything about my problem
• There are no open or closed issues that are related to my problem

Description

I have noticed that every once in a while pulling cached layers from ecr is suddenly very slow (<0.1MB/s), even with previous layers in the same build successfully pulled much faster.

I am using a large ubuntu runner.
the ecr is eu-west-01
the build is using an authenticated service account to pull the images.

I could not find anything on the internet so I am posting here, hoping maybe someone is having the same issue?

Expected behaviour

consistently pulling images at fast pace

Actual behaviour

sometimes suddenly bandwith breaks when pulling from ecr

Repository URL

No response

Workflow run URL

No response

YAML workflow

- name: Build and push Docker image
        id: build-and-push
        uses: docker/build-push-action@0565240e2d4ab88bba5387d719585280857ece09 # v5.0.0
        with:
          context: .
          push: ${{ github.event_name != 'pull_request' }}
          platforms: "${{ matrix.os }}/${{ matrix.arch }}"
          tags: ${{ steps.login-ecr.outputs.registry }}/${{ env.IMAGE_NAME }}
          labels: ${{ steps.meta.outputs.labels }}
          # use the GitHub App token to be able to clone private repos during the build
          secrets: |
            github_token=${{ steps.app-token.outputs.token }}
            phraseapp_token=${{ secrets.PHRASEAPP_TOKEN }}
          # Cache to is set to the base branch if merged
          # otherwise to the current branch
          cache-to: type=registry,mode=max,image-manifest=true,oci-mediatypes=true,ref=${{ steps.login-ecr.outputs.registry }}/${{ env.IMAGE_NAME }}:${{ github.event.pull_request.merged && github.event.base_ref || env.BRANCH_NAME }}-${{matrix.os}}-${{ matrix.arch }}-cache
          # multiple cache-from entries to maximize cache hits
          # 1. try to find a cache for the current branch
          # 2. if this is a merge use the cache from the cache it is merged from
          # 3. worst case fallback to main/master branch cache
          cache-from: |
            type=registry,ref=${{ steps.login-ecr.outputs.registry }}/${{ env.IMAGE_NAME }}:${{ env.BRANCH_NAME }}-${{matrix.os}}-${{ matrix.arch }}-cache
            type=registry,ref=${{ steps.login-ecr.outputs.registry }}/${{ env.IMAGE_NAME }}:${{ github.event.pull_request.head.ref || github.event.base_ref || github.event.repository.default_branch }}-${{matrix.os}}-${{ matrix.arch }}-cache
            type=registry,ref=${{ steps.login-ecr.outputs.registry }}/${{ env.IMAGE_NAME }}:${{ github.event.repository.default_branch }}-${{matrix.os}}-${{ matrix.arch }}-cache
          outputs: type=image,push-by-digest=true,name-canonical=true,push=true

Workflow logs

No response

BuildKit logs

Additional info

No response