Project

Profile

Help

Issue #8901

Fails to pull images from AWS ecr to pulp with unknown blob error

Added by sandeepc1988 4 months ago. Updated 3 months ago.

Status:
CLOSED - WORKSFORME
Priority:
Normal
Assignee:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Platform Release:
OS:
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Quarter:

Description

We are trying to sync Amazon ECR container registry images to pulp

Followed pulp workflow:

  1. created the repo
  2. created remote --> which points to ECR registry, pull policy is "on_demand"
  3. Synced remote to repo
  4. created distribution.

All the API calls were successful.

Issue :

  1. when docker pulls API is executed we are getting an unknown blob error.

# docker pull <pulp_server>/quay.io/maprtech/csi-nfsplugin:1.0.1.2 1.0.1.2: Pulling from quay.io/maprtech/csi-nfsplugin d9e72d058dc5: Pulling fs layer cca21acb641a: Pulling fs layer 8f3bec2e0ffe: Downloading d4889023e8ff: Waiting 6fce7d7675ec: Waiting 342937d06bfb: Waiting fb8dcb3732a4: Waiting 2bc41d47dd50: Waiting c2a40f5494a9: Waiting unknown blob*

  1. When the repo is deleted, partially downloaded blobs are not deleted/ cleaned up. And when we recreate the repo, remote, and resync, it points to the same corrupted or partially downloaded blobs. This results in an unknown blob error when Docker pull command is executed.

  2. Amazon ECR token required to pull the images will expire in 11 hrs, Is there any efficient way to rotate the token in pulp? We used PUT remote API call to update the AWS ECR token and did a resync to the repo, the issue remains the same.

Let me know if you need more information.

History

#1 Updated by ipanova@redhat.com 4 months ago

When the repo is deleted, partially downloaded blobs are not deleted/ cleaned up. And when we recreate the repo, remote, and resync, it points to the same corrupted or partially downloaded blobs. This results in an unknown blob error when Docker pull command is executed.

Have you run orphan clean up after repo deletion? https://docs.pulpproject.org/pulpcore/restapi.html#tag/Orphans

Amazon ECR token required to pull the images will expire in 11 hrs, Is there any efficient way to rotate the token in pulp? We used PUT remote API call to update the AWS ECR token and did a resync to the repo, the issue remains the same.

Can you share the details of this call? Pulp container registry supports mirroring from registries that are compliant v2 api spec with Basic auth or jwt token based auth. Are you saying that username and password expires every 11 hours and so you updated those through the PUT remote api call?

#2 Updated by ipanova@redhat.com 4 months ago

Also, can you share the media_type for the unknown blob?

#3 Updated by ipanova@redhat.com 3 months ago

  • Status changed from NEW to CLOSED - WORKSFORME

please re-open if you still face the issue.

Please register to edit this issue

Also available in: Atom PDF