Project

Profile

Help

Issue #5173

Should handle "429" response code appropriately e.g. from Quay.io

Added by twaugh 4 months ago. Updated about 2 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Sprint/Milestone:
-
Start date:
Due date:
Severity:
2. Medium
Version - Nectar:
Platform Release:
2.21.0
Blocks Release:
Target Release - Nectar:
OS:
Backwards Incompatible:
No
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:
Sprint 56

Description

When syncing from a registry which applies rate-limiting to requests, Pulp fails to handle the 429 response code ("Too many requests") and the sync task fails.

A better behaviour would be to back off when seeing 429, then retry.

In the particular instance of Quay.io, request limiting is applied per client IP address per second.

Related: DELIVERY-7214

2019-07-23 03:34:59 +0000 [ERROR   ] Pulp task [0d11937c-f11a-4ba5-b082-245febe02e1c] failed: PLP0000: Too Many Requests:
  Traceback (most recent call last):
    File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
      R = retval = fun(*args, **kwargs)
    File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 529, in __call__
      return super(Task, self).__call__(*args, **kwargs)
    File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 107, in __call__
      return super(PulpTask, self).__call__(*args, **kwargs)
    File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
      return self.run(*args, **kwargs)
    File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 769, in sync
      sync_report = sync_repo(transfer_repo, conduit, call_config)
    File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 737, in wrap_f
      return f(*args, **kwargs)
    File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/importer.py", line 85, in sync_repo
      return self.sync_step.process_lifecycle()
    File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 572, in process_lifecycle
      super(PluginStep, self).process_lifecycle()
    File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 163, in process_lifecycle
      step.process()
    File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 256, in process
      self._process_block()
    File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 303, in _process_block
      self.process_main()
    File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/importers/sync.py", line 224, in process_main
      available_tags = self.parent.index_repository.get_tags()
    File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/registry.py", line 473, in get_tags
      headers, tags = self._get_path(link)
    File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/registry.py", line 528, in _get_path
      self._raise_path_error(report)
    File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/registry.py", line 550, in _raise_path_error
      raise IOError(report.error_msg)
  IOError: Too Many Requests
clipboard-201909191620-w8vsd.png (2.34 KB) merlangsejahtera, 09/19/2019 01:21 PM clipboard-201909191620-w8vsd.png
clipboard-201909191622-lypni.png (4.23 KB) merlangsejahtera, 09/19/2019 01:23 PM clipboard-201909191622-lypni.png
250
250

Associated revisions

Revision d94087de View on GitHub
Added by twaugh 4 months ago

Always retry on 429 response, even without a retry-after header

Requests with 429 responses ('Too many requests') are only retried by
default if the retry-after header is present.

Instead, always retry these requests. This fixes interoperability with
Quay.io, which presents 429 requests without a retry-after header.

Fixes #5173.

Signed-off-by: Tim Waugh <>

History

#1 Updated by ipanova@redhat.com 4 months ago

  • Description updated (diff)

#2 Updated by ipanova@redhat.com 4 months ago

  • Tags Pulp 2 added

#3 Updated by ipanova@redhat.com 4 months ago

If hitting 429, check Retry-After response header that specifies how long to wait before retrying.
If registy does not say how long to wait use Exponential backoff http://docs.celeryproject.org/en/latest/userguide/tasks.html#Task.retry_backoff

#4 Updated by twaugh 4 months ago

I don't think Quay.io gives any Retry-After response header.

#5 Updated by ipanova@redhat.com 4 months ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to twaugh
  • Triaged changed from No to Yes

#7 Updated by ipanova@redhat.com 4 months ago

  • Project changed from Docker Support to Nectar
  • Status changed from ASSIGNED to MODIFIED

#8 Updated by ipanova@redhat.com 4 months ago

  • Sprint set to Sprint 56

#9 Updated by twaugh 3 months ago

#10 Updated by dalley 3 months ago

  • Platform Release set to 2.21.0

#11 Updated by phonedatapoints 3 months ago

Thanks for the great thread. Need more info regarding this. https://www.phonedatapoints.com.au/

#12 Updated by merlangsejahtera 2 months ago

250
250

Іt's really a cool and helpful piece of information. I am satisfied that you just shared this helpful information about response code appropriately with us. Please keep us up to date like this. Thank you for sharing.
https://www.indomascemerlangsejahtera.com/kratom-powder/

#13 Updated by dalley about 2 months ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Please register to edit this issue

Also available in: Atom PDF