Project

Profile

Help

Story #7965

As an user, I can configure a remote to rate limit to req/sec

Added by fao89 8 months ago. Updated 6 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 88
Quarter:

Description

Problem

Servers like galaxy.ansible.com and cloud.redhat.com have rate limits, and they are pretty low. This causes sync's to fail because they do not correctly respond with 429 when rate limiting is in effect. Instead during rate limiting they respond with 500+ errors which causes the sync to fail.

Why won't concurrency resolve this?

The Galaxy API causes many, very small requests to occur, so even a rate limit of 1 can cause so many requests to occur in a short amount of time to trigger the rate limit situation.

Solution

Add a rate_limit option to the CollectionRemote which will be disabled by default. It will be set in terms of an integer number of requests per second.

This will include all requests including token fetches and actual data fetches, etc.

Implementation

The Model Field

rate_limit = models.IntegerField(null=True)

The Serializer

rate_limit = serializers.IntegerField(help_text=_("Limits total download rate in requests per second"))

Associated revisions

Revision a70ab191 View on GitHub
Added by Fabricio Aguiar 6 months ago

Add rate_limit option to Remote

https://pulp.plan.io/issues/7965 closes #7965

History

#1 Updated by bmbouter 7 months ago

  • Subject changed from As an user I want sync machinery to deal with rate limiting to As an user, I can configure a remote to rate limit to req/min
  • Description updated (diff)

#2 Updated by bmbouter 7 months ago

  • Subject changed from As an user, I can configure a remote to rate limit to req/min to As an user, I can configure a remote to rate limit to req/sec
  • Description updated (diff)

#3 Updated by fao89 7 months ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to fao89
  • Sprint set to Sprint 88

#4 Updated by pulpbot 7 months ago

  • Status changed from ASSIGNED to POST

#5 Updated by Anonymous 6 months ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

Applied in changeset commit:pulp_ansible|c2e353c746c2b478486acb78c852e48d2b2a82f5.

#6 Updated by bmbouter 6 months ago

  • Project changed from Ansible Plugin to Pulp
  • Status changed from MODIFIED to NEW

#7 Updated by bmbouter 6 months ago

  • Sprint/Milestone set to 3.10.0

#8 Updated by pulpbot 6 months ago

  • Status changed from NEW to POST

#9 Updated by Anonymous 6 months ago

  • Status changed from POST to MODIFIED

#10 Updated by pulpbot 6 months ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Please register to edit this issue

Also available in: Atom PDF