Project

Profile

Help

Story #7965

closed

As an user, I can configure a remote to rate limit to req/sec

Added by fao89 almost 2 years ago. Updated over 1 year ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 88
Quarter:

Description

Problem

Servers like galaxy.ansible.com and cloud.redhat.com have rate limits, and they are pretty low. This causes sync's to fail because they do not correctly respond with 429 when rate limiting is in effect. Instead during rate limiting they respond with 500+ errors which causes the sync to fail.

Why won't concurrency resolve this?

The Galaxy API causes many, very small requests to occur, so even a rate limit of 1 can cause so many requests to occur in a short amount of time to trigger the rate limit situation.

Solution

Add a rate_limit option to the CollectionRemote which will be disabled by default. It will be set in terms of an integer number of requests per second.

This will include all requests including token fetches and actual data fetches, etc.

Implementation

The Model Field

rate_limit = models.IntegerField(null=True)

The Serializer

rate_limit = serializers.IntegerField(help_text=_("Limits total download rate in requests per second"))

Actions #1

Updated by bmbouter over 1 year ago

  • Subject changed from As an user I want sync machinery to deal with rate limiting to As an user, I can configure a remote to rate limit to req/min
  • Description updated (diff)
Actions #2

Updated by bmbouter over 1 year ago

  • Subject changed from As an user, I can configure a remote to rate limit to req/min to As an user, I can configure a remote to rate limit to req/sec
  • Description updated (diff)
Actions #3

Updated by fao89 over 1 year ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to fao89
  • Sprint set to Sprint 88
Actions #4

Updated by pulpbot over 1 year ago

  • Status changed from ASSIGNED to POST
Actions #5

Updated by Anonymous over 1 year ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

Applied in changeset commit:pulp_ansible|c2e353c746c2b478486acb78c852e48d2b2a82f5.

Actions #6

Updated by bmbouter over 1 year ago

  • Project changed from Ansible Plugin to Pulp
  • Status changed from MODIFIED to NEW
Actions #7

Updated by bmbouter over 1 year ago

  • Sprint/Milestone set to 3.10.0
Actions #8

Updated by pulpbot over 1 year ago

  • Status changed from NEW to POST

Added by Fabricio Aguiar over 1 year ago

Revision a70ab191

Add rate_limit option to Remote

https://pulp.plan.io/issues/7965 closes #7965

Actions #9

Updated by Anonymous over 1 year ago

  • Status changed from POST to MODIFIED
Actions #10

Updated by pulpbot over 1 year ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF