Project

Profile

Help

Story #7965

closed

As an user, I can configure a remote to rate limit to req/sec

Added by fao89 over 3 years ago. Updated about 3 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 88
Quarter:

Description

Problem

Servers like galaxy.ansible.com and cloud.redhat.com have rate limits, and they are pretty low. This causes sync's to fail because they do not correctly respond with 429 when rate limiting is in effect. Instead during rate limiting they respond with 500+ errors which causes the sync to fail.

Why won't concurrency resolve this?

The Galaxy API causes many, very small requests to occur, so even a rate limit of 1 can cause so many requests to occur in a short amount of time to trigger the rate limit situation.

Solution

Add a rate_limit option to the CollectionRemote which will be disabled by default. It will be set in terms of an integer number of requests per second.

This will include all requests including token fetches and actual data fetches, etc.

Implementation

The Model Field

rate_limit = models.IntegerField(null=True)

The Serializer

rate_limit = serializers.IntegerField(help_text=_("Limits total download rate in requests per second"))

Also available in: Atom PDF