Pulp

Problem¶

Servers like galaxy.ansible.com and cloud.redhat.com have rate limits, and they are pretty low. This causes sync's to fail because they do not correctly respond with 429 when rate limiting is in effect. Instead during rate limiting they respond with 500+ errors which causes the sync to fail.

Why won't concurrency resolve this?¶

The Galaxy API causes many, very small requests to occur, so even a rate limit of 1 can cause so many requests to occur in a short amount of time to trigger the rate limit situation.

Solution¶

Add a rate_limit option to the CollectionRemote which will be disabled by default. It will be set in terms of an integer number of requests per second.

This will include all requests including token fetches and actual data fetches, etc.

Implementation¶

The Model Field¶

rate_limit = models.IntegerField(null=True)

The Serializer¶

rate_limit = serializers.IntegerField(help_text=_("Limits total download rate in requests per second"))