Story #3180
closed
As a user, I can mirror Galaxy roles to a local repo
0%
Description
This is designed to develop the code that will find roles on Galaxy and sync them.
1. Find roles through https://galaxy.ansible.com/api/v1/roles/
2. Sync associated repos using python-git. Save them as a bare repo into /var/lib/pulp/shared/pulp_ansible/<repo_name>
3. Create a content unit for each role as a unit. Note a unit contains a version number, so when syncing a repo with 10 versions, you'll get 10 content units for that role sync.
Updated by bmbouter about 7 years ago
- Blocked by Story #3179: As an user, I can view information for a role in Pulp added
Updated by bmbouter about 7 years ago
- Blocks Story #3181: As a user, I have a roles API for published distribution base paths added
Updated by daviddavis about 7 years ago
- Groomed changed from No to Yes
- Sprint Candidate changed from No to Yes
Updated by daviddavis about 7 years ago
- Sprint/Milestone changed from 0.1 Release to 52
Updated by daviddavis almost 7 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to daviddavis
Updated by daviddavis almost 7 years ago
I'm wondering how to best deal with the pagination of the Ansible Galaxy API. By default the page size is 10 which right now is about 1463 for all roles. That means we'll have to do 1463 requests to pull down info for all galaxy roles. I tried bumping up the page size to 1000 (ie https://galaxy.ansible.com/api/v1/roles/?page_size=1000) but then the time of a request slows down quite a bit (to 10-20s). So timeouts become a potential problem.
Updated by bmbouter almost 7 years ago
What about somewhere in the middle, like 100? That would reduce the query count by an order of magnitude and still keep the times to < 10s. If not that, then I would be in favor of just doing a lot of requests with the normal pagination size of 10.
Updated by sean797 almost 7 years ago
We need to also be careful in case the Galaxy API has any rate limiting in place.
Added by daviddavis almost 7 years ago
Updated by daviddavis almost 7 years ago
- Status changed from ASSIGNED to MODIFIED
Added by bmbouter over 6 years ago
Revision d0eb9d12 | View on GitHub
Metadata Fetching Improvements
-
New progress reporting has been added which shows the page-by-page fetching from Galaxy's role API.
-
The page fetching from Galaxy is now asyncrhonous with a parallelization of 20.
-
Adds retry support for 504 errors with logging when fetching Role metadata
Updated by bmbouter about 6 years ago
- Blocks deleted (Story #3181: As a user, I have a roles API for published distribution base paths)
Updated by bmbouter about 6 years ago
- Blocked by deleted (Story #3179: As an user, I can view information for a role in Pulp)
Updated by bmbouter about 6 years ago
- Status changed from MODIFIED to CLOSED - CURRENTRELEASE
.
As a user, I can mirror Galaxy roles to a local repo
fixes #3180 https://pulp.plan.io/issues/3180