Project

Profile

Help

Story #3180

As a user, I can mirror Galaxy roles to a local repo

Added by bmbouter almost 2 years ago. Updated 7 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Sprint/Milestone:
Start date:
Due date:
% Done:

0%

Platform Release:
Blocks Release:
Backwards Incompatible:
No
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:
Sprint 33

Description

This is designed to develop the code that will find roles on Galaxy and sync them.

1. Find roles through https://galaxy.ansible.com/api/v1/roles/
2. Sync associated repos using python-git. Save them as a bare repo into /var/lib/pulp/shared/pulp_ansible/<repo_name>
3. Create a content unit for each role as a unit. Note a unit contains a version number, so when syncing a repo with 10 versions, you'll get 10 content units for that role sync.


Checklist

Associated revisions

Revision d405c1bf View on GitHub
Added by daviddavis over 1 year ago

As a user, I can mirror Galaxy roles to a local repo

fixes #3180
https://pulp.plan.io/issues/3180

Revision d0eb9d12 View on GitHub
Added by bmbouter over 1 year ago

Metadata Fetching Improvements

  • New progress reporting has been added which shows the page-by-page
    fetching from Galaxy's role API.
  • The page fetching from Galaxy is now asyncrhonous with a
    parallelization of 20.
  • Adds retry support for 504 errors with logging when fetching Role
    metadata

https://pulp.plan.io/issues/3180
re #3180

History

#1 Updated by bmbouter almost 2 years ago

  • Blocked by Story #3179: As an user, I can view information for a role in Pulp added

#2 Updated by bmbouter almost 2 years ago

  • Blocks Story #3181: As a user, I have a roles API for published distribution base paths added

#3 Updated by daviddavis almost 2 years ago

  • Groomed changed from No to Yes
  • Sprint Candidate changed from No to Yes

#4 Updated by daviddavis almost 2 years ago

  • Sprint/Milestone changed from 0.1 Release to 52

#5 Updated by rchan almost 2 years ago

  • Sprint/Milestone changed from 52 to 53

#6 Updated by daviddavis almost 2 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to daviddavis

#7 Updated by daviddavis almost 2 years ago

I'm wondering how to best deal with the pagination of the Ansible Galaxy API. By default the page size is 10 which right now is about 1463 for all roles. That means we'll have to do 1463 requests to pull down info for all galaxy roles. I tried bumping up the page size to 1000 (ie https://galaxy.ansible.com/api/v1/roles/?page_size=1000) but then the time of a request slows down quite a bit (to 10-20s). So timeouts become a potential problem.

#8 Updated by bmbouter almost 2 years ago

What about somewhere in the middle, like 100? That would reduce the query count by an order of magnitude and still keep the times to < 10s. If not that, then I would be in favor of just doing a lot of requests with the normal pagination size of 10.

#9 Updated by bmbouter almost 2 years ago

  • Sprint/Milestone changed from 53 to 54

#10 Updated by sean797 over 1 year ago

We need to also be careful in case the Galaxy API has any rate limiting in place.

#11 Updated by daviddavis over 1 year ago

@sean797 great point.

#12 Updated by rchan over 1 year ago

  • Sprint/Milestone changed from 54 to 56

#13 Updated by bmbouter over 1 year ago

  • Sprint set to Sprint 33

#14 Updated by bmbouter over 1 year ago

  • Sprint/Milestone deleted (56)

#15 Updated by daviddavis over 1 year ago

  • Status changed from ASSIGNED to MODIFIED

#16 Updated by ttereshc over 1 year ago

  • Tags Pulp 3 added

#17 Updated by daviddavis over 1 year ago

  • Sprint/Milestone set to 0.1 Release

#18 Updated by bmbouter about 1 year ago

  • Blocks deleted (Story #3181: As a user, I have a roles API for published distribution base paths)

#19 Updated by bmbouter about 1 year ago

  • Blocked by deleted (Story #3179: As an user, I can view information for a role in Pulp)

#20 Updated by bmbouter about 1 year ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

#21 Updated by bmbouter 7 months ago

  • Tags deleted (Pulp 3)

Please register to edit this issue

Also available in: Atom PDF