As a user, I can mirror Galaxy roles to a local repo
This is designed to develop the code that will find roles on Galaxy and sync them.
1. Find roles through https://galaxy.ansible.com/api/v1/roles/
2. Sync associated repos using python-git. Save them as a bare repo into
3. Create a content unit for each role as a unit. Note a unit contains a version number, so when syncing a repo with 10 versions, you'll get 10 content units for that role sync.
Metadata Fetching Improvements
New progress reporting has been added which shows the page-by-page fetching from Galaxy's role API.
The page fetching from Galaxy is now asyncrhonous with a parallelization of 20.
Adds retry support for 504 errors with logging when fetching Role metadata
#7 Updated by daviddavis almost 3 years ago
I'm wondering how to best deal with the pagination of the Ansible Galaxy API. By default the page size is 10 which right now is about 1463 for all roles. That means we'll have to do 1463 requests to pull down info for all galaxy roles. I tried bumping up the page size to 1000 (ie
https://galaxy.ansible.com/api/v1/roles/?page_size=1000) but then the time of a request slows down quite a bit (to 10-20s). So timeouts become a potential problem.
Please register to edit this issue