Project

Profile

Help

Story #7942

closed

Task #7938: [EPIC] As a user, I have improved Galaxy V3 sync performance

As a sync user, I get no-op when possible

Added by bmbouter over 3 years ago. Updated about 3 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
Groomed:
Yes
Sprint Candidate:
No
Tags:
Sprint:
Sprint 90
Quarter:

Description

The first stage needs to keep all the existing code and fall back to it if the "new metadata endpoints are not available". They are only available on the Galaxy V3 API and if the server side implements them.

Model Changes

Add to the AnsibleRemote a new attribute named last_synced_metadata_time. Allow it to be None since in many cases, e.g. V2 sync and V3 sync's that don't yet support the new "metadata optimizations" won't be saving data into this attribute.

No migration is needed, since a None value will cause a sync to occur, and only subsequent sync's will be no-operation optimized.

Sync Workflow Changes

  1. Check at the /pulp_ansible/galaxy/<path:path>/api/v3/ endpoint and check the build time. If 404, fall back to the existing sync implementation.
  2. If the build time is the same as the one stored on the Repository object, exit the sync doing nothing (no-operation)
  3. Save the last_synced_metadata_time to the AnsibleRepository so that the next time sync occurs we can determine if the server side has changed.

Related issues

Has duplicate Ansible Plugin - Story #7672: Sync Repository Metadata at sync time and parse and save itCLOSED - DUPLICATE

Actions
Blocked by Ansible Plugin - Story #7939: As a user, the /v3/ endpoint returns the repository_version.created timeCLOSED - CURRENTRELEASEfao89

Actions
Blocked by Ansible Plugin - Story #7940: As a user, I have a new v3/metadata/collections/ endpoint returning all collections unpaginatedCLOSED - CURRENTRELEASEfao89

Actions
Blocked by Ansible Plugin - Story #7941: As a user, I have a new v3/metadata/collection_versions/ endpoint returning all collection_versions unpaginatedCLOSED - CURRENTRELEASE

Actions
Copied to Ansible Plugin - Story #8177: As a sync user, I use the efficient unpaginated metadata endpoints if they are availableCLOSED - CURRENTRELEASEfao89

Actions
Actions #1

Updated by bmbouter over 3 years ago

  • Blocked by Story #7939: As a user, the /v3/ endpoint returns the repository_version.created time added
Actions #2

Updated by bmbouter over 3 years ago

  • Blocked by Story #7940: As a user, I have a new v3/metadata/collections/ endpoint returning all collections unpaginated added
Actions #3

Updated by bmbouter over 3 years ago

  • Blocked by Story #7941: As a user, I have a new v3/metadata/collection_versions/ endpoint returning all collection_versions unpaginated added
Actions #4

Updated by fao89 about 3 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to fao89
  • Groomed changed from No to Yes
  • Sprint set to Sprint 89
Actions #5

Updated by pulpbot about 3 years ago

  • Status changed from ASSIGNED to POST
Actions #6

Updated by fao89 about 3 years ago

  • Copied to Story #8177: As a sync user, I use the efficient unpaginated metadata endpoints if they are available added
Actions #7

Updated by fao89 about 3 years ago

  • Subject changed from As a sync user, I use the efficient metadata endpoints if they are available and no-op when possible to As a sync user, I get no-op when possible
  • Description updated (diff)
Actions #8

Updated by rchan about 3 years ago

  • Sprint changed from Sprint 89 to Sprint 90
Actions #10

Updated by fao89 about 3 years ago

  • Sprint/Milestone set to 0.7.0

Added by Fabricio Aguiar about 3 years ago

Revision e09fee30 | View on GitHub

Improve sync performance with no-op when possible

https://pulp.plan.io/issues/7942 closes #7942

Actions #11

Updated by Anonymous about 3 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100
Actions #12

Updated by pulpbot about 3 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE
Actions #13

Updated by bmbouter about 3 years ago

  • Has duplicate Story #7672: Sync Repository Metadata at sync time and parse and save it added

Also available in: Atom PDF