Project

Profile

Help

Story #1983

As a user, an importer config change or content removal will cause the next sync to be full

Added by mhrivnak about 5 years ago. Updated about 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
2.11.0
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Pulp 2
Sprint:
Sprint 6
Quarter:

Description

__A sync optimization exists that checks the remote metadata's revision number, and if it hasn't increased since the last successful sync, skips most of the current sync. It tries to accommodate several edge cases where the full sync should happen, but it's difficult and cumbersome to try catching them all. Here's what the decision looks like today:

        if 0 < metadata_files.revision <= previous_revision \
                and not self.config.override_config.get(importer_constants.KEY_FEED) \
                and previous_skip_set - current_skip_set == set() \
                and (self.download_deferred or not missing_units) \
                and not sync_due_to_unit_removal:

It would be valuable in this importer, and in others, to simplify some of the conditions down to these questions:

The sync should be full in case one of these answers are true:

1. Did the config change in any way since the last sync?
- A new field should be added to the Importer model 'last_updated'. As a consequence a migration should be written.
2. Has content been removed since the last sync?
3. If there was something specified in the override_config( because it changes current sync).
- A new field should be added to the Importer model 'last_override_config'. As a consequence a migration should be written.
4. Force-full option was specified.

Ideally, the platform should make it easy to answer those two questions. Then individual imports like this one can use that knowledge, combined with plugin-specific logic (like the repomd revision number) to decide if some or all of the sync can be skipped.

There is already code in the yum importer to answer question 2. It's only about 6 lines of code, but it would be easy and convenient to put that in the platform as a function, perhaps as a method on the Importer model.

Similar work was done here for distributors: https://pulp.plan.io/issues/1724


Checklist


Related issues

Related to RPM Support - Story #1982: As a user, I can force a full syncCLOSED - CURRENTRELEASE

<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Related to RPM Support - Issue #1971: '--retain-old-count' option not retroactiveCLOSED - WONTFIX<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Related to RPM Support - Issue #2027: remove_missing option does not work unless repository is updatedCLOSED - WONTFIX<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision 6868481c View on GitHub
Added by dalley almost 5 years ago

Add generic checks for whether a full sync should be forced to the repo controller. closes #1983

Revision 6868481c View on GitHub
Added by dalley almost 5 years ago

Add generic checks for whether a full sync should be forced to the repo controller. closes #1983

History

#1 Updated by mhrivnak about 5 years ago

  • Related to Story #1982: As a user, I can force a full sync added

#2 Updated by mhrivnak about 5 years ago

  • Related to Issue #1971: '--retain-old-count' option not retroactive added

#3 Updated by ttereshc about 5 years ago

  • Groomed changed from No to Yes

#4 Updated by ipanova@redhat.com about 5 years ago

  • Checklist item write needed migrations added
  • Checklist item implement a pre_save_signal for importer added
  • Description updated (diff)

#5 Updated by mhrivnak about 5 years ago

  • Sprint/Milestone set to 22

#6 Updated by ipanova@redhat.com about 5 years ago

  • Related to Issue #2027: remove_missing option does not work unless repository is updated added

#7 Updated by ipanova@redhat.com almost 5 years ago

  • Checklist item implement a pre_save_signal for importer set to Done

#8 Updated by ipanova@redhat.com almost 5 years ago

  • Checklist item implement a pre_save_signal for importer set to Not done

#9 Updated by mhrivnak almost 5 years ago

  • Sprint/Milestone changed from 22 to 23

#10 Updated by dalley almost 5 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dalley

#11 Updated by ipanova@redhat.com almost 5 years ago

  • Checklist item Put logic in the platform that can answer the two key questions set to Done
  • Checklist item Modify the skip decision logic to use that information set to Done
  • Checklist item Add release notes set to Done
  • Checklist item write needed migrations set to Done
  • Checklist item implement a pre_save_signal for importer set to Done

#12 Updated by mhrivnak almost 5 years ago

  • Sprint/Milestone changed from 23 to 24

#13 Updated by dalley almost 5 years ago

  • Status changed from ASSIGNED to POST
  • Platform Release set to 2.11.0

#15 Updated by dalley almost 5 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#16 Updated by semyers over 4 years ago

  • Status changed from MODIFIED to 5

#17 Updated by semyers over 4 years ago

  • Project changed from RPM Support to Pulp

#18 Updated by pcreech over 4 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE

#20 Updated by bmbouter over 3 years ago

  • Sprint set to Sprint 6

#21 Updated by bmbouter over 3 years ago

  • Sprint/Milestone deleted (24)

#22 Updated by bmbouter about 2 years ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF