Project

Profile

Help

Story #2113

closed

As a user, I can perform an incremental publish using ISO rsync distributor

Added by dkliban@redhat.com over 7 years ago. Updated almost 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Sprint/Milestone:
-
Start date:
Due date:
% Done:

0%

Estimated time:
Platform Release:
2.11.0
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Pulp 2
Sprint:
Sprint 7
Quarter:

Description

As a result of this story the ISO rsync distributor will perform incremental publishes when the following criteria are met:

- The repository has been published with the ISO rsync distributor before
- New units have been added to the repository(and published by predistributor) since the last publish by the ISO rsync distributor
- Units have NOT been removed from the repository since the last publish by the ISO rsync distributor

The user will also be able to specify a 'force_full' distributor config. When 'force_full' is true, the ISO rsync distributor will perform a full publish. The default value will be False. ( this will be done in platform)


Related issues

Related to Docker Support - Story #2114: As a user, I can perform incremental publish with Docker rsync distributorCLOSED - CURRENTRELEASEipanova@redhat.com

Actions
Actions #1

Updated by dkliban@redhat.com over 7 years ago

  • Sprint Candidate changed from No to Yes
Actions #2

Updated by dkliban@redhat.com over 7 years ago

  • Description updated (diff)
Actions #3

Updated by dkliban@redhat.com over 7 years ago

  • Description updated (diff)
Actions #4

Updated by dkliban@redhat.com over 7 years ago

  • Description updated (diff)
Actions #5

Updated by bmbouter over 7 years ago

  • Sprint/Milestone set to 25
  • Groomed changed from No to Yes
Actions #6

Updated by ipanova@redhat.com over 7 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ipanova@redhat.com
Actions #7

Updated by bmbouter over 7 years ago

  • Tags 2.11 added
Actions #8

Updated by ipanova@redhat.com over 7 years ago

  • Description updated (diff)
Actions #9

Updated by ipanova@redhat.com over 7 years ago

1. create repo with a feed and sync it
2. remove 1 ISO and publish repo with iso_distributor
3. add iso_rsync_distributor and publish repo with it
4. sync repo to restore previously removed ISO
5, publish repo with iso_distributor
6. remove on the remote server contents and repos directories
7. publish repo with iso_rsync_distributor
8. observe on the remote host just 1 ISO published

Actions #10

Updated by mhrivnak over 7 years ago

There is likely not much value to the iso_distributor itself in being able to do an incremental publish. Hopefully there is a simple way to implement this feature just in the iso_rsync_distributor.

I'm preaching to the choir for sure, but just to re-cap: the reason incremental publish is useful in yum repos is because of the sheer number of units, and the number of file operations that must happen for each. For each rpm, and as we know there could be 10k+, pulp must:

- make a symlink in one place
- copy that symlink to another place
- load the unit from the DB
- render three templates
- write the resulting XML to three different files

So limiting that to just newly-added rpms helps performance a lot.

For ISOs, the normal use case involves an order of magnitude fewer files. And the metadata file (PULP_MANIFEST) is very small and fast to re-generate.

Not knowing much detail about how the rsync distributor works, perhaps in the incremental case, it can just rsync over the new files and symlinks, and replace the PULP_MANIFEST.

In any case, my main reason for writing all this is to suggest that we not invest a lot of effort in adding incremental publish to the iso_distributor itself unless we really have to, or if it really is the path of least resistance.

Actions #11

Updated by ipanova@redhat.com over 7 years ago

mhrivnak wrote:

There is likely not much value to the iso_distributor itself in being able to do an incremental publish. Hopefully there is a simple way to implement this feature just in the iso_rsync_distributor.

I'm preaching to the choir for sure, but just to re-cap: the reason incremental publish is useful in yum repos is because of the sheer number of units, and the number of file operations that must happen for each. For each rpm, and as we know there could be 10k+, pulp must:

- make a symlink in one place
- copy that symlink to another place
- load the unit from the DB
- render three templates
- write the resulting XML to three different files

So limiting that to just newly-added rpms helps performance a lot.

For ISOs, the normal use case involves an order of magnitude fewer files. And the metadata file (PULP_MANIFEST) is very small and fast to re-generate.

Not knowing much detail about how the rsync distributor works, perhaps in the incremental case, it can just rsync over the new files and symlinks, and replace the PULP_MANIFEST.

In any case, my main reason for writing all this is to suggest that we not invest a lot of effort in adding incremental publish to the iso_distributor itself unless we really have to, or if it really is the path of least resistance.

Hey Mike,

we were not even considering to add incremental publish to the iso_distributor(if that would not block the iso_rsync_dist). I checked out today in the morning that there is no need to do that, we can still implement iso_rsync_distributor incremental publish without first making iso_dist incremental.

Added by ipanova@redhat.com over 7 years ago

Revision c6c6dd44 | View on GitHub

As a user, I can perform an incremental publish using ISO rsync distributor

closes#2113 https://pulp.plan.io/issues/2113

Actions #12

Updated by ipanova@redhat.com over 7 years ago

  • Related to Story #2114: As a user, I can perform incremental publish with Docker rsync distributor added
Actions #13

Updated by ipanova@redhat.com over 7 years ago

  • Status changed from ASSIGNED to POST
Actions #14

Updated by ipanova@redhat.com over 7 years ago

  • Platform Release set to 2.11.0
Actions #15

Updated by bmbouter over 7 years ago

  • Platform Release deleted (2.11.0)

Since we're doing time based releases we should only set the target platform release when the issue is a MODIFIED.

Actions #16

Updated by ipanova@redhat.com over 7 years ago

  • Status changed from POST to MODIFIED
  • Platform Release set to 2.11.0
Actions #17

Updated by semyers over 7 years ago

  • Status changed from MODIFIED to 5
Actions #19

Updated by pcreech over 7 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE
Actions #21

Updated by bmbouter over 6 years ago

  • Tags deleted (2.11)
Actions #22

Updated by bmbouter about 6 years ago

  • Sprint set to Sprint 7
Actions #23

Updated by bmbouter about 6 years ago

  • Sprint/Milestone deleted (25)
Actions #24

Updated by bmbouter almost 5 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF