Story #2113
closedAs a user, I can perform an incremental publish using ISO rsync distributor
0%
Description
As a result of this story the ISO rsync distributor will perform incremental publishes when the following criteria are met:
- The repository has been published with the ISO rsync distributor before
- New units have been added to the repository(and published by predistributor) since the last publish by the ISO rsync distributor
- Units have NOT been removed from the repository since the last publish by the ISO rsync distributor
The user will also be able to specify a 'force_full' distributor config. When 'force_full' is true, the ISO rsync distributor will perform a full publish. The default value will be False. ( this will be done in platform)
Related issues
Updated by dkliban@redhat.com over 7 years ago
- Sprint Candidate changed from No to Yes
Updated by bmbouter over 7 years ago
- Sprint/Milestone set to 25
- Groomed changed from No to Yes
Updated by ipanova@redhat.com over 7 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to ipanova@redhat.com
Updated by ipanova@redhat.com over 7 years ago
1. create repo with a feed and sync it
2. remove 1 ISO and publish repo with iso_distributor
3. add iso_rsync_distributor and publish repo with it
4. sync repo to restore previously removed ISO
5, publish repo with iso_distributor
6. remove on the remote server contents and repos directories
7. publish repo with iso_rsync_distributor
8. observe on the remote host just 1 ISO published
Updated by mhrivnak over 7 years ago
There is likely not much value to the iso_distributor itself in being able to do an incremental publish. Hopefully there is a simple way to implement this feature just in the iso_rsync_distributor.
I'm preaching to the choir for sure, but just to re-cap: the reason incremental publish is useful in yum repos is because of the sheer number of units, and the number of file operations that must happen for each. For each rpm, and as we know there could be 10k+, pulp must:
- make a symlink in one place
- copy that symlink to another place
- load the unit from the DB
- render three templates
- write the resulting XML to three different files
So limiting that to just newly-added rpms helps performance a lot.
For ISOs, the normal use case involves an order of magnitude fewer files. And the metadata file (PULP_MANIFEST) is very small and fast to re-generate.
Not knowing much detail about how the rsync distributor works, perhaps in the incremental case, it can just rsync over the new files and symlinks, and replace the PULP_MANIFEST.
In any case, my main reason for writing all this is to suggest that we not invest a lot of effort in adding incremental publish to the iso_distributor itself unless we really have to, or if it really is the path of least resistance.
Updated by ipanova@redhat.com over 7 years ago
mhrivnak wrote:
There is likely not much value to the iso_distributor itself in being able to do an incremental publish. Hopefully there is a simple way to implement this feature just in the iso_rsync_distributor.
I'm preaching to the choir for sure, but just to re-cap: the reason incremental publish is useful in yum repos is because of the sheer number of units, and the number of file operations that must happen for each. For each rpm, and as we know there could be 10k+, pulp must:
- make a symlink in one place
- copy that symlink to another place
- load the unit from the DB
- render three templates
- write the resulting XML to three different filesSo limiting that to just newly-added rpms helps performance a lot.
For ISOs, the normal use case involves an order of magnitude fewer files. And the metadata file (PULP_MANIFEST) is very small and fast to re-generate.
Not knowing much detail about how the rsync distributor works, perhaps in the incremental case, it can just rsync over the new files and symlinks, and replace the PULP_MANIFEST.
In any case, my main reason for writing all this is to suggest that we not invest a lot of effort in adding incremental publish to the iso_distributor itself unless we really have to, or if it really is the path of least resistance.
Hey Mike,
we were not even considering to add incremental publish to the iso_distributor(if that would not block the iso_rsync_dist). I checked out today in the morning that there is no need to do that, we can still implement iso_rsync_distributor incremental publish without first making iso_dist incremental.
Added by ipanova@redhat.com over 7 years ago
Updated by ipanova@redhat.com over 7 years ago
- Related to Story #2114: As a user, I can perform incremental publish with Docker rsync distributor added
Updated by ipanova@redhat.com over 7 years ago
- Status changed from ASSIGNED to POST
Updated by bmbouter over 7 years ago
- Platform Release deleted (
2.11.0)
Since we're doing time based releases we should only set the target platform release when the issue is a MODIFIED.
Updated by ipanova@redhat.com over 7 years ago
- Status changed from POST to MODIFIED
- Platform Release set to 2.11.0
Updated by pthomas@redhat.com over 7 years ago
Updated by pcreech over 7 years ago
- Status changed from 5 to CLOSED - CURRENTRELEASE
As a user, I can perform an incremental publish using ISO rsync distributor
closes#2113 https://pulp.plan.io/issues/2113