RPM Support

1. create repo with a feed and sync it
2. remove 1 ISO and publish repo with iso_distributor
3. add iso_rsync_distributor and publish repo with it
4. sync repo to restore previously removed ISO
5, publish repo with iso_distributor
6. remove on the remote server contents and repos directories
7. publish repo with iso_rsync_distributor
8. observe on the remote host just 1 ISO published

There is likely not much value to the iso_distributor itself in being able to do an incremental publish. Hopefully there is a simple way to implement this feature just in the iso_rsync_distributor.

I'm preaching to the choir for sure, but just to re-cap: the reason incremental publish is useful in yum repos is because of the sheer number of units, and the number of file operations that must happen for each. For each rpm, and as we know there could be 10k+, pulp must:

- make a symlink in one place
- copy that symlink to another place
- load the unit from the DB
- render three templates
- write the resulting XML to three different files

So limiting that to just newly-added rpms helps performance a lot.

For ISOs, the normal use case involves an order of magnitude fewer files. And the metadata file (PULP_MANIFEST) is very small and fast to re-generate.

Not knowing much detail about how the rsync distributor works, perhaps in the incremental case, it can just rsync over the new files and symlinks, and replace the PULP_MANIFEST.

In any case, my main reason for writing all this is to suggest that we not invest a lot of effort in adding incremental publish to the iso_distributor itself unless we really have to, or if it really is the path of least resistance.

mhrivnak wrote:

There is likely not much value to the iso_distributor itself in being able to do an incremental publish. Hopefully there is a simple way to implement this feature just in the iso_rsync_distributor.

I'm preaching to the choir for sure, but just to re-cap: the reason incremental publish is useful in yum repos is because of the sheer number of units, and the number of file operations that must happen for each. For each rpm, and as we know there could be 10k+, pulp must:

- make a symlink in one place
- copy that symlink to another place
- load the unit from the DB
- render three templates
- write the resulting XML to three different files

So limiting that to just newly-added rpms helps performance a lot.

For ISOs, the normal use case involves an order of magnitude fewer files. And the metadata file (PULP_MANIFEST) is very small and fast to re-generate.

Not knowing much detail about how the rsync distributor works, perhaps in the incremental case, it can just rsync over the new files and symlinks, and replace the PULP_MANIFEST.

In any case, my main reason for writing all this is to suggest that we not invest a lot of effort in adding incremental publish to the iso_distributor itself unless we really have to, or if it really is the path of least resistance.

Hey Mike,

we were not even considering to add incremental publish to the iso_distributor(if that would not block the iso_rsync_dist). I checked out today in the morning that there is no need to do that, we can still implement iso_rsync_distributor incremental publish without first making iso_dist incremental.

https://github.com/PulpQE/pulp-smash/issues/454