Issue #2606
closedFix the ability to forge importer to mirror upstream repository on sync
Description
There is a need to be able to mirror Puppet repositories, including the optional ability to remove any local modules that no longer are present in the remote repo. This is called the "remove_missing" setting in the yum importer.
Currently Katello has a number of steps to do in order to achieve that if when the sync is not the first one:
- unassociate all the puppet modules from repo
- perform sync
- clean orphans
When multiple Katello syncs run in parallel, it leads to race condition between one Pulp sync task and the orphan removal task. So that sync notices nothing but during publish one can see the error like this:
pulp: pulp_puppet.plugins.distributors.publish:ERROR: (12867-29760) IOError: [Errno 2] No such file or directory: u'/var/lib/pulp/content/units/puppet_module/4c/052a353708dab8210898dd297f74cf9e4db4e896ae2bbca487719f02e62a3d/smals-mid_modules_wls_12-1.0.54.tar.gz'
Adding Fixing the remove_missing option will solve the issue because the described steps won't be needed anymore. It will have the side benefit of reducing the amount of work that needs to happen during a "smart proxy" sync.
This should be done for the forge importer case.
Fix remove_missing option and unnecessary downloads for forge importer
Fix remove_missing option for the forge and directory importers. Do not re-download units if they are in DB and on a disk for the forge importer.
closes #2606 https://pulp.plan.io/issues/2606