Pulp

"versions": [ { "component": "core", "version": "3.16.0" }, { "component": "rpm", "version": "3.16.1" }, { "component": "python", "version": "3.5.2" }, { "component": "file", "version": "1.10.1" }, { "component": "deb", "version": "2.16.0" }, { "component": "container", "version": "2.9.0" }, { "component": "certguard", "version": "1.5.1" }, { "component": "ansible", "version": "0.10.1" } ],

I can confirm syncs were successful and that a version 1 of the repositories were created. My expectation is that once an artifact is created in pulp. that it is not downloaded again, regardless of whether other repositories/remotes/content types have content units that reference the same sha256 (artifact) hash using different metadata.

Currently I am noticing that every time a sync is done between rpm remotes/repositories and the sync optimization determines a a sync is warranted, the tasking redownloads every package into tmp then only updates those that changed. This is incredibly problematic and has made pulp completely unusable as a daily sync job against remote mirrors.

If someone could spend some time helping me get up to speed on the sync process I'd be happy to help troubleshoot the code further. I have built three new pulp deployments to test how it behaves with a brand new database with the same result. Every sync that isn't skipped results in downloading every package. Syncing a new remote/repository aimed at an identical mirror (for instance two different mirrors of Centos8 Appstream) results in downloading everything twice with no consideration of existing artifacts.

I've also written bash scripts to add all artifacts in the artifacts folder of a preexisting pulp deployment to a new pulp deployment as artifact objects. Then followed with adding a remote/repository that has all its referenced artifacts among the artifacts I added manually. This still results in the sync ignoring anything currently existing as an artifact in the database, downloading the entire remote repository to the tmp folder, and only then determining what content units and artifacts need to be added.

Any assistance someone can offer me would be greatly appreciated as this has made syncs of RPM repositories unviable.

Related Katello forum post: https://discourse.pulpproject.org/t/syncs-of-rpm-repositories-are-downloading-every-artifact-every-time-in-3-16/213

I believe we do have an issue here. I used this script below:

#!/bin/bash

set -ev

repo_name="repo$RANDOM"
remote_name="remote$RANDOM"
url="https://fixtures.pulpproject.org/file/PULP_MANIFEST"

pulp file remote create --name $remote_name --url $url
pulp file repository create --name $repo_name --remote $remote_name
pulp file repository sync --name $repo_name

repo_name="repo$RANDOM"
remote_name="remote$RANDOM"

pulp file remote create --name $remote_name --url $url
pulp file repository create --name $repo_name --remote $remote_name
pulp file repository sync --name $repo_name

And added this diff to pulpcore to make it easy to monitor the downloads occuring:

diff --git a/pulpcore/download/http.py b/pulpcore/download/http.py
index 725a9c8d2..68a7a4149 100644
--- a/pulpcore/download/http.py
+++ b/pulpcore/download/http.py
@@ -156,6 +156,7 @@ class HttpDownloader(BaseDownloader):
             kwargs (dict): This accepts the parameters of
                 :class:`~pulpcore.plugin.download.BaseDownloader`.
         """
+        log.warning(f"Downloading url: {url}")
         if session:
             self.session = session
             self._close_session_on_finalize = False

And when I run the script I see:

pulp [2b50c00cf8c54f2983b7b7cdd066a522]: pulpcore.download.http:WARNING: Downloading url: https://fixtures.pulpproject.org/file/PULP_MANIFEST
pulp [2b50c00cf8c54f2983b7b7cdd066a522]: pulpcore.download.http:WARNING: Downloading url: https://fixtures.pulpproject.org/file/1.iso
pulp [2b50c00cf8c54f2983b7b7cdd066a522]: pulpcore.download.http:WARNING: Downloading url: https://fixtures.pulpproject.org/file/2.iso
pulp [2b50c00cf8c54f2983b7b7cdd066a522]: pulpcore.download.http:WARNING: Downloading url: https://fixtures.pulpproject.org/file/3.iso
pulp [dcf3453f2444431ba47fb3d19524c5fd]: pulpcore.download.http:WARNING: Downloading url: https://fixtures.pulpproject.org/file/PULP_MANIFEST
pulp [dcf3453f2444431ba47fb3d19524c5fd]: pulpcore.download.http:WARNING: Downloading url: https://fixtures.pulpproject.org/file/2.iso
pulp [dcf3453f2444431ba47fb3d19524c5fd]: pulpcore.download.http:WARNING: Downloading url: https://fixtures.pulpproject.org/file/3.iso

I expected only to see:

pulp [2b50c00cf8c54f2983b7b7cdd066a522]: pulpcore.download.http:WARNING: Downloading url: https://fixtures.pulpproject.org/file/PULP_MANIFEST
pulp [2b50c00cf8c54f2983b7b7cdd066a522]: pulpcore.download.http:WARNING: Downloading url: https://fixtures.pulpproject.org/file/1.iso
pulp [2b50c00cf8c54f2983b7b7cdd066a522]: pulpcore.download.http:WARNING: Downloading url: https://fixtures.pulpproject.org/file/2.iso
pulp [2b50c00cf8c54f2983b7b7cdd066a522]: pulpcore.download.http:WARNING: Downloading url: https://fixtures.pulpproject.org/file/3.iso
pulp [dcf3453f2444431ba47fb3d19524c5fd]: pulpcore.download.http:WARNING: Downloading url: https://fixtures.pulpproject.org/file/PULP_MANIFEST

What's interesting is that in running it over and over (with a full reset in between) I never see 1.iso redownloaded, but I always see 2.iso and 3.iso.