Pulp: Issueshttps://pulp.plan.io/https://pulp.plan.io/favicon.ico2021-12-09T18:59:35ZPulp
Planio RPM Support - Issue #9627 (MODIFIED): publish fails on MD5-checksummed repos, on FIPShttps://pulp.plan.io/issues/96272021-12-09T18:59:35Zggainey
<p>See associated BZ for details, reproducer</p> RPM Support - Test #9622 (MODIFIED): Add a repo signed using 'sha' as alias for 'sha1'https://pulp.plan.io/issues/96222021-12-08T19:00:00Zggainey
<p>'sha' support exists in the wild, is the same as 'sha1', and has broken us several times now, Let's make it possible to write tests for it.</p> RPM Support - Issue #9559 (MODIFIED): Unable to sync Fedora 35 repositoryhttps://pulp.plan.io/issues/95592021-11-08T02:09:29Zhyu
<p>Cloned from <a href="https://bugzilla.redhat.com/show_bug.cgi?id=2020473" class="external">https://bugzilla.redhat.com/show_bug.cgi?id=2020473</a></p>
<h2>Description of problem:
When publishing the Fedora 35 repository, Pulp failed with the following error:</h2>
<h2>celery.app.trace:ERROR: [56fc62c9] (24505-91520) Task pulp.server.managers.repo.publish.publish[56fc62c9-29b0-41bf-a820-ba3625ef4a00] raised unexpected: TemplateSyntaxError(u'Empty variable tag on line 548',)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) Traceback (most recent call last):
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
celery.app.trace:ERROR: [56fc62c9] (24505-91520) R = retval = fun(*args, **kwargs)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 688, in <strong>call</strong>
celery.app.trace:ERROR: [56fc62c9] (24505-91520) return super(Task, self).<strong>call</strong>(*args, **kwargs)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 110, in <strong>call</strong>
celery.app.trace:ERROR: [56fc62c9] (24505-91520) return super(PulpTask, self).<strong>call</strong>(*args, **kwargs)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in <strong>protected_call</strong>
celery.app.trace:ERROR: [56fc62c9] (24505-91520) return self.run(*args, **kwargs)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1142, in publish
celery.app.trace:ERROR: [56fc62c9] (24505-91520) result = check_publish(repo_obj, dist_id, dist_inst, transfer_repo, conduit, call_config)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1251, in check_publish
celery.app.trace:ERROR: [56fc62c9] (24505-91520) result = _do_publish(repo_obj, dist_id, dist_inst, transfer_repo, conduit, call_config)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1303, in _do_publish
celery.app.trace:ERROR: [56fc62c9] (24505-91520) publish_report = publish_repo(transfer_repo, conduit, call_config)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 901, in wrap_f
celery.app.trace:ERROR: [56fc62c9] (24505-91520) return f(*args, **kwargs)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/distributor.py", line 185, in publish_repo
celery.app.trace:ERROR: [56fc62c9] (24505-91520) return self._publisher.process_lifecycle()
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 573, in process_lifecycle
celery.app.trace:ERROR: [56fc62c9] (24505-91520) super(PluginStep, self).process_lifecycle()
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 164, in process_lifecycle
celery.app.trace:ERROR: [56fc62c9] (24505-91520) step.process()
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 240, in process
celery.app.trace:ERROR: [56fc62c9] (24505-91520) self._process_block(item=item)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 302, in _process_block
celery.app.trace:ERROR: [56fc62c9] (24505-91520) self.process_main(item=item)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/publish.py", line 500, in process_main
celery.app.trace:ERROR: [56fc62c9] (24505-91520) context.add_unit_metadata(unit)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/metadata/filelists.py", line 42, in add_unit_metadata
celery.app.trace:ERROR: [56fc62c9] (24505-91520) self.metadata_file_handle.write(unit.render_filelists(self.checksum_type))
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/db/models.py", line 869, in render_filelists
celery.app.trace:ERROR: [56fc62c9] (24505-91520) return self._render(metadata, context)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/db/models.py", line 884, in _render
celery.app.trace:ERROR: [56fc62c9] (24505-91520) t = Template(template)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/django/template/base.py", line 191, in <strong>init</strong>
celery.app.trace:ERROR: [56fc62c9] (24505-91520) self.nodelist = self.compile_nodelist()
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/django/template/base.py", line 230, in compile_nodelist
celery.app.trace:ERROR: [56fc62c9] (24505-91520) return parser.parse()
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/django/template/base.py", line 482, in parse
celery.app.trace:ERROR: [56fc62c9] (24505-91520) raise self.error(token, 'Empty variable tag on line %d' % token.lineno)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) TemplateSyntaxError: Empty variable tag on line 548</h2>
<p>This is because one of the rpm in the repo has as "{{" filename causing the Django template syntax error.</p>
<p>In "6dc8dae1be904c2613d5aa3667dacd2554d05077eb1ce4296b6edfa3c4db3a46-filelists.xml.gz"
/usr/lib/.build-id
/usr/lib/.build-id/42
/usr/lib/.build-id/42/5049f5a9e0d1ac25c21de795f28fe886bfa0ca
/usr/lib64/R/library/rlang
/usr/lib64/R/library/rlang/DESCRIPTION
/usr/lib64/R/library/rlang/INDEX
/usr/lib64/R/library/rlang/LICENSE
/usr/lib64/R/library/rlang/Meta
snip...
/usr/lib64/R/library/rlang/help/wref_key.html
/usr/lib64/R/library/rlang/help/wref_value.html
/usr/lib64/R/library/rlang/help/zap.html
/usr/lib64/R/library/rlang/help/zap_srcref.html
/usr/lib64/R/library/rlang/help/{{.html <================= TemplateSyntaxError: Empty variable tag on line 548\n",
/usr/lib64/R/library/rlang/help/{{}}.html <=================</p>
<p>$ rpm -qlp R-rlang-0.4.11-3.fc35.x86_64.rpm | grep "{{"
/usr/lib64/R/library/rlang/help/{{.html
/usr/lib64/R/library/rlang/help/{{}}.html</p>
<p>help]$ cat {{.html</p>
<p>help]$ cat {{}}.html</p>
<p>Steps to Reproduce:</p>
<ol>
<li>Create a custom repository call fedora 35 and add the following feed url</li>
</ol>
<p><a href="https://dl.fedoraproject.org/pub/fedora/linux/releases/35/Everything/x86_64/os/" class="external">https://dl.fedoraproject.org/pub/fedora/linux/releases/35/Everything/x86_64/os/</a></p>
<ol start="2">
<li>Sync the repo</li>
</ol>
<p>Actual results:
Failed to publish repo</p>
<p>Expected results:
Repo can be published successfully</p>
<p>Additional info:
Tested on Satellite 6.10 and the repository can be synced successfully without issue.</p> RPM Support - Issue #9553 (MODIFIED): Publishing repository with large metadata may consume high ...https://pulp.plan.io/issues/95532021-11-03T06:10:34Zhyu
<p>Pulp is consuming high memory when publishing RHEL 7 repository. This is happening when Pulp is calculating the checksum of the metadata. It reads the whole metadata file into memory at once to calculates the checksum. For example, the other.xml.gz (compressed ) for RHEL 7 repository is about 837MB size. Reading the entire file into memory will cause Pulp worker to consume more than 1GB for RAM.</p>
<p><a href="https://github.com/pulp/pulp/blob/2-master/server/pulp/plugins/util/metadata_writer.py#L99-L101" class="external">https://github.com/pulp/pulp/blob/2-master/server/pulp/plugins/util/metadata_writer.py#L99-L101</a>.</p>
<p>How to reproduce:</p>
<ol>
<li>Sync the RHEL 7 repository.</li>
<li>After that manually force full publish it and run the below command to observe the memory usage.</li>
</ol>
<p>watch -n 1 'ps -aux | grep resource_worker</p>
<ol start="3">
<li>The memory usage should be stable between 200MB to 350MB all the time, but will suddenly go up to about 1.1GB for about 3 seconds (around finalizing the publish rpms step) then back to 200MB+.</li>
</ol> RPM Support - Task #9435 (MODIFIED): Update ACS docs to use the CLIhttps://pulp.plan.io/issues/94352021-09-23T13:22:02Zppicka
<p>ssia</p> RPM Support - Story #9358 (MODIFIED): As a user I can use Alternate Content Sourceshttps://pulp.plan.io/issues/93582021-09-08T16:40:29Zppicka
<p>Add refresh endpoint with slightly adjust synchronize method and first stage to support ACS.</p> RPM Support - Issue #9021 (MODIFIED): No such file or directory: '/var/lib/pulp/sync_imports/test...https://pulp.plan.io/issues/90212021-07-06T18:41:26Ziballou
<p>Pulp should not error out when repomd.xml exists but repomd.xml.asc doesn't. This error occurred when syncing from the filesystem with the following patch applied to Pulpcore 3.14: <a href="https://github.com/pulp/pulpcore/pull/1463.patch" class="external">https://github.com/pulp/pulpcore/pull/1463.patch</a></p>
<p>Occurred on pulp-rpm (3.13.2)</p> RPM Support - Issue #8893 (MODIFIED): 3rd party repository sync fails with 'InvalidStringData: st...https://pulp.plan.io/issues/88932021-06-14T14:14:36Zggainey
<p>ModulemdDefaults are BSON-encoded before being saved into MongoDB because MongoDB (apparently) has restrictions on valid keys which are incompatible with the module data we need to store.</p>
<p><a href="https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/importers/yum/repomd/modules.py#L94-L95" class="external">https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/importers/yum/repomd/modules.py#L94-L95</a></p>
<p>...But the serialized BSON strings being created by the encoding function we are using are not always valid UTF-8... Presumably, it works the vast majority of the time, enough so that it wasn't noticed.</p>
<p>Only specific permutations of the input data appear to trigger this. If I delete key from the profiles dictionary, all of a sudden it's UTF-8 compatible. This is presumably why the problem pops into and out of existence.</p>
<p>At some point we attempt to save this string to MongoDB, and Mongo decides to apply a UTF-8 validation to it, and it blows up.</p> RPM Support - Issue #8890 (MODIFIED): Publishing a repository can take longer time to finish if m...https://pulp.plan.io/issues/88902021-06-14T06:00:43Zhyu
<p>Description of problem:
Pulp can take more than an hour to publish a repository when a large number of repositories have been synced from upstream and same errata are existed in the synced repositories, such as RHEL 7.x, RHEL 7 EUS and different aches.</p>
<p>The more repositories have the same errata the more "erratum pkglist" entries will be created in the mongodb which can cause the performance degradation.</p>
<p>For example:</p>
<blockquote>
<p>db.erratum_pkglists.find({errata_id: "RHSA-2018:2557"}).count()
387
db.erratum_pkglists.find({errata_id: "RHBA-2019:2180"}).count()
217</p>
</blockquote>
<p>When publishing errata, Pulp will use the above query to get all package lists of the errata. This will take long time to process when they are many package lists returned by the query and each package list is consist of many packages.</p>
<p>As we can see below, the "Publish Errata" step is very slow. 53 minutes has passed, it has only processed about 2073 errata. It will take more than an hour to finish.
...
{
"description": "Publishing Errata",
"details": "",
"error_details": [],
"items_total": 4789,
"num_failures": 0,
"num_processed": 2073,
"num_success": 2073,
"state": "IN_PROGRESS",
"step_id": "2f09190d-013a-4300-9445-eccb52ad94fe",
"step_type": "errata"
},
...
"start_time": "2021-06-09T12:41:13Z",</p>
<a name="date"></a>
<h1 >date<a href="#date" class="wiki-anchor">¶</a></h1>
<p>Wed Jun 9 13:32:41 UTC 2021</p> RPM Support - Test #7350 (MODIFIED): Test syncing from a repository located on a local diskhttps://pulp.plan.io/issues/73502020-08-19T15:40:07Zdalleydalley@redhat.com
<p>e.g. a repository with a file:// url such as file:///var/lib/pulp/sync_imports/test_repos/zoo/</p> RPM Support - Test #4952 (CLOSED - COMPLETE): Missing rpms in erratum pkglist when an erratum app...https://pulp.plan.io/issues/49522019-06-11T12:15:26Zbherring
<p>Description of problem:<br>
Pulp will returns wrong 'pkglist' if an erratum exists in multiple repositories and user syncs multiple of these repositories</p>
<p>For example:<br>
Erratum 'RHSA-2019:1235' exists in both 'rhel-7-server-rpms' repo and 'rhel-7-server-optional-debug-rpms' repo with the following 'pkglist' respectively</p>
<p>rhel-7-server-rpms:<br>
"ruby-libs-2.0.0.648-35.el7_6.x86_64.rpm",<br>
"ruby-libs-2.0.0.648-35.el7_6.i686.rpm",<br>
"ruby-2.0.0.648-35.el7_6.x86_64.rpm",<br>
"rubygem-bigdecimal-1.2.0-35.el7_6.x86_64.rpm",<br>
"rubygem-psych-2.0.0-35.el7_6.x86_64.rpm",<br>
"rubygem-rdoc-4.0.0-35.el7_6.noarch.rpm",<br>
"ruby-irb-2.0.0.648-35.el7_6.noarch.rpm",<br>
"rubygem-io-console-0.4.2-35.el7_6.x86_64.rpm",<br>
"rubygems-2.0.14.1-35.el7_6.noarch.rpm",<br>
"rubygem-json-1.7.7-35.el7_6.x86_64.rpm"</p>
<p>rhel-7-server-optional-debug-rpms:<br>
"ruby-debuginfo-2.0.0.648-35.el7_6.x86_64.rpm",</p>
<p>After syncing these 2 repos to the Satellite, Pulp will only list the 'ruby-debuginfo rpm' in the erratum 'pkglist'. Pulp will calculate the wrong applicability for the consumer because the erratum is listing only 1 rpm.</p>
<p>Test in "foreman-rake console":</p>
<ol>
<li>Before enabling debug repo, the erratum shows 10 packages:<br>
pp Katello::pulp_server.resources.repository.unit_search("046aa6b4-6369-4ef7-a3f7-3959250bd86f", {:type_ids => ['erratum'], filters: {unit: {"errata_id": "RHSA-2019:1235"}}})[0]['metadata']['pkglist'][0]['packages'].size<br>
10</li>
</ol>
<ol>
<li>After enabling and sync debug repo. the erratum shows only 1 package.<br>
pp Katello::pulp_server.resources.repository.unit_search("046aa6b4-6369-4ef7-a3f7-3959250bd86f", {:type_ids => ['erratum'], filters: {unit: {"errata_id": "RHSA-2019:1235"}}})[0]['metadata']['pkglist'][0]['packages'].size<br>
1</li>
</ol>
<p>pp Katello::pulp_server.resources.repository.unit_search("046aa6b4-6369-4ef7-a3f7-3959250bd86f", {:type_ids => ['erratum'], filters: {unit: {"errata_id": "RHSA-2019:1235"}}})[0]['metadata']['pkglist'][0]['packages'][0]['filename']<br>
"ruby-debuginfo-2.0.0.648-35.el7_6.x86_64.rpm"</p>
<p>exit</p>
<p>Steps to Reproduce:<br>
1. Ensure you have a RHEL 7 client with older ruby version. Lets say Client 'A'.<br>
2. Enable and sync 'rhel-7-server-rpms' repo<br>
3. Go to Web UI -> Hosts -> Content Hosts -> Client 'A' -> Errata tab -> Filter "id = RHSA-2019:1235" should show this erratum is applicable/installable.<br>
4. Enable and sync 'rhel-7-server-optional-debug-rpms' repo<br>
5. Now need to do something so that we can FORCE recalculate the applicability for Client 'A'</p>
<p>On client 'A' do:<br>
subscription-manager repos --enable="rhel-7-server-optional-debug-rpms"<br>
katello-enabled-repos-upload -f</p>
<p>6. Wait for the regenerate applicability task to finish<br>
7. Go to Web UI -> Hosts -> Content Hosts -> Client 'A' -> Errata tab -> Filter "id = RHSA-2019:1235" shows empty result.</p>
<p>Actual results:<br>
Errata RHSA-2019:1235 is not applicable to Client A</p>
<p>Expected results:<br>
Errata RHSA-2019:1235 is applicable to the Client A</p> RPM Support - Test #4727 (CLOSED - COMPLETE): rpm_rsync_distributor is broken for yum_repo_metada...https://pulp.plan.io/issues/47272019-04-23T19:41:52Zbherring
<p>rpm_rsync_distributor doesn't work at all for units of type yum_repo_metadata_file. It will publish broken symlinks to the remote host.</p>
<p>This is due to the following interaction between yum_distributor and rpm_rsync_distributor:</p>
<p>First, in yum_distributor, the metadata file is <strong>symlinked</strong> from repo publish dir to storage path:</p>
<pre><code class="python syntaxhl" data-language="python"><span class="k">class</span> <span class="nc">PublishMetadataStep</span><span class="p">(</span><span class="n">platform_steps</span><span class="p">.</span><span class="n">UnitModelPluginStep</span><span class="p">):</span>
<span class="p">...</span>
<span class="k">def</span> <span class="nf">process_main</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">item</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
<span class="s">"""
Copy the metadata file into place and add it to the repomd file.
:param item: The unit to process
:type item: pulp.server.db.model.ContentUnit
"""</span>
<span class="n">unit</span> <span class="o">=</span> <span class="n">item</span>
<span class="c1"># Copy the file to the location on disk where the published repo is built
</span> <span class="n">publish_location_relative_path</span> <span class="o">=</span> <span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">get_working_dir</span><span class="p">(),</span>
<span class="n">REPO_DATA_DIR_NAME</span><span class="p">)</span>
<span class="n">metadata_file_name</span> <span class="o">=</span> <span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">basename</span><span class="p">(</span><span class="n">unit</span><span class="p">.</span><span class="n">_storage_path</span><span class="p">)</span>
<span class="n">link_path</span> <span class="o">=</span> <span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="n">publish_location_relative_path</span><span class="p">,</span> <span class="n">metadata_file_name</span><span class="p">)</span>
<span class="n">plugin_misc</span><span class="p">.</span><span class="n">create_symlink</span><span class="p">(</span><span class="n">unit</span><span class="p">.</span><span class="n">_storage_path</span><span class="p">,</span> <span class="n">link_path</span><span class="p">)</span>
<span class="c1"># Add the proper relative reference to the metadata file to repomd
</span> <span class="bp">self</span><span class="p">.</span><span class="n">parent</span><span class="p">.</span><span class="n">repomd_file_context</span><span class="p">.</span>\
<span class="n">add_metadata_file_metadata</span><span class="p">(</span><span class="n">unit</span><span class="p">.</span><span class="n">data_type</span><span class="p">,</span> <span class="n">link_path</span><span class="p">)</span>
</code></pre>
<p>That results in a yum repo with a symlink as in following example:</p>
<pre><code>$ ls -l /var/lib/pulp/published/yum/master/yum_distributor/rhel-ha-for-rhel-7-for-system-z-rpms__7Server__s390x/1552949091.82/repodata/
total 200
-rw-r--r--. 1 apache apache 2087 Mar 18 22:44 29e7003fe4c25a3291e80c16006ca0edc315c8b89f930f8be9f435bb10c1000b-updateinfo.xml.gz
-rw-r--r--. 1 apache apache 160494 Mar 18 22:44 792d01f805670173c58c8c55b1b960576de6b72215bd61d845b077af254d7e14-other.xml.gz
-rw-r--r--. 1 apache apache 16643 Mar 18 22:44 79f6997620d82b1085fb9dc22499b81d1b4ccb8a6113372039273d0dbab5be9f-filelists.xml.gz
lrwxrwxrwx. 1 apache apache 153 Mar 18 22:44 85771b84-06dd-4e4d-9e88-1b1d5616b3cc -> /var/lib/pulp/content/units/yum_repo_metadata_file/6b/74066bca70043c0cec6f1c43bc973a4bb84aa9ff6579757b8b0a9be1bcf81b/85771b84-06dd-4e4d-9e88-1b1d5616b3cc
-rw-r--r--. 1 apache apache 124 Mar 18 22:44 a27718cc28ec6d71432e0ef3e6da544b7f9d93f6bb7d0a55aacd592d03144b70-comps.xml
-rw-r--r--. 1 apache apache 3575 Mar 18 22:44 a57cc25570ba03b0c7fc82ad4502fad14fb20b404f065801563bc58ea8340eda-primary.xml.gz
-rw-r--r--. 1 apache apache 2405 Mar 18 22:44 repomd.xml
</code></pre>
<p>Then, rpm_rsync_distributor performs rsync of repodata directory <strong>with links=True</strong>, meaning rsync --links, meaning "copy symlinks as symlinks":</p>
<pre><code class="python syntaxhl" data-language="python"> <span class="bp">self</span><span class="p">.</span><span class="n">add_child</span><span class="p">(</span><span class="n">RSyncPublishStep</span><span class="p">(</span><span class="s">"Rsync step (repodata)"</span><span class="p">,</span>
<span class="n">repodata_file_list</span><span class="p">,</span>
<span class="s">"%s/"</span> <span class="o">%</span> <span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="n">master_dir</span><span class="p">,</span> <span class="s">'repodata'</span><span class="p">),</span>
<span class="s">"%s/"</span> <span class="o">%</span> <span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="n">remote_repo_path</span><span class="p">,</span> <span class="s">'repodata'</span><span class="p">),</span>
<span class="n">exclude</span><span class="o">=</span><span class="p">[</span><span class="s">".*"</span><span class="p">,</span> <span class="s">"repodata.old"</span><span class="p">],</span>
<span class="n">config</span><span class="o">=</span><span class="n">config</span><span class="p">,</span> <span class="n">links</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span>
<span class="n">delete</span><span class="o">=</span><span class="bp">self</span><span class="p">.</span><span class="n">config</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">"delete"</span><span class="p">)))</span>
</code></pre>
<p>This indeed rsyncs the symlink onto the remote host exactly as it is:</p>
<pre><code>$ ls -l /mnt/redhat/cdn/content/dist/rhel/system-z/7/7Server/s390x/highavailability/os/repodata | grep 85771b84-06dd-4e4d-9e88-1b1d5616b3cc
lrwxrwxrwx. 1 vagrant vagrant 153 Mar 18 22:44 85771b84-06dd-4e4d-9e88-1b1d5616b3cc -> /var/lib/pulp/content/units/yum_repo_metadata_file/6b/74066bca70043c0cec6f1c43bc973a4bb84aa9ff6579757b8b0a9be1bcf81b/85771b84-06dd-4e4d-9e88-1b1d5616b3cc
</code></pre>
<p>That's a broken symlink to /var/lib/pulp, which is not expected to exist on the remote. Hence the file is not able to be accessed and the published repo is incomplete.</p>
<a name="Steps-to-reproduce"></a>
<h3 >Steps to reproduce<a href="#Steps-to-reproduce" class="wiki-anchor">¶</a></h3>
<p>- Create a repo with yum_distributor, rpm_rsync_distributor<br>
- Add any yum_repo_metadata_file to the repo<br>
- Publish the repo with yum_distributor and then rpm_rsync_distributor</p>
<a name="Expected-result"></a>
<h3 >Expected result<a href="#Expected-result" class="wiki-anchor">¶</a></h3>
<p>All yum_repo_metadata_file in the repo are accessible in the repo published to remote host via rpm_rsync_distributor</p>
<a name="Actual-result"></a>
<h3 >Actual result<a href="#Actual-result" class="wiki-anchor">¶</a></h3>
<p>All yum_repo_metadata_file in published repo are broken symlinks.</p> RPM Support - Test #4632 (CLOSED - COMPLETE): Publish YumMetadataFiles as files and not symlinkshttps://pulp.plan.io/issues/46322019-04-03T12:06:21Zbherring
<p>YumMetadata File should be copied, not symlinked into the publish location. (check the associated bz for more info.)</p> RPM Support - Test #4566 (CLOSED - COMPLETE): Improve performance of rpm duplicate nevra checkhttps://pulp.plan.io/issues/45662019-03-25T12:53:30Zbherring
<p>In current versions of Pulp 2.x, uploading an RPM to a repo will remove other RPMs with the same NEVRA.</p>
<p>Currently, we are upgrading from an old version of Pulp 2.7, and I've found that performance of import_uploaded_unit tasks for RPMs has regressed significantly. In Pulp 2.7, imports would usually take around 0.5s. In Pulp 2-master, imports to the same repos have taken from 8 to 130 seconds, depending on the size of the repo.</p>
<p>By debugging I've found most of the time is spent in this duplicate check (remove_unit_duplicate_nevra).</p>
<p>This issue is for improving the performance of remove_unit_duplicate_nevra to reduce the severity of the performance regression.</p> RPM Support - Test #4548 (CLOSED - COMPLETE): Ensure modular_errata.py includes cases for --recur...https://pulp.plan.io/issues/45482019-03-18T16:46:33Zbherring
<a name="Parent-Tickets"></a>
<h3 >Parent Tickets<a href="#Parent-Tickets" class="wiki-anchor">¶</a></h3>
<ul>
<li><a class="issue tracker-3 status-11 priority-6 priority-default closed parent" title="Story: Implement modularity content dependency solving (CLOSED - CURRENTRELEASE)" href="https://pulp.plan.io/issues/3740">#3740</a> && <a class="issue tracker-2 status-11 priority-6 priority-default closed" title="Task: Document how to use the newly added recursive_conservative (CLOSED - CURRENTRELEASE)" href="https://pulp.plan.io/issues/4371">#4371</a></li>
</ul>
<a name="Affected-Files"></a>
<h3 >Affected File(s)<a href="#Affected-Files" class="wiki-anchor">¶</a></h3>
<ul>
<li>test_modular_errata.py::ManageModularErrataTestCase::test_copy_errata</li>
</ul>
<a name="Update"></a>
<h3 >Update<a href="#Update" class="wiki-anchor">¶</a></h3>
<ul>
<li>Evaluate if the following are covered already and if any additional updates are required
<ul>
<li>Ensure test_modular_errata test includes --recursive and --recursive_conservative cases</li>
<li>Ensure that <strong>latest</strong> RPM copy test case is included</li>
</ul>
</li>
</ul>