Pulp: Issueshttps://pulp.plan.io/https://pulp.plan.io/favicon.ico2021-06-25T13:46:59ZPulp
Planio RPM Support - Test #8972 (CLOSED - COMPLETE): Add test for properly reading and writing the metadatahttps://pulp.plan.io/issues/89722021-06-25T13:46:59Zdalleydalley@redhat.com
<p>Verify that package metadata in == package metadata out</p> RPM Support - Test #8301 (CLOSED - COMPLETE): Pulp to Pulp sync test uses sha1 checksum, fails on...https://pulp.plan.io/issues/83012021-02-23T23:44:11Zdalleydalley@redhat.com
<p>I tried to run the tests and got some failures due to SHA1 checksums</p>
<p><a href="https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/tests/functional/api/test_pulp_to_pulp.py#L91" class="external">https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/tests/functional/api/test_pulp_to_pulp.py#L91</a></p>
<p>E HTTP response body: {"non_field_errors":["Checksum must be one of allowed types: ['sha224', 'sha256', 'sha384', 'sha512'].\nYou can adjust these with the 'ALLOWED_CONTENT_CHECKSUMS' setting."]}</p> Ansible Plugin - Test #5333 (CLOSED - COMPLETE): As a user, I can sync collections from another P...https://pulp.plan.io/issues/53332019-08-22T17:28:11Zdaviddavis
<p>In lieu of setting up two Pulp servers, I would just use two repositories.</p>
<p>Test steps:</p>
<p>1. Create a collections repo, remote, sync and publish<br>
2. Create a second repository and a remote which points to the distribution in step 1<br>
3. Sync the second repo and confirm it has the same contents as the first.</p> Pulp - Test #5242 (CLOSED - COMPLETE): Test copy using "additional_repos" to provide multiple sou...https://pulp.plan.io/issues/52422019-08-09T09:20:25Zdalleydalley@redhat.com
<p>See: <a href="https://pulp.plan.io/issues/5067" class="external">https://pulp.plan.io/issues/5067</a></p> RPM Support - Test #4727 (CLOSED - COMPLETE): rpm_rsync_distributor is broken for yum_repo_metada...https://pulp.plan.io/issues/47272019-04-23T19:41:52Zbherring
<p>rpm_rsync_distributor doesn't work at all for units of type yum_repo_metadata_file. It will publish broken symlinks to the remote host.</p>
<p>This is due to the following interaction between yum_distributor and rpm_rsync_distributor:</p>
<p>First, in yum_distributor, the metadata file is <strong>symlinked</strong> from repo publish dir to storage path:</p>
<pre><code class="python syntaxhl" data-language="python"><span class="k">class</span> <span class="nc">PublishMetadataStep</span><span class="p">(</span><span class="n">platform_steps</span><span class="p">.</span><span class="n">UnitModelPluginStep</span><span class="p">):</span>
<span class="p">...</span>
<span class="k">def</span> <span class="nf">process_main</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">item</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
<span class="s">"""
Copy the metadata file into place and add it to the repomd file.
:param item: The unit to process
:type item: pulp.server.db.model.ContentUnit
"""</span>
<span class="n">unit</span> <span class="o">=</span> <span class="n">item</span>
<span class="c1"># Copy the file to the location on disk where the published repo is built
</span> <span class="n">publish_location_relative_path</span> <span class="o">=</span> <span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">get_working_dir</span><span class="p">(),</span>
<span class="n">REPO_DATA_DIR_NAME</span><span class="p">)</span>
<span class="n">metadata_file_name</span> <span class="o">=</span> <span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">basename</span><span class="p">(</span><span class="n">unit</span><span class="p">.</span><span class="n">_storage_path</span><span class="p">)</span>
<span class="n">link_path</span> <span class="o">=</span> <span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="n">publish_location_relative_path</span><span class="p">,</span> <span class="n">metadata_file_name</span><span class="p">)</span>
<span class="n">plugin_misc</span><span class="p">.</span><span class="n">create_symlink</span><span class="p">(</span><span class="n">unit</span><span class="p">.</span><span class="n">_storage_path</span><span class="p">,</span> <span class="n">link_path</span><span class="p">)</span>
<span class="c1"># Add the proper relative reference to the metadata file to repomd
</span> <span class="bp">self</span><span class="p">.</span><span class="n">parent</span><span class="p">.</span><span class="n">repomd_file_context</span><span class="p">.</span>\
<span class="n">add_metadata_file_metadata</span><span class="p">(</span><span class="n">unit</span><span class="p">.</span><span class="n">data_type</span><span class="p">,</span> <span class="n">link_path</span><span class="p">)</span>
</code></pre>
<p>That results in a yum repo with a symlink as in following example:</p>
<pre><code>$ ls -l /var/lib/pulp/published/yum/master/yum_distributor/rhel-ha-for-rhel-7-for-system-z-rpms__7Server__s390x/1552949091.82/repodata/
total 200
-rw-r--r--. 1 apache apache 2087 Mar 18 22:44 29e7003fe4c25a3291e80c16006ca0edc315c8b89f930f8be9f435bb10c1000b-updateinfo.xml.gz
-rw-r--r--. 1 apache apache 160494 Mar 18 22:44 792d01f805670173c58c8c55b1b960576de6b72215bd61d845b077af254d7e14-other.xml.gz
-rw-r--r--. 1 apache apache 16643 Mar 18 22:44 79f6997620d82b1085fb9dc22499b81d1b4ccb8a6113372039273d0dbab5be9f-filelists.xml.gz
lrwxrwxrwx. 1 apache apache 153 Mar 18 22:44 85771b84-06dd-4e4d-9e88-1b1d5616b3cc -> /var/lib/pulp/content/units/yum_repo_metadata_file/6b/74066bca70043c0cec6f1c43bc973a4bb84aa9ff6579757b8b0a9be1bcf81b/85771b84-06dd-4e4d-9e88-1b1d5616b3cc
-rw-r--r--. 1 apache apache 124 Mar 18 22:44 a27718cc28ec6d71432e0ef3e6da544b7f9d93f6bb7d0a55aacd592d03144b70-comps.xml
-rw-r--r--. 1 apache apache 3575 Mar 18 22:44 a57cc25570ba03b0c7fc82ad4502fad14fb20b404f065801563bc58ea8340eda-primary.xml.gz
-rw-r--r--. 1 apache apache 2405 Mar 18 22:44 repomd.xml
</code></pre>
<p>Then, rpm_rsync_distributor performs rsync of repodata directory <strong>with links=True</strong>, meaning rsync --links, meaning "copy symlinks as symlinks":</p>
<pre><code class="python syntaxhl" data-language="python"> <span class="bp">self</span><span class="p">.</span><span class="n">add_child</span><span class="p">(</span><span class="n">RSyncPublishStep</span><span class="p">(</span><span class="s">"Rsync step (repodata)"</span><span class="p">,</span>
<span class="n">repodata_file_list</span><span class="p">,</span>
<span class="s">"%s/"</span> <span class="o">%</span> <span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="n">master_dir</span><span class="p">,</span> <span class="s">'repodata'</span><span class="p">),</span>
<span class="s">"%s/"</span> <span class="o">%</span> <span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="n">remote_repo_path</span><span class="p">,</span> <span class="s">'repodata'</span><span class="p">),</span>
<span class="n">exclude</span><span class="o">=</span><span class="p">[</span><span class="s">".*"</span><span class="p">,</span> <span class="s">"repodata.old"</span><span class="p">],</span>
<span class="n">config</span><span class="o">=</span><span class="n">config</span><span class="p">,</span> <span class="n">links</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span>
<span class="n">delete</span><span class="o">=</span><span class="bp">self</span><span class="p">.</span><span class="n">config</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">"delete"</span><span class="p">)))</span>
</code></pre>
<p>This indeed rsyncs the symlink onto the remote host exactly as it is:</p>
<pre><code>$ ls -l /mnt/redhat/cdn/content/dist/rhel/system-z/7/7Server/s390x/highavailability/os/repodata | grep 85771b84-06dd-4e4d-9e88-1b1d5616b3cc
lrwxrwxrwx. 1 vagrant vagrant 153 Mar 18 22:44 85771b84-06dd-4e4d-9e88-1b1d5616b3cc -> /var/lib/pulp/content/units/yum_repo_metadata_file/6b/74066bca70043c0cec6f1c43bc973a4bb84aa9ff6579757b8b0a9be1bcf81b/85771b84-06dd-4e4d-9e88-1b1d5616b3cc
</code></pre>
<p>That's a broken symlink to /var/lib/pulp, which is not expected to exist on the remote. Hence the file is not able to be accessed and the published repo is incomplete.</p>
<a name="Steps-to-reproduce"></a>
<h3 >Steps to reproduce<a href="#Steps-to-reproduce" class="wiki-anchor">¶</a></h3>
<p>- Create a repo with yum_distributor, rpm_rsync_distributor<br>
- Add any yum_repo_metadata_file to the repo<br>
- Publish the repo with yum_distributor and then rpm_rsync_distributor</p>
<a name="Expected-result"></a>
<h3 >Expected result<a href="#Expected-result" class="wiki-anchor">¶</a></h3>
<p>All yum_repo_metadata_file in the repo are accessible in the repo published to remote host via rpm_rsync_distributor</p>
<a name="Actual-result"></a>
<h3 >Actual result<a href="#Actual-result" class="wiki-anchor">¶</a></h3>
<p>All yum_repo_metadata_file in published repo are broken symlinks.</p> RPM Support - Test #4632 (CLOSED - COMPLETE): Publish YumMetadataFiles as files and not symlinkshttps://pulp.plan.io/issues/46322019-04-03T12:06:21Zbherring
<p>YumMetadata File should be copied, not symlinked into the publish location. (check the associated bz for more info.)</p> Pulp - Test #4628 (CLOSED - COMPLETE): Test pulp_streamer stream decodes responses, but sends the...https://pulp.plan.io/issues/46282019-04-02T17:28:31Zkersom
<p>Steps to reproduce it:</p>
<pre><code>pulp-admin rpm repo create --download-policy on_demand --feed https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-kickstart/ --repo-id ks
pulp-admin rpm repo sync run --repo-id ks
curl -sH 'Accept-encoding: gzip' -k -L https://localhost/pulp/repos/pulp/pulp/fixtures/rpm-kickstart/images/pxeboot/vmlinuz | gunzip -
gzip: stdin: not in gzip format
</code></pre> RPM Support - Test #4566 (CLOSED - COMPLETE): Improve performance of rpm duplicate nevra checkhttps://pulp.plan.io/issues/45662019-03-25T12:53:30Zbherring
<p>In current versions of Pulp 2.x, uploading an RPM to a repo will remove other RPMs with the same NEVRA.</p>
<p>Currently, we are upgrading from an old version of Pulp 2.7, and I've found that performance of import_uploaded_unit tasks for RPMs has regressed significantly. In Pulp 2.7, imports would usually take around 0.5s. In Pulp 2-master, imports to the same repos have taken from 8 to 130 seconds, depending on the size of the repo.</p>
<p>By debugging I've found most of the time is spent in this duplicate check (remove_unit_duplicate_nevra).</p>
<p>This issue is for improving the performance of remove_unit_duplicate_nevra to reduce the severity of the performance regression.</p> RPM Support - Test #4548 (CLOSED - COMPLETE): Ensure modular_errata.py includes cases for --recur...https://pulp.plan.io/issues/45482019-03-18T16:46:33Zbherring
<a name="Parent-Tickets"></a>
<h3 >Parent Tickets<a href="#Parent-Tickets" class="wiki-anchor">¶</a></h3>
<ul>
<li><a class="issue tracker-3 status-11 priority-6 priority-default closed parent" title="Story: Implement modularity content dependency solving (CLOSED - CURRENTRELEASE)" href="https://pulp.plan.io/issues/3740">#3740</a> && <a class="issue tracker-2 status-11 priority-6 priority-default closed" title="Task: Document how to use the newly added recursive_conservative (CLOSED - CURRENTRELEASE)" href="https://pulp.plan.io/issues/4371">#4371</a></li>
</ul>
<a name="Affected-Files"></a>
<h3 >Affected File(s)<a href="#Affected-Files" class="wiki-anchor">¶</a></h3>
<ul>
<li>test_modular_errata.py::ManageModularErrataTestCase::test_copy_errata</li>
</ul>
<a name="Update"></a>
<h3 >Update<a href="#Update" class="wiki-anchor">¶</a></h3>
<ul>
<li>Evaluate if the following are covered already and if any additional updates are required
<ul>
<li>Ensure test_modular_errata test includes --recursive and --recursive_conservative cases</li>
<li>Ensure that <strong>latest</strong> RPM copy test case is included</li>
</ul>
</li>
</ul> Pulp - Test #4359 (CLOSED - COMPLETE): 2.18.1 Testinghttps://pulp.plan.io/issues/43592019-01-29T16:02:43Zbherring
<a name="Notes"></a>
<h2 >Notes<a href="#Notes" class="wiki-anchor">¶</a></h2>
<p>Testing Coverage for 2.18.1 will consist of the following</p>
<ul>
<li>Migration/Upgrade Testing
<ul>
<li>2.17.0 --> 2.18.1
<ul>
<li>FIPS</li>
<li>Non-FIPS</li>
</ul>
</li>
</ul>
</li>
<li>2.18.1 Functional Testing Suite
<ul>
<li>FIPS</li>
<li>Non-FIPS</li>
</ul>
</li>
</ul> Pulp - Test #4357 (CLOSED - COMPLETE): Pulp 2.19 master pulled epel django instead of pulp 2 djangohttps://pulp.plan.io/issues/43572019-01-29T12:41:38Zbherring
<a name="Problem"></a>
<h2 >Problem<a href="#Problem" class="wiki-anchor">¶</a></h2>
<p>When running nightly rhel76-fips testing, there were 500 errors [0] on jenkins tests since 1/23 and issues in journalctl with django [1].</p>
<p>After talking with asmacdo and daviddavis, it appears this may be a django being pulled from EPEL and not from our Pulp 2 repos on build [2][3]</p>
<a name="Testing"></a>
<h2 >Testing<a href="#Testing" class="wiki-anchor">¶</a></h2>
<p>It was determined this was the result of EPEL django precedence over the pulp.repo version that was built. This issue has been seen before.</p>
<p>Forced installation of the <strong>python2-django-1.11.17-1</strong> from the fedorapeople repo [3] and restart of services (or with a hammer of <strong>shutdown now -r</strong>) resulted in a functioning pulp install.</p>
<pre><code>[root@rhel76-fips ~]# journalctl -f
-- Logs begin at Mon 2019-01-28 12:33:13 EST. --
Jan 28 12:34:38 rhel76-fips pulp[5046]: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Jan 28 12:34:38 rhel76-fips pulp[5046]: django.request:ERROR: (5046-27424) Internal Server Error: /pulp/api/v2/status/
Jan 28 12:34:38 rhel76-fips pulp[5046]: django.request:ERROR: (5046-27424) Traceback (most recent call last):
Jan 28 12:34:38 rhel76-fips pulp[5046]: django.request:ERROR: (5046-27424) File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 131, in get_response
Jan 28 12:34:38 rhel76-fips pulp[5046]: django.request:ERROR: (5046-27424) response = middleware_method(request, response)
Jan 28 12:34:38 rhel76-fips pulp[5046]: django.request:ERROR: (5046-27424) File "/usr/lib/python2.7/site-packages/django/middleware/http.py", line 23, in process_response
Jan 28 12:34:38 rhel76-fips pulp[5046]: django.request:ERROR: (5046-27424) set_response_etag(response)
Jan 28 12:34:38 rhel76-fips pulp[5046]: django.request:ERROR: (5046-27424) File "/usr/lib/python2.7/site-packages/django/utils/cache.py", line 109, in set_response_etag
Jan 28 12:34:38 rhel76-fips pulp[5046]: django.request:ERROR: (5046-27424) response['ETag'] = quote_etag(hashlib.md5(response.content).hexdigest())
Jan 28 12:34:38 rhel76-fips pulp[5046]: django.request:ERROR: (5046-27424) ValueError: error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips
</code></pre>
<a name="Actions"></a>
<h2 >Actions<a href="#Actions" class="wiki-anchor">¶</a></h2>
<p>Pulp-dev determined this issue as a low priority issue where documentation exercise may happen as a result of this issue.</p>
<p>For QE, this is a critical breakage of CI as the ansible install fails on FIPS installation and puts the box into a non-testable state. QE will be opening a related Urgent test tracker to fix the issue in the pulp-ci installer to fix mission critical work.</p>
<a name="References"></a>
<h2 >References<a href="#References" class="wiki-anchor">¶</a></h2>
<p>[0] - <a href="https://paste.fedoraproject.org/paste/bPRtYPa4ZCLShrZ7VUMqMQ" class="external">https://paste.fedoraproject.org/paste/bPRtYPa4ZCLShrZ7VUMqMQ</a><br>
[1] - <a href="https://pulp-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/Pulp%202%20-%20Master/job/pulp-2-master-dev-rhel7-fips/228/" class="external">https://pulp-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/Pulp%202%20-%20Master/job/pulp-2-master-dev-rhel7-fips/228/</a><br>
[2] - <a href="https://paste.fedoraproject.org/paste/6eHTS1Uc3NkC0hohvKe-TA" class="external">https://paste.fedoraproject.org/paste/6eHTS1Uc3NkC0hohvKe-TA</a> [from the affected 2.19a1-nightly box under test<br>
[3] - 2.11.17-1 <a href="https://repos.fedorapeople.org/repos/pulp/pulp/v2/testing/automation/2.19/stage/7Server/x86_64/" class="external">https://repos.fedorapeople.org/repos/pulp/pulp/v2/testing/automation/2.19/stage/7Server/x86_64/</a><br>
[4] - <a href="https://wiki.centos.org/PackageManagement/Yum/Priorities" class="external">https://wiki.centos.org/PackageManagement/Yum/Priorities</a></p> RPM Support - Test #4351 (CLOSED - COMPLETE): modules.yaml reference in repomd.xml does not use s...https://pulp.plan.io/issues/43512019-01-24T14:13:07Zbherring
<p>A report from upstream katello (<a href="https://projects.theforeman.org/issues/25529" class="external">https://projects.theforeman.org/issues/25529</a>) indicates that pulp generates a repomd.xml file that claims to use the checksum type that has been set on the publisher, but in fact is not using that:</p>
<p>In this example it purports to have used a sha1 checksum, but in fact its actually a sha256 checksum:</p>
<pre><code> <data type="modules">
<location href="repodata/824ffe238f202a0612ecbb2b0c0459dc289a7ef47adb1f26406453d41f476449-modules.yaml.gz"/>
<timestamp>1542811864</timestamp>
<size>33</size>
<checksum type="sha1">824ffe238f202a0612ecbb2b0c0459dc289a7ef47adb1f26406453d41f476449</checksum>
<open-size>0</open-size>
<open-checksum type="sha1">da39a3ee5e6b4b0d3255bfef95601890afd80709</open-checksum>
</data>
</code></pre>
<p>To reproduce:</p>
<p>1. create and sync a yum repository<br>
2. set a checksum type of sha1<br>
3. publish the repository</p> RPM Support - Test #4350 (CLOSED - COMPLETE): modules.yaml file is generated on repository with n...https://pulp.plan.io/issues/43502019-01-24T14:12:25Zbherring
<p>It seems that syncing and publishing a repository without any modularity info, still results in a modules.yaml being generated.</p>
<p>See: <a href="https://projects.theforeman.org/issues/25529" class="external">https://projects.theforeman.org/issues/25529</a> for more information</p> RPM Support - Test #4269 (CLOSED - COMPLETE): Recursive and conservative recursive copyhttps://pulp.plan.io/issues/42692018-12-14T17:36:04Zmilan
<a name="Description"></a>
<h2 >Description<a href="#Description" class="wiki-anchor">¶</a></h2>
<p>With the <a href="https://github.com/pulp/pulp_rpm/pull/1226" class="external">fix</a> for the Issue <a class="issue tracker-1 status-11 priority-6 priority-default closed" title="Issue: Regression Pulp 2.17.1: recursive copy of RPMs does not copy partially resolvable dependencies (CLOSED - CURRENTRELEASE)" href="https://pulp.plan.io/issues/4152">#4152</a>, the default behaviour of recursive copy changes back to the state before 2.17, rendering the test case <a href="https://github.com/PulpQE/Pulp-2-Tests/blob/c9277928c2788bced8d0ae3f630a46b6162fc4c9/pulp_2_tests/tests/rpm/api_v2/test_rich_weak_dependencies.py#L183" class="external">CopyRecursiveUnitsTestCase</a> broken.<br>
The behaviour specified in the <a href="https://github.com/PulpQE/pulp-smash/issues/1090" class="external">pulp smash issue 1090</a>, as requested in the Issue <a class="issue tracker-3 status-11 priority-6 priority-default closed" title="Story: Support more conservative dependency solving (CLOSED - CURRENTRELEASE)" href="https://pulp.plan.io/issues/2478">#2478</a>, is now optional and can be triggered by providing the <code>"recursive_conservative": true</code> config override instead of the default <code>"recursive": true</code>.<br>
With that setting, the test case <a href="https://github.com/PulpQE/Pulp-2-Tests/blob/c9277928c2788bced8d0ae3f630a46b6162fc4c9/pulp_2_tests/tests/rpm/api_v2/test_rich_weak_dependencies.py#L183" class="external">CopyRecursiveUnitsTestCase</a> should agin pass as is.</p>
<a name="Proposed-solution"></a>
<h2 >Proposed solution<a href="#Proposed-solution" class="wiki-anchor">¶</a></h2>
<p>Split the test case <a href="https://github.com/PulpQE/Pulp-2-Tests/blob/c9277928c2788bced8d0ae3f630a46b6162fc4c9/pulp_2_tests/tests/rpm/api_v2/test_rich_weak_dependencies.py#L183" class="external">CopyRecursiveUnitsTestCase</a> such that it asserts:</p>
<ul>
<li>a failure in the terms described in <a href="https://github.com/PulpQE/pulp-smash/issues/1090" class="external">pulp smash issue 1090</a> with the default <code>"recursive": true</code> config override setting</li>
<li>a success once the non-default <code>"recursive_conservative": true</code> config override setting is provided</li>
</ul> Pulp - Test #4258 (CLOSED - COMPLETE): Publishing incorrect branch head.https://pulp.plan.io/issues/42582018-12-11T13:33:31Zbherring
<a name="Notes"></a>
<h2 >Notes<a href="#Notes" class="wiki-anchor">¶</a></h2>
<ul>
<li>ostree smash test? ping jeff</li>
</ul>