Pulp: Issueshttps://pulp.plan.io/https://pulp.plan.io/favicon.ico2019-06-11T12:15:26ZPulp
Planio RPM Support - Test #4952 (CLOSED - COMPLETE): Missing rpms in erratum pkglist when an erratum app...https://pulp.plan.io/issues/49522019-06-11T12:15:26Zbherring
<p>Description of problem:<br>
Pulp will returns wrong 'pkglist' if an erratum exists in multiple repositories and user syncs multiple of these repositories</p>
<p>For example:<br>
Erratum 'RHSA-2019:1235' exists in both 'rhel-7-server-rpms' repo and 'rhel-7-server-optional-debug-rpms' repo with the following 'pkglist' respectively</p>
<p>rhel-7-server-rpms:<br>
"ruby-libs-2.0.0.648-35.el7_6.x86_64.rpm",<br>
"ruby-libs-2.0.0.648-35.el7_6.i686.rpm",<br>
"ruby-2.0.0.648-35.el7_6.x86_64.rpm",<br>
"rubygem-bigdecimal-1.2.0-35.el7_6.x86_64.rpm",<br>
"rubygem-psych-2.0.0-35.el7_6.x86_64.rpm",<br>
"rubygem-rdoc-4.0.0-35.el7_6.noarch.rpm",<br>
"ruby-irb-2.0.0.648-35.el7_6.noarch.rpm",<br>
"rubygem-io-console-0.4.2-35.el7_6.x86_64.rpm",<br>
"rubygems-2.0.14.1-35.el7_6.noarch.rpm",<br>
"rubygem-json-1.7.7-35.el7_6.x86_64.rpm"</p>
<p>rhel-7-server-optional-debug-rpms:<br>
"ruby-debuginfo-2.0.0.648-35.el7_6.x86_64.rpm",</p>
<p>After syncing these 2 repos to the Satellite, Pulp will only list the 'ruby-debuginfo rpm' in the erratum 'pkglist'. Pulp will calculate the wrong applicability for the consumer because the erratum is listing only 1 rpm.</p>
<p>Test in "foreman-rake console":</p>
<ol>
<li>Before enabling debug repo, the erratum shows 10 packages:<br>
pp Katello::pulp_server.resources.repository.unit_search("046aa6b4-6369-4ef7-a3f7-3959250bd86f", {:type_ids => ['erratum'], filters: {unit: {"errata_id": "RHSA-2019:1235"}}})[0]['metadata']['pkglist'][0]['packages'].size<br>
10</li>
</ol>
<ol>
<li>After enabling and sync debug repo. the erratum shows only 1 package.<br>
pp Katello::pulp_server.resources.repository.unit_search("046aa6b4-6369-4ef7-a3f7-3959250bd86f", {:type_ids => ['erratum'], filters: {unit: {"errata_id": "RHSA-2019:1235"}}})[0]['metadata']['pkglist'][0]['packages'].size<br>
1</li>
</ol>
<p>pp Katello::pulp_server.resources.repository.unit_search("046aa6b4-6369-4ef7-a3f7-3959250bd86f", {:type_ids => ['erratum'], filters: {unit: {"errata_id": "RHSA-2019:1235"}}})[0]['metadata']['pkglist'][0]['packages'][0]['filename']<br>
"ruby-debuginfo-2.0.0.648-35.el7_6.x86_64.rpm"</p>
<p>exit</p>
<p>Steps to Reproduce:<br>
1. Ensure you have a RHEL 7 client with older ruby version. Lets say Client 'A'.<br>
2. Enable and sync 'rhel-7-server-rpms' repo<br>
3. Go to Web UI -> Hosts -> Content Hosts -> Client 'A' -> Errata tab -> Filter "id = RHSA-2019:1235" should show this erratum is applicable/installable.<br>
4. Enable and sync 'rhel-7-server-optional-debug-rpms' repo<br>
5. Now need to do something so that we can FORCE recalculate the applicability for Client 'A'</p>
<p>On client 'A' do:<br>
subscription-manager repos --enable="rhel-7-server-optional-debug-rpms"<br>
katello-enabled-repos-upload -f</p>
<p>6. Wait for the regenerate applicability task to finish<br>
7. Go to Web UI -> Hosts -> Content Hosts -> Client 'A' -> Errata tab -> Filter "id = RHSA-2019:1235" shows empty result.</p>
<p>Actual results:<br>
Errata RHSA-2019:1235 is not applicable to Client A</p>
<p>Expected results:<br>
Errata RHSA-2019:1235 is applicable to the Client A</p> Pulp - Test #4942 (CLOSED - COMPLETE): [EPIC] - Pulp2 - 2.20 https://pulp.plan.io/issues/49422019-06-10T12:33:32Zbherring
<a name="Information"></a>
<h2 >Information<a href="#Information" class="wiki-anchor">¶</a></h2>
<p>As new issues are added for release candidate to P2 for 2.20, compiling all of those issues as related issues to this parent issue.</p>
<p>This ticket allows the release shepherd to keep up with the status of each ticket as QE is heading to Dev Freeze to attempt to stay ahead.</p>
<a name="Current-Topics"></a>
<h3 >Current Topics<a href="#Current-Topics" class="wiki-anchor">¶</a></h3>
<p>These topics will be a subset of Dev release notes.</p>
<p>Just here for quick FYI on scanning:</p>
<ul>
<li>Modularity</li>
<li>Unit Copy Duplication</li>
<li>Issue with libsolv even in building: <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1718015" class="external">https://bugzilla.redhat.com/show_bug.cgi?id=1718015</a>
</li>
</ul> Python Support - Test #4748 (CLOSED - DUPLICATE): Improve Publications functional testshttps://pulp.plan.io/issues/47482019-04-30T12:13:07Zamacdona@redhat.comaustin@redhat.com
<p>The tests written during the feature change are a stopgap, and have some problems:<br>
<a href="https://github.com/pulp/pulp_python/pull/242/" class="external">https://github.com/pulp/pulp_python/pull/242/</a></p>
<p>The tests need to be broken up. Currently they are fragile and fully dependent on each other to pass. Ideally, the preparation should be moved to the setup of each class, but this could cause slow tests since a lot of steps need to be in place before a publication can be created. Please have a look at the utilities, they probably need to be refactored.</p> RPM Support - Test #4727 (CLOSED - COMPLETE): rpm_rsync_distributor is broken for yum_repo_metada...https://pulp.plan.io/issues/47272019-04-23T19:41:52Zbherring
<p>rpm_rsync_distributor doesn't work at all for units of type yum_repo_metadata_file. It will publish broken symlinks to the remote host.</p>
<p>This is due to the following interaction between yum_distributor and rpm_rsync_distributor:</p>
<p>First, in yum_distributor, the metadata file is <strong>symlinked</strong> from repo publish dir to storage path:</p>
<pre><code class="python syntaxhl" data-language="python"><span class="k">class</span> <span class="nc">PublishMetadataStep</span><span class="p">(</span><span class="n">platform_steps</span><span class="p">.</span><span class="n">UnitModelPluginStep</span><span class="p">):</span>
<span class="p">...</span>
<span class="k">def</span> <span class="nf">process_main</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">item</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
<span class="s">"""
Copy the metadata file into place and add it to the repomd file.
:param item: The unit to process
:type item: pulp.server.db.model.ContentUnit
"""</span>
<span class="n">unit</span> <span class="o">=</span> <span class="n">item</span>
<span class="c1"># Copy the file to the location on disk where the published repo is built
</span> <span class="n">publish_location_relative_path</span> <span class="o">=</span> <span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">get_working_dir</span><span class="p">(),</span>
<span class="n">REPO_DATA_DIR_NAME</span><span class="p">)</span>
<span class="n">metadata_file_name</span> <span class="o">=</span> <span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">basename</span><span class="p">(</span><span class="n">unit</span><span class="p">.</span><span class="n">_storage_path</span><span class="p">)</span>
<span class="n">link_path</span> <span class="o">=</span> <span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="n">publish_location_relative_path</span><span class="p">,</span> <span class="n">metadata_file_name</span><span class="p">)</span>
<span class="n">plugin_misc</span><span class="p">.</span><span class="n">create_symlink</span><span class="p">(</span><span class="n">unit</span><span class="p">.</span><span class="n">_storage_path</span><span class="p">,</span> <span class="n">link_path</span><span class="p">)</span>
<span class="c1"># Add the proper relative reference to the metadata file to repomd
</span> <span class="bp">self</span><span class="p">.</span><span class="n">parent</span><span class="p">.</span><span class="n">repomd_file_context</span><span class="p">.</span>\
<span class="n">add_metadata_file_metadata</span><span class="p">(</span><span class="n">unit</span><span class="p">.</span><span class="n">data_type</span><span class="p">,</span> <span class="n">link_path</span><span class="p">)</span>
</code></pre>
<p>That results in a yum repo with a symlink as in following example:</p>
<pre><code>$ ls -l /var/lib/pulp/published/yum/master/yum_distributor/rhel-ha-for-rhel-7-for-system-z-rpms__7Server__s390x/1552949091.82/repodata/
total 200
-rw-r--r--. 1 apache apache 2087 Mar 18 22:44 29e7003fe4c25a3291e80c16006ca0edc315c8b89f930f8be9f435bb10c1000b-updateinfo.xml.gz
-rw-r--r--. 1 apache apache 160494 Mar 18 22:44 792d01f805670173c58c8c55b1b960576de6b72215bd61d845b077af254d7e14-other.xml.gz
-rw-r--r--. 1 apache apache 16643 Mar 18 22:44 79f6997620d82b1085fb9dc22499b81d1b4ccb8a6113372039273d0dbab5be9f-filelists.xml.gz
lrwxrwxrwx. 1 apache apache 153 Mar 18 22:44 85771b84-06dd-4e4d-9e88-1b1d5616b3cc -> /var/lib/pulp/content/units/yum_repo_metadata_file/6b/74066bca70043c0cec6f1c43bc973a4bb84aa9ff6579757b8b0a9be1bcf81b/85771b84-06dd-4e4d-9e88-1b1d5616b3cc
-rw-r--r--. 1 apache apache 124 Mar 18 22:44 a27718cc28ec6d71432e0ef3e6da544b7f9d93f6bb7d0a55aacd592d03144b70-comps.xml
-rw-r--r--. 1 apache apache 3575 Mar 18 22:44 a57cc25570ba03b0c7fc82ad4502fad14fb20b404f065801563bc58ea8340eda-primary.xml.gz
-rw-r--r--. 1 apache apache 2405 Mar 18 22:44 repomd.xml
</code></pre>
<p>Then, rpm_rsync_distributor performs rsync of repodata directory <strong>with links=True</strong>, meaning rsync --links, meaning "copy symlinks as symlinks":</p>
<pre><code class="python syntaxhl" data-language="python"> <span class="bp">self</span><span class="p">.</span><span class="n">add_child</span><span class="p">(</span><span class="n">RSyncPublishStep</span><span class="p">(</span><span class="s">"Rsync step (repodata)"</span><span class="p">,</span>
<span class="n">repodata_file_list</span><span class="p">,</span>
<span class="s">"%s/"</span> <span class="o">%</span> <span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="n">master_dir</span><span class="p">,</span> <span class="s">'repodata'</span><span class="p">),</span>
<span class="s">"%s/"</span> <span class="o">%</span> <span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="n">remote_repo_path</span><span class="p">,</span> <span class="s">'repodata'</span><span class="p">),</span>
<span class="n">exclude</span><span class="o">=</span><span class="p">[</span><span class="s">".*"</span><span class="p">,</span> <span class="s">"repodata.old"</span><span class="p">],</span>
<span class="n">config</span><span class="o">=</span><span class="n">config</span><span class="p">,</span> <span class="n">links</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span>
<span class="n">delete</span><span class="o">=</span><span class="bp">self</span><span class="p">.</span><span class="n">config</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">"delete"</span><span class="p">)))</span>
</code></pre>
<p>This indeed rsyncs the symlink onto the remote host exactly as it is:</p>
<pre><code>$ ls -l /mnt/redhat/cdn/content/dist/rhel/system-z/7/7Server/s390x/highavailability/os/repodata | grep 85771b84-06dd-4e4d-9e88-1b1d5616b3cc
lrwxrwxrwx. 1 vagrant vagrant 153 Mar 18 22:44 85771b84-06dd-4e4d-9e88-1b1d5616b3cc -> /var/lib/pulp/content/units/yum_repo_metadata_file/6b/74066bca70043c0cec6f1c43bc973a4bb84aa9ff6579757b8b0a9be1bcf81b/85771b84-06dd-4e4d-9e88-1b1d5616b3cc
</code></pre>
<p>That's a broken symlink to /var/lib/pulp, which is not expected to exist on the remote. Hence the file is not able to be accessed and the published repo is incomplete.</p>
<a name="Steps-to-reproduce"></a>
<h3 >Steps to reproduce<a href="#Steps-to-reproduce" class="wiki-anchor">¶</a></h3>
<p>- Create a repo with yum_distributor, rpm_rsync_distributor<br>
- Add any yum_repo_metadata_file to the repo<br>
- Publish the repo with yum_distributor and then rpm_rsync_distributor</p>
<a name="Expected-result"></a>
<h3 >Expected result<a href="#Expected-result" class="wiki-anchor">¶</a></h3>
<p>All yum_repo_metadata_file in the repo are accessible in the repo published to remote host via rpm_rsync_distributor</p>
<a name="Actual-result"></a>
<h3 >Actual result<a href="#Actual-result" class="wiki-anchor">¶</a></h3>
<p>All yum_repo_metadata_file in published repo are broken symlinks.</p> Pulp - Test #4659 (CLOSED - DUPLICATE): Add RHEL to the ansible-pulp molecule CIhttps://pulp.plan.io/issues/46592019-04-09T15:14:09Zamacdona@redhat.comaustin@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulpcore/1822":<a href="https://github.com/pulp/pulpcore/issues/1822" class="external">https://github.com/pulp/pulpcore/issues/1822</a></p>
<hr>
<p>Unlike the other OSes that are currently tested, RHEL needs some additional steps, notably, enabling the subscription.</p>
<p>This test will be complete when RHEL is included in the test matrix for the ansible installer.</p> RPM Support - Test #4632 (CLOSED - COMPLETE): Publish YumMetadataFiles as files and not symlinkshttps://pulp.plan.io/issues/46322019-04-03T12:06:21Zbherring
<p>YumMetadata File should be copied, not symlinked into the publish location. (check the associated bz for more info.)</p> RPM Support - Test #4548 (CLOSED - COMPLETE): Ensure modular_errata.py includes cases for --recur...https://pulp.plan.io/issues/45482019-03-18T16:46:33Zbherring
<a name="Parent-Tickets"></a>
<h3 >Parent Tickets<a href="#Parent-Tickets" class="wiki-anchor">¶</a></h3>
<ul>
<li><a class="issue tracker-3 status-11 priority-6 priority-default closed parent" title="Story: Implement modularity content dependency solving (CLOSED - CURRENTRELEASE)" href="https://pulp.plan.io/issues/3740">#3740</a> && <a class="issue tracker-2 status-11 priority-6 priority-default closed" title="Task: Document how to use the newly added recursive_conservative (CLOSED - CURRENTRELEASE)" href="https://pulp.plan.io/issues/4371">#4371</a></li>
</ul>
<a name="Affected-Files"></a>
<h3 >Affected File(s)<a href="#Affected-Files" class="wiki-anchor">¶</a></h3>
<ul>
<li>test_modular_errata.py::ManageModularErrataTestCase::test_copy_errata</li>
</ul>
<a name="Update"></a>
<h3 >Update<a href="#Update" class="wiki-anchor">¶</a></h3>
<ul>
<li>Evaluate if the following are covered already and if any additional updates are required
<ul>
<li>Ensure test_modular_errata test includes --recursive and --recursive_conservative cases</li>
<li>Ensure that <strong>latest</strong> RPM copy test case is included</li>
</ul>
</li>
</ul> Container Support - Test #4460 (CLOSED - COMPLETE): Test pull from container registryhttps://pulp.plan.io/issues/44602019-02-25T14:21:27Zbherring
<a name="Test-Notes"></a>
<h2 >Test Notes<a href="#Test-Notes" class="wiki-anchor">¶</a></h2>
<ul>
<li>testing `docker pull` against published docker repo... ;</li>
<li>API tests against a published distribution</li>
</ul> RPM Support - Test #4351 (CLOSED - COMPLETE): modules.yaml reference in repomd.xml does not use s...https://pulp.plan.io/issues/43512019-01-24T14:13:07Zbherring
<p>A report from upstream katello (<a href="https://projects.theforeman.org/issues/25529" class="external">https://projects.theforeman.org/issues/25529</a>) indicates that pulp generates a repomd.xml file that claims to use the checksum type that has been set on the publisher, but in fact is not using that:</p>
<p>In this example it purports to have used a sha1 checksum, but in fact its actually a sha256 checksum:</p>
<pre><code> <data type="modules">
<location href="repodata/824ffe238f202a0612ecbb2b0c0459dc289a7ef47adb1f26406453d41f476449-modules.yaml.gz"/>
<timestamp>1542811864</timestamp>
<size>33</size>
<checksum type="sha1">824ffe238f202a0612ecbb2b0c0459dc289a7ef47adb1f26406453d41f476449</checksum>
<open-size>0</open-size>
<open-checksum type="sha1">da39a3ee5e6b4b0d3255bfef95601890afd80709</open-checksum>
</data>
</code></pre>
<p>To reproduce:</p>
<p>1. create and sync a yum repository<br>
2. set a checksum type of sha1<br>
3. publish the repository</p> RPM Support - Test #4350 (CLOSED - COMPLETE): modules.yaml file is generated on repository with n...https://pulp.plan.io/issues/43502019-01-24T14:12:25Zbherring
<p>It seems that syncing and publishing a repository without any modularity info, still results in a modules.yaml being generated.</p>
<p>See: <a href="https://projects.theforeman.org/issues/25529" class="external">https://projects.theforeman.org/issues/25529</a> for more information</p> RPM Support - Test #4297 (CLOSED - COMPLETE): Prevent duplicate Package content in RPM reposhttps://pulp.plan.io/issues/42972019-01-07T16:37:45Zbherring
<p>Packages are unique in a repo by NEVRA. When implemented, this will cause the most recently added unique NEVRA to be kept and other duplicate NEVRA to be removed. Duplicate NEVRA could have different checksums.</p>
<pre><code>[{
'model': Package,
'field_names': ['name', 'epoch', 'version', 'release', 'arch']
}]
</code></pre>
<p>Use a <code>RemoveDuplicates</code> stage provided by pulpcore-plugin.</p> Pulp - Test #4259 (CLOSED - COMPLETE): As a user I can sync from a docker registryhttps://pulp.plan.io/issues/42592018-12-11T13:37:49Zbherring
<a name="Notes"></a>
<h2 >Notes<a href="#Notes" class="wiki-anchor">¶</a></h2>
<ul>
<li>evaluate what more we can do in this area (see <a class="issue tracker-5 status-13 priority-6 priority-default closed" title="Test: Test sync of a repository that is missing blobs (CLOSED - COMPLETE)" href="https://pulp.plan.io/issues/4128">#4128</a> and <a class="issue tracker-5 status-13 priority-6 priority-default closed" title="Test: Test sync of a repository that returns a 403 response for a download request (CLOSED - COMPLETE)" href="https://pulp.plan.io/issues/4129">#4129</a> for more information in this area)</li>
</ul> Docker Support - Test #4129 (CLOSED - COMPLETE): Test sync of a repository that returns a 403 res...https://pulp.plan.io/issues/41292018-11-06T19:56:22Zamacdona@redhat.comaustin@redhat.com
<p>This test will require the creation of a new fixture for pulp_docker that is a corrupted repository. This fixture is primarily intended to test <a href="https://pulp.plan.io/issues/2966" class="external">https://pulp.plan.io/issues/2966</a> (pulp 2) but could also be useful for testing pulp 3.</p>
<p>From 2966, it appears that a 403 can be caused by a missing symlink in the published repository.</p> Docker Support - Test #4128 (CLOSED - COMPLETE): Test sync of a repository that is missing blobshttps://pulp.plan.io/issues/41282018-11-06T19:52:42Zamacdona@redhat.comaustin@redhat.com
<p>This test will require the creation of a new fixture for pulp_docker that is a corrupted repository. This fixture is primarily intended to test <a href="https://pulp.plan.io/issues/2849" class="external">https://pulp.plan.io/issues/2849</a> (pulp 2) but could also be useful for testing pulp 3.</p>
<p>The "corrupted repository" can be a fully published non-corrupt repository that is simply missing one or more Images referenced by or or more of the Image Manifests.</p> Pulp - Test #3634 (CLOSED - COMPLETE): Write functional tests for task delete and cancel codehttps://pulp.plan.io/issues/36342018-04-30T20:59:51Zamacdona@redhat.comaustin@redhat.com
<p>Tests should successfully delete tasks in final states, and should not be able to delete tasks in waiting or running states.</p>
<p>Tests should successfully cancel tasks in waiting and running states, but should not be able to cancel tasks in final states.</p>