Pulp: Issueshttps://pulp.plan.io/https://pulp.plan.io/favicon.ico2018-12-14T17:36:04ZPulp
Planio RPM Support - Test #4269 (CLOSED - COMPLETE): Recursive and conservative recursive copyhttps://pulp.plan.io/issues/42692018-12-14T17:36:04Zmilan
<a name="Description"></a>
<h2 >Description<a href="#Description" class="wiki-anchor">¶</a></h2>
<p>With the <a href="https://github.com/pulp/pulp_rpm/pull/1226" class="external">fix</a> for the Issue <a class="issue tracker-1 status-11 priority-6 priority-default closed" title="Issue: Regression Pulp 2.17.1: recursive copy of RPMs does not copy partially resolvable dependencies (CLOSED - CURRENTRELEASE)" href="https://pulp.plan.io/issues/4152">#4152</a>, the default behaviour of recursive copy changes back to the state before 2.17, rendering the test case <a href="https://github.com/PulpQE/Pulp-2-Tests/blob/c9277928c2788bced8d0ae3f630a46b6162fc4c9/pulp_2_tests/tests/rpm/api_v2/test_rich_weak_dependencies.py#L183" class="external">CopyRecursiveUnitsTestCase</a> broken.<br>
The behaviour specified in the <a href="https://github.com/PulpQE/pulp-smash/issues/1090" class="external">pulp smash issue 1090</a>, as requested in the Issue <a class="issue tracker-3 status-11 priority-6 priority-default closed" title="Story: Support more conservative dependency solving (CLOSED - CURRENTRELEASE)" href="https://pulp.plan.io/issues/2478">#2478</a>, is now optional and can be triggered by providing the <code>"recursive_conservative": true</code> config override instead of the default <code>"recursive": true</code>.<br>
With that setting, the test case <a href="https://github.com/PulpQE/Pulp-2-Tests/blob/c9277928c2788bced8d0ae3f630a46b6162fc4c9/pulp_2_tests/tests/rpm/api_v2/test_rich_weak_dependencies.py#L183" class="external">CopyRecursiveUnitsTestCase</a> should agin pass as is.</p>
<a name="Proposed-solution"></a>
<h2 >Proposed solution<a href="#Proposed-solution" class="wiki-anchor">¶</a></h2>
<p>Split the test case <a href="https://github.com/PulpQE/Pulp-2-Tests/blob/c9277928c2788bced8d0ae3f630a46b6162fc4c9/pulp_2_tests/tests/rpm/api_v2/test_rich_weak_dependencies.py#L183" class="external">CopyRecursiveUnitsTestCase</a> such that it asserts:</p>
<ul>
<li>a failure in the terms described in <a href="https://github.com/PulpQE/pulp-smash/issues/1090" class="external">pulp smash issue 1090</a> with the default <code>"recursive": true</code> config override setting</li>
<li>a success once the non-default <code>"recursive_conservative": true</code> config override setting is provided</li>
</ul> RPM Support - Story #4162 (CLOSED - CURRENTRELEASE): As a user, I have dependency solving when co...https://pulp.plan.io/issues/41622018-11-15T15:20:58Zmilan
<p>When modules are copied between repos, the following are also copied:</p>
<ul>
<li>RPM artifacts</li>
<li>Other modules marked as dependencies</li>
<li>The module default for that module, if one exists</li>
</ul> RPM Support - Issue #4144 (CLOSED - CURRENTRELEASE): Duplicated module dependnecy entries both af...https://pulp.plan.io/issues/41442018-11-10T19:26:08Zmilan
<p>Having run the migration <code>0047_add_modulemd_dependencies_field.py</code> in my pulp deployment that syncs Fedora 29 content I saw that the dependencies of modules can be duplicated:</p>
<p><code>db.units_modulemd.find({"dependencies": {$ne: []}})[6]</code></p>
<pre><code class="JSON syntaxhl" data-language="JSON"><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"_id"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"042a0ffd-0475-40ef-bba6-cf9380ab931c"</span><span class="p">,</span><span class="w">
</span><span class="nl">"pulp_user_metadata"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"_last_updated"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">1539260489</span><span class="p">,</span><span class="w">
</span><span class="nl">"_storage_path"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"/var/lib/pulp/content/units/modulemd/b0/f1a8a4a6229d2531c4004f0d70499933a040f4d6428827d6c41aeaa81ed66b"</span><span class="p">,</span><span class="w">
</span><span class="nl">"downloaded"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w">
</span><span class="nl">"name"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"reviewboard"</span><span class="p">,</span><span class="w">
</span><span class="nl">"stream"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"2.5"</span><span class="p">,</span><span class="w">
</span><span class="nl">"version"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="err">NumberLong(</span><span class="s2">"20180828143308"</span><span class="err">)</span><span class="p">,</span><span class="w">
</span><span class="nl">"context"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"083bce86"</span><span class="p">,</span><span class="w">
</span><span class="nl">"arch"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"x86_64"</span><span class="p">,</span><span class="w">
</span><span class="nl">"summary"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"A web-based code review tool"</span><span class="p">,</span><span class="w">
</span><span class="nl">"description"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"Review Board is a powerful web-based code review tool that offers developers an easy way to handle code reviews. It scales well from small projects to large companies and offers a varie$y of tools to take much of the stress and time out of the code review process."</span><span class="p">,</span><span class="w">
</span><span class="nl">"profiles"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"default"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"ReviewBoard"</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"server"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"ReviewBoard"</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"artifacts"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"ReviewBoard-0:2.5.17-17.module_1631+4353a891.noarch"</span><span class="p">,</span><span class="w">
</span><span class="s2">"python-django-haystack-docs-0:2.4.1-12.module_1631+4353a891.noarch"</span><span class="p">,</span><span class="w">
</span><span class="s2">"python2-django-evolution-1:0.7.7-12.module_1631+4353a891.noarch"</span><span class="p">,</span><span class="w">
</span><span class="s2">"python2-django-haystack-0:2.4.1-12.module_1631+4353a891.noarch"</span><span class="p">,</span><span class="w">
</span><span class="s2">"python2-django-multiselectfield-0:0.1.3-10.module_1631+4353a891.noarch"</span><span class="p">,</span><span class="w">
</span><span class="s2">"python2-django-pipeline-0:1.3.27-11.module_1631+4353a891.noarch"</span><span class="p">,</span><span class="w">
</span><span class="s2">"python2-djblets-0:0.9.9-13.module_1631+4353a891.noarch"</span><span class="p">,</span><span class="w">
</span><span class="s2">"python2-markdown-0:2.4.1-12.module_2085+40241970.noarch"</span><span class="p">,</span><span class="w">
</span><span class="s2">"python3-markdown-0:2.4.1-12.module_2085+40241970.noarch"</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"checksum"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"92c01be4ef52dab5aa75e0d31562b1bbb45e088b013900c8fe90f0c744cf8431"</span><span class="p">,</span><span class="w">
</span><span class="nl">"_ns"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"units_modulemd"</span><span class="p">,</span><span class="w">
</span><span class="nl">"_content_type_id"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"modulemd"</span><span class="p">,</span><span class="w">
</span><span class="nl">"dependencies"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"platform"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"f29"</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"django"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"1.6"</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"platform"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"f29"</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"django"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"1.6"</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre>
<p>For comparison, the original modular metadata:</p>
<pre><code class="YAML syntaxhl" data-language="YAML"><span class="na">document</span><span class="pi">:</span> <span class="s">modulemd</span>
<span class="na">version</span><span class="pi">:</span> <span class="m">2</span>
<span class="na">data</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">reviewboard</span>
<span class="na">stream</span><span class="pi">:</span> <span class="m">2.5</span>
<span class="na">version</span><span class="pi">:</span> <span class="m">20180828143308</span>
<span class="na">context</span><span class="pi">:</span> <span class="s">083bce86</span>
<span class="na">arch</span><span class="pi">:</span> <span class="s">x86_64</span>
<span class="na">summary</span><span class="pi">:</span> <span class="s">A web-based code review tool</span>
<span class="na">description</span><span class="pi">:</span> <span class="pi">>-</span>
<span class="s">Review Board is a powerful web-based code review tool that offers developers an</span>
<span class="s">easy way to handle code reviews. It scales well from small projects to large companies</span>
<span class="s">and offers a variety of tools to take much of the stress and time out of the code</span>
<span class="s">review process.</span>
<span class="na">license</span><span class="pi">:</span>
<span class="na">module</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">MIT</span>
<span class="na">content</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">BSD</span>
<span class="pi">-</span> <span class="s">LGPLv3</span>
<span class="pi">-</span> <span class="s">MIT</span>
<span class="pi">-</span> <span class="s">MIT and (MIT or GPLv2)</span>
<span class="na">xmd</span><span class="pi">:</span>
<span class="na">mbs</span><span class="pi">:</span>
<span class="na">mse</span><span class="pi">:</span> <span class="s">TRUE</span>
<span class="na">scmurl</span><span class="pi">:</span> <span class="s">https://src.fedoraproject.org/modules/reviewboard.git?#0fc31243aa85666e4e584aebda7a74e430cd0698</span>
<span class="na">commit</span><span class="pi">:</span> <span class="s">0fc31243aa85666e4e584aebda7a74e430cd0698</span>
<span class="na">buildrequires</span><span class="pi">:</span>
<span class="na">platform</span><span class="pi">:</span>
<span class="na">ref</span><span class="pi">:</span> <span class="s">f29</span>
<span class="na">stream</span><span class="pi">:</span> <span class="s">f29</span>
<span class="na">context</span><span class="pi">:</span> <span class="m">00000000</span>
<span class="na">version</span><span class="pi">:</span> <span class="m">5</span>
<span class="na">filtered_rpms</span><span class="pi">:</span> <span class="pi">[]</span>
<span class="na">django</span><span class="pi">:</span>
<span class="na">ref</span><span class="pi">:</span> <span class="s">da53d66d3db1aa5f6afd75887b00fa26849bcc0a</span>
<span class="na">stream</span><span class="pi">:</span> <span class="m">1.6</span>
<span class="na">context</span><span class="pi">:</span> <span class="s">6c81f848</span>
<span class="na">version</span><span class="pi">:</span> <span class="m">20180828135711</span>
<span class="na">filtered_rpms</span><span class="pi">:</span> <span class="pi">[]</span>
<span class="na">rpms</span><span class="pi">:</span>
<span class="na">python-django-multiselectfield</span><span class="pi">:</span>
<span class="na">ref</span><span class="pi">:</span> <span class="s">149bf58875fb7b55efe29e1735baf96d44eb99a9</span>
<span class="na">python-markdown</span><span class="pi">:</span>
<span class="na">ref</span><span class="pi">:</span> <span class="s">96c1b2f925c2ad3b7aae04ab7ebf25ae1d714ebf</span>
<span class="na">python-django-evolution</span><span class="pi">:</span>
<span class="na">ref</span><span class="pi">:</span> <span class="s">512424e1fc4b99f6f74c01a4130a4d9402b56b4e</span>
<span class="na">python-django-pipeline</span><span class="pi">:</span>
<span class="na">ref</span><span class="pi">:</span> <span class="s">f019137be96cf86f49a81001fef47a0c7ab6aa35</span>
<span class="na">ReviewBoard</span><span class="pi">:</span>
<span class="na">ref</span><span class="pi">:</span> <span class="s">5d28213f6a797e5ce28ad05ab23f80fe67353da8</span>
<span class="na">python-djblets</span><span class="pi">:</span>
<span class="na">ref</span><span class="pi">:</span> <span class="s">d5634779089456ff3d0ac7b78eec81e13ff4c733</span>
<span class="na">dependencies</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">buildrequires</span><span class="pi">:</span>
<span class="na">django</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">1.6</span><span class="pi">]</span>
<span class="na">platform</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">f29</span><span class="pi">]</span>
<span class="na">requires</span><span class="pi">:</span>
<span class="na">django</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">1.6</span><span class="pi">]</span>
<span class="na">platform</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">f29</span><span class="pi">]</span>
<span class="c1"># ------->%---------------------</span>
</code></pre> RPM Support - Issue #4002 (CLOSED - WONTFIX): pulp-admin doesn't support uploading modular erratahttps://pulp.plan.io/issues/40022018-09-12T14:00:44Zmilan
<p>The pulp-admin tool doesn't currently support uploading modular errata.<br>
This requires enhancement of the <code>--pkglist-csv packagelist.csv</code> switch of the errata upload subcommand because at the moment there's no way to assign a package list to a modular erratum.<br>
Possible solution might add a new, nargs switch that would accept a JSON structure, similar to the one the upload API uses, e.g:<br>
<code>$ pulp-admin rpm repo uploads erratum --repo-id zoo --erratum-id RHEA-2018:0042 ... --module-package-collection gorilla.json --module-package-collection walrus.json</code><br>
<code>$ cat walrus.json</code></p>
<pre><code class="json syntaxhl" data-language="json"><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"collection 0"</span><span class="p">,</span><span class="w">
</span><span class="nl">"short"</span><span class="p">:</span><span class="w"> </span><span class="s2">"0"</span><span class="p">,</span><span class="w">
</span><span class="nl">"module"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"walrus"</span><span class="p">,</span><span class="w">
</span><span class="nl">"stream"</span><span class="p">:</span><span class="w"> </span><span class="s2">"5.21"</span><span class="p">,</span><span class="w">
</span><span class="nl">"version"</span><span class="p">:</span><span class="w"> </span><span class="s2">"20180704144203"</span><span class="p">,</span><span class="w">
</span><span class="nl">"arch"</span><span class="p">:</span><span class="w"> </span><span class="s2">"x86_64"</span><span class="p">,</span><span class="w">
</span><span class="nl">"context"</span><span class="p">:</span><span class="w"> </span><span class="s2">"deadbeef"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"packages"</span><span class="p">:</span><span class="w"> </span><span class="p">[]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre> RPM Support - Story #3959 (CLOSED - CURRENTRELEASE): Modular errata sync and publishhttps://pulp.plan.io/issues/39592018-08-31T09:08:19Zmilan
<p>This sub-task tracks the implementation of sync and publish of modular errata, just to better coordinate the engineering work.<br>
For the details about the sync&publish implementation, see the Issue <a class="issue tracker-3 status-11 priority-6 priority-default closed parent" title="Story: As a user, I can manage modular Errata content (CLOSED - CURRENTRELEASE)" href="https://pulp.plan.io/issues/3919">#3919</a>.</p> RPM Support - Issue #3903 (CLOSED - CURRENTRELEASE): Redundant solver instantiated and loaded dur...https://pulp.plan.io/issues/39032018-08-06T20:29:27Zmilan
<p>It seems a solver is being unnecessarily <a href="https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/importers/yum/associate.py#L53#L55">instantiated and loaded</a> in a <a href="https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/importers/yum/associate.py#L92" class="external">recursive associate call</a><br>
This wastes a lot of RAM and CPU with big repositories and might be avoided.</p> RPM Support - Issue #3902 (CLOSED - NOTABUG): Recursive module copy doesn't follow intended modul...https://pulp.plan.io/issues/39022018-08-06T20:15:32Zmilan
<p>Recursive copy of module artifacts should <a href="https://docs.fedoraproject.org/en-US/modularity/architecture/consuming/#_updating_the_system" class="external">prefer the modular dependencies implied</a> while at the same time, Pulp should keep the <a href="https://pulp.plan.io/issues/2478" class="external">conservative dependency solving</a> ; these requirements contradict each other.</p>
<p>Say module Frog has a frog-1.rpm and fly-1.rpm artifacts and that frog-1.rpm has the Requires: fly dependency.<br>
If a source repository contains frog-1.rpm, fly-1.rpm and the destination repository contains fly-2.rpm, the fly-1.rpm won't be recursively copied because of an already satisfied dependency fly-2.rpm being present in the target repository.<br>
This breaks the expected modular behavior.</p>
<a name="Notes"></a>
<h3 >Notes<a href="#Notes" class="wiki-anchor">¶</a></h3>
<p>This actually seems to work OK because Pulp always copies first-level module dependencies and unless the module is broken by not shipping/listing custom artifact versions necessary for the module to (buind and) run, this might be just enough:</p>
<ul>
<li><a href="https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/importers/yum/associate.py#L97#L101">get the module artifact nevras to copy</a></li>
<li><a href="https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/importers/yum/associate.py#L227#L228">identify dependent units outside of the module artifacts</a></li>
</ul> RPM Support - Task #3858 (CLOSED - WONTFIX): Clean up old solver and unittestshttps://pulp.plan.io/issues/38582018-07-17T12:10:42Zmilan
<p>Let's remove the old solver code and unit tests once the Issue <a class="issue tracker-3 status-11 priority-6 priority-default closed" title="Story: Reimplement unit dependency solving with the libsolv library (CLOSED - CURRENTRELEASE)" href="https://pulp.plan.io/issues/3715">#3715</a> PR <a href="https://github.com/pulp/pulp_rpm/pull/1122" class="external">https://github.com/pulp/pulp_rpm/pull/1122</a> lands.</p> RPM Support - Issue #3853 (CLOSED - WONTFIX): Recursive copy doesn't take conflicts into accounthttps://pulp.plan.io/issues/38532018-07-13T16:28:15Zmilan
<p>Recursive copy doesn't take conflicts into account. This leads into broken repositories when only few units a time are being copied.<br>
Imagine following hypothetical recursive copy scenario:</p>
<ul>
<li>foo requires webserver and conflicts with libnginx</li>
<li>nginx provides webserver and libnginx</li>
<li>httpd provides webserver while it doesn't provide libnginx</li>
<li>the source repository contains both httpd and nginx rpms</li>
<li>pulp not processing the conflicts attribute, a user copying foo to the empty repository might end up with nginx being pulled in</li>
<li>dnf panics trying to install foo from the target repository</li>
</ul>
<p>This might not have been reported sooner as the usual use case seems to be many units are being copied at a time.<br>
Or perhaps nothing ever conflicts with libnginx ;)</p> RPM Support - Story #3847 (CLOSED - CURRENTRELEASE): Introduce weak dependency solvinghttps://pulp.plan.io/issues/38472018-07-10T16:41:13Zmilan
<p>Pulp lacks resolving of <a href="http://rpm.org/user_doc/dependencies.html" class="external">RPM weak forward dependency</a> <code>Recommends</code> during content association in case of e.g recursively copying RPMs between repositories, which by default is respected by <code>dnf</code> A content gap might therefore be induced on a consumer machine when installing from a repository containing just the recursive-copy calculated dependencies of an RPM unit. This isn't a breaking discrepancy as <a href="https://fedoraproject.org/wiki/Packaging:WeakDependencies#Weak_dependencies" class="external">by definition, weak dependencies missing don't cause an unit to break</a></p>
<p>The very weak/hint <code>Suggests</code> field is out of scope <a href="https://fedoraproject.org/wiki/Packaging:WeakDependencies#Hints" class="external">as hints are by default not processed by <code>dnf</code></a><br>
Bot the weak and very weak/hint backward fields: <code>Supplements</code> and <code>Enhances</code> are out of scope too.</p>
<p>Publishing the weak dependencies in repository metadata is out of scope of this story as this is already supported trivially.</p>
<a name="Implementation"></a>
<h3 >Implementation<a href="#Implementation" class="wiki-anchor">¶</a></h3>
<ul>
<li>the <a href="https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/db/models.py#L743,#L772">RPM model</a> needs to track the following "forward" weak dependency field <code>Recommends</code>
</li>
<li>the <a href="https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/importers/yum/repomd/primary.py" class="external">primary XML repo metadata parsing code</a> has to be updated in order to populate the <code>Recommends</code> field</li>
</ul>
<a name="Examples"></a>
<h3 >Examples<a href="#Examples" class="wiki-anchor">¶</a></h3>
<p>I've found few <code>Recommends</code> field samples in the Fedora28--Workstation flavor <a href="http://download.eng.brq.redhat.com/released/F-28/GOLD/Workstation/x86_64/os/repodata/6f44e3c7dc24e8943cdc1a384fbbb8d7af3f2150724af202393ec065ee971e61-primary.xml.gz" class="external">primary.xml</a> repodata file. The <code>dnf</code> unit recommends installing these two items, with rich dependency conditioning:</p>
<pre><code class="xml syntaxhl" data-language="xml"><span class="nt"><rpm:recommends></span>
<span class="nt"><rpm:entry</span> <span class="na">name=</span><span class="s">"(/usr/bin/sqlite3 if bash-completion)"</span><span class="nt">/></span>
<span class="nt"><rpm:entry</span> <span class="na">name=</span><span class="s">"(python3-dbus if NetworkManager)"</span><span class="nt">/></span>
<span class="nt"></rpm:recommends></span>
</code></pre>
<a name="Notes"></a>
<h3 >Notes<a href="#Notes" class="wiki-anchor">¶</a></h3>
<ul>
<li>the RPM content upload code needn't be updated as it <a href="https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/importers/yum/upload.py#L375,#L384">reuses the XML parsing code</a>
</li>
<li>the backwards dependencies needn't be processed because the Pulp workflow is closer to a repo closure calculation than to an actual content installation</li>
<li>the processing of the <code>Recommends</code> field is going to be <a href="https://github.com/pulp/pulp_rpm/pull/1122/files#diff-cfe0bb646220ae499249edd39626a096R232" class="external">handled thru <code>libsolv</code></a> once that PR lands</li>
<li>weak dependencies processing can be <a href="https://bgstack15.wordpress.com/2017/05/13/dnf-ignore-weak-dependencies/" class="external">switched off</a> in <code>dnf</code>; see also: <code>man dnf.conf</code>
</li>
<li>very weak dependencies/hints are most likely just completely ignored by <code>dnf</code>
</li>
</ul> RPM Support - Issue #3787 (CLOSED - CURRENTRELEASE): Unhashable type: list while recursively copy...https://pulp.plan.io/issues/37872018-06-25T20:11:18Zmilan
<p>While testing the pulp_rpm PR 1122 for Issue <a class="issue tracker-3 status-11 priority-6 priority-default closed" title="Story: Reimplement unit dependency solving with the libsolv library (CLOSED - CURRENTRELEASE)" href="https://pulp.plan.io/issues/3715">#3715</a> ran into following traceback:</p>
<pre><code>Jun 25 22:04:08 pulp2F28 pulp[8750]: pulp.server.managers.repo.unit_association:ERROR: [5e075c5f] (8750-23936) Exception from importer [yum_importer] while importing units into repository [fo
o]
Jun 25 22:04:08 pulp2F28 pulp[8750]: pulp.server.managers.repo.unit_association:ERROR: [5e075c5f] (8750-23936) Traceback (most recent call last):
Jun 25 22:04:08 pulp2F28 pulp[8750]: pulp.server.managers.repo.unit_association:ERROR: [5e075c5f] (8750-23936) File "/home/milan/devel/pulp/server/pulp/server/managers/repo/unit_association
.py", line 273, in associate_from_repo
Jun 25 22:04:08 pulp2F28 pulp[8750]: pulp.server.managers.repo.unit_association:ERROR: [5e075c5f] (8750-23936) units=transfer_units)
Jun 25 22:04:08 pulp2F28 pulp[8750]: pulp.server.managers.repo.unit_association:ERROR: [5e075c5f] (8750-23936) File "/home/milan/devel/pulp_rpm/plugins/pulp_rpm/plugins/importers/yum/import
er.py", line 58, in import_units
Jun 25 22:04:08 pulp2F28 pulp[8750]: pulp.server.managers.repo.unit_association:ERROR: [5e075c5f] (8750-23936) return associate.associate(source_repo, dest_repo, import_conduit, config, u
nits)
Jun 25 22:04:08 pulp2F28 pulp[8750]: pulp.server.managers.repo.unit_association:ERROR: [5e075c5f] (8750-23936) File "/home/milan/devel/pulp_rpm/plugins/pulp_rpm/plugins/importers/yum/associ
ate.py", line 88, in associate
Jun 25 22:04:08 pulp2F28 pulp[8750]: pulp.server.managers.repo.unit_association:ERROR: [5e075c5f] (8750-23936) associate(source_repo, dest_repo, import_conduit, config, group_units))
Jun 25 22:04:08 pulp2F28 pulp[8750]: pulp.server.managers.repo.unit_association:ERROR: [5e075c5f] (8750-23936) TypeError: unhashable type: 'list'
Jun 25 22:04:08 pulp2F28 pulp[8750]: pulp.server.async.tasks:INFO: [5e075c5f] Task failed : [5e075c5f-f93f-4fd9-a9db-38c70f430007]
Jun 25 22:04:08 pulp2F28 pulp[8750]: celery.app.trace:ERROR: [5e075c5f] (8750-23936) Task pulp.server.managers.repo.unit_association.associate_from_repo[5e075c5f-f93f-4fd9-a9db-38c70f430007]
raised unexpected: TypeError("unhashable type: 'list'",)
Jun 25 22:04:08 pulp2F28 pulp[8750]: celery.app.trace:ERROR: [5e075c5f] (8750-23936) Traceback (most recent call last):
Jun 25 22:04:08 pulp2F28 pulp[8750]: celery.app.trace:ERROR: [5e075c5f] (8750-23936) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
Jun 25 22:04:08 pulp2F28 pulp[8750]: celery.app.trace:ERROR: [5e075c5f] (8750-23936) R = retval = fun(*args, **kwargs)
Jun 25 22:04:08 pulp2F28 pulp[8750]: celery.app.trace:ERROR: [5e075c5f] (8750-23936) File "/home/milan/devel/pulp/server/pulp/server/async/tasks.py", line 529, in __call__
Jun 25 22:04:08 pulp2F28 pulp[8750]: celery.app.trace:ERROR: [5e075c5f] (8750-23936) return super(Task, self).__call__(*args, **kwargs)
Jun 25 22:04:08 pulp2F28 pulp[8750]: celery.app.trace:ERROR: [5e075c5f] (8750-23936) File "/home/milan/devel/pulp/server/pulp/server/async/tasks.py", line 107, in __call__
Jun 25 22:04:08 pulp2F28 pulp[8750]: celery.app.trace:ERROR: [5e075c5f] (8750-23936) return super(PulpTask, self).__call__(*args, **kwargs)
Jun 25 22:04:08 pulp2F28 pulp[8750]: celery.app.trace:ERROR: [5e075c5f] (8750-23936) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
Jun 25 22:04:08 pulp2F28 pulp[8750]: celery.app.trace:ERROR: [5e075c5f] (8750-23936) return self.run(*args, **kwargs)
Jun 25 22:04:08 pulp2F28 pulp[8750]: celery.app.trace:ERROR: [5e075c5f] (8750-23936) File "/home/milan/devel/pulp/server/pulp/server/managers/repo/unit_association.py", line 291, in associa
te_from_repo
Jun 25 22:04:08 pulp2F28 pulp[8750]: celery.app.trace:ERROR: [5e075c5f] (8750-23936) raise (e, None, sys.exc_info()[2])
Jun 25 22:04:08 pulp2F28 pulp[8750]: celery.app.trace:ERROR: [5e075c5f] (8750-23936) TypeError: unhashable type: 'list'
</code></pre>
<p>In the <a href="https://github.com/pulp/pulp_rpm/blob/master/plugins/pulp_rpm/plugins/importers/yum/associate.py#L88" class="external">offending line</a>, the <code>associate</code> function is called recursively with found content category group rpm units but returns the whole tuple instead of just the list of associated units into the <code>set</code> call, causing the traceback.</p> RPM Support - Task #3746 (CLOSED - COMPLETE): Investigate, whether exposing the @pool->considered...https://pulp.plan.io/issues/37462018-06-08T10:19:31Zmilan
<p>It might be the case that to support the functionality described in issue <a class="issue tracker-3 status-11 priority-6 priority-default closed parent" title="Story: Implement modularity content dependency solving (CLOSED - CURRENTRELEASE)" href="https://pulp.plan.io/issues/3740">#3740</a>, exposing the <code>pool->considered</code> bitmap needn't be necessary.</p>
<p>Considering the <a href="https://github.com/dparalen/pulp_solv" class="external">POC</a>, it might be sufficient to just ignore the target repository content while building the pool to resolve the copy query.</p> RPM Support - Story #3740 (CLOSED - CURRENTRELEASE): Implement modularity content dependency solvinghttps://pulp.plan.io/issues/37402018-06-07T12:53:47Zmilan
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>Pulp lacks the ability to perform dependency solving at a module level and to copy dependent modules and their artifacts during copy operation.</p>
<p>Pulp (@2.18) ignores module dependencies when copying modules recursively.<br>
Pulp doesn't distinguish between modular and non-modular artifacts when recursively copying modules to a repository. Pulp however always copies all module artifacts, including the artifacts rpm dependencies.<br>
A potential problem and a rare case:<br>
- a mixture of modular and non-modular packages with the same NEVRA are present in a repo<br>
- unlucky coincidence with versions<br>
- as a result - a "bad" repo (client will likely have a problem with a module which doesn't have all its <strong>modular</strong> rpms in a repo, since some of the copied rpms were non-modular ones).</p>
<a name="Proposed-solution"></a>
<h2 >Proposed solution<a href="#Proposed-solution" class="wiki-anchor">¶</a></h2>
<p>It's important for modules to have all their artifacts present in a repo.<br>
Regardless of the depsolving strategy for RPMs, <strong>all</strong> module artifacts and related modules should be copied to a target repo. E.g if the target repository already contains units newer than a module requires, the module-required units need to be copied in the older version, in addition to the already present units.</p>
<p><a href="https://docs.fedoraproject.org/en-US/modularity/making-modules/defining-modules/#_advanced_dependencies_optional" class="external">Modules may depend on other modules</a>, either for the build- or the run-time. Pulp should perform dependency solving based on runtime dependencies only. They are already available on the Modulemd model.</p>
<p>To support these usecases, a fake <code>libsolv</code> solvable can be created for each module unit. This will allow tracking dependencies between the modules. To prevent non-modular content from satisfying dependencies in target repository, it seems exposing the <code>pool->considered</code> bitmap from the <code>libsolv</code> library to its Python binding will be required. This is how DNF deals with solving modular dependencies when installing and upgrading content. This effort is tracked in the sub-task #3741.</p>
<a name="References"></a>
<h2 >References<a href="#References" class="wiki-anchor">¶</a></h2>
<ul>
<li>the base pulp_modularity support tracker; Issue <a class="issue tracker-3 status-11 priority-6 priority-default closed" title="Story: Support for module metadata in pulp_rpm (CLOSED - CURRENTRELEASE)" href="https://pulp.plan.io/issues/3206">#3206</a></li>
<li>the <a href="https://docs.fedoraproject.org/fedora-project/subprojects/fesco/en-US/index.html" class="external">Modularity project docs page</a>
</li>
<li>the depsolving library investigation <a class="issue tracker-2 status-13 priority-6 priority-default closed" title="Task: Investigate which library to use for dep solving (CLOSED - COMPLETE)" href="https://pulp.plan.io/issues/3528">#3528</a> <a href="https://github.com/dparalen/pulp_solv" class="external">POC</a>
</li>
<li><a href="https://github.com/openSUSE/libsolv" class="external">libsolv</a></li>
<li><a href="https://pulp.plan.io/issues/3919" class="external">modular errata</a></li>
<li>A cross-team, <a href="https://docs.google.com/document/d/1U-kkciwohRTuTF12V-uc0TrIrm0YDQxpqtD3feJJANo/edit" class="external">modular-depsolving document</a>
</li>
</ul> RPM Support - Story #3715 (CLOSED - CURRENTRELEASE): Reimplement unit dependency solving with the...https://pulp.plan.io/issues/37152018-05-31T15:23:12Zmilan
<p>This story tracks tasks required to implement dependency solving using the <code>libsolv</code> library, based on the investigation and on the POC from the Issue <a class="issue tracker-2 status-13 priority-6 priority-default closed" title="Task: Investigate which library to use for dep solving (CLOSED - COMPLETE)" href="https://pulp.plan.io/issues/3528">#3528</a>.</p>
<p>The goal is to implement a solver suitable for the importer <a href="https://github.com/pulp/pulp_rpm/blob/ef5fc5b2af47736114b68bc08658d9b2a94b84e1/plugins/pulp_rpm/plugins/importers/yum/associate.py#L45#L101">associate interface</a> but some refactoring of the current internal solver call interface might be necessary.</p>
<p>The solver should support the Yum-specific content unit types (recursive) dependency solving, namely:</p>
<ul>
<li>RPM</li>
<li>Errata</li>
<li>Package Group</li>
<li>Package Category</li>
</ul>
<p>on content unit attributes, such as <code>provides, requires</code>, as relevant to a particular content unit model, including boolean&weak dependencies solving, <em>in the context of a source and target repository</em> to <a href="https://pulp.plan.io/issues/2478" class="external">Support more conservative dependency solving</a></p>
<p>A separate story will track the <a href="https://docs.pagure.org/modularity/" class="external">modular content</a> as it requires a <code>libsolv</code> patch to expose the <code>pool->considered</code> bitmap to the Python bindings.</p>
<p>Implementation design is described on the Issue <a class="issue tracker-2 status-13 priority-6 priority-default closed" title="Task: Investigate which library to use for dep solving (CLOSED - COMPLETE)" href="https://pulp.plan.io/issues/3528">#3528</a>; see the <a href="https://pulp.plan.io/issues/3528#note-8" class="external">Note #8</a><br>
The <a href="https://github.com/dparalen/pulp_solv" class="external">POC</a> code can be used as an example.</p>
<p>The functionality exposed by this implementation should be feature-compatible with current Pulp2 implementation.<br>
It implies that no support for either SRPM or DRPM is required and that the REST API must not change when it comes to reporting the units copied.</p> Pulp - Story #3520 (CLOSED - CURRENTRELEASE): As a Sys-admin/Deployer/DevOps/Developer person, I'...https://pulp.plan.io/issues/35202018-03-22T21:11:31Zmilan
<p><a href="https://www.redhat.com/archives/pulp-list/2018-March/msg00013.html" class="external">The Community may be interested</a> in this</p>
<a name="Scope"></a>
<h2 >Scope<a href="#Scope" class="wiki-anchor">¶</a></h2>
<p>This can be based on <a class="user active" href="https://pulp.plan.io/users/7">mhrivnak</a> 's <a href="https://github.com/mhrivnak/pulp-k8s" class="external">work</a></p>
<p>Figure out how to create Kubernetes stateful (micro) services, probably by with the help of a <a href="https://kubernetes.io/docs/tutorials/stateful-application/mysql-wordpress-persistent-volume/" class="external">tutorial</a></p>
<p>Create containers for:</p>
<ul>
<li>workers</li>
<li>wsgi pulp core</li>
<li>crane</li>
</ul>
<p>Figure out how to plug pulp services/containers into existing environment:</p>
<ul>
<li>certificates management</li>
<li>database</li>
<li>shared filesystem (GlusterFS, Ceph, Nfs)</li>
<li>(tasking) queue (backend)</li>
<li>Proxy services (for e.g delayed download)</li>
<li>load-balancers</li>
</ul>
<p>Share the findings as repository of artifacts and documentation reusable by folks to deploy Pulp themselves</p>