Pulp: Issueshttps://pulp.plan.io/https://pulp.plan.io/favicon.ico2021-08-19T14:09:05ZPulp
Planio RPM Support - Task #9259 (CLOSED - DUPLICATE): workflow-docs should use pulp-cli instead of httpiehttps://pulp.plan.io/issues/92592021-08-19T14:09:05Zggainey
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2314":<a href="https://github.com/pulp/pulp_rpm/issues/2314" class="external">https://github.com/pulp/pulp_rpm/issues/2314</a></p>
<hr>
<p>Probably shouldn't do this unless/until <strong>all</strong> workflows are supported by pulp-cli commands - mixing pulp-cli and httpie is Very Confusing to the user.</p>
<p>Putting this up as a placeholder for docs-day</p> RPM Support - Issue #9233 (CLOSED - DUPLICATE): Downloaded content seems to be removed when a tas...https://pulp.plan.io/issues/92332021-08-11T22:27:01Zlmjachky
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2294":<a href="https://github.com/pulp/pulp_rpm/issues/2294" class="external">https://github.com/pulp/pulp_rpm/issues/2294</a></p>
<hr>
<p>For large repositories, this means that when performing <strong>immediate</strong> synchronization, all the downloaded content is lost after the task's failure. Pulp is then trying to download the content (artifacts) once again from scratch when issuing re-syncing, rather than associating the orphaned artifacts with corresponding content units.</p>
<p>More info can be found here: <a href="https://community.theforeman.org/t/oracle-linux-8-appstream-sync-fails/24676/9" class="external">https://community.theforeman.org/t/oracle-linux-8-appstream-sync-fails/24676/9</a>. The following part is the most important to us:</p>
<blockquote>
<p>If you are performing one of these large syncs, and get part way through before the sync fails with the timeout, it seems everything which was downloaded into the tmp directory is thrown away, and not kept. So, the next time the sync is tried, it has to download everything again.</p>
</blockquote>
<p>Used versions: pulpcore 3.14.3 and pulp_rpm 3.14.0</p>
<p>This behaviour could not be reproduced for pulp_file (<a href="https://fixtures.pulpproject.org/file/" class="external">https://fixtures.pulpproject.org/file/</a>; pulpcore 3.15.0.dev and pulp_file 1.9.0.dev); therefore, we should focus on the pulp_rpm plugin.</p> RPM Support - Issue #9208 (CLOSED - DUPLICATE): Published .treeinfo metadata not matching expecta...https://pulp.plan.io/issues/92082021-08-04T04:48:45Zdalleydalley@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2292":<a href="https://github.com/pulp/pulp_rpm/issues/2292" class="external">https://github.com/pulp/pulp_rpm/issues/2292</a></p>
<hr>
<ol>
<li>"packages" field on variants may not match with reality in the mirror case. You can see that the "repository" and "packages" fields for the "variant-External" variant which previously had both the repository and packages listed as "../rpm-signed/" now maps to "External" and "External/Packages", respectively. But in the event of mirror sync, this is incorrect.</li>
</ol>
<p>Likewise, the Land variant is wrong - but this may be a fixture issue. There is no such "Packages" directory. <a href="https://fixtures.pulpproject.org/rpm-distribution-tree/variants/land/" class="external">https://fixtures.pulpproject.org/rpm-distribution-tree/variants/land/</a></p>
<pre><code> ('change', 'variant-Land.packages', ('Packages', 'Land/Packages')),
('change', 'variant-Land.repository', ('variants/land', 'Land')),
('change',
'variant-External.packages',
('../rpm-signed/', 'External/Packages')),
('change', 'variant-External.repository', ('../rpm-signed/', 'External'))]
</code></pre>
<ol start="2">
<li>The order of the "variants" is different from the original - since productmd sometimes places significance on the first variant in the list, this could possibly have an impact in some cases.</li>
</ol>
<pre><code>[('change', 'general.variants', ('Land,Sea,External', 'External,Land,Sea')),
('change', 'tree.variants', ('Land,Sea,External', 'External,Land,Sea')),
</code></pre>
<p><code>test_publish.py::DistributionTreeMetadataTestCase</code> needs to be updated for both of these</p> RPM Support - Issue #8992 (CLOSED - DUPLICATE): Memory usage when destroy repository; The memory ...https://pulp.plan.io/issues/89922021-06-30T07:11:25Zwilful
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2280":<a href="https://github.com/pulp/pulp_rpm/issues/2280" class="external">https://github.com/pulp/pulp_rpm/issues/2280</a></p>
<hr>
<p>When i try to destroy one of repositories, pulp (RQ basically) are using more 3G RAM and crashed by OOM-Killer</p>
<pre><code class="c syntaxhl" data-language="c"><span class="n">Jun</span> <span class="mi">30</span> <span class="mo">06</span><span class="o">:</span><span class="mi">59</span><span class="o">:</span><span class="mi">16</span> <span class="o">***</span> <span class="n">kernel</span><span class="o">:</span> <span class="n">Out</span> <span class="n">of</span> <span class="n">memory</span><span class="o">:</span> <span class="n">Kill</span> <span class="n">process</span> <span class="mi">81458</span> <span class="p">(</span><span class="n">rq</span><span class="p">)</span> <span class="n">score</span> <span class="mi">493</span> <span class="n">or</span> <span class="n">sacrifice</span> <span class="n">child</span>
<span class="n">Jun</span> <span class="mi">30</span> <span class="mo">06</span><span class="o">:</span><span class="mi">59</span><span class="o">:</span><span class="mi">16</span> <span class="o">***</span> <span class="n">kernel</span><span class="o">:</span> <span class="n">Killed</span> <span class="n">process</span> <span class="mi">81458</span> <span class="p">(</span><span class="n">rq</span><span class="p">),</span> <span class="n">UID</span> <span class="mi">1002</span><span class="p">,</span> <span class="n">total</span><span class="o">-</span><span class="n">vm</span><span class="o">:</span><span class="mi">4434448</span><span class="n">kB</span><span class="p">,</span> <span class="n">anon</span><span class="o">-</span><span class="n">rss</span><span class="o">:</span><span class="mi">3936104</span><span class="n">kB</span><span class="p">,</span> <span class="n">file</span><span class="o">-</span><span class="n">rss</span><span class="o">:</span><span class="mi">0</span><span class="n">kB</span><span class="p">,</span> <span class="n">shmem</span><span class="o">-</span><span class="n">rss</span><span class="o">:</span><span class="mi">0</span><span class="n">kB</span>
</code></pre>
<p>State before on my node:</p>
<pre><code class="c syntaxhl" data-language="c"><span class="cp"># free -h
</span> <span class="n">total</span> <span class="n">used</span> <span class="n">free</span> <span class="n">shared</span> <span class="n">buff</span><span class="o">/</span><span class="n">cache</span> <span class="n">available</span>
<span class="n">Mem</span><span class="o">:</span> <span class="mi">7</span><span class="p">.</span><span class="mi">6</span><span class="n">G</span> <span class="mi">1</span><span class="p">.</span><span class="mi">5</span><span class="n">G</span> <span class="mi">3</span><span class="p">.</span><span class="mi">6</span><span class="n">G</span> <span class="mi">2</span><span class="p">.</span><span class="mi">2</span><span class="n">G</span> <span class="mi">2</span><span class="p">.</span><span class="mi">6</span><span class="n">G</span> <span class="mi">3</span><span class="p">.</span><span class="mi">7</span><span class="n">G</span>
<span class="n">Swap</span><span class="o">:</span> <span class="mi">0</span><span class="n">B</span> <span class="mi">0</span><span class="n">B</span> <span class="mi">0</span><span class="n">B</span>
</code></pre>
<pre><code class="c syntaxhl" data-language="c"> <span class="s">"versions"</span><span class="o">:</span> <span class="p">[</span>
<span class="p">{</span>
<span class="s">"component"</span><span class="o">:</span> <span class="s">"core"</span><span class="p">,</span>
<span class="s">"version"</span><span class="o">:</span> <span class="s">"3.13.0"</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="s">"component"</span><span class="o">:</span> <span class="s">"rpm"</span><span class="p">,</span>
<span class="s">"version"</span><span class="o">:</span> <span class="s">"3.13.0"</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="s">"component"</span><span class="o">:</span> <span class="s">"python"</span><span class="p">,</span>
<span class="s">"version"</span><span class="o">:</span> <span class="s">"3.4.0"</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="s">"component"</span><span class="o">:</span> <span class="s">"file"</span><span class="p">,</span>
<span class="s">"version"</span><span class="o">:</span> <span class="s">"1.8.0"</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="s">"component"</span><span class="o">:</span> <span class="s">"deb"</span><span class="p">,</span>
<span class="s">"version"</span><span class="o">:</span> <span class="s">"2.13.0"</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="s">"component"</span><span class="o">:</span> <span class="s">"container"</span><span class="p">,</span>
<span class="s">"version"</span><span class="o">:</span> <span class="s">"2.6.0"</span>
<span class="p">}</span>
<span class="p">],</span>
</code></pre> RPM Support - Test #8809 (CLOSED - DUPLICATE): Better tests for metadata mirroringhttps://pulp.plan.io/issues/88092021-05-24T19:54:30Zdalleydalley@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2277":<a href="https://github.com/pulp/pulp_rpm/issues/2277" class="external">https://github.com/pulp/pulp_rpm/issues/2277</a></p>
<hr>
<p>We need a fixture repository with some of the extra files, such as repomd.xml.asc (metadata signature), extra_files.json, .treeinfo, possibly licenses, multiple package directories / package locations, extra repomd entries that Pulp doesn't natively care about, etc. And then we need to test that mirroring works properly with such repos.</p> RPM Support - Issue #8720 (CLOSED - DUPLICATE): Published RPM metadata isn't sorted properlyhttps://pulp.plan.io/issues/87202021-05-07T22:09:20Zdalleydalley@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2274":<a href="https://github.com/pulp/pulp_rpm/issues/2274" class="external">https://github.com/pulp/pulp_rpm/issues/2274</a></p>
<hr>
<p>RPM metadata should be published in-order, which helps with compression efficency (see associated BZ). createrepo_c does this, but not via the library itself, so Pulp is still publishing unordered metadata.</p>
<p>Note that the metadata is "fine", it works, it's just inefficient to compress.</p>
<p>createrepo_c uses location_href as the sort key.</p>
<p>Problem: Pulp mixes location_href's together from many different repositories, and because they are meaningless, it basically ignores them. So we store useless data in the database.</p>
<p>We should remove the location_href and location_base fields (the latter is entirely unused), and replace them with just a filename, which we can possibly use to reconstruct a location_href if we need to keep it for backwards compatibility. Then we can properly sort by it, and we can use it directly in various places without needing to rewrite the value constantly.</p>
<p>It is not a "real" part of the RPM package metadata, only a value which createrepo_c happens to provide on the objects which we copied over. This probably shouldn't have been done.</p>
<p>We should therefore sort by the filename, which is basically equivalent to sorting by location_href since within a repository all the packages should have the same directory.</p> RPM Support - Story #8673 (CLOSED - DUPLICATE): Auto-publishing should be more fault-toleranthttps://pulp.plan.io/issues/86732021-04-30T14:56:08Zsskracic@redhat.comsskracic@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2273":<a href="https://github.com/pulp/pulp_rpm/issues/2273" class="external">https://github.com/pulp/pulp_rpm/issues/2273</a></p>
<hr>
<p>I admit the title is a bit vague.</p>
<p>During auto-publishing sync of a very large repository (rhel-7-server-rpms), the <code>rq</code> process got killed by oom-killer sometime in the middle of the publishing step. So the new repository version (1) got created, but accompanying
publication did not.</p>
<p>On the subsequent sync runs, the repository did not get published, as no new content was available to sync, hence a new repository version was not created, which in turn should trigger publication and distribution update.</p>
<p>Of course, the repo can still be published and distributed using the non-autopublishing REST API, but I wonder whether the behavior was intended.</p> RPM Support - Test #8335 (CLOSED - DUPLICATE): Need a new fixture - advisory, same date/version, ...https://pulp.plan.io/issues/83352021-03-03T20:04:27Zggainey
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2268":<a href="https://github.com/pulp/pulp_rpm/issues/2268" class="external">https://github.com/pulp/pulp_rpm/issues/2268</a></p>
<hr>
<p><a href="https://pulp.plan.io/issues/8249" class="external">https://pulp.plan.io/issues/8249</a> changed the failure-case in advisory.resolve_advisory_conflict - need a fixture to reflect the failure-case, and an update to test_sync.py to re-enable test_sync_advisory_incomplete_pgk_list</p> RPM Support - Issue #8229 (CLOSED - NOTABUG): Do not replace advisory if only 'updated_date' has ...https://pulp.plan.io/issues/82292021-02-09T07:18:23Zadam.winberg@smhi.se
<p>It seems that EPEL sets a new 'updated_date' every day, so when I run my daily sync I get:</p>
<pre><code> "content_summary": {
"added": {
"rpm.advisory": {
"count": 2540,
"href": "/pulp/api/v3/content/rpm/advisories/?repository_version_added=/pulp/api/v3/repositories/rpm/rpm/10e51ae6-65c7-42aa-8ab1-ffebdf752500/versions/19/"
},
"rpm.package": {
"count": 6,
"href": "/pulp/api/v3/content/rpm/packages/?repository_version_added=/pulp/api/v3/repositories/rpm/rpm/10e51ae6-65c7-42aa-8ab1-ffebdf752500/versions/19/"
}
},
"removed": {
"rpm.advisory": {
"count": 2536,
"href": "/pulp/api/v3/content/rpm/advisories/I've tested this and it works?repository_version_removed=/pulp/api/v3/repositories/rpm/rpm/10e51ae6-65c7-42aa-8ab1-ffebdf752500/versions/19/"
}
}
},
</code></pre>
<p>Inspecting the advisories, the only thing changed is the 'updated_date' metadata, version and package lists are identical. EPEL seems to set a new 'updated_date' each day on existing advisories, and in such a case there is not much sense in replacing the existing advisory with the 'new' one. I run weekly scripts to copy new advisories to a frozen repo which is a lot harder to do when thousands of new advisories are coming in every day.</p>
<p>I looked at the code ("pulp_rpm/app/advisory.py") and propose that an additional test is made in the "resolve_advisory_conflict" function, a crude example:</p>
<pre><code> elif not same_dates and pkgs_intersection:
if same_version and previous_pkglist == added_pkglist:
# Ignore new advisory if only 'updated_date' has changed
to_exclude.append(added_advisory.pk)
</code></pre> RPM Support - Issue #7283 (CLOSED - NOTABUG): new ruby bindings send default checksum typeshttps://pulp.plan.io/issues/72832020-08-05T17:53:53Zjsherril@redhat.comjsherril@redhat.com
<p>new bindings send a default checksum type when creating a publication.</p>
<p>Previously the 3.4 ruby bindings when creating a publication resulted in a publication request body of:</p>
<p>{"repository_version":"/pulp/api/v3/repositories/rpm/rpm/03b20f89-a2a3-4a02-8ea5-9af641623d15/versions/0/"}</p>
<p>after updating to pulp_rpm_client (3.6.0.dev01596557138), we now get a request body of:</p>
<p>{"repository_version":"/pulp/api/v3/repositories/rpm/rpm/7fcae337-d94c-4422-bd35-295943f48533/versions/0/","metadata_checksum_type":"sha256","package_checksum_type":"sha256"}</p>
<p>This might be fine, but i wanted to make sure it was intentional.</p> RPM Support - Test #6605 (CLOSED - DUPLICATE): Re-enable test_sync_advisory_no_updated_datehttps://pulp.plan.io/issues/66052020-04-29T18:37:26Zppicka
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2245":<a href="https://github.com/pulp/pulp_rpm/issues/2245" class="external">https://github.com/pulp/pulp_rpm/issues/2245</a></p>
<hr>
<p>If advisory has same ID, version but update_date missing sync will fail.</p>
<p>If updated_date missing in advisory issued_date is not take into consideration in advisory conflict resolution time.</p>
<p>How to reproduce:</p>
<pre><code>git clone https://github.com/pulp/pulp-fixtures
cd pulp-fixtures
# create fixtures
make fixtures/rpm-unsigned
make fixtures/rpm-advisory-no-update-date
</code></pre>
<ol>
<li>create repository</li>
<li>sync remote rpm-unsigned</li>
<li>re-sync same repo with rpm-advisory-no-update-date remote</li>
</ol>
<pre><code>"error" {
"description": "'<' not supported between instances of 'datetime.datetime' and 'NoneType'"
...}
</code></pre> RPM Support - Test #6425 (CLOSED - DUPLICATE): Support compressed and uncompressed version of mod...https://pulp.plan.io/issues/64252020-03-31T19:50:25Zfao89
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2240":<a href="https://github.com/pulp/pulp_rpm/issues/2240" class="external">https://github.com/pulp/pulp_rpm/issues/2240</a></p> RPM Support - Test #6349 (CLOSED - DUPLICATE): Test that upload of RPM with large filelists does ...https://pulp.plan.io/issues/63492020-03-16T19:33:49Zfao89
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2236":<a href="https://github.com/pulp/pulp_rpm/issues/2236" class="external">https://github.com/pulp/pulp_rpm/issues/2236</a></p>
<hr>
<ol>
<li>Create an RPM repo.</li>
<li>Verify whether the file <code>RPM_LARGE_METADATA</code> can be uploaded into the repo without errors.</li>
</ol>
<p>RPM_LARGE_METADATA = <a href="https://repos.fedorapeople.org/pulp/pulp/rpm_large_metadata/nodejs-babel-preset-es2015-6.6.0-2.el6.noarch.rpm" class="external">https://repos.fedorapeople.org/pulp/pulp/rpm_large_metadata/nodejs-babel-preset-es2015-6.6.0-2.el6.noarch.rpm</a></p> RPM Support - Story #5740 (CLOSED - CURRENTRELEASE): As a user, advisory collection names are uni...https://pulp.plan.io/issues/57402019-11-18T11:56:24Zttereshcttereshc@redhat.com
<a name="Background"></a>
<h3 >Background<a href="#Background" class="wiki-anchor">¶</a></h3>
<p>As a part of <a class="issue tracker-3 status-11 priority-6 priority-default closed" title="Story: As a user, a repository version has no advisories with the same id (CLOSED - CURRENTRELEASE)" href="https://pulp.plan.io/issues/4295">#4295</a>, two advisories can be merged into a new one which contains combined package list.</p>
<a name="Motivation"></a>
<h3 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h3>
<p>Each package list consists of collections (usually 1 , but can be 0 or 2+ as well). Each collection has a name which is expected to be unique by dnf/yum client. If the name is not unique, only a subset of packages will be updated when advisory is applied on a user system.</p>
<a name="Solution"></a>
<h3 >Solution<a href="#Solution" class="wiki-anchor">¶</a></h3>
<p>Ensure the uniqueness of collection names at the time of combining two advisories into one.<br>
If names are unique, preserve original names.<br>
If names are the same, make them unique (e.g. append an ordinal number to the existing name)</p> RPM Support - Story #5356 (CLOSED - CURRENTRELEASE): As a user, I can download a configuration fo...https://pulp.plan.io/issues/53562019-08-27T12:28:24Zdkliban@redhat.com
<p>It is common for repositories to include a repo file that can be used to add the repository to a client. This file can be manually placed into /etc/yum.repos.d/ or it can be specified as a parameter to yum/dnf. e.g.</p>
<pre><code>dnf config-manager --add-repo http://example.com/pulp/content/some/repo/path/config.repo
</code></pre>