Pulp: Issueshttps://pulp.plan.io/https://pulp.plan.io/favicon.ico2021-08-19T14:09:05ZPulp
Planio RPM Support - Task #9259 (CLOSED - DUPLICATE): workflow-docs should use pulp-cli instead of httpiehttps://pulp.plan.io/issues/92592021-08-19T14:09:05Zggainey
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2314":<a href="https://github.com/pulp/pulp_rpm/issues/2314" class="external">https://github.com/pulp/pulp_rpm/issues/2314</a></p>
<hr>
<p>Probably shouldn't do this unless/until <strong>all</strong> workflows are supported by pulp-cli commands - mixing pulp-cli and httpie is Very Confusing to the user.</p>
<p>Putting this up as a placeholder for docs-day</p> RPM Support - Issue #9233 (CLOSED - DUPLICATE): Downloaded content seems to be removed when a tas...https://pulp.plan.io/issues/92332021-08-11T22:27:01Zlmjachky
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2294":<a href="https://github.com/pulp/pulp_rpm/issues/2294" class="external">https://github.com/pulp/pulp_rpm/issues/2294</a></p>
<hr>
<p>For large repositories, this means that when performing <strong>immediate</strong> synchronization, all the downloaded content is lost after the task's failure. Pulp is then trying to download the content (artifacts) once again from scratch when issuing re-syncing, rather than associating the orphaned artifacts with corresponding content units.</p>
<p>More info can be found here: <a href="https://community.theforeman.org/t/oracle-linux-8-appstream-sync-fails/24676/9" class="external">https://community.theforeman.org/t/oracle-linux-8-appstream-sync-fails/24676/9</a>. The following part is the most important to us:</p>
<blockquote>
<p>If you are performing one of these large syncs, and get part way through before the sync fails with the timeout, it seems everything which was downloaded into the tmp directory is thrown away, and not kept. So, the next time the sync is tried, it has to download everything again.</p>
</blockquote>
<p>Used versions: pulpcore 3.14.3 and pulp_rpm 3.14.0</p>
<p>This behaviour could not be reproduced for pulp_file (<a href="https://fixtures.pulpproject.org/file/" class="external">https://fixtures.pulpproject.org/file/</a>; pulpcore 3.15.0.dev and pulp_file 1.9.0.dev); therefore, we should focus on the pulp_rpm plugin.</p> RPM Support - Issue #9208 (CLOSED - DUPLICATE): Published .treeinfo metadata not matching expecta...https://pulp.plan.io/issues/92082021-08-04T04:48:45Zdalleydalley@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2292":<a href="https://github.com/pulp/pulp_rpm/issues/2292" class="external">https://github.com/pulp/pulp_rpm/issues/2292</a></p>
<hr>
<ol>
<li>"packages" field on variants may not match with reality in the mirror case. You can see that the "repository" and "packages" fields for the "variant-External" variant which previously had both the repository and packages listed as "../rpm-signed/" now maps to "External" and "External/Packages", respectively. But in the event of mirror sync, this is incorrect.</li>
</ol>
<p>Likewise, the Land variant is wrong - but this may be a fixture issue. There is no such "Packages" directory. <a href="https://fixtures.pulpproject.org/rpm-distribution-tree/variants/land/" class="external">https://fixtures.pulpproject.org/rpm-distribution-tree/variants/land/</a></p>
<pre><code> ('change', 'variant-Land.packages', ('Packages', 'Land/Packages')),
('change', 'variant-Land.repository', ('variants/land', 'Land')),
('change',
'variant-External.packages',
('../rpm-signed/', 'External/Packages')),
('change', 'variant-External.repository', ('../rpm-signed/', 'External'))]
</code></pre>
<ol start="2">
<li>The order of the "variants" is different from the original - since productmd sometimes places significance on the first variant in the list, this could possibly have an impact in some cases.</li>
</ol>
<pre><code>[('change', 'general.variants', ('Land,Sea,External', 'External,Land,Sea')),
('change', 'tree.variants', ('Land,Sea,External', 'External,Land,Sea')),
</code></pre>
<p><code>test_publish.py::DistributionTreeMetadataTestCase</code> needs to be updated for both of these</p> Migration Plugin - Issue #9191 (CLOSED - DUPLICATE): Migrated advisories with dates-as-timestamps...https://pulp.plan.io/issues/91912021-08-02T19:37:16Zggainey
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp-2to3-migration/issues/483": <a href="https://github.com/pulp/pulp-2to3-migration/issues/483" class="external">https://github.com/pulp/pulp-2to3-migration/issues/483</a></p> Pulp - Issue #9009 (CLOSED - DUPLICATE): CacheBasicOperationsTestCase::test_04_basic_expires unit...https://pulp.plan.io/issues/90092021-07-03T14:52:45Zdkliban@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulpcore/2023":<a href="https://github.com/pulp/pulpcore/issues/2023" class="external">https://github.com/pulp/pulpcore/issues/2023</a></p>
<hr>
<p>I am seeing the following in our CI from time to time:</p>
<pre><code># Run unit tests.
cmd_prefix bash -c "PULP_DATABASES__default__USER=postgres django-admin test --noinput /usr/local/lib/python3.6/site-packages/pulpcore/tests/unit/"
Creating test database for alias 'default'...
Access policy for groups/users created.
Access policy for groups created.
Access policy for tasks updated.
Deleting Guardians' AnonymousUser
System check identified no issues (0 silenced).
.............................F.......s.............................................pulp [None]: asyncio:ERROR: Task was destroyed but it is pending!
task: <Task pending coro=<ArtifactDownloader._handle_content_unit() running at /usr/local/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py:207> wait_for=<_GatheringFuture pending cb=[<TaskWakeupMethWrapper object at 0x7f8fd9a270d8>()]>>
pulp [None]: asyncio:ERROR: Task was destroyed but it is pending!
task: <Task pending coro=<ArtifactDownloader._handle_content_unit() running at /usr/local/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py:207> wait_for=<_GatheringFuture pending cb=[<TaskWakeupMethWrapper object at 0x7f8fd9a27468>()]>>
.............
======================================================================
FAIL: test_04_basic_expires (pulpcore.tests.unit.test_cache.CacheBasicOperationsTestCase)
Tests setting values with expiration times
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/pulpcore/tests/unit/test_cache.py", line 42, in test_04_basic_expires
self.assertIsNone(ret)
AssertionError: b'hi' is not None
----------------------------------------------------------------------
Ran 96 tests in 6.627s
</code></pre> RPM Support - Issue #8992 (CLOSED - DUPLICATE): Memory usage when destroy repository; The memory ...https://pulp.plan.io/issues/89922021-06-30T07:11:25Zwilful
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2280":<a href="https://github.com/pulp/pulp_rpm/issues/2280" class="external">https://github.com/pulp/pulp_rpm/issues/2280</a></p>
<hr>
<p>When i try to destroy one of repositories, pulp (RQ basically) are using more 3G RAM and crashed by OOM-Killer</p>
<pre><code class="c syntaxhl" data-language="c"><span class="n">Jun</span> <span class="mi">30</span> <span class="mo">06</span><span class="o">:</span><span class="mi">59</span><span class="o">:</span><span class="mi">16</span> <span class="o">***</span> <span class="n">kernel</span><span class="o">:</span> <span class="n">Out</span> <span class="n">of</span> <span class="n">memory</span><span class="o">:</span> <span class="n">Kill</span> <span class="n">process</span> <span class="mi">81458</span> <span class="p">(</span><span class="n">rq</span><span class="p">)</span> <span class="n">score</span> <span class="mi">493</span> <span class="n">or</span> <span class="n">sacrifice</span> <span class="n">child</span>
<span class="n">Jun</span> <span class="mi">30</span> <span class="mo">06</span><span class="o">:</span><span class="mi">59</span><span class="o">:</span><span class="mi">16</span> <span class="o">***</span> <span class="n">kernel</span><span class="o">:</span> <span class="n">Killed</span> <span class="n">process</span> <span class="mi">81458</span> <span class="p">(</span><span class="n">rq</span><span class="p">),</span> <span class="n">UID</span> <span class="mi">1002</span><span class="p">,</span> <span class="n">total</span><span class="o">-</span><span class="n">vm</span><span class="o">:</span><span class="mi">4434448</span><span class="n">kB</span><span class="p">,</span> <span class="n">anon</span><span class="o">-</span><span class="n">rss</span><span class="o">:</span><span class="mi">3936104</span><span class="n">kB</span><span class="p">,</span> <span class="n">file</span><span class="o">-</span><span class="n">rss</span><span class="o">:</span><span class="mi">0</span><span class="n">kB</span><span class="p">,</span> <span class="n">shmem</span><span class="o">-</span><span class="n">rss</span><span class="o">:</span><span class="mi">0</span><span class="n">kB</span>
</code></pre>
<p>State before on my node:</p>
<pre><code class="c syntaxhl" data-language="c"><span class="cp"># free -h
</span> <span class="n">total</span> <span class="n">used</span> <span class="n">free</span> <span class="n">shared</span> <span class="n">buff</span><span class="o">/</span><span class="n">cache</span> <span class="n">available</span>
<span class="n">Mem</span><span class="o">:</span> <span class="mi">7</span><span class="p">.</span><span class="mi">6</span><span class="n">G</span> <span class="mi">1</span><span class="p">.</span><span class="mi">5</span><span class="n">G</span> <span class="mi">3</span><span class="p">.</span><span class="mi">6</span><span class="n">G</span> <span class="mi">2</span><span class="p">.</span><span class="mi">2</span><span class="n">G</span> <span class="mi">2</span><span class="p">.</span><span class="mi">6</span><span class="n">G</span> <span class="mi">3</span><span class="p">.</span><span class="mi">7</span><span class="n">G</span>
<span class="n">Swap</span><span class="o">:</span> <span class="mi">0</span><span class="n">B</span> <span class="mi">0</span><span class="n">B</span> <span class="mi">0</span><span class="n">B</span>
</code></pre>
<pre><code class="c syntaxhl" data-language="c"> <span class="s">"versions"</span><span class="o">:</span> <span class="p">[</span>
<span class="p">{</span>
<span class="s">"component"</span><span class="o">:</span> <span class="s">"core"</span><span class="p">,</span>
<span class="s">"version"</span><span class="o">:</span> <span class="s">"3.13.0"</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="s">"component"</span><span class="o">:</span> <span class="s">"rpm"</span><span class="p">,</span>
<span class="s">"version"</span><span class="o">:</span> <span class="s">"3.13.0"</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="s">"component"</span><span class="o">:</span> <span class="s">"python"</span><span class="p">,</span>
<span class="s">"version"</span><span class="o">:</span> <span class="s">"3.4.0"</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="s">"component"</span><span class="o">:</span> <span class="s">"file"</span><span class="p">,</span>
<span class="s">"version"</span><span class="o">:</span> <span class="s">"1.8.0"</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="s">"component"</span><span class="o">:</span> <span class="s">"deb"</span><span class="p">,</span>
<span class="s">"version"</span><span class="o">:</span> <span class="s">"2.13.0"</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="s">"component"</span><span class="o">:</span> <span class="s">"container"</span><span class="p">,</span>
<span class="s">"version"</span><span class="o">:</span> <span class="s">"2.6.0"</span>
<span class="p">}</span>
<span class="p">],</span>
</code></pre> Pulp - Issue #8988 (CLOSED - CURRENTRELEASE): `pulpcore-worker` startup should remove old worker ...https://pulp.plan.io/issues/89882021-06-29T19:37:18Zbmbouterbmbouter@redhat.com
<a name="To-reproduce"></a>
<h2 >To reproduce<a href="#To-reproduce" class="wiki-anchor">¶</a></h2>
<ol>
<li>Start a pulpcore-worker against an empty database</li>
<li>Observe that the status API shows that worker</li>
<li>kill -9 your worker</li>
<li>Observe the status API after 30 seconds no longer shows your worker</li>
<li>Start your pulpcore-worker again</li>
<li>Go into shell_plus and observe this query shows 2 workers present: <code>Workers.objects.count()</code>
</li>
</ol>
<a name="Expected-result"></a>
<h2 >Expected result<a href="#Expected-result" class="wiki-anchor">¶</a></h2>
<pre><code>In [1]: Worker.objects.count()
Out[1]: 1
</code></pre>
<a name="Solution"></a>
<h2 >Solution<a href="#Solution" class="wiki-anchor">¶</a></h2>
<p>Have workers that startup run a query that delete any workers that haven't issued a heartbeat in say 7 days. This does not need to be configurable. The 7 day idea is to make something that won't let records accumulate on the long term, but leave them in place for someone to look at the db post-mortem and still see them for investigation.</p> RPM Support - Test #8809 (CLOSED - DUPLICATE): Better tests for metadata mirroringhttps://pulp.plan.io/issues/88092021-05-24T19:54:30Zdalleydalley@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2277":<a href="https://github.com/pulp/pulp_rpm/issues/2277" class="external">https://github.com/pulp/pulp_rpm/issues/2277</a></p>
<hr>
<p>We need a fixture repository with some of the extra files, such as repomd.xml.asc (metadata signature), extra_files.json, .treeinfo, possibly licenses, multiple package directories / package locations, extra repomd entries that Pulp doesn't natively care about, etc. And then we need to test that mirroring works properly with such repos.</p> RPM Support - Issue #8720 (CLOSED - DUPLICATE): Published RPM metadata isn't sorted properlyhttps://pulp.plan.io/issues/87202021-05-07T22:09:20Zdalleydalley@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2274":<a href="https://github.com/pulp/pulp_rpm/issues/2274" class="external">https://github.com/pulp/pulp_rpm/issues/2274</a></p>
<hr>
<p>RPM metadata should be published in-order, which helps with compression efficency (see associated BZ). createrepo_c does this, but not via the library itself, so Pulp is still publishing unordered metadata.</p>
<p>Note that the metadata is "fine", it works, it's just inefficient to compress.</p>
<p>createrepo_c uses location_href as the sort key.</p>
<p>Problem: Pulp mixes location_href's together from many different repositories, and because they are meaningless, it basically ignores them. So we store useless data in the database.</p>
<p>We should remove the location_href and location_base fields (the latter is entirely unused), and replace them with just a filename, which we can possibly use to reconstruct a location_href if we need to keep it for backwards compatibility. Then we can properly sort by it, and we can use it directly in various places without needing to rewrite the value constantly.</p>
<p>It is not a "real" part of the RPM package metadata, only a value which createrepo_c happens to provide on the objects which we copied over. This probably shouldn't have been done.</p>
<p>We should therefore sort by the filename, which is basically equivalent to sorting by location_href since within a repository all the packages should have the same directory.</p> RPM Support - Story #8673 (CLOSED - DUPLICATE): Auto-publishing should be more fault-toleranthttps://pulp.plan.io/issues/86732021-04-30T14:56:08Zsskracic@redhat.comsskracic@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2273":<a href="https://github.com/pulp/pulp_rpm/issues/2273" class="external">https://github.com/pulp/pulp_rpm/issues/2273</a></p>
<hr>
<p>I admit the title is a bit vague.</p>
<p>During auto-publishing sync of a very large repository (rhel-7-server-rpms), the <code>rq</code> process got killed by oom-killer sometime in the middle of the publishing step. So the new repository version (1) got created, but accompanying
publication did not.</p>
<p>On the subsequent sync runs, the repository did not get published, as no new content was available to sync, hence a new repository version was not created, which in turn should trigger publication and distribution update.</p>
<p>Of course, the repo can still be published and distributed using the non-autopublishing REST API, but I wonder whether the behavior was intended.</p> Debian Support - Task #8643 (CLOSED - DUPLICATE): Rework the API docshttps://pulp.plan.io/issues/86432021-04-28T07:07:15Zquba42
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_deb/405":<a href="https://github.com/pulp/pulp_deb/issues/405" class="external">https://github.com/pulp/pulp_deb/issues/405</a></p>
<hr>
<p>I would like to systematically go through the strings used for the API docs, to ensure they provide all the critical information for users.</p> Debian Support - Task #8641 (CLOSED - DUPLICATE): Review handling of ALLOWED_CONTENT_CHECKSUMS in...https://pulp.plan.io/issues/86412021-04-28T07:02:46Zquba42
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_deb/403":<a href="https://github.com/pulp/pulp_deb/issues/403" class="external">https://github.com/pulp/pulp_deb/issues/403</a></p>
<hr>
<p>Right now we are testing using checksum settings that are not what most production setups will use.
The current settings also require manual intervention in devel boxes to run tests locally.</p>
<p>Ideally we would have different tests using different settings for full coverage.</p> Debian Support - Issue #8544 (CLOSED - DUPLICATE): Add some translation files to at least one tes...https://pulp.plan.io/issues/85442021-04-12T15:36:57Zquba42
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_deb/401":<a href="https://github.com/pulp/pulp_deb/issues/401" class="external">https://github.com/pulp/pulp_deb/issues/401</a></p>
<hr>
<p>Should provide test coverage for the following:</p>
<ul>
<li><a href="https://pulp.plan.io/issues/8410" class="external">https://pulp.plan.io/issues/8410</a></li>
<li><a href="https://pulp.plan.io/issues/8096" class="external">https://pulp.plan.io/issues/8096</a></li>
</ul> RPM Support - Test #8335 (CLOSED - DUPLICATE): Need a new fixture - advisory, same date/version, ...https://pulp.plan.io/issues/83352021-03-03T20:04:27Zggainey
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2268":<a href="https://github.com/pulp/pulp_rpm/issues/2268" class="external">https://github.com/pulp/pulp_rpm/issues/2268</a></p>
<hr>
<p><a href="https://pulp.plan.io/issues/8249" class="external">https://pulp.plan.io/issues/8249</a> changed the failure-case in advisory.resolve_advisory_conflict - need a fixture to reflect the failure-case, and an update to test_sync.py to re-enable test_sync_advisory_incomplete_pgk_list</p> RPM Support - Issue #8229 (CLOSED - NOTABUG): Do not replace advisory if only 'updated_date' has ...https://pulp.plan.io/issues/82292021-02-09T07:18:23Zadam.winberg@smhi.se
<p>It seems that EPEL sets a new 'updated_date' every day, so when I run my daily sync I get:</p>
<pre><code> "content_summary": {
"added": {
"rpm.advisory": {
"count": 2540,
"href": "/pulp/api/v3/content/rpm/advisories/?repository_version_added=/pulp/api/v3/repositories/rpm/rpm/10e51ae6-65c7-42aa-8ab1-ffebdf752500/versions/19/"
},
"rpm.package": {
"count": 6,
"href": "/pulp/api/v3/content/rpm/packages/?repository_version_added=/pulp/api/v3/repositories/rpm/rpm/10e51ae6-65c7-42aa-8ab1-ffebdf752500/versions/19/"
}
},
"removed": {
"rpm.advisory": {
"count": 2536,
"href": "/pulp/api/v3/content/rpm/advisories/I've tested this and it works?repository_version_removed=/pulp/api/v3/repositories/rpm/rpm/10e51ae6-65c7-42aa-8ab1-ffebdf752500/versions/19/"
}
}
},
</code></pre>
<p>Inspecting the advisories, the only thing changed is the 'updated_date' metadata, version and package lists are identical. EPEL seems to set a new 'updated_date' each day on existing advisories, and in such a case there is not much sense in replacing the existing advisory with the 'new' one. I run weekly scripts to copy new advisories to a frozen repo which is a lot harder to do when thousands of new advisories are coming in every day.</p>
<p>I looked at the code ("pulp_rpm/app/advisory.py") and propose that an additional test is made in the "resolve_advisory_conflict" function, a crude example:</p>
<pre><code> elif not same_dates and pkgs_intersection:
if same_version and previous_pkglist == added_pkglist:
# Ignore new advisory if only 'updated_date' has changed
to_exclude.append(added_advisory.pk)
</code></pre>