Pulp: Issueshttps://pulp.plan.io/https://pulp.plan.io/favicon.ico2021-07-29T09:20:22ZPulp
Planio Debian Support - Task #9175 (CLOSED - CURRENTRELEASE): Backport 9162 to 2.14https://pulp.plan.io/issues/91752021-07-29T09:20:22Zquba42
<p><a href="https://github.com/pulp/community/discussions/59" class="external">https://github.com/pulp/community/discussions/59</a></p> Debian Support - Task #9162 (CLOSED - CURRENTRELEASE): Comply with orphan clean changes made in p...https://pulp.plan.io/issues/91622021-07-28T08:52:57Zquba42
<p><a href="https://github.com/pulp/community/discussions/59" class="external">https://github.com/pulp/community/discussions/59</a></p> Debian Support - Task #8682 (CLOSED - CURRENTRELEASE): Migrate to new Distribution model for pulp...https://pulp.plan.io/issues/86822021-05-03T12:21:35Zquba42Debian Support - Task #8388 (CLOSED - CURRENTRELEASE): Handle ALLOWED_CONTENT_CHECKSUMS settinghttps://pulp.plan.io/issues/83882021-03-11T14:59:24Zquba42
<p>Right now the pulp_deb plugin is dependend on the following hashs being present on artifacts: md5, sha1, sha256, sha512.</p>
<p>Starting with pulpcore 3.11 will exclude md5 and sha1 from ALLOWED_CONTENT_CHECKSUMS by default.</p>
<p>Pulp deb will need to handle this gracefully. There is still some debate on whether pulp_deb makes sense without md5 feature wise. As a result, the solution may need to include demanding or encouraging a different configuration from pulp_deb users than the pulpcore default configuration.</p>
<p>See the following mailing list threads:</p>
<ul>
<li><a href="https://listman.redhat.com/archives/pulp-dev/2021-February/msg00017.html" class="external">https://listman.redhat.com/archives/pulp-dev/2021-February/msg00017.html</a></li>
<li><a href="https://listman.redhat.com/archives/pulp-dev/2021-March/msg00013.html" class="external">https://listman.redhat.com/archives/pulp-dev/2021-March/msg00013.html</a></li>
</ul> Pulp - Test #4359 (CLOSED - COMPLETE): 2.18.1 Testinghttps://pulp.plan.io/issues/43592019-01-29T16:02:43Zbherring
<a name="Notes"></a>
<h2 >Notes<a href="#Notes" class="wiki-anchor">¶</a></h2>
<p>Testing Coverage for 2.18.1 will consist of the following</p>
<ul>
<li>Migration/Upgrade Testing
<ul>
<li>2.17.0 --> 2.18.1
<ul>
<li>FIPS</li>
<li>Non-FIPS</li>
</ul>
</li>
</ul>
</li>
<li>2.18.1 Functional Testing Suite
<ul>
<li>FIPS</li>
<li>Non-FIPS</li>
</ul>
</li>
</ul> Pulp - Test #4357 (CLOSED - COMPLETE): Pulp 2.19 master pulled epel django instead of pulp 2 djangohttps://pulp.plan.io/issues/43572019-01-29T12:41:38Zbherring
<a name="Problem"></a>
<h2 >Problem<a href="#Problem" class="wiki-anchor">¶</a></h2>
<p>When running nightly rhel76-fips testing, there were 500 errors [0] on jenkins tests since 1/23 and issues in journalctl with django [1].</p>
<p>After talking with asmacdo and daviddavis, it appears this may be a django being pulled from EPEL and not from our Pulp 2 repos on build [2][3]</p>
<a name="Testing"></a>
<h2 >Testing<a href="#Testing" class="wiki-anchor">¶</a></h2>
<p>It was determined this was the result of EPEL django precedence over the pulp.repo version that was built. This issue has been seen before.</p>
<p>Forced installation of the <strong>python2-django-1.11.17-1</strong> from the fedorapeople repo [3] and restart of services (or with a hammer of <strong>shutdown now -r</strong>) resulted in a functioning pulp install.</p>
<pre><code>[root@rhel76-fips ~]# journalctl -f
-- Logs begin at Mon 2019-01-28 12:33:13 EST. --
Jan 28 12:34:38 rhel76-fips pulp[5046]: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Jan 28 12:34:38 rhel76-fips pulp[5046]: django.request:ERROR: (5046-27424) Internal Server Error: /pulp/api/v2/status/
Jan 28 12:34:38 rhel76-fips pulp[5046]: django.request:ERROR: (5046-27424) Traceback (most recent call last):
Jan 28 12:34:38 rhel76-fips pulp[5046]: django.request:ERROR: (5046-27424) File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 131, in get_response
Jan 28 12:34:38 rhel76-fips pulp[5046]: django.request:ERROR: (5046-27424) response = middleware_method(request, response)
Jan 28 12:34:38 rhel76-fips pulp[5046]: django.request:ERROR: (5046-27424) File "/usr/lib/python2.7/site-packages/django/middleware/http.py", line 23, in process_response
Jan 28 12:34:38 rhel76-fips pulp[5046]: django.request:ERROR: (5046-27424) set_response_etag(response)
Jan 28 12:34:38 rhel76-fips pulp[5046]: django.request:ERROR: (5046-27424) File "/usr/lib/python2.7/site-packages/django/utils/cache.py", line 109, in set_response_etag
Jan 28 12:34:38 rhel76-fips pulp[5046]: django.request:ERROR: (5046-27424) response['ETag'] = quote_etag(hashlib.md5(response.content).hexdigest())
Jan 28 12:34:38 rhel76-fips pulp[5046]: django.request:ERROR: (5046-27424) ValueError: error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips
</code></pre>
<a name="Actions"></a>
<h2 >Actions<a href="#Actions" class="wiki-anchor">¶</a></h2>
<p>Pulp-dev determined this issue as a low priority issue where documentation exercise may happen as a result of this issue.</p>
<p>For QE, this is a critical breakage of CI as the ansible install fails on FIPS installation and puts the box into a non-testable state. QE will be opening a related Urgent test tracker to fix the issue in the pulp-ci installer to fix mission critical work.</p>
<a name="References"></a>
<h2 >References<a href="#References" class="wiki-anchor">¶</a></h2>
<p>[0] - <a href="https://paste.fedoraproject.org/paste/bPRtYPa4ZCLShrZ7VUMqMQ" class="external">https://paste.fedoraproject.org/paste/bPRtYPa4ZCLShrZ7VUMqMQ</a><br>
[1] - <a href="https://pulp-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/Pulp%202%20-%20Master/job/pulp-2-master-dev-rhel7-fips/228/" class="external">https://pulp-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/Pulp%202%20-%20Master/job/pulp-2-master-dev-rhel7-fips/228/</a><br>
[2] - <a href="https://paste.fedoraproject.org/paste/6eHTS1Uc3NkC0hohvKe-TA" class="external">https://paste.fedoraproject.org/paste/6eHTS1Uc3NkC0hohvKe-TA</a> [from the affected 2.19a1-nightly box under test<br>
[3] - 2.11.17-1 <a href="https://repos.fedorapeople.org/repos/pulp/pulp/v2/testing/automation/2.19/stage/7Server/x86_64/" class="external">https://repos.fedorapeople.org/repos/pulp/pulp/v2/testing/automation/2.19/stage/7Server/x86_64/</a><br>
[4] - <a href="https://wiki.centos.org/PackageManagement/Yum/Priorities" class="external">https://wiki.centos.org/PackageManagement/Yum/Priorities</a></p> RPM Support - Test #4351 (CLOSED - COMPLETE): modules.yaml reference in repomd.xml does not use s...https://pulp.plan.io/issues/43512019-01-24T14:13:07Zbherring
<p>A report from upstream katello (<a href="https://projects.theforeman.org/issues/25529" class="external">https://projects.theforeman.org/issues/25529</a>) indicates that pulp generates a repomd.xml file that claims to use the checksum type that has been set on the publisher, but in fact is not using that:</p>
<p>In this example it purports to have used a sha1 checksum, but in fact its actually a sha256 checksum:</p>
<pre><code> <data type="modules">
<location href="repodata/824ffe238f202a0612ecbb2b0c0459dc289a7ef47adb1f26406453d41f476449-modules.yaml.gz"/>
<timestamp>1542811864</timestamp>
<size>33</size>
<checksum type="sha1">824ffe238f202a0612ecbb2b0c0459dc289a7ef47adb1f26406453d41f476449</checksum>
<open-size>0</open-size>
<open-checksum type="sha1">da39a3ee5e6b4b0d3255bfef95601890afd80709</open-checksum>
</data>
</code></pre>
<p>To reproduce:</p>
<p>1. create and sync a yum repository<br>
2. set a checksum type of sha1<br>
3. publish the repository</p> RPM Support - Test #4350 (CLOSED - COMPLETE): modules.yaml file is generated on repository with n...https://pulp.plan.io/issues/43502019-01-24T14:12:25Zbherring
<p>It seems that syncing and publishing a repository without any modularity info, still results in a modules.yaml being generated.</p>
<p>See: <a href="https://projects.theforeman.org/issues/25529" class="external">https://projects.theforeman.org/issues/25529</a> for more information</p> RPM Support - Test #4269 (CLOSED - COMPLETE): Recursive and conservative recursive copyhttps://pulp.plan.io/issues/42692018-12-14T17:36:04Zmilan
<a name="Description"></a>
<h2 >Description<a href="#Description" class="wiki-anchor">¶</a></h2>
<p>With the <a href="https://github.com/pulp/pulp_rpm/pull/1226" class="external">fix</a> for the Issue <a class="issue tracker-1 status-11 priority-6 priority-default closed" title="Issue: Regression Pulp 2.17.1: recursive copy of RPMs does not copy partially resolvable dependencies (CLOSED - CURRENTRELEASE)" href="https://pulp.plan.io/issues/4152">#4152</a>, the default behaviour of recursive copy changes back to the state before 2.17, rendering the test case <a href="https://github.com/PulpQE/Pulp-2-Tests/blob/c9277928c2788bced8d0ae3f630a46b6162fc4c9/pulp_2_tests/tests/rpm/api_v2/test_rich_weak_dependencies.py#L183" class="external">CopyRecursiveUnitsTestCase</a> broken.<br>
The behaviour specified in the <a href="https://github.com/PulpQE/pulp-smash/issues/1090" class="external">pulp smash issue 1090</a>, as requested in the Issue <a class="issue tracker-3 status-11 priority-6 priority-default closed" title="Story: Support more conservative dependency solving (CLOSED - CURRENTRELEASE)" href="https://pulp.plan.io/issues/2478">#2478</a>, is now optional and can be triggered by providing the <code>"recursive_conservative": true</code> config override instead of the default <code>"recursive": true</code>.<br>
With that setting, the test case <a href="https://github.com/PulpQE/Pulp-2-Tests/blob/c9277928c2788bced8d0ae3f630a46b6162fc4c9/pulp_2_tests/tests/rpm/api_v2/test_rich_weak_dependencies.py#L183" class="external">CopyRecursiveUnitsTestCase</a> should agin pass as is.</p>
<a name="Proposed-solution"></a>
<h2 >Proposed solution<a href="#Proposed-solution" class="wiki-anchor">¶</a></h2>
<p>Split the test case <a href="https://github.com/PulpQE/Pulp-2-Tests/blob/c9277928c2788bced8d0ae3f630a46b6162fc4c9/pulp_2_tests/tests/rpm/api_v2/test_rich_weak_dependencies.py#L183" class="external">CopyRecursiveUnitsTestCase</a> such that it asserts:</p>
<ul>
<li>a failure in the terms described in <a href="https://github.com/PulpQE/pulp-smash/issues/1090" class="external">pulp smash issue 1090</a> with the default <code>"recursive": true</code> config override setting</li>
<li>a success once the non-default <code>"recursive_conservative": true</code> config override setting is provided</li>
</ul> Pulp - Test #4258 (CLOSED - COMPLETE): Publishing incorrect branch head.https://pulp.plan.io/issues/42582018-12-11T13:33:31Zbherring
<a name="Notes"></a>
<h2 >Notes<a href="#Notes" class="wiki-anchor">¶</a></h2>
<ul>
<li>ostree smash test? ping jeff</li>
</ul> Python Support - Refactor #4132 (CLOSED - NOTABUG): Metadata is not downloaded in parallelhttps://pulp.plan.io/issues/41322018-11-07T17:01:10Zamacdona@redhat.comaustin@redhat.com
<p>Metadata for the python plugin is all retrieved in the PythonFirstStage[0]. Project level metadata is downloaded in a for loop [1]. Since we are using asyncio, the `await` does cede control while it is downloading a metadata file, so after the first metadata file is downloaded and processed, the PythonFirstStage continues to download 1 metadata file at a time while all later stages from pulpcore continue.</p>
<p>My assumption is that downloading the project level metadata in parallel would be a relatively small performance improvement when each project (corresponding to 1 metadata file) has many Python distributions associated with it. However, for lazy sync or cases where each project has a small number of Python distributions, downloading the metadata in parallel could be a very large performance increase.</p>
<a name="Background"></a>
<h3 >Background<a href="#Background" class="wiki-anchor">¶</a></h3>
<p>Unless a Stage implements parallel calls with asyncio, each Stage only operates 1 at a time (or 1 batch at a time). The only pulpcore Stage that runs multiple calls in parallel is the ArtifactDownloader[1], which uses the ArtifactDownloaderRunner to ensure_futures[2], which is what enables many calls to happen simultaneously.[2]</p>
<a name="Design-Options"></a>
<h3 >Design Options<a href="#Design-Options" class="wiki-anchor">¶</a></h3>
<p>We can download the metadata in parallel in at least 2 ways.</p>
<ol>
<li>Split up the PythonFirstStage. Even though the Python plugin doesn't have a "Project" Content unit, we can still create DeclarativeContent objects that can flow to later stages.
<ol>
<li>ProjectListStage - Rather than downloading metadata in the first stage[3], we could create a DeclarativeContent object for each project, and out_q.put(dc).</li>
<li>ArtifactDownloader (from core, unchanged) - This would download (in parallel) each of the project metadata files.</li>
<li>ProcessProjectMetadata -This Stage would open and read the project dc.d_artifact file, and create dcs for PythonPackageContent, and out_q.put(python_package_content_dc). It would <strong>not</strong> continue to pass project dcs down the pipeline. This would roughly correspond to part of the first stage[4], but would also need to include package filtering[5].</li>
<li>Suggested Pipeline: ProjectListStage -> ArtifactDownloader -> ProcessProjectMetadata -> ArtifactDownloader -> save artifacts, save content,etc</li>
</ol>
</li>
<li>It's probably also possible to implement parallel downloads directly into a monolithic first stage by using asyncio.ensure_future, but IMO this should not be idiomatic use of the Stages API. It would be complex and would require knowledge of asyncio that is not really expected of plugin writers. Besides, the tricky part of this is already implemented by the ArtifactDownloader.</li>
</ol>
<p>[0]: <a href="https://github.com/pulp/pulp_python/blob/master/pulp_python/app/tasks/sync.py#L57" class="external">https://github.com/pulp/pulp_python/blob/master/pulp_python/app/tasks/sync.py#L57</a><br>
[1]: <a href="https://github.com/pulp/pulp/blob/master/plugin/pulpcore/plugin/stages/artifact_stages.py#L215" class="external">https://github.com/pulp/pulp/blob/master/plugin/pulpcore/plugin/stages/artifact_stages.py#L215</a><br>
[2]: <a href="https://github.com/pulp/pulp/blob/master/plugin/pulpcore/plugin/stages/artifact_stages.py#L121-L132" class="external">https://github.com/pulp/pulp/blob/master/plugin/pulpcore/plugin/stages/artifact_stages.py#L121-L132</a><br>
[3]: <a href="https://github.com/pulp/pulp_python/blob/master/pulp_python/app/tasks/sync.py#L96" class="external">https://github.com/pulp/pulp_python/blob/master/pulp_python/app/tasks/sync.py#L96</a><br>
[4]: <a href="https://github.com/pulp/pulp_python/blob/master/pulp_python/app/tasks/sync.py#L121-L129" class="external">https://github.com/pulp/pulp_python/blob/master/pulp_python/app/tasks/sync.py#L121-L129</a><br>
[5]: <a href="https://github.com/pulp/pulp_python/blob/master/pulp_python/app/tasks/sync.py#L151" class="external">https://github.com/pulp/pulp_python/blob/master/pulp_python/app/tasks/sync.py#L151</a></p> Pulp - Story #3778 (CLOSED - CURRENTRELEASE): [Epic] As a user, I can run Pulp 3 in a FIPS-enable...https://pulp.plan.io/issues/37782018-06-21T16:13:49ZdaviddavisPackaging - Task #2228 (CLOSED - CURRENTRELEASE): Improve nodepool node launch demand calculationhttps://pulp.plan.io/issues/22282016-09-02T19:12:45Zsemyerssean.myers@redhat.com
<p>Because of a needed tweak to nodepool[0] we already running a fork. After chatting with elyezer, there are some further improvements I'd like to include on the fork.</p>
<p>Currently, nodepoold uses a tool called "geard" to determine node launch demand, based on the configure label demands of queued jobs. I believe that when nodepool was first being used, this information was not obtainable via the Jenkins API. Now, however, it is[1]. Furthermore, the geard integration simply doesn't work, afaict, for our use-case, unless we want to go nuts and bring in more pieces of openstack automation (e.g. zuul).</p>
<p>I've already done some preliminary work[2] to try to determine label demand based on the current jenkins build queue, so all that remains is wiring it into the nodepool fork and getting rid of the geard stuff that doesn't meet our needs.</p>
<p>[0]: <a href="https://github.com/seandst/nodepool/commit/1a807c8def36d62e80aaaa0e4e961a176e9ce746" class="external">https://github.com/seandst/nodepool/commit/1a807c8def36d62e80aaaa0e4e961a176e9ce746</a><br>
[1]: <a href="https://python-jenkins.readthedocs.io/en/latest/api.html#jenkins.Jenkins.get_queue_info" class="external">https://python-jenkins.readthedocs.io/en/latest/api.html#jenkins.Jenkins.get_queue_info</a><br>
[2]: <a href="https://gist.github.com/seandst/8e9b62f7b46bb5baa88a8bf5fbb11215" class="external">https://gist.github.com/seandst/8e9b62f7b46bb5baa88a8bf5fbb11215</a></p> Pulp - Task #1941 (CLOSED - CURRENTRELEASE): Strict/nitpicky sphinx settings break our rpm buildshttps://pulp.plan.io/issues/19412016-05-23T22:40:42Zsemyerssean.myers@redhat.com
<p>This is a post-mortem report of the task of diagnosing and fixing broken builds.</p>
<hr>
<p>In <a class="issue tracker-2 status-11 priority-6 priority-default closed" title="Task: Consolidate platform and plugin docs into a static site. (CLOSED - CURRENTRELEASE)" href="https://pulp.plan.io/issues/950">#950</a>, we turned on strict docs builds, but this ended up breaking our builds.</p>
<p>The problem was that there's no internet access inside the mock buildroot, so when sphinx went to download the intersphinx inventory files it would fail. Normally this triggers a warning, but enabling strict mode promotes all build warnings to errors. It's worth pointing out that the intersphinx downloads have always been failing during RPM build, the only thing new was our strictness.</p>
<p>Strict building is good; at the moment work is ongoing in <a class="issue tracker-2 status-11 priority-6 priority-default closed" title="Task: Consolidate platform and plugin docs into a static site. (CLOSED - CURRENTRELEASE)" href="https://pulp.plan.io/issues/950">#950</a> to modify our docs build process to hopefully (among many other goals) allow us to once again do strict builds with sphinx without breaking the build.</p> Pulp - Refactor #9 (CLOSED - DUPLICATE): As a developer, I would like /etc/pulp/repo_auth.conf to...https://pulp.plan.io/issues/92014-12-18T16:12:34ZAnonymous
<p>Some of the repo auth code in pulp_rpm but should be in platform so other plugins can use it without depending on pulp_rpm being installed.This was originally related to <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1140823" class="external">https://bugzilla.redhat.com/show_bug.cgi?id=1140823</a></p>