Pulp: Issueshttps://pulp.plan.io/https://pulp.plan.io/favicon.ico2021-12-09T18:59:35ZPulp
Planio RPM Support - Issue #9627 (MODIFIED): publish fails on MD5-checksummed repos, on FIPShttps://pulp.plan.io/issues/96272021-12-09T18:59:35Zggainey
<p>See associated BZ for details, reproducer</p> RPM Support - Test #9622 (MODIFIED): Add a repo signed using 'sha' as alias for 'sha1'https://pulp.plan.io/issues/96222021-12-08T19:00:00Zggainey
<p>'sha' support exists in the wild, is the same as 'sha1', and has broken us several times now, Let's make it possible to write tests for it.</p> RPM Support - Issue #9559 (MODIFIED): Unable to sync Fedora 35 repositoryhttps://pulp.plan.io/issues/95592021-11-08T02:09:29Zhyu
<p>Cloned from <a href="https://bugzilla.redhat.com/show_bug.cgi?id=2020473" class="external">https://bugzilla.redhat.com/show_bug.cgi?id=2020473</a></p>
<h2>Description of problem:
When publishing the Fedora 35 repository, Pulp failed with the following error:</h2>
<h2>celery.app.trace:ERROR: [56fc62c9] (24505-91520) Task pulp.server.managers.repo.publish.publish[56fc62c9-29b0-41bf-a820-ba3625ef4a00] raised unexpected: TemplateSyntaxError(u'Empty variable tag on line 548',)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) Traceback (most recent call last):
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
celery.app.trace:ERROR: [56fc62c9] (24505-91520) R = retval = fun(*args, **kwargs)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 688, in <strong>call</strong>
celery.app.trace:ERROR: [56fc62c9] (24505-91520) return super(Task, self).<strong>call</strong>(*args, **kwargs)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 110, in <strong>call</strong>
celery.app.trace:ERROR: [56fc62c9] (24505-91520) return super(PulpTask, self).<strong>call</strong>(*args, **kwargs)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in <strong>protected_call</strong>
celery.app.trace:ERROR: [56fc62c9] (24505-91520) return self.run(*args, **kwargs)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1142, in publish
celery.app.trace:ERROR: [56fc62c9] (24505-91520) result = check_publish(repo_obj, dist_id, dist_inst, transfer_repo, conduit, call_config)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1251, in check_publish
celery.app.trace:ERROR: [56fc62c9] (24505-91520) result = _do_publish(repo_obj, dist_id, dist_inst, transfer_repo, conduit, call_config)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1303, in _do_publish
celery.app.trace:ERROR: [56fc62c9] (24505-91520) publish_report = publish_repo(transfer_repo, conduit, call_config)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 901, in wrap_f
celery.app.trace:ERROR: [56fc62c9] (24505-91520) return f(*args, **kwargs)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/distributor.py", line 185, in publish_repo
celery.app.trace:ERROR: [56fc62c9] (24505-91520) return self._publisher.process_lifecycle()
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 573, in process_lifecycle
celery.app.trace:ERROR: [56fc62c9] (24505-91520) super(PluginStep, self).process_lifecycle()
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 164, in process_lifecycle
celery.app.trace:ERROR: [56fc62c9] (24505-91520) step.process()
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 240, in process
celery.app.trace:ERROR: [56fc62c9] (24505-91520) self._process_block(item=item)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 302, in _process_block
celery.app.trace:ERROR: [56fc62c9] (24505-91520) self.process_main(item=item)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/publish.py", line 500, in process_main
celery.app.trace:ERROR: [56fc62c9] (24505-91520) context.add_unit_metadata(unit)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/metadata/filelists.py", line 42, in add_unit_metadata
celery.app.trace:ERROR: [56fc62c9] (24505-91520) self.metadata_file_handle.write(unit.render_filelists(self.checksum_type))
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/db/models.py", line 869, in render_filelists
celery.app.trace:ERROR: [56fc62c9] (24505-91520) return self._render(metadata, context)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/db/models.py", line 884, in _render
celery.app.trace:ERROR: [56fc62c9] (24505-91520) t = Template(template)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/django/template/base.py", line 191, in <strong>init</strong>
celery.app.trace:ERROR: [56fc62c9] (24505-91520) self.nodelist = self.compile_nodelist()
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/django/template/base.py", line 230, in compile_nodelist
celery.app.trace:ERROR: [56fc62c9] (24505-91520) return parser.parse()
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/django/template/base.py", line 482, in parse
celery.app.trace:ERROR: [56fc62c9] (24505-91520) raise self.error(token, 'Empty variable tag on line %d' % token.lineno)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) TemplateSyntaxError: Empty variable tag on line 548</h2>
<p>This is because one of the rpm in the repo has as "{{" filename causing the Django template syntax error.</p>
<p>In "6dc8dae1be904c2613d5aa3667dacd2554d05077eb1ce4296b6edfa3c4db3a46-filelists.xml.gz"
/usr/lib/.build-id
/usr/lib/.build-id/42
/usr/lib/.build-id/42/5049f5a9e0d1ac25c21de795f28fe886bfa0ca
/usr/lib64/R/library/rlang
/usr/lib64/R/library/rlang/DESCRIPTION
/usr/lib64/R/library/rlang/INDEX
/usr/lib64/R/library/rlang/LICENSE
/usr/lib64/R/library/rlang/Meta
snip...
/usr/lib64/R/library/rlang/help/wref_key.html
/usr/lib64/R/library/rlang/help/wref_value.html
/usr/lib64/R/library/rlang/help/zap.html
/usr/lib64/R/library/rlang/help/zap_srcref.html
/usr/lib64/R/library/rlang/help/{{.html <================= TemplateSyntaxError: Empty variable tag on line 548\n",
/usr/lib64/R/library/rlang/help/{{}}.html <=================</p>
<p>$ rpm -qlp R-rlang-0.4.11-3.fc35.x86_64.rpm | grep "{{"
/usr/lib64/R/library/rlang/help/{{.html
/usr/lib64/R/library/rlang/help/{{}}.html</p>
<p>help]$ cat {{.html</p>
<p>help]$ cat {{}}.html</p>
<p>Steps to Reproduce:</p>
<ol>
<li>Create a custom repository call fedora 35 and add the following feed url</li>
</ol>
<p><a href="https://dl.fedoraproject.org/pub/fedora/linux/releases/35/Everything/x86_64/os/" class="external">https://dl.fedoraproject.org/pub/fedora/linux/releases/35/Everything/x86_64/os/</a></p>
<ol start="2">
<li>Sync the repo</li>
</ol>
<p>Actual results:
Failed to publish repo</p>
<p>Expected results:
Repo can be published successfully</p>
<p>Additional info:
Tested on Satellite 6.10 and the repository can be synced successfully without issue.</p> RPM Support - Issue #9553 (MODIFIED): Publishing repository with large metadata may consume high ...https://pulp.plan.io/issues/95532021-11-03T06:10:34Zhyu
<p>Pulp is consuming high memory when publishing RHEL 7 repository. This is happening when Pulp is calculating the checksum of the metadata. It reads the whole metadata file into memory at once to calculates the checksum. For example, the other.xml.gz (compressed ) for RHEL 7 repository is about 837MB size. Reading the entire file into memory will cause Pulp worker to consume more than 1GB for RAM.</p>
<p><a href="https://github.com/pulp/pulp/blob/2-master/server/pulp/plugins/util/metadata_writer.py#L99-L101" class="external">https://github.com/pulp/pulp/blob/2-master/server/pulp/plugins/util/metadata_writer.py#L99-L101</a>.</p>
<p>How to reproduce:</p>
<ol>
<li>Sync the RHEL 7 repository.</li>
<li>After that manually force full publish it and run the below command to observe the memory usage.</li>
</ol>
<p>watch -n 1 'ps -aux | grep resource_worker</p>
<ol start="3">
<li>The memory usage should be stable between 200MB to 350MB all the time, but will suddenly go up to about 1.1GB for about 3 seconds (around finalizing the publish rpms step) then back to 200MB+.</li>
</ol> RPM Support - Task #9435 (MODIFIED): Update ACS docs to use the CLIhttps://pulp.plan.io/issues/94352021-09-23T13:22:02Zppicka
<p>ssia</p> RPM Support - Story #9358 (MODIFIED): As a user I can use Alternate Content Sourceshttps://pulp.plan.io/issues/93582021-09-08T16:40:29Zppicka
<p>Add refresh endpoint with slightly adjust synchronize method and first stage to support ACS.</p> RPM Support - Issue #9021 (MODIFIED): No such file or directory: '/var/lib/pulp/sync_imports/test...https://pulp.plan.io/issues/90212021-07-06T18:41:26Ziballou
<p>Pulp should not error out when repomd.xml exists but repomd.xml.asc doesn't. This error occurred when syncing from the filesystem with the following patch applied to Pulpcore 3.14: <a href="https://github.com/pulp/pulpcore/pull/1463.patch" class="external">https://github.com/pulp/pulpcore/pull/1463.patch</a></p>
<p>Occurred on pulp-rpm (3.13.2)</p> RPM Support - Issue #8893 (MODIFIED): 3rd party repository sync fails with 'InvalidStringData: st...https://pulp.plan.io/issues/88932021-06-14T14:14:36Zggainey
<p>ModulemdDefaults are BSON-encoded before being saved into MongoDB because MongoDB (apparently) has restrictions on valid keys which are incompatible with the module data we need to store.</p>
<p><a href="https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/importers/yum/repomd/modules.py#L94-L95" class="external">https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/importers/yum/repomd/modules.py#L94-L95</a></p>
<p>...But the serialized BSON strings being created by the encoding function we are using are not always valid UTF-8... Presumably, it works the vast majority of the time, enough so that it wasn't noticed.</p>
<p>Only specific permutations of the input data appear to trigger this. If I delete key from the profiles dictionary, all of a sudden it's UTF-8 compatible. This is presumably why the problem pops into and out of existence.</p>
<p>At some point we attempt to save this string to MongoDB, and Mongo decides to apply a UTF-8 validation to it, and it blows up.</p> RPM Support - Issue #8890 (MODIFIED): Publishing a repository can take longer time to finish if m...https://pulp.plan.io/issues/88902021-06-14T06:00:43Zhyu
<p>Description of problem:
Pulp can take more than an hour to publish a repository when a large number of repositories have been synced from upstream and same errata are existed in the synced repositories, such as RHEL 7.x, RHEL 7 EUS and different aches.</p>
<p>The more repositories have the same errata the more "erratum pkglist" entries will be created in the mongodb which can cause the performance degradation.</p>
<p>For example:</p>
<blockquote>
<p>db.erratum_pkglists.find({errata_id: "RHSA-2018:2557"}).count()
387
db.erratum_pkglists.find({errata_id: "RHBA-2019:2180"}).count()
217</p>
</blockquote>
<p>When publishing errata, Pulp will use the above query to get all package lists of the errata. This will take long time to process when they are many package lists returned by the query and each package list is consist of many packages.</p>
<p>As we can see below, the "Publish Errata" step is very slow. 53 minutes has passed, it has only processed about 2073 errata. It will take more than an hour to finish.
...
{
"description": "Publishing Errata",
"details": "",
"error_details": [],
"items_total": 4789,
"num_failures": 0,
"num_processed": 2073,
"num_success": 2073,
"state": "IN_PROGRESS",
"step_id": "2f09190d-013a-4300-9445-eccb52ad94fe",
"step_type": "errata"
},
...
"start_time": "2021-06-09T12:41:13Z",</p>
<a name="date"></a>
<h1 >date<a href="#date" class="wiki-anchor">¶</a></h1>
<p>Wed Jun 9 13:32:41 UTC 2021</p> RPM Support - Test #7350 (MODIFIED): Test syncing from a repository located on a local diskhttps://pulp.plan.io/issues/73502020-08-19T15:40:07Zdalleydalley@redhat.com
<p>e.g. a repository with a file:// url such as file:///var/lib/pulp/sync_imports/test_repos/zoo/</p> RPM Support - Task #3746 (CLOSED - COMPLETE): Investigate, whether exposing the @pool->considered...https://pulp.plan.io/issues/37462018-06-08T10:19:31Zmilan
<p>It might be the case that to support the functionality described in issue <a class="issue tracker-3 status-11 priority-6 priority-default closed parent" title="Story: Implement modularity content dependency solving (CLOSED - CURRENTRELEASE)" href="https://pulp.plan.io/issues/3740">#3740</a>, exposing the <code>pool->considered</code> bitmap needn't be necessary.</p>
<p>Considering the <a href="https://github.com/dparalen/pulp_solv" class="external">POC</a>, it might be sufficient to just ignore the target repository content while building the pool to resolve the copy query.</p> RPM Support - Task #3528 (CLOSED - COMPLETE): Investigate which library to use for dep solvinghttps://pulp.plan.io/issues/35282018-03-27T12:35:33Zttereshcttereshc@redhat.com
<p>Current implementation of a dep solving supports only strong (<code>Requires:</code>) simple (<code>package_name</code>) dependencies.<br>
In order to support strong (<code>Requires:</code>) rich (aka <a href="http://rpm.org/user_doc/boolean_dependencies.html" class="external">boolean</a>) dependencies an external library should be used.<br>
The suggestion is to drop the current implementation of dep solving and for any kind of a dep solving start using:<br>
- either <code>libdnf</code> via its Python interface (python2-dnf)<br>
- or <code>libsolv</code> via its Python bindings (python2-solv)</p>
<p><code>libdnf</code> uses <code>libsolv</code> and exposes a higher level API to a user.</p>
<p>Both <code>libdnf</code> and <code>libsolv</code> requires <code>repodata</code> to perform dep solving.<br>
Some temporary working directory should be provided to a solver to store its cache there.</p>
<p>Functionality of a solver we are looking for:</p>
<ul>
<li>list dependencies for a given rpm package (can be a part of a synchronous task)</li>
<li>list fulfilled dependencies (from a given list of available packages) for a given rpm package</li>
<li>validate that all the dependencies (from a given list of available packages) for a given rpm package are fulfilled.</li>
<li>perform dep solving against multiple repositories</li>
</ul>
<p>The actions mentioned above will be a part of an asynchronous task (see a copy example below).</p>
<p>An example of the dep solving use case in Pulp is a <code>recursive</code> flag during a copy: <a href="https://docs.pulpproject.org/plugins/pulp_rpm/user-guide/recipes.html?highlight=recursive#copy-errata-from-one-repository-to-another" class="external">https://docs.pulpproject.org/plugins/pulp_rpm/user-guide/recipes.html?highlight=recursive#copy-errata-from-one-repository-to-another</a><br>
Dep solving related functionality to support: <a href="https://pulp.plan.io/issues/2478" class="external">https://pulp.plan.io/issues/2478</a></p>
<p>Example usage of the <code>libsolv</code> Python bindings: <a href="https://github.com/fedora-modularity/depchase" class="external">https://github.com/fedora-modularity/depchase</a><br>
Ping ttereshc for more details about examples for <code>libdnf</code> case.</p>
<p>Figure out which library supports all required functionality. Keep in mind that performance is important as well. There could be repositories with 100K+ packages against which dep solving can be requested.</p> RPM Support - Task #2937 (CLOSED - COMPLETE): Add pep8speakshttps://pulp.plan.io/issues/29372017-07-24T13:45:04ZdaviddavisRPM Support - Task #2702 (CLOSED - COMPLETE): Install Plugins into Pulp 3 development envhttps://pulp.plan.io/issues/27022017-04-12T18:13:42Zsemyerssean.myers@redhat.com
<p>I <a href="https://github.com/pulp/devel/commit/3a49e1325e0ea7fac3281b510a45b78493196bc8#diff-c674e6be1d4fd8bed7f77bad58146d18" class="external">recently added a task</a> to the 3.0 dev playbook to install pulp_file into the platform virtualenv on install. As seen in the comments on that block, this is a bit of a placeholder just to get pulp_file into platform so that you can do fun things with the platform API.</p>
<p>We need to come up with a mechanism that ansible can easily use for installing plugins for Pulp 3. Maybe this is continuing to rely on each plugin providing a "pulp-dev.py" executable, maybe we require all plugins to be installable with "python setup.py develop"; whatever we go with would ideally be consistent across plugins.</p>
<p>In addition to installing plugins into the platform virtualenv for use together, we should also make sure to install plugins into their <em>own</em> virtualenvs, allowing them to be worked on in isolation from other plugins.</p> RPM Support - Task #2266 (CLOSED - COMPLETE): Figure out how to test Pulp 3 Platform and Pluginshttps://pulp.plan.io/issues/22662016-09-19T13:17:16Zsemyerssean.myers@redhat.com
<p>In trying to tackle <a class="issue tracker-2 status-11 priority-6 priority-default closed" title="Task: Fix bash aliases for pulp 3 w/ postgres (CLOSED - CURRENTRELEASE)" href="https://pulp.plan.io/issues/2194">#2194</a>, I started to go down a little bit of a rabbit hole related to getting the "ptests" alias to work. Each plugin has its own "run-tests.py", which brings in test runner code from pulp.devel, and each plugin then does something a little bit different when it comes to running its tests, with some running pep257, and other minor inconsistencies.</p>
<p>I'm not sure how to deal with this, but have some ideas:</p>
<ul>
<li>pulp.devel (currently devel in the pulp platform repo, packaged as pulp-devel RPM) should go away, and common dev-related lib code should live in the pulp_dev lib from the devel repo</li>
<li>Platform and plugins should register an executable, either as a pulp-dev subcommand or as a standalone executable, to handle running pulp's tests with uniform standards across plugins</li>
<li>Django introduces a new dimension to testing (<code>python manage.py test</code>, specifically), with the plugins exposed as apps that depend on the platform app that must be taken into account when testing.</li>
<li>Travis CI should run the same set of acceptance tests that developers do prior to committing</li>
</ul>
<p>Crazy idea:<br>
Assuming we officially adopt a less coverage-oriented testing scheme (probatio gratia probationis? I'm not sure we have an issue open about this yet), we could potentially opt to run every installed plugin's tests (plus platform) when testing PRs instead of having a different tester for every plugin. Even if we do have thorough unit-testing with a 100% coverage goal, the Django tests should be written to <em>run quickly</em>.</p>