Pulp: Issueshttps://pulp.plan.io/https://pulp.plan.io/favicon.ico2021-12-09T18:59:35ZPulp
Planio RPM Support - Issue #9627 (MODIFIED): publish fails on MD5-checksummed repos, on FIPShttps://pulp.plan.io/issues/96272021-12-09T18:59:35Zggainey
<p>See associated BZ for details, reproducer</p> RPM Support - Test #9622 (MODIFIED): Add a repo signed using 'sha' as alias for 'sha1'https://pulp.plan.io/issues/96222021-12-08T19:00:00Zggainey
<p>'sha' support exists in the wild, is the same as 'sha1', and has broken us several times now, Let's make it possible to write tests for it.</p> RPM Support - Issue #9559 (MODIFIED): Unable to sync Fedora 35 repositoryhttps://pulp.plan.io/issues/95592021-11-08T02:09:29Zhyu
<p>Cloned from <a href="https://bugzilla.redhat.com/show_bug.cgi?id=2020473" class="external">https://bugzilla.redhat.com/show_bug.cgi?id=2020473</a></p>
<h2>Description of problem:
When publishing the Fedora 35 repository, Pulp failed with the following error:</h2>
<h2>celery.app.trace:ERROR: [56fc62c9] (24505-91520) Task pulp.server.managers.repo.publish.publish[56fc62c9-29b0-41bf-a820-ba3625ef4a00] raised unexpected: TemplateSyntaxError(u'Empty variable tag on line 548',)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) Traceback (most recent call last):
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
celery.app.trace:ERROR: [56fc62c9] (24505-91520) R = retval = fun(*args, **kwargs)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 688, in <strong>call</strong>
celery.app.trace:ERROR: [56fc62c9] (24505-91520) return super(Task, self).<strong>call</strong>(*args, **kwargs)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 110, in <strong>call</strong>
celery.app.trace:ERROR: [56fc62c9] (24505-91520) return super(PulpTask, self).<strong>call</strong>(*args, **kwargs)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in <strong>protected_call</strong>
celery.app.trace:ERROR: [56fc62c9] (24505-91520) return self.run(*args, **kwargs)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1142, in publish
celery.app.trace:ERROR: [56fc62c9] (24505-91520) result = check_publish(repo_obj, dist_id, dist_inst, transfer_repo, conduit, call_config)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1251, in check_publish
celery.app.trace:ERROR: [56fc62c9] (24505-91520) result = _do_publish(repo_obj, dist_id, dist_inst, transfer_repo, conduit, call_config)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1303, in _do_publish
celery.app.trace:ERROR: [56fc62c9] (24505-91520) publish_report = publish_repo(transfer_repo, conduit, call_config)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 901, in wrap_f
celery.app.trace:ERROR: [56fc62c9] (24505-91520) return f(*args, **kwargs)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/distributor.py", line 185, in publish_repo
celery.app.trace:ERROR: [56fc62c9] (24505-91520) return self._publisher.process_lifecycle()
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 573, in process_lifecycle
celery.app.trace:ERROR: [56fc62c9] (24505-91520) super(PluginStep, self).process_lifecycle()
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 164, in process_lifecycle
celery.app.trace:ERROR: [56fc62c9] (24505-91520) step.process()
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 240, in process
celery.app.trace:ERROR: [56fc62c9] (24505-91520) self._process_block(item=item)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 302, in _process_block
celery.app.trace:ERROR: [56fc62c9] (24505-91520) self.process_main(item=item)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/publish.py", line 500, in process_main
celery.app.trace:ERROR: [56fc62c9] (24505-91520) context.add_unit_metadata(unit)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/metadata/filelists.py", line 42, in add_unit_metadata
celery.app.trace:ERROR: [56fc62c9] (24505-91520) self.metadata_file_handle.write(unit.render_filelists(self.checksum_type))
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/db/models.py", line 869, in render_filelists
celery.app.trace:ERROR: [56fc62c9] (24505-91520) return self._render(metadata, context)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/db/models.py", line 884, in _render
celery.app.trace:ERROR: [56fc62c9] (24505-91520) t = Template(template)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/django/template/base.py", line 191, in <strong>init</strong>
celery.app.trace:ERROR: [56fc62c9] (24505-91520) self.nodelist = self.compile_nodelist()
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/django/template/base.py", line 230, in compile_nodelist
celery.app.trace:ERROR: [56fc62c9] (24505-91520) return parser.parse()
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/django/template/base.py", line 482, in parse
celery.app.trace:ERROR: [56fc62c9] (24505-91520) raise self.error(token, 'Empty variable tag on line %d' % token.lineno)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) TemplateSyntaxError: Empty variable tag on line 548</h2>
<p>This is because one of the rpm in the repo has as "{{" filename causing the Django template syntax error.</p>
<p>In "6dc8dae1be904c2613d5aa3667dacd2554d05077eb1ce4296b6edfa3c4db3a46-filelists.xml.gz"
/usr/lib/.build-id
/usr/lib/.build-id/42
/usr/lib/.build-id/42/5049f5a9e0d1ac25c21de795f28fe886bfa0ca
/usr/lib64/R/library/rlang
/usr/lib64/R/library/rlang/DESCRIPTION
/usr/lib64/R/library/rlang/INDEX
/usr/lib64/R/library/rlang/LICENSE
/usr/lib64/R/library/rlang/Meta
snip...
/usr/lib64/R/library/rlang/help/wref_key.html
/usr/lib64/R/library/rlang/help/wref_value.html
/usr/lib64/R/library/rlang/help/zap.html
/usr/lib64/R/library/rlang/help/zap_srcref.html
/usr/lib64/R/library/rlang/help/{{.html <================= TemplateSyntaxError: Empty variable tag on line 548\n",
/usr/lib64/R/library/rlang/help/{{}}.html <=================</p>
<p>$ rpm -qlp R-rlang-0.4.11-3.fc35.x86_64.rpm | grep "{{"
/usr/lib64/R/library/rlang/help/{{.html
/usr/lib64/R/library/rlang/help/{{}}.html</p>
<p>help]$ cat {{.html</p>
<p>help]$ cat {{}}.html</p>
<p>Steps to Reproduce:</p>
<ol>
<li>Create a custom repository call fedora 35 and add the following feed url</li>
</ol>
<p><a href="https://dl.fedoraproject.org/pub/fedora/linux/releases/35/Everything/x86_64/os/" class="external">https://dl.fedoraproject.org/pub/fedora/linux/releases/35/Everything/x86_64/os/</a></p>
<ol start="2">
<li>Sync the repo</li>
</ol>
<p>Actual results:
Failed to publish repo</p>
<p>Expected results:
Repo can be published successfully</p>
<p>Additional info:
Tested on Satellite 6.10 and the repository can be synced successfully without issue.</p> RPM Support - Issue #9553 (MODIFIED): Publishing repository with large metadata may consume high ...https://pulp.plan.io/issues/95532021-11-03T06:10:34Zhyu
<p>Pulp is consuming high memory when publishing RHEL 7 repository. This is happening when Pulp is calculating the checksum of the metadata. It reads the whole metadata file into memory at once to calculates the checksum. For example, the other.xml.gz (compressed ) for RHEL 7 repository is about 837MB size. Reading the entire file into memory will cause Pulp worker to consume more than 1GB for RAM.</p>
<p><a href="https://github.com/pulp/pulp/blob/2-master/server/pulp/plugins/util/metadata_writer.py#L99-L101" class="external">https://github.com/pulp/pulp/blob/2-master/server/pulp/plugins/util/metadata_writer.py#L99-L101</a>.</p>
<p>How to reproduce:</p>
<ol>
<li>Sync the RHEL 7 repository.</li>
<li>After that manually force full publish it and run the below command to observe the memory usage.</li>
</ol>
<p>watch -n 1 'ps -aux | grep resource_worker</p>
<ol start="3">
<li>The memory usage should be stable between 200MB to 350MB all the time, but will suddenly go up to about 1.1GB for about 3 seconds (around finalizing the publish rpms step) then back to 200MB+.</li>
</ol> RPM Support - Task #9435 (MODIFIED): Update ACS docs to use the CLIhttps://pulp.plan.io/issues/94352021-09-23T13:22:02Zppicka
<p>ssia</p> RPM Support - Task #9422 (CLOSED - COMPLETE): Create a fixture repo that has only metadatahttps://pulp.plan.io/issues/94222021-09-17T16:19:34Zipanova@redhat.comipanova@redhat.com
<p>ssia</p> RPM Support - Story #9358 (MODIFIED): As a user I can use Alternate Content Sourceshttps://pulp.plan.io/issues/93582021-09-08T16:40:29Zppicka
<p>Add refresh endpoint with slightly adjust synchronize method and first stage to support ACS.</p> RPM Support - Task #9091 (CLOSED - COMPLETE): Test that as a user I can use RHUI as a remote and/...https://pulp.plan.io/issues/90912021-07-15T13:52:41Zppicka
<p>Setup a private RHUI and confirm Pulp can set it up as ACS</p> RPM Support - Issue #9021 (MODIFIED): No such file or directory: '/var/lib/pulp/sync_imports/test...https://pulp.plan.io/issues/90212021-07-06T18:41:26Ziballou
<p>Pulp should not error out when repomd.xml exists but repomd.xml.asc doesn't. This error occurred when syncing from the filesystem with the following patch applied to Pulpcore 3.14: <a href="https://github.com/pulp/pulpcore/pull/1463.patch" class="external">https://github.com/pulp/pulpcore/pull/1463.patch</a></p>
<p>Occurred on pulp-rpm (3.13.2)</p> RPM Support - Issue #8893 (MODIFIED): 3rd party repository sync fails with 'InvalidStringData: st...https://pulp.plan.io/issues/88932021-06-14T14:14:36Zggainey
<p>ModulemdDefaults are BSON-encoded before being saved into MongoDB because MongoDB (apparently) has restrictions on valid keys which are incompatible with the module data we need to store.</p>
<p><a href="https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/importers/yum/repomd/modules.py#L94-L95" class="external">https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/importers/yum/repomd/modules.py#L94-L95</a></p>
<p>...But the serialized BSON strings being created by the encoding function we are using are not always valid UTF-8... Presumably, it works the vast majority of the time, enough so that it wasn't noticed.</p>
<p>Only specific permutations of the input data appear to trigger this. If I delete key from the profiles dictionary, all of a sudden it's UTF-8 compatible. This is presumably why the problem pops into and out of existence.</p>
<p>At some point we attempt to save this string to MongoDB, and Mongo decides to apply a UTF-8 validation to it, and it blows up.</p> RPM Support - Issue #8890 (MODIFIED): Publishing a repository can take longer time to finish if m...https://pulp.plan.io/issues/88902021-06-14T06:00:43Zhyu
<p>Description of problem:
Pulp can take more than an hour to publish a repository when a large number of repositories have been synced from upstream and same errata are existed in the synced repositories, such as RHEL 7.x, RHEL 7 EUS and different aches.</p>
<p>The more repositories have the same errata the more "erratum pkglist" entries will be created in the mongodb which can cause the performance degradation.</p>
<p>For example:</p>
<blockquote>
<p>db.erratum_pkglists.find({errata_id: "RHSA-2018:2557"}).count()
387
db.erratum_pkglists.find({errata_id: "RHBA-2019:2180"}).count()
217</p>
</blockquote>
<p>When publishing errata, Pulp will use the above query to get all package lists of the errata. This will take long time to process when they are many package lists returned by the query and each package list is consist of many packages.</p>
<p>As we can see below, the "Publish Errata" step is very slow. 53 minutes has passed, it has only processed about 2073 errata. It will take more than an hour to finish.
...
{
"description": "Publishing Errata",
"details": "",
"error_details": [],
"items_total": 4789,
"num_failures": 0,
"num_processed": 2073,
"num_success": 2073,
"state": "IN_PROGRESS",
"step_id": "2f09190d-013a-4300-9445-eccb52ad94fe",
"step_type": "errata"
},
...
"start_time": "2021-06-09T12:41:13Z",</p>
<a name="date"></a>
<h1 >date<a href="#date" class="wiki-anchor">¶</a></h1>
<p>Wed Jun 9 13:32:41 UTC 2021</p> RPM Support - Task #8841 (CLOSED - COMPLETE): Release 3.11.1https://pulp.plan.io/issues/88412021-05-31T09:59:06Zttereshcttereshc@redhat.comRPM Support - Task #8780 (CLOSED - COMPLETE): Release 3.12.0https://pulp.plan.io/issues/87802021-05-19T13:46:26Zttereshcttereshc@redhat.comRPM Support - Task #8771 (CLOSED - COMPLETE): Release 3.11.0https://pulp.plan.io/issues/87712021-05-18T11:22:40Zttereshcttereshc@redhat.comRPM Support - Test #7350 (MODIFIED): Test syncing from a repository located on a local diskhttps://pulp.plan.io/issues/73502020-08-19T15:40:07Zdalleydalley@redhat.com
<p>e.g. a repository with a file:// url such as file:///var/lib/pulp/sync_imports/test_repos/zoo/</p>