Pulp: Issueshttps://pulp.plan.io/https://pulp.plan.io/favicon.ico2021-11-08T02:09:29ZPulp
Planio RPM Support - Issue #9559 (MODIFIED): Unable to sync Fedora 35 repositoryhttps://pulp.plan.io/issues/95592021-11-08T02:09:29Zhyu
<p>Cloned from <a href="https://bugzilla.redhat.com/show_bug.cgi?id=2020473" class="external">https://bugzilla.redhat.com/show_bug.cgi?id=2020473</a></p>
<h2>Description of problem:
When publishing the Fedora 35 repository, Pulp failed with the following error:</h2>
<h2>celery.app.trace:ERROR: [56fc62c9] (24505-91520) Task pulp.server.managers.repo.publish.publish[56fc62c9-29b0-41bf-a820-ba3625ef4a00] raised unexpected: TemplateSyntaxError(u'Empty variable tag on line 548',)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) Traceback (most recent call last):
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
celery.app.trace:ERROR: [56fc62c9] (24505-91520) R = retval = fun(*args, **kwargs)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 688, in <strong>call</strong>
celery.app.trace:ERROR: [56fc62c9] (24505-91520) return super(Task, self).<strong>call</strong>(*args, **kwargs)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 110, in <strong>call</strong>
celery.app.trace:ERROR: [56fc62c9] (24505-91520) return super(PulpTask, self).<strong>call</strong>(*args, **kwargs)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in <strong>protected_call</strong>
celery.app.trace:ERROR: [56fc62c9] (24505-91520) return self.run(*args, **kwargs)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1142, in publish
celery.app.trace:ERROR: [56fc62c9] (24505-91520) result = check_publish(repo_obj, dist_id, dist_inst, transfer_repo, conduit, call_config)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1251, in check_publish
celery.app.trace:ERROR: [56fc62c9] (24505-91520) result = _do_publish(repo_obj, dist_id, dist_inst, transfer_repo, conduit, call_config)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1303, in _do_publish
celery.app.trace:ERROR: [56fc62c9] (24505-91520) publish_report = publish_repo(transfer_repo, conduit, call_config)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 901, in wrap_f
celery.app.trace:ERROR: [56fc62c9] (24505-91520) return f(*args, **kwargs)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/distributor.py", line 185, in publish_repo
celery.app.trace:ERROR: [56fc62c9] (24505-91520) return self._publisher.process_lifecycle()
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 573, in process_lifecycle
celery.app.trace:ERROR: [56fc62c9] (24505-91520) super(PluginStep, self).process_lifecycle()
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 164, in process_lifecycle
celery.app.trace:ERROR: [56fc62c9] (24505-91520) step.process()
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 240, in process
celery.app.trace:ERROR: [56fc62c9] (24505-91520) self._process_block(item=item)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 302, in _process_block
celery.app.trace:ERROR: [56fc62c9] (24505-91520) self.process_main(item=item)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/publish.py", line 500, in process_main
celery.app.trace:ERROR: [56fc62c9] (24505-91520) context.add_unit_metadata(unit)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/metadata/filelists.py", line 42, in add_unit_metadata
celery.app.trace:ERROR: [56fc62c9] (24505-91520) self.metadata_file_handle.write(unit.render_filelists(self.checksum_type))
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/db/models.py", line 869, in render_filelists
celery.app.trace:ERROR: [56fc62c9] (24505-91520) return self._render(metadata, context)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/db/models.py", line 884, in _render
celery.app.trace:ERROR: [56fc62c9] (24505-91520) t = Template(template)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/django/template/base.py", line 191, in <strong>init</strong>
celery.app.trace:ERROR: [56fc62c9] (24505-91520) self.nodelist = self.compile_nodelist()
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/django/template/base.py", line 230, in compile_nodelist
celery.app.trace:ERROR: [56fc62c9] (24505-91520) return parser.parse()
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/django/template/base.py", line 482, in parse
celery.app.trace:ERROR: [56fc62c9] (24505-91520) raise self.error(token, 'Empty variable tag on line %d' % token.lineno)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) TemplateSyntaxError: Empty variable tag on line 548</h2>
<p>This is because one of the rpm in the repo has as "{{" filename causing the Django template syntax error.</p>
<p>In "6dc8dae1be904c2613d5aa3667dacd2554d05077eb1ce4296b6edfa3c4db3a46-filelists.xml.gz"
/usr/lib/.build-id
/usr/lib/.build-id/42
/usr/lib/.build-id/42/5049f5a9e0d1ac25c21de795f28fe886bfa0ca
/usr/lib64/R/library/rlang
/usr/lib64/R/library/rlang/DESCRIPTION
/usr/lib64/R/library/rlang/INDEX
/usr/lib64/R/library/rlang/LICENSE
/usr/lib64/R/library/rlang/Meta
snip...
/usr/lib64/R/library/rlang/help/wref_key.html
/usr/lib64/R/library/rlang/help/wref_value.html
/usr/lib64/R/library/rlang/help/zap.html
/usr/lib64/R/library/rlang/help/zap_srcref.html
/usr/lib64/R/library/rlang/help/{{.html <================= TemplateSyntaxError: Empty variable tag on line 548\n",
/usr/lib64/R/library/rlang/help/{{}}.html <=================</p>
<p>$ rpm -qlp R-rlang-0.4.11-3.fc35.x86_64.rpm | grep "{{"
/usr/lib64/R/library/rlang/help/{{.html
/usr/lib64/R/library/rlang/help/{{}}.html</p>
<p>help]$ cat {{.html</p>
<p>help]$ cat {{}}.html</p>
<p>Steps to Reproduce:</p>
<ol>
<li>Create a custom repository call fedora 35 and add the following feed url</li>
</ol>
<p><a href="https://dl.fedoraproject.org/pub/fedora/linux/releases/35/Everything/x86_64/os/" class="external">https://dl.fedoraproject.org/pub/fedora/linux/releases/35/Everything/x86_64/os/</a></p>
<ol start="2">
<li>Sync the repo</li>
</ol>
<p>Actual results:
Failed to publish repo</p>
<p>Expected results:
Repo can be published successfully</p>
<p>Additional info:
Tested on Satellite 6.10 and the repository can be synced successfully without issue.</p> RPM Support - Issue #9553 (MODIFIED): Publishing repository with large metadata may consume high ...https://pulp.plan.io/issues/95532021-11-03T06:10:34Zhyu
<p>Pulp is consuming high memory when publishing RHEL 7 repository. This is happening when Pulp is calculating the checksum of the metadata. It reads the whole metadata file into memory at once to calculates the checksum. For example, the other.xml.gz (compressed ) for RHEL 7 repository is about 837MB size. Reading the entire file into memory will cause Pulp worker to consume more than 1GB for RAM.</p>
<p><a href="https://github.com/pulp/pulp/blob/2-master/server/pulp/plugins/util/metadata_writer.py#L99-L101" class="external">https://github.com/pulp/pulp/blob/2-master/server/pulp/plugins/util/metadata_writer.py#L99-L101</a>.</p>
<p>How to reproduce:</p>
<ol>
<li>Sync the RHEL 7 repository.</li>
<li>After that manually force full publish it and run the below command to observe the memory usage.</li>
</ol>
<p>watch -n 1 'ps -aux | grep resource_worker</p>
<ol start="3">
<li>The memory usage should be stable between 200MB to 350MB all the time, but will suddenly go up to about 1.1GB for about 3 seconds (around finalizing the publish rpms step) then back to 200MB+.</li>
</ol> RPM Support - Issue #8890 (MODIFIED): Publishing a repository can take longer time to finish if m...https://pulp.plan.io/issues/88902021-06-14T06:00:43Zhyu
<p>Description of problem:
Pulp can take more than an hour to publish a repository when a large number of repositories have been synced from upstream and same errata are existed in the synced repositories, such as RHEL 7.x, RHEL 7 EUS and different aches.</p>
<p>The more repositories have the same errata the more "erratum pkglist" entries will be created in the mongodb which can cause the performance degradation.</p>
<p>For example:</p>
<blockquote>
<p>db.erratum_pkglists.find({errata_id: "RHSA-2018:2557"}).count()
387
db.erratum_pkglists.find({errata_id: "RHBA-2019:2180"}).count()
217</p>
</blockquote>
<p>When publishing errata, Pulp will use the above query to get all package lists of the errata. This will take long time to process when they are many package lists returned by the query and each package list is consist of many packages.</p>
<p>As we can see below, the "Publish Errata" step is very slow. 53 minutes has passed, it has only processed about 2073 errata. It will take more than an hour to finish.
...
{
"description": "Publishing Errata",
"details": "",
"error_details": [],
"items_total": 4789,
"num_failures": 0,
"num_processed": 2073,
"num_success": 2073,
"state": "IN_PROGRESS",
"step_id": "2f09190d-013a-4300-9445-eccb52ad94fe",
"step_type": "errata"
},
...
"start_time": "2021-06-09T12:41:13Z",</p>
<a name="date"></a>
<h1 >date<a href="#date" class="wiki-anchor">¶</a></h1>
<p>Wed Jun 9 13:32:41 UTC 2021</p> RPM Support - Test #7134 (CLOSED - COMPLETE): Test against CDNhttps://pulp.plan.io/issues/71342020-07-14T13:59:20Zppicka
<p>When core issue <a href="https://pulp.plan.io/issues/6735" class="external">https://pulp.plan.io/issues/6735</a> is solved, lets add the test to sync against the CDN both baseos and appstream repositories.</p>
<p>This will test same repo meta data with different filename as same productid is shipped intentionally with both of mentioned repositories.</p>
<p>Also this test is testing using a certificates which are needed for CDN.</p> RPM Support - Test #6503 (CLOSED - COMPLETE): assert checksums for on_demand publicationhttps://pulp.plan.io/issues/65032020-04-15T17:17:25Zfao89
<p>we need to test publication behavior when setting: metadata_checksum_type and package_checksum_type for on demand publication</p>
<p>Note: mixing on_demand and immediate policies on the same test case could lead to intermittent failures</p> RPM Support - Test #6015 (CLOSED - COMPLETE): Fix the "SyncMutatedUpdateRecordTestCase" testhttps://pulp.plan.io/issues/60152020-01-17T15:37:52Zdalleydalley@redhat.com
<p>This test was written long before the feature was implemented and has been disabled since. The feature is now implemented but when the test is enabled, it fails with the following exception thrown:</p>
<pre><code>pulp_rpm.app.exceptions.AdvisoryConflict: Incoming and existing advisories have the same id and timestamp but different and intersecting package lists. At least one of them is wrong. Advisory id: RHEA-2012:0055
</code></pre>
<p>This most likely means that Pulp 3 is more conservative than the test is designed for, and rejects the advisory in our test fixture because both versions of the advisory have identical timestamps but different package lists.</p>
<p>We need to find (or create) a fixture advisory where the two versions of the advisory have differing timestamps, so that the package lists can be successfully merged.</p> RPM Support - Task #5619 (CLOSED - COMPLETE): pulp-rpm-prerequisites (git) and pulp.pulp_rpm_prer...https://pulp.plan.io/issues/56192019-10-25T08:59:26Zmdepaulo@redhat.com
<p>If you clone "pulp-rpm-prerequisites" from github, it has dashes. You specify that quoted string in your vars file for the role name (prereq_role).</p>
<p>if you install it from ansible galaxy, it must be specified as "pulp.pulp_rpm_prerequisites" instead.</p>
<p>Both names are specified across different docs.</p>
<p>The "pulp." prefix is kind of obvious, but the dashes vs underscores is not. It's easy to make this mistake.</p>
<p>The underscores are <strong>required</strong> by ansible galaxy, but not ansible core/engine. Dashes and dots be auto-converted to underscores during galaxy upload.</p>
<p>The git repo shouldd be renamed with underscores.</p>
<p>We should also determine if it is possible for users to specify it without the prefix. Perhaps "role_name" needs to be set? Or perhaps the new Ansible Galaxy FQDNs for rolenames will enable this.</p> RPM Support - Task #4925 (CLOSED - COMPLETE): ensure that CharFields have appropriate max_lengthhttps://pulp.plan.io/issues/49252019-06-06T15:59:18Zdkliban@redhat.comRPM Support - Test #4721 (CLOSED - COMPLETE): Test simple content copyhttps://pulp.plan.io/issues/47212019-04-20T05:16:54Zdalleydalley@redhat.com
<ol>
<li>Create an RPM repository</li>
<li>Populate it with content</li>
<li>Create a second, empty RPM repository</li>
<li>Use the copy API endpoint to copy all content from the first repository into the second repository</li>
<li>Check that a new repository version was created in the second repository, and that it has identical content to the first</li>
</ol>
<p>The copy API works as follows</p>
<p>http --form POST <a href="http://localhost:24817/pulp/api/v3/rpm/copy/" class="external">http://localhost:24817/pulp/api/v3/rpm/copy/</a> source_repository=${SRC_REPO_HREF} destination_repository=${DEST_REPO_HREF}</p>
<p>(exact parameter names TBD)</p> RPM Support - Test #4469 (CLOSED - COMPLETE): Test checksum algorithm when publishinghttps://pulp.plan.io/issues/44692019-02-25T21:05:27Zkersom
<p>checksum_type type will be an option at publish time. Explore this new option. This test should be added to the RPM plugin.</p> RPM Support - Test #4457 (CLOSED - COMPLETE): Test out S3 supporthttps://pulp.plan.io/issues/44572019-02-25T12:33:30Zdaviddavis
<p>Make sure that the pulp rpm plugin works when using Pulp with S3. I see no reason why it shouldn't but it's worth testing to confirm.</p>
<p>The docs for configuring pulp w/ S3 are here: <a href="https://docs.pulpproject.org/en/3.0/nightly/installation/storage.html#amazon-s3" class="external">https://docs.pulpproject.org/en/3.0/nightly/installation/storage.html#amazon-s3</a></p> RPM Support - Test #4257 (CLOSED - COMPLETE): QueryExistingArtifacts stage does not prevent dupli...https://pulp.plan.io/issues/42572018-12-11T13:31:01Zbherring
<a name="Notes"></a>
<h2 >Notes<a href="#Notes" class="wiki-anchor">¶</a></h2>
<ul>
<li>See <a class="issue tracker-1 status-11 priority-6 priority-default closed" title="Issue: Checksum constraint error when syncing a package (CLOSED - CURRENTRELEASE)" href="https://pulp.plan.io/issues/4170">#4170</a> Note <a class="issue tracker-3 status-9 priority-6 priority-default closed" title="Story: As a user, I can rest easy in the knowledge that my celery workers will ensure that their AMQP me... (CLOSED - WONTFIX)" href="https://pulp.plan.io/issues/15">#15</a></li>
</ul> RPM Support - Test #4210 (CLOSED - COMPLETE): Test upload of RPM with invalid byte sequences embe...https://pulp.plan.io/issues/42102018-11-28T20:52:00Zkersom
<ul>
<li>Upload an RPM with non-ascii metadata.</li>
<li>Upload an RPM with non-utf-8 metadata - users should get a friendly error message</li>
</ul> RPM Support - Test #4158 (CLOSED - COMPLETE): Test applicability for modular or mixed contenthttps://pulp.plan.io/issues/41582018-11-14T17:49:54Zttereshcttereshc@redhat.com
<a name="Ingredients"></a>
<h3 >Ingredients<a href="#Ingredients" class="wiki-anchor">¶</a></h3>
<ul>
<li>consumer</li>
<li>consumer profile<br>
It used to be of <code>rpm</code> type only, now there can be a second active profile for the same consumer, "modulemd" type.<br>
So two types are of consumer profiles are now allowed for uploading. Their format is different depending on type but it's not tested validated anywhere, so don't test with different weird fields in the profile :) See examples and expected format in <a class="issue tracker-3 status-11 priority-6 priority-default closed" title="Story: As a user, I can upload information about Consumer (CLOSED - CURRENTRELEASE)" href="https://pulp.plan.io/issues/3927">#3927</a>.</li>
</ul>
<ul>
<li>repository with modular content, like <a href="https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-with-modules/" class="external">our fixtures</a>
</li>
</ul>
<p>Applicability calculates which RPMs, modules and errata are applicable.</p>
<a name="Conditions-when-each-of-the-content-types-are-applicable"></a>
<h3 >Conditions when each of the content types are applicable<a href="#Conditions-when-each-of-the-content-types-are-applicable" class="wiki-anchor">¶</a></h3>
<p>1. RPM is applicable:</p>
<ul>
<li>in case of modular RPM:</li>
</ul>
<ul>
<li>if its corresponding module is enabled</li>
<li>if the first-level module dependencies of that module are enabled</li>
<li>if that RPM is newer than the installed one</li>
</ul>
<ul>
<li>in case of non-modular RPM:</li>
</ul>
<ul>
<li>if RPM name doesn't match the name of any of the modular RPMs from enabled modules</li>
<li>if that RPM is newer than the installed one</li>
</ul>
<p>2. Module is applicable:</p>
<ul>
<li>if at least one RPM which is referred in its artifacts is present in a repo</li>
<li>if at least one RPM which is referred in its artifacts is applicable</li>
</ul>
<p>3. Erratum is applicable:</p>
<ul>
<li>if at least one RPM which is referred in the erratum pkglist is present in a repo</li>
<li>if at least one RPM which is referred in the erratum pkglist is applicable</li>
</ul>
<a name="Behavioral-changes"></a>
<h3 >Behavioral changes<a href="#Behavioral-changes" class="wiki-anchor">¶</a></h3>
<ul>
<li>It was required to request consumer applicability first (to create an applicability profile) and only then it was possible to update it with repo applicability. It's no longer a requirement, repo applicability will create applicability profiles if they don't exist.</li>
</ul>
<a name="Suggested-tests-for-fixtures-repo"></a>
<h3 >Suggested tests (for fixtures repo)<a href="#Suggested-tests-for-fixtures-repo" class="wiki-anchor">¶</a></h3>
<ul>
<li>Test applicability when only modular RPMs are installed on a consumer</li>
<li>Test applicability when modular and non-modular RPMs are installed on a consumer</li>
<li>Test applicability when enabled module has module dependencies (see <code>duck</code> modules)</li>
<li>Test applicability when enabled module has multiple versions, the same stream but multiple versions (see duck and kangaroo)</li>
<li>Test applicability when no modules are enabled</li>
<li>Test applicability when module is present in an erratum</li>
<li>Test applicability when module is absent from an erratum but still applicable (see <code>walrus</code> modules)</li>
<li>Test applicability when a non-modular erratum refers to an RPM with the same name as modular applicable RPM (see <code>walrus</code> RPM and <code>Sea_Erratum</code>)</li>
<li>any other useful scenarios</li>
</ul> RPM Support - Task #4137 (CLOSED - COMPLETE): Add Installer docs to rpm installationhttps://pulp.plan.io/issues/41372018-11-08T15:59:51Zbmbouterbmbouter@redhat.com
<a name="Problem"></a>
<h2 >Problem<a href="#Problem" class="wiki-anchor">¶</a></h2>
<p>The pulp_rpm plugin is too hard to install. <a href="https://pulp-rpm.readthedocs.io/en/latest/installation.html" class="external">https://pulp-rpm.readthedocs.io/en/latest/installation.html</a></p>
<a name="Solution"></a>
<h2 >Solution<a href="#Solution" class="wiki-anchor">¶</a></h2>
<p>Use the Ansible installer from core to, configured to also install pulp_rpm via the <a href="https://github.com/pulp/ansible-pulp3/tree/master/roles/pulp3#role-variables" class="external">pulp_install_plugins variable</a></p>
<p>We also need to somehow have the installer pip install the createrepo_c from fedorapeople.</p>
<p>These docs should be the main installation docs, but it should not delete the "manual" install docs we already have.</p>