Pulp: Issueshttps://pulp.plan.io/https://pulp.plan.io/favicon.ico2021-12-07T18:13:47ZPulp
Planio Migration Plugin - Backport #9612 (MODIFIED): Backport #8968 "'NoneType' object has no attribute ...https://pulp.plan.io/issues/96122021-12-07T18:13:47Zttereshcttereshc@redhat.com
<p>Backtrace:</p>
<pre>
"error"=>
{"traceback"=>
" File \"/usr/lib/python3.6/site-packages/rq/worker.py\", line 936, in perform_job\n" +
" rv = job.perform()\n" +
" File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 684, in perform\n" +
" self._result = self._execute()\n" +
" File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 690, in _execute\n" +
" return self.func(*self.args, **self.kwargs)\n" +
" File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py\", line 76, in migrate_from_pulp2\n" +
" pre_migrate_all_without_content(plan)\n" +
" File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/pre_migration.py\", line 493, in pre_migrate_all_without_content\n" +
" pre_migrate_importer(repo_id, importer_types)\n" +
" File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/pre_migration.py\", line 601, in pre_migrate_importer\n" +
" importer.pulp3_remote.delete()\n",
"description"=>"'NoneType' object has no attribute 'delete'"},
"worker"=>"/pulp/api/v3/workers/fc6ba1d6-ddc6-494d-9385-2368544a09ef/",
</pre>
<p>This happened on Katello 3.18.3 which uses:</p>
<pre>
pulp-2to3-migration (0.11.1)
pulp-certguard (1.0.3)
pulp-container (2.1.2)
pulp-deb (2.7.0)
pulp-file (1.3.0)
pulp-rpm (3.10.0)
pulpcore (3.7.6)
</pre> RPM Support - Issue #9559 (MODIFIED): Unable to sync Fedora 35 repositoryhttps://pulp.plan.io/issues/95592021-11-08T02:09:29Zhyu
<p>Cloned from <a href="https://bugzilla.redhat.com/show_bug.cgi?id=2020473" class="external">https://bugzilla.redhat.com/show_bug.cgi?id=2020473</a></p>
<h2>Description of problem:
When publishing the Fedora 35 repository, Pulp failed with the following error:</h2>
<h2>celery.app.trace:ERROR: [56fc62c9] (24505-91520) Task pulp.server.managers.repo.publish.publish[56fc62c9-29b0-41bf-a820-ba3625ef4a00] raised unexpected: TemplateSyntaxError(u'Empty variable tag on line 548',)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) Traceback (most recent call last):
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
celery.app.trace:ERROR: [56fc62c9] (24505-91520) R = retval = fun(*args, **kwargs)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 688, in <strong>call</strong>
celery.app.trace:ERROR: [56fc62c9] (24505-91520) return super(Task, self).<strong>call</strong>(*args, **kwargs)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 110, in <strong>call</strong>
celery.app.trace:ERROR: [56fc62c9] (24505-91520) return super(PulpTask, self).<strong>call</strong>(*args, **kwargs)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in <strong>protected_call</strong>
celery.app.trace:ERROR: [56fc62c9] (24505-91520) return self.run(*args, **kwargs)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1142, in publish
celery.app.trace:ERROR: [56fc62c9] (24505-91520) result = check_publish(repo_obj, dist_id, dist_inst, transfer_repo, conduit, call_config)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1251, in check_publish
celery.app.trace:ERROR: [56fc62c9] (24505-91520) result = _do_publish(repo_obj, dist_id, dist_inst, transfer_repo, conduit, call_config)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1303, in _do_publish
celery.app.trace:ERROR: [56fc62c9] (24505-91520) publish_report = publish_repo(transfer_repo, conduit, call_config)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 901, in wrap_f
celery.app.trace:ERROR: [56fc62c9] (24505-91520) return f(*args, **kwargs)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/distributor.py", line 185, in publish_repo
celery.app.trace:ERROR: [56fc62c9] (24505-91520) return self._publisher.process_lifecycle()
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 573, in process_lifecycle
celery.app.trace:ERROR: [56fc62c9] (24505-91520) super(PluginStep, self).process_lifecycle()
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 164, in process_lifecycle
celery.app.trace:ERROR: [56fc62c9] (24505-91520) step.process()
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 240, in process
celery.app.trace:ERROR: [56fc62c9] (24505-91520) self._process_block(item=item)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 302, in _process_block
celery.app.trace:ERROR: [56fc62c9] (24505-91520) self.process_main(item=item)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/publish.py", line 500, in process_main
celery.app.trace:ERROR: [56fc62c9] (24505-91520) context.add_unit_metadata(unit)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/metadata/filelists.py", line 42, in add_unit_metadata
celery.app.trace:ERROR: [56fc62c9] (24505-91520) self.metadata_file_handle.write(unit.render_filelists(self.checksum_type))
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/db/models.py", line 869, in render_filelists
celery.app.trace:ERROR: [56fc62c9] (24505-91520) return self._render(metadata, context)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/db/models.py", line 884, in _render
celery.app.trace:ERROR: [56fc62c9] (24505-91520) t = Template(template)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/django/template/base.py", line 191, in <strong>init</strong>
celery.app.trace:ERROR: [56fc62c9] (24505-91520) self.nodelist = self.compile_nodelist()
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/django/template/base.py", line 230, in compile_nodelist
celery.app.trace:ERROR: [56fc62c9] (24505-91520) return parser.parse()
celery.app.trace:ERROR: [56fc62c9] (24505-91520) File "/usr/lib/python2.7/site-packages/django/template/base.py", line 482, in parse
celery.app.trace:ERROR: [56fc62c9] (24505-91520) raise self.error(token, 'Empty variable tag on line %d' % token.lineno)
celery.app.trace:ERROR: [56fc62c9] (24505-91520) TemplateSyntaxError: Empty variable tag on line 548</h2>
<p>This is because one of the rpm in the repo has as "{{" filename causing the Django template syntax error.</p>
<p>In "6dc8dae1be904c2613d5aa3667dacd2554d05077eb1ce4296b6edfa3c4db3a46-filelists.xml.gz"
/usr/lib/.build-id
/usr/lib/.build-id/42
/usr/lib/.build-id/42/5049f5a9e0d1ac25c21de795f28fe886bfa0ca
/usr/lib64/R/library/rlang
/usr/lib64/R/library/rlang/DESCRIPTION
/usr/lib64/R/library/rlang/INDEX
/usr/lib64/R/library/rlang/LICENSE
/usr/lib64/R/library/rlang/Meta
snip...
/usr/lib64/R/library/rlang/help/wref_key.html
/usr/lib64/R/library/rlang/help/wref_value.html
/usr/lib64/R/library/rlang/help/zap.html
/usr/lib64/R/library/rlang/help/zap_srcref.html
/usr/lib64/R/library/rlang/help/{{.html <================= TemplateSyntaxError: Empty variable tag on line 548\n",
/usr/lib64/R/library/rlang/help/{{}}.html <=================</p>
<p>$ rpm -qlp R-rlang-0.4.11-3.fc35.x86_64.rpm | grep "{{"
/usr/lib64/R/library/rlang/help/{{.html
/usr/lib64/R/library/rlang/help/{{}}.html</p>
<p>help]$ cat {{.html</p>
<p>help]$ cat {{}}.html</p>
<p>Steps to Reproduce:</p>
<ol>
<li>Create a custom repository call fedora 35 and add the following feed url</li>
</ol>
<p><a href="https://dl.fedoraproject.org/pub/fedora/linux/releases/35/Everything/x86_64/os/" class="external">https://dl.fedoraproject.org/pub/fedora/linux/releases/35/Everything/x86_64/os/</a></p>
<ol start="2">
<li>Sync the repo</li>
</ol>
<p>Actual results:
Failed to publish repo</p>
<p>Expected results:
Repo can be published successfully</p>
<p>Additional info:
Tested on Satellite 6.10 and the repository can be synced successfully without issue.</p> RPM Support - Issue #9553 (MODIFIED): Publishing repository with large metadata may consume high ...https://pulp.plan.io/issues/95532021-11-03T06:10:34Zhyu
<p>Pulp is consuming high memory when publishing RHEL 7 repository. This is happening when Pulp is calculating the checksum of the metadata. It reads the whole metadata file into memory at once to calculates the checksum. For example, the other.xml.gz (compressed ) for RHEL 7 repository is about 837MB size. Reading the entire file into memory will cause Pulp worker to consume more than 1GB for RAM.</p>
<p><a href="https://github.com/pulp/pulp/blob/2-master/server/pulp/plugins/util/metadata_writer.py#L99-L101" class="external">https://github.com/pulp/pulp/blob/2-master/server/pulp/plugins/util/metadata_writer.py#L99-L101</a>.</p>
<p>How to reproduce:</p>
<ol>
<li>Sync the RHEL 7 repository.</li>
<li>After that manually force full publish it and run the below command to observe the memory usage.</li>
</ol>
<p>watch -n 1 'ps -aux | grep resource_worker</p>
<ol start="3">
<li>The memory usage should be stable between 200MB to 350MB all the time, but will suddenly go up to about 1.1GB for about 3 seconds (around finalizing the publish rpms step) then back to 200MB+.</li>
</ol> RPM Support - Issue #9021 (MODIFIED): No such file or directory: '/var/lib/pulp/sync_imports/test...https://pulp.plan.io/issues/90212021-07-06T18:41:26Ziballou
<p>Pulp should not error out when repomd.xml exists but repomd.xml.asc doesn't. This error occurred when syncing from the filesystem with the following patch applied to Pulpcore 3.14: <a href="https://github.com/pulp/pulpcore/pull/1463.patch" class="external">https://github.com/pulp/pulpcore/pull/1463.patch</a></p>
<p>Occurred on pulp-rpm (3.13.2)</p> Migration Plugin - Issue #8968 (MODIFIED): "'NoneType' object has no attribute 'delete'" during m...https://pulp.plan.io/issues/89682021-06-24T20:55:28Ziballou
<p>Backtrace:</p>
<pre>
"error"=>
{"traceback"=>
" File \"/usr/lib/python3.6/site-packages/rq/worker.py\", line 936, in perform_job\n" +
" rv = job.perform()\n" +
" File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 684, in perform\n" +
" self._result = self._execute()\n" +
" File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 690, in _execute\n" +
" return self.func(*self.args, **self.kwargs)\n" +
" File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py\", line 76, in migrate_from_pulp2\n" +
" pre_migrate_all_without_content(plan)\n" +
" File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/pre_migration.py\", line 493, in pre_migrate_all_without_content\n" +
" pre_migrate_importer(repo_id, importer_types)\n" +
" File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/pre_migration.py\", line 601, in pre_migrate_importer\n" +
" importer.pulp3_remote.delete()\n",
"description"=>"'NoneType' object has no attribute 'delete'"},
"worker"=>"/pulp/api/v3/workers/fc6ba1d6-ddc6-494d-9385-2368544a09ef/",
</pre>
<p>This happened on Katello 3.18.3 which uses:</p>
<pre>
pulp-2to3-migration (0.11.1)
pulp-certguard (1.0.3)
pulp-container (2.1.2)
pulp-deb (2.7.0)
pulp-file (1.3.0)
pulp-rpm (3.10.0)
pulpcore (3.7.6)
</pre> RPM Support - Issue #8893 (MODIFIED): 3rd party repository sync fails with 'InvalidStringData: st...https://pulp.plan.io/issues/88932021-06-14T14:14:36Zggainey
<p>ModulemdDefaults are BSON-encoded before being saved into MongoDB because MongoDB (apparently) has restrictions on valid keys which are incompatible with the module data we need to store.</p>
<p><a href="https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/importers/yum/repomd/modules.py#L94-L95" class="external">https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/importers/yum/repomd/modules.py#L94-L95</a></p>
<p>...But the serialized BSON strings being created by the encoding function we are using are not always valid UTF-8... Presumably, it works the vast majority of the time, enough so that it wasn't noticed.</p>
<p>Only specific permutations of the input data appear to trigger this. If I delete key from the profiles dictionary, all of a sudden it's UTF-8 compatible. This is presumably why the problem pops into and out of existence.</p>
<p>At some point we attempt to save this string to MongoDB, and Mongo decides to apply a UTF-8 validation to it, and it blows up.</p> Pulp - Backport #8485 (MODIFIED): Backport to 3.7 artifact stage fixes which affect pulp 2to3 mig...https://pulp.plan.io/issues/84852021-03-31T15:01:11Zttereshcttereshc@redhat.com
<p>Fixes to backport:</p>
<p><a href="https://pulp.plan.io/issues/7876" class="external">https://pulp.plan.io/issues/7876</a><br>
<a href="https://pulp.plan.io/issues/8377" class="external">https://pulp.plan.io/issues/8377</a><br>
<a href="https://pulp.plan.io/issues/8489" class="external">https://pulp.plan.io/issues/8489</a> (this is important only together with 8377)</p>
<p>Cherry-pick is clean is they are performed in the order.</p> Docker Support - Issue #8344 (MODIFIED): pulp2pulp sync fails when schema1 is not presenthttps://pulp.plan.io/issues/83442021-03-04T19:31:29Zipanova@redhat.comipanova@redhat.com
<ol>
<li>on pulp1 sync registry.redhat.io/dotnet/dotnetcore-10-rhel7</li>
<li>try with pulp2 sync from pulp1</li>
<li>observe error : PLP0000: 404 Client Error: 'Not Found' for url: <a href="https://localhost:5000/dotnetcore-10-rhel7/manifests/1.0" class="external">https://localhost:5000/dotnetcore-10-rhel7/manifests/1.0</a>
</li>
</ol> Ansible Plugin - Story #8205 (MODIFIED): Add AnsibleCollectionDeprecated resource to export/importhttps://pulp.plan.io/issues/82052021-02-03T23:48:12Zgerrod
<a name="The-Two-Related-Problem"></a>
<h2 >The Two Related Problem<a href="#The-Two-Related-Problem" class="wiki-anchor">¶</a></h2>
<ol>
<li>The 'deprecated' data is mutable.</li>
<li>The 'deprecated' data is not correctly included in import/export data</li>
</ol>
<a name="Solution"></a>
<h2 >Solution<a href="#Solution" class="wiki-anchor">¶</a></h2>
<p>From a high level, make the <a href="https://github.com/pulp/pulp_ansible/blob/4b3d63f14dd85b9bfd689074defb99a23ee72f31/pulp_ansible/app/models.py#L281" class="external"><code>AnsibleCollectionDeprecated</code></a> content.</p>
<a name="Modeling"></a>
<h3 >Modeling<a href="#Modeling" class="wiki-anchor">¶</a></h3>
<p>Here are the rough steps involved:</p>
<ol>
<li>switch <a href="https://github.com/pulp/pulp_ansible/blob/4b3d63f14dd85b9bfd689074defb99a23ee72f31/pulp_ansible/app/models.py#L281" class="external">the existing AnsibleCollectionDeprecated</a> to inherit from Content</li>
<li>Remove the <a href="https://github.com/pulp/pulp_ansible/blob/4b3d63f14dd85b9bfd689074defb99a23ee72f31/pulp_ansible/app/models.py#L286-L288" class="external"><code>repository_version</code> FK</a> entirely.</li>
<li>Change the <a href="https://github.com/pulp/pulp_ansible/blob/4b3d63f14dd85b9bfd689074defb99a23ee72f31/pulp_ansible/app/models.py#L289" class="external"><code>collection</code></a> field to be a string with the collection name instead</li>
<li>Add a string field called namespace.</li>
<li>The uniqueness constraint should be the tuple "namespace", "collection_name"</li>
</ol>
<a name="Sync-Implementation"></a>
<h3 >Sync Implementation<a href="#Sync-Implementation" class="wiki-anchor">¶</a></h3>
<p>The sync code will continue to parse the deprecated data in FirstStage in the same way, but instead of <a href="https://github.com/pulp/pulp_ansible/blob/master/pulp_ansible/app/tasks/collections.py#L140-L153" class="external">saving/managing them in first-stage</a>, have the stages pipeline handle it all by making <code>AnsibleCollectionDeprecated</code> in-memory and associating with a declarative content object.</p>
<a name="The-Galaxy-API-viewset"></a>
<h3 >The Galaxy API viewset<a href="#The-Galaxy-API-viewset" class="wiki-anchor">¶</a></h3>
<p><a href="https://github.com/pulp/pulp_ansible/blob/8cd963d769e5b25f957d241571e21c371264a8f4/pulp_ansible/app/galaxy/v3/views.py#L172" class="external">Port this portion</a> to look for the objects in the set of content of the RepositoryVersion.</p>
<a name="The-viewset-for-setunset-deprecation-handling"></a>
<h3 >The viewset for set/unset deprecation handling<a href="#The-viewset-for-setunset-deprecation-handling" class="wiki-anchor">¶</a></h3>
<p><a href="https://github.com/pulp/pulp_ansible/blob/8cd963d769e5b25f957d241571e21c371264a8f4/pulp_ansible/app/galaxy/v3/views.py#L298-L306" class="external">Port this portion</a> to generate a task that creates a RepositoryVersion</p>
<a name="Additional-work-needed-to-get-this-working-with-ImportExport"></a>
<h3 >Additional work needed to get this working with Import/Export<a href="#Additional-work-needed-to-get-this-working-with-ImportExport" class="wiki-anchor">¶</a></h3>
<p>Add a model to model resources: <a href="https://github.com/pulp/pulp_ansible/blob/master/pulp_ansible/app/modelresource.py" class="external">https://github.com/pulp/pulp_ansible/blob/master/pulp_ansible/app/modelresource.py</a></p> Pulp - Backport #8122 (CLOSED - COMPLETE): Backport Request: 8099 (File upload causes django.sec...https://pulp.plan.io/issues/81222021-01-19T16:37:41Ziballou
<p>We'll need this backported to use Pulpcore 3.9 with Katello 4.0 once the issue is closed.</p> Migration Plugin - Task #7782 (CLOSED - COMPLETE): Ensure migration plugin runs in FIPS mode and ...https://pulp.plan.io/issues/77822020-11-03T15:51:55Zttereshcttereshc@redhat.com
<p>Katello needs to run in a FIPS-enabled environment with the migration plugin.</p>
<p>The prerequisite for this task would be pulpcore, pulp_file, pulp_container and pulp_rpm being able to function in the FIPS environment.</p> Migration Plugin - Task #7778 (CLOSED - COMPLETE): Ensure a migration can be interrupted within r...https://pulp.plan.io/issues/77782020-11-03T15:25:58Zttereshcttereshc@redhat.com
<a name="Motivation"></a>
<h3 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h3>
<p>Users who have pulp 2 and pulp 3 on the same machine need to have a way to control when migration is running.
The migration plugin inevitably provides certain load on the system and uses resources.
Users might want to choose windows when they run a migration. E.g. Run every Saturday for 4 hours at most.</p>
<p>If the system is large, it can take hours to migrate it, users need a way to cancel the task and free all the related resource within a reasonable timeframe.</p>
<a name="Potential-issues"></a>
<h3 >Potential issues<a href="#Potential-issues" class="wiki-anchor">¶</a></h3>
<p>When there is a very slow db query and the task which triggered that query is cancelled, how quickly the db query will be aborted as well?</p> RPM Support - Test #7350 (MODIFIED): Test syncing from a repository located on a local diskhttps://pulp.plan.io/issues/73502020-08-19T15:40:07Zdalleydalley@redhat.com
<p>e.g. a repository with a file:// url such as file:///var/lib/pulp/sync_imports/test_repos/zoo/</p> Ansible Plugin - Test #5256 (CLOSED - COMPLETE): Test - Ansible collection remote doesn't handle ...https://pulp.plan.io/issues/52562019-08-13T14:42:32Zkersom
<p>Ex:<br>
With a url <a href="https://galaxy.ansible.com" class="external">https://galaxy.ansible.com</a> it works fine.<br>
With a url <a href="https://galaxy.ansible.com/" class="external">https://galaxy.ansible.com/</a> , throws the following error: <a href="https://gist.github.com/sjha4/0afdd783bac7cea56055b706f6b02155" class="external">https://gist.github.com/sjha4/0afdd783bac7cea56055b706f6b02155</a></p> RPM Support - Task #3824 (CLOSED - COMPLETE): Analyze repository version creation performance for...https://pulp.plan.io/issues/38242018-07-06T16:41:41Zdkliban@redhat.com
<p>After the user has migrated content from Pulp 2 to Pulp 3, it will be necesary to migrate the repositories and the association between repos and content. There are 2 options for migrating the association:</p>
<p>- create a repository version by performing a sync from Pulp 2 repository<br>
- create a repository version by querying Pulp 2, find units in Pulp 3, create a repository version by specifying the pulp 3 units.</p>
<p>This task is to analyze the performance of the two methods of creating repository versions and provide guidance on how this part of the migration should be handled.</p>