Pulp: Issueshttps://pulp.plan.io/https://pulp.plan.io/favicon.ico2021-09-17T14:10:06ZPulp
Planio Pulp - Issue #9421 (MODIFIED): Add support for sslmode to settings in pulp and pulp-operator.https://pulp.plan.io/issues/94212021-09-17T14:10:06Zbenthomasson
<p>In order to run AAP on Azure I need to enable the sslmode option for the database connection for automation hub.</p>
<p>Automation Hub runs as a django app and so needs a DATABASES configuration like this:</p>
<pre><code class="python syntaxhl" data-language="python"><span class="n">DATABASES</span> <span class="o">=</span> <span class="p">{</span>
<span class="s">'default'</span><span class="p">:</span> <span class="p">{</span>
<span class="s">'ENGINE'</span><span class="p">:</span> <span class="s">'django.db.backends.postgresql'</span><span class="p">,</span>
<span class="s">'NAME'</span><span class="p">:</span> <span class="s">"db_name"</span><span class="p">,</span>
<span class="s">'USER'</span><span class="p">:</span> <span class="s">"db_username"</span><span class="p">,</span>
<span class="s">'PASSWORD'</span><span class="p">:</span> <span class="s">"db_password"</span><span class="p">,</span>
<span class="s">'HOST'</span><span class="p">:</span> <span class="s">"db_host"</span><span class="p">,</span>
<span class="s">'OPTIONS'</span><span class="p">:</span> <span class="p">{</span><span class="s">'sslmode'</span><span class="p">:</span> <span class="s">'require'</span><span class="p">},</span>
<span class="p">},</span>
<span class="p">}</span>
</code></pre>
<p>I am using the pulp-operator to install galaxy_ng and I would like the pulp-operator to look for this option in the databases secret:</p>
<pre><code class="yaml syntaxhl" data-language="yaml">
<span class="nn">---</span>
<span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Secret</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">galaxy-pulp-postgres-configuration</span>
<span class="na">namespace</span><span class="pi">:</span> <span class="s">default</span>
<span class="na">stringData</span><span class="pi">:</span>
<span class="na">host</span><span class="pi">:</span> <span class="s">postgres</span>
<span class="na">port</span><span class="pi">:</span> <span class="s2">"</span><span class="s">5432"</span>
<span class="na">database</span><span class="pi">:</span> <span class="s">pulp</span>
<span class="na">username</span><span class="pi">:</span> <span class="s">pulp</span>
<span class="na">password</span><span class="pi">:</span> <span class="s">XXX</span>
<span class="na">sslmode</span><span class="pi">:</span> <span class="s">require</span>
<span class="na">type</span><span class="pi">:</span> <span class="s">unmanaged</span>
<span class="na">type</span><span class="pi">:</span> <span class="s">Opaque</span>
</code></pre> Pulp - Issue #9395 (CLOSED - CURRENTRELEASE): RemoteArtifacts are not being saved properlyhttps://pulp.plan.io/issues/93952021-09-14T03:24:43Zdalleydalley@redhat.com
<p>This is the result of a long discussion on the Katello forums: <a href="https://community.theforeman.org/t/katello-4-1-2-1-404-error-through-content-proxy-due-to-incorrect-location-href/24812/26?u=dralley" class="external">https://community.theforeman.org/t/katello-4-1-2-1-404-error-through-content-proxy-due-to-incorrect-location-href/24812/26?u=dralley</a></p>
<p>TL;DR if you sync a repository on-demand multiple times against different repos, only the first set of RemoteArtifacts is saved. If the layout of the repository changes or the repository disappears, all of these URLs get broken, and the RemoteArtifacts are thus broken even if they were supposed to have multiple different potential sources.</p>
<p>I've confirmed this by syncing a single repository, changing the layout of that repository, and resyncing. Only the original RemoteArtifact will exist and the new ones will not. A script is attached to demonstrate this (if you look in the DB afterwards)</p>
<p>This is an especially severe issue because "metadata mirroring" and standard syncs have entirely different layouts, so re-publishing a mirrored repository or mirroring it after having otherwise not been doing so results in broken repositories due to all of the URLs changing.</p> RPM Support - Issue #9335 (CLOSED - DUPLICATE): Huge memory consumption when performing depsolvin...https://pulp.plan.io/issues/93352021-09-02T14:53:23Zdalleydalley@redhat.comPulp - Issue #9269 (CLOSED - CURRENTRELEASE): Allow content types that declare their own custom m...https://pulp.plan.io/issues/92692021-08-21T17:15:36Zdalleydalley@redhat.com
<p>Content subclasses that declare their own custom managers don't have access to touch() functionality, we need to export it through the API so that they can subclass it.</p> Pulp - Issue #8603 (CLOSED - CURRENTRELEASE): possible tasking race condition: update or delete ...https://pulp.plan.io/issues/86032021-04-20T19:30:22Zjsherril@redhat.comjsherril@redhat.com
<p>With pulpcore 3.11.0</p>
<pre><code>pulp [34656514-03a4-4255-b3b7-8986492b87b1]: rq.worker:ERROR: Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/django/db/backends/base/base.py", line 240, in _commit
return self.connection.commit()
psycopg2.errors.ForeignKeyViolation: update or delete on table "core_reservedresource" violates foreign key constraint "core_taskreservedres_resource_id_ee0b7c62_fk_core_rese" on table "core_taskreservedresource"
DETAIL: Key (pulp_id)=(c0b7db83-401e-4a46-8d9e-07f5ab70de1e) is still referenced from table "core_taskreservedresource".
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/rq/worker.py", line 984, in perform_job
started_job_registry=started_job_registry)
File "/usr/lib/python3.6/site-packages/pulpcore/tasking/worker.py", line 146, in handle_job_success
task.release_resources()
File "/usr/lib/python3.6/site-packages/pulpcore/app/models/task.py", line 430, in release_resources
reservation.delete()
File "/usr/lib/python3.6/site-packages/django_lifecycle/mixins.py", line 146, in delete
value = super().delete(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 922, in delete
return collector.delete()
File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 318, in delete
sender=model, instance=obj, using=self.using
File "/usr/lib/python3.6/site-packages/django/db/transaction.py", line 240, in __exit__
connection.commit()
File "/usr/lib/python3.6/site-packages/django/db/backends/base/base.py", line 262, in commit
self._commit()
File "/usr/lib/python3.6/site-packages/django/db/backends/base/base.py", line 240, in _commit
return self.connection.commit()
File "/usr/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/usr/lib/python3.6/site-packages/django/db/backends/base/base.py", line 240, in _commit
return self.connection.commit()
django.db.utils.IntegrityError: update or delete on table "core_reservedresource" violates foreign key constraint "core_taskreservedres_resource_id_ee0b7c62_fk_core_rese" on table "core_taskreservedresource"
DETAIL: Key (pulp_id)=(c0b7db83-401e-4a46-8d9e-07f5ab70de1e) is still referenced from table "core_taskreservedresource".
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/django/db/backends/base/base.py", line 240, in _commit
return self.connection.commit()
psycopg2.errors.ForeignKeyViolation: update or delete on table "core_reservedresource" violates foreign key constraint "core_taskreservedres_resource_id_ee0b7c62_fk_core_rese" on table "core_taskreservedresource"
DETAIL: Key (pulp_id)=(c0b7db83-401e-4a46-8d9e-07f5ab70de1e) is still referenced from table "core_taskreservedresource".
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/rq/worker.py", line 984, in perform_job
started_job_registry=started_job_registry)
File "/usr/lib/python3.6/site-packages/pulpcore/tasking/worker.py", line 146, in handle_job_success
task.release_resources()
File "/usr/lib/python3.6/site-packages/pulpcore/app/models/task.py", line 430, in release_resources
reservation.delete()
File "/usr/lib/python3.6/site-packages/django_lifecycle/mixins.py", line 146, in delete
value = super().delete(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 922, in delete
return collector.delete()
File "/usr/lib/python3.6/site-packages/django/db/models/deletion.py", line 318, in delete
sender=model, instance=obj, using=self.using
File "/usr/lib/python3.6/site-packages/django/db/transaction.py", line 240, in __exit__
connection.commit()
File "/usr/lib/python3.6/site-packages/django/db/backends/base/base.py", line 262, in commit
self._commit()
File "/usr/lib/python3.6/site-packages/django/db/backends/base/base.py", line 240, in _commit
return self.connection.commit()
File "/usr/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/usr/lib/python3.6/site-packages/django/db/backends/base/base.py", line 240, in _commit
return self.connection.commit()
django.db.utils.IntegrityError: update or delete on table "core_reservedresource" violates foreign key constraint "core_taskreservedres_resource_id_ee0b7c62_fk_core_rese" on table "core_taskreservedresource"
DETAIL: Key (pulp_id)=(c0b7db83-401e-4a46-8d9e-07f5ab70de1e) is still referenced from table "core_taskreservedresource".
pulp [None]: rq.worker:INFO: 28341@pipe-up-katello-proxy-nightly-centos7.n60.example.com: a9f5653c-8d52-47ea-b36b-52f86d91e9a3
</code></pre> RPM Support - Issue #7786 (CLOSED - WORKSFORME): Pulp_rpm sync reports "completed" but does not u...https://pulp.plan.io/issues/77862020-11-04T07:51:35ZAant
<p>This is a copy of <a href="https://pulp.plan.io/issues/7535?next_issue_id=7534" class="external">https://pulp.plan.io/issues/7535?next_issue_id=7534</a>. The issue persists.
Even if I delete and recreate the publication and distribution after synchronization I still experience this issue.
This is a very severe bug as it makes me believe that my systems are up to date while they are not.</p>
<a name="What-would-be-the-proper-workflow-to-resynchronise-a-repo-after-it-is-created"></a>
<h2 >What would be the proper workflow to resynchronise a repo after it is created?<a href="#What-would-be-the-proper-workflow-to-resynchronise-a-repo-after-it-is-created" class="wiki-anchor">¶</a></h2>
<p>I have set up Pulpcore v3.6.3 with Pulp_rpm v3.6.2 - later updated both to 3.7.0.
I created the following remote:</p>
<pre><code>{
"pulp_href": "/pulp/api/v3/remotes/rpm/rpm/d8fb524c-e73b-4afc-ba2e-05eea7f5b6a8/",
"pulp_created": "2020-08-18T08:47:19.183836Z",
"name": "centos8_base",
"url": "http://mirror.centos.org/centos-8/8/BaseOS/x86_64/os/",
"ca_cert": null,
"client_cert": null,
"client_key": null,
"tls_validation": true,
"proxy_url": "http://<proxy server>",
"username": null,
"password": null,
"pulp_last_updated": "2020-08-18T08:47:19.183855Z",
"download_concurrency": 10,
"policy": "immediate",
"sles_auth_token": null
}
</code></pre>
<p>and this repository:</p>
<pre><code>{
"pulp_href": "/pulp/api/v3/repositories/rpm/rpm/7696b222-3788-4754-b4a6-747e64228d7a/",
"pulp_created": "2020-08-18T08:47:17.704104Z",
"versions_href": "/pulp/api/v3/repositories/rpm/rpm/7696b222-3788-4754-b4a6-747e64228d7a/versions/",
"latest_version_href": "/pulp/api/v3/repositories/rpm/rpm/7696b222-3788-4754-b4a6-747e64228d7a/versions/1/",
"name": "centos8_base",
"description": null,
"remote": null,
"metadata_signing_service": null,
"retain_package_versions": 0
}
</code></pre>
<p>If I sync, it reports success:</p>
<pre><code>http --auth admin --auth-type basic :9000/pulp/api/v3/repositories/rpm/rpm/7696b222-3788-4754-b4a6-747e64228d7a/sync/ remote=/pulp/api/v3/remotes/rpm/rpm/d8fb524c-e73b-4afc-ba2e-05eea7f5b6a8/
http: password for admin@localhost:9000:
HTTP/1.1 202 Accepted
Allow: POST, OPTIONS
Connection: close
Content-Length: 67
Content-Type: application/json
Date: Mon, 21 Sep 2020 08:10:11 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN
{
"task": "/pulp/api/v3/tasks/655d2e7b-2783-4496-a851-3bcd8d5e442e/"
}
</code></pre>
<p>Let's take a closer look at the task:</p>
<pre><code> http --auth admin --auth-type basic :9000/pulp/api/v3/tasks/655d2e7b-2783-4496-a851-3bcd8d5e442e/
http: password for admin@localhost:9000:
HTTP/1.1 200 OK
Allow: GET, PATCH, DELETE, HEAD, OPTIONS
Connection: close
Content-Length: 743
Content-Type: application/json
Date: Mon, 21 Sep 2020 08:10:29 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN
{
"child_tasks": [],
"created_resources": [],
"error": null,
"finished_at": "2020-09-21T08:10:11.585792Z",
"name": "pulp_rpm.app.tasks.synchronizing.synchronize",
"parent_task": null,
"progress_reports": [
{
"code": "optimizing.sync",
"done": 1,
"message": "Optimizing Sync",
"state": "completed",
"suffix": null,
"total": null
}
],
"pulp_created": "2020-09-21T08:10:11.257796Z",
"pulp_href": "/pulp/api/v3/tasks/655d2e7b-2783-4496-a851-3bcd8d5e442e/",
"reserved_resources_record": [
"/pulp/api/v3/remotes/rpm/rpm/d8fb524c-e73b-4afc-ba2e-05eea7f5b6a8/",
"/pulp/api/v3/repositories/rpm/rpm/7696b222-3788-4754-b4a6-747e64228d7a/"
],
"started_at": "2020-09-21T08:10:11.507912Z",
"state": "completed",
"task_group": null,
"worker": "/pulp/api/v3/workers/80ca382e-fdec-4eef-94b4-6c1c67433511/"
}
</code></pre>
<p>Let's use this repo on the same machine:</p>
<pre><code>cat /etc/yum.repos.d/CentOS-Base.repo
[Base]
name=CentOS-$releasever - Base
enabled=1
baseurl=https://<myurl>/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
</code></pre>
<p>Now if I do <code>yum check-update</code> it does not find any package to update.
If I change <code>baseurl=</code> to <code>http://mirror.centos.org/centos-8/8/BaseOS/x86_64/os/</code> (which is the same url I have in the remote) and I do <code>yum check-update</code>, then it finds 28 outdated packages. How is it possible?</p> Pulp - Issue #7676 (CLOSED - CURRENTRELEASE): django-cleanup can silently delete files being used...https://pulp.plan.io/issues/76762020-10-08T14:15:28Zdalleydalley@redhat.com
<pre><code>Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: pulp: pulp_rpm.app.tasks.publishing:INFO: Publishing: repository=centos8-baseos, version=1
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: pulp: rq.worker:ERROR: Traceback (most recent call last):
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: rv = job.perform()
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 684, in perform
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: self._result = self._execute()
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 690, in _execute
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: return self.func(*self.args, **self.kwargs)
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/tasks/migrate.py", line 142, in migrate_from_pulp2
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: create_repoversions_publications_distributions(plan)
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/migration.py", line 323, in create_repoversions_publications_distributions
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: task_func(*task_args)
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/migration.py", line 270, in complex_repo_migration
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: migrated_repo.pulp3_repository_version
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/migration.py", line 391, in migrate_repo_distributor
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: pulp2dist, repo_version)
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/rpm/repository.py", line 74, in migrate_to_pulp3
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: publish(repo_version.pk, checksum_types=checksum_types)
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: File "/home/vagrant/devel/pulp_rpm/pulp_rpm/app/tasks/publishing.py", line 283, in publish
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: publication_data.populate()
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: File "/home/vagrant/devel/pulp_rpm/pulp_rpm/app/tasks/publishing.py", line 224, in populate
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: self.handle_sub_repos(distribution_tree)
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: File "/home/vagrant/devel/pulp_rpm/pulp_rpm/app/tasks/publishing.py", line 165, in handle_sub_repos
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: artifact_file = storage.open(original_treeinfo_content_artifact.artifact.file.name)
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/files/storage.py", line 36, in open
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: return self._open(name, mode)
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/files/storage.py", line 224, in _open
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: return File(open(self.path(name), mode))
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/pulp/artifact/48/66eceaf78ffabb1d25774932085c6653f9f654411d7669db7b91e2ef0fd2a3'
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: Traceback (most recent call last):
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: rv = job.perform()
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 684, in perform
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: self._result = self._execute()
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 690, in _execute
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: return self.func(*self.args, **self.kwargs)
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/tasks/migrate.py", line 142, in migrate_from_pulp2
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: create_repoversions_publications_distributions(plan)
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/migration.py", line 323, in create_repoversions_publications_distributions
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: task_func(*task_args)
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/migration.py", line 270, in complex_repo_migration
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: migrated_repo.pulp3_repository_version
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/migration.py", line 391, in migrate_repo_distributor
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: pulp2dist, repo_version)
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/plugin/rpm/repository.py", line 74, in migrate_to_pulp3
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: publish(repo_version.pk, checksum_types=checksum_types)
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: File "/home/vagrant/devel/pulp_rpm/pulp_rpm/app/tasks/publishing.py", line 283, in publish
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: publication_data.populate()
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: File "/home/vagrant/devel/pulp_rpm/pulp_rpm/app/tasks/publishing.py", line 224, in populate
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: self.handle_sub_repos(distribution_tree)
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: File "/home/vagrant/devel/pulp_rpm/pulp_rpm/app/tasks/publishing.py", line 165, in handle_sub_repos
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: artifact_file = storage.open(original_treeinfo_content_artifact.artifact.file.name)
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/files/storage.py", line 36, in open
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: return self._open(name, mode)
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/files/storage.py", line 224, in _open
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: return File(open(self.path(name), mode))
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/pulp/artifact/48/66eceaf78ffabb1d25774932085c6653f9f654411d7669db7b91e2ef0fd2a3'
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: pulp: rq.worker:INFO: 26855@pulp2-nightly-pulp3-source-centos7.localhost.example.com: c009a655-bb3e-4665-9b2e-b3100b82b060
Oct 08 13:30:37 pulp2-nightly-pulp3-source-centos7.localhost.example.com rq[26855]: pulp: rq.worker:INFO: 26855@pulp2-nightly-pulp3-source-centos7.localhost.example.com: Job OK (c009a655-bb3e-4665-9b2e-b3100b82b060)
</code></pre> RPM Support - Issue #7535 (CLOSED - WORKSFORME): Pulp_rpm sync reports "completed" but does not u...https://pulp.plan.io/issues/75352020-09-21T08:28:31ZAant
<p>I have set up Pulpcore v3.6.3 with Pulp_rpm v3.6.2.
I created the following remote:</p>
<pre><code>{
"pulp_href": "/pulp/api/v3/remotes/rpm/rpm/d8fb524c-e73b-4afc-ba2e-05eea7f5b6a8/",
"pulp_created": "2020-08-18T08:47:19.183836Z",
"name": "centos8_base",
"url": "http://mirror.centos.org/centos-8/8/BaseOS/x86_64/os/",
"ca_cert": null,
"client_cert": null,
"client_key": null,
"tls_validation": true,
"proxy_url": "http://<proxy server>",
"username": null,
"password": null,
"pulp_last_updated": "2020-08-18T08:47:19.183855Z",
"download_concurrency": 10,
"policy": "immediate",
"sles_auth_token": null
}
</code></pre>
<p>and this repository:</p>
<pre><code>{
"pulp_href": "/pulp/api/v3/repositories/rpm/rpm/7696b222-3788-4754-b4a6-747e64228d7a/",
"pulp_created": "2020-08-18T08:47:17.704104Z",
"versions_href": "/pulp/api/v3/repositories/rpm/rpm/7696b222-3788-4754-b4a6-747e64228d7a/versions/",
"latest_version_href": "/pulp/api/v3/repositories/rpm/rpm/7696b222-3788-4754-b4a6-747e64228d7a/versions/1/",
"name": "centos8_base",
"description": null,
"remote": null,
"metadata_signing_service": null,
"retain_package_versions": 0
}
</code></pre>
<p>If I sync, it reports success:</p>
<pre><code>http --auth admin --auth-type basic :9000/pulp/api/v3/repositories/rpm/rpm/7696b222-3788-4754-b4a6-747e64228d7a/sync/ remote=/pulp/api/v3/remotes/rpm/rpm/d8fb524c-e73b-4afc-ba2e-05eea7f5b6a8/
http: password for admin@localhost:9000:
HTTP/1.1 202 Accepted
Allow: POST, OPTIONS
Connection: close
Content-Length: 67
Content-Type: application/json
Date: Mon, 21 Sep 2020 08:10:11 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN
{
"task": "/pulp/api/v3/tasks/655d2e7b-2783-4496-a851-3bcd8d5e442e/"
}
</code></pre>
<p>Let's take a closer look at the task:</p>
<pre><code> http --auth admin --auth-type basic :9000/pulp/api/v3/tasks/655d2e7b-2783-4496-a851-3bcd8d5e442e/
http: password for admin@localhost:9000:
HTTP/1.1 200 OK
Allow: GET, PATCH, DELETE, HEAD, OPTIONS
Connection: close
Content-Length: 743
Content-Type: application/json
Date: Mon, 21 Sep 2020 08:10:29 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN
{
"child_tasks": [],
"created_resources": [],
"error": null,
"finished_at": "2020-09-21T08:10:11.585792Z",
"name": "pulp_rpm.app.tasks.synchronizing.synchronize",
"parent_task": null,
"progress_reports": [
{
"code": "optimizing.sync",
"done": 1,
"message": "Optimizing Sync",
"state": "completed",
"suffix": null,
"total": null
}
],
"pulp_created": "2020-09-21T08:10:11.257796Z",
"pulp_href": "/pulp/api/v3/tasks/655d2e7b-2783-4496-a851-3bcd8d5e442e/",
"reserved_resources_record": [
"/pulp/api/v3/remotes/rpm/rpm/d8fb524c-e73b-4afc-ba2e-05eea7f5b6a8/",
"/pulp/api/v3/repositories/rpm/rpm/7696b222-3788-4754-b4a6-747e64228d7a/"
],
"started_at": "2020-09-21T08:10:11.507912Z",
"state": "completed",
"task_group": null,
"worker": "/pulp/api/v3/workers/80ca382e-fdec-4eef-94b4-6c1c67433511/"
}
</code></pre>
<p>Let's use this repo on the same machine:</p>
<pre><code>cat /etc/yum.repos.d/CentOS-Base.repo
[Base]
name=CentOS-$releasever - Base
enabled=1
baseurl=https://<myurl>/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
</code></pre>
<p>Now if I do <code>yum check-update</code> it does not find any package to update.
If I change <code>baseurl=</code> to <code>http://mirror.centos.org/centos-8/8/BaseOS/x86_64/os/</code> (which is the same url I have in the remote) and I do <code>yum check-update</code>, then it finds 28 outdated packages. How is it possible?</p> RPM Support - Issue #6861 (CLOSED - CURRENTRELEASE): Migration from 3.2.0 to 3.3.z is brokenhttps://pulp.plan.io/issues/68612020-05-29T16:37:58Zdalleydalley@redhat.com
<p>In trying to resolve the migration issues between 3.3.0 and 3.3.1, we accidentally introduced a new blocking issue between 3.2.0 and 3.3.z.</p>
<p>Both migration 0005 and 0009 need to be modified, again, such that they both specify that 'last_sync_revision_number' is nullable.</p> RPM Support - Issue #6743 (CLOSED - CURRENTRELEASE): Installations migrating from 3.3.0 to 3.3.1 ...https://pulp.plan.io/issues/67432020-05-14T19:27:32Zdalleydalley@redhat.com
<p>By modifying a prior migration, we accidentally created a mismatch between the schema we expect and the schema Django seems to interpret and enforce, presumably due to some kind of optimization being employed.</p>
<p>It seems that because Django sees that migration 0005 made the 'last_sync_revision_number' column nullable (it didn't, it was modified afterwards) [0], it ignores when migration 0009 tells Django to perform that migration later. Thus the 'last_sync_revision_number' column remains non-nullable.</p>
<p>This impacts Pulp users by throwing the following error when creating new RPM repositories.</p>
<pre><code>django.db.utils.IntegrityError: null value in column "last_sync_revision_number" violates not-null constraint
</code></pre>
<p>[0] <a href="https://github.com/pulp/pulp_rpm/pull/1694/files#diff-8882c765a6805ada6c8a8bafe3e3e5b8" class="external">https://github.com/pulp/pulp_rpm/pull/1694/files#diff-8882c765a6805ada6c8a8bafe3e3e5b8</a></p> Pulp - Issue #6438 (CLOSED - WORKSFORME): Pulp 3.2.1 not able to sync to redhathttps://pulp.plan.io/issues/64382020-04-02T13:47:18Zbinlinf0
<p>After upgrading from 3.1 to 3.2.1, we no longer able to sync to redhat repo. Below is the errors:</p>
<p>"error": {
"description": "[PEM: NO_START_LINE] no start line (_ssl.c:3947)",
"traceback": " File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py", line 884, in perform_job\n rv = job.perform()\n File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line 664, in perform\n self._result = self._execute()\n File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line 670, in _execute\n return self.func(*self.args, **self.kwargs)\n File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 126, in synchronize\n treeinfo = get_treeinfo_data(remote)\n File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/kickstart/treeinfo.py", line 21, in get_treeinfo_data\n downloader = remote.get_downloader(url=urljoin(remote_url, namespace))\n File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/app/models/repository.py", line 287, in get_downloader\n return self.download_factory.build(url, **kwargs)\n File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/app/models/repository.py", line 245, in download_factory\n self._download_factory = DownloaderFactory(self)\n File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/factory.py", line 68, in <strong>init</strong>\n self._session = self._make_aiohttp_session_from_remote()\n File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/download/factory.py", line 85, in _make_aiohttp_session_from_remote\n sslcontext = ssl.create_default_context(cadata=self._remote.ca_cert)\n File "/opt/python/3.7.3/lib64/python3.7/ssl.py", line 573, in create_default_context\n context.load_verify_locations(cafile, capath, cadata)\n"</p>
<p>The works fine before updating to 3.2.1. We also tried overwrite all cert in pulp 3.2.1 and it didn't help.</p> Ansible Plugin - Issue #5571 (CLOSED - CURRENTRELEASE): Add the ability to filter collection vers...https://pulp.plan.io/issues/55712019-10-14T20:13:14Znewswangerd
<p>We need a parameter on `/pulp/api/v3/content/ansible/collection_versions/` that allows for us to filter collection versions by tag, such that we can specify `tags=?network,cloud` and get a list of collection versions that have the network AND cloud tags.</p> Debian Support - Issue #5377 (CLOSED - CURRENTRELEASE): Cannot modify, delete, or duplicate contenthttps://pulp.plan.io/issues/53772019-08-30T18:52:56Zjaymzh
<p>Once an artifact has been made into apt content, that content cannot be modified or deleted. Further, you can't work around this by creating new content because multiple contents cannot point to the same artifact because content cannot have the same sha256 sum.</p>
<p>As such if you create content incorrectly, there is literally now way out of it.</p> Debian Support - Issue #3094 (CLOSED - DUPLICATE): Unable to cancel pending unstarted taskshttps://pulp.plan.io/issues/30942017-10-23T21:40:42ZAnonymous
<p>1. Created Consumers<br>
2. Created Consumer Groups for Patching<br>
3. Deleted some Consumers<br>
4. Run Patching on the Consumer Group<br>
5. Orphan Tasks that cannot be deleted</p>
<p>pulp-admin tasks cancel --task-id <task-id></p>
<p>The following resources(s) could not be found</p>
<p><consumer.hostname> (consumer)</p>
<p>Is there a workaround for this kind of issue?</p> Python Support - Issue #2334 (CLOSED - NOTABUG): Python egg uploads failhttps://pulp.plan.io/issues/23342016-10-13T15:21:02ZIchimonji10jerebear@protonmail.com
<p>Let's say one decides to upload a Python egg with pulp-admin. The following script for doing so succeeds:</p>
<pre><code class="bash syntaxhl" data-language="bash">wget https://repos.fedorapeople.org/pulp/pulp/fixtures/python/packages/source/s/shelf-reader/shelf-reader-0.1.tar.gz
pulp-admin python repo create <span class="nt">--repo-id</span> foo
pulp-admin python repo upload <span class="nt">--repo-id</span> foo <span class="nt">--file</span> shelf-reader-0.1.tar.gz
</code></pre>
<p>On the other hand, let's say one uploads directly via the API. Doing so is a four-step process:</p>
<p>1. Create an upload request.<br>
2. Make one or more PUT requests, sending a chunk of the file each time.<br>
3. Import the file into a repository.<br>
4. Close the upload request.</p>
<p>This has worked since at least Pulp 2.8. Unfortunately, the code for accomplishing step 3 has changed in Pulp 2.11. A POST request to <code>/{repo_href}/actions/import_upload/</code> with the following body will succeed in Pulp 2.8, 2.9 and 2.10:</p>
<pre><code class="json syntaxhl" data-language="json"><span class="p">{</span><span class="w">
</span><span class="nl">"unit_key"</span><span class="p">:</span><span class="w"> </span><span class="p">{},</span><span class="w">
</span><span class="nl">"unit_type_id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"python_package"</span><span class="p">,</span><span class="w">
</span><span class="nl">"upload_id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"..."</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre>
<p>In Pulp 2.11, this request will instead cause Pulp to respond with the following failed task:</p>
<pre><code>{'_href': '/pulp/api/v2/tasks/37de3871-e878-4abc-9c6f-0d39c86cd4bb/',
'_id': {'$oid': '57ff9b425137beaefc1f9d0b'},
'_ns': 'task_status',
'error': {'code': 'PLP0000',
'data': {},
'description': 'Pulp exception occurred: PulpExecutionException',
'sub_errors': []},
'exception': None,
'finish_time': '2016-10-13T14:33:38Z',
'id': '57ff9b425137beaefc1f9d0b',
'progress_report': {},
'queue': 'reserved_resource_worker-0@localhost.localdomain.dq',
'result': None,
'spawned_tasks': [],
'start_time': '2016-10-13T14:33:38Z',
'state': 'error',
'tags': ['pulp:repository:a962599e-2dfd-4560-8dfa-d6d391d141fa',
'pulp:action:import_upload'],
'task_id': '37de3871-e878-4abc-9c6f-0d39c86cd4bb',
'task_type': 'pulp.server.managers.content.upload.import_uploaded_unit',
'traceback': 'Traceback (most recent call last):\n'
' File "/usr/lib/python2.7/site-packages/celery/app/trace.py", '
'line 240, in trace_task\n'
' R = retval = fun(*args, **kwargs)\n'
' File '
'"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", '
'line 488, in __call__\n'
' return super(Task, self).__call__(*args, **kwargs)\n'
' File '
'"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", '
'line 103, in __call__\n'
' return super(PulpTask, self).__call__(*args, **kwargs)\n'
' File "/usr/lib/python2.7/site-packages/celery/app/trace.py", '
'line 437, in __protected_call__\n'
' return self.run(*args, **kwargs)\n'
' File '
'"/usr/lib/python2.7/site-packages/pulp/server/managers/content/upload.py", '
'line 218, in import_uploaded_unit\n'
' unit_metadata, file_path, conduit, call_config)\n'
' File '
'"/usr/lib/python2.7/site-packages/pulp_python/plugins/importers/importer.py", '
'line 191, in upload_unit\n'
' new_file_path = os.path.join(working_dir, '
"unit_key['filename'])\n"
'PulpExecutionException: Pulp exception occurred: '
'PulpExecutionException\n',
'worker_name': 'reserved_resource_worker-0@localhost.localdomain'}
</code></pre>
<p>Here's a snippet from journalctl:</p>
<pre><code>Oct 13 10:33:38 localhost.localdomain pulp[7917]: pulp.server.managers.content.upload:ERROR: (7917-13792) Error from the importer while importing uploaded unit to repository [a962599e-2dfd-4560-8dfa-d6d391d141fa]
Oct 13 10:33:38 localhost.localdomain pulp[7917]: pulp.server.managers.content.upload:ERROR: (7917-13792) Traceback (most recent call last):
Oct 13 10:33:38 localhost.localdomain pulp[7917]: pulp.server.managers.content.upload:ERROR: (7917-13792) File "/usr/lib/python2.7/site-packages/pulp/server/managers/content/upload.py", line 218, in import_uploaded_unit
Oct 13 10:33:38 localhost.localdomain pulp[7917]: pulp.server.managers.content.upload:ERROR: (7917-13792) unit_metadata, file_path, conduit, call_config)
Oct 13 10:33:38 localhost.localdomain pulp[7917]: pulp.server.managers.content.upload:ERROR: (7917-13792) File "/usr/lib/python2.7/site-packages/pulp_python/plugins/importers/importer.py", line 191, in upload_unit
Oct 13 10:33:38 localhost.localdomain pulp[7917]: pulp.server.managers.content.upload:ERROR: (7917-13792) new_file_path = os.path.join(working_dir, unit_key['filename'])
Oct 13 10:33:38 localhost.localdomain pulp[7917]: pulp.server.managers.content.upload:ERROR: (7917-13792) KeyError: 'filename'
Oct 13 10:33:38 localhost.localdomain pulp[7917]: pulp.server.async.tasks:INFO: Task failed : [fe698de5-9172-4994-ad37-d0c14188b505]
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792) Task pulp.server.managers.content.upload.import_uploaded_unit[fe698de5-9172-4994-ad37-d0c14188b505] raised unexpected: PulpExecutionException(KeyError('filename',),)
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792) Traceback (most recent call last):
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792) R = retval = fun(*args, **kwargs)
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 488, in __call__
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792) return super(Task, self).__call__(*args, **kwargs)
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 103, in __call__
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792) return super(PulpTask, self).__call__(*args, **kwargs)
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792) return self.run(*args, **kwargs)
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792) File "/usr/lib/python2.7/site-packages/pulp/server/managers/content/upload.py", line 218, in import_uploaded_unit
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792) unit_metadata, file_path, conduit, call_config)
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792) File "/usr/lib/python2.7/site-packages/pulp_python/plugins/importers/importer.py", line 191, in upload_unit
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792) new_file_path = os.path.join(working_dir, unit_key['filename'])
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792) PulpExecutionException: Pulp exception occurred: PulpExecutionException
</code></pre>
<p>The culprit seems to be that Pulp now expects the following POST request body:</p>
<pre><code class="json syntaxhl" data-language="json"><span class="p">{</span><span class="w">
</span><span class="nl">"unit_key"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="nl">"filename"</span><span class="p">:</span><span class="w"> </span><span class="s2">"shelf-reader-0.1.tar.gz"</span><span class="p">},</span><span class="w">
</span><span class="nl">"unit_type_id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"python_package"</span><span class="p">,</span><span class="w">
</span><span class="nl">"upload_id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"..."</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre>
<p>This change is backward-incompatible. It also seems unnecessary, as the filename can be derived from the package metadata.</p>
<p>If this information is needed when uploading packages, it should be documented in one of the following locations:</p>
<ul>
<li><a href="http://docs.pulpproject.org/dev-guide/integration/rest-api/content/upload.html#import-into-a-repository" class="external">http://docs.pulpproject.org/dev-guide/integration/rest-api/content/upload.html#import-into-a-repository</a></li>
<li><a href="http://docs.pulpproject.org/plugins/pulp_python/user-docs/getting_started.html#upload-a-python-package" class="external">http://docs.pulpproject.org/plugins/pulp_python/user-docs/getting_started.html#upload-a-python-package</a></li>
</ul>