Issue #8411
closed
Inconsistent data in upstream remote repo causes inconsistent of pulp content
Description
Ticket moved to GitHub: "pulp/pulpcore/1980":https://github.com/pulp/pulpcore/issues/1980
The issues started inconstancy of repo metadata checksum and actual checksum in upstream remote repo. With the patch https://gist.github.com/daviddavis/2e6ab1872d97230d144a6cd1f9d05e31, the sync to upstream failed with following errors: Received checksum b8b257c32135daf51e703d439594f1a676871d7d for http://something/something/flume-1.9.0-1.noarch.rpm but expected c281a94a354178c42800d47b63479c2621772351
Once the upstream repo was fixed, the subsequence sync keep failing with: "error": { "description": "Package matching query does not exist.", "traceback": " File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/worker.py", line 886, in perform_job\n rv = job.perform()\n File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line 664, in perform\n self._result = self._execute()\n File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/rq/job.py", line 670, in _execute\n return self.func(*self.args, **self.kwargs)\n File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 266, in synchronize\n dv.create()\n File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/declarative_version.py", line 148, in create\n loop.run_until_complete(pipeline)\n File "/opt/python/3.7.3/lib64/python3.7/asyncio/base_events.py", line 584, in run_until_complete\n return future.result()\n File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline\n await asyncio.gather(*futures)\n File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/api.py", line 43, in call\n await self.run()\n File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/pulpcore/plugin/stages/content_stages.py", line 105, in run\n d_content.content.q()\n File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/manager.py", line 82, in manager_method\n return getattr(self.get_queryset(), name)(*args, **kwargs)\n File "/opt/utils/venv/pulp/3.7.3/lib64/python3.7/site-packages/django/db/models/query.py", line 408, in get\n self.model._meta.object_name\n"
We deleted the repo which failed sync and run "delete localhost/pulp/api/v3/orphans/" but keep getting the same errors.
Query the database doesn't provide any result of the package which cause the issue originally
=> select name from rpm_package where name like 'flume%' limit 100; name
(0 rows)
=> select checksum from rpm_checksum where checksum like '%594f1a676871d7d' OR checksum like '%63479c2621772351'; checksum
(0 rows)
=> select name from rpm_package where name like 'flume%' limit 100; name
(0 rows)
=> select checksum from rpm_checksum where checksum like '%594f1a676871d7d' OR checksum like '%63479c2621772351'; checksum
(0 rows)
Updated by dalley about 3 years ago
I think step 1 is to figure out why a DigestValidationError that was (or should have been) raised didn't prevent the invalid artifact file from working its way through the pipeline and being saved.
Adding David's patch to make sure it gets logged is probably also a good idea.
Updated by fao89 about 3 years ago
- Triaged changed from No to Yes
- Sprint set to Sprint 93
Updated by rchan almost 3 years ago
- Sprint changed from Sprint 100 to Sprint 101
Updated by ipanova@redhat.com over 2 years ago
- Sprint changed from Sprint 101 to Sprint 102
Updated by macado over 2 years ago
I seem to also be running into this issue
[root@puppetmaster-prod-01 foreman]# rpm -qa| grep pulpcore tfm-rubygem-pulpcore_client-3.7.1-1.el7.noarch python3-pulpcore-3.7.9-1.el7.noarch
Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: pulp: rq.worker:ERROR: Traceback (most recent call last): Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: return self.cursor.execute(sql, params) Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "rpm_package_pkgId_key" Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: DETAIL: Key ("pkgId")=(2f4224dee658e772cd0b7fbf81a6d2a62871440b) already exists. Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: The above exception was the direct cause of the following exception: Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: Traceback (most recent call last): Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/content_stages.py", line 102, in run Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: d_content.content.save() Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulpcore/app/models/base.py", line 115, in save Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: return super().save(*args, **kwargs) Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/django_lifecycle/mixins.py", line 129, in save Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: save(*args, **kwargs) Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 744, in save Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: force_update=force_update, update_fields=update_fields) Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 782, in save_base Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: force_update, using, update_fields, Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 873, in _save_table Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: result = self._do_insert(cls._base_manager, using, fields, update_pk, raw) Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 911, in _do_insert Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: using=using, raw=raw) Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: return getattr(self.get_queryset(), name)(*args, **kwargs) Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 1186, in _insert Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: return query.get_compiler(using=using).execute_sql(return_id) Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1377, in execute_sql Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: cursor.execute(sql, params) Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-content: [07/Dec/2021:20:15:20 +0000] "GET /pulp/content/Tufts_University/Library/RHEL6-ESAI-Default/custom/Extra_Packages_for_Enterprise_Linux/EPEL-RHEL6/repodata/repomd.xml HTTP/1.1" 200 4089 "-" "urlgrabber/3.9.1 yum/3.2.29" Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: return executor(sql, params, many, context) Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: return self.cursor.execute(sql, params) Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/django/db/utils.py", line 89, in exit Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: raise dj_exc_value.with_traceback(traceback) from exc_value Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: return self.cursor.execute(sql, params) Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: django.db.utils.IntegrityError: duplicate key value violates unique constraint "rpm_package_pkgId_key" Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: DETAIL: Key ("pkgId")=(2f4224dee658e772cd0b7fbf81a6d2a62871440b) already exists. Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: During handling of the above exception, another exception occurred: Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: Traceback (most recent call last): Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: rv = job.perform() Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: self._result = self._execute() Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: return self.func(*self.args, **self.kwargs) Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 269, in synchronize Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: dv.create() Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 148, in create Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: loop.run_until_complete(pipeline) Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: return future.result() Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: await asyncio.gather(*futures) Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in call Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: await self.run() Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/content_stages.py", line 105, in run Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: d_content.content.q() Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: return getattr(self.get_queryset(), name)(*args, **kwargs) Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 408, in get Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: self.model._meta.object_name Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: pulp_rpm.app.models.package.Package.DoesNotExist: Package matching query does not exist. Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: Traceback (most recent call last): Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: return self.cursor.execute(sql, params) Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "rpm_package_pkgId_key" Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: DETAIL: Key ("pkgId")=(2f4224dee658e772cd0b7fbf81a6d2a62871440b) already exists. Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: The above exception was the direct cause of the following exception: Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: Traceback (most recent call last): Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/content_stages.py", line 102, in run Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: d_content.content.save() Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulpcore/app/models/base.py", line 115, in save Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: return super().save(*args, **kwargs) Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/django_lifecycle/mixins.py", line 129, in save Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: save(*args, **kwargs) Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 744, in save Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: force_update=force_update, update_fields=update_fields) Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 782, in save_base Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: force_update, using, update_fields, Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 873, in _save_table Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: result = self._do_insert(cls._base_manager, using, fields, update_pk, raw) Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 911, in _do_insert Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: using=using, raw=raw) Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: return getattr(self.get_queryset(), name)(*args, **kwargs) Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 1186, in _insert Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: return query.get_compiler(using=using).execute_sql(return_id) Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1377, in execute_sql Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: cursor.execute(sql, params) Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: return executor(sql, params, many, context) Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: return self.cursor.execute(sql, params) Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/django/db/utils.py", line 89, in exit Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: raise dj_exc_value.with_traceback(traceback) from exc_value Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: return self.cursor.execute(sql, params) Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: django.db.utils.IntegrityError: duplicate key value violates unique constraint "rpm_package_pkgId_key" Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: DETAIL: Key ("pkgId")=(2f4224dee658e772cd0b7fbf81a6d2a62871440b) already exists. Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: During handling of the above exception, another exception occurred: Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: Traceback (most recent call last): Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: rv = job.perform() Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: self._result = self._execute() Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: return self.func(*self.args, **self.kwargs) Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 269, in synchronize Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: dv.create() Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 148, in create Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: loop.run_until_complete(pipeline) Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: return future.result() Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: await asyncio.gather(*futures) Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in call Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: await self.run() Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/content_stages.py", line 105, in run Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: d_content.content.q() Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: return getattr(self.get_queryset(), name)(*args, **kwargs) Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 408, in get Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: self.model._meta.object_name Dec 7 15:15:20 puppetmaster-prod-01 pulpcore-worker-2: pulp_rpm.app.models.package.Package.DoesNotExist: Package matching query does not exist.
Updated by pulpbot over 2 years ago
- Description updated (diff)
- Status changed from NEW to CLOSED - DUPLICATE
.