Issue #8400
Updated by ttereshc over 3 years ago
Hi, After upgrade foreman and katello, (based on https://theforeman.org/plugins/katello/3.18/upgrade/index.html) the Pulp3 Migration failed. With this error: ForemanTasks::TaskError: Task 3568639a-17d0-4198-a12a-8b0a7d29cfb0: Katello::Errors::Pulp3Error: 1 subtask(s) failed for task group /pulp/api/v3/task-groups/2e11a580-8850-4f2f-ade4-d27d5f9e9481/. They, Foreman team, have already verified the logs and suggested me that I open a case with you. * pip3 list | grep pulp * pulp-2to3-migration (0.8.0) * pulp-certguard (1.0.3) * pulp-container (2.1.0) * pulp-deb (2.7.0) * pulp-file (1.3.0) * pulp-rpm (3.7.0) * pulpcore (3.7.3) I show you here, fragments of /var/log/messages around the time of the last error: ~~~ … Mar 11 15:07:10 cpt-infra22l-d pulpcore-api[25051]: - - [11/Mar/2021:20:07:10 +0000] “GET /pulp/api/v3/task-groups/2e11a580-8850-4f2f-ade4-d27d5f9e9481/ HTTP/1.1” 200 440 “-” “OpenAPI-Generator/3.7.1/ruby” Mar 11 15:07:12 cpt-infra22l-d pulpcore-api[25051]: - - [11/Mar/2021:20:07:12 +0000] “GET /pulp/api/v3/tasks/ea052d37-c235-4b4a-9cef-4ce62718c1c6/ HTTP/1.1” 200 3960 “-” “OpenAPI-Generator/3.7.1/ruby” Mar 11 15:07:12 cpt-infra22l-d pulpcore-api[25051]: - - [11/Mar/2021:20:07:12 +0000] “GET /pulp/api/v3/task-groups/2e11a580-8850-4f2f-ade4-d27d5f9e9481/ HTTP/1.1” 200 440 “-” “OpenAPI-Generator/3.7.1/ruby” Mar 11 15:07:13 cpt-infra22l-d pulpcore-worker-1[25158]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without an entry in the lazy catalog, pulp2 unit_id: 6b7f5719-ff27-4246-b86b-b53fd996881b Mar 11 15:07:13 cpt-infra22l-d pulpcore-worker-1[25158]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without an entry in the lazy catalog, pulp2 unit_id: 28d28583-a22b-43c2-b6d7-25263a1d36bf Mar 11 15:07:13 cpt-infra22l-d pulpcore-worker-1[25158]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without an entry in the lazy catalog, pulp2 unit_id: 51f5d2fd-e1cc-4e19-9389-f4d7da27ee7b … many similar lines … Mar 11 15:07:18 cpt-infra22l-d pulpcore-worker-1[25158]: pulp: pulp_2to3_migration.app.plugin.content:WARNING: On_demand content cannot be migrated without an entry in the lazy catalog, pulp2 unit_id: edb8dd22-8893-48a2-bf83-7e5f4cc206f9 Mar 11 15:07:19 cpt-infra22l-d pulpcore-resource-manager[25633]: pulp: rq.worker:INFO: resource-manager: 66f04d27-c44d-4eab-b78e-079370434c02 Mar 11 15:07:19 cpt-infra22l-d pulpcore-worker-3[25160]: pulp: rq.worker:INFO: 25160@cpt-infra22l-d.acceo.com: 5e8e263c-daaa-4d45-857e-de4a1698aa51 Mar 11 15:07:19 cpt-infra22l-d pulpcore-resource-manager[25633]: pulp: rq.worker:INFO: resource-manager: Job OK (66f04d27-c44d-4eab-b78e-079370434c02) Mar 11 15:07:19 cpt-infra22l-d pulpcore-resource-manager[25633]: pulp: rq.worker:INFO: resource-manager: 97a5e73c-794c-486d-8774-948bcd28559f … many similar lines … Mar 11 15:07:26 cpt-infra22l-d pulpcore-worker-2[25156]: pulp: rq.worker:INFO: 25156@cpt-infra22l-d.acceo.com: 584131d3-1f44-4927-af32-73bd6f432fe4 Mar 11 15:07:26 cpt-infra22l-d pulpcore-worker-2[25156]: pulp: rq.worker:INFO: 25156@cpt-infra22l-d.acceo.com: Job OK (584131d3-1f44-4927-af32-73bd6f432fe4) Mar 11 15:07:27 cpt-infra22l-d pulpcore-worker-3[25160]: pulp: pulp_rpm.app.tasks.publishing:INFO: Publishing: repository=f5b6756b-bd2e-402f-8269-8847b8bea678, version=3 Mar 11 15:07:29 cpt-infra22l-d pulpcore-api[25051]: - - [11/Mar/2021:20:07:29 +0000] “GET /pulp/api/v3/tasks/ea052d37-c235-4b4a-9cef-4ce62718c1c6/ HTTP/1.1” 200 11587 “-” “OpenAPI-Generator/3.7.1/ruby” Mar 11 15:07:29 cpt-infra22l-d pulpcore-api[25051]: - - [11/Mar/2021:20:07:29 +0000] “GET /pulp/api/v3/task-groups/2e11a580-8850-4f2f-ade4-d27d5f9e9481/ HTTP/1.1” 200 440 “-” “OpenAPI-Generator/3.7.1/ruby” Mar 11 15:07:33 cpt-infra22l-d pulpcore-api[25051]: - - [11/Mar/2021:20:07:33 +0000] “GET /pulp/api/v3/tasks/ea052d37-c235-4b4a-9cef-4ce62718c1c6/ HTTP/1.1” 200 11587 “-” “OpenAPI-Generator/3.7.1/ruby” … many similar lines … Mar 11 15:08:30 cpt-infra22l-d pulpcore-api[25051]: - - [11/Mar/2021:20:08:30 +0000] “GET /pulp/api/v3/tasks/ea052d37-c235-4b4a-9cef-4ce62718c1c6/ HTTP/1.1” 200 11587 “-” “OpenAPI-Generator/3.7.1/ruby” Mar 11 15:08:30 cpt-infra22l-d pulpcore-api[25051]: - - [11/Mar/2021:20:08:30 +0000] “GET /pulp/api/v3/task-groups/2e11a580-8850-4f2f-ade4-d27d5f9e9481/ HTTP/1.1” 200 440 “-” “OpenAPI-Generator/3.7.1/ruby” Mar 11 15:08:44 cpt-infra22l-d pulpcore-worker-4[25159]: pulp: rq.worker:ERROR: Traceback (most recent call last): Mar 11 15:08:44 cpt-infra22l-d pulpcore-worker-4[25159]: File “/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/repository.py”, line 78, in migrate_to_pulp3 Mar 11 15:08:44 cpt-infra22l-d pulpcore-worker-4[25159]: publish(repo_version.pk, checksum_types=checksum_types, sqlite_metadata=sqlite) Mar 11 15:08:44 cpt-infra22l-d pulpcore-worker-4[25159]: TypeError: publish() got an unexpected keyword argument ‘sqlite_metadata’ Mar 11 15:08:44 cpt-infra22l-d pulpcore-worker-4[25159]: During handling of the above exception, another exception occurred: Mar 11 15:08:44 cpt-infra22l-d pulpcore-worker-4[25159]: Traceback (most recent call last): Mar 11 15:08:44 cpt-infra22l-d pulpcore-worker-4[25159]: File “/usr/lib/python3.6/site-packages/rq/worker.py”, line 936, in perform_job Mar 11 15:08:44 cpt-infra22l-d pulpcore-worker-4[25159]: rv = job.perform() Mar 11 15:08:44 cpt-infra22l-d pulpcore-worker-4[25159]: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 684, in perform Mar 11 15:08:44 cpt-infra22l-d pulpcore-worker-4[25159]: self._result = self._execute() Mar 11 15:08:44 cpt-infra22l-d pulpcore-worker-4[25159]: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 690, in _execute Mar 11 15:08:44 cpt-infra22l-d pulpcore-worker-4[25159]: return self.func(*self.args, **self.kwargs) Mar 11 15:08:44 cpt-infra22l-d pulpcore-worker-4[25159]: File “/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py”, line 240, in complex_repo_migration Mar 11 15:08:44 cpt-infra22l-d pulpcore-worker-4[25159]: migrated_repo.pulp3_repository_version Mar 11 15:08:44 cpt-infra22l-d pulpcore-worker-4[25159]: File “/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py”, line 360, in migrate_repo_distributor Mar 11 15:08:44 cpt-infra22l-d pulpcore-worker-4[25159]: pulp2dist, repo_version) Mar 11 15:08:44 cpt-infra22l-d pulpcore-worker-4[25159]: File “/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/repository.py”, line 81, in migrate_to_pulp3 Mar 11 15:08:44 cpt-infra22l-d pulpcore-worker-4[25159]: publish(repo_version.pk, checksum_types=checksum_types) Mar 11 15:08:44 cpt-infra22l-d pulpcore-worker-4[25159]: File “/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py”, line 290, in publish Mar 11 15:08:44 cpt-infra22l-d pulpcore-worker-4[25159]: metadata_signing_service=metadata_signing_service Mar 11 15:08:44 cpt-infra22l-d pulpcore-worker-4[25159]: File “/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py”, line 483, in create_repomd_xml Mar 11 15:08:44 cpt-infra22l-d pulpcore-worker-4[25159]: record.fill(checksum_type) Mar 11 15:08:44 cpt-infra22l-d pulpcore-worker-4[25159]: OSError: Error while computing stat of compressed content of updateinfo.xml.zck:Cannot open a file updateinfo.xml.zck: createrepo_c wasn’t compiled with zchunk support Mar 11 15:08:44 cpt-infra22l-d pulpcore-worker-4[25159]: Traceback (most recent call last): Mar 11 15:08:44 cpt-infra22l-d pulpcore-worker-4[25159]: File “/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/repository.py”, line 78, in migrate_to_pulp3 Mar 11 15:08:44 cpt-infra22l-d pulpcore-worker-4[25159]: publish(repo_version.pk, checksum_types=checksum_types, sqlite_metadata=sqlite) Mar 11 15:08:44 cpt-infra22l-d pulpcore-worker-4[25159]: TypeError: publish() got an unexpected keyword argument ‘sqlite_metadata’ Mar 11 15:08:44 cpt-infra22l-d pulpcore-worker-4[25159]: During handling of the above exception, another exception occurred: Mar 11 15:08:44 cpt-infra22l-d pulpcore-worker-4[25159]: Traceback (most recent call last): … Mar 11 15:08:44 cpt-infra22l-d pulpcore-worker-4[25159]: record.fill(checksum_type) Mar 11 15:08:44 cpt-infra22l-d pulpcore-worker-4[25159]: OSError: Error while computing stat of compressed content of updateinfo.xml.zck:Cannot open a file updateinfo.xml.zck: createrepo_c wasn’t compiled with zchunk support Mar 11 15:08:44 cpt-infra22l-d pulpcore-worker-4[25159]: pulp: rq.worker:INFO: 25159@cpt-infra22l-d.acceo.com: 88d7a555-20bb-4ce3-9408-dd05005f9fcf Mar 11 15:08:44 cpt-infra22l-d pulpcore-worker-4[25159]: pulp: rq.worker:INFO: 25159@cpt-infra22l-d.acceo.com: Job OK (88d7a555-20bb-4ce3-9408-dd05005f9fcf) Mar 11 15:08:46 cpt-infra22l-d pulpcore-api[25051]: - - [11/Mar/2021:20:08:46 +0000] “GET /pulp/api/v3/tasks/ea052d37-c235-4b4a-9cef-4ce62718c1c6/ HTTP/1.1” 200 11587 “-” “OpenAPI-Generator/3.7.1/ruby” Mar 11 15:08:46 cpt-infra22l-d pulpcore-api[25051]: - - [11/Mar/2021:20:08:46 +0000] “GET /pulp/api/v3/task-groups/2e11a580-8850-4f2f-ade4-d27d5f9e9481/ HTTP/1.1” 200 440 “-” “OpenAPI-Generator/3.7.1/ruby” Mar 11 15:08:50 cpt-infra22l-d pulpcore-worker-1[25158]: pulp: rq.worker:ERROR: Traceback (most recent call last): Mar 11 15:08:50 cpt-infra22l-d pulpcore-worker-1[25158]: File “/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/repository.py”, line 78, in migrate_to_pulp3 Mar 11 15:08:50 cpt-infra22l-d pulpcore-worker-1[25158]: publish(repo_version.pk, checksum_types=checksum_types, sqlite_metadata=sqlite) Mar 11 15:08:50 cpt-infra22l-d pulpcore-worker-1[25158]: TypeError: publish() got an unexpected keyword argument ‘sqlite_metadata’ Mar 11 15:08:50 cpt-infra22l-d pulpcore-worker-1[25158]: During handling of the above exception, another exception occurred: Mar 11 15:08:50 cpt-infra22l-d pulpcore-worker-1[25158]: Traceback (most recent call last): Mar 11 15:08:50 cpt-infra22l-d pulpcore-worker-1[25158]: File “/usr/lib/python3.6/site-packages/rq/worker.py”, line 936, in perform_job Mar 11 15:08:50 cpt-infra22l-d pulpcore-worker-1[25158]: rv = job.perform() Mar 11 15:08:50 cpt-infra22l-d pulpcore-worker-1[25158]: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 684, in perform Mar 11 15:08:50 cpt-infra22l-d pulpcore-worker-1[25158]: self._result = self._execute() Mar 11 15:08:50 cpt-infra22l-d pulpcore-worker-1[25158]: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 690, in _execute Mar 11 15:08:50 cpt-infra22l-d pulpcore-worker-1[25158]: return self.func(*self.args, **self.kwargs) Mar 11 15:08:50 cpt-infra22l-d pulpcore-worker-1[25158]: File “/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py”, l ine 240, in complex_repo_migration Mar 11 15:08:50 cpt-infra22l-d pulpcore-worker-1[25158]: migrated_repo.pulp3_repository_version Mar 11 15:08:50 cpt-infra22l-d pulpcore-worker-1[25158]: File “/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py”, l ine 360, in migrate_repo_distributor Mar 11 15:08:50 cpt-infra22l-d pulpcore-worker-1[25158]: pulp2dist, repo_version) Mar 11 15:08:50 cpt-infra22l-d pulpcore-worker-1[25158]: File “/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/repos itory.py”, line 81, in migrate_to_pulp3 Mar 11 15:08:50 cpt-infra22l-d pulpcore-worker-1[25158]: publish(repo_version.pk, checksum_types=checksum_types) Mar 11 15:08:50 cpt-infra22l-d pulpcore-worker-1[25158]: File “/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py”, line 290, in publish Mar 11 15:08:50 cpt-infra22l-d pulpcore-worker-1[25158]: metadata_signing_service=metadata_signing_service Mar 11 15:08:50 cpt-infra22l-d pulpcore-worker-1[25158]: File “/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py”, line 483, in create_repomd_xml Mar 11 15:08:50 cpt-infra22l-d pulpcore-worker-1[25158]: record.fill(checksum_type) Mar 11 15:08:50 cpt-infra22l-d pulpcore-worker-1[25158]: OSError: Error while computing stat of compressed content of updateinfo.xml.zck :Cannot open a file updateinfo.xml.zck: createrepo_c wasn’t compiled with zchunk support … ~~~ Thanks