Project

Profile

Help

Issue #8100

Error when publishing repository Red Hat Enterprise Linux 6 Server RPMs x86_64 6Server - pulpcore-worker-3: FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/pulp/media/artifact/5b/290c0e700df6f4b4669d9e7366f6aa4b6817f96e1dcb9b81284409ae64

Added by pieta 9 months ago. Updated 9 months ago.

Status:
CLOSED - WORKSFORME
Priority:
Normal
Assignee:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Platform Release:
OS:
CentOS 7
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Katello
Sprint:
Quarter:

Description

Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: pulp: pulp_rpm.app.tasks.publishing:INFO: Publishing: repository=Red_Hat_Enterprise_Linux_6_Server_RPMs_x86_64_6Server-189772, version=1
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: pulp: rq.worker:ERROR: Traceback (most recent call last):
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: rv = job.perform()
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: self._result = self._execute()
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: return self.func(*self.args, **self.kwargs)
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 283, in publish
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: publication_data.populate()
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 209, in populate
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: self.repomdrecords = self.prepare_metadata_files(main_content)
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 89, in prepare_metadata_files
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: current_file = content_artifact.artifact.file.file
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/django/db/models/fields/files.py", line 43, in _get_file
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: self._file = self.storage.open(self.name, 'rb')
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/django/core/files/storage.py", line 36, in open
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: return self._open(name, mode)
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/django/core/files/storage.py", line 224, in _open
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: return File(open(self.path(name), mode))
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/pulp/media/artifact/5b/290c0e700df6f4b4669d9e7366f6aa4b6817f96e1dcb9b81284409ae640972'
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: Traceback (most recent call last):
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: rv = job.perform()
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: self._result = self._execute()
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: return self.func(*self.args, **self.kwargs)
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 283, in publish
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: publication_data.populate()
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 209, in populate
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: self.repomdrecords = self.prepare_metadata_files(main_content)
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 89, in prepare_metadata_files
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: current_file = content_artifact.artifact.file.file
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/django/db/models/fields/files.py", line 43, in _get_file
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: self._file = self.storage.open(self.name, 'rb')
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/django/core/files/storage.py", line 36, in open
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: return self._open(name, mode)
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/django/core/files/storage.py", line 224, in _open
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: return File(open(self.path(name), mode))
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/pulp/media/artifact/5b/290c0e700df6f4b4669d9e7366f6aa4b6817f96e1dcb9b81284409ae640972'
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: pulp: rq.worker:INFO: 32205@oplrepo-cpd-dmz.corpnet.pl: cb6fefb5-5347-4d56-a933-c9716438a13c
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: pulp: rq.worker:INFO: 32205@oplrepo-cpd-dmz.corpnet.pl: Job OK (cb6fefb5-5347-4d56-a933-c9716438a13c)
Jan 14 12:10:08 oplrepo-cpd-dmz pulpcore-api: - - [14/Jan/2021:11:10:08 +0000] "GET /pulp/api/v3/tasks/ed9ab0b6-1878-4f3e-ba36-552963eb92eb/ HTTP/1.1" 200 1957 "-" "OpenAPI-Generator/3.7.1/ruby"
Jan 14 12:10:08 oplrepo-cpd-dmz pulpcore-resource-manager: pulp: rq.worker:INFO: resource-manager: e780fd48-487e-4bde-b6ad-f46eae938da8
Jan 14 12:10:08 oplrepo-cpd-dmz pulpcore-api: - - [14/Jan/2021:11:10:08 +0000] "PATCH /pulp/api/v3/distributions/rpm/rpm/1d113bbc-48b9-4897-b3e8-ae5e18546b6a/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.7.0/ruby"
Jan 14 12:10:08 oplrepo-cpd-dmz pulpcore-api: - - [14/Jan/2021:11:10:08 +0000] "GET /pulp/api/v3/tasks/dd30cc46-38fd-4ae3-8935-6e94fc967400/ HTTP/1.1" 200 401 "-" "OpenAPI-Generator/3.7.1/ruby"
Jan 14 12:10:08 oplrepo-cpd-dmz pulpcore-worker-1: pulp: rq.worker:INFO: 32146@oplrepo-cpd-dmz.corpnet.pl: dd30cc46-38fd-4ae3-8935-6e94fc967400
Jan 14 12:10:08 oplrepo-cpd-dmz pulpcore-resource-manager: pulp: rq.worker:INFO: resource-manager: Job OK (e780fd48-487e-4bde-b6ad-f46eae938da8)
Jan 14 12:10:09 oplrepo-cpd-dmz pulpcore-api: - - [14/Jan/2021:11:10:09 +0000] "GET /pulp/api/v3/tasks/dd30cc46-38fd-4ae3-8935-6e94fc967400/ HTTP/1.1" 200 482 "-" "OpenAPI-Generator/3.7.1/ruby"
Jan 14 12:10:09 oplrepo-cpd-dmz pulpcore-worker-1: pulp: rq.worker:INFO: 32146@oplrepo-cpd-dmz.corpnet.pl: Job OK (dd30cc46-38fd-4ae3-8935-6e94fc967400)
Jan 14 12:10:09 oplrepo-cpd-dmz pulpcore-worker-1: pulp: rq.worker:INFO: 32146@oplrepo-cpd-dmz.corpnet.pl: f33e4dc9-4dea-4487-80c8-7f705a0c4571
Jan 14 12:10:09 oplrepo-cpd-dmz pulpcore-worker-1: pulp: rq.worker:INFO: 32146@oplrepo-cpd-dmz.corpnet.pl: Job OK (f33e4dc9-4dea-4487-80c8-7f705a0c4571)
Jan 14 12:10:09 oplrepo-cpd-dmz pulpcore-api: - - [14/Jan/2021:11:10:09 +0000] "GET /pulp/api/v3/tasks/dd30cc46-38fd-4ae3-8935-6e94fc967400/ HTTP/1.1" 200 509 "-" "OpenAPI-Generator/3.7.1/ruby"
Jan 14 12:10:09 oplrepo-cpd-dmz pulpcore-resource-manager: pulp: rq.worker:INFO: resource-manager: cc602b56-95da-449d-ab86-f9d16580a5d4
Jan 14 12:10:09 oplrepo-cpd-dmz pulpcore-api: - - [14/Jan/2021:11:10:09 +0000] "PATCH /pulp/api/v3/distributions/rpm/rpm/1d113bbc-48b9-4897-b3e8-ae5e18546b6a/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.7.0/ruby"
Jan 14 12:10:09 oplrepo-cpd-dmz pulpcore-worker-4: pulp: rq.worker:INFO: 32227@oplrepo-cpd-dmz.corpnet.pl: 5733c987-6ebb-4b23-a759-68efa2ced7e4
Jan 14 12:10:09 oplrepo-cpd-dmz pulpcore-resource-manager: pulp: rq.worker:INFO: resource-manager: Job OK (cc602b56-95da-449d-ab86-f9d16580a5d4)
Jan 14 12:10:10 oplrepo-cpd-dmz pulpcore-worker-4: pulp: rq.worker:INFO: 32227@oplrepo-cpd-dmz.corpnet.pl: Job OK (5733c987-6ebb-4b23-a759-68efa2ced7e4)
Jan 14 12:10:10 oplrepo-cpd-dmz pulpcore-worker-4: pulp: rq.worker:INFO: 32227@oplrepo-cpd-dmz.corpnet.pl: b2df0b58-9ce4-4a6f-ab02-8c4cf0732ebe
Jan 14 12:10:10 oplrepo-cpd-dmz pulpcore-worker-4: pulp: rq.worker:INFO: 32227@oplrepo-cpd-dmz.corpnet.pl: Job OK (b2df0b58-9ce4-4a6f-ab02-8c4cf0732ebe)
Jan 14 12:10:40 oplrepo-cpd-dmz pulpcore-api: [2021-01-14 12:10:40 +0100] [1995] [CRITICAL] WORKER TIMEOUT (pid:2058)
Jan 14 12:10:42 oplrepo-cpd-dmz pulpcore-api: [2021-01-14 12:10:42 +0100] [12482] [INFO] Booting worker with pid: 12482
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-resource-manager: pulp: rq.worker:INFO: resource-manager: 31df0561-106d-4edc-ba58-eaacf58bf71e
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-api: - - [14/Jan/2021:11:10:47 +0000] "POST /pulp/api/v3/publications/rpm/rpm/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.7.0/ruby"
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: pulp: rq.worker:INFO: 32227@oplrepo-cpd-dmz.corpnet.pl: 004d6d1e-02a3-4805-9ba1-4379be2a620d
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-resource-manager: pulp: rq.worker:INFO: resource-manager: Job OK (31df0561-106d-4edc-ba58-eaacf58bf71e)
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-api: - - [14/Jan/2021:11:10:47 +0000] "GET /pulp/api/v3/tasks/004d6d1e-02a3-4805-9ba1-4379be2a620d/ HTTP/1.1" 200 449 "-" "OpenAPI-Generator/3.7.1/ruby"
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: pulp: pulp_rpm.app.tasks.publishing:INFO: Publishing: repository=Red_Hat_Enterprise_Linux_6_Server_RPMs_x86_64_6Server-189772, version=1
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: pulp: rq.worker:ERROR: Traceback (most recent call last):
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: rv = job.perform()
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: self._result = self._execute()
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: return self.func(*self.args, **self.kwargs)
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 283, in publish
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: publication_data.populate()
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 209, in populate
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: self.repomdrecords = self.prepare_metadata_files(main_content)
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 89, in prepare_metadata_files
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: current_file = content_artifact.artifact.file.file
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/django/db/models/fields/files.py", line 43, in _get_file
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: self._file = self.storage.open(self.name, 'rb')
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/django/core/files/storage.py", line 36, in open
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: return self._open(name, mode)
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/django/core/files/storage.py", line 224, in _open
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: return File(open(self.path(name), mode))
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/pulp/media/artifact/5b/290c0e700df6f4b4669d9e7366f6aa4b6817f96e1dcb9b81284409ae640972'
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: Traceback (most recent call last):
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: rv = job.perform()
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: self._result = self._execute()
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: return self.func(*self.args, **self.kwargs)
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 283, in publish
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: publication_data.populate()
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 209, in populate
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: self.repomdrecords = self.prepare_metadata_files(main_content)
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 89, in prepare_metadata_files
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: current_file = content_artifact.artifact.file.file
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/django/db/models/fields/files.py", line 43, in _get_file
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: self._file = self.storage.open(self.name, 'rb')
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/django/core/files/storage.py", line 36, in open
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: return self._open(name, mode)
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/django/core/files/storage.py", line 224, in _open
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: return File(open(self.path(name), mode))
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/pulp/media/artifact/5b/290c0e700df6f4b4669d9e7366f6aa4b6817f96e1dcb9b81284409ae640972'

History

#1 Updated by ttereshc 9 months ago

  • Project changed from Pulp to RPM Support
  • Description updated (diff)

Pre-d the descritpion

#2 Updated by ttereshc 9 months ago

Which version of Pulp are you running? Why is it marked as Katello item? Are you running Katello? Which version?

#3 Updated by dalley 9 months ago

This could possibly be an instance of https://pulp.plan.io/issues/7676

@pieta please do tell us what version you're running, but also, you can try to use the "repository version repair" steps described here to see if that fixes your problems. The global repair feature was added recently so if you're running a very new version you might have it available - in which case, use that instead.

https://docs.pulpproject.org/pulpcore/workflows/repairing-pulp.html

#4 Updated by pieta 9 months ago

I'm sorry I didn't answer for so long.

I use this version:

[root@oplrepo-cpd-dmz ~]# rpm -q -a | grep katello katello-default-ca-1.0-1.noarch katello-agent-3.5.4-1.el7.noarch katello-common-3.18.1-1.el7.noarch katello-server-ca-1.0-4.noarch katello-host-tools-3.5.4-1.el7.noarch katello-repos-3.18.1-1.el7.noarch katello-ca-consumer-oplrepo-cpd-dmz.corpnet.pl-1.0-2.noarch katello-client-bootstrap-1.7.5-1.el7.noarch katello-debug-3.18.1-1.el7.noarch foreman-installer-katello-2.3.2-1.el7.noarch tfm-rubygem-hammer_cli_katello-0.24.2-1.el7.noarch katello-certs-tools-2.7.3-1.el7.noarch katello-3.18.1-1.el7.noarch tfm-rubygem-katello-3.18.1-1.el7.noarch katello-host-tools-fact-plugin-3.5.4-1.el7.noarch pulp-katello-1.0.3-1.el7.noarch [root@oplrepo-cpd-dmz ~]# rpm -q -a | grep pulpcore python3-pulpcore-3.7.3-1.el7.noarch tfm-rubygem-pulpcore_client-3.7.1-1.el7.noarch

#5 Updated by dalley 9 months ago

No problem. Early versions of Pulp 3.7 were impacted by that bug I linked earlier, and although we did backport the fix, it's possible that you experienced some data loss before the fix landed.

The repair functionality I mentioned should most likely be able to resolve the issue however. Let us know if it works for you. Unfortunately the global repair isn't available in 3.7 so you'd need to repair a specific repository version that contains the content unit triggering your error.

#6 Updated by dalley 9 months ago

@pieta any luck?

#7 Updated by dalley 9 months ago

  • Status changed from NEW to CLOSED - WORKSFORME

@pieta, If you continue encountering this problem and it is not resolved by executing a repair on the affected repositories, please reopen.

Please register to edit this issue

Also available in: Atom PDF