Issue #8100: Error when publishing repository Red Hat Enterprise Linux 6 Server RPMs x86_64 6Server - pulpcore-worker-3: FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/pulp/media/artifact/5b/290c0e700df6f4b4669d9e7366f6aa4b6817f96e1dcb9b81284409ae64 - RPM Support - Pulp

Actions

Send by e-mail Copy link

Issue #8100

closed

Error when publishing repository Red Hat Enterprise Linux 6 Server RPMs x86_64 6Server - pulpcore-worker-3: FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/pulp/media/artifact/5b/290c0e700df6f4b4669d9e7366f6aa4b6817f96e1dcb9b81284409ae64

Added by pieta over 3 years ago. Updated about 3 years ago.

Status:

CLOSED - WORKSFORME

Priority:

Normal

Assignee:

Sprint/Milestone:

Start date:

Due date:

Estimated time:

Severity:

3. High

Version:

Platform Release:

OS:

CentOS 7

Triaged:

Groomed:

Sprint Candidate:

Tags:

Katello

Sprint:

Quarter:

Description

Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: pulp: pulp_rpm.app.tasks.publishing:INFO: Publishing: repository=Red_Hat_Enterprise_Linux_6_Server_RPMs_x86_64_6Server-189772, version=1
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: pulp: rq.worker:ERROR: Traceback (most recent call last):
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: rv = job.perform()
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: self._result = self._execute()
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: return self.func(*self.args, **self.kwargs)
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 283, in publish
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: publication_data.populate()
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 209, in populate
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: self.repomdrecords = self.prepare_metadata_files(main_content)
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 89, in prepare_metadata_files
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: current_file = content_artifact.artifact.file.file
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/django/db/models/fields/files.py", line 43, in _get_file
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: self._file = self.storage.open(self.name, 'rb')
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/django/core/files/storage.py", line 36, in open
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: return self._open(name, mode)
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/django/core/files/storage.py", line 224, in _open
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: return File(open(self.path(name), mode))
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/pulp/media/artifact/5b/290c0e700df6f4b4669d9e7366f6aa4b6817f96e1dcb9b81284409ae640972'
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: Traceback (most recent call last):
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: rv = job.perform()
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: self._result = self._execute()
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: return self.func(*self.args, **self.kwargs)
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 283, in publish
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: publication_data.populate()
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 209, in populate
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: self.repomdrecords = self.prepare_metadata_files(main_content)
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 89, in prepare_metadata_files
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: current_file = content_artifact.artifact.file.file
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/django/db/models/fields/files.py", line 43, in _get_file
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: self._file = self.storage.open(self.name, 'rb')
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/django/core/files/storage.py", line 36, in open
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: return self._open(name, mode)
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/django/core/files/storage.py", line 224, in _open
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: return File(open(self.path(name), mode))
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/pulp/media/artifact/5b/290c0e700df6f4b4669d9e7366f6aa4b6817f96e1dcb9b81284409ae640972'
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: pulp: rq.worker:INFO: 32205@oplrepo-cpd-dmz.corpnet.pl: cb6fefb5-5347-4d56-a933-c9716438a13c
Jan 14 12:10:07 oplrepo-cpd-dmz pulpcore-worker-3: pulp: rq.worker:INFO: 32205@oplrepo-cpd-dmz.corpnet.pl: Job OK (cb6fefb5-5347-4d56-a933-c9716438a13c)
Jan 14 12:10:08 oplrepo-cpd-dmz pulpcore-api: - - [14/Jan/2021:11:10:08 +0000] "GET /pulp/api/v3/tasks/ed9ab0b6-1878-4f3e-ba36-552963eb92eb/ HTTP/1.1" 200 1957 "-" "OpenAPI-Generator/3.7.1/ruby"
Jan 14 12:10:08 oplrepo-cpd-dmz pulpcore-resource-manager: pulp: rq.worker:INFO: resource-manager: e780fd48-487e-4bde-b6ad-f46eae938da8
Jan 14 12:10:08 oplrepo-cpd-dmz pulpcore-api: - - [14/Jan/2021:11:10:08 +0000] "PATCH /pulp/api/v3/distributions/rpm/rpm/1d113bbc-48b9-4897-b3e8-ae5e18546b6a/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.7.0/ruby"
Jan 14 12:10:08 oplrepo-cpd-dmz pulpcore-api: - - [14/Jan/2021:11:10:08 +0000] "GET /pulp/api/v3/tasks/dd30cc46-38fd-4ae3-8935-6e94fc967400/ HTTP/1.1" 200 401 "-" "OpenAPI-Generator/3.7.1/ruby"
Jan 14 12:10:08 oplrepo-cpd-dmz pulpcore-worker-1: pulp: rq.worker:INFO: 32146@oplrepo-cpd-dmz.corpnet.pl: dd30cc46-38fd-4ae3-8935-6e94fc967400
Jan 14 12:10:08 oplrepo-cpd-dmz pulpcore-resource-manager: pulp: rq.worker:INFO: resource-manager: Job OK (e780fd48-487e-4bde-b6ad-f46eae938da8)
Jan 14 12:10:09 oplrepo-cpd-dmz pulpcore-api: - - [14/Jan/2021:11:10:09 +0000] "GET /pulp/api/v3/tasks/dd30cc46-38fd-4ae3-8935-6e94fc967400/ HTTP/1.1" 200 482 "-" "OpenAPI-Generator/3.7.1/ruby"
Jan 14 12:10:09 oplrepo-cpd-dmz pulpcore-worker-1: pulp: rq.worker:INFO: 32146@oplrepo-cpd-dmz.corpnet.pl: Job OK (dd30cc46-38fd-4ae3-8935-6e94fc967400)
Jan 14 12:10:09 oplrepo-cpd-dmz pulpcore-worker-1: pulp: rq.worker:INFO: 32146@oplrepo-cpd-dmz.corpnet.pl: f33e4dc9-4dea-4487-80c8-7f705a0c4571
Jan 14 12:10:09 oplrepo-cpd-dmz pulpcore-worker-1: pulp: rq.worker:INFO: 32146@oplrepo-cpd-dmz.corpnet.pl: Job OK (f33e4dc9-4dea-4487-80c8-7f705a0c4571)
Jan 14 12:10:09 oplrepo-cpd-dmz pulpcore-api: - - [14/Jan/2021:11:10:09 +0000] "GET /pulp/api/v3/tasks/dd30cc46-38fd-4ae3-8935-6e94fc967400/ HTTP/1.1" 200 509 "-" "OpenAPI-Generator/3.7.1/ruby"
Jan 14 12:10:09 oplrepo-cpd-dmz pulpcore-resource-manager: pulp: rq.worker:INFO: resource-manager: cc602b56-95da-449d-ab86-f9d16580a5d4
Jan 14 12:10:09 oplrepo-cpd-dmz pulpcore-api: - - [14/Jan/2021:11:10:09 +0000] "PATCH /pulp/api/v3/distributions/rpm/rpm/1d113bbc-48b9-4897-b3e8-ae5e18546b6a/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.7.0/ruby"
Jan 14 12:10:09 oplrepo-cpd-dmz pulpcore-worker-4: pulp: rq.worker:INFO: 32227@oplrepo-cpd-dmz.corpnet.pl: 5733c987-6ebb-4b23-a759-68efa2ced7e4
Jan 14 12:10:09 oplrepo-cpd-dmz pulpcore-resource-manager: pulp: rq.worker:INFO: resource-manager: Job OK (cc602b56-95da-449d-ab86-f9d16580a5d4)
Jan 14 12:10:10 oplrepo-cpd-dmz pulpcore-worker-4: pulp: rq.worker:INFO: 32227@oplrepo-cpd-dmz.corpnet.pl: Job OK (5733c987-6ebb-4b23-a759-68efa2ced7e4)
Jan 14 12:10:10 oplrepo-cpd-dmz pulpcore-worker-4: pulp: rq.worker:INFO: 32227@oplrepo-cpd-dmz.corpnet.pl: b2df0b58-9ce4-4a6f-ab02-8c4cf0732ebe
Jan 14 12:10:10 oplrepo-cpd-dmz pulpcore-worker-4: pulp: rq.worker:INFO: 32227@oplrepo-cpd-dmz.corpnet.pl: Job OK (b2df0b58-9ce4-4a6f-ab02-8c4cf0732ebe)
Jan 14 12:10:40 oplrepo-cpd-dmz pulpcore-api: [2021-01-14 12:10:40 +0100] [1995] [CRITICAL] WORKER TIMEOUT (pid:2058)
Jan 14 12:10:42 oplrepo-cpd-dmz pulpcore-api: [2021-01-14 12:10:42 +0100] [12482] [INFO] Booting worker with pid: 12482
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-resource-manager: pulp: rq.worker:INFO: resource-manager: 31df0561-106d-4edc-ba58-eaacf58bf71e
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-api: - - [14/Jan/2021:11:10:47 +0000] "POST /pulp/api/v3/publications/rpm/rpm/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.7.0/ruby"
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: pulp: rq.worker:INFO: 32227@oplrepo-cpd-dmz.corpnet.pl: 004d6d1e-02a3-4805-9ba1-4379be2a620d
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-resource-manager: pulp: rq.worker:INFO: resource-manager: Job OK (31df0561-106d-4edc-ba58-eaacf58bf71e)
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-api: - - [14/Jan/2021:11:10:47 +0000] "GET /pulp/api/v3/tasks/004d6d1e-02a3-4805-9ba1-4379be2a620d/ HTTP/1.1" 200 449 "-" "OpenAPI-Generator/3.7.1/ruby"
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: pulp: pulp_rpm.app.tasks.publishing:INFO: Publishing: repository=Red_Hat_Enterprise_Linux_6_Server_RPMs_x86_64_6Server-189772, version=1
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: pulp: rq.worker:ERROR: Traceback (most recent call last):
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: rv = job.perform()
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: self._result = self._execute()
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: return self.func(*self.args, **self.kwargs)
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 283, in publish
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: publication_data.populate()
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 209, in populate
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: self.repomdrecords = self.prepare_metadata_files(main_content)
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 89, in prepare_metadata_files
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: current_file = content_artifact.artifact.file.file
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/django/db/models/fields/files.py", line 43, in _get_file
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: self._file = self.storage.open(self.name, 'rb')
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/django/core/files/storage.py", line 36, in open
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: return self._open(name, mode)
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/django/core/files/storage.py", line 224, in _open
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: return File(open(self.path(name), mode))
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/pulp/media/artifact/5b/290c0e700df6f4b4669d9e7366f6aa4b6817f96e1dcb9b81284409ae640972'
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: Traceback (most recent call last):
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: rv = job.perform()
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: self._result = self._execute()
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: return self.func(*self.args, **self.kwargs)
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 283, in publish
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: publication_data.populate()
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 209, in populate
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: self.repomdrecords = self.prepare_metadata_files(main_content)
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 89, in prepare_metadata_files
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: current_file = content_artifact.artifact.file.file
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/django/db/models/fields/files.py", line 43, in _get_file
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: self._file = self.storage.open(self.name, 'rb')
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/django/core/files/storage.py", line 36, in open
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: return self._open(name, mode)
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/django/core/files/storage.py", line 224, in _open
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: return File(open(self.path(name), mode))
Jan 14 12:10:47 oplrepo-cpd-dmz pulpcore-worker-4: FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/pulp/media/artifact/5b/290c0e700df6f4b4669d9e7366f6aa4b6817f96e1dcb9b81284409ae640972'

Actions

Copy link

Updated by ttereshc over 3 years ago

Project changed from Pulp to RPM Support
Description updated (diff)

Pre-d the descritpion

Actions

Copy link

Updated by ttereshc over 3 years ago

Which version of Pulp are you running? Why is it marked as Katello item? Are you running Katello? Which version?

Actions

Copy link

Updated by dalley about 3 years ago

This could possibly be an instance of https://pulp.plan.io/issues/7676

@pieta please do tell us what version you're running, but also, you can try to use the "repository version repair" steps described here to see if that fixes your problems. The global repair feature was added recently so if you're running a very new version you might have it available - in which case, use that instead.

https://docs.pulpproject.org/pulpcore/workflows/repairing-pulp.html

Actions

Copy link

Updated by pieta about 3 years ago

I'm sorry I didn't answer for so long.

I use this version:

[root@oplrepo-cpd-dmz ~]# rpm -q -a | grep katello katello-default-ca-1.0-1.noarch katello-agent-3.5.4-1.el7.noarch katello-common-3.18.1-1.el7.noarch katello-server-ca-1.0-4.noarch katello-host-tools-3.5.4-1.el7.noarch katello-repos-3.18.1-1.el7.noarch katello-ca-consumer-oplrepo-cpd-dmz.corpnet.pl-1.0-2.noarch katello-client-bootstrap-1.7.5-1.el7.noarch katello-debug-3.18.1-1.el7.noarch foreman-installer-katello-2.3.2-1.el7.noarch tfm-rubygem-hammer_cli_katello-0.24.2-1.el7.noarch katello-certs-tools-2.7.3-1.el7.noarch katello-3.18.1-1.el7.noarch tfm-rubygem-katello-3.18.1-1.el7.noarch katello-host-tools-fact-plugin-3.5.4-1.el7.noarch pulp-katello-1.0.3-1.el7.noarch [root@oplrepo-cpd-dmz ~]# rpm -q -a | grep pulpcore python3-pulpcore-3.7.3-1.el7.noarch tfm-rubygem-pulpcore_client-3.7.1-1.el7.noarch

Actions

Copy link

Updated by dalley about 3 years ago

No problem. Early versions of Pulp 3.7 were impacted by that bug I linked earlier, and although we did backport the fix, it's possible that you experienced some data loss before the fix landed.

The repair functionality I mentioned should most likely be able to resolve the issue however. Let us know if it works for you. Unfortunately the global repair isn't available in 3.7 so you'd need to repair a specific repository version that contains the content unit triggering your error.

Actions

Copy link