Project

Profile

Help

Issue #9146

closed

sync of file:// repos tries to move files from /var/lib/pulp/sync_imports/

Added by jsherril@redhat.com over 2 years ago. Updated over 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Urgent
Category:
-
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Katello
Sprint:
Sprint 102
Quarter:

Description

When syncing file:// repos it appears that pulp is trying to move files from that directory. In our case we copy files that are world readable, but not writable by the pulp user and see a permission error:

Jul 26 18:27:11 devel2 pulpcore-worker-1: pulp [994770242e2842ecade44f158e7de8d0]: pulpcore.tasking.pulpcore_worker:INFO: Task 4c40418c-8a76-4a83-a6ee-4d2b5326c9d4 failed ([Errno 13] Pe
rmission denied: '/var/lib/pulp/sync_imports/test_repos/zoo/kangaroo-0.2-1.noarch.rpm')
Jul 26 18:27:11 devel2 pulpcore-worker-1: pulp [994770242e2842ecade44f158e7de8d0]: pulpcore.tasking.pulpcore_worker:INFO:   File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpc
ore_worker.py", line 272, in _perform_task
Jul 26 18:27:11 devel2 pulpcore-worker-1: result = func(*args, **kwargs)
Jul 26 18:27:11 devel2 pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 427, in synchronize
Jul 26 18:27:11 devel2 pulpcore-worker-1: version = dv.create()
Jul 26 18:27:11 devel2 pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 151, in create
Jul 26 18:27:11 devel2 pulpcore-worker-1: loop.run_until_complete(pipeline)
Jul 26 18:27:11 devel2 pulpcore-worker-1: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
Jul 26 18:27:11 devel2 pulpcore-worker-1: return future.result()
Jul 26 18:27:11 devel2 pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Jul 26 18:27:11 devel2 pulpcore-worker-1: await asyncio.gather(*futures)
Jul 26 18:27:11 devel2 pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
Jul 26 18:27:11 devel2 pulpcore-worker-1: await self.run()
Jul 26 18:27:11 devel2 pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 248, in run
Jul 26 18:27:11 devel2 pulpcore-worker-1: d_artifact.artifact for d_artifact in da_to_save
Jul 26 18:27:11 devel2 pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulpcore/app/models/content.py", line 82, in bulk_get_or_create
Jul 26 18:27:11 devel2 pulpcore-worker-1: return super().bulk_create(objs, batch_size=batch_size)
Jul 26 18:27:11 devel2 pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
Jul 26 18:27:11 devel2 pulpcore-worker-1: return getattr(self.get_queryset(), name)(*args, **kwargs)
Jul 26 18:27:11 devel2 pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 468, in bulk_create
Jul 26 18:27:11 devel2 pulpcore-worker-1: self._batched_insert(objs_with_pk, fields, batch_size, ignore_conflicts=ignore_conflicts)
Jul 26 18:27:11 devel2 pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 1204, in _batched_insert
Jul 26 18:27:11 devel2 pulpcore-worker-1: ignore_conflicts=ignore_conflicts,
Jul 26 18:27:11 devel2 pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 1186, in _insert
Jul 26 18:27:11 devel2 pulpcore-worker-1: return query.get_compiler(using=using).execute_sql(return_id)
Jul 26 18:27:11 devel2 pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1376, in execute_sql
Jul 26 18:27:11 devel2 pulpcore-worker-1: for sql, params in self.as_sql():
Jul 26 18:27:11 devel2 pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/django_readonly_field/compiler.py", line 31, in as_sql
Jul 26 18:27:11 devel2 pulpcore-worker-1: return super(ReadonlySQLCompilerMixin, self).as_sql()
Jul 26 18:27:11 devel2 pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1320, in as_sql
Jul 26 18:27:11 devel2 pulpcore-worker-1: for obj in self.query.objs
Jul 26 18:27:11 devel2 pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1320, in <listcomp>
Jul 26 18:27:11 devel2 pulpcore-worker-1: for obj in self.query.objs
Jul 26 18:27:11 devel2 pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1319, in <listcomp>
Jul 26 18:27:11 devel2 pulpcore-worker-1: [self.prepare_value(field, self.pre_save_val(field, obj)) for field in fields]
Jul 26 18:27:11 devel2 pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1270, in pre_save_val
Jul 26 18:27:11 devel2 pulpcore-worker-1: return field.pre_save(obj, add=True)
Jul 26 18:27:11 devel2 pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulpcore/app/models/fields.py", line 68, in pre_save
Jul 26 18:27:11 devel2 pulpcore-worker-1: return super().pre_save(model_instance, add)
Jul 26 18:27:11 devel2 pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/django/db/models/fields/files.py", line 289, in pre_save
Jul 26 18:27:11 devel2 pulpcore-worker-1: file.save(file.name, file.file, save=False)
Jul 26 18:27:11 devel2 pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/django/db/models/fields/files.py", line 88, in save
Jul 26 18:27:11 devel2 pulpcore-worker-1: self.name = self.storage.save(name, content, max_length=self.field.max_length)
Jul 26 18:27:11 devel2 pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/django/core/files/storage.py", line 54, in save
Jul 26 18:27:11 devel2 pulpcore-worker-1: return self._save(name, content)
Jul 26 18:27:11 devel2 pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/pulpcore/app/models/storage.py", line 68, in _save
Jul 26 18:27:11 devel2 pulpcore-worker-1: file_move_safe(content.temporary_file_path(), full_path)
Jul 26 18:27:11 devel2 pulpcore-worker-1: File "/usr/lib/python3.6/site-packages/django/core/files/move.py", line 80, in file_move_safe
Jul 26 18:27:11 devel2 pulpcore-worker-1: os.remove(old_file_name)

These repos are in /var/lib/pulp/sync_imports/ (and settings.py is configured properly).


Related issues

Copied to Pulp - Backport #9202: Backport #9146 "sync of file:// repos tries to move files from /var/lib/pulp/sync_imports/" to 3.14.zCLOSED - CURRENTRELEASE

Actions
Actions #1

Updated by jsherril@redhat.com over 2 years ago

Also, this was seen in pulp 3.14.3

Actions #2

Updated by dalley over 2 years ago

This is a bit of a weird use case that nobody (myself included) really forsaw when looking at https://github.com/pulp/pulpcore/commit/b4a373064231c2df605f37c2dd35b222520b0014

But the fix should be just as simple as the original PR.

Actions #3

Updated by dkliban@redhat.com over 2 years ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 101
Actions #5

Updated by ipanova@redhat.com over 2 years ago

  • Sprint changed from Sprint 101 to Sprint 102
Actions #6

Updated by dkliban@redhat.com over 2 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dkliban@redhat.com
Actions #7

Updated by pulpbot over 2 years ago

  • Status changed from ASSIGNED to POST
Actions #8

Updated by dalley over 2 years ago

  • Sprint/Milestone set to 3.15.0
Actions #9

Updated by dalley over 2 years ago

  • Copied to Backport #9202: Backport #9146 "sync of file:// repos tries to move files from /var/lib/pulp/sync_imports/" to 3.14.z added

Added by dkliban@redhat.com over 2 years ago

Revision c828ca59 | View on GitHub

Move files to artifact storage from WORKING_DIRECTORY.

Copy files from all other sources. Since users are able to change the WORKING_DIRECTORY settings, this patch also ensures that we check that the WORKING_DIRECTORY and MEDIA_ROOT are on the same device. Django's FILE_UPLOAD_TEMP_DIR is now set to the value of WORKING_DIRECTORY.

fixes: #9146 https://pulp.plan.io/issues/9146/

Actions #10

Updated by dkliban@redhat.com over 2 years ago

  • Status changed from POST to MODIFIED
Actions #11

Updated by pulpbot over 2 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF