Issue #1312
closed
Rpm repo sync fails for a centos feed
Description
Rpm repo sync fails during distribution sync
[root@qe-blade-12 ~]# pulp-admin rpm repo create --repo-id centos7 --feed http://mirror.mojohost.com/centos/7/os/x86_64/
Successfully created repository [centos7]
[root@qe-blade-12 ~]# pulp-admin rpm repo sync run --repo-id centos7
+----------------------------------------------------------------------+
Synchronizing Repository [centos7]
+----------------------------------------------------------------------+
This command may be exited via ctrl+c without affecting the request.
Downloading metadata...
[|]
... completed
Downloading repository content...
[==================================================] 100%
RPMs: 8652/8652 items
Delta RPMs: 0/0 items
... completed
Downloading distribution files...
[==================================================] 100%
Distributions: 6/6 items
Errors encountered during distribution sync:
Task Failed
Importer indicated a failed response
Oct 12 15:55:15 qe-blade-12.idmqe.lab.eng.bos.redhat.com pulp[2475]: pulp.server.async.tasks:INFO: Task failed : [56bb5291-2066-4e96-89ca-8ef2d3bfbcf1]
Oct 12 15:55:15 qe-blade-12.idmqe.lab.eng.bos.redhat.com pulp[2359]: celery.worker.job:ERROR: (2359-15904) Task pulp.server.managers.repo.sync.sync[56bb5291-2066-4e96-89ca-8ef2d3bfbcf1] raised unexpected: PulpExecutionException('Importer indicated a failed response',)
Oct 12 15:55:15 qe-blade-12.idmqe.lab.eng.bos.redhat.com pulp[2359]: celery.worker.job:ERROR: (2359-15904) Traceback (most recent call last):
Oct 12 15:55:15 qe-blade-12.idmqe.lab.eng.bos.redhat.com pulp[2359]: celery.worker.job:ERROR: (2359-15904) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Oct 12 15:55:15 qe-blade-12.idmqe.lab.eng.bos.redhat.com pulp[2359]: celery.worker.job:ERROR: (2359-15904) R = retval = fun(*args, **kwargs)
Oct 12 15:55:15 qe-blade-12.idmqe.lab.eng.bos.redhat.com pulp[2359]: celery.worker.job:ERROR: (2359-15904) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 392, in __call__
Oct 12 15:55:15 qe-blade-12.idmqe.lab.eng.bos.redhat.com pulp[2359]: celery.worker.job:ERROR: (2359-15904) return super(Task, self).__call__(*args, **kwargs)
Oct 12 15:55:15 qe-blade-12.idmqe.lab.eng.bos.redhat.com pulp[2359]: celery.worker.job:ERROR: (2359-15904) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
Oct 12 15:55:15 qe-blade-12.idmqe.lab.eng.bos.redhat.com pulp[2359]: celery.worker.job:ERROR: (2359-15904) return self.run(*args, **kwargs)
Oct 12 15:55:15 qe-blade-12.idmqe.lab.eng.bos.redhat.com pulp[2359]: celery.worker.job:ERROR: (2359-15904) File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/sync.py", line 100, in sync
Oct 12 15:55:15 qe-blade-12.idmqe.lab.eng.bos.redhat.com pulp[2359]: celery.worker.job:ERROR: (2359-15904) raise PulpExecutionException(_('Importer indicated a failed response'))
Oct 12 15:55:15 qe-blade-12.idmqe.lab.eng.bos.redhat.com pulp[2359]: celery.worker.job:ERROR: (2359-15904) PulpExecutionException: Importer indicated a failed response
Oct 12 15:55:15 qe-blade-12.idmqe.lab.eng.bos.redhat.com pulp[2359]: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[986d332f-9ebf-4ad9-a5f8-c3608a0a7900] succeeded in 0.00721029104898s: None
Updated by dkliban@redhat.com over 8 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to dkliban@redhat.com
- Severity changed from 2. Medium to 3. High
- Platform Release set to 2.7.0
- Triaged changed from No to Yes
Updated by pthomas@redhat.com over 8 years ago
pulp-server-2.7.0-0.6.beta.el7.noarch.rpm
Updated by dkliban@redhat.com over 8 years ago
The problem lies right here [0]. Can't use copy because we are copying a directory. We can't use copytree because we don't want the SELinux security context labels copied. So the only option we have is to implement our own version of copytree() which does not call copystat() after copying files. Here [1] is an example of how to implement copytree(). We just need to exclude the try statement at the end.
[0] https://github.com/pulp/pulp_rpm/blob/2.7-testing/plugins/pulp_rpm/plugins/importers/yum/parse/treeinfo.py#L89
[1] https://docs.python.org/2.7/library/shutil.html#copytree-example
Updated by bmbouter over 8 years ago
I think you're right that we probably need to carry our own. I considered we might be able to run chcon or restorecon after the copy occurs, but we would need to have the python bindings for libselinux available, or shell out. I don't want to do either of those things.
Interestingly Python has issue 1540112 filed about a capability to resolve this problem. That was fixed with this commit in Python 3.2.
Note, in addition to removing the last try/except block which contains copystat, we also need to replace copy2() with copy(). If we don't do both then some directories will receive the correct labels but files won't.
Which module in Pulp should contain this new function?
Updated by dkliban@redhat.com over 8 years ago
I'd like to add it to here [0].
[0] https://github.com/pulp/pulp/blob/2.7-testing/common/pulp/common/util.py
Updated by bmbouter over 8 years ago
Sounds good to me.
Also maybe find other usages of copytree and adjust those use the one we are carrying also? That way these SELinux issues will be fixed in many places. Since it's an equivalent copytree behavior it should be safe to replace. It just won't have the attributes from the original files, which I think is OK.
Do others think this is ok?
Added by dkliban@redhat.com over 8 years ago
Updated by dkliban@redhat.com over 8 years ago
- Status changed from ASSIGNED to MODIFIED
- % Done changed from 0 to 100
Applied in changeset c74d3c79ef07d423db0e68e9ae6ce49be021fede.
Updated by dkliban@redhat.com over 8 years ago
- Status changed from MODIFIED to 5
Updated by pthomas@redhat.com over 8 years ago
- Status changed from 5 to 6
verified
[root@ibm-x3550m3-06 ~]# pulp-admin rpm repo create --repo-id centos --feed http://mirror.mojohost.com/centos/7/os/x86_64/
Successfully created repository [centos]
[root@ibm-x3550m3-06 ~]# pulp-admin rpm repo sync run --repo-id centos
+----------------------------------------------------------------------+
Synchronizing Repository [centos]
+----------------------------------------------------------------------+
This command may be exited via ctrl+c without affecting the request.
Downloading metadata...
[\]
... completed
Downloading repository content...
[==================================================] 100%
RPMs: 8652/8652 items
Delta RPMs: 0/0 items
... completed
Downloading distribution files...
[==================================================] 100%
Distributions: 6/6 items
... completed
Importing errata...
[-]
... completed
Importing package groups/categories...
[\]
... completed
Task Succeeded
Initializing repo metadata
[-]
... completed
Publishing Distribution files
[-]
... completed
Publishing RPMs
[==================================================] 100%
8652 of 8652 items
... completed
Publishing Delta RPMs
... skipped
Publishing Errata
[-]
... completed
Publishing Comps file
[==================================================] 100%
102 of 102 items
... completed
Publishing Metadata.
[-]
... completed
Closing repo metadata
[-]
... completed
Generating sqlite files
... skipped
Publishing files to web
[|]
... completed
Writing Listings File
[-]
... completed
Task Succeeded
[root@ibm-x3550m3-06 ~]#
Updated by amacdona@redhat.com over 8 years ago
- Status changed from 6 to CLOSED - CURRENTRELEASE
.
Switches copy to copytree
https://pulp.plan.io/issues/1312 closes #1312