Project

Profile

Help

Issue #1312

Rpm repo sync fails for a centos feed

Added by pthomas@redhat.com about 6 years ago. Updated over 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Platform Release:
2.7.0
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

Rpm repo sync fails during distribution sync


[root@qe-blade-12 ~]#  pulp-admin rpm repo create --repo-id centos7 --feed http://mirror.mojohost.com/centos/7/os/x86_64/
Successfully created repository [centos7]

[root@qe-blade-12 ~]# pulp-admin rpm  repo  sync run  --repo-id centos7
+----------------------------------------------------------------------+
                   Synchronizing Repository [centos7]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[|]
... completed

Downloading repository content...
[==================================================] 100%
RPMs:       8652/8652 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 6/6 items

Errors encountered during distribution sync:

Task Failed

Importer indicated a failed response

Oct 12 15:55:15 qe-blade-12.idmqe.lab.eng.bos.redhat.com pulp[2475]: pulp.server.async.tasks:INFO: Task failed : [56bb5291-2066-4e96-89ca-8ef2d3bfbcf1]
Oct 12 15:55:15 qe-blade-12.idmqe.lab.eng.bos.redhat.com pulp[2359]: celery.worker.job:ERROR: (2359-15904) Task pulp.server.managers.repo.sync.sync[56bb5291-2066-4e96-89ca-8ef2d3bfbcf1] raised unexpected: PulpExecutionException('Importer indicated a failed response',)
Oct 12 15:55:15 qe-blade-12.idmqe.lab.eng.bos.redhat.com pulp[2359]: celery.worker.job:ERROR: (2359-15904) Traceback (most recent call last):
Oct 12 15:55:15 qe-blade-12.idmqe.lab.eng.bos.redhat.com pulp[2359]: celery.worker.job:ERROR: (2359-15904)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Oct 12 15:55:15 qe-blade-12.idmqe.lab.eng.bos.redhat.com pulp[2359]: celery.worker.job:ERROR: (2359-15904)     R = retval = fun(*args, **kwargs)
Oct 12 15:55:15 qe-blade-12.idmqe.lab.eng.bos.redhat.com pulp[2359]: celery.worker.job:ERROR: (2359-15904)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 392, in __call__
Oct 12 15:55:15 qe-blade-12.idmqe.lab.eng.bos.redhat.com pulp[2359]: celery.worker.job:ERROR: (2359-15904)     return super(Task, self).__call__(*args, **kwargs)
Oct 12 15:55:15 qe-blade-12.idmqe.lab.eng.bos.redhat.com pulp[2359]: celery.worker.job:ERROR: (2359-15904)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
Oct 12 15:55:15 qe-blade-12.idmqe.lab.eng.bos.redhat.com pulp[2359]: celery.worker.job:ERROR: (2359-15904)     return self.run(*args, **kwargs)
Oct 12 15:55:15 qe-blade-12.idmqe.lab.eng.bos.redhat.com pulp[2359]: celery.worker.job:ERROR: (2359-15904)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/sync.py", line 100, in sync
Oct 12 15:55:15 qe-blade-12.idmqe.lab.eng.bos.redhat.com pulp[2359]: celery.worker.job:ERROR: (2359-15904)     raise PulpExecutionException(_('Importer indicated a failed response'))
Oct 12 15:55:15 qe-blade-12.idmqe.lab.eng.bos.redhat.com pulp[2359]: celery.worker.job:ERROR: (2359-15904) PulpExecutionException: Importer indicated a failed response
Oct 12 15:55:15 qe-blade-12.idmqe.lab.eng.bos.redhat.com pulp[2359]: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[986d332f-9ebf-4ad9-a5f8-c3608a0a7900] succeeded in 0.00721029104898s: None

Associated revisions

History

#1 Updated by dkliban@redhat.com about 6 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dkliban@redhat.com
  • Severity changed from 2. Medium to 3. High
  • Platform Release set to 2.7.0
  • Triaged changed from No to Yes

#2 Updated by pthomas@redhat.com about 6 years ago

pulp-server-2.7.0-0.6.beta.el7.noarch.rpm

#3 Updated by dkliban@redhat.com about 6 years ago

The problem lies right here [0]. Can't use copy because we are copying a directory. We can't use copytree because we don't want the SELinux security context labels copied. So the only option we have is to implement our own version of copytree() which does not call copystat() after copying files. Here [1] is an example of how to implement copytree(). We just need to exclude the try statement at the end.

[0] https://github.com/pulp/pulp_rpm/blob/2.7-testing/plugins/pulp_rpm/plugins/importers/yum/parse/treeinfo.py#L89
[1] https://docs.python.org/2.7/library/shutil.html#copytree-example

#4 Updated by bmbouter about 6 years ago

I think you're right that we probably need to carry our own. I considered we might be able to run chcon or restorecon after the copy occurs, but we would need to have the python bindings for libselinux available, or shell out. I don't want to do either of those things.

Interestingly Python has issue 1540112 filed about a capability to resolve this problem. That was fixed with this commit in Python 3.2.

Note, in addition to removing the last try/except block which contains copystat, we also need to replace copy2() with copy(). If we don't do both then some directories will receive the correct labels but files won't.

Which module in Pulp should contain this new function?

#6 Updated by bmbouter about 6 years ago

Sounds good to me.

Also maybe find other usages of copytree and adjust those use the one we are carrying also? That way these SELinux issues will be fixed in many places. Since it's an equivalent copytree behavior it should be safe to replace. It just won't have the attributes from the original files, which I think is OK.

Do others think this is ok?

#7 Updated by mhrivnak almost 6 years ago

Seems reasonable to me.

#8 Updated by dkliban@redhat.com almost 6 years ago

  • Status changed from ASSIGNED to MODIFIED
  • % Done changed from 0 to 100

#9 Updated by dkliban@redhat.com almost 6 years ago

  • Status changed from MODIFIED to 5

#10 Updated by pthomas@redhat.com almost 6 years ago

  • Status changed from 5 to 6

verified

[root@ibm-x3550m3-06 ~]# pulp-admin rpm repo create --repo-id centos --feed http://mirror.mojohost.com/centos/7/os/x86_64/
Successfully created repository [centos]

[root@ibm-x3550m3-06 ~]# pulp-admin rpm repo sync run --repo-id centos
+----------------------------------------------------------------------+
                   Synchronizing Repository [centos]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[\]
... completed

Downloading repository content...
[==================================================] 100%
RPMs:       8652/8652 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 6/6 items
... completed

Importing errata...
[-]
... completed

Importing package groups/categories...
[\]
... completed

Task Succeeded

Initializing repo metadata
[-]
... completed

Publishing Distribution files
[-]
... completed

Publishing RPMs
[==================================================] 100%
8652 of 8652 items
... completed

Publishing Delta RPMs
... skipped

Publishing Errata
[-]
... completed

Publishing Comps file
[==================================================] 100%
102 of 102 items
... completed

Publishing Metadata.
[-]
... completed

Closing repo metadata
[-]
... completed

Generating sqlite files
... skipped

Publishing files to web
[|]
... completed

Writing Listings File
[-]
... completed

Task Succeeded

[root@ibm-x3550m3-06 ~]# 

#11 Updated by amacdona@redhat.com almost 6 years ago

  • Status changed from 6 to CLOSED - CURRENTRELEASE

#13 Updated by bmbouter over 2 years ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF