Project

Profile

Help

Actions
Send by e-mail Copy link

Issue #1869

closed

Resynchronizing rhel repos seems to be failing after upgrade

Added by pthomas@redhat.com almost 7 years ago. Updated almost 4 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
dkliban@redhat.com
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Platform Release:
2.8.3
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 2
Quarter:

Description

Resynchronizing a rhel repo that was setup pre upgrade failing

1. Setup 2.7 pulp
2. Create a rhel6 kickstart repo
3. Sync the repo 
4. Upgrade pulp to 2.8
5. Re sync rhel6 repo

$ pulp-admin rpm repo create     --feed https://cdn.redhat.com/content/dist/rhel/rhui/server/6/6.7/x86_64/kickstart/ --feed-ca-cert CDN/cdn.redhat.com-chain.crt     --feed-cert CDN/914f702153514b06c1ef279db9dcadce.crt     --feed-key CDN/914f702153514b06c1ef279db9dcadce.key 

[root@qe-blade-11 ~]# pulp-admin rpm repo sync run --repo-id rhel6
+----------------------------------------------------------------------+
                    Synchronizing Repository [rhel6]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[\]
... completed

Downloading repository content...
[-]
[==================================================] 100%
RPMs:       0/0 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items
... completed

Importing errata...
[-]
... completed

Importing package groups/categories...
[-]
... completed

Cleaning duplicate packages...
[-]
... completed

Task Succeeded

Copying files
[-]
... completed

Initializing repo metadata
[-]
... completed

Publishing Distribution files
[-]

Task Failed

[Errno 2] No such file or directory:
u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-11.idmqe.lab.eng.bos.redha
t.com/4c2fff49-ea28-4629-b7ef-d4c9f1f54a2f/PULP_DISTRIBUTION.xml'

[root@qe-blade-11 ~]# pulp-admin tasks details  --task-id 00f3aed9-e059-4874-9206-2439ed5e877b
+----------------------------------------------------------------------+
                              Task Details
+----------------------------------------------------------------------+

Operations:       publish
Resources:        rhel6 (repository)
State:            Failed
Start Time:       2016-04-27T20:48:12Z
Finish Time:      2016-04-27T20:48:33Z
Result:           N/A
Task Id:          00f3aed9-e059-4874-9206-2439ed5e877b
Progress Report:  
  Yum Distributor: 
    Description:   Copying files
    Details:       
    Error Details: 
    Items Total:   1
    Num Failures:  0
    Num Processed: 1
    Num Success:   1
    State:         FINISHED
    Step Id:       f857604b-23d0-4c39-b3ea-5bf68af327d5
    Step Type:     save_tar
    Description:   Initializing repo metadata
    Details:       
    Error Details: 
    Items Total:   1
    Num Failures:  0
    Num Processed: 1
    Num Success:   1
    State:         FINISHED
    Step Id:       f49b9cc3-c630-49e8-a096-a42af344d6c1
    Step Type:     initialize_repo_metadata
    Description:   Publishing Distribution files
    Details:       
    Error Details: 
      Error:     [Errno 2] No such file or directory:
                 u'/var/cache/pulp/reserved_resource_worker-6@qe-blade-11.idmqe.
                 lab.eng.bos.redhat.com/00f3aed9-e059-4874-9206-2439ed5e877b/PUL
                 P_DISTRIBUTION.xml'
      Traceback:   File
                 "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_ste
                 p.py", line 232, in process     self._process_block(item=item) 
                 File
                 "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_ste
                 p.py", line 291, in _process_block    
                 self.process_main(item=item)    File
                 "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors
                 /yum/publish.py", line 748, in process_main    
                 self._publish_distribution_files(unit)    File
                 "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors
                 /yum/publish.py", line 816, in _publish_distribution_files    
                 self._write_pulp_distribution_file(distro_files, xml_file_path)
                 File
                 "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors
                 /yum/publish.py", line 843, in _write_pulp_distribution_file   
                 tree.write(os.path.join(self.get_working_dir(),
                 constants.DISTRIBUTION_XML))    File
                 "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 802, in
                 write     file = open(file_or_filename, "wb") 
    Items Total:   1
    Num Failures:  1
    Num Processed: 1
    Num Success:   0
    State:         FAILED
    Step Id:       71d29c22-485f-45b8-81b0-42f45bf8485f
    Step Type:     distribution
    Description:   Publishing RPMs
    Details:       
    Error Details: 
    Items Total:   1
    Num Failures:  0
    Num Processed: 0
    Num Success:   0
    State:         NOT_STARTED
    Step Id:       8ca87714-ae7e-4eac-9037-45a80b10291d
    Step Type:     rpms
    Description:   Publishing Delta RPMs
    Details:       
    Error Details: 
    Items Total:   1
    Num Failures:  0
    Num Processed: 0
    Num Success:   0
    State:         NOT_STARTED
    Step Id:       f83412c9-783a-4f98-9938-0378cd7bcbf6
    Step Type:     drpms
    Description:   Publishing Errata
    Details:       
    Error Details: 
    Items Total:   1
    Num Failures:  0
    Num Processed: 0
    Num Success:   0
    State:         NOT_STARTED
    Step Id:       a31e06e4-4bcb-4859-a099-f49d20ef197e
    Step Type:     errata
    Description:   Publishing Comps file
    Details:       
    Error Details: 
    Items Total:   1
    Num Failures:  0
    Num Processed: 0
    Num Success:   0
    State:         NOT_STARTED
    Step Id:       07906e28-291d-4be0-a201-e8b98e236de1
    Step Type:     comps
    Description:   Publishing Metadata.
    Details:       
    Error Details: 
    Items Total:   1
    Num Failures:  0
    Num Processed: 0
    Num Success:   0
    State:         NOT_STARTED
    Step Id:       ce8b596f-d90c-46d1-9eaa-7ac8baf7b8bf
    Step Type:     metadata
    Description:   Closing repo metadata
    Details:       
    Error Details: 
    Items Total:   1
    Num Failures:  0
    Num Processed: 0
    Num Success:   0
    State:         NOT_STARTED
    Step Id:       765166ad-1e67-4685-a30c-ad44e82b8679
    Step Type:     close_repo_metadata
    Description:   Generating sqlite files
    Details:       
    Error Details: 
    Items Total:   1
    Num Failures:  0
    Num Processed: 0
    Num Success:   0
    State:         NOT_STARTED
    Step Id:       2fb24b43-c2f4-4764-9237-2a6cbde8c9c9
    Step Type:     generate sqlite
    Description:   Publishing files to web
    Details:       
    Error Details: 
    Items Total:   1
    Num Failures:  0
    Num Processed: 0
    Num Success:   0
    State:         NOT_STARTED
    Step Id:       e1d9bfc7-39b5-43df-bffb-106442314b73
    Step Type:     publish_directory
    Description:   Writing Listings File
    Details:       
    Error Details: 
    Items Total:   1
    Num Failures:  0
    Num Processed: 0
    Num Success:   0
    State:         NOT_STARTED
    Step Id:       2ab94019-dc10-40c2-9c0c-f1fa1b358cde
    Step Type:     initialize_repo_metadata
    Description:   Writing Listings File
    Details:       
    Error Details: 
    Items Total:   1
    Num Failures:  0
    Num Processed: 0
    Num Success:   0
    State:         NOT_STARTED
    Step Id:       f2a016a2-1019-425f-b4c9-bdbb63a1ce2e
    Step Type:     initialize_repo_metadata
Traceback:        Traceback (most recent call last):   File
                  "/usr/lib/python2.7/site-packages/celery/app/trace.py", line
                  240, in trace_task     R = retval = fun(*args, **kwargs)  
                  File
                  "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py",
                  line 473, in __call__     return super(Task,
                  self).__call__(*args, **kwargs)   File
                  "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py",
                  line 103, in __call__     return super(PulpTask,
                  self).__call__(*args, **kwargs)   File
                  "/usr/lib/python2.7/site-packages/celery/app/trace.py", line
                  437, in __protected_call__     return self.run(*args,
                  **kwargs)   File
                  "/usr/lib/python2.7/site-packages/pulp/server/controllers/repo
                  sitory.py", line 971, in publish     result =
                  _do_publish(repo_obj, dist_id, dist_inst, transfer_repo,
                  conduit, call_config)   File
                  "/usr/lib/python2.7/site-packages/pulp/server/controllers/repo
                  sitory.py", line 1023, in _do_publish     publish_report =
                  publish_repo(transfer_repo, conduit, call_config)   File
                  "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py",
                  line 658, in wrap_f     return f(*args, **kwargs)   File
                  "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributor
                  s/yum/distributor.py", line 174, in publish_repo     return
                  self._publisher.process_lifecycle()   File
                  "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_st
                  ep.py", line 562, in process_lifecycle     super(PluginStep,
                  self).process_lifecycle()   File
                  "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_st
                  ep.py", line 159, in process_lifecycle     step.process()  
                  File
                  "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_st
                  ep.py", line 232, in process    
                  self._process_block(item=item)   File
                  "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_st
                  ep.py", line 291, in _process_block    
                  self.process_main(item=item)   File
                  "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributor
                  s/yum/publish.py", line 748, in process_main    
                  self._publish_distribution_files(unit)   File
                  "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributor
                  s/yum/publish.py", line 816, in _publish_distribution_files   
                  self._write_pulp_distribution_file(distro_files,
                  xml_file_path)   File
                  "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributor
                  s/yum/publish.py", line 843, in _write_pulp_distribution_file 
                  tree.write(os.path.join(self.get_working_dir(),
                  constants.DISTRIBUTION_XML))   File
                  "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 802, in
                  write     file = open(file_or_filename, "wb") IOError: [Errno
                  2] No such file or directory:
                  u'/var/cache/pulp/reserved_resource_worker-6@qe-blade-11.idmqe
                  .lab.eng.bos.redhat.com/00f3aed9-e059-4874-9206-2439ed5e877b/P
                  ULP_DISTRIBUTION.xml' 

[root@qe-blade-11 ~]# 

Apr 26 16:36:35 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12386]: pulp.plugins.pulp_rpm.plugins.distributors.yum.metadata.metadata:WARNING: Overwriting existing metadata file [/var/cache/pulp/reserved_resource_worker-7@qe-blade-11.idmqe.lab.eng.bos.redhat.com/99635eab-72d1-421c-b231-a0a8c70eccad/repodata/repomd.xml]
Apr 26 16:36:49 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12386]: pulp.server.controllers.repository:ERROR: (12386-96928) Exception caught from plugin during publish for repo [rhel6]
Apr 26 16:36:49 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12386]: pulp.server.controllers.repository:ERROR: (12386-96928) Traceback (most recent call last):
Apr 26 16:36:49 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12386]: pulp.server.controllers.repository:ERROR: (12386-96928)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1023, in _do_publish
Apr 26 16:36:49 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12386]: pulp.server.controllers.repository:ERROR: (12386-96928)     publish_report = publish_repo(transfer_repo, conduit, call_config)
Apr 26 16:36:49 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12386]: pulp.server.controllers.repository:ERROR: (12386-96928)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 658, in wrap_f
Apr 26 16:36:49 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12386]: pulp.server.controllers.repository:ERROR: (12386-96928)     return f(*args, **kwargs)
Apr 26 16:36:49 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12386]: pulp.server.controllers.repository:ERROR: (12386-96928)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/distributor.py", line 174, in publish_repo
Apr 26 16:36:49 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12386]: pulp.server.controllers.repository:ERROR: (12386-96928)     return self._publisher.process_lifecycle()
Apr 26 16:36:49 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12386]: pulp.server.controllers.repository:ERROR: (12386-96928)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 562, in process_lifecycle
Apr 26 16:36:49 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12386]: pulp.server.controllers.repository:ERROR: (12386-96928)     super(PluginStep, self).process_lifecycle()
Apr 26 16:36:49 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12386]: pulp.server.controllers.repository:ERROR: (12386-96928)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 159, in process_lifecycle
Apr 26 16:36:49 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12386]: pulp.server.controllers.repository:ERROR: (12386-96928)     step.process()
Apr 26 16:36:49 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12386]: pulp.server.controllers.repository:ERROR: (12386-96928)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 232, in process
Apr 26 16:36:49 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12386]: pulp.server.controllers.repository:ERROR: (12386-96928)     self._process_block(item=item)
Apr 26 16:36:49 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12386]: pulp.server.controllers.repository:ERROR: (12386-96928)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 291, in _process_block
Apr 26 16:36:49 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12386]: pulp.server.controllers.repository:ERROR: (12386-96928)     self.process_main(item=item)
Apr 26 16:36:49 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12386]: pulp.server.controllers.repository:ERROR: (12386-96928)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/publish.py", line 748, in process_main
Apr 26 16:36:49 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12386]: pulp.server.controllers.repository:ERROR: (12386-96928)     self._publish_distribution_files(unit)
Apr 26 16:36:49 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12386]: pulp.server.controllers.repository:ERROR: (12386-96928)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/publish.py", line 816, in _publish_distribution_files
Apr 26 16:36:49 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12386]: pulp.server.controllers.repository:ERROR: (12386-96928)     self._write_pulp_distribution_file(distro_files, xml_file_path)
Apr 26 16:36:49 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12386]: pulp.server.controllers.repository:ERROR: (12386-96928)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/publish.py", line 843, in _write_pulp_distribution_file
Apr 26 16:36:49 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12386]: pulp.server.controllers.repository:ERROR: (12386-96928)     tree.write(os.path.join(self.get_working_dir(), constants.DISTRIBUTION_XML))
Apr 26 16:36:49 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12386]: pulp.server.controllers.repository:ERROR: (12386-96928)   File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 802, in write
Apr 26 16:36:49 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12386]: pulp.server.controllers.repository:ERROR: (12386-96928)     file = open(file_or_filename, "wb")
Apr 26 16:36:49 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12386]: pulp.server.controllers.repository:ERROR: (12386-96928) IOError: [Errno 2] No such file or directory: u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-11.idmqe.lab.eng.bos.redhat.com/99635eab-72d1-421c-b231-a0a8c70eccad/PULP_DISTRIBUTION.xml'
Apr 26 16:36:49 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12386]: pulp.server.async.tasks:INFO: Task failed : [99635eab-72d1-421c-b231-a0a8c70eccad]
Apr 26 16:36:50 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928) Task pulp.server.managers.repo.publish.publish[99635eab-72d1-421c-b231-a0a8c70eccad] raised unexpected: IOError(2, 'No such file or directory')
Apr 26 16:36:50 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928) Traceback (most recent call last):
Apr 26 16:36:50 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Apr 26 16:36:50 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928)     R = retval = fun(*args, **kwargs)
Apr 26 16:36:50 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 473, in __call__
Apr 26 16:36:50 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928)     return super(Task, self).__call__(*args, **kwargs)
Apr 26 16:36:50 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 103, in __call__
Apr 26 16:36:50 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928)     return super(PulpTask, self).__call__(*args, **kwargs)
Apr 26 16:36:50 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
Apr 26 16:36:50 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928)     return self.run(*args, **kwargs)
Apr 26 16:36:50 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 971, in publish
Apr 26 16:36:50 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928)     result = _do_publish(repo_obj, dist_id, dist_inst, transfer_repo, conduit, call_config)
Apr 26 16:36:50 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1023, in _do_publish
Apr 26 16:36:50 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928)     publish_report = publish_repo(transfer_repo, conduit, call_config)
Apr 26 16:36:50 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 658, in wrap_f
Apr 26 16:36:50 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928)     return f(*args, **kwargs)
Apr 26 16:36:50 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/distributor.py", line 174, in publish_repo
Apr 26 16:36:50 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928)     return self._publisher.process_lifecycle()
Apr 26 16:36:50 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 562, in process_lifecycle
Apr 26 16:36:50 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928)     super(PluginStep, self).process_lifecycle()
Apr 26 16:36:50 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 159, in process_lifecycle
Apr 26 16:36:50 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928)     step.process()
Apr 26 16:36:50 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 232, in process
Apr 26 16:36:50 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928)     self._process_block(item=item)
Apr 26 16:36:50 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 291, in _process_block
Apr 26 16:36:50 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928)     self.process_main(item=item)
Apr 26 16:36:50 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/publish.py", line 748, in process_main
Apr 26 16:36:50 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928)     self._publish_distribution_files(unit)
Apr 26 16:36:50 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/publish.py", line 816, in _publish_distribution_files
Apr 26 16:36:50 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928)     self._write_pulp_distribution_file(distro_files, xml_file_path)
Apr 26 16:36:50 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/publish.py", line 843, in _write_pulp_distribution_file
Apr 26 16:36:50 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928)     tree.write(os.path.join(self.get_working_dir(), constants.DISTRIBUTION_XML))
Apr 26 16:36:50 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928)   File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 802, in write
Apr 26 16:36:50 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928)     file = open(file_or_filename, "wb")
Apr 26 16:36:50 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928) IOError: [Errno 2] No such file or directory: u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-11.idmqe.lab.eng.bos.redhat.com/99635eab-72d1-421c-b231-a0a8c70eccad/PULP_DISTRIBUTION.xml'

root@qe-blade-11 ~]# pulp-admin rpm repo sync run --repo-id centos5
+----------------------------------------------------------------------+
                   Synchronizing Repository [centos5]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[|]
... completed

Downloading repository content...
[-]
[==================================================] 100%
RPMs:       46/46 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[===========================================       ] 87%
Distributions: 7/8 items

Task Failed

Importer indicated a failed response

[root@qe-blade-11 ~]# 

Apr 26 16:40:05 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12071]: celery.worker.job:INFO: Task pulp.server.managers.repo.publish.publish[970b68a8-055a-4e95-8e38-a4cba3493642] succeeded in 0.509529267001s: {'exception': None, 'repo_id': u'zoo', 'traceback': None, 'started': '2016-04-26T20:40:05Z', '_ns': u'repo_publish_results',...
Apr 26 16:40:05 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12071]: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[f1876a82-545c-475c-8273-13b6ef9ab7e1] succeeded in 0.0230738100072s: None
Apr 26 16:40:42 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12386]: pulp_rpm.plugins.importers.yum.parse.treeinfo:ERROR: some distro file downloads failed
Apr 26 16:40:42 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12386]: pulp_rpm.plugins.importers.yum.sync:ERROR: (12386-96928)
Apr 26 16:40:42 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12386]: pulp_rpm.plugins.importers.yum.sync:ERROR: (12386-96928) Traceback (most recent call last):
Apr 26 16:40:42 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12386]: pulp_rpm.plugins.importers.yum.sync:ERROR: (12386-96928)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 228, in run
Apr 26 16:40:42 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12386]: pulp_rpm.plugins.importers.yum.sync:ERROR: (12386-96928)     dist_sync.run()
Apr 26 16:40:42 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12386]: pulp_rpm.plugins.importers.yum.sync:ERROR: (12386-96928)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/parse/treeinfo.py", line 125, in run
Apr 26 16:40:42 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12386]: pulp_rpm.plugins.importers.yum.sync:ERROR: (12386-96928)     self._run(tmp_dir)
Apr 26 16:40:42 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12386]: pulp_rpm.plugins.importers.yum.sync:ERROR: (12386-96928)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/parse/treeinfo.py", line 195, in _run
Apr 26 16:40:42 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12386]: pulp_rpm.plugins.importers.yum.sync:ERROR: (12386-96928)     for destination, location in downloaded:
Apr 26 16:40:42 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12386]: pulp_rpm.plugins.importers.yum.sync:ERROR: (12386-96928)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/parse/treeinfo.py", line 284, in download_files
Apr 26 16:40:42 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12386]: pulp_rpm.plugins.importers.yum.sync:ERROR: (12386-96928)     raise DownloadFailed()
Apr 26 16:40:42 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12386]: pulp_rpm.plugins.importers.yum.sync:ERROR: (12386-96928) DownloadFailed
Apr 26 16:40:42 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12386]: pulp.server.async.tasks:INFO: Task failed : [f3ee2e57-842b-40f3-861d-9b65c4e634e8]
Apr 26 16:40:42 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928) Task pulp.server.managers.repo.sync.sync[f3ee2e57-842b-40f3-861d-9b65c4e634e8] raised unexpected: PulpExecutionException('Importer indicated a failed response',)
Apr 26 16:40:42 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928) Traceback (most recent call last):
Apr 26 16:40:42 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Apr 26 16:40:42 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928)     R = retval = fun(*args, **kwargs)
Apr 26 16:40:42 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 473, in __call__
Apr 26 16:40:42 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928)     return super(Task, self).__call__(*args, **kwargs)
Apr 26 16:40:42 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 103, in __call__
Apr 26 16:40:42 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928)     return super(PulpTask, self).__call__(*args, **kwargs)
Apr 26 16:40:42 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
Apr 26 16:40:42 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928)     return self.run(*args, **kwargs)
Apr 26 16:40:42 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 810, in sync
Apr 26 16:40:42 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928)     raise pulp_exceptions.PulpExecutionException(_('Importer indicated a failed response'))
Apr 26 16:40:42 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:ERROR: (12100-96928) PulpExecutionException: Importer indicated a failed response
Apr 26 16:40:42 qe-blade-11.idmqe.lab.eng.bos.redhat.com pulp[12100]: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[dd76746d-f541-4414-892a-a5b9352a0ae0] succeeded in 0.0075548599998s: None
Actions
Copy link
 #1

Updated by pthomas@redhat.com almost 7 years ago

  • Description updated (diff)
Actions
Copy link
 #2

Updated by pthomas@redhat.com almost 7 years ago

  • Description updated (diff)
Actions
Copy link
 #3

Updated by dkliban@redhat.com almost 7 years ago

I am setting up a reproducer. Let me know if you pick up this task so I can point you at the machine that has it.

Actions
Copy link
 #4

Updated by dkliban@redhat.com almost 7 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dkliban@redhat.com
  • Priority changed from Normal to High
  • Severity changed from 2. Medium to 3. High
  • Platform Release set to 2.8.3
  • Triaged changed from No to Yes
Actions
Copy link
 #5

Updated by dkliban@redhat.com almost 7 years ago

  • Status changed from ASSIGNED to NEW

This is a regression from 2.8.2, but 2.8.2 had issue #1843[0] related to this.

Before the fix for issue #1843, Pulp would sync PULP_DISTRIBUTION.xml file from upstream repository and then publish it. The solution to #1843 was to start generating PULP_DISTRIBUTION.xml during the publish. This works well the first time a repository is published. On a subsequent publish, when a the pulp worker attempts to write out the PULP_DISTRIBUTION.xml unit to it's working directory, it is not able to open the file for writing. The failure occurs because PULP_DISTRIBUTION.xml already exists in the working directory as a symlink pointing to /var/lib/pulp/content/units/rpm/......./PULP_DISTRIBUTION.xml.

If we delete the unit first, there is a chance that the publish won't finish for some other reason and the deleted unit will not be replaced. I am not ready to proceed with a fix for this.

[0]https://pulp.plan.io/issues/1843

Actions
Copy link
 #6

Updated by dkliban@redhat.com almost 7 years ago

This is probably a problem on the first publish also. After the PULP_DISTRIBUTION.xml unit is synced, a symlink to it will be present in the working directory of worker doing the publish.

Actions
Copy link
 #7

Updated by bmbouter almost 7 years ago

@dkliban, your description makes sense. It does sound like the distribution unit is already published (the copy of the upstream one) and when Pulp goes to write one it can't overwrite the symlink. I think we need info from @jcline.

Actions
Copy link
 #8

Updated by dkliban@redhat.com almost 7 years ago

  • Assignee deleted (dkliban@redhat.com)
Actions
Copy link
 #9

Updated by mhrivnak almost 7 years ago

  • Sprint/Milestone set to 20
Actions
Copy link
 #10

Updated by dkliban@redhat.com almost 7 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dkliban@redhat.com
Actions
Copy link
 #11

Updated by dkliban@redhat.com almost 7 years ago

  • Status changed from ASSIGNED to POST

Added by dkliban@redhat.com almost 7 years ago

Revision 409dbf84

Removes symlink/file PULP_DISTRIBUTION.xml before writing a new file

fixes #1869 https://pulp.plan.io/issues/1869

Actions
Copy link
 #12

Updated by dkliban@redhat.com almost 7 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100
Actions
Copy link
 #13

Updated by semyers almost 7 years ago

  • Status changed from MODIFIED to 5
Actions
Copy link
 #14

Updated by pthomas@redhat.com almost 7 years ago

  • Status changed from 5 to 6

verified in 2.6->2.7 upgrade


[root@ibm-x3250m4-03 ~]# rpm -qa pulp-server
pulp-server-2.8.3-0.2.beta.el6.noarch
[root@ibm-x3250m4-03 ~]# 

[root@ibm-x3250m4-03 ~]# pulp-admin rpm repo sync run --repo-id rhel5
+----------------------------------------------------------------------+
                    Synchronizing Repository [rhel5]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[|]
... completed

Downloading repository content...
[-]
[==================================================] 100%
RPMs:       0/0 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items
... completed

Importing errata...
[-]
... completed

Importing package groups/categories...
[-]
... completed

Cleaning duplicate packages...
[-]
... completed

Task Succeeded

Copying files
[-]
... completed

Initializing repo metadata
[-]
... completed

Publishing Distribution files
[-]
... completed

Publishing RPMs
[/]
... completed

Publishing Delta RPMs
... skipped

Publishing Errata
[-]
... completed

Publishing Comps file
[-]
... completed

Publishing Metadata.
[-]
... completed

Closing repo metadata
[-]
... completed

Generating sqlite files
... skipped

Publishing files to web
[\]
... completed

Writing Listings File
[-]
... completed

Writing Listings File
[-]
... completed

Task Succeeded
Actions
Copy link
 #15

Updated by semyers almost 7 years ago

  • Status changed from 6 to CLOSED - CURRENTRELEASE
Actions
Copy link
 #17

Updated by bmbouter about 5 years ago

  • Sprint set to Sprint 2
Actions
Copy link
 #18

Updated by bmbouter about 5 years ago

  • Sprint/Milestone deleted (20)
Actions
Copy link
 #19

Updated by bmbouter almost 4 years ago

  • Tags Pulp 2 added
Actions
Send by e-mail Copy link

Also available in: Atom PDF