Project

Profile

Help

Actions
Send by e-mail Copy link

Issue #1824

closed

iso repo publish fails for file in subdirectories

Added by paji@redhat.com about 8 years ago. Updated about 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
mhrivnak
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Master
Platform Release:
2.8.3
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 1
Quarter:

Description

Download the attached pulp-manifest.sh

Setup the following repo

$ chmod 700 pulp-manifest.sh 
$ mkdir -p foo/bar
$ touch foo/blah.txt
$ touch foo/bar/boo.txt
$ cd foo
$ ../pulp-manifest.sh blah.txt bar/boo.txt > PULP_MANIFEST
$ python -m SimpleHTTPServer 5555

on a diff window

$ pulp-admin iso repo create --repo-id=foo --feed=http://localhost:5555/foo
$ pulp-admin iso repo sync run --repo-id=foo

Expected:
Everything synced fine and dandy

Actual:


+----------------------------------------------------------------------+
                 Synchronizing Repository [foo]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

The Pulp Manifest was downloaded successfully.

There are no ISOs that need to be downloaded.

Task Succeeded

Task Failed

The distributor iso_distributor indicated a failed response when publishing
repository atomic-iso.

More stack trace from /var/log/messages

Apr  7 15:17:09 galaxy pulp: celery.worker.job:INFO: Task pulp.server.managers.repo.sync.sync[27f0b948-afcf-4adb-9c5e-2661655b4329] succeeded in 46.302700247s: <pulp.server.async.tasks.TaskResult object at 0x34497d0>
Apr  7 15:17:09 galaxy pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[d3ddd96d-a901-4985-997e-ba0d6a3a9546] succeeded in 0.00477200599562s: None
Apr  7 15:17:09 galaxy pulp: pulp.plugins.file.distributor:INFO: Beginning publish for repository <foo>
Apr  7 15:17:09 galaxy pulp: pulp.plugins.file.distributor:ERROR: (2419-02304) [Errno 2] No such file or directory
Apr  7 15:17:09 galaxy pulp: pulp.plugins.file.distributor:ERROR: (2419-02304) Traceback (most recent call last):
Apr  7 15:17:09 galaxy pulp: pulp.plugins.file.distributor:ERROR: (2419-02304)   File "/usr/lib/python2.7/site-packages/pulp/plugins/file/distributor.py", line 90, in publish_repo
Apr  7 15:17:09 galaxy pulp: pulp.plugins.file.distributor:ERROR: (2419-02304)     self._symlink_unit(build_dir, unit, links_to_create)
Apr  7 15:17:09 galaxy pulp: pulp.plugins.file.distributor:ERROR: (2419-02304)   File "/usr/lib/python2.7/site-packages/pulp/plugins/file/distributor.py", line 239, in _symlink_unit
Apr  7 15:17:09 galaxy pulp: pulp.plugins.file.distributor:ERROR: (2419-02304)     os.symlink(unit.storage_path, symlink_filename)
Apr  7 15:17:09 galaxy pulp: pulp.plugins.file.distributor:ERROR: (2419-02304) OSError: [Errno 2] No such file or directory
Apr  7 15:17:10 galaxy pulp: pulp.server.controllers.repository:INFO: {'state': 'failed', 'error_message': '[Errno 2] No such file or directory', 'traceback': 'Traceback (most recent call last):\n  File "/usr/lib/python2.7/site-packages/pulp/plugins/file/distributor.py", line 90, in publish_repo\n    self._symlink_unit(build_dir, unit, links_to_create)\n  File "/usr/lib/python2.7/site-packages/pulp/plugins/file/distributor.py", line 239, in _symlink_unit\n    os.symlink(unit.storage_path, symlink_filename)\nOSError: [Errno 2] No such file or directory\n', 'state_times': {'failed': '2016-04-07T19:17:09', 'not_started': '2016-04-07T19:17:09', 'in_progress': '2016-04-07T19:17:09'}}
Apr  7 15:17:10 galaxy pulp: py.warnings:WARNING: (2419-02304) /usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py:1039: DeprecationWarning: save is deprecated. Use insert_one or replace_one instead
Apr  7 15:17:10 galaxy pulp: py.warnings:WARNING: (2419-02304)   publish_result_coll.save(result)
Apr  7 15:17:10 galaxy pulp: py.warnings:WARNING: (2419-02304)
Apr  7 15:17:10 galaxy pulp: pulp.server.controllers.repository:ERROR: (2419-02304) Exception caught from plugin during publish for repo [foo]
Apr  7 15:17:10 galaxy pulp: pulp.server.controllers.repository:ERROR: (2419-02304) Traceback (most recent call last):
Apr  7 15:17:10 galaxy pulp: pulp.server.controllers.repository:ERROR: (2419-02304)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1026, in _do_publish
Apr  7 15:17:10 galaxy pulp: pulp.server.controllers.repository:ERROR: (2419-02304)     distributor_id=dist_id, summary=publish_report.summary
Apr  7 15:17:10 galaxy pulp: pulp.server.controllers.repository:ERROR: (2419-02304) PulpCodedException: The distributor iso_distributor indicated a failed response when publishing repository atomic-iso.
Apr  7 15:17:10 galaxy pulp: pulp.server.async.tasks:INFO: Task failed : [0657e309-bf23-49eb-80b2-e02e2d7649f0] : The distributor iso_distributor indicated a failed response when publishing repository atomic-iso.
Apr  7 15:17:10 galaxy pulp: celery.worker.job:INFO: Task pulp.server.managers.repo.publish.publish[0657e309-bf23-49eb-80b2-e02e2d7649f0] raised expected: PulpCodedException()

Files

pulp-manifest.sh (119 Bytes) pulp-manifest.sh paji@redhat.com, 04/07/2016 10:37 PM
Actions
Copy link
 #1

Updated by paji@redhat.com about 8 years ago

This was the recommended fix that seems to work

diff --git a/server/pulp/plugins/file/distributor.py b/server/pulp/plugins/file/distributor.py
index fd5c3cd..e30011b 100644
--- a/server/pulp/plugins/file/distributor.py
+++ b/server/pulp/plugins/file/distributor.py
@@ -236,6 +236,9 @@ class FileDistributor(Distributor):
                     os.remove(symlink_filename)
             # If we've gotten here, we've removed any existing file at the symlink_filename path,
             # so now we should recreate it.
+            parent = os.path.dirname(symlink_filename)
+            if not os.path.exists(parent):
+                os.makedirs(parent)
             os.symlink(unit.storage_path, symlink_filename)

     def _rmtree_if_exists(self, path):
Actions
Copy link
 #2

Updated by mhrivnak about 8 years ago

  • Subject changed from pulp iso repo sync run failing on distro publish to iso repo publish fails for file in subdirectories
  • Status changed from NEW to ASSIGNED
  • Assignee set to mhrivnak
  • Sprint/Milestone set to 19
Actions
Copy link
 #3

Updated by mhrivnak about 8 years ago

  • Status changed from ASSIGNED to POST
Actions
Copy link
 #4

Updated by mhrivnak about 8 years ago

  • Triaged changed from No to Yes
Actions
Copy link
 #5

Updated by mhrivnak about 8 years ago

  • Platform Release set to 2.8.3

Added by mhrivnak about 8 years ago

Revision 6345568b | View on GitHub

FileDistributor now works with files that have to be in subdirectories.

https://pulp.plan.io/issues/1824

fixes #1824

Added by mhrivnak about 8 years ago

Revision 6345568b | View on GitHub

FileDistributor now works with files that have to be in subdirectories.

https://pulp.plan.io/issues/1824

fixes #1824

Actions
Copy link
 #7

Updated by mhrivnak about 8 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100
Actions
Copy link
 #8

Updated by semyers almost 8 years ago

  • Status changed from MODIFIED to 5
Actions
Copy link
 #9

Updated by pthomas@redhat.com almost 8 years ago

Michael,

I am getting an error trying the reproducer steps. Not sure if its an actual failure or something I have missed in setup

[root@ibm-x3550m3-09 ~]# chmod 700 pulp-manifest.sh 
[root@ibm-x3550m3-09 ~]# mkdir -p foo/bar
[root@ibm-x3550m3-09 ~]# touch foo/blah.txt
[root@ibm-x3550m3-09 ~]# touch foo/bar/boo.txt
[root@ibm-x3550m3-09 foo]# ../pulp-manifest.sh blah.txt bar/boo.txt > PULP_MANIFEST
[root@ibm-x3550m3-09 foo]#  python -m SimpleHTTPServer 5555
Serving HTTP on 0.0.0.0 port 5555 ...
127.0.0.1 - - [11/May/2016 19:42:24] code 404, message File not found
127.0.0.1 - - [11/May/2016 19:42:24] "GET /foo/PULP_MANIFEST HTTP/1.1" 404 -

[root@ibm-x3550m3-09 ~]# pulp-admin iso repo sync run --repo-id=foo1
+----------------------------------------------------------------------+
                    Synchronizing Repository [foo1]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Task Failed

Importer indicated a failed response
Actions
Copy link
 #10

Updated by mhrivnak almost 8 years ago

It looks like you started the http server from within the "foo" directory, but you were expecting it to be serving that dir. I think if you change your working directory to ../ and then start the http server, it may work.

Actions
Copy link
 #11

Updated by pthomas@redhat.com almost 8 years ago

  • Status changed from 5 to 6

verified

[root@ibm-x3550m3-09 ~]# pulp-admin iso repo sync run --repo-id=foo1
+----------------------------------------------------------------------+
                    Synchronizing Repository [foo1]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

The Pulp Manifest was downloaded successfully.

Downloading 2 ISOs...

Successfully downloaded 2 ISOs.

Task Succeeded

The repository was successfully published.

Task Succeeded
Actions
Copy link
 #12

Updated by semyers almost 8 years ago

  • Status changed from 6 to CLOSED - CURRENTRELEASE
Actions
Copy link
 #14

Updated by bmbouter about 6 years ago

  • Sprint set to Sprint 1
Actions
Copy link
 #15

Updated by bmbouter about 6 years ago

  • Sprint/Milestone deleted (19)
Actions
Copy link
 #16

Updated by bmbouter about 5 years ago

  • Tags Pulp 2 added
Actions
Send by e-mail Copy link

Also available in: Atom PDF