Issue #1824: iso repo publish fails for file in subdirectories - Pulp

Actions

Send by e-mail Copy link

Issue #1824

closed

iso repo publish fails for file in subdirectories

Added by paji@redhat.com almost 7 years ago. Updated almost 4 years ago.

Status:

CLOSED - CURRENTRELEASE

Priority:

Normal

Assignee:

mhrivnak

Category:

Sprint/Milestone:

Start date:

Due date:

Estimated time:

Severity:

2. Medium

Version:

Master

Platform Release:

2.8.3

OS:

Triaged:

Yes

Groomed:

Sprint Candidate:

Tags:

Pulp 2

Sprint:

Sprint 1

Quarter:

Description

Download the attached pulp-manifest.sh

Setup the following repo

$ chmod 700 pulp-manifest.sh 
$ mkdir -p foo/bar
$ touch foo/blah.txt
$ touch foo/bar/boo.txt
$ cd foo
$ ../pulp-manifest.sh blah.txt bar/boo.txt > PULP_MANIFEST
$ python -m SimpleHTTPServer 5555

on a diff window

$ pulp-admin iso repo create --repo-id=foo --feed=http://localhost:5555/foo
$ pulp-admin iso repo sync run --repo-id=foo

Expected:
Everything synced fine and dandy

Actual:


+----------------------------------------------------------------------+
                 Synchronizing Repository [foo]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

The Pulp Manifest was downloaded successfully.

There are no ISOs that need to be downloaded.

Task Succeeded

Task Failed

The distributor iso_distributor indicated a failed response when publishing
repository atomic-iso.

More stack trace from /var/log/messages

Apr  7 15:17:09 galaxy pulp: celery.worker.job:INFO: Task pulp.server.managers.repo.sync.sync[27f0b948-afcf-4adb-9c5e-2661655b4329] succeeded in 46.302700247s: <pulp.server.async.tasks.TaskResult object at 0x34497d0>
Apr  7 15:17:09 galaxy pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[d3ddd96d-a901-4985-997e-ba0d6a3a9546] succeeded in 0.00477200599562s: None
Apr  7 15:17:09 galaxy pulp: pulp.plugins.file.distributor:INFO: Beginning publish for repository <foo>
Apr  7 15:17:09 galaxy pulp: pulp.plugins.file.distributor:ERROR: (2419-02304) [Errno 2] No such file or directory
Apr  7 15:17:09 galaxy pulp: pulp.plugins.file.distributor:ERROR: (2419-02304) Traceback (most recent call last):
Apr  7 15:17:09 galaxy pulp: pulp.plugins.file.distributor:ERROR: (2419-02304)   File "/usr/lib/python2.7/site-packages/pulp/plugins/file/distributor.py", line 90, in publish_repo
Apr  7 15:17:09 galaxy pulp: pulp.plugins.file.distributor:ERROR: (2419-02304)     self._symlink_unit(build_dir, unit, links_to_create)
Apr  7 15:17:09 galaxy pulp: pulp.plugins.file.distributor:ERROR: (2419-02304)   File "/usr/lib/python2.7/site-packages/pulp/plugins/file/distributor.py", line 239, in _symlink_unit
Apr  7 15:17:09 galaxy pulp: pulp.plugins.file.distributor:ERROR: (2419-02304)     os.symlink(unit.storage_path, symlink_filename)
Apr  7 15:17:09 galaxy pulp: pulp.plugins.file.distributor:ERROR: (2419-02304) OSError: [Errno 2] No such file or directory
Apr  7 15:17:10 galaxy pulp: pulp.server.controllers.repository:INFO: {'state': 'failed', 'error_message': '[Errno 2] No such file or directory', 'traceback': 'Traceback (most recent call last):\n  File "/usr/lib/python2.7/site-packages/pulp/plugins/file/distributor.py", line 90, in publish_repo\n    self._symlink_unit(build_dir, unit, links_to_create)\n  File "/usr/lib/python2.7/site-packages/pulp/plugins/file/distributor.py", line 239, in _symlink_unit\n    os.symlink(unit.storage_path, symlink_filename)\nOSError: [Errno 2] No such file or directory\n', 'state_times': {'failed': '2016-04-07T19:17:09', 'not_started': '2016-04-07T19:17:09', 'in_progress': '2016-04-07T19:17:09'}}
Apr  7 15:17:10 galaxy pulp: py.warnings:WARNING: (2419-02304) /usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py:1039: DeprecationWarning: save is deprecated. Use insert_one or replace_one instead
Apr  7 15:17:10 galaxy pulp: py.warnings:WARNING: (2419-02304)   publish_result_coll.save(result)
Apr  7 15:17:10 galaxy pulp: py.warnings:WARNING: (2419-02304)
Apr  7 15:17:10 galaxy pulp: pulp.server.controllers.repository:ERROR: (2419-02304) Exception caught from plugin during publish for repo [foo]
Apr  7 15:17:10 galaxy pulp: pulp.server.controllers.repository:ERROR: (2419-02304) Traceback (most recent call last):
Apr  7 15:17:10 galaxy pulp: pulp.server.controllers.repository:ERROR: (2419-02304)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1026, in _do_publish
Apr  7 15:17:10 galaxy pulp: pulp.server.controllers.repository:ERROR: (2419-02304)     distributor_id=dist_id, summary=publish_report.summary
Apr  7 15:17:10 galaxy pulp: pulp.server.controllers.repository:ERROR: (2419-02304) PulpCodedException: The distributor iso_distributor indicated a failed response when publishing repository atomic-iso.
Apr  7 15:17:10 galaxy pulp: pulp.server.async.tasks:INFO: Task failed : [0657e309-bf23-49eb-80b2-e02e2d7649f0] : The distributor iso_distributor indicated a failed response when publishing repository atomic-iso.
Apr  7 15:17:10 galaxy pulp: celery.worker.job:INFO: Task pulp.server.managers.repo.publish.publish[0657e309-bf23-49eb-80b2-e02e2d7649f0] raised expected: PulpCodedException()

Files

pulp-manifest.sh (119 Bytes) pulp-manifest.sh

paji@redhat.com, 04/07/2016 10:37 PM

Actions

Copy link

Updated by paji@redhat.com almost 7 years ago

This was the recommended fix that seems to work

diff --git a/server/pulp/plugins/file/distributor.py b/server/pulp/plugins/file/distributor.py
index fd5c3cd..e30011b 100644
--- a/server/pulp/plugins/file/distributor.py
+++ b/server/pulp/plugins/file/distributor.py
@@ -236,6 +236,9 @@ class FileDistributor(Distributor):
                     os.remove(symlink_filename)
             # If we've gotten here, we've removed any existing file at the symlink_filename path,
             # so now we should recreate it.
+            parent = os.path.dirname(symlink_filename)
+            if not os.path.exists(parent):
+                os.makedirs(parent)
             os.symlink(unit.storage_path, symlink_filename)

     def _rmtree_if_exists(self, path):

Actions

Copy link

Updated by mhrivnak almost 7 years ago

Subject changed from pulp iso repo sync run failing on distro publish to iso repo publish fails for file in subdirectories
Status changed from NEW to ASSIGNED
Assignee set to mhrivnak
Sprint/Milestone set to 19

Actions

Copy link

Updated by mhrivnak almost 7 years ago

Status changed from ASSIGNED to POST

https://github.com/pulp/pulp/pull/2519

Actions

Copy link

Updated by mhrivnak almost 7 years ago

Triaged changed from No to Yes

Actions

Copy link

Updated by mhrivnak almost 7 years ago

Platform Release set to 2.8.3

Added by mhrivnak almost 7 years ago

Revision 6345568b

FileDistributor now works with files that have to be in subdirectories.

https://pulp.plan.io/issues/1824

fixes #1824

Added by mhrivnak almost 7 years ago

Revision 6345568b

FileDistributor now works with files that have to be in subdirectories.

https://pulp.plan.io/issues/1824

fixes #1824

Actions

Copy link

Updated by mhrivnak almost 7 years ago

Status changed from POST to MODIFIED
% Done changed from 0 to 100

Applied in changeset pulp|6345568b8f7ee5b9b90c80ce2addc6c7fd8d5c89.

Actions

Copy link

Updated by semyers almost 7 years ago

Status changed from MODIFIED to 5

Actions

Copy link

Updated by pthomas@redhat.com almost 7 years ago

Michael,

I am getting an error trying the reproducer steps. Not sure if its an actual failure or something I have missed in setup

[root@ibm-x3550m3-09 ~]# chmod 700 pulp-manifest.sh 
[root@ibm-x3550m3-09 ~]# mkdir -p foo/bar
[root@ibm-x3550m3-09 ~]# touch foo/blah.txt
[root@ibm-x3550m3-09 ~]# touch foo/bar/boo.txt
[root@ibm-x3550m3-09 foo]# ../pulp-manifest.sh blah.txt bar/boo.txt > PULP_MANIFEST
[root@ibm-x3550m3-09 foo]#  python -m SimpleHTTPServer 5555
Serving HTTP on 0.0.0.0 port 5555 ...
127.0.0.1 - - [11/May/2016 19:42:24] code 404, message File not found
127.0.0.1 - - [11/May/2016 19:42:24] "GET /foo/PULP_MANIFEST HTTP/1.1" 404 -

[root@ibm-x3550m3-09 ~]# pulp-admin iso repo sync run --repo-id=foo1
+----------------------------------------------------------------------+
                    Synchronizing Repository [foo1]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Task Failed

Importer indicated a failed response

Actions

Copy link

#10

Updated by mhrivnak almost 7 years ago

It looks like you started the http server from within the "foo" directory, but you were expecting it to be serving that dir. I think if you change your working directory to ../ and then start the http server, it may work.

Actions

Copy link

#11

Updated by pthomas@redhat.com almost 7 years ago

Status changed from 5 to 6

verified

[root@ibm-x3550m3-09 ~]# pulp-admin iso repo sync run --repo-id=foo1
+----------------------------------------------------------------------+
                    Synchronizing Repository [foo1]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

The Pulp Manifest was downloaded successfully.

Downloading 2 ISOs...

Successfully downloaded 2 ISOs.

Task Succeeded

The repository was successfully published.

Task Succeeded

Actions

Copy link

#12