Actions
Issue #2967
closed
Unable to sync debian repos using URL as feed
Start date:
Due date:
Estimated time:
Severity:
1. Low
Version - Debian:
Platform Release:
Target Release - Debian:
OS:
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:
Description
As a user, I should be able to specify an URL to a feed, and be able to use the sync command to synchronize debian packages from an URL.
A guest during test day attempted to create and sync a debian repo using a debian mirror and sync fails.
- Pulp packages and plugins:
rpm -qa | grep pulp
python-pulp-puppet-common-2.14.0-0.3.rc.fc25.noarch
pulp-puppet-plugins-2.14.0-0.3.rc.fc25.noarch
pulp-ostree-plugins-1.3.0-0.3.rc.fc25.noarch
python-pulp-streamer-2.14.0-0.3.rc.fc25.noarch
python-pulp-bindings-2.14.0-0.3.rc.fc25.noarch
pulp-deb-plugins-1.5.2-0.2.beta.fc25.noarch
python-pulp-docker-common-3.0.0-0.3.rc.fc25.noarch
python-pulp-oid_validation-2.14.0-0.3.rc.fc25.noarch
pulp-docker-plugins-3.0.0-0.3.rc.fc25.noarch
pulp-puppet-admin-extensions-2.14.0-0.3.rc.fc25.noarch
pulp-ostree-admin-extensions-1.3.0-0.3.rc.fc25.noarch
python-pulp-common-2.14.0-0.3.rc.fc25.noarch
pulp-docker-admin-extensions-3.0.0-0.3.rc.fc25.noarch
python-pulp-python-common-2.0.1-0.3.rc.fc25.noarch
pulp-python-plugins-2.0.1-0.3.rc.fc25.noarch
python-pulp-rpm-common-2.14.0-0.3.rc.fc25.noarch
pulp-rpm-plugins-2.14.0-0.3.rc.fc25.noarch
pulp-admin-client-2.14.0-0.3.rc.fc25.noarch
pulp-deb-admin-extensions-1.5.2-0.2.beta.fc25.noarch
python-pulp-repoauth-2.14.0-0.3.rc.fc25.noarch
pulp-rpm-admin-extensions-2.14.0-0.3.rc.fc25.noarch
python-pulp-client-lib-2.14.0-0.3.rc.fc25.noarch
python-pulp-deb-common-1.5.2-0.2.beta.fc25.noarch
pulp-selinux-2.14.0-0.3.rc.fc25.noarch
pulp-server-2.14.0-0.3.rc.fc25.noarch
python-pulp-ostree-common-1.3.0-0.3.rc.fc25.noarch
pulp-python-admin-extensions-2.0.1-0.3.rc.fc25.noarch
[test@fedora25-base214 ~]$ cat /etc/redhat-release
Fedora release 25 (Twenty Five)
pulp-admin status
+----------------------------------------------------------------------+
Status of the server
+----------------------------------------------------------------------+
Api Version: 2
Database Connection:
Connected: True
Known Workers:
_id: scheduler@fedora25-base214
_ns: workers
Last Heartbeat: 2017-08-13T15:43:56Z
_id: resource_manager@fedora25-base214
_ns: workers
Last Heartbeat: 2017-08-13T15:43:57Z
_id: reserved_resource_worker-0@fedora25-base214
_ns: workers
Last Heartbeat: 2017-08-13T15:43:57Z
_id: reserved_resource_worker-1@fedora25-base214
_ns: workers
Last Heartbeat: 2017-08-13T15:43:57Z
Messaging Connection:
Connected: True
Versions:
Platform Version: 2.14rc3
[test@fedora25-base214 ~]$ pulp-admin deb repo create --repo-id debian-repo --feed http://ftp.debian.org/debian/dists/stable/
Successfully created repository [debian-repo]
[test@fedora25-base214 ~]$ pulp-admin deb repo sync run --repo-id=debian-repo
+----------------------------------------------------------------------+
Synchronizing Repository [debian-repo]
+----------------------------------------------------------------------+
This command may be exited via ctrl+c without affecting the request.
Task Failed
[Errno 2] No such file or directory:
u'/var/cache/pulp/reserved_resource_worker-0@fedora25-base214/21a48ad3-6a97-4738
-9294-4ed9f37b8e15/Release'
[test@fedora25-base214 ~]$ journalctl -f > debian.test.logs &
Aug 13 12:09:10 fedora25-base214 pulp[2584]: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[19b9b8b9-01ed-4f1e-b1d3-9882b90b536a] succeeded in 0.0236528830001s: None
Aug 13 12:09:30 fedora25-base214 gnome-terminal-[2370]: Allocating size to GtkScrollbar 0x7caf495390 without calling gtk_widget_get_preferred_width/height(). How does the code know the size to allocate?
Aug 13 12:09:32 fedora25-base214 gnome-terminal-[2370]: Allocating size to GtkScrollbar 0x7caf2508e0 without calling gtk_widget_get_preferred_width/height(). How does the code know the size to allocate?
Aug 13 12:10:16 fedora25-base214 pulp[2577]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[043bcbd4-3f65-4d6b-9247-912e476590c8]
Aug 13 12:10:16 fedora25-base214 pulp[2584]: celery.worker.strategy:INFO: Received task: pulp.server.tasks.repository.delete[1e7f1ec0-bb4a-4497-877b-5afa335df288]
Aug 13 12:10:16 fedora25-base214 pulp[2584]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[35f9bdbd-91ba-4a86-8f66-fde9d3e17645]
Aug 13 12:10:16 fedora25-base214 pulp[2577]: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[043bcbd4-3f65-4d6b-9247-912e476590c8] succeeded in 0.0389802519999s: None
Aug 13 12:10:16 fedora25-base214 pulp[2584]: celery.worker.job:INFO: Task pulp.server.tasks.repository.delete[1e7f1ec0-bb4a-4497-877b-5afa335df288] succeeded in 0.0342449720001s: <pulp.server.async.tasks.TaskResult object at 0x7f80ec85d450>
Aug 13 12:10:16 fedora25-base214 pulp[2584]: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[35f9bdbd-91ba-4a86-8f66-fde9d3e17645] succeeded in 0.0255510260004s: None
Aug 13 12:10:20 fedora25-base214 pulp[2592]: pulp.server.webservices.middleware.exception:INFO: Missing resource(s): repository=debian-repo1
Aug 13 12:10:43 fedora25-base214 tracker-extract[2140]: unable to create file '/run/user/1000/dconf/user': Permission denied. dconf will not work properly.
Aug 13 12:10:46 fedora25-base214 tracker-extract[2140]: unable to create file '/run/user/1000/dconf/user': Permission denied. dconf will not work properly.
Aug 13 12:10:49 fedora25-base214 tracker-extract[2140]: unable to create file '/run/user/1000/dconf/user': Permission denied. dconf will not work properly.
Aug 13 12:10:52 fedora25-base214 tracker-extract[2140]: unable to create file '/run/user/1000/dconf/user': Permission denied. dconf will not work properly.
Aug 13 12:10:55 fedora25-base214 tracker-extract[2140]: unable to create file '/run/user/1000/dconf/user': Permission denied. dconf will not work properly.
Aug 13 12:10:58 fedora25-base214 tracker-extract[2140]: unable to create file '/run/user/1000/dconf/user': Permission denied. dconf will not work properly.
Aug 13 12:11:01 fedora25-base214 tracker-extract[2140]: unable to create file '/run/user/1000/dconf/user': Permission denied. dconf will not work properly.
Aug 13 12:11:04 fedora25-base214 tracker-extract[2140]: unable to create file '/run/user/1000/dconf/user': Permission denied. dconf will not work properly.
Aug 13 12:11:07 fedora25-base214 pulp[2577]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[c54c2e95-7d7e-4c09-bf12-d9d84594d7cb]
Aug 13 12:11:07 fedora25-base214 pulp[2584]: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[21a48ad3-6a97-4738-9294-4ed9f37b8e15]
Aug 13 12:11:07 fedora25-base214 pulp[2713]: pulp_deb.plugins.importers.importer:INFO: [21a48ad3] Repo sync started.
Aug 13 12:11:07 fedora25-base214 pulp[2713]: pulp_deb.plugins.importers.sync:INFO: [21a48ad3] Downloading http://ftp.debian.org/debian/dists/stable/dists/stable/Release
Aug 13 12:11:07 fedora25-base214 pulp[2584]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[9207f966-2c78-4200-a530-e7de56f2f5b2]
Aug 13 12:11:07 fedora25-base214 pulp[2713]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): ftp.debian.org
Aug 13 12:11:07 fedora25-base214 pulp[2577]: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[c54c2e95-7d7e-4c09-bf12-d9d84594d7cb] succeeded in 0.0389599970003s: None
Aug 13 12:11:07 fedora25-base214 tracker-extract[2140]: unable to create file '/run/user/1000/dconf/user': Permission denied. dconf will not work properly.
Aug 13 12:11:08 fedora25-base214 pulp[2713]: nectar.downloaders.threaded:INFO: Download failed: Download of http://ftp.debian.org/debian/dists/stable/dists/stable/Release failed with code 404: Not Found
Aug 13 12:11:08 fedora25-base214 pulp[2713]: pulp.server.async.tasks:INFO: [21a48ad3] Task failed : [21a48ad3-6a97-4738-9294-4ed9f37b8e15]
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800) Task pulp.server.managers.repo.sync.sync[21a48ad3-6a97-4738-9294-4ed9f37b8e15] raised unexpected: IOError(2, 'No such file or directory')
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800) Traceback (most recent call last):
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800) R = retval = fun(*args, **kwargs)
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 522, in __call__
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800) return super(Task, self).__call__(*args, **kwargs)
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 107, in __call__
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800) return super(PulpTask, self).__call__(*args, **kwargs)
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800) return self.run(*args, **kwargs)
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800) File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 769, in sync
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800) sync_report = sync_repo(transfer_repo, conduit, call_config)
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 725, in wrap_f
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800) return f(*args, **kwargs)
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800) File "/usr/lib/python2.7/site-packages/pulp_deb/plugins/importers/importer.py", line 135, in sync_repo
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800) report = self._current_sync.process_lifecycle()
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800) File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 572, in process_lifecycle
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800) super(PluginStep, self).process_lifecycle()
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800) File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 163, in process_lifecycle
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800) step.process()
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800) File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 256, in process
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800) self._process_block()
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800) File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 303, in _process_block
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800) self.process_main()
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800) File "/usr/lib/python2.7/site-packages/pulp_deb/plugins/importers/sync.py", line 95, in process_main
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800) release=open(self.parent.release_file, "rb"),
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800) IOError: [Errno 2] No such file or directory: u'/var/cache/pulp/reserved_resource_worker-0@fedora25-base214/21a48ad3-6a97-4738-9294-4ed9f37b8e15/Release'
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[9207f966-2c78-4200-a530-e7de56f2f5b2] succeeded in 0.00361970600034s: None
Aug 13 12:11:10 fedora25-base214 tracker-extract[2140]: unable to create file '/run/user/1000/dconf/user': Permission denied. dconf will not work properly.
Aug 13 12:11:16 fedora25-base214 tracker-extract[2140]: unable to create file '/run/user/1000/dconf/user': Permission denied. dconf will not work properly.
Related story: https://pulp.plan.io/issues/2763#change-21246
Updated by kersom over 7 years ago
- Subject changed from Unable to sync debian repo using URL to Unable to sync debian repo using URL as feed
Updated by mhrivnak over 7 years ago
Not having used pulp_deb, I'm not understanding what the problem is. Is it not possible to do a sync at all because you can't provide a feed? Or does it expect the feed to be something other than a URL? Or is there a problem with the expectations around constructing the URL?
Perhaps it would help to describe 1) what behavior do you want to see and why, vs 2) what behavior do you actually see
Updated by kersom over 7 years ago
- Subject changed from Unable to sync debian repo using URL as feed to Unable to sync debian repos using URL as feed
- Description updated (diff)
Updated by mihai.ibanescu@gmail.com about 7 years ago
- Status changed from NEW to ASSIGNED
I don't know why I didn't get notified when this issue was created, I apologize.
With the changes slated to go out with pulp 2.15, I expect syncing of upstream repos to work much better.
If you have the chance, please test one of the 2.15 nightlies, I would really appreciate that.
README.md has been updated in https://github.com/pulp/pulp_deb/commit/e5302c46a844fc579ca7e336d98fd59a5609778c to describe how to selectively sync a release/component/arch.
Updated by mihai.ibanescu@gmail.com about 7 years ago
- Assignee set to mihai.ibanescu@gmail.com
Updated by quba42 about 6 years ago
Does this issue need to remain open?
As far as I can tell this works fine using the commands
pulp-admin deb repo create --repo-id 'debian-repo' --feed 'http://ftp.debian.org/debian/' --releases 'stable'
pulp-admin deb repo sync run --repo-id=debian-repo1
On a general note, it is my understanding that the feed URL for debian repositories should always be to the base of the repository. Any desired releases, components, and arches should then be chosen via the --releases, --components, and --architectures flags?
Updated by mdellweg about 6 years ago
- Status changed from ASSIGNED to CLOSED - NOTABUG
Actions
.