Debian Support

As a user, I should be able to specify an URL to a feed, and be able to use the sync command to synchronize debian packages from an URL.

A guest during test day attempted to create and sync a debian repo using a debian mirror and sync fails.

• Pulp packages and plugins:

rpm -qa | grep pulp

python-pulp-puppet-common-2.14.0-0.3.rc.fc25.noarch
pulp-puppet-plugins-2.14.0-0.3.rc.fc25.noarch
pulp-ostree-plugins-1.3.0-0.3.rc.fc25.noarch
python-pulp-streamer-2.14.0-0.3.rc.fc25.noarch
python-pulp-bindings-2.14.0-0.3.rc.fc25.noarch
pulp-deb-plugins-1.5.2-0.2.beta.fc25.noarch
python-pulp-docker-common-3.0.0-0.3.rc.fc25.noarch
python-pulp-oid_validation-2.14.0-0.3.rc.fc25.noarch
pulp-docker-plugins-3.0.0-0.3.rc.fc25.noarch
pulp-puppet-admin-extensions-2.14.0-0.3.rc.fc25.noarch
pulp-ostree-admin-extensions-1.3.0-0.3.rc.fc25.noarch
python-pulp-common-2.14.0-0.3.rc.fc25.noarch
pulp-docker-admin-extensions-3.0.0-0.3.rc.fc25.noarch
python-pulp-python-common-2.0.1-0.3.rc.fc25.noarch
pulp-python-plugins-2.0.1-0.3.rc.fc25.noarch
python-pulp-rpm-common-2.14.0-0.3.rc.fc25.noarch
pulp-rpm-plugins-2.14.0-0.3.rc.fc25.noarch
pulp-admin-client-2.14.0-0.3.rc.fc25.noarch
pulp-deb-admin-extensions-1.5.2-0.2.beta.fc25.noarch
python-pulp-repoauth-2.14.0-0.3.rc.fc25.noarch
pulp-rpm-admin-extensions-2.14.0-0.3.rc.fc25.noarch
python-pulp-client-lib-2.14.0-0.3.rc.fc25.noarch
python-pulp-deb-common-1.5.2-0.2.beta.fc25.noarch
pulp-selinux-2.14.0-0.3.rc.fc25.noarch
pulp-server-2.14.0-0.3.rc.fc25.noarch
python-pulp-ostree-common-1.3.0-0.3.rc.fc25.noarch
pulp-python-admin-extensions-2.0.1-0.3.rc.fc25.noarch

[test@fedora25-base214 ~]$ cat /etc/redhat-release
Fedora release 25 (Twenty Five)

pulp-admin status
+----------------------------------------------------------------------+
                          Status of the server
+----------------------------------------------------------------------+

Api Version:           2
Database Connection:   
  Connected: True
Known Workers:         
  _id:            scheduler@fedora25-base214
  _ns:            workers
  Last Heartbeat: 2017-08-13T15:43:56Z
  _id:            resource_manager@fedora25-base214
  _ns:            workers
  Last Heartbeat: 2017-08-13T15:43:57Z
  _id:            reserved_resource_worker-0@fedora25-base214
  _ns:            workers
  Last Heartbeat: 2017-08-13T15:43:57Z
  _id:            reserved_resource_worker-1@fedora25-base214
  _ns:            workers
  Last Heartbeat: 2017-08-13T15:43:57Z
Messaging Connection:  
  Connected: True
Versions:              
  Platform Version: 2.14rc3

[test@fedora25-base214 ~]$ pulp-admin deb repo create --repo-id debian-repo --feed http://ftp.debian.org/debian/dists/stable/
Successfully created repository [debian-repo]

[test@fedora25-base214 ~]$ pulp-admin deb repo sync run --repo-id=debian-repo
+----------------------------------------------------------------------+
                 Synchronizing Repository [debian-repo]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Task Failed

[Errno 2] No such file or directory:
u'/var/cache/pulp/reserved_resource_worker-0@fedora25-base214/21a48ad3-6a97-4738
-9294-4ed9f37b8e15/Release'

[test@fedora25-base214 ~]$ journalctl -f > debian.test.logs &

Aug 13 12:09:10 fedora25-base214 pulp[2584]: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[19b9b8b9-01ed-4f1e-b1d3-9882b90b536a] succeeded in 0.0236528830001s: None
Aug 13 12:09:30 fedora25-base214 gnome-terminal-[2370]: Allocating size to GtkScrollbar 0x7caf495390 without calling gtk_widget_get_preferred_width/height(). How does the code know the size to allocate?
Aug 13 12:09:32 fedora25-base214 gnome-terminal-[2370]: Allocating size to GtkScrollbar 0x7caf2508e0 without calling gtk_widget_get_preferred_width/height(). How does the code know the size to allocate?
Aug 13 12:10:16 fedora25-base214 pulp[2577]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[043bcbd4-3f65-4d6b-9247-912e476590c8]
Aug 13 12:10:16 fedora25-base214 pulp[2584]: celery.worker.strategy:INFO: Received task: pulp.server.tasks.repository.delete[1e7f1ec0-bb4a-4497-877b-5afa335df288]
Aug 13 12:10:16 fedora25-base214 pulp[2584]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[35f9bdbd-91ba-4a86-8f66-fde9d3e17645]
Aug 13 12:10:16 fedora25-base214 pulp[2577]: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[043bcbd4-3f65-4d6b-9247-912e476590c8] succeeded in 0.0389802519999s: None
Aug 13 12:10:16 fedora25-base214 pulp[2584]: celery.worker.job:INFO: Task pulp.server.tasks.repository.delete[1e7f1ec0-bb4a-4497-877b-5afa335df288] succeeded in 0.0342449720001s: <pulp.server.async.tasks.TaskResult object at 0x7f80ec85d450>
Aug 13 12:10:16 fedora25-base214 pulp[2584]: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[35f9bdbd-91ba-4a86-8f66-fde9d3e17645] succeeded in 0.0255510260004s: None
Aug 13 12:10:20 fedora25-base214 pulp[2592]: pulp.server.webservices.middleware.exception:INFO: Missing resource(s): repository=debian-repo1
Aug 13 12:10:43 fedora25-base214 tracker-extract[2140]: unable to create file '/run/user/1000/dconf/user': Permission denied.  dconf will not work properly.
Aug 13 12:10:46 fedora25-base214 tracker-extract[2140]: unable to create file '/run/user/1000/dconf/user': Permission denied.  dconf will not work properly.
Aug 13 12:10:49 fedora25-base214 tracker-extract[2140]: unable to create file '/run/user/1000/dconf/user': Permission denied.  dconf will not work properly.
Aug 13 12:10:52 fedora25-base214 tracker-extract[2140]: unable to create file '/run/user/1000/dconf/user': Permission denied.  dconf will not work properly.
Aug 13 12:10:55 fedora25-base214 tracker-extract[2140]: unable to create file '/run/user/1000/dconf/user': Permission denied.  dconf will not work properly.
Aug 13 12:10:58 fedora25-base214 tracker-extract[2140]: unable to create file '/run/user/1000/dconf/user': Permission denied.  dconf will not work properly.
Aug 13 12:11:01 fedora25-base214 tracker-extract[2140]: unable to create file '/run/user/1000/dconf/user': Permission denied.  dconf will not work properly.
Aug 13 12:11:04 fedora25-base214 tracker-extract[2140]: unable to create file '/run/user/1000/dconf/user': Permission denied.  dconf will not work properly.
Aug 13 12:11:07 fedora25-base214 pulp[2577]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[c54c2e95-7d7e-4c09-bf12-d9d84594d7cb]
Aug 13 12:11:07 fedora25-base214 pulp[2584]: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[21a48ad3-6a97-4738-9294-4ed9f37b8e15]
Aug 13 12:11:07 fedora25-base214 pulp[2713]: pulp_deb.plugins.importers.importer:INFO: [21a48ad3] Repo sync started.
Aug 13 12:11:07 fedora25-base214 pulp[2713]: pulp_deb.plugins.importers.sync:INFO: [21a48ad3] Downloading http://ftp.debian.org/debian/dists/stable/dists/stable/Release
Aug 13 12:11:07 fedora25-base214 pulp[2584]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[9207f966-2c78-4200-a530-e7de56f2f5b2]
Aug 13 12:11:07 fedora25-base214 pulp[2713]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): ftp.debian.org
Aug 13 12:11:07 fedora25-base214 pulp[2577]: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[c54c2e95-7d7e-4c09-bf12-d9d84594d7cb] succeeded in 0.0389599970003s: None
Aug 13 12:11:07 fedora25-base214 tracker-extract[2140]: unable to create file '/run/user/1000/dconf/user': Permission denied.  dconf will not work properly.
Aug 13 12:11:08 fedora25-base214 pulp[2713]: nectar.downloaders.threaded:INFO: Download failed: Download of http://ftp.debian.org/debian/dists/stable/dists/stable/Release failed with code 404: Not Found
Aug 13 12:11:08 fedora25-base214 pulp[2713]: pulp.server.async.tasks:INFO: [21a48ad3] Task failed : [21a48ad3-6a97-4738-9294-4ed9f37b8e15]
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800) Task pulp.server.managers.repo.sync.sync[21a48ad3-6a97-4738-9294-4ed9f37b8e15] raised unexpected: IOError(2, 'No such file or directory')
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800) Traceback (most recent call last):
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800)     R = retval = fun(*args, **kwargs)
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 522, in __call__
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800)     return super(Task, self).__call__(*args, **kwargs)
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 107, in __call__
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800)     return super(PulpTask, self).__call__(*args, **kwargs)
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800)     return self.run(*args, **kwargs)
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 769, in sync
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800)     sync_report = sync_repo(transfer_repo, conduit, call_config)
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 725, in wrap_f
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800)     return f(*args, **kwargs)
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800)   File "/usr/lib/python2.7/site-packages/pulp_deb/plugins/importers/importer.py", line 135, in sync_repo
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800)     report = self._current_sync.process_lifecycle()
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 572, in process_lifecycle
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800)     super(PluginStep, self).process_lifecycle()
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 163, in process_lifecycle
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800)     step.process()
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 256, in process
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800)     self._process_block()
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 303, in _process_block
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800)     self.process_main()
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800)   File "/usr/lib/python2.7/site-packages/pulp_deb/plugins/importers/sync.py", line 95, in process_main
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800)     release=open(self.parent.release_file, "rb"),
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:ERROR: (2584-56800) IOError: [Errno 2] No such file or directory: u'/var/cache/pulp/reserved_resource_worker-0@fedora25-base214/21a48ad3-6a97-4738-9294-4ed9f37b8e15/Release'
Aug 13 12:11:08 fedora25-base214 pulp[2584]: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[9207f966-2c78-4200-a530-e7de56f2f5b2] succeeded in 0.00361970600034s: None
Aug 13 12:11:10 fedora25-base214 tracker-extract[2140]: unable to create file '/run/user/1000/dconf/user': Permission denied.  dconf will not work properly.
Aug 13 12:11:16 fedora25-base214 tracker-extract[2140]: unable to create file '/run/user/1000/dconf/user': Permission denied.  dconf will not work properly.

Related story: https://pulp.plan.io/issues/2763#change-21246