Docker Support

I created a very small filesystem, mounted it to /var/cache/pulp/, and made sure the apache user had permission to write to it. I then tried to do a docker sync of a centos repo from docker hub.

$ dd if=/dev/zero of=foofs bs=1024 count=1000
$ mke2fs foofs 
$ sudo mount -o loop ./foofs /var/cache/pulp/
$ prestart
$ pulp-admin docker repo create --repo-id=centos --feed=https://registry-1.docker.io/ --upstream-name=centos
$ pulp-admin docker repo sync run --repo-id=centos

I saw the task fail with an error message in the terminal and lots of errors in the log. But after it was done, there were 11 blob units in the database, 10 of which were associated to the repo. The files were in /var/lib/pulp/content/ but most had a length of 0 bytes.

$ find /var/lib/pulp/content/units/docker_blob/ -type f | xargs ls -lh
-rw-r--r-- 1 apache apache   32 Aug  9 13:24 /var/lib/pulp/content/units/docker_blob/12/86843087e8774c31f670616e3c7e693a725d5615fe02d1043030e936d0e4f9/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
-rw-r--r-- 1 apache apache    0 Aug  9 13:24 /var/lib/pulp/content/units/docker_blob/20/5ecdb17746b9c6e24e9c7c7ab6751d0315a78b265edb05df2924009da9f90a/sha256:f6e4e6bc8376818ff75e2818d38b9451b87e3348e1d8256bef64ecd9000e7305
-rw-r--r-- 1 apache apache    0 Aug  9 13:24 /var/lib/pulp/content/units/docker_blob/52/c7b2a0896b367f1bcfa2e736f32fed0ccf459a7ad41ca3bf1aa23eb31f516b/sha256:3eaa9b70c44a88d9d4c2bcac5f8af1f7f63e12eae72fd21a029a54ad95a47f50
-rw-r--r-- 1 apache apache    0 Aug  9 13:24 /var/lib/pulp/content/units/docker_blob/5d/02bff11fee510f89fa10a43ef3b219f929ac6596477b86dc44fd03a2970512/sha256:f19ed9d73b3ef2c54aef640d150f40de66926016b485da5d086f9666679ce5a0
-rw-r--r-- 1 apache apache    0 Aug  9 13:24 /var/lib/pulp/content/units/docker_blob/62/6628be11abffb60b82481339a748fe11752b36aad0d03c7e49166ba8a05ed3/sha256:d129ecdb0249b2540429964ecf04a8f426199d27bfec517ecd9f19b2c7ae21ba
-rw-r--r-- 1 apache apache    0 Aug  9 13:24 /var/lib/pulp/content/units/docker_blob/75/9230be77c479edac688ee64f2481561d2edcdd68a4b0307c8b79945778e993/sha256:5237cae4d62996a5c7e0dec4eec6a221b5cad630c4d7de33db57004d6cd2501b
-rw-r--r-- 1 apache apache    0 Aug  9 13:24 /var/lib/pulp/content/units/docker_blob/87/66436ca847c58b74b7e4b03fcb65d20e061ca4312ed92a2c6e5c63ee868612/sha256:2fbe3a450f5d5be9ac8ed64d7a5e2434adc4156ab2530bcf74e32376de0aa371
-rw-r--r-- 1 apache apache    0 Aug  9 13:24 /var/lib/pulp/content/units/docker_blob/d5/de1456ada563c480f721bd94951b569df51e2e31e9185f59fcda7ccd60b850/sha256:ebd69957142bd0a24ecd4d5293bcfc18c2732741afb2e2401c60d13c3e80798c
-rw-r--r-- 1 apache apache 804K Aug  9 13:24 /var/lib/pulp/content/units/docker_blob/db/ac044223053fc10dff79e714751d4a94b6a01e80ee0ebb6523f2302ee091de/sha256:b6d7b2ebc0a7f5c0e87724be3904839d65562ec69909232ce1b7524257eb41af
-rw-r--r-- 1 apache apache    0 Aug  9 13:24 /var/lib/pulp/content/units/docker_blob/fa/5119032836af90a70da36533618d5742d382938bfc24acec2f30d466a8f3c4/sha256:3d8673bd162aafaf70e94029801a409fa3210363707ea1e7956b6f8ff9351cc8

This was the terminal output:

$ pulp-admin docker repo sync run --repo-id=centos
+----------------------------------------------------------------------+
                   Synchronizing Repository [centos]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading manifests
[-]
... completed

Copying units already in pulp
[-]
... completed

Copying units already in pulp
[-]
... completed

Downloading remote files
[====                                              ] 9%
1 of 11 items
... failed

Task Failed

Content import of
/var/cache/pulp/reserved_resource_worker-0@dev/52cdb2fd-8e19-4b33-bab8-8cb75715a
737/sha256:08a7a0bb61228a03e099486f3ffbb4e617a7b61d69aa7e8e803d925e1c5ebaaa
failed - must be an existing file.

and the log output;

Aug 09 13:23:22 dev pulp[18984]: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Aug 09 13:23:22 dev pulp[19004]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[95175ded-1bf6-4c04-a390-6e46175bf2ce]
Aug 09 13:23:22 dev pulp[19015]: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[5533191e-d489-4b90-bdab-10b160920bd6]
Aug 09 13:23:22 dev pulp[19015]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[8f566b88-baa1-4d29-b2ce-e0fdbde32fe4]
Aug 09 13:23:22 dev pulp[19004]: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[95175ded-1bf6-4c04-a390-6e46175bf2ce] succeeded in 0.0363659770228s: None
Aug 09 13:23:22 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): index.docker.io
Aug 09 13:23:22 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): auth.docker.io
Aug 09 13:23:22 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (2): index.docker.io
Aug 09 13:23:22 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (3): index.docker.io
Aug 09 13:23:22 dev pulp[19105]: pulp.server.async.tasks:INFO: Task failed : [5533191e-d489-4b90-bdab-10b160920bd6] : Could not fetch repository centos from registry https://index.docker.io/ - Unauthorized or Not Found
Aug 09 13:23:22 dev pulp[19015]: celery.worker.job:INFO: Task pulp.server.managers.repo.sync.sync[5533191e-d489-4b90-bdab-10b160920bd6] raised expected: PulpCodedException()
Aug 09 13:23:22 dev pulp[19015]: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[8f566b88-baa1-4d29-b2ce-e0fdbde32fe4] succeeded in 0.00693614489865s: None
Aug 09 13:24:08 dev pulp[18983]: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Aug 09 13:24:08 dev pulp[19004]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[d3c84ccd-8d9d-42d4-b500-b37c724722a9]
Aug 09 13:24:08 dev pulp[19015]: celery.worker.strategy:INFO: Received task: pulp.server.tasks.repository.distributor_update[0d6c6f10-d84b-4e6a-bba3-2b348f6623ce]
Aug 09 13:24:08 dev pulp[19015]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[9761fa09-1a1f-4a96-8bad-251904a23425]
Aug 09 13:24:08 dev pulp[19004]: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[d3c84ccd-8d9d-42d4-b500-b37c724722a9] succeeded in 0.0386713229818s: None
Aug 09 13:24:08 dev pulp[19015]: celery.worker.job:INFO: Task pulp.server.tasks.repository.distributor_update[0d6c6f10-d84b-4e6a-bba3-2b348f6623ce] succeeded in 0.074527513003s: <pulp.server.async.tasks.TaskResult object at 0x7effcbba6510>
Aug 09 13:24:08 dev pulp[19015]: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[9761fa09-1a1f-4a96-8bad-251904a23425] succeeded in 0.00498440302908s: None
Aug 09 13:24:11 dev pulp[19004]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[fe4c8d10-534d-4263-a369-9ad8ffad43ba]
Aug 09 13:24:11 dev pulp[19015]: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[52cdb2fd-8e19-4b33-bab8-8cb75715a737]
Aug 09 13:24:11 dev pulp[19015]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[f8db0c7b-13f9-42ab-9bf2-d5c0eec19737]
Aug 09 13:24:11 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): registry-1.docker.io
Aug 09 13:24:11 dev pulp[19004]: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[fe4c8d10-534d-4263-a369-9ad8ffad43ba] succeeded in 0.0350315800169s: None
Aug 09 13:24:11 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): auth.docker.io
Aug 09 13:24:11 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (2): registry-1.docker.io
Aug 09 13:24:12 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (3): registry-1.docker.io
Aug 09 13:24:16 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): registry-1.docker.io
Aug 09 13:24:16 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (2): registry-1.docker.io
Aug 09 13:24:16 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (3): registry-1.docker.io
Aug 09 13:24:16 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (4): registry-1.docker.io
Aug 09 13:24:16 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (5): registry-1.docker.io
Aug 09 13:24:16 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (6): registry-1.docker.io
Aug 09 13:24:16 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): dseasb33srnrn.cloudfront.net
Aug 09 13:24:16 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816) [Errno 28] No space left on device
Aug 09 13:24:16 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816) Traceback (most recent call last):
Aug 09 13:24:16 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 243, in _fetch
Aug 09 13:24:16 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816)     file_handle.write(chunk)
Aug 09 13:24:16 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816) IOError: [Errno 28] No space left on device
Aug 09 13:24:16 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (7): registry-1.docker.io
Aug 09 13:24:16 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (2): dseasb33srnrn.cloudfront.net
Aug 09 13:24:16 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (3): dseasb33srnrn.cloudfront.net
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-71520) [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-71520) Traceback (most recent call last):
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-71520)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 243, in _fetch
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-71520)     file_handle.write(chunk)
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-71520) IOError: [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (4): dseasb33srnrn.cloudfront.net
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656) [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656) Traceback (most recent call last):
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 243, in _fetch
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656)     file_handle.write(chunk)
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656) IOError: [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (5): dseasb33srnrn.cloudfront.net
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-86112) [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-86112) Traceback (most recent call last):
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-86112)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 243, in _fetch
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-86112)     file_handle.write(chunk)
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-86112) IOError: [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (8): registry-1.docker.io
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816) [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816) Traceback (most recent call last):
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 243, in _fetch
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816)     file_handle.write(chunk)
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816) IOError: [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (9): registry-1.docker.io
Aug 09 13:24:17 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (10): registry-1.docker.io
Aug 09 13:24:17 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (6): dseasb33srnrn.cloudfront.net
Aug 09 13:24:17 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (7): dseasb33srnrn.cloudfront.net
Aug 09 13:24:17 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (8): dseasb33srnrn.cloudfront.net
Aug 09 13:24:17 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (9): dseasb33srnrn.cloudfront.net
Aug 09 13:24:17 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (10): dseasb33srnrn.cloudfront.net
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-64224) [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-64224) Traceback (most recent call last):
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-64224)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 243, in _fetch
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-64224)     file_handle.write(chunk)
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-64224) IOError: [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-86112) [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-86112) Traceback (most recent call last):
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-86112)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 243, in _fetch
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-86112)     file_handle.write(chunk)
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-86112) IOError: [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816) [Errno 28] No space left on device: u'/var/cache/pulp/reserved_resource_worker-0@dev/52cdb2fd-8e19-4b33-bab8-8cb75715a737/sha256:08a7a0bb61228a03e099486f3ffbb4e617a7b61d69aa7e8e803d925e1c5ebaaa'
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816) Traceback (most recent call last):
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 229, in _fetch
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816)     file_handle = request.initialize_file_handle()
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816)   File "/usr/lib/python2.7/site-packages/nectar/request.py", line 46, in initialize_file_handle
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816)     self._file_handle = open(self.destination, 'wb')  # cache the handle
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816) IOError: [Errno 28] No space left on device: u'/var/cache/pulp/reserved_resource_worker-0@dev/52cdb2fd-8e19-4b33-bab8-8cb75715a737/sha256:08a7a0bb61228a03e099486f3ffbb4e617a7b61d69aa7e8e803d925e1c5ebaaa'
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656) [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656) Traceback (most recent call last):
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 243, in _fetch
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656)     file_handle.write(chunk)
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656) IOError: [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-71520) [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-71520) Traceback (most recent call last):
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-71520)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 243, in _fetch
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-71520)     file_handle.write(chunk)
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-71520) IOError: [Errno 28] No space left on device
Aug 09 13:24:18 dev pulp[19105]: pulp.server.async.tasks:INFO: Task failed : [52cdb2fd-8e19-4b33-bab8-8cb75715a737] : Content import of /var/cache/pulp/reserved_resource_worker-0@dev/52cdb2fd-8e19-4b33-bab8-8cb75715a737/sha256:08a7a0bb61228a03e099486f3ffbb4e617a7b61d69aa7e8e803d925e1c5ebaaa failed - must be an existing file.
Aug 09 13:24:18 dev pulp[19015]: celery.worker.job:INFO: Task pulp.server.managers.repo.sync.sync[52cdb2fd-8e19-4b33-bab8-8cb75715a737] raised expected: PulpCodedException()