Issue #2142: Units created with 0-byte files when sync runs out of disk space - Docker Support - Pulp

Actions

Send by e-mail Copy link

Issue #2142

closed

Units created with 0-byte files when sync runs out of disk space

Added by mhrivnak over 7 years ago. Updated about 5 years ago.

Status:

CLOSED - CURRENTRELEASE

Priority:

High

Assignee:

dkliban@redhat.com

Start date:

Due date:

Estimated time:

Severity:

3. High

Version - Docker:

master

Platform Release:

2.9.3

Target Release - Docker:

OS:

Triaged:

Yes

Groomed:

Sprint Candidate:

Tags:

Pulp 2

Sprint:

Sprint 7

Quarter:

Description

I created a very small filesystem, mounted it to /var/cache/pulp/, and made sure the apache user had permission to write to it. I then tried to do a docker sync of a centos repo from docker hub.

$ dd if=/dev/zero of=foofs bs=1024 count=1000
$ mke2fs foofs 
$ sudo mount -o loop ./foofs /var/cache/pulp/
$ prestart
$ pulp-admin docker repo create --repo-id=centos --feed=https://registry-1.docker.io/ --upstream-name=centos
$ pulp-admin docker repo sync run --repo-id=centos

I saw the task fail with an error message in the terminal and lots of errors in the log. But after it was done, there were 11 blob units in the database, 10 of which were associated to the repo. The files were in /var/lib/pulp/content/ but most had a length of 0 bytes.

$ find /var/lib/pulp/content/units/docker_blob/ -type f | xargs ls -lh
-rw-r--r-- 1 apache apache   32 Aug  9 13:24 /var/lib/pulp/content/units/docker_blob/12/86843087e8774c31f670616e3c7e693a725d5615fe02d1043030e936d0e4f9/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
-rw-r--r-- 1 apache apache    0 Aug  9 13:24 /var/lib/pulp/content/units/docker_blob/20/5ecdb17746b9c6e24e9c7c7ab6751d0315a78b265edb05df2924009da9f90a/sha256:f6e4e6bc8376818ff75e2818d38b9451b87e3348e1d8256bef64ecd9000e7305
-rw-r--r-- 1 apache apache    0 Aug  9 13:24 /var/lib/pulp/content/units/docker_blob/52/c7b2a0896b367f1bcfa2e736f32fed0ccf459a7ad41ca3bf1aa23eb31f516b/sha256:3eaa9b70c44a88d9d4c2bcac5f8af1f7f63e12eae72fd21a029a54ad95a47f50
-rw-r--r-- 1 apache apache    0 Aug  9 13:24 /var/lib/pulp/content/units/docker_blob/5d/02bff11fee510f89fa10a43ef3b219f929ac6596477b86dc44fd03a2970512/sha256:f19ed9d73b3ef2c54aef640d150f40de66926016b485da5d086f9666679ce5a0
-rw-r--r-- 1 apache apache    0 Aug  9 13:24 /var/lib/pulp/content/units/docker_blob/62/6628be11abffb60b82481339a748fe11752b36aad0d03c7e49166ba8a05ed3/sha256:d129ecdb0249b2540429964ecf04a8f426199d27bfec517ecd9f19b2c7ae21ba
-rw-r--r-- 1 apache apache    0 Aug  9 13:24 /var/lib/pulp/content/units/docker_blob/75/9230be77c479edac688ee64f2481561d2edcdd68a4b0307c8b79945778e993/sha256:5237cae4d62996a5c7e0dec4eec6a221b5cad630c4d7de33db57004d6cd2501b
-rw-r--r-- 1 apache apache    0 Aug  9 13:24 /var/lib/pulp/content/units/docker_blob/87/66436ca847c58b74b7e4b03fcb65d20e061ca4312ed92a2c6e5c63ee868612/sha256:2fbe3a450f5d5be9ac8ed64d7a5e2434adc4156ab2530bcf74e32376de0aa371
-rw-r--r-- 1 apache apache    0 Aug  9 13:24 /var/lib/pulp/content/units/docker_blob/d5/de1456ada563c480f721bd94951b569df51e2e31e9185f59fcda7ccd60b850/sha256:ebd69957142bd0a24ecd4d5293bcfc18c2732741afb2e2401c60d13c3e80798c
-rw-r--r-- 1 apache apache 804K Aug  9 13:24 /var/lib/pulp/content/units/docker_blob/db/ac044223053fc10dff79e714751d4a94b6a01e80ee0ebb6523f2302ee091de/sha256:b6d7b2ebc0a7f5c0e87724be3904839d65562ec69909232ce1b7524257eb41af
-rw-r--r-- 1 apache apache    0 Aug  9 13:24 /var/lib/pulp/content/units/docker_blob/fa/5119032836af90a70da36533618d5742d382938bfc24acec2f30d466a8f3c4/sha256:3d8673bd162aafaf70e94029801a409fa3210363707ea1e7956b6f8ff9351cc8

This was the terminal output:

$ pulp-admin docker repo sync run --repo-id=centos
+----------------------------------------------------------------------+
                   Synchronizing Repository [centos]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading manifests
[-]
... completed

Copying units already in pulp
[-]
... completed

Copying units already in pulp
[-]
... completed

Downloading remote files
[====                                              ] 9%
1 of 11 items
... failed

Task Failed

Content import of
/var/cache/pulp/reserved_resource_worker-0@dev/52cdb2fd-8e19-4b33-bab8-8cb75715a
737/sha256:08a7a0bb61228a03e099486f3ffbb4e617a7b61d69aa7e8e803d925e1c5ebaaa
failed - must be an existing file.

and the log output;

Aug 09 13:23:22 dev pulp[18984]: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Aug 09 13:23:22 dev pulp[19004]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[95175ded-1bf6-4c04-a390-6e46175bf2ce]
Aug 09 13:23:22 dev pulp[19015]: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[5533191e-d489-4b90-bdab-10b160920bd6]
Aug 09 13:23:22 dev pulp[19015]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[8f566b88-baa1-4d29-b2ce-e0fdbde32fe4]
Aug 09 13:23:22 dev pulp[19004]: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[95175ded-1bf6-4c04-a390-6e46175bf2ce] succeeded in 0.0363659770228s: None
Aug 09 13:23:22 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): index.docker.io
Aug 09 13:23:22 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): auth.docker.io
Aug 09 13:23:22 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (2): index.docker.io
Aug 09 13:23:22 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (3): index.docker.io
Aug 09 13:23:22 dev pulp[19105]: pulp.server.async.tasks:INFO: Task failed : [5533191e-d489-4b90-bdab-10b160920bd6] : Could not fetch repository centos from registry https://index.docker.io/ - Unauthorized or Not Found
Aug 09 13:23:22 dev pulp[19015]: celery.worker.job:INFO: Task pulp.server.managers.repo.sync.sync[5533191e-d489-4b90-bdab-10b160920bd6] raised expected: PulpCodedException()
Aug 09 13:23:22 dev pulp[19015]: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[8f566b88-baa1-4d29-b2ce-e0fdbde32fe4] succeeded in 0.00693614489865s: None
Aug 09 13:24:08 dev pulp[18983]: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Aug 09 13:24:08 dev pulp[19004]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[d3c84ccd-8d9d-42d4-b500-b37c724722a9]
Aug 09 13:24:08 dev pulp[19015]: celery.worker.strategy:INFO: Received task: pulp.server.tasks.repository.distributor_update[0d6c6f10-d84b-4e6a-bba3-2b348f6623ce]
Aug 09 13:24:08 dev pulp[19015]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[9761fa09-1a1f-4a96-8bad-251904a23425]
Aug 09 13:24:08 dev pulp[19004]: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[d3c84ccd-8d9d-42d4-b500-b37c724722a9] succeeded in 0.0386713229818s: None
Aug 09 13:24:08 dev pulp[19015]: celery.worker.job:INFO: Task pulp.server.tasks.repository.distributor_update[0d6c6f10-d84b-4e6a-bba3-2b348f6623ce] succeeded in 0.074527513003s: <pulp.server.async.tasks.TaskResult object at 0x7effcbba6510>
Aug 09 13:24:08 dev pulp[19015]: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[9761fa09-1a1f-4a96-8bad-251904a23425] succeeded in 0.00498440302908s: None
Aug 09 13:24:11 dev pulp[19004]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[fe4c8d10-534d-4263-a369-9ad8ffad43ba]
Aug 09 13:24:11 dev pulp[19015]: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[52cdb2fd-8e19-4b33-bab8-8cb75715a737]
Aug 09 13:24:11 dev pulp[19015]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[f8db0c7b-13f9-42ab-9bf2-d5c0eec19737]
Aug 09 13:24:11 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): registry-1.docker.io
Aug 09 13:24:11 dev pulp[19004]: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[fe4c8d10-534d-4263-a369-9ad8ffad43ba] succeeded in 0.0350315800169s: None
Aug 09 13:24:11 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): auth.docker.io
Aug 09 13:24:11 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (2): registry-1.docker.io
Aug 09 13:24:12 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (3): registry-1.docker.io
Aug 09 13:24:16 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): registry-1.docker.io
Aug 09 13:24:16 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (2): registry-1.docker.io
Aug 09 13:24:16 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (3): registry-1.docker.io
Aug 09 13:24:16 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (4): registry-1.docker.io
Aug 09 13:24:16 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (5): registry-1.docker.io
Aug 09 13:24:16 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (6): registry-1.docker.io
Aug 09 13:24:16 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): dseasb33srnrn.cloudfront.net
Aug 09 13:24:16 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816) [Errno 28] No space left on device
Aug 09 13:24:16 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816) Traceback (most recent call last):
Aug 09 13:24:16 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 243, in _fetch
Aug 09 13:24:16 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816)     file_handle.write(chunk)
Aug 09 13:24:16 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816) IOError: [Errno 28] No space left on device
Aug 09 13:24:16 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (7): registry-1.docker.io
Aug 09 13:24:16 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (2): dseasb33srnrn.cloudfront.net
Aug 09 13:24:16 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (3): dseasb33srnrn.cloudfront.net
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-71520) [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-71520) Traceback (most recent call last):
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-71520)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 243, in _fetch
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-71520)     file_handle.write(chunk)
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-71520) IOError: [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (4): dseasb33srnrn.cloudfront.net
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656) [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656) Traceback (most recent call last):
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 243, in _fetch
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656)     file_handle.write(chunk)
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656) IOError: [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (5): dseasb33srnrn.cloudfront.net
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-86112) [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-86112) Traceback (most recent call last):
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-86112)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 243, in _fetch
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-86112)     file_handle.write(chunk)
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-86112) IOError: [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (8): registry-1.docker.io
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816) [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816) Traceback (most recent call last):
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 243, in _fetch
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816)     file_handle.write(chunk)
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816) IOError: [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (9): registry-1.docker.io
Aug 09 13:24:17 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (10): registry-1.docker.io
Aug 09 13:24:17 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (6): dseasb33srnrn.cloudfront.net
Aug 09 13:24:17 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (7): dseasb33srnrn.cloudfront.net
Aug 09 13:24:17 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (8): dseasb33srnrn.cloudfront.net
Aug 09 13:24:17 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (9): dseasb33srnrn.cloudfront.net
Aug 09 13:24:17 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (10): dseasb33srnrn.cloudfront.net
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-64224) [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-64224) Traceback (most recent call last):
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-64224)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 243, in _fetch
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-64224)     file_handle.write(chunk)
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-64224) IOError: [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-86112) [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-86112) Traceback (most recent call last):
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-86112)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 243, in _fetch
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-86112)     file_handle.write(chunk)
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-86112) IOError: [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816) [Errno 28] No space left on device: u'/var/cache/pulp/reserved_resource_worker-0@dev/52cdb2fd-8e19-4b33-bab8-8cb75715a737/sha256:08a7a0bb61228a03e099486f3ffbb4e617a7b61d69aa7e8e803d925e1c5ebaaa'
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816) Traceback (most recent call last):
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 229, in _fetch
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816)     file_handle = request.initialize_file_handle()
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816)   File "/usr/lib/python2.7/site-packages/nectar/request.py", line 46, in initialize_file_handle
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816)     self._file_handle = open(self.destination, 'wb')  # cache the handle
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816) IOError: [Errno 28] No space left on device: u'/var/cache/pulp/reserved_resource_worker-0@dev/52cdb2fd-8e19-4b33-bab8-8cb75715a737/sha256:08a7a0bb61228a03e099486f3ffbb4e617a7b61d69aa7e8e803d925e1c5ebaaa'
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656) [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656) Traceback (most recent call last):
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 243, in _fetch
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656)     file_handle.write(chunk)
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656) IOError: [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-71520) [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-71520) Traceback (most recent call last):
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-71520)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 243, in _fetch
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-71520)     file_handle.write(chunk)
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-71520) IOError: [Errno 28] No space left on device
Aug 09 13:24:18 dev pulp[19105]: pulp.server.async.tasks:INFO: Task failed : [52cdb2fd-8e19-4b33-bab8-8cb75715a737] : Content import of /var/cache/pulp/reserved_resource_worker-0@dev/52cdb2fd-8e19-4b33-bab8-8cb75715a737/sha256:08a7a0bb61228a03e099486f3ffbb4e617a7b61d69aa7e8e803d925e1c5ebaaa failed - must be an existing file.
Aug 09 13:24:18 dev pulp[19015]: celery.worker.job:INFO: Task pulp.server.managers.repo.sync.sync[52cdb2fd-8e19-4b33-bab8-8cb75715a737] raised expected: PulpCodedException()

Related issues

Actions

Copy link

Updated by amacdona@redhat.com over 7 years ago

Triaged changed from No to Yes

Actions

Copy link

Updated by mhrivnak over 7 years ago

Per discussion in IRC, if a change is required in platform, it should go on 2.8-dev.

Actions

Copy link

Updated by dkliban@redhat.com over 7 years ago

Status changed from NEW to ASSIGNED
Assignee set to dkliban@redhat.com

Actions

Copy link

Updated by mhrivnak over 7 years ago

Related to Issue #1823: RPMs partially downloaded added

Actions

Copy link

Updated by dkliban@redhat.com over 7 years ago

Sprint/Milestone set to 24

Actions

Copy link

Updated by mhrivnak over 7 years ago

Sprint/Milestone changed from 24 to 25

Actions

Copy link

Updated by dkliban@redhat.com over 7 years ago

The sync of a docker repository consists of multiple steps. One of the steps is responsible for downloading files into the working directory. This work is done by nectar. Nectar uses threads to perform the downloads in parallel. After all the files are downloaded, the next step creates each unit in the database and copies the unit's file from the working directory to the content directory.

To perform the download, nectar creates an empty file and then writes to it the content it is retrieving from upstream repository. The "No space left on device" exception occurs during the download step of the sync. When nectar encounters any exception, it calls a failure handler. The failure handler increments the number of download failures and the sync process continues running. After all the downloads are finished, the next steps copies the empty files into the content directory in Pulp. The task is marked as successfully completed even though the content is not usable.

Since the out of space exception should stop the sync process completely, it is necessary to raise an exception that will propagate to the main thread running the task. The current nectar design does not allow for such propagation. However, it is possible to check the exception in the failure handler that that nectar calls on download failure. If the failure handler sees the "No space left on device" exception, it will kill the process. As a result, the worker will restart and the task status for the sync will be updated with a failed status and a message stating the worker exited unexpectedly. The logs will contain "No space left on device" exception.

Actions

Copy link