Project

Profile

Help

Issue #2142

Units created with 0-byte files when sync runs out of disk space

Added by mhrivnak about 4 years ago. Updated over 1 year ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Start date:
Due date:
Estimated time:
Severity:
3. High
Version - Docker:
master
Platform Release:
2.9.3
Target Release - Docker:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 7
Quarter:

Description

I created a very small filesystem, mounted it to /var/cache/pulp/, and made sure the apache user had permission to write to it. I then tried to do a docker sync of a centos repo from docker hub.

$ dd if=/dev/zero of=foofs bs=1024 count=1000
$ mke2fs foofs 
$ sudo mount -o loop ./foofs /var/cache/pulp/
$ prestart
$ pulp-admin docker repo create --repo-id=centos --feed=https://registry-1.docker.io/ --upstream-name=centos
$ pulp-admin docker repo sync run --repo-id=centos

I saw the task fail with an error message in the terminal and lots of errors in the log. But after it was done, there were 11 blob units in the database, 10 of which were associated to the repo. The files were in /var/lib/pulp/content/ but most had a length of 0 bytes.

$ find /var/lib/pulp/content/units/docker_blob/ -type f | xargs ls -lh
-rw-r--r-- 1 apache apache   32 Aug  9 13:24 /var/lib/pulp/content/units/docker_blob/12/86843087e8774c31f670616e3c7e693a725d5615fe02d1043030e936d0e4f9/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
-rw-r--r-- 1 apache apache    0 Aug  9 13:24 /var/lib/pulp/content/units/docker_blob/20/5ecdb17746b9c6e24e9c7c7ab6751d0315a78b265edb05df2924009da9f90a/sha256:f6e4e6bc8376818ff75e2818d38b9451b87e3348e1d8256bef64ecd9000e7305
-rw-r--r-- 1 apache apache    0 Aug  9 13:24 /var/lib/pulp/content/units/docker_blob/52/c7b2a0896b367f1bcfa2e736f32fed0ccf459a7ad41ca3bf1aa23eb31f516b/sha256:3eaa9b70c44a88d9d4c2bcac5f8af1f7f63e12eae72fd21a029a54ad95a47f50
-rw-r--r-- 1 apache apache    0 Aug  9 13:24 /var/lib/pulp/content/units/docker_blob/5d/02bff11fee510f89fa10a43ef3b219f929ac6596477b86dc44fd03a2970512/sha256:f19ed9d73b3ef2c54aef640d150f40de66926016b485da5d086f9666679ce5a0
-rw-r--r-- 1 apache apache    0 Aug  9 13:24 /var/lib/pulp/content/units/docker_blob/62/6628be11abffb60b82481339a748fe11752b36aad0d03c7e49166ba8a05ed3/sha256:d129ecdb0249b2540429964ecf04a8f426199d27bfec517ecd9f19b2c7ae21ba
-rw-r--r-- 1 apache apache    0 Aug  9 13:24 /var/lib/pulp/content/units/docker_blob/75/9230be77c479edac688ee64f2481561d2edcdd68a4b0307c8b79945778e993/sha256:5237cae4d62996a5c7e0dec4eec6a221b5cad630c4d7de33db57004d6cd2501b
-rw-r--r-- 1 apache apache    0 Aug  9 13:24 /var/lib/pulp/content/units/docker_blob/87/66436ca847c58b74b7e4b03fcb65d20e061ca4312ed92a2c6e5c63ee868612/sha256:2fbe3a450f5d5be9ac8ed64d7a5e2434adc4156ab2530bcf74e32376de0aa371
-rw-r--r-- 1 apache apache    0 Aug  9 13:24 /var/lib/pulp/content/units/docker_blob/d5/de1456ada563c480f721bd94951b569df51e2e31e9185f59fcda7ccd60b850/sha256:ebd69957142bd0a24ecd4d5293bcfc18c2732741afb2e2401c60d13c3e80798c
-rw-r--r-- 1 apache apache 804K Aug  9 13:24 /var/lib/pulp/content/units/docker_blob/db/ac044223053fc10dff79e714751d4a94b6a01e80ee0ebb6523f2302ee091de/sha256:b6d7b2ebc0a7f5c0e87724be3904839d65562ec69909232ce1b7524257eb41af
-rw-r--r-- 1 apache apache    0 Aug  9 13:24 /var/lib/pulp/content/units/docker_blob/fa/5119032836af90a70da36533618d5742d382938bfc24acec2f30d466a8f3c4/sha256:3d8673bd162aafaf70e94029801a409fa3210363707ea1e7956b6f8ff9351cc8

This was the terminal output:

$ pulp-admin docker repo sync run --repo-id=centos
+----------------------------------------------------------------------+
                   Synchronizing Repository [centos]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading manifests
[-]
... completed

Copying units already in pulp
[-]
... completed

Copying units already in pulp
[-]
... completed

Downloading remote files
[====                                              ] 9%
1 of 11 items
... failed

Task Failed

Content import of
/var/cache/pulp/reserved_resource_worker-0@dev/52cdb2fd-8e19-4b33-bab8-8cb75715a
737/sha256:08a7a0bb61228a03e099486f3ffbb4e617a7b61d69aa7e8e803d925e1c5ebaaa
failed - must be an existing file.

and the log output;

Aug 09 13:23:22 dev pulp[18984]: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Aug 09 13:23:22 dev pulp[19004]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[95175ded-1bf6-4c04-a390-6e46175bf2ce]
Aug 09 13:23:22 dev pulp[19015]: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[5533191e-d489-4b90-bdab-10b160920bd6]
Aug 09 13:23:22 dev pulp[19015]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[8f566b88-baa1-4d29-b2ce-e0fdbde32fe4]
Aug 09 13:23:22 dev pulp[19004]: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[95175ded-1bf6-4c04-a390-6e46175bf2ce] succeeded in 0.0363659770228s: None
Aug 09 13:23:22 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): index.docker.io
Aug 09 13:23:22 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): auth.docker.io
Aug 09 13:23:22 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (2): index.docker.io
Aug 09 13:23:22 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (3): index.docker.io
Aug 09 13:23:22 dev pulp[19105]: pulp.server.async.tasks:INFO: Task failed : [5533191e-d489-4b90-bdab-10b160920bd6] : Could not fetch repository centos from registry https://index.docker.io/ - Unauthorized or Not Found
Aug 09 13:23:22 dev pulp[19015]: celery.worker.job:INFO: Task pulp.server.managers.repo.sync.sync[5533191e-d489-4b90-bdab-10b160920bd6] raised expected: PulpCodedException()
Aug 09 13:23:22 dev pulp[19015]: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[8f566b88-baa1-4d29-b2ce-e0fdbde32fe4] succeeded in 0.00693614489865s: None
Aug 09 13:24:08 dev pulp[18983]: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Aug 09 13:24:08 dev pulp[19004]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[d3c84ccd-8d9d-42d4-b500-b37c724722a9]
Aug 09 13:24:08 dev pulp[19015]: celery.worker.strategy:INFO: Received task: pulp.server.tasks.repository.distributor_update[0d6c6f10-d84b-4e6a-bba3-2b348f6623ce]
Aug 09 13:24:08 dev pulp[19015]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[9761fa09-1a1f-4a96-8bad-251904a23425]
Aug 09 13:24:08 dev pulp[19004]: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[d3c84ccd-8d9d-42d4-b500-b37c724722a9] succeeded in 0.0386713229818s: None
Aug 09 13:24:08 dev pulp[19015]: celery.worker.job:INFO: Task pulp.server.tasks.repository.distributor_update[0d6c6f10-d84b-4e6a-bba3-2b348f6623ce] succeeded in 0.074527513003s: <pulp.server.async.tasks.TaskResult object at 0x7effcbba6510>
Aug 09 13:24:08 dev pulp[19015]: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[9761fa09-1a1f-4a96-8bad-251904a23425] succeeded in 0.00498440302908s: None
Aug 09 13:24:11 dev pulp[19004]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[fe4c8d10-534d-4263-a369-9ad8ffad43ba]
Aug 09 13:24:11 dev pulp[19015]: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[52cdb2fd-8e19-4b33-bab8-8cb75715a737]
Aug 09 13:24:11 dev pulp[19015]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[f8db0c7b-13f9-42ab-9bf2-d5c0eec19737]
Aug 09 13:24:11 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): registry-1.docker.io
Aug 09 13:24:11 dev pulp[19004]: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[fe4c8d10-534d-4263-a369-9ad8ffad43ba] succeeded in 0.0350315800169s: None
Aug 09 13:24:11 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): auth.docker.io
Aug 09 13:24:11 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (2): registry-1.docker.io
Aug 09 13:24:12 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (3): registry-1.docker.io
Aug 09 13:24:16 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): registry-1.docker.io
Aug 09 13:24:16 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (2): registry-1.docker.io
Aug 09 13:24:16 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (3): registry-1.docker.io
Aug 09 13:24:16 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (4): registry-1.docker.io
Aug 09 13:24:16 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (5): registry-1.docker.io
Aug 09 13:24:16 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (6): registry-1.docker.io
Aug 09 13:24:16 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): dseasb33srnrn.cloudfront.net
Aug 09 13:24:16 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816) [Errno 28] No space left on device
Aug 09 13:24:16 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816) Traceback (most recent call last):
Aug 09 13:24:16 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 243, in _fetch
Aug 09 13:24:16 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816)     file_handle.write(chunk)
Aug 09 13:24:16 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816) IOError: [Errno 28] No space left on device
Aug 09 13:24:16 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (7): registry-1.docker.io
Aug 09 13:24:16 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (2): dseasb33srnrn.cloudfront.net
Aug 09 13:24:16 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (3): dseasb33srnrn.cloudfront.net
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-71520) [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-71520) Traceback (most recent call last):
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-71520)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 243, in _fetch
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-71520)     file_handle.write(chunk)
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-71520) IOError: [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (4): dseasb33srnrn.cloudfront.net
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656) [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656) Traceback (most recent call last):
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 243, in _fetch
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656)     file_handle.write(chunk)
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656) IOError: [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (5): dseasb33srnrn.cloudfront.net
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-86112) [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-86112) Traceback (most recent call last):
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-86112)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 243, in _fetch
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-86112)     file_handle.write(chunk)
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-86112) IOError: [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (8): registry-1.docker.io
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816) [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816) Traceback (most recent call last):
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 243, in _fetch
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816)     file_handle.write(chunk)
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816) IOError: [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (9): registry-1.docker.io
Aug 09 13:24:17 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (10): registry-1.docker.io
Aug 09 13:24:17 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (6): dseasb33srnrn.cloudfront.net
Aug 09 13:24:17 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (7): dseasb33srnrn.cloudfront.net
Aug 09 13:24:17 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (8): dseasb33srnrn.cloudfront.net
Aug 09 13:24:17 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (9): dseasb33srnrn.cloudfront.net
Aug 09 13:24:17 dev pulp[19105]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (10): dseasb33srnrn.cloudfront.net
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-64224) [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-64224) Traceback (most recent call last):
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-64224)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 243, in _fetch
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-64224)     file_handle.write(chunk)
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-64224) IOError: [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-86112) [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-86112) Traceback (most recent call last):
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-86112)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 243, in _fetch
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-86112)     file_handle.write(chunk)
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-86112) IOError: [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816) [Errno 28] No space left on device: u'/var/cache/pulp/reserved_resource_worker-0@dev/52cdb2fd-8e19-4b33-bab8-8cb75715a737/sha256:08a7a0bb61228a03e099486f3ffbb4e617a7b61d69aa7e8e803d925e1c5ebaaa'
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816) Traceback (most recent call last):
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 229, in _fetch
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816)     file_handle = request.initialize_file_handle()
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816)   File "/usr/lib/python2.7/site-packages/nectar/request.py", line 46, in initialize_file_handle
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816)     self._file_handle = open(self.destination, 'wb')  # cache the handle
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78816) IOError: [Errno 28] No space left on device: u'/var/cache/pulp/reserved_resource_worker-0@dev/52cdb2fd-8e19-4b33-bab8-8cb75715a737/sha256:08a7a0bb61228a03e099486f3ffbb4e617a7b61d69aa7e8e803d925e1c5ebaaa'
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656) [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656) Traceback (most recent call last):
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 243, in _fetch
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656)     file_handle.write(chunk)
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-78656) IOError: [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-71520) [Errno 28] No space left on device
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-71520) Traceback (most recent call last):
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-71520)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 243, in _fetch
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-71520)     file_handle.write(chunk)
Aug 09 13:24:17 dev pulp[19105]: nectar.downloaders.threaded:ERROR: (19105-71520) IOError: [Errno 28] No space left on device
Aug 09 13:24:18 dev pulp[19105]: pulp.server.async.tasks:INFO: Task failed : [52cdb2fd-8e19-4b33-bab8-8cb75715a737] : Content import of /var/cache/pulp/reserved_resource_worker-0@dev/52cdb2fd-8e19-4b33-bab8-8cb75715a737/sha256:08a7a0bb61228a03e099486f3ffbb4e617a7b61d69aa7e8e803d925e1c5ebaaa failed - must be an existing file.
Aug 09 13:24:18 dev pulp[19015]: celery.worker.job:INFO: Task pulp.server.managers.repo.sync.sync[52cdb2fd-8e19-4b33-bab8-8cb75715a737] raised expected: PulpCodedException()

Related issues

Related to RPM Support - Issue #1823: RPMs partially downloadedCLOSED - CURRENTRELEASE<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision b7c8bfde View on GitHub
Added by dkliban@redhat.com about 4 years ago

Adds a check for 'No space left on device' on download failures

re #2142 https://pulp.plan.io/issues/2142

Revision b7c8bfde View on GitHub
Added by dkliban@redhat.com about 4 years ago

Adds a check for 'No space left on device' on download failures

re #2142 https://pulp.plan.io/issues/2142

History

#1 Updated by amacdona@redhat.com about 4 years ago

  • Triaged changed from No to Yes

#2 Updated by mhrivnak about 4 years ago

Per discussion in IRC, if a change is required in platform, it should go on 2.8-dev.

#3 Updated by dkliban@redhat.com about 4 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dkliban@redhat.com

#4 Updated by mhrivnak about 4 years ago

  • Related to Issue #1823: RPMs partially downloaded added

#5 Updated by dkliban@redhat.com about 4 years ago

  • Sprint/Milestone set to 24

#6 Updated by mhrivnak about 4 years ago

  • Sprint/Milestone changed from 24 to 25

#7 Updated by dkliban@redhat.com about 4 years ago

The sync of a docker repository consists of multiple steps. One of the steps is responsible for downloading files into the working directory. This work is done by nectar. Nectar uses threads to perform the downloads in parallel. After all the files are downloaded, the next step creates each unit in the database and copies the unit's file from the working directory to the content directory.

To perform the download, nectar creates an empty file and then writes to it the content it is retrieving from upstream repository. The "No space left on device" exception occurs during the download step of the sync. When nectar encounters any exception, it calls a failure handler. The failure handler increments the number of download failures and the sync process continues running. After all the downloads are finished, the next steps copies the empty files into the content directory in Pulp. The task is marked as successfully completed even though the content is not usable.

Since the out of space exception should stop the sync process completely, it is necessary to raise an exception that will propagate to the main thread running the task. The current nectar design does not allow for such propagation. However, it is possible to check the exception in the failure handler that that nectar calls on download failure. If the failure handler sees the "No space left on device" exception, it will kill the process. As a result, the worker will restart and the task status for the sync will be updated with a failed status and a message stating the worker exited unexpectedly. The logs will contain "No space left on device" exception.

#8 Updated by dkliban@redhat.com about 4 years ago

  • Status changed from ASSIGNED to POST
  • Platform Release set to 2.9.3

The fix specific to docker:
https://github.com/pulp/pulp_docker/pull/174

The more general fix in platform:
https://github.com/pulp/pulp/pull/2702

#9 Updated by dkliban@redhat.com about 4 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#10 Updated by semyers about 4 years ago

  • Status changed from MODIFIED to 5

#11 Updated by semyers about 4 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE

#12 Updated by bmbouter over 2 years ago

  • Sprint set to Sprint 7

#13 Updated by bmbouter over 2 years ago

  • Sprint/Milestone deleted (25)

#14 Updated by bmbouter over 1 year ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF