Issue #9645
Updated by fao89 over 2 years ago
**Ticket moved to GitHub**: "pulp/pulpcore/2072":https://github.com/pulp/pulpcore/issues/2072 ---- Clone from https://bugzilla.redhat.com/show_bug.cgi?id=2033847 Description of problem: Pulp raises memory error when exporting repositories with large contents. hammer content-export incremental version --content-view=rhel --version=2.0 --organization="Default Organization" --lifecycle-environment Library --chunk-size-gb 2 [....................................................................................................................................................................] [100%] Error: Pulp task error undefined method `first' for nil:NilClas ~~~ python # Traceback in /var/log/messages pulpcore-worker-7: pulp [029c96b2-78a8-48ca-8605-9d59872f1be0]: pulpcore.tasking.pulpcore_worker:INFO: Task 508915bc-d4f5-49e3-aedb-d838bb0b2d14 failed () <========= Raised error without message which is normally MemoryError ... pulpcore-worker-7: pulp [029c96b2-78a8-48ca-8605-9d59872f1be0]: pulpcore.tasking.pulpcore_worker:INFO: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 317, in _perform_task pulpcore-worker-7: result = func(*args, **kwargs) pulpcore-worker-7: File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 204, in pulp_export pulpcore-worker-7: _do_export(pulp_exporter, tar, the_export) pulpcore-worker-7: File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 313, in _do_export pulpcore-worker-7: export_content(the_export, version) pulpcore-worker-7: File "/usr/lib/python3.6/site-packages/pulpcore/app/importexport.py", line 134, in export_content pulpcore-worker-7: _write_export(export.tarfile, resource, dest_dir) pulpcore-worker-7: File "/usr/lib/python3.6/site-packages/pulpcore/app/importexport.py", line 41, in _write_export pulpcore-worker-7: data = dataset.json.encode("utf8") pulpcore-worker-7: File "/usr/lib/python3.6/site-packages/tablib/formats/__init__.py", line 62, in __get__ pulpcore-worker-7: return self._format.export_set(obj, **kwargs) pulpcore-worker-7: File "/usr/lib/python3.6/site-packages/tablib/formats/_json.py", line 27, in export_set pulpcore-worker-7: dataset.dict, default=serialize_objects_handler, ensure_ascii=False pulpcore-worker-7: File "/usr/lib64/python3.6/json/__init__.py", line 238, in dumps pulpcore-worker-7: **kw).encode(obj) pulpcore-worker-7: File "/usr/lib64/python3.6/json/encoder.py", line 202, in encode pulpcore-worker-7: return ''.join(chunks) ~~~ ~~~ python # Add a logging to Pulp so that it will print the Exception type; pulpcore-worker-6: pulp [da8e4925-df12-4fae-851e-f6352b5ae425]: pulpcore.tasking.pulpcore_worker:INFO: Task b246ce3c-a805-4a0f-8393-2fbf7cdd73a7 failed () pulpcore-worker-6: pulp [da8e4925-df12-4fae-851e-f6352b5ae425]: pulpcore.tasking.pulpcore_worker:INFO: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 317, in _perform_task pulpcore-worker-6: result = func(*args, **kwargs) pulpcore-worker-6: File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 204, in pulp_export pulpcore-worker-6: _do_export(pulp_exporter, tar, the_export) pulpcore-worker-6: File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 313, in _do_export pulpcore-worker-6: export_content(the_export, version) pulpcore-worker-6: File "/usr/lib/python3.6/site-packages/pulpcore/app/importexport.py", line 134, in export_content pulpcore-worker-6: _write_export(export.tarfile, resource, dest_dir) pulpcore-worker-6: File "/usr/lib/python3.6/site-packages/pulpcore/app/importexport.py", line 41, in _write_export pulpcore-worker-6: data = dataset.json.encode("utf8") pulpcore-worker-6: pulp [da8e4925-df12-4fae-851e-f6352b5ae425]: pulpcore.tasking.pulpcore_worker:INFO: <class 'MemoryError'> <======= ~~~ ~~~ text # While the task is running, we can see that the memory consumption is very high total used free shared buff/cache available Mem: 33010716 19796992 221744 579536 12991980 12230452 Swap: 12058620 3917188 8141432 total used free shared buff/cache available Mem: 33010716 20831688 331752 579536 11847276 11196116 Swap: 12058620 3916932 8141688 total used free shared buff/cache available Mem: 33010716 21926716 224460 579540 10859540 10100732 Swap: 12058620 3916932 8141688 total used free shared buff/cache available Mem: 33010716 23173152 219596 579540 9617968 8854304 Swap: 12058620 3916932 8141688 total used free shared buff/cache available Mem: 33010716 24317448 238176 579660 8455092 7709900 Swap: 12058620 3916932 8141688 total used free shared buff/cache available Mem: 33010716 25737804 230592 579612 7042320 6289292 Swap: 12058620 3916932 8141688 total used free shared buff/cache available Mem: 33010716 27344320 226584 579612 5439812 4683004 Swap: 12058620 3916932 8141688 total used free shared buff/cache available Mem: 33010716 30193200 234504 579612 2583012 1834164 <=============== Observed free memory dropped to 1.8GB from 20GB Swap: 12058620 3916932 8141688 total used free shared buff/cache available Mem: 33010716 24422544 7876732 334776 711440 7886420 Swap: 12058620 8729032 3329588 <===================== Free swap dropped 3G total used free shared buff/cache available Mem: 33010716 23213200 9030436 336976 767080 9067412 Swap: 12058620 8674504 3384116 total used free shared buff/cache available Mem: 33010716 22871236 9357400 339484 782080 9400532 Swap: 12058620 8637896 3420724 total used free shared buff/cache available Mem: 33010716 7216620 25034040 339872 760056 25053004 <======== Memory freed Swap: 12058620 8431560 3627060 total used free shared buff/cache available Mem: 33010716 7234448 25014092 339928 762176 25033868 Swap: 12058620 8414408 3644212 ~~~ ~~~ text # As we can see that Pulp worker consumed 22.3G of RAM when writing the resource file top - 00:31:24 up 1 day, 12:52, 6 users, load average: 60.50, 26.92, 13.02 Tasks: 373 total, 1 running, 372 sleeping, 0 stopped, 0 zombie KiB Mem : 99.8/33010716 [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||] KiB Swap: 52.1/12058620 [|||||||||||||||||||||||||||||||||||||||||||||||||||| ] PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 8442 pulp 20 0 23.9g 22.3g 636 D 17.2 70.7 13:38.63 pulpcore-worker <============= And it consumes 10G of RAM when writing a smaller file (I think) KiB Mem : 56.3/33010716 [|||||||||||||||||||||||||||||||||||||||||||||||||||||||| ] KiB Swap: 34.2/12058620 [|||||||||||||||||||||||||||||||||| ] PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 8442 pulp 20 0 10.4g 10.0g 856 R 99.7 31.7 17:25.61 pulpcore-worker <============= ~~~ ~~~ text Steps to Reproduce: 1. Enable and sync the following 2 repos Red_Hat_Enterprise_Linux_6_Server_-_Extended_Life_Cycle_Support_RPMs_x86_64 Red_Hat_Enterprise_Linux_7_Server_RPMs_x86_64_7Server 2. Create a content view and add the above repos 3. Publish the content view 4. Export the content view hammer content-export incremental version --content-view=rhel --version=1.0 --organization="Default Organization" --lifecycle-environment Library --chunk-size-gb 2 Actual results: Error: Pulp task error undefined method `first' for nil:NilClass Expected results: Success ~~~
.