Actions
Issue #9645
closed
System runs out of memory when exporting repository with many packages, such as rhel 7 server rpms repo
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Master
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 112
Quarter:
Description
Ticket moved to GitHub: "pulp/pulpcore/2072":https://github.com/pulp/pulpcore/issues/2072
Clone from https://bugzilla.redhat.com/show_bug.cgi?id=2033847
Description of problem:
Pulp raises memory error when exporting repositories with large contents.
hammer content-export incremental version --content-view=rhel --version=2.0 --organization="Default Organization" --lifecycle-environment Library --chunk-size-gb 2 [....................................................................................................................................................................] [100%] Error: Pulp task error undefined method `first' for nil:NilClas
# Traceback in /var/log/messages
pulpcore-worker-7: pulp [029c96b2-78a8-48ca-8605-9d59872f1be0]: pulpcore.tasking.pulpcore_worker:INFO: Task 508915bc-d4f5-49e3-aedb-d838bb0b2d14 failed () <========= Raised error without message which is normally MemoryError
...
pulpcore-worker-7: pulp [029c96b2-78a8-48ca-8605-9d59872f1be0]: pulpcore.tasking.pulpcore_worker:INFO: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 317, in _perform_task
pulpcore-worker-7: result = func(*args, **kwargs)
pulpcore-worker-7: File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 204, in pulp_export
pulpcore-worker-7: _do_export(pulp_exporter, tar, the_export)
pulpcore-worker-7: File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 313, in _do_export
pulpcore-worker-7: export_content(the_export, version)
pulpcore-worker-7: File "/usr/lib/python3.6/site-packages/pulpcore/app/importexport.py", line 134, in export_content
pulpcore-worker-7: _write_export(export.tarfile, resource, dest_dir)
pulpcore-worker-7: File "/usr/lib/python3.6/site-packages/pulpcore/app/importexport.py", line 41, in _write_export
pulpcore-worker-7: data = dataset.json.encode("utf8")
pulpcore-worker-7: File "/usr/lib/python3.6/site-packages/tablib/formats/__init__.py", line 62, in __get__
pulpcore-worker-7: return self._format.export_set(obj, **kwargs)
pulpcore-worker-7: File "/usr/lib/python3.6/site-packages/tablib/formats/_json.py", line 27, in export_set
pulpcore-worker-7: dataset.dict, default=serialize_objects_handler, ensure_ascii=False
pulpcore-worker-7: File "/usr/lib64/python3.6/json/__init__.py", line 238, in dumps
pulpcore-worker-7: **kw).encode(obj)
pulpcore-worker-7: File "/usr/lib64/python3.6/json/encoder.py", line 202, in encode
pulpcore-worker-7: return ''.join(chunks)
# Add a logging to Pulp so that it will print the Exception type;
pulpcore-worker-6: pulp [da8e4925-df12-4fae-851e-f6352b5ae425]: pulpcore.tasking.pulpcore_worker:INFO: Task b246ce3c-a805-4a0f-8393-2fbf7cdd73a7 failed ()
pulpcore-worker-6: pulp [da8e4925-df12-4fae-851e-f6352b5ae425]: pulpcore.tasking.pulpcore_worker:INFO: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 317, in _perform_task
pulpcore-worker-6: result = func(*args, **kwargs)
pulpcore-worker-6: File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 204, in pulp_export
pulpcore-worker-6: _do_export(pulp_exporter, tar, the_export)
pulpcore-worker-6: File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 313, in _do_export
pulpcore-worker-6: export_content(the_export, version)
pulpcore-worker-6: File "/usr/lib/python3.6/site-packages/pulpcore/app/importexport.py", line 134, in export_content
pulpcore-worker-6: _write_export(export.tarfile, resource, dest_dir)
pulpcore-worker-6: File "/usr/lib/python3.6/site-packages/pulpcore/app/importexport.py", line 41, in _write_export
pulpcore-worker-6: data = dataset.json.encode("utf8")
pulpcore-worker-6: pulp [da8e4925-df12-4fae-851e-f6352b5ae425]: pulpcore.tasking.pulpcore_worker:INFO: <class 'MemoryError'> <=======
# While the task is running, we can see that the memory consumption is very high
total used free shared buff/cache available
Mem: 33010716 19796992 221744 579536 12991980 12230452
Swap: 12058620 3917188 8141432
total used free shared buff/cache available
Mem: 33010716 20831688 331752 579536 11847276 11196116
Swap: 12058620 3916932 8141688
total used free shared buff/cache available
Mem: 33010716 21926716 224460 579540 10859540 10100732
Swap: 12058620 3916932 8141688
total used free shared buff/cache available
Mem: 33010716 23173152 219596 579540 9617968 8854304
Swap: 12058620 3916932 8141688
total used free shared buff/cache available
Mem: 33010716 24317448 238176 579660 8455092 7709900
Swap: 12058620 3916932 8141688
total used free shared buff/cache available
Mem: 33010716 25737804 230592 579612 7042320 6289292
Swap: 12058620 3916932 8141688
total used free shared buff/cache available
Mem: 33010716 27344320 226584 579612 5439812 4683004
Swap: 12058620 3916932 8141688
total used free shared buff/cache available
Mem: 33010716 30193200 234504 579612 2583012 1834164 <=============== Observed free memory dropped to 1.8GB from 20GB
Swap: 12058620 3916932 8141688
total used free shared buff/cache available
Mem: 33010716 24422544 7876732 334776 711440 7886420
Swap: 12058620 8729032 3329588 <===================== Free swap dropped 3G
total used free shared buff/cache available
Mem: 33010716 23213200 9030436 336976 767080 9067412
Swap: 12058620 8674504 3384116
total used free shared buff/cache available
Mem: 33010716 22871236 9357400 339484 782080 9400532
Swap: 12058620 8637896 3420724
total used free shared buff/cache available
Mem: 33010716 7216620 25034040 339872 760056 25053004 <======== Memory freed
Swap: 12058620 8431560 3627060
total used free shared buff/cache available
Mem: 33010716 7234448 25014092 339928 762176 25033868
Swap: 12058620 8414408 3644212
# As we can see that Pulp worker consumed 22.3G of RAM when writing the resource file
top - 00:31:24 up 1 day, 12:52, 6 users, load average: 60.50, 26.92, 13.02
Tasks: 373 total, 1 running, 372 sleeping, 0 stopped, 0 zombie
KiB Mem : 99.8/33010716 [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||]
KiB Swap: 52.1/12058620 [|||||||||||||||||||||||||||||||||||||||||||||||||||| ]
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
8442 pulp 20 0 23.9g 22.3g 636 D 17.2 70.7 13:38.63 pulpcore-worker <=============
And it consumes 10G of RAM when writing a smaller file (I think)
KiB Mem : 56.3/33010716 [|||||||||||||||||||||||||||||||||||||||||||||||||||||||| ]
KiB Swap: 34.2/12058620 [|||||||||||||||||||||||||||||||||| ]
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
8442 pulp 20 0 10.4g 10.0g 856 R 99.7 31.7 17:25.61 pulpcore-worker <=============
Steps to Reproduce:
1. Enable and sync the following 2 repos
Red_Hat_Enterprise_Linux_6_Server_-_Extended_Life_Cycle_Support_RPMs_x86_64
Red_Hat_Enterprise_Linux_7_Server_RPMs_x86_64_7Server
2. Create a content view and add the above repos
3. Publish the content view
4. Export the content view
hammer content-export incremental version --content-view=rhel --version=1.0 --organization="Default Organization" --lifecycle-environment Library --chunk-size-gb 2
Actual results:
Error: Pulp task error
undefined method `first' for nil:NilClass
Expected results:
Success
Updated by pulpbot almost 3 years ago
- Status changed from NEW to POST
Updated by dkliban@redhat.com almost 3 years ago
- Assignee set to hyu
- Triaged changed from No to Yes
- Sprint set to Sprint 111
Updated by Johndavidson9 over 2 years ago
I have the same issue. Any tips for solution?
Updated by fao89 over 2 years ago
- Description updated (diff)
- Status changed from POST to CLOSED - DUPLICATE
Actions
.