Project

Profile

Help

Actions
Send by e-mail Copy link

Issue #9645

closed

System runs out of memory when exporting repository with many packages, such as rhel 7 server rpms repo

Added by hyu about 3 years ago. Updated almost 3 years ago.

Status:
CLOSED - DUPLICATE
Priority:
Normal
Assignee:
hyu
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Master
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 112
Quarter:

Description

Ticket moved to GitHub: "pulp/pulpcore/2072":https://github.com/pulp/pulpcore/issues/2072

Clone from https://bugzilla.redhat.com/show_bug.cgi?id=2033847

Description of problem:

Pulp raises memory error when exporting repositories with large contents.

hammer content-export incremental version --content-view=rhel --version=2.0 --organization="Default Organization" --lifecycle-environment Library --chunk-size-gb 2 [....................................................................................................................................................................] [100%] Error: Pulp task error undefined method `first' for nil:NilClas

# Traceback in /var/log/messages
pulpcore-worker-7: pulp [029c96b2-78a8-48ca-8605-9d59872f1be0]: pulpcore.tasking.pulpcore_worker:INFO: Task 508915bc-d4f5-49e3-aedb-d838bb0b2d14 failed ()  <========= Raised error without message which is normally MemoryError
...
pulpcore-worker-7: pulp [029c96b2-78a8-48ca-8605-9d59872f1be0]: pulpcore.tasking.pulpcore_worker:INFO:   File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 317, in _perform_task
pulpcore-worker-7: result = func(*args, **kwargs)
pulpcore-worker-7: File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 204, in pulp_export
pulpcore-worker-7: _do_export(pulp_exporter, tar, the_export)
pulpcore-worker-7: File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 313, in _do_export
pulpcore-worker-7: export_content(the_export, version)
pulpcore-worker-7: File "/usr/lib/python3.6/site-packages/pulpcore/app/importexport.py", line 134, in export_content
pulpcore-worker-7: _write_export(export.tarfile, resource, dest_dir)
pulpcore-worker-7: File "/usr/lib/python3.6/site-packages/pulpcore/app/importexport.py", line 41, in _write_export
pulpcore-worker-7: data = dataset.json.encode("utf8")
pulpcore-worker-7: File "/usr/lib/python3.6/site-packages/tablib/formats/__init__.py", line 62, in __get__
pulpcore-worker-7: return self._format.export_set(obj, **kwargs)
pulpcore-worker-7: File "/usr/lib/python3.6/site-packages/tablib/formats/_json.py", line 27, in export_set
pulpcore-worker-7: dataset.dict, default=serialize_objects_handler, ensure_ascii=False
pulpcore-worker-7: File "/usr/lib64/python3.6/json/__init__.py", line 238, in dumps
pulpcore-worker-7: **kw).encode(obj)
pulpcore-worker-7: File "/usr/lib64/python3.6/json/encoder.py", line 202, in encode
pulpcore-worker-7: return ''.join(chunks)

# Add a logging to Pulp so that it will print the Exception type;
pulpcore-worker-6: pulp [da8e4925-df12-4fae-851e-f6352b5ae425]: pulpcore.tasking.pulpcore_worker:INFO: Task b246ce3c-a805-4a0f-8393-2fbf7cdd73a7 failed ()
pulpcore-worker-6: pulp [da8e4925-df12-4fae-851e-f6352b5ae425]: pulpcore.tasking.pulpcore_worker:INFO:   File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 317, in _perform_task
pulpcore-worker-6: result = func(*args, **kwargs)
pulpcore-worker-6: File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 204, in pulp_export
pulpcore-worker-6: _do_export(pulp_exporter, tar, the_export)
pulpcore-worker-6: File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/export.py", line 313, in _do_export
pulpcore-worker-6: export_content(the_export, version)
pulpcore-worker-6: File "/usr/lib/python3.6/site-packages/pulpcore/app/importexport.py", line 134, in export_content
pulpcore-worker-6: _write_export(export.tarfile, resource, dest_dir)
pulpcore-worker-6: File "/usr/lib/python3.6/site-packages/pulpcore/app/importexport.py", line 41, in _write_export
pulpcore-worker-6: data = dataset.json.encode("utf8")
pulpcore-worker-6: pulp [da8e4925-df12-4fae-851e-f6352b5ae425]: pulpcore.tasking.pulpcore_worker:INFO: <class 'MemoryError'>  <=======

# While the task is running, we can see that the memory consumption is very high

              total        used        free      shared  buff/cache   available
Mem:       33010716    19796992      221744      579536    12991980    12230452
Swap:      12058620     3917188     8141432

              total        used        free      shared  buff/cache   available
Mem:       33010716    20831688      331752      579536    11847276    11196116
Swap:      12058620     3916932     8141688

              total        used        free      shared  buff/cache   available
Mem:       33010716    21926716      224460      579540    10859540    10100732
Swap:      12058620     3916932     8141688

              total        used        free      shared  buff/cache   available
Mem:       33010716    23173152      219596      579540     9617968     8854304
Swap:      12058620     3916932     8141688

              total        used        free      shared  buff/cache   available
Mem:       33010716    24317448      238176      579660     8455092     7709900
Swap:      12058620     3916932     8141688

              total        used        free      shared  buff/cache   available
Mem:       33010716    25737804      230592      579612     7042320     6289292
Swap:      12058620     3916932     8141688

              total        used        free      shared  buff/cache   available
Mem:       33010716    27344320      226584      579612     5439812     4683004
Swap:      12058620     3916932     8141688

              total        used        free      shared  buff/cache   available
Mem:       33010716    30193200      234504      579612     2583012     1834164    <=============== Observed free memory dropped to 1.8GB from 20GB
Swap:      12058620     3916932     8141688

              total        used        free      shared  buff/cache   available
Mem:       33010716    24422544     7876732      334776      711440     7886420
Swap:      12058620     8729032     3329588             <===================== Free swap dropped 3G

              total        used        free      shared  buff/cache   available
Mem:       33010716    23213200     9030436      336976      767080     9067412
Swap:      12058620     8674504     3384116

              total        used        free      shared  buff/cache   available
Mem:       33010716    22871236     9357400      339484      782080     9400532
Swap:      12058620     8637896     3420724

              total        used        free      shared  buff/cache   available
Mem:       33010716     7216620    25034040      339872      760056    25053004   <======== Memory freed
Swap:      12058620     8431560     3627060

              total        used        free      shared  buff/cache   available
Mem:       33010716     7234448    25014092      339928      762176    25033868
Swap:      12058620     8414408     3644212

# As we can see that Pulp worker consumed 22.3G of RAM when writing the resource file

top - 00:31:24 up 1 day, 12:52,  6 users,  load average: 60.50, 26.92, 13.02
Tasks: 373 total,   1 running, 372 sleeping,   0 stopped,   0 zombie
KiB Mem : 99.8/33010716 [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||]
KiB Swap: 52.1/12058620 [||||||||||||||||||||||||||||||||||||||||||||||||||||                                                ]

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                            
 8442 pulp      20   0   23.9g  22.3g    636 D  17.2 70.7  13:38.63 pulpcore-worker  <=============


And it consumes 10G of RAM when writing a smaller file (I think)
KiB Mem : 56.3/33010716 [||||||||||||||||||||||||||||||||||||||||||||||||||||||||                                            ]
KiB Swap: 34.2/12058620 [||||||||||||||||||||||||||||||||||                                                                  ]

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                                                  
 8442 pulp      20   0   10.4g  10.0g    856 R  99.7 31.7  17:25.61 pulpcore-worker  <=============

Steps to Reproduce:
1. Enable and sync the following 2 repos

Red_Hat_Enterprise_Linux_6_Server_-_Extended_Life_Cycle_Support_RPMs_x86_64
Red_Hat_Enterprise_Linux_7_Server_RPMs_x86_64_7Server

2. Create a content view and add the above repos
3. Publish the content view
4. Export the content view

hammer content-export incremental version --content-view=rhel --version=1.0 --organization="Default Organization" --lifecycle-environment Library --chunk-size-gb 2


Actual results:
Error: Pulp task error
undefined method `first' for nil:NilClass

Expected results:
Success
Actions
Copy link
 #1

Updated by pulpbot about 3 years ago

  • Status changed from NEW to POST
Actions
Copy link
 #2

Updated by dkliban@redhat.com almost 3 years ago

  • Assignee set to hyu
  • Triaged changed from No to Yes
  • Sprint set to Sprint 111
Actions
Copy link
 #3

Updated by Johndavidson9 almost 3 years ago

I have the same issue. Any tips for solution?

https://get-9apps.com https://cartoonhd.onl/index/

Actions
Copy link
 #4

Updated by rchan almost 3 years ago

  • Sprint changed from Sprint 111 to Sprint 112
Actions
Copy link
 #5

Updated by fao89 almost 3 years ago

  • Assignee deleted (hyu)
Actions
Copy link
 #6

Updated by fao89 almost 3 years ago

  • Assignee set to hyu
Actions
Copy link
 #7

Updated by fao89 almost 3 years ago

  • Description updated (diff)
  • Status changed from POST to CLOSED - DUPLICATE
Actions
Send by e-mail Copy link

Also available in: Atom PDF