Project

Profile

Help

Issue #8992

Memory usage when destroy repository; The memory leaks?

Added by wilful 3 months ago. Updated 1 day ago.

Status:
NEW
Priority:
Low
Assignee:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Platform Release:
OS:
CentOS 7
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 106
Quarter:
Q4-2021

Description

When i try to destroy one of repositories, pulp (RQ basically) are using more 3G RAM and crashed by OOM-Killer

Jun 30 06:59:16 *** kernel: Out of memory: Kill process 81458 (rq) score 493 or sacrifice child
Jun 30 06:59:16 *** kernel: Killed process 81458 (rq), UID 1002, total-vm:4434448kB, anon-rss:3936104kB, file-rss:0kB, shmem-rss:0kB

State before on my node:

# free -h
              total        used        free      shared  buff/cache   available
Mem:           7.6G        1.5G        3.6G        2.2G        2.6G        3.7G
Swap:            0B          0B          0B
  "versions": [
    {
      "component": "core",
      "version": "3.13.0"
    },
    {
      "component": "rpm",
      "version": "3.13.0"
    },
    {
      "component": "python",
      "version": "3.4.0"
    },
    {
      "component": "file",
      "version": "1.8.0"
    },
    {
      "component": "deb",
      "version": "2.13.0"
    },
    {
      "component": "container",
      "version": "2.6.0"
    }
  ],

History

#1 Updated by wilful 3 months ago

More 6G...

#2 Updated by ggainey 3 months ago

  • Priority changed from Normal to High

Hi @wilful - I have some questions that will help us narrow this down.

  • Is there a specific repo that exhibits this behavior?
  • Do you have an idea of how many repo-versions/publications there are on the repo being deleted?
  • Are you doing things in parallel? (e.g., multiple deletes/syncs/copies)
  • Can you get us the pulp-logs from, say, 1 minute before thru 1 minute after the OOM, and attach to this issue? That will help debugging a lot!

Thanks!

#3 Updated by wilful 2 months ago

Hi! Try to answer:

  • This was our repository, which have more what 3-4k packages and more 100 versions
  • Publications was 20-30
  • No, only one task
  • I can't found logs, but when i deleting big repository, i see big RAM usage. Basically when destroyed repository with a lot versions/publication

#4 Updated by dalley 2 months ago

  • Severity changed from 2. Medium to 3. High
  • Triaged changed from No to Yes
  • Sprint set to Sprint 101

#5 Updated by dalley 2 months ago

Hi wilful, unrelated to this issue, I just wanted to warn you that pulp_rpm 3.13.0 had a data corruption bug - you should not use it.

Anything after 3.13.1 is OK, and it might be a good idea to delete any repository versions created after you installed pulp_rpm 3.13.0 and then run orphan cleanup to delete the content that was synced during that time period.

#6 Updated by ipanova@redhat.com about 2 months ago

  • Sprint changed from Sprint 101 to Sprint 102

#7 Updated by dalley about 2 months ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dalley

#8 Updated by rchan about 1 month ago

  • Sprint changed from Sprint 102 to Sprint 103

#9 Updated by rchan 29 days ago

  • Sprint changed from Sprint 103 to Sprint 104

#10 Updated by dalley 24 days ago

  • Status changed from ASSIGNED to NEW
  • Assignee deleted (dalley)

We'll get to this, I'm just unassiging it because of other issues that are higher priority

#11 Updated by dalley 23 days ago

  • Priority changed from High to Normal
  • Sprint deleted (Sprint 104)
  • Quarter set to Q4-2021

#12 Updated by dalley 1 day ago

  • Priority changed from Normal to Low
  • Sprint set to Sprint 106

Please register to edit this issue

Also available in: Atom PDF