Project

Profile

Help

Issue #8992

closed

Memory usage when destroy repository; The memory leaks?

Added by wilful over 2 years ago. Updated about 2 years ago.

Status:
CLOSED - DUPLICATE
Priority:
Low
Assignee:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Platform Release:
OS:
CentOS 7
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 111
Quarter:
Q4-2021

Description

Ticket moved to GitHub: "pulp/pulp_rpm/2280":https://github.com/pulp/pulp_rpm/issues/2280


When i try to destroy one of repositories, pulp (RQ basically) are using more 3G RAM and crashed by OOM-Killer

Jun 30 06:59:16 *** kernel: Out of memory: Kill process 81458 (rq) score 493 or sacrifice child
Jun 30 06:59:16 *** kernel: Killed process 81458 (rq), UID 1002, total-vm:4434448kB, anon-rss:3936104kB, file-rss:0kB, shmem-rss:0kB

State before on my node:

# free -h
              total        used        free      shared  buff/cache   available
Mem:           7.6G        1.5G        3.6G        2.2G        2.6G        3.7G
Swap:            0B          0B          0B
  "versions": [
    {
      "component": "core",
      "version": "3.13.0"
    },
    {
      "component": "rpm",
      "version": "3.13.0"
    },
    {
      "component": "python",
      "version": "3.4.0"
    },
    {
      "component": "file",
      "version": "1.8.0"
    },
    {
      "component": "deb",
      "version": "2.13.0"
    },
    {
      "component": "container",
      "version": "2.6.0"
    }
  ],

Files

Actions #1

Updated by wilful over 2 years ago

More 6G...

Actions #2

Updated by ggainey over 2 years ago

  • Priority changed from Normal to High

Hi @wilful - I have some questions that will help us narrow this down.

  • Is there a specific repo that exhibits this behavior?
  • Do you have an idea of how many repo-versions/publications there are on the repo being deleted?
  • Are you doing things in parallel? (e.g., multiple deletes/syncs/copies)
  • Can you get us the pulp-logs from, say, 1 minute before thru 1 minute after the OOM, and attach to this issue? That will help debugging a lot!

Thanks!

Actions #3

Updated by wilful over 2 years ago

Hi! Try to answer:

  • This was our repository, which have more what 3-4k packages and more 100 versions
  • Publications was 20-30
  • No, only one task
  • I can't found logs, but when i deleting big repository, i see big RAM usage. Basically when destroyed repository with a lot versions/publication
Actions #4

Updated by dalley over 2 years ago

  • Severity changed from 2. Medium to 3. High
  • Triaged changed from No to Yes
  • Sprint set to Sprint 101
Actions #5

Updated by dalley over 2 years ago

Hi wilful, unrelated to this issue, I just wanted to warn you that pulp_rpm 3.13.0 had a data corruption bug - you should not use it.

Anything after 3.13.1 is OK, and it might be a good idea to delete any repository versions created after you installed pulp_rpm 3.13.0 and then run orphan cleanup to delete the content that was synced during that time period.

Actions #6

Updated by ipanova@redhat.com over 2 years ago

  • Sprint changed from Sprint 101 to Sprint 102
Actions #7

Updated by dalley over 2 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dalley
Actions #8

Updated by rchan over 2 years ago

  • Sprint changed from Sprint 102 to Sprint 103
Actions #9

Updated by rchan over 2 years ago

  • Sprint changed from Sprint 103 to Sprint 104
Actions #10

Updated by dalley over 2 years ago

  • Status changed from ASSIGNED to NEW
  • Assignee deleted (dalley)

We'll get to this, I'm just unassiging it because of other issues that are higher priority

Actions #11

Updated by dalley over 2 years ago

  • Priority changed from High to Normal
  • Sprint deleted (Sprint 104)
  • Quarter set to Q4-2021
Actions #12

Updated by dalley over 2 years ago

  • Priority changed from Normal to Low
  • Sprint set to Sprint 106
Actions #13

Updated by rchan over 2 years ago

  • Sprint changed from Sprint 106 to Sprint 107
Actions #14

Updated by rchan over 2 years ago

  • Sprint changed from Sprint 107 to Sprint 108
Actions #15

Updated by rchan over 2 years ago

  • Sprint changed from Sprint 108 to Sprint 109
Actions #16

Updated by rchan over 2 years ago

  • Sprint changed from Sprint 109 to Sprint 110
Actions #17

Updated by wilful over 2 years ago

Today i can't remove repository with more what 15G RAM, please up these issue for medium or hight. Its big trouble for multiversion repository

Actions #18

Updated by rchan over 2 years ago

  • Sprint changed from Sprint 110 to Sprint 111
Actions #19

Updated by pulpbot about 2 years ago

  • Description updated (diff)
  • Status changed from NEW to CLOSED - DUPLICATE

Also available in: Atom PDF