Project

Profile

Help

Story #2371

Use process recycling by default

Added by bmbouter almost 3 years ago. Updated 6 months ago.

Status:
CLOSED - WONTFIX
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

0%

Platform Release:
Blocks Release:
Backwards Incompatible:
Yes
Groomed:
No
Sprint Candidate:
Yes
Tags:
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:

Description

On the 2.y line a feature was introduced which is disabled by default. The feature was process recycling for celery workers. This was done as Issue #2172.

This issue is to update the conf file defaults from 0 to 2 which will enable this feature by default.


Related issues

Related to Pulp - Story #2172: Memory Improvements with Process Recycling CLOSED - CURRENTRELEASE Actions

History

#1 Updated by bmbouter almost 3 years ago

This is blocked until the commits from #2172 are merged from master to 3.0-dev branch.

#2 Updated by bmbouter almost 3 years ago

  • Related to Story #2172: Memory Improvements with Process Recycling added

#3 Updated by mhrivnak almost 3 years ago

Can you comment on why the value of 2 was chosen? Just based on gut reaction, that seems aggressive. In addition to normal overhead of destroying a process and creating a new one, in this case that also means tearing down and re-creating connections to the database and message broker.

As these things go, the price is likely very small on a mostly-idle system, but grows as resource contention occurs. We haven't quantified that total cost, but even facing an unknown (probably small) cost, we do get to choose how often we pay it. Paying the cost almost as often as possible may be a fine choice, but wouldn't be my personal starting point. What's the thinking?

#4 Updated by bmbouter almost 3 years ago

Almost all tasks in Pulp require a reservation. Each "reservation task" is actually 2 celery tasks to be processed by a worker. The first is the task itself, the second is a task to release the reservation for that task in the database. Because of this a value of 1 would be unproductive.

Why 2? Anecdotally, Pulp tasks processing real data have service times probably have an average on the order of minutes. Even in our dev environments with no-op tasks it takes multiple seconds. The additional delay caused by process recycling is small probably < 0.5 seconds. Even with a conservative average runtime of 60 seconds, a value of 2 would make that runtime 60.5 seconds which is an overhead of 0.8 %.

I also think the common case of Pulp installations is a mostly idle worker so this optimizes on that by aggressively freeing memory since it may not get more work soon.

Note that the parent process is not torn down, so it won't have to establish a new broker connection in most cases. The parent process does most of the broker communication. That overhead of having to make a new db connection is true. Also note that the process recycling is done by re-forking, which causes the subsequent process to not start from scratch in terms of its Python state. For example the Pulp tasking code was already imported by the parent process.

#5 Updated by dalley over 1 year ago

  • Status changed from NEW to CLOSED - WONTFIX

RQ re-forks for each task, therefore, this issue can be closed.

#6 Updated by daviddavis 6 months ago

  • Sprint/Milestone set to 3.0

#7 Updated by bmbouter 6 months ago

  • Tags deleted (Pulp 3)

Please register to edit this issue

Also available in: Atom PDF