Project

Profile

Help

Task #8495

[EPIC] As a user, I have a new-improved tasking system with new-style workers

Added by bmbouter 7 months ago. Updated 4 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
% Done:

100%

Estimated time:
(Total: 0:00 h)
Platform Release:
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Quarter:

Description

Current issues

Goals

  1. Eliminate the resource manager. This will address the architectural throughput issue while also simplifying Pulp deployments.
  2. Introduce a new worker type that is not RQ based and instead works entirely out of the Task table in Postgresql. This will resolve the high availability issue and the data synchronization data.

Plan for introducing this to users

Both the new-style and the old, RQ-style workers will be available in the codebase at the same time. This will allow users to try the new-style workers, but also be able to go back to the old style if any issues come up.

The new-style workers will be introduced in the following phases over a variety of releases:

  1. Introduce the new-style workers and leave the old-style workers as the default. A new setting named USE_NEW_WORKER_TYPE will allow users to opt-in to the new-style.
  2. Add support to the installer, containers (s6), and operator to allow users to also opt-in
  3. In a specific release, change the default of USE_NEW_WORKER_TYPE=False to USE_NEW_WORKER_TYPE=True. This will still keep the old-style workers around so users can opt-out back to the old style if they need to.
  4. Remove the old-style worker and its codepaths entirely. Drop RQ as a dependency.

Subtasks

Task #8496: Deprecate `enqueue_with_reservation` from plugin API and introduce new `dispatch` interface to replace itCLOSED - CURRENTRELEASEbmbouter

Actions
Task #8497: Remove the deprecated `enqueue_with_reservation` from the plugin APICLOSED - CURRENTRELEASEipanova@redhat.com

Actions
Story #8501: As a user, I have new pulp workers that do not require a resource managerCLOSED - CURRENTRELEASEmdellweg

Actions
Story #8502: As an installer user, I can opt-in to the new-style workersCLOSED - NOTABUG

Actions
Story #8503: As a single-container user, I can opt-in to the new-style workersCLOSED - CURRENTRELEASEmdellweg

Actions
Story #8504: As an operator user, I can opt-in to the new-style workersCLOSED - NOTABUG

Actions
Story #8505: As a plugin developer, I am notified that passing non-JSON serializable arguments is deprecatedCLOSED - CURRENTRELEASEbmbouter

Actions
Story #8930: As system admin i can gracefully shutdown new style workersCLOSED - CURRENTRELEASEmdellweg

Actions
Task #8948: Switch default tasking system and adjust documentationCLOSED - CURRENTRELEASEmdellweg

Actions

History

#1 Updated by bmbouter 7 months ago

  • Description updated (diff)

#2 Updated by bmbouter 4 months ago

  • Status changed from NEW to CLOSED - CURRENTRELEASE

This epic is done.

Please register to edit this issue

Also available in: Atom PDF