Project

Profile

Help

Task #8495

[EPIC] As a user, I have a new-improved tasking system with new-style workers

Added by bmbouter 14 days ago. Updated 9 days ago.

Status:
NEW
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
% Done:

29%

Estimated time:
(Total: 0:00 h)
Platform Release:
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Quarter:

Description

Current issues

Goals

  1. Eliminate the resource manager. This will address the architectural throughput issue while also simplifying Pulp deployments.
  2. Introduce a new worker type that is not RQ based and instead works entirely out of the Task table in Postgresql. This will resolve the high availability issue and the data synchronization data.

Plan for introducing this to users

Both the new-style and the old, RQ-style workers will be available in the codebase at the same time. This will allow users to try the new-style workers, but also be able to go back to the old style if any issues come up.

The new-style workers will be introduced in the following phases over a variety of releases:

  1. Introduce the new-style workers and leave the old-style workers as the default. A new setting named USE_NEW_WORKER_TYPE will allow users to opt-in to the new-style.
  2. Add support to the installer, containers (s6), and operator to allow users to also opt-in
  3. In a specific release, change the default of USE_NEW_WORKER_TYPE=False to USE_NEW_WORKER_TYPE=True. This will still keep the old-style workers around so users can opt-out back to the old style if they need to.
  4. Remove the old-style worker and its codepaths entirely. Drop RQ as a dependency.

Subtasks

Task #8496: Deprecate `enqueue_with_reservation` from plugin API and introduce new `dispatch` interface to replace itCLOSED - CURRENTRELEASEbmbouter

Actions
Task #8497: Remove the deprecated `enqueue_with_reservation` from the plugin APINEW

Actions
Story #8501: As a user, I have new pulp workers that do not require a resource managerASSIGNEDbmbouter

Actions
Story #8502: As an installer user, I can opt-in to the new-style workersNEW

Actions
Story #8503: As a single-container user, I can opt-in to the new-style workersNEW

Actions
Story #8504: As an operator user, I can opt-in to the new-style workersNEW

Actions
Story #8505: As a plugin developer, I am notified that passing non-JSON serializable arguments is deprecatedCLOSED - CURRENTRELEASEbmbouter

Actions

History

#1 Updated by bmbouter 13 days ago

  • Description updated (diff)

Please register to edit this issue

Also available in: Atom PDF