Project

Profile

Help

Issue #1891

Distribution of 'import_uploaded_unit' tasks is unbalanced on Pulp workers

Added by yuzheng over 4 years ago. Updated over 1 year ago.

Status:
CLOSED - NOTABUG
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
2.7.1
Platform Release:
OS:
RHEL 6
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

How reproducible:
100%

Builds:
pulp-server-2.7.1-1.el6.noarch
pulp-rpm-plugins-2.7.1-1.el6.noarch

Steps:
1. Prepare 10 uploaded rpms in /var/lib/pulp/uploads/
2. Run 10 'POST /pulp/api/v2/repositories/{REPO_ID}/actions/import_upload/' requests in threads (thread_number=5) by attached script
3. Check pulp task status from db client

Actual result:
There were 4 running workers. But all 'import_uploaded_unit' tasks were allocated to the same pulp worker - reserved_resource_worker-3.

[root@dhcp-136-194 tmp]# service pulp_workers status
celery init v10.0.
Using config script: /etc/default/pulp_workers
node reserved_resource_worker-0 (pid 845) is running...
node reserved_resource_worker-1 (pid 870) is running...
node reserved_resource_worker-2 (pid 899) is running...
node reserved_resource_worker-3 (pid 927) is running...

db.task_status.find({task_type:"pulp.server.managers.content.upload.import_uploaded_unit"}).count()
10
db.task_status.find({task_type:"pulp.server.managers.content.upload.import_uploaded_unit", worker_name:"reserved_resource_worker-3@dhcp-136-194***"}).count()
10

Expected result:
Distribution of 10 tasks should be balanced on 4 workers. Otherwise, tasks can not be run in parallel.

import_test.py (1.19 KB) import_test.py yuzheng, 05/04/2016 09:01 AM

History

#1 Updated by mhrivnak over 4 years ago

  • Status changed from NEW to CLOSED - NOTABUG

Thanks for the report, and especially for the reproducer script!

For data safety and consistency, pulp can only execute one "write" task on a repository at a time. Those tasks include sync, upload, and copy. Since all of the tasks here are for the same repo, they each get put in a queue for the same worker. That pattern guarantees that only one will execute at a time.

If you queued upload tasks for 4 or more different repositories, you would see all of the workers busy.

#2 Updated by yuzheng over 4 years ago

mhrivnak wrote:

Thanks for the report, and especially for the reproducer script!

For data safety and consistency, pulp can only execute one "write" task on a repository at a time. Those tasks include sync, upload, and copy. Since all of the tasks here are for the same repo, they each get put in a queue for the same worker. That pattern guarantees that only one will execute at a time.

If you queued upload tasks for 4 or more different repositories, you would see all of the workers busy.

That makes sense. Thanks for quick response!

#3 Updated by bmbouter over 1 year ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF