Project

Profile

Help

Story #898

closed

As a user I can run multiple pulp_resource_managers concurrently with all of them actively participating

Added by bmbouter over 9 years ago. Updated over 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
2.10.0
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Pulp 2
Sprint:
Sprint 5
Quarter:

Description

This story has a some database schema changes, a small coding part, and a good deal of testing. The FCFS ordering should be tested first before any of the coding changes happen.

Current ReservedResource Design

The ReservedResource collection needs to have its database schema adjusted so that we can have an opportunity to interact with it in a race-condition free way. Currently the mongoengine model defines:

<task_id, worker_name, resource_id>

The only restriction is that task_id needs to be unique. Generally one record is created for each task id.

Schema Changes

Having multiple records entered prevents us from adding the necessary indexes which would make writing atomic code for this collection more feasible. Consider this schema:

class ReservedResource(Document):
    resource_id = StringField(primary_key=True, unique=True) # ensures correctness that each resource_type can only have one reservation across all workers
    worker_name = StringField(required=True) # ensures that each worker can only receive one reservation at a time
    task_ids = ListField() # notice, this is now a list

    # For backward compatibility
    _ns = StringField(default='reserved_resources')

    meta = {'collection': 'reserved_resources', allow_inheritance': False}

The existing indexes would all be removed.

Coding Deliverables

while:
    # assume we need to update an existing reservation
    find an existing worker by querying for an existing ReservedResource by the resource_id
    if there is a worker found:
        attempt to add the task_id that needs reservation for any found existing worker, but do not allow upsert
        assert that the number of documents updated is either 0 or 1. If not it should raise a loud exception
        if 1:
            query for the worker name that goes with the reservation just updated
    else if no worker found or if no worker was updated:
        # find an available worker, relying on uniqueness constraints to ensure atomic operation 
        query for the list of workers
        for each worker in the list:
            try:
                create a new ReservedResource(worker_name, resource_id, taskids=[task_id]).save()
            except ConstraintViolation:
                continue 
    if a ReservationResource was updated:
        break
    else:
        sleep(0.25)

Testing

This needs to be tested to ensure two main this:

1. That a highly concurrent environment calling reserve resource and release resource maintains correctness. This should be done with a simple concurrency driver which would be left to run in a bunch of processes in a test environment for a few days.

2. That the FCFS ordering is strictly preserved even when failures occur.


Related issues

Blocks Pulp - Issue #1114: If an instance of pulp_resource_manager dies unexpectedly, Pulp incorrectly tries to "cancel all tasks in its queue"CLOSED - WONTFIXActions

Also available in: Atom PDF