Story #898
closed
As a user I can run multiple pulp_resource_managers concurrently with all of them actively participating
100%
Description
This story has a some database schema changes, a small coding part, and a good deal of testing. The FCFS ordering should be tested first before any of the coding changes happen.
Current ReservedResource Design¶
The ReservedResource collection needs to have its database schema adjusted so that we can have an opportunity to interact with it in a race-condition free way. Currently the mongoengine model defines:
<task_id, worker_name, resource_id>
The only restriction is that task_id needs to be unique. Generally one record is created for each task id.
Schema Changes¶
Having multiple records entered prevents us from adding the necessary indexes which would make writing atomic code for this collection more feasible. Consider this schema:
class ReservedResource(Document):
resource_id = StringField(primary_key=True, unique=True) # ensures correctness that each resource_type can only have one reservation across all workers
worker_name = StringField(required=True) # ensures that each worker can only receive one reservation at a time
task_ids = ListField() # notice, this is now a list
# For backward compatibility
_ns = StringField(default='reserved_resources')
meta = {'collection': 'reserved_resources', allow_inheritance': False}
The existing indexes would all be removed.
Coding Deliverables¶
- Rewrite the reservation creation code so that it has roughly the following logic:
while:
# assume we need to update an existing reservation
find an existing worker by querying for an existing ReservedResource by the resource_id
if there is a worker found:
attempt to add the task_id that needs reservation for any found existing worker, but do not allow upsert
assert that the number of documents updated is either 0 or 1. If not it should raise a loud exception
if 1:
query for the worker name that goes with the reservation just updated
else if no worker found or if no worker was updated:
# find an available worker, relying on uniqueness constraints to ensure atomic operation
query for the list of workers
for each worker in the list:
try:
create a new ReservedResource(worker_name, resource_id, taskids=[task_id]).save()
except ConstraintViolation:
continue
if a ReservationResource was updated:
break
else:
sleep(0.25)
Testing¶
This needs to be tested to ensure two main this:
1. That a highly concurrent environment calling reserve resource and release resource maintains correctness. This should be done with a simple concurrency driver which would be left to run in a bunch of processes in a test environment for a few days.
2. That the FCFS ordering is strictly preserved even when failures occur.
Related issues
.
Added mechanism to prevent two resource managers from being active at once - duplicate resource managers will wait until the original dies off. closes #898