Project

Profile

Help

Story #3186

Updated by bmbouter about 2 years ago

h2. Motivation

Versioned repositories will have significant impact on the locking system. Due to the bookkeeping approach, anytime you modify content or system, in a RepositoryVersion, you need to serialize those operations good but unavoidable way. Pulp2 locked on repositories, but with other repo versions being immutable and content manipulations on that repo.

Also, the decision to not associate an importer directly
units associating with a repo versions, we need to rethink what locking means that for Pulp3. The same tasking system can be used exactly as is, we just need to think about the right lock differently when updating importer settings. labels for different task types.

Finally, publication manipulation also needs to lock on repository versions.

h2. Design

The existing tasking system prevents two tasks from running at the same time that have the same lock label. For correctness, I think we will need to introduce multi-resource locking, which is the ability to have one task require lock N resources. This is easily added to the tasking system. The resource locked and in what situation are listed here:

* an update/delete importer task locks on the importer instance
* an update/delete publisher task locks on the publisher instance

*
a sync task locks on the importer instance if one is specified
* a sync task locks on the versioned repo
* a sync task locks on the repository it is associated with


* a publish an update/delete publisher task locks on the publisher instance
* a publish task locks on the versioned repo

publisher instance
* a delete versioned repo publish task locks on the versioned repo
* a delete versioned repo task locks on the repository versioned repo

h2. A three task example

Say a user updates an importer with a PUT to /api/v3/importers/file-importer/55. That would cause the update task to lock on just that importer ["/api/v3/importers/file-importer/55"]

Then a user will POST to /api/v3/repository/foo/repo_version/ with {"importer": "/api/v3/importers/file-importer/55"}

That task would multi-lock on the importer and the repo version: ["/api/v3/importers/file-importer/55", "/api/v3/repository/foo/repo_version/72"] <---- Note that the viewset has to create the RepoVersion so it can know the detail view to lock on

Then a user will DELETE /api/v3/repository/foo/repo_version/72 which will cause a lock of ["/api/v3/repository/foo/repo_version/72"] to be used.

h4. Example Run Order

Even on a Pulp system with 3+ workers, the above 3 tasks will be executed fully serially because each one requires a resource locked by the one previous.

h2. Impact

I think this will cause Pulp's tasking throughput to increase significantly due to more concurrent opportunities such as concurrent sync+publishing.

Back