Story #3186
Updated by bmbouter over 6 years ago
h2. Motivation
Versioned repositories will have significant impact on the locking system, in a good but unavoidable way. Pulp2 locked on repositories, but with repo versions being immutable and content units associating with repo versions, we need to rethink what locking means for Pulp3. The same tasking system can be used exactly as is, we just need to think about the right lock labels for different task types.
h2. Design
The existing tasking system prevents two tasks from running at the same time that have the same lock label. For correctness, I think we will need to introduce multi-resource locking, which is the ability to have one task require lock N resources. This is easily added to the tasking system. The resource locked and in what situation are listed here:
* an update/delete importer task locks on the importer instance
* a sync task locks on the importer instance if one is specified
* a sync task locks on the versioned repo
* a sync task locks on the repository
* an update/delete publisher task locks on the publisher instance
* a publish task locks on the publisher instance
* a publish task locks on the versioned repo
* a delete versioned repo task locks on the versioned repo
h2. A three task example
Say a user updates an importer with a PUT to /api/v3/importers/file-importer/55. That would cause the update task to lock on just that importer ["/api/v3/importers/file-importer/55"]
Then a user will POST to /api/v3/repository/foo/repo_version/ with {"importer": "/api/v3/importers/file-importer/55"}
That task would multi-lock on the importer and the repo version: ["/api/v3/importers/file-importer/55", "/api/v3/repository/foo/repo_version/72"] <---- Note that the viewset has to create the RepoVersion so it can know the detail view to lock on
Then a user will DELETE /api/v3/repository/foo/repo_version/72 which will cause a lock of ["/api/v3/repository/foo/repo_version/72"] to be used.
h4. Example Run Order
Even on a Pulp system with 3+ workers, the above 3 tasks will be executed fully serially because each one requires a resource locked by the one previous.
h2. Impact
I think this will cause Pulp's tasking throughput to increase significantly due to more concurrent opportunities such as concurrent sync+publishing.
.