Project

Profile

Help

Story #3186

Updated by bmbouter almost 7 years ago

h2. Motivation 

 Versioned repositories will have significant impact on the locking system, in a good but unavoidable way. Pulp2 locked on repositories, but with repo versions being immutable and content units associating with repo versions, we need to rethink what locking means for Pulp3. The same tasking system can be used exactly as is, we just need to think about the right lock labels for different task types. 

 h2. Design 

 The existing tasking system prevents two tasks from running at the same time that have the same lock label. For correctness, I think we will need to introduce multi-resource locking, which is the ability to have one task require lock N resources. This is easily added to the tasking system. The resource locked and in what situation are listed here: 

 * an update/delete importer task locks on the importer instance 
 * a sync task locks on the importer instance if one is specified 
 * a sync task locks on the versioned repo 
 * a sync task locks on the repository 
 * an update/delete publisher task locks on the publisher instance 
 * a publish task locks on the publisher instance 
 * a publish task locks on the versioned repo 
 * a delete versioned repo task locks on the versioned repo 

 h2. A three task example 

 Say a user updates an importer with a PUT to /api/v3/importers/file-importer/55. That would cause the update task to lock on just that importer ["/api/v3/importers/file-importer/55"] 

 Then a user will POST to /api/v3/repository/foo/repo_version/ with {"importer": "/api/v3/importers/file-importer/55"} 

 That task would multi-lock on the importer and the repo version: ["/api/v3/importers/file-importer/55", "/api/v3/repository/foo/repo_version/72"]       <---- Note that the viewset has to create the RepoVersion so it can know the detail view to lock on 

 Then a user will DELETE /api/v3/repository/foo/repo_version/72 which will cause a lock of ["/api/v3/repository/foo/repo_version/72"] to be used. 

 h4. Example Run Order 

 Even on a Pulp system with 3+ workers, the above 3 tasks will be executed fully serially because each one requires a resource locked by the one previous. 

 h2. Impact 

 I think this will cause Pulp's tasking throughput to increase significantly due to more concurrent opportunities such as concurrent sync+publishing.

Back