Project

Profile

Help

Issue #5668

Using futures in sysncpipeline basically deactivates batching

Added by mdellweg 8 months ago. Updated 23 days ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Performance
Sprint:
Sprint 63

Description

Currently, when using await on the result of get_or_create_future of a DeclarativeContent without marking it as unable to batch, will most likely lead to deadlocks, when the future is awaited, while the content is waiting for a batch to be filled.

The related PR: https://github.com/pulp/pulpcore/pull/365
provides the possibility that a DeclarativeContent can be batched and will wait until its associated future is actually awaited. At that very moment it is marked as no longer able to batch up, and it signals the queue it is in to flush the current (incomplete) batch.

This change is completely backwards compatible. It only adds the possibility to use the futures without specifying does_batch=False.

If used reasonably, it can help to increase sync performance in some cases.

Associated revisions

Revision 338deb90 View on GitHub
Added by mdellweg 7 months ago

Add resolution callback to DeclarativeContent

If it is currently waiting in a queue, unfreeze that. Remove old interfaces does_batch and get_or_create_future.

fixes #5668

Revision d98259db View on GitHub
Added by bmbouter 7 months ago

Update release notes for Batching change

https://pulp.plan.io/issues/5668 closes #5668

History

#1 Updated by daviddavis 8 months ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 61

#2 Updated by dalley 8 months ago

  • Status changed from NEW to POST
  • Assignee set to mdellweg

#3 Updated by rchan 8 months ago

  • Sprint changed from Sprint 61 to Sprint 62

#4 Updated by rchan 7 months ago

  • Sprint changed from Sprint 62 to Sprint 63

#5 Updated by mdellweg 7 months ago

  • Status changed from POST to MODIFIED

#6 Updated by mdellweg 7 months ago

mdellweg wrote:

This change is completely backwards compatible. It only adds the possibility to use the futures without specifying does_batch=False.

This is no longer true. The change removes both does_batch as well as get_or_create_future and adds the resolution coroutine to replace them.

#8 Updated by bmbouter 5 months ago

  • Sprint/Milestone set to 3.1.0

#9 Updated by bmbouter 5 months ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

#10 Updated by bmbouter 23 days ago

  • Tags Performance added
  • Tags deleted (Sync Performance)

Please register to edit this issue

Also available in: Atom PDF