Project

Profile

Help

Issue #5668

Using futures in sysncpipeline basically deactivates batching

Added by mdellweg 12 months ago. Updated 4 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Performance
Sprint:
Sprint 63
Quarter:

Description

Currently, when using await on the result of get_or_create_future of a DeclarativeContent without marking it as unable to batch, will most likely lead to deadlocks, when the future is awaited, while the content is waiting for a batch to be filled.

The related PR: https://github.com/pulp/pulpcore/pull/365
provides the possibility that a DeclarativeContent can be batched and will wait until its associated future is actually awaited. At that very moment it is marked as no longer able to batch up, and it signals the queue it is in to flush the current (incomplete) batch.

This change is completely backwards compatible. It only adds the possibility to use the futures without specifying does_batch=False.

If used reasonably, it can help to increase sync performance in some cases.

Associated revisions

Revision 338deb90 View on GitHub
Added by mdellweg 11 months ago

Add resolution callback to DeclarativeContent

If it is currently waiting in a queue, unfreeze that. Remove old interfaces does_batch and get_or_create_future.

fixes #5668

Revision d98259db View on GitHub
Added by bmbouter 11 months ago

Update release notes for Batching change

https://pulp.plan.io/issues/5668 closes #5668

History

#1 Updated by daviddavis 12 months ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 61

#2 Updated by dalley 12 months ago

  • Status changed from NEW to POST
  • Assignee set to mdellweg

#3 Updated by rchan 12 months ago

  • Sprint changed from Sprint 61 to Sprint 62

#4 Updated by rchan 11 months ago

  • Sprint changed from Sprint 62 to Sprint 63

#5 Updated by mdellweg 11 months ago

  • Status changed from POST to MODIFIED

#6 Updated by mdellweg 11 months ago

mdellweg wrote:

This change is completely backwards compatible. It only adds the possibility to use the futures without specifying does_batch=False.

This is no longer true. The change removes both does_batch as well as get_or_create_future and adds the resolution coroutine to replace them.

#8 Updated by bmbouter 9 months ago

  • Sprint/Milestone set to 3.1.0

#9 Updated by bmbouter 9 months ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

#10 Updated by bmbouter 4 months ago

  • Tags Performance added
  • Tags deleted (Sync Performance)

Please register to edit this issue

Also available in: Atom PDF