Project

Profile

Help

Issue #5668

closed

Using futures in sysncpipeline basically deactivates batching

Added by mdellweg over 4 years ago. Updated over 3 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Performance
Sprint:
Sprint 63
Quarter:

Description

Currently, when using await on the result of get_or_create_future of a DeclarativeContent without marking it as unable to batch, will most likely lead to deadlocks, when the future is awaited, while the content is waiting for a batch to be filled.

The related PR: https://github.com/pulp/pulpcore/pull/365
provides the possibility that a DeclarativeContent can be batched and will wait until its associated future is actually awaited. At that very moment it is marked as no longer able to batch up, and it signals the queue it is in to flush the current (incomplete) batch.

This change is completely backwards compatible. It only adds the possibility to use the futures without specifying does_batch=False.

If used reasonably, it can help to increase sync performance in some cases.

Also available in: Atom PDF