Project

Profile

Help

Story #6024

As a plugin writer, I can iterate over repository versions in a memory efficient manner

Added by gmbnomis 7 months ago. Updated 6 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:

Description

Motivation

Plugins need to iterate over the content in a repository version to create a publication. Pulp
should offer a batching iterator over content in order to limit the memory footprint and to allow
efficient use of content batches in DB operations.

Solution

We could make batch_qs() available to plugin writers. But this generator is hard to use correctly because of the ordering requirement (even in pulpcore it has not been used correctly).

Instead, add a generator to RepositoryVersion, which does "the right thing" by default and is customizable by plugin writers if necessary.

For an example use case, see https://github.com/gmbnomis/pulp_cookbook/blob/batched_publication_upstream/pulp_cookbook/app/tasks/publishing.py#L87-L117

Associated revisions

Revision 2a9b2882 View on GitHub
Added by gmbnomis 7 months ago

Add a batching content iterator content_batch_qs() to RepositoryVersion

Plugins need to iterate over the content in a repository version to create a publication.

RepositoryVersion.content_batch_qs() generates content batches to efficiently iterate over all content.

fixes: #6024 https://pulp.plan.io/issues/6024

History

#1 Updated by gmbnomis 7 months ago

  • Description updated (diff)

#3 Updated by bmbouter 7 months ago

  • Status changed from NEW to POST

#4 Updated by gmbnomis 7 months ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#5 Updated by bmbouter 6 months ago

  • Sprint/Milestone set to 3.1.0

#6 Updated by bmbouter 6 months ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Please register to edit this issue

Also available in: Atom PDF