Project

Profile

Help

Story #6024

closed

As a plugin writer, I can iterate over repository versions in a memory efficient manner

Added by gmbnomis over 2 years ago. Updated over 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Quarter:

Description

Motivation

Plugins need to iterate over the content in a repository version to create a publication. Pulp
should offer a batching iterator over content in order to limit the memory footprint and to allow
efficient use of content batches in DB operations.

Solution

We could make batch_qs() available to plugin writers. But this generator is hard to use correctly because of the ordering requirement (even in pulpcore it has not been used correctly).

Instead, add a generator to RepositoryVersion, which does "the right thing" by default and is customizable by plugin writers if necessary.

For an example use case, see https://github.com/gmbnomis/pulp_cookbook/blob/batched_publication_upstream/pulp_cookbook/app/tasks/publishing.py#L87-L117

Actions #1

Updated by gmbnomis over 2 years ago

  • Description updated (diff)
Actions #3

Updated by bmbouter over 2 years ago

  • Status changed from NEW to POST

Added by gmbnomis over 2 years ago

Revision 2a9b2882

Add a batching content iterator content_batch_qs() to RepositoryVersion

Plugins need to iterate over the content in a repository version to create a publication.

RepositoryVersion.content_batch_qs() generates content batches to efficiently iterate over all content.

fixes: #6024 https://pulp.plan.io/issues/6024

Actions #4

Updated by gmbnomis over 2 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100
Actions #5

Updated by bmbouter over 2 years ago

  • Sprint/Milestone set to 3.1.0
Actions #6

Updated by bmbouter over 2 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF