Project

Profile

Help

Story #3934

closed

RPM Support - Story #3202: As a user, I can sync RPM/SRPM/Erratum from a remote Yum/DNF repository

As a plugin writer, I can have a stage that removes duplicates

Added by daviddavis over 5 years ago. Updated over 4 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Sprint:
Sprint 46
Quarter:

Description

Problem

Both rpm and docker have a situation where content units can mutate. The DeclarativeVersion will treat a mutated content unit as a new unit and add it to a RepositoryVersion with sync. This is in addition to the previous version (the unmutated one). This effectively adds the same unit to the RepositoryVersion twice.

What would be great is if the older one would be removed by DeclarativeVersion as part of the pipeline in some kind of configurable way.

Solution

Make a new stage called RemoveDuplicates that takes two parameters 'type' and 'field_list' (or tuple). 'type' is the content unit type that the stage should inspect. 'field_list' is the list of field names that needs to be unique within the RepositoryVersion. For example for RPM it will configure this stage with type=pulp_rpm.UpdateRecord and field_list=['id']. A Docker example would use the stage twice, first: `type=pulp_docker.Tag`, `field_list=['name', 'manifest']`; second: `type=pulp_docker.Tag`, field_list=`['name', 'manifest_list']`

This new stage will unassociate any units that are of type=type with the same field names as one of the units emitted in the DeclarativeContent stream. It will be a batching stage, handling batches of units at a time. (Note, batches might perform poorly here, since multiple types may be flowing through the stream.)

The stage can be used directly by plugin writers. This funcitonality will also be added as an option to DeclarativeVersion called remove_duplicates which will take the following form:

[{
    'type': 'pulp_rpm.UpdateRecord',
    'field_names': ['id']
}]

Notice how the stage takes only 1 duplicate type, but the DeclarativeVersion takes a list of them. The DeclarativeVersion will create one RemoveDuplicates stage for each item in the list, making the pipeline a variable length depending on the data passed into DeclarativeVersion.

These extra stages should be run before the AssociateContent stage.


Related issues

Blocks RPM Support - Task #3954: Prevent duplicate Package content in reposCLOSED - CURRENTRELEASEttereshc

Actions
Blocks File Support - Task #4028: Prevent duplicate files in repositoriesCLOSED - CURRENTRELEASEbmbouter

Actions
Blocks Container Support - Story #4172: Remove duplicate tags from repository during syncCLOSED - CURRENTRELEASEamacdona@redhat.com

Actions

Also available in: Atom PDF