Project

Profile

Help

Story #5344

closed

As a user, I have a consistant API experience for copying content

Added by dalley over 4 years ago. Updated over 4 years ago.

Status:
CLOSED - DUPLICATE
Priority:
Normal
Assignee:
Sprint/Milestone:
-
Start date:
Due date:
% Done:

0%

Estimated time:
Platform Release:
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Quarter:

Description

Problem Statement:

API design for copy and upload has thus far been somewhat haphazard and it would be useful to make it more consistent. Additionally there are some deficiencies in some of the current APIs that need to be addressed.

For example, the copy API currently looks something like this

POST /pulp/api/v3/rpm/copy/ types=['package', 'modulemd']

The problem with this API is that it will be difficult to make it play nicely with filtering, which has yet to be implemented. The filtering use case is essentially "copy units for which metadata matches these filters, e.g. name=walrus and version>1.1.0". However, because different unit types have different metadata fields, allowing the user to specify filters while also allowing more than one type of content to be copied at once would be problematic because there's no good way to determine what filters should apply to what content types. Possibly it could be done, but it would probably not be a good idea. Therefore it is best that the "types" parameter be removed entirely.

Sidenote: In various discussions we've mentioned the utility (and possibly necessity) of being able to have a group of tasks that are atomic and transactional together. Something like this could be applied here in order to replace this functionality (however, it is out of scope for this Story).

There are 2 different alternative APIs I would like to propose:

1. The same API, but with a single type copied at a time. If "all" is specified as the type, it would copy everything. This would IMO cover the most important use cases.

POST /pulp/api/v3/rpm/copy/ type='package'

Pros:

- More flexible, more similar to the Pulp 2 API

Cons:

- Dissimilar to the other API's in Pulp 3

- "type" is going to be the name of a field on the RPM content itself (inherited from core, formerly _type)

2. Make the type to be copied part of the name of the endpoint itself, like so:

POST /pulp/api/v3/rpm/package/copy/

Pros:

- Consistent with other Pulp 3 APIs

Cons:

- Not as flexible, no way to copy all units at once via this API

3. Use a complex filter endpoint

POST /pulp/api/v3/rpm/copy/ filter={'package': "name='foo', version>='4.5', arch='x86_64'", 'modulemd': "name=ripgrep, stream=master"}

Pros:

- Might be able to support all the use cases we want in one API endpoint

Cons:

- Complicated to implement

Use cases

  • As a user, I can copy all content of a specific type from one repository to another repository
  • As a user, I can copy content matching certain "search criteria" from one repository to another repository
  • As a user, when copying content that directly references other content, the referenced content is ?always? copied
    * e.g. Modules referencing Modules, Modules referencing RPMs, Erratum referencing RPMs, and {{other types}}
  • As a user, I can optionally choose to copy all indirect dependencies of content that is being copied (recursive copy)
    * Should this be the default?
  • Some content types create a new content unit in the destination repo instead of just copying, e.g. yum_repo_metadata_file
  • Special copy needed for modules? (multi-repo)
  • Special copy needed for distribution trees?

Related issues

Related to RPM Support - Test #5276: Test - As a user, I have simple content copy between repositoriesCLOSED - DUPLICATEActions
Related to RPM Support - Task #5681: Remove /pulp/api/v3/rpm/copy/ endpointCLOSED - CURRENTRELEASECodeHeeler

Actions
Actions #1

Updated by dalley over 4 years ago

  • Tracker changed from Task to Story
  • Subject changed from Implement a consistent copy and upload API compatible with filtering to As a user, I have a consistant API experience for copying and uploading content
  • Description updated (diff)
Actions #2

Updated by dalley over 4 years ago

  • Subject changed from As a user, I have a consistant API experience for copying and uploading content to As a user, I have a consistant API experience for copying and uploading content [EPIC]
  • Description updated (diff)
Actions #3

Updated by dalley over 4 years ago

  • Subject changed from As a user, I have a consistant API experience for copying and uploading content [EPIC] to [EPIC] As a user, I have a consistant API experience for copying and uploading content
  • Description updated (diff)
Actions #4

Updated by dalley over 4 years ago

  • Description updated (diff)
Actions #5

Updated by dalley over 4 years ago

  • Subject changed from [EPIC] As a user, I have a consistant API experience for copying and uploading content to [EPIC] As a user, I have a consistant API experience for copying content
  • Description updated (diff)
Actions #6

Updated by dalley over 4 years ago

  • Description updated (diff)
Actions #7

Updated by ttereshc over 4 years ago

  • Tags Pulp 3 RPM blocker added
Actions #8

Updated by ttereshc over 4 years ago

  • Subject changed from [EPIC] As a user, I have a consistant API experience for copying content to As a user, I have a consistant API experience for copying content
Actions #9

Updated by kersom over 4 years ago

  • Related to Test #5276: Test - As a user, I have simple content copy between repositories added
Actions #10

Updated by ggainey over 4 years ago

Discussion:

  • We will always want an easy 'complete copy/clone' API. You can get this currently via creating a repo-version and specifying a base-version
  • There is a use-case for filtering, either by-type (or possibly diff-filters-per-type?)
  • There is a use-case for "one version created for a copy even when multiple filters are in play"

Proposal:

  • For initial GA, remove/hide /copy endpoint
  • For initial GA, let 'clone this repo' be supported via the create-repo-version-from-base mentioned above.
  • Further investigation on how to do filtering needed, with an additive API change post-GA. Need sub-tasks for this story.
Actions #11

Updated by ggainey over 4 years ago

Possible proposal for multi-type copy and filtering: have filter= be not a string, but a structure/dictionary, key by content-type, value is 'filter string for that type'

Terrible non-real example:

POST /pulp/api/v3/rpm/copy/ types=['package', 'modulemd'] filter={'package': "name='foo', version>='4.5', arch='x86_64'", 'modulemd': "<insert modulemd filtering here?>"}

Obviously, filter-syntax could be Very Complicated. Edge-cases between types (ie, type-1's filter and type-2's filter result in a non-sane repository) are going to be frequent and sharp. Needs some serious thought and doc, with real/working examples.

Actions #12

Updated by ttereshc over 4 years ago

  • Related to Task #5681: Remove /pulp/api/v3/rpm/copy/ endpoint added
Actions #13

Updated by ttereshc over 4 years ago

  • Tags deleted (Pulp 3 RPM blocker)

Removing a blocker tag for 3.0. These endpoints will be added after 3.0 GA.

Actions #14

Updated by dalley over 4 years ago

  • Parent issue set to #4762

Use cases, from the RPM plugin roadmap

  • As a user, I can copy a content of a specific type from one repository to another repository
  • As a user, I can copy RPM packages and all their dependencies from one repository to another repository
  • As a user, I can copy an Erratum and all the RPM packages it references from one repository to another repository
  • As a user, I can copy an Erratum and all the RPM packages it references and all the RPM dependencies from one repository to another repository
  • Some content types create a new content unit in the destination repo instead of just copying, e.g. yum_repo_metadata_file
  • Special copy for modules [TBD]
Actions #15

Updated by dalley over 4 years ago

  • Description updated (diff)
Actions #16

Updated by dalley over 4 years ago

  • Status changed from NEW to CLOSED - DUPLICATE
  • Assignee set to dalley
  • Parent issue deleted (#4762)

Design discussion happening here: https://hackmd.io/OIIpE4GUTXKYoQa8lRMgww?both

Planning happening on the epic https://pulp.plan.io/issues/4762

Also available in: Atom PDF