Story #5344
closedAs a user, I have a consistant API experience for copying content
0%
Description
Problem Statement:
API design for copy and upload has thus far been somewhat haphazard and it would be useful to make it more consistent. Additionally there are some deficiencies in some of the current APIs that need to be addressed.
For example, the copy API currently looks something like this
POST /pulp/api/v3/rpm/copy/ types=['package', 'modulemd']
The problem with this API is that it will be difficult to make it play nicely with filtering, which has yet to be implemented. The filtering use case is essentially "copy units for which metadata matches these filters, e.g. name=walrus and version>1.1.0". However, because different unit types have different metadata fields, allowing the user to specify filters while also allowing more than one type of content to be copied at once would be problematic because there's no good way to determine what filters should apply to what content types. Possibly it could be done, but it would probably not be a good idea. Therefore it is best that the "types" parameter be removed entirely.
Sidenote: In various discussions we've mentioned the utility (and possibly necessity) of being able to have a group of tasks that are atomic and transactional together. Something like this could be applied here in order to replace this functionality (however, it is out of scope for this Story).
There are 2 different alternative APIs I would like to propose:
1. The same API, but with a single type copied at a time. If "all" is specified as the type, it would copy everything. This would IMO cover the most important use cases.
POST /pulp/api/v3/rpm/copy/ type='package'
Pros:
- More flexible, more similar to the Pulp 2 API
Cons:
- Dissimilar to the other API's in Pulp 3
- "type" is going to be the name of a field on the RPM content itself (inherited from core, formerly _type)
2. Make the type to be copied part of the name of the endpoint itself, like so:
POST /pulp/api/v3/rpm/package/copy/
Pros:
- Consistent with other Pulp 3 APIs
Cons:
- Not as flexible, no way to copy all units at once via this API
3. Use a complex filter endpoint
POST /pulp/api/v3/rpm/copy/ filter={'package': "name='foo', version>='4.5', arch='x86_64'", 'modulemd': "name=ripgrep, stream=master"}
Pros:
- Might be able to support all the use cases we want in one API endpoint
Cons:
- Complicated to implement
Use cases¶
- As a user, I can copy all content of a specific type from one repository to another repository
- As a user, I can copy content matching certain "search criteria" from one repository to another repository
- As a user, when copying content that directly references other content, the referenced content is ?always? copied
* e.g. Modules referencing Modules, Modules referencing RPMs, Erratum referencing RPMs, and {{other types}} - As a user, I can optionally choose to copy all indirect dependencies of content that is being copied (recursive copy)
* Should this be the default? - Some content types create a new content unit in the destination repo instead of just copying, e.g. yum_repo_metadata_file
- Special copy needed for modules? (multi-repo)
- Special copy needed for distribution trees?
Related issues
Updated by dalley about 5 years ago
- Tracker changed from Task to Story
- Subject changed from Implement a consistent copy and upload API compatible with filtering to As a user, I have a consistant API experience for copying and uploading content
- Description updated (diff)
Updated by dalley about 5 years ago
- Subject changed from As a user, I have a consistant API experience for copying and uploading content to As a user, I have a consistant API experience for copying and uploading content [EPIC]
- Description updated (diff)
Updated by dalley about 5 years ago
- Subject changed from As a user, I have a consistant API experience for copying and uploading content [EPIC] to [EPIC] As a user, I have a consistant API experience for copying and uploading content
- Description updated (diff)
Updated by dalley about 5 years ago
- Subject changed from [EPIC] As a user, I have a consistant API experience for copying and uploading content to [EPIC] As a user, I have a consistant API experience for copying content
- Description updated (diff)
Updated by ttereshc about 5 years ago
- Subject changed from [EPIC] As a user, I have a consistant API experience for copying content to As a user, I have a consistant API experience for copying content
Updated by kersom about 5 years ago
- Related to Test #5276: Test - As a user, I have simple content copy between repositories added
Updated by ggainey about 5 years ago
Discussion:
- We will always want an easy 'complete copy/clone' API. You can get this currently via creating a repo-version and specifying a base-version
- There is a use-case for filtering, either by-type (or possibly diff-filters-per-type?)
- There is a use-case for "one version created for a copy even when multiple filters are in play"
Proposal:
- For initial GA, remove/hide /copy endpoint
- For initial GA, let 'clone this repo' be supported via the create-repo-version-from-base mentioned above.
- Further investigation on how to do filtering needed, with an additive API change post-GA. Need sub-tasks for this story.
Updated by ggainey about 5 years ago
Possible proposal for multi-type copy and filtering: have filter= be not a string, but a structure/dictionary, key by content-type, value is 'filter string for that type'
Terrible non-real example:
POST /pulp/api/v3/rpm/copy/ types=['package', 'modulemd'] filter={'package': "name='foo', version>='4.5', arch='x86_64'", 'modulemd': "<insert modulemd filtering here?>"}
Obviously, filter-syntax could be Very Complicated. Edge-cases between types (ie, type-1's filter and type-2's filter result in a non-sane repository) are going to be frequent and sharp. Needs some serious thought and doc, with real/working examples.
Updated by ttereshc about 5 years ago
- Related to Task #5681: Remove /pulp/api/v3/rpm/copy/ endpoint added
Updated by ttereshc about 5 years ago
- Tags deleted (
Pulp 3 RPM blocker)
Removing a blocker tag for 3.0. These endpoints will be added after 3.0 GA.
Updated by dalley almost 5 years ago
- Parent issue set to #4762
Use cases, from the RPM plugin roadmap
- As a user, I can copy a content of a specific type from one repository to another repository
- As a user, I can copy RPM packages and all their dependencies from one repository to another repository
- As a user, I can copy an Erratum and all the RPM packages it references from one repository to another repository
- As a user, I can copy an Erratum and all the RPM packages it references and all the RPM dependencies from one repository to another repository
- Some content types create a new content unit in the destination repo instead of just copying, e.g. yum_repo_metadata_file
- Special copy for modules [TBD]
Updated by dalley almost 5 years ago
- Status changed from NEW to CLOSED - DUPLICATE
- Assignee set to dalley
- Parent issue deleted (
#4762)
Design discussion happening here: https://hackmd.io/OIIpE4GUTXKYoQa8lRMgww?both
Planning happening on the epic https://pulp.plan.io/issues/4762