Story #3209

Updated by almost 3 years ago

This is the parent task for all stories/tasks related to adding RepositoryVersions.

h2. Repository Version CRUD


New Repository Version can be created in 2 ways, (1) directly with the Repository Versions Viewset (3234) and (2) action endpoints defined by each plugin (see "actions" below).

Manual add/remove is done by a POST to /v3/repositories/<repo_id>/versions/. The user specifies the content to add and the content to remove in the POST body. The RepositoryVersions Viewset will deploy the task that creates a RepositoryVersion.

Stories: Create a new Repository Version
/v3/repositories/<repo_uuid>/versions/ POST

h3. LIST:

list (v3/repositories/<repo_uuid>/versions/) and read (v3/repositories/<repo_uuid>/versions/<number>/) is handled by the Repository Version serializer and provides content href, added href, and removed href.

Stories: Add detail routes to viewset and serializer
/v3/repositories/<repo_uuid>/versions/<number>/ should include links to detail routes, including:
* content href
* added content href
* removed content href On Repositories Serializer, return href for latest version Filters for RepositoryVersions

Bugs: - bugfix - v3/repositories/<deleted_repo>/versions/ should return 404

h3. UPDATE: disabled

RepositoryVersions are immutable, so the Viewset should not inherrit from the update mixin, so PUT and PATCH requests return a 405 (Method not allowed)


All versions are able to be deleted. The latest version is simply deleted and the latest version becomes the previous latest. When a version other than latest is deleted, the changes made by that version are "squashed" into the following version.

For example, Repository Av1 contains 2 packages (a and b). Av2 Adds version package c. Av3 removes package a. Av4 removes package b.
Av1 (a, b)
Av2 (a, b, c) [+c]
Av3 (b, c) [-a]
Av4 (c) [-b]

If Av2 is deleted, Av3 now also adds "c". Av3(b, c) [+c, -a]
If Av4 is deleted, A.latest() become Av3 and contains (b, c)

When a user deletes a repository version, the content contained by all following versions does not change, but the content added and removed in the following version changes.
When the latest version is deleted, Repository.latest rolls back to the previous version.

Stories: Delete Repository Versions:
RepositoryVersion.latest is deleted
RepositoryVersion that is not latest is "squashed"

Bugs Squash bug

h2. Actions that create a Repository Version (including sync)

If Pulp performs an action to create a Repository Version (as opposed to a manual add/remove), all of the logic is owned by each plugin. Interactions with the pulpcore database (RepositoryVersion, Repository, RepositoryContent) are handled by pulpcore utilities that are part of the plugin API. The utilities will provide a working directory, ensure that writes happen in transactions, clean up pulpcore database after failures and crashes.

This change alters the plugin API. "sync", "publish" and other actions will be tasks that are written by each plugin (instead of a function on the model). The tasks are deployed by Plugin Viewsets which create action endpoints in the REST API.

Transactions and Cleanup stories: Add complete flag, and mark complete when action is finished. Cleanup incomplete Repository Versions after a worker crash Update Changeset with new arguments (repo_version, not old_version and new_version)

Plugin API changes: Remove sync code from pulpcore. Also acts as parent task for Plugin Stories: Provide context manager to handle RepositoryVersion creation, finalization, and cleanup.

Plugin Stories File plugin sync - Example plugin sync - Python plugin sync

h2. Actions that create a Publication (including publish)

Actions that create Publications are handled similarly to actions the create RepositoryVersions. All of the logic is owned by each plugin. Interactions with the pulpcore database (Publication, Repository) are handled by pulpcore utilities that are part of the plugin API.

Stories: Move publish code from model to task Write util to assist with publish (context manager?)

Plugin stories: file publish story example publish story

h2. Docs for Plugin Writer: Plugin writer HOWTO guide to implement sync (and copy) Plugin writer HOWTO guide to implement publish

h2. Plugin Docs: Update file plugin docs for repository versions Update example plugin docs for repository versions Update python plugin docs for repository versions

h2. Why this design?


The ContentUnits added to a repository change over time, but Pulp only stores the current state. This prevents a lot of great things like rolling back to a particluar snapshot, or understanding exactly what content is in place at any given time.

h3. h2. Solution

Introducing the Repository Version, which represents the ContentUnits in a repository after any given operation. For example, each of the following operations will produce a repository version.

* Sync with from an importer
* Add a set of units to a repository
* Remove a set of units from a repository

Any repository version can be deleted. Publications are changed to publish a specific RepositoryVersion, not just the latest content in a given repo.

h3. h2. API examples

POST /api/v3/importers/{type}/{importer_id}/sync/ Use plugin to create a new RepositoryVersion.
/api/v3/repositories/{repo_id}/versions/ # create the new RepositoryVersion. specify the importer here, and/or units to add, and/or units to remove
GET /api/v3/repository/{repo_id}/versions/{number} # The resource for a single repo version
GET /api/v3/repositories/{repo_id}/versions/<id>/content/ # paginated list of all content in particular version
GET /api/v3/repositories/{repo_id}/versions/<id>/added_content/ # paginated list of content added when a particular version was created
GET /api/v3/repositories/{repo_id}/versions/<id>/removed_content/ # paginated list of content removed when a particular version was created

h2. Technical details

The _RepositoryVersion_ table stores all repository versions for all repositories. A repository version is a set of content.

* [pk] *id* - The primary key.
* [fk] *content* - The associated repository.
* *created* - When the repository version was created.
* *number* - A positive integer that uniquely identifies a version of a specific repository. Each new version for a repository should have this field set to 1 + the most recent version.
* *complete* - When true, the action that creates the RepositoryVersion is finished and the version is ready to be used.

The _RepositoryContent_ table stores association between a repository version and its contained content.

* [pk] *id* - The primary key.
* [fk] *content* - The associated content.
* [fk] *repository* - The associated repository.
* [fk] *version_added* - The RepositoryVersion which added the referenced Content.
* [fk] *version_removed* - The RepositoryVersion which removed the referenced Content.