Story #3209
Updated by amacdona@redhat.com almost 7 years ago
This is the parent task for all stories/tasks related to adding RepositoryVersions.
h2. Repository Version CRUD
h3. CREATE:
New Repository Version can be created in 2 ways, (1) directly with the Repository Versions Viewset (3234) and (2) action endpoints defined by each plugin (see "actions" below).
Manual add/remove is done by a POST to /v3/repositories/<repo_id>/versions/. The user specifies the content to add and the content to remove in the POST body. The RepositoryVersions Viewset will deploy the task that creates a RepositoryVersion.
Stories:
https://pulp.plan.io/issues/3234 Create a new Repository Version
/v3/repositories/<repo_uuid>/versions/ POST
h3. LIST:
list (v3/repositories/<repo_uuid>/versions/) and read (v3/repositories/<repo_uuid>/versions/<number>/) is handled by the Repository Version serializer and provides content href, added href, and removed href.
Stories:
https://pulp.plan.io/issues/3218 Add detail routes to viewset and serializer
/v3/repositories/<repo_uuid>/versions/<number>/ should include links to detail routes, including:
* content href
* added content href
* removed content href
https://pulp.plan.io/issues/3235 On Repositories Serializer, return href for latest version
https://pulp.plan.io/issues/3238 Filters for RepositoryVersions
Bugs:
https://pulp.plan.io/issues/3230 - bugfix - v3/repositories/<deleted_repo>/versions/ should return 404
h3. UPDATE: disabled
RepositoryVersions are immutable, so the Viewset should not inherrit from the update mixin, so PUT and PATCH requests return a 405 (Method not allowed)
h3. DELETE:
All versions are able to be deleted. The latest version is simply deleted and the latest version becomes the previous latest. When a version other than latest is deleted, the changes made by that version are "squashed" into the following version.
For example, Repository Av1 contains 2 packages (a and b). Av2 Adds version package c. Av3 removes package a. Av4 removes package b.
Av1 (a, b)
Av2 (a, b, c) [+c]
Av3 (b, c) [-a]
Av4 (c) [-b]
If Av2 is deleted, Av3 now also adds "c". Av3(b, c) [+c, -a]
If Av4 is deleted, A.latest() become Av3 and contains (b, c)
When a user deletes a repository version, the content contained by all following versions does not change, but the content added and removed in the following version changes.
When the latest version is deleted, Repository.latest rolls back to the previous version.
Stories:
https://pulp.plan.io/issues/3219 Delete Repository Versions:
RepositoryVersion.latest is deleted
RepositoryVersion that is not latest is "squashed"
Bugs
https://pulp.plan.io/issues/3233 Squash bug
h2. Actions that create a Repository Version (including sync)
If Pulp performs an action to create a Repository Version (as opposed to a manual add/remove), all of the logic is owned by each plugin. Interactions with the pulpcore database (RepositoryVersion, Repository, RepositoryContent) are handled by pulpcore utilities that are part of the plugin API. The utilities will provide a working directory, ensure that writes happen in transactions, clean up pulpcore database after failures and crashes.
This change alters the plugin API. "sync", "publish" and other actions will be tasks that are written by each plugin (instead of a function on the model). The tasks are deployed by Plugin Viewsets which create action endpoints in the REST API.
Transactions and Cleanup stories:
https://pulp.plan.io/issues/3222 Add complete flag, and mark complete when action is finished.
https://pulp.plan.io/issues/3226 Cleanup incomplete Repository Versions after a worker crash
https://pulp.plan.io/issues/3225 Update Changeset with new arguments (repo_version, not old_version and new_version)
Plugin API changes:
https://pulp.plan.io/issues/3074 Remove sync code from pulpcore. Also acts as parent task for Plugin Stories:
https://pulp.plan.io/issues/3285 Provide context manager to handle RepositoryVersion creation, finalization, and cleanup.
Plugin Stories
https://pulp.plan.io/issues/3260- File plugin sync
https://pulp.plan.io/issues/3243 - Example plugin sync
https://pulp.plan.io/issues/3294 - Python plugin sync
h2. Actions that create a Publication (including publish)
Actions that create Publications are handled similarly to actions the create RepositoryVersions. All of the logic is owned by each plugin. Interactions with the pulpcore database (Publication, Repository) are handled by pulpcore utilities that are part of the plugin API.
Stories:
https://pulp.plan.io/issues/3221 Move publish code from model to task
https://pulp.plan.io/issues/3295 Write util to assist with publish (context manager?)
Plugin stories:
https://pulp.plan.io/issues/3296 file publish story
https://pulp.plan.io/issues/3297 example publish story
h2. Docs for Plugin Writer:
https://pulp.plan.io/issues/3220 Plugin writer HOWTO guide to implement sync (and copy)
https://pulp.plan.io/issues/3298 Plugin writer HOWTO guide to implement publish
h2. Plugin Docs:
https://pulp.plan.io/issues/3249 Update file plugin docs for repository versions
https://pulp.plan.io/issues/3299 Update example plugin docs for repository versions
https://pulp.plan.io/issues/3300 Update python plugin docs for repository versions
h2. Why this design?
h3. Motivation
The ContentUnits added to a repository change over time, but Pulp only stores the current state. This prevents a lot of great things like rolling back to a particluar snapshot, or understanding exactly what content is in place at any given time.
h3. Solution
Introducing the Repository Version, which represents the ContentUnits in a repository after any given operation. For example, each of the following operations will produce a repository version.
* Sync with an importer
* Add a set units to a repository
* Remove a set of units from a repository
Any repository version can be deleted. Publications are changed to publish a specific RepositoryVersion, not just the latest content in a given repo.
h3. API examples
<pre>
POST /api/v3/importers/{type}/{importer_id}/sync/ Use plugin to create a new RepositoryVersion.
POST /api/v3/repositories/{repo_id}/versions/ # create the new RepositoryVersion. specify units to add, and/or units to remove
GET /api/v3/repository/{repo_id}/versions/{number} # The resource for a single repo version
GET /api/v3/repositories/{repo_id}/versions/<id>/content/ # paginated list of all content in particular version
GET /api/v3/repositories/{repo_id}/versions/<id>/added_content/ # paginated list of content added when a particular version was created
GET /api/v3/repositories/{repo_id}/versions/<id>/removed_content/ # paginated list of content removed when a particular version was created
</pre>
h2. Technical details
The _RepositoryVersion_ table stores all repository versions for all repositories. A repository version is a set of content.
*RepositoryVersion*
* [pk] *id* - The primary key.
* [fk] *content* - The associated repository.
* *created* - When the repository version was created.
* *number* - A positive integer that uniquely identifies a version of a specific repository. Each new version for a repository should have this field set to 1 + the most recent version.
* *complete* - When true, the action that creates the RepositoryVersion is finished and the version is ready to be used.
The _RepositoryContent_ table stores association between a repository version and its contained content.
*RepositoryContent*
* [pk] *id* - The primary key.
* [fk] *content* - The associated content.
* [fk] *repository* - The associated repository.
* [fk] *version_added* - The RepositoryVersion which added the referenced Content.
* [fk] *version_removed* - The RepositoryVersion which removed the referenced Content.
.