Pulp 3 Minimum Viable Product » History » Revision 133
Revision 132 (bmbouter, 12/12/2017 05:15 PM) → Revision 133/167 (amacdona@redhat.com, 12/12/2017 06:34 PM)
# Pulp 3.0.0 Minimum Viable Product (MVP)
<span style="color:red;">Lines highlighted in red need more attention.</span>
## Overall Guarantees
- This is not a direct replacement for Pulp 2.
- This is the minimum required for a 3.0.0 beta and GA.
- All REST API calls will update the DB using transactions as necessary to ensure data integrity.
## Legend
\[done\] means merged and documented
\[in-progress\] means started but not fully done
If there is no label the effort has not yet been started
## Authentication
As an authenticated user I can manage user(s). \[done\]
- Add a user
- View user(s)
- Update any user detail
- Delete a user
As an API user, I can have documentation to generate a JSON Web Token (JWT) without the server being online. \[done\]
As an administrator, I can disable JWT token expiration. This configuration is in the settings file and is system-wide. \[done\]
As an administrator, I can configure the JWT tokens to expire after a configurable amount of time. This configuration is in the settings file and is system-wide. \[done\]
The JWT shall have a username identifier \[done\]
As an API user, I can authenticate any API call with a valid username and password \[done\]
As an API user, I can authenticate any API call with a valid JWT \[[3163](https://pulp.plan.io/issues/3163)\]
As a JWT authenticated user, I can refresh my JWT token if Pulp is configured with JWT_ALLOW_REFRESH set to True (default is False) \[[3163](https://pulp.plan.io/issues/3163)\]
As an API user, I can invalidate all existing JWT tokens for a given user. \[done\]
As an authenticated user, when deleting a user 'foo', all of user 'foo's existing JWTs are invalidated. \[done\]
As an un-authenticated user, I can obtain a JWT token by using a username and password. \[done\]
As an authenticated user, I can filter users by: \[[3142](https://pulp.plan.io/issues/3142)\]
- username: (equality, username_in_list)
## Repositories
As an authenticated user, I can list all repos.
- All fields are included \[done\]
- Pagination is supported \[done\]
As an authenticated user I can use filters on Repositories list: \[[3079](https://pulp.plan.io/issues/3079)\]
- id: (id_in_list) \# id equality is not necessary, v3/repositories/\<UUID\>/
- name: (equality, name_in_list)
As an authenticated user, I can CRUD a repository
- Create a repo \[done\]
- Read a repo \[done\]
- Update all mutable repo fields \[done\]
- Delete a repo (asynchronous)
As an authenticated user, I can list a repository's associated importers and publishers
- All fields are included \[done\]
- Pagination is supported \[done\]
As an authenticated user, I can see the number of content unit types with counts for each \[done\]\[[3059](https://pulp.plan.io/issues/3059)\]
As an authenticated user, when viewing a repository, I can discover a URL to the latest version of a repository.
## Importers
note: Importer attributes will commonly be available on importers, but aren't guaranteed to be used by all importers.
As an authenticated user, I can CRUD an importer
- Create an importer
- Read an importer
- Update all mutable importer fields
- Delete an importer (asynchronous)
As an authenticated user, I have filters on the Importer list: \[[3080](https://pulp.plan.io/issues/3080)\]
- id: (id_in_list) \# id equality is not necessary, objects are referenced by id
- name: (equality, name_in_list)
As an authenticated user I can configure the following attributes on an Importer: \[done\]
- validate (bool) \[optional: defaults to True\]: If true, the plugin will validate imported content.
- ssl_ca_certificate (str) \[optional\] String containing a PEM encoded CA certificate used to validate the server certificate presented by the external source.
- ssl_client_certificate (str) \[optional\] Contains a PEM encoded client certificate used for authentication.
- ssl_client_key (str) \[optional\] Contains a PEM encoded private key used for authentication.
- ssl_validation (bool) \[optional: defaults to True\]: If true, SSL peer validation must be performed.
- proxy_url (str) \[optional\] Contains the proxy URL. Format: scheme://user:password@host:port.
- username (str) \[optional\] Contains the username to be used for authentication when syncing.
- password (str) \[optional\] Contains the password to be used for authentication when syncing.
- <span style="color:orange;">download_policy (str) \[optional\] Contains the downloading policy name. This is a choice of three options:
\- immediate (default) - Downloading occurs during sync. The sync task does not complete until downloading is done.
\- background - Downloading is started by the sync but occurs in the background. The sync task completes before downloading is complete.
\- on-demand - The sync task records everything that would be downloaded but does not download content. Downloading occurs on demand as driven by client requests for content.</span>
- feed_url (str) \[optional\] Contains the URL of an external content source. This is optional.
- <span style="color:orange;">sync mode: (str) \[optional\]. It has two choices 'additive' and 'mirror':
\- additive (default) - all remote content is added to the base repo version on sync. During sync no content is removed from the base repo version.
\- mirror - the local content will mirror the remote content exactly, removing local content if not also present in the remote content.</span>
- name (str) \[required\] Contains the name.
- last_updated (datetime) \[read-only\] Contains the datetime of the last importer update.
- last_synced (datetime) \[read-only\] Contains the datetime of the last importer sync.
As an authenticated user, I can define a default importer on a repo via href to be used whenever a new repo version is created.
## Publishers
note: Publisher attributes will commonly be available on publishers, but aren't guaranteed to be used by all publishers.
As an authenticated user, I can CRUD a publisher
- Create a publisher
- Read a publisher
- Update all mutable publisher fields
- Delete a publisher (asynchronous)
As an authenticated user, I have filters on the Publisher list: \[[3081](https://pulp.plan.io/issues/3081)\]
- id: (id_in_list) \# id equality is not necessary, objects are referenced by id
- name: (equality, name_in_list)
As an authenticated user I can configure the following attributes on a Publisher:
- relative_path (str) \[optional\] The (relative) path component of the published url.
- name - (str) \[required\] contains the name.
- last_published (datetime) \[read-only\] When the last successful publis%{color:red}h occurred.
- last_updated (datetime) \[read-only\] The datetime of the last publisher update.
- <span style="color:red;">auto_publish(bool) - ??? consider adding auto-publish feature to MVP</span>
## Distributions
As an authenticated user, I can CRUD Distributions:
- Create a Distribution.
- Read a Distribution
- List Distributions
- Update all mutable Distribution fields (synchronous)
\- base path
\- http
\- https
- Delete an Distribution (synchronous)
As a user, my distribution base paths don't conflict and my create/update is rejected identifying the conflicting distributions \[[2987](https://pulp.plan.io/issues/2987)\]
As an authenticated user, I can create or update a distribution that is not associated with any publication (NULL)
As an authenticated user, I can create or update a distribution that is not associated with any publisher (NULL)
As a user, I can update a Distribution to distribute a specific Publication
As a user, I want a newly created publication to be automatically served by the content as defined by distributions.
As a user, I can see the full urls my base path is served at (one for http and one for https depending on what is on)
As an authenticated user, I have filters on the Distribution list: \[[3082](https://pulp.plan.io/issues/3082)\]
- name: (equality, name_in_list)
- base_path: (equality, substring, base_path_in_list)
## Publications
As an authenticated user, when creating a Publication, I can post a repo version href to be published.
As an authenticated user, I can publish a repository's latest version by posting a repository href to be published.
As an authenticated user, I can view which repository version was used to create a particular publication.
As an authenticated user, I can read Publication(s)
- Read a Publication - id, created datetime, list of distribution hrefs, repo version
- List all Publications - ordered by created datetime in descending order
As an authenticated user, I can delete publications.
\- asynchronously with a lock on the repository version.
\- prevented if associated with a distribution.
\- single object only.
As an authenticated user, I can list publications and I have enough information to choose which ones to delete.
- choose by created (older ones or perhaps latest)
- choose not associated to a distribution.
- does not imply filtering
As an authenticated user, I can list publications and i have enough information to select a publication to be associated with a distribution.
- choose by created (latest or just by publish date)
\- manual promotion. "My rawhide publication has been tested and now I want to promote it to stable".
\- rollback to an earlier publication.
- does not imply filtering
As an authenticated user, I can determine if and how a publication is distributed.
## Sync and Publish
As an authenticated user, I can trigger an importer to sync. \[done\]
- I can follow the progress of all syncs. (Syncs are asynchronous.)
- I cannot pass "sync" options.
- Auto-publish is not included as an importer property.
As an authenticated user, I can trigger a publisher to publish. \[done\]
- I can follow the progress of all publishes. (Publishes are asynchronous.)
- I cannot pass "publish" options.
## Content Manipulation
#### Uploading Artifacts
As an authenticated user, I can create an Artifact by uploading a file. \[done\]
As an authenticated user, I can specify a size and/or digest to validate the uploaded file. \[done\]
#### Creating Content Units
As an authenticated user, I can create a content unit by providing the content type (in the URL), references to Artifacts, and the metadata supplied in the POST body. \[done\]
#### Simple Copy
As an authenticated user, I can search (synchronous call) a repository's content using filtering.
<span style="color:red;">\* note: In a separate call (async), I can add this content to another repository. This is effectively a copy operation in two calls.</span>
As an authenticated user, I can import all content from one repository into another repository in a single async call. (Clone use case)
- I can follow the progress. (adding are asynchronous).
#### Complex Copy
As a plugin writer I can provide a rich search features with abitrary viewsets. e.g. depsolving, verisoning, etc
Examples of specific plugin use cases motivating the above general viewset
- As an authenticated user, I can add an Errata from one repository to another repository along with packages mentioned in the Errata and all their dependencies that are present in the source repository.
- As an authenticated user, I can add bunch of dependencies and store n versions back for each RPM.
- As an authenticated user, I can use depsolving and versioning copy features together
- As an authenticated user, I can copy a puppet module and all of it's dependencies from one repository to another.
- As an authenticated user, I can depsolve units to be added to a destination repo based on an errata
#### Delete
As an authenticated user, I can delete a specific content unit
- If the content unit is still in at least one repository version the delete fails
- Error message saying that the unit is in use by a repo version and a link to the filter to return all of the repo versions.
- Content unit deletion needs to be race condition free. For example: cannot delete a content unit during (for example)
\- sync that may be assuming the content exits.
\- copy operation
<!-- end list -->
- As a user, I know that files (Artifacts) associated with the Content unit are not removed by this call (docs)
#### Filtering
As an authenticated user, I can filter Content by repository version information when plugin writers have provided such a filter
As an authenticated user, I can filter content by repository version
As an authenticated user, I can filter content by type
As an authenticated user, I can filter content on plugin specific attributes when plugin writers have provided such a filter
## Versioned Repositories
#### CRD
As an authenticated user, I can create a new repository version. \[[3173](https://pulp.plan.io/issues/3173)\]
As an authenticated user, I can list versions for a particular repository.
As an authenticated user, I can filter repository versions by:
- number (equality, lt/lte, gt/gte)
- content id (equality, content_in_list)
- <span style="color:red;">created datetime (range)</span>
As an authenticated user, I can delete any repository version.
As an authenticated user, I can view attributes on a repository version including its base_version, importer, added/removed content ids.
#### Repository Version Content
As an authenticated user, I can retrieve read-only immutable content sets by href.
As an authenticated user, I can list the content in a particular repository version
- All fields are included
- Pagination is supported
- <span style="color:red;">Filtering support</span>
As an autheticated user, I can create a new version by adding or removing content to a particular base version (default is latest).
- I can follow the progress. (adding/removing are asynchronous).
- can add and remove in a single call
As an authenticated user, I can specify the base_version of content to be the basis for the new repository version. This defaults to the latest repo version associated with the repository.
## Orphan Content Units and Artifacts
As an authenticated user, I can cause an action that cleans up **both** orphaned content units and orphaned artifacts.
- I cannot specify the units specifically (all types).
- I can follow the progress of all cleanups. (Cleanups are asynchronous.)
## Task Management
As an authenticated user, I can list all tasks
- <span style="color:orange;">Filtering support on \['state', 'id', 'group'\]</span>
- <span style="color:orange;">This does not include associated progress reports</span>
As an authenticated user, I can see a detail view for a specific task \[done\]
- all attributes of a task
- all associated progress reports
As an authenticated user, I can cancel a task \[done\]
- don't dare to use the DELETE verb!
As an authenticated user, I can delete tasks.
As an authenticated user, I can filter tags by: \[[3144](https://pulp.plan.io/issues/3144)\]
- state (equality, state_in_list)
- started_at(started_in_range)
- finished_at(finished_in_range)
- worker (equality)
- parent (equality)
## Task Group
<span style="color:red;">I can view a summary of the status of all tasks in a group</span>
## Status
As an unauthenticated user I can view the status of Pulp workers and resource managers. \[done\]
As an unauthenticated user I can view the status of the web server's connection to the database and message broker. \[done\]
As an unauthenticated user I can view the versions of core and each installed plugin.
## Workers
As an authenticated user, I can filter workers by: \[[3143](https://pulp.plan.io/issues/3143)\]
- last_heartbeat (range)
- name (substring)
- gracefully_stopped (equality)
## Plugin API
As a plugin writer, I have a plugin API that is semantically versioned at 0.x separate from the REST API \[done\]
As a plugin writer, I can report progress with a message and state \[done\]
As a plugin writer, I can report progress with an optional suffix \[done\]
As a plugin writer, I can report progress with a total count of things to do an the current count of things done \[done\]
As a plugin writer, non-fatal exceptions on the Task and are included in the Task detail. non_fatal exceptions do not cause the Task to be marked as failed, but may be interpreted by the user as not fully successful. \[done\]
As a plugin writer, the working directory is set before Task work is done and cleaned up afterwards. I should not need to interact with the file system outside of the working dir. \[done\]
<span style="color:red;">As a plugin writer, I can provide a subclassed Importer. The importer's responsibility is to synchronize the content of a Pulp repository with the content of a remote repository. (a circular import problem needs to be discussed and may cause this to change) \[done\]</span>
<span style="color:red;">As a plugin writer, I can provide a subclassed Publisher. The publisher's responsibility is to publish content. (a circular import problem needs to be discussed and may cause this to change) \[done\]</span>
As a plugin writer, I can define unit types by subclassing Content models to provide concrete content unit types to be manged by the platform. \[done\]
As a plugin writer, I can interact with and create Artifacts \[done\]
As a plugin writer, my app will be discovered by Pulp's app via an entry point provided by the plugin writer \[done\]
As a plugin writer, I can use the plugin API to query content units/artifacts associated with a repository. \[done\]
As a plugin writer, I can add and remove content units to and from a repository. \[done\]
As a plugin writer, I have documentation that shows how I can add filters to filter content responsibly.
As a plugin writer, I have documentation on how to write a filter for my Content that can use the RepositoryContent through table.
- note: This will allow users to filter content by repository information
## Plugin Writer "live APIs"
As a plugin writer, I can provide a Live API with arbitrary views and viewsets.
As a plugin writer, I have documentation on what URLs I should -not use for my views and viewsets
Here are some concrete use cases driving the very Live API use cases above:
~~~
# Concrete user use cases:
As an authenticated user, I can use the puppet client to fetch content from Pulp using the Forge API
As an authenticated user I can use the docker client to fetch content from Pulp using the Docker v1 API
As an authenticated user I can use the docker client to fetch content from Pulp using the Docker v2 API
# Concrete plugin writer use cases
As a puppet plugin developer, I can provide a viewset which handles the server side of the puppet Forge v3 API
As a docker plugin developer, I can provide a viewset which handles the server side of the docker v1 API
As a docker plugin developer, I can provide a viewset which handles the server side of the docker v2 API
~~~
## Webserver Deployment
As a system administrator, I can deploy all Pulp web applications on one process
As a system administrator, I can deploy the Pulp REST API exclusively in one process
As a system administrator, I can deploy the Pulp content serving view exclusively in one process
As a system administrator, I can deploy all Pulp web applications inside a virtualenv.
As a system administrator, I can deploy all Pulp web applications without root permissions.
## CLI
We will use coreapi-cli to generate a one to one mapping of cli commands to rest api schema #3068
<span style="color:orange;">We will have a wrapper for coreapi-cli. This wrapper will handle parallel progress reporting</span>
## Download API
As a plugin writer, I can download files via
- http://
- https://
- file://
As a plugin writer, I can configure a downloader with:
- Basic Auth
- SSL Cert Client Auth
- Custom CAs will be configured via a "trust store" either on the system or similar. Pulp will not do anything to read/load/manage CAs directly.
As a plugin writer, I can provide arbitrary behaviors for customized downloaders
- For example token authentication in the docker plugin
As a plugin writer, I can have connection pooling/reuse
As a plugin writer, I have proxy settings
- proxy url (containing basic auth info)
As a plugin writer, I can have great logs
As an authenticated user, I have documentation about how to use something for bandwidth limiting
As a plugin writer, I can configure the validation mechanisms used at download time
- checksum validation - minimum (md5, sha1, sha256, sha512)
- size validation
<span style="color:red;">As a plugin writer, I expect units that are missing from the remote repository to not be created in Pulp when using the immediate download policy.</span>
<span style="color:red;">As a plugin writer, I expect units that are missing from the remote repository to be created in Pulp when using background or on_demand download policies.</span>
As a plugin writer, the plugin API provides tooling whereby I can provide the content to be added and removed from the repository. This tooling supports both immediate and deferred downloading.
As a plugin writer I can manage the catalog by using ChangeSets
As a plugin writer, the plugin can participate in adding content for cases where the decision to add additional content is based content that has been downloaded.
As a plugin writer, I can fetch content myself (but I am not encouraged to do so) with code I write
As a plugin writer, I can CRUD content units
## <span style="color:red;">Consumer Profile Applicability</span>
<span style="color:red;">Using Consumer Profiles and repository bindings I can compute applicability with 2.y parity</span>
- Performance needs to be awesome%
## Migrations only involving Pulp 3
Users can run "pulp-manager migrate" to migrate the database and adjust state in other locations (filesystem, message broker, ...). \[done\]
## Web Server Integration
As a user, I can have content efficiently served to me by Apache by Pulp using the X-SEND response headers. \[done\]
As a user, I can have content efficiently served to me by Nginx by Pulp using the X-Accel-Redirect response headers. \[done\]
As a user, I can have an Ansible role to install Apache which enables Apache integration for Pulp and configures Apache to serve Pulp. \[[2921](https://pulp.plan.io/issues/2921)\]
As a user, I can have an Ansible role to install Nginx which enables Nginx integration for Pulp and configures Nginx to serve Pulp. \[[2922](https://pulp.plan.io/issues/2922)\]
## Glossary
Add (Content Unit): An operation causing a new repository version to contain a content unit(s)
Applicability - A plugin defined term meaning when a package update available in a repository is applicable to a given consumer as determined by the Consumer Profile.
Artifact - A file associated with one content (unit). Artifacts are not shared between content (units). Create a content unit using an uploaded file ID as the source for its metadata. Create Artifacts associated with the content unit using an uploaded file ID for each; commit as a single transaction.
Consumer Profile - A set of installed units on a specific machine. In Pulp3 this machine is not a "consumer" in the same sense as Pulp2. Pulp is not "managing" the machine anymore; Pulp3 only collects Profile information.
Content (unit) - A single piece of content manged by Pulp. Each file associated with a content (unit) is called an Artifact. Each content (unit) may have zero or many Artifacts.
Distribution: Where and how the content app serves a Publication. i.e. http vs https and base path component of the URL. A Distribution defines:
- the base path of the repository publication (required)
- serving via http (default=False)
- serving via https (default=True)
- relationship w/ a Publisher for auto-distribution (should be allowed to be NULL)
- relationship with Publication (should be allowed to be NULL)
Live API: a viewset endpoint contributed by plugin. For examples see the [associated MVP section](https://pulp.plan.io/projects/pulp/wiki/Pulp_3\_Minimum_Viable_Product#Plugin-Writer-live-APIs)
Orphan Artifact: An Artifact that is associated with 0 Content Units and 0 Publications
Orphan Content (unit): A content unit that is a member of 0 repository versions
Remove (content unit): An operation causing a new repository version to not contain a content unit(s)
Repository - A named collection of repository versions.
Repository Version - An immutable set of content which is versioned by a sequential number.
.