Pulp 3 Minimum Viable Product » History » Revision 94
Revision 93 (bizhang, 10/10/2017 11:07 PM) → Revision 94/167 (bizhang, 10/12/2017 08:10 PM)
# Pulp 3.0.0 Minimum Viable Product (MVP) <span style="color:red;">Lines highlighted in red need more attention.</span> ## Overall Guarantees - This is not a direct replacement for Pulp 2. - This is the minimum required for a 3.0.0 beta and GA. - All REST API calls will update the DB using transactions as necessary to ensure data integrity. ## Legend \[done\] means merged and documented \[in-progress\] means started but not fully done If there is no label the effort has not yet been started ## Authentication As an authenticated user I can manage user(s). \[done\] - Add a user - View user(s) - Update any user detail - Delete a user As an API user, I can have documentation to generate a JSON Web Token (JWT) without the server being online. \[done\] As an administrator, I can disable JWT token expiration. This configuration is in the settings file and is system-wide. \[done\] As an administrator, I can configure the JWT tokens to expire after a configurable amount of time. This configuration is in the settings file and is system-wide. \[done\] The JWT shall have a username identifier \[done\] <span style="color:red;">As an API user, I can authenticate any API call (except to request a JWT) with a JWT. (not certain if this should be the behavior) \[in progress\]</span> As an API user, I can invalidate all existing JWT tokens for a given user. \[done\] As an authenticated user, when deleting a user 'foo', all of user 'foo's existing JWTs are invalidated. \[done\] As an autheticated user, I can invalidate a user's JWTs in the same operation as updating the password. \[done\] As an un-authenticated user, I can obtain a JWT token by using a username and password. \[done\] ## Repositories As an authenticated user, I can list all repos. - All fields are included \[done\] - Pagination is supported \[done\] - <span style="color:orange;">Filtering support</span> As an authenticated user, I can CRUD a repository - Create a repo \[done\] - Read a repo \[done\] - Update all mutable repo fields \[done\] - Delete a repo (asynchronous) \[done\] As an authenticated user, I can list a repository's associated importers and publishers - All fields are included \[done\] - Pagination is supported \[done\] <span style="color:orange;">As an authenticated user, I can see the number of content unit types with counts for each</span> ## Importers note: Importer attributes will commonly be available on importers, but aren't guaranteed to be used by all importers. As an authenticated user, I can CRUD an importer - Create an importer - Read an importer - Update all mutable importer fields - Delete an importer (asynchronous) As an authenticated user I can configure the following attributes on an Importer: \[done\] - validate (bool) \[optional: defaults to True\]: If true, the plugin will validate imported content. - ssl_ca_certificate (str) \[optional\] String containing a PEM encoded CA certificate used to validate the server certificate presented by the external source. - ssl_client_certificate (str) \[optional\] Contains a PEM encoded client certificate used for authentication. - ssl_client_key (str) \[optional\] Contains a PEM encoded private key used for authentication. - ssl_validation (bool) \[optional: defaults to True\]: If true, SSL peer validation must be performed. - proxy_url (str) \[optional\] Contains the proxy URL. Format: scheme://user:password@host:port. - username (str) \[optional\] Contains the username to be used for authentication when syncing. - password (str) \[optional\] Contains the password to be used for authentication when syncing. - <span style="color:orange;">download_policy (str) \[optional\] Contains the downloading policy name. This is a choice of three options: \- immediate (default) - Downloading occurs during sync. The sync task does not complete until downloading is done. \- background - Downloading is started by the sync but occurs in the background. The sync task completes before downloading is complete. \- on-demand - The sync task records everything that would be downloaded but does not download content. Downloading occurs on demand as driven by client requests for content.</span> - feed_url (str) \[optional\] Contains the URL of an external content source. This is optional. - <span style="color:orange;">sync mode: (str) \[optional\]. It has two choices 'additive' and 'mirror': \- additive (default) - all remote content is added to the local repository on sync. During sync no content is removed from the local repository. \- mirror - the local content will mirror the remote content exactly, removing local content if not also present in the remote content.</span> - name (str) \[required\] Contains the name. - last_updated (datetime) \[read-only\] Contains the datetime of the last importer update. - last_synced (datetime) \[read-only\] Contains the datetime of the last importer sync. ## Publishers note: Publisher attributes will commonly be available on publishers, but aren't guaranteed to be used by all publishers. As an authenticated user, I can CRUD a publisher - Create a publisher - Read a publisher - Update all mutable publisher fields - Delete a publisher (asynchronous) As an authenticated user I can configure the following attributes on a Publisher: - relative_path (str) \[optional\] The (relative) path component of the published url. - name - (str) \[required\] contains the name. - last_published (datetime) \[read-only\] When the last successful publish occurred. - last_updated (datetime) \[read-only\] The datetime of the last publisher update. - <span style="color:red;">auto_publish(bool) - ??? consider adding auto-publish feature to MVP</span> ## Sync and Publish As an authenticated user, I can trigger an importer to sync. \[done\] - I can follow the progress of all syncs. (Syncs are asynchronous.) - I cannot pass "sync" options. - Auto-publish is not included as an importer property. As an authenticated user, I can trigger a publisher to publish. \[done\] - I can follow the progress of all publishes. (Publishes are asynchronous.) - I cannot pass "publish" options. ## Content Manipulation #### Uploading Artifacts As an authenticated user, I can create an Artifact by uploading a file. \[done\] As an authenticated user, I can specify a size and/or digest to validate the uploaded file. \[done\] #### Creating Content Units As an authenticated user, I can create a content unit by providing the content type (in the URL), references to Artifacts, and the metadata supplied in the POST body. \[done\] #### Content Management / Copy <span style="color:red;">As an authenticated user, I can add and remove one or more units to and from a destination repo.</span> - <span style="color:red;">Filtering support for specifying the unit(s)</span> - <span style="color:red;">I can follow the progress. (adding and removing are asynchronous).</span> #### <span style="color:red;">Content Removal</span> ## <span style="color:red;">Versioned Repositories</span> As an authenticated user, I can list the content in a particular repository version - All fields are included - Pagination is supported - <span style="color:red;">Filtering support</span> As an authenticated user, I can discover a URL to the latest version of a repository <span class="resource repository the on attributes or endpoint, API dedicated a through \^ Is" style="color:red;"></span> As an authenticated user, I can run a publisher without a repository version and have it default to the latest version. <span style="color:red;">As an authenticated user, I can delete a repository version by specifying the version</span> <span style="color:red;">As an authenticated user, I can upload multiple content(s?) and add create a single new version that adds all of them.</span> ## Orphans <span style="color:red;">As an authenticated user, I can clean up orphaned content units</span> <span style="color:red;">\* I can follow the progress of all cleanups. (Cleanups are asynchronous.)</span> <span style="color:red;">As an authenticated user, I can delete a specific content unit</span> <span style="color:red;">\* If the content unit is still in at least one repository the delete fails with a listing of all repositories the unit is part of.</span> <span style="color:red;">\* Artifacts and associated files from the deleted unit are cleaned up</span> <span style="color:red;">As an authenticated user, I can delete multiple content units with filtering</span> <span style="color:red;">\* If a content unit is still in at least one repository the delete fails with a listing of all repositories the unit is part of.</span> <span style="color:red;">\* Artifacts and associated files from deleted units are cleaned up</span> <span style="color:red;">As an authenticated user, I see all (orphans) units that are not in any repositories</span> ## Task Management As an authenticated user, I can list all tasks - <span style="color:orange;">Filtering support on \['state', 'id', 'group'\]</span> - <span style="color:orange;">This does not include associated progress reports</span> As an authenticated user, I can see a detail view for a specific task \[done\] - all attributes of a task - all associated progress reports As an authenticated user, I can cancel a task \[done\] - don't dare to use the DELETE verb! As an authenticated user, I can delete tasks. ## Task Group <span style="color:red;">I can view a summary of the status of all tasks in a group</span> ## Status As an unauthenticated user I can view the status of Pulp workers, resource managers, and celerybeats. \[done\] As an unauthenticated user I can view the status of the web server's connection to the database and message broker. \[done\] As an unauthenticated user I can view the versions of core and each installed plugin. ## Plugin API As a plugin writer, I have a plugin API that is semantically versioned at 0.x separate from the REST API \[done\] As a plugin writer, I can report progress with a message and state \[done\] As a plugin writer, I can report progress with an optional suffix \[done\] As a plugin writer, I can report progress with a total count of things to do an the current count of things done \[done\] As a plugin writer, non-fatal exceptions on the Task and are included in the Task detail. non_fatal exceptions do not cause the Task to be marked as failed, but may be interpreted by the user as not fully successful. \[done\] As a plugin writer, the working directory is set before Task work is done and cleaned up afterwards. I should not need to interact with the file system outside of the working dir. \[done\] <span style="color:red;">As a plugin writer, I can provide a subclassed Importer. The importer's responsibility is to synchronize the content of a Pulp repository with the content of a remote repository. (a circular import problem needs to be discussed and may cause this to change) \[done\]</span> <span style="color:red;">As a plugin writer, I can provide a subclassed Publisher. The publisher's responsibility is to publish content. (a circular import problem needs to be discussed and may cause this to change) \[done\]</span> As a plugin writer, I can define unit types by subclassing Content models to provide concrete content unit types to be manged by the platform. \[done\] As a plugin writer, I can interact with and create Artifacts \[done\] As a plugin writer, my app will be discovered by Pulp's app via an entry point provided by the plugin writer \[done\] As a plugin writer, I can use the plugin API to query content units/artifacts associated with a repository. \[done\] As a plugin writer, I can add and remove content units to and from a repository. \[done\] ## CLI We <span style="color:orange;">We will use coreapi-cli to generate a one to one mapping of cli commands to rest api schema #3068 schema</span> <span style="color:orange;">We will have a wrapper for coreapi-cli. This wrapper will handle parallel progress reporting</span> ## Download API As a plugin writer, I can download files via - http:// - https:// - file:// As a plugin writer, I can configure a downloader with: - Basic Auth - SSL Cert Client Auth - Custom CAs will be configured via a "trust store" either on the system or similar. Pulp will not do anything to read/load/manage CAs directly. As a plugin writer, I can provide arbitrary behaviors for customized downloaders - For example token authentication in the docker plugin As a plugin writer, I can have connection pooling/reuse As a plugin writer, I have proxy settings - proxy url (containing basic auth info) As a plugin writer, I can have great logs As a user, I have documentation about how to use something for bandwidth limiting As a plugin writer, I can configure the validation mechanisms used at download time - checksum validation - minimum (md5, sha1, sha256, sha512) - size validation <span style="color:red;">As a plugin writer, I expect units that are missing from the remote repository to not be created in Pulp when using the immediate download policy.</span> <span style="color:red;">As a plugin writer, I expect units that are missing from the remote repository to be created in Pulp when using background or on_demand download policies.</span> As a plugin writer I can configure mirror lists and rotate between the mirrors - round robin - nearest mirror support As a plugin writer, the plugin API provides tooling whereby I can provide the content to be added and removed from the repository. This tooling supports both immediate and deferred downloading. As a plugin writer I can manage the catalog by using ChangeSets As a plugin writer, the plugin can participate in adding content for cases where the decision to add additional content is based content that has been downloaded. As a plugin writer, I can fetch content myself (but I am not encouraged to do so) with code I write As a plugin writer, I can CRUD content units ## {color:red} Consumer Applicability <span style="color:red;">Using consumer profiles and repo bindings I can compute applicability with 2.y parity Performance needs to be awesome</span> <span style="color:red;">\_(Is the Pulp Consumer going away in Pulp 3? If so, is this section still appropriate?)\_</span> ## Migrations only involving Pulp 3 Users can run "pulp-manager migrate" to migrate the database and adjust state in other locations (filesystem, message broker, ...). \[done\] ## Web Server Integration <span style="color:red;">The content app when deployed with Apache web server or nginx uses their respective features to serve files efficiently.</span> ## Glossary Repository - A named collection of content. Artifact - A file associated with one content (unit). Artifacts are not shared between content (units). Create a content unit using an uploaded file ID as the source for its metadata. Create Artifacts associated with the content unit using an uploaded file ID for each; commit as a single transaction. Content (unit) - A single piece of content manged by Pulp. Each file associated with a content (unit) is called an Artifact. Each content (unit) may have zero or many Artifacts.
.