Pulp

Pulp 3 Minimum Viable Product » History » Revision 117

Revision 116 (daviddavis, 11/27/2017 08:50 PM) → Revision 117/167 (bmbouter, 11/27/2017 09:31 PM)

# Pulp 3.0.0 Minimum Viable Product (MVP) 

 <span style="color:red;">Lines highlighted in red need more attention.</span> 

 ## Overall Guarantees 

   - This is not a direct replacement for Pulp 2. 
   - This is the minimum required for a 3.0.0 beta and GA. 
   - All REST API calls will update the DB using transactions as necessary to ensure data integrity. 

 ## Legend 

 \[done\] means merged and documented   
 \[in-progress\] means started but not fully done   
 If there is no label the effort has not yet been started 

 ## Authentication 

 As an authenticated user I can manage user(s). \[done\] 

   - Add a user 
   - View user(s) 
   - Update any user detail 
   - Delete a user 

 As an API user, I can have documentation to generate a JSON Web Token (JWT) without the server being online. \[done\] 

 As an administrator, I can disable JWT token expiration. This configuration is in the settings file and is system-wide. \[done\] 

 As an administrator, I can configure the JWT tokens to expire after a configurable amount of time. This configuration is in the settings file and is system-wide. \[done\] 

 The JWT shall have a username identifier \[done\] 

 <span style="color:red;">As an API user, I can authenticate any API call (except to request a JWT) with a JWT. (not certain if this should be the behavior) \[in progress\]</span> 

 As an API user, I can invalidate all existing JWT tokens for a given user. \[done\] 

 As an authenticated user, when deleting a user 'foo', all of user 'foo's existing JWTs are invalidated. \[done\] 

 As an un-authenticated user, I can obtain a JWT token by using a username and password. \[done\] 

 As an authenticated user, I can filter users by: \[[3142](https://pulp.plan.io/issues/3142)\] 

   - username: (equality, username_in_list) 

 ## Repositories 

 As an authenticated user, I can list all repos. 

   - All fields are included \[done\] 
   - Pagination is supported \[done\] 

 As an authenticated user I can use filters on Repositories list: \[[3079](https://pulp.plan.io/issues/3079)\] 

   - id: (id_in_list) \# id equality is not necessary, v3/repositories/\<UUID\>/ 
   - name: (equality, name_in_list) 

 As an authenticated user, I can CRUD a repository 

   - Create a repo \[done\] 
   - Read a repo \[done\] 
   - Update all mutable repo fields \[done\] 
   - Delete a repo (asynchronous) \[done\] 

 As an authenticated user, I can view content for a repository. 

 As an authenticated user, I can list a repository's associated importers and publishers 

   - All fields are included \[done\] 
   - Pagination is supported \[done\] 

 As an authenticated user, I can see the number of content unit types with counts for each \[done\]\[[3059](https://pulp.plan.io/issues/3059)\] 

 ## Importers 

 note: Importer attributes will commonly be available on importers, but aren't guaranteed to be used by all importers. 

 As an authenticated user, I can CRUD an importer 

   - Create an importer 
   - Read an importer 
   - Update all mutable importer fields 
   - Delete an importer (asynchronous) 

 As an authenticated user, I have filters on the Importer list: \[[3080](https://pulp.plan.io/issues/3080)\] 

   - id: (id_in_list) \# id equality is not necessary, objects are referenced by id 
   - name: (equality, name_in_list) 

 As an authenticated user I can configure the following attributes on an Importer: \[done\] 

   - validate (bool) \[optional: defaults to True\]: If true, the plugin will validate imported content. 
   - ssl_ca_certificate (str) \[optional\] String containing a PEM encoded CA certificate used to validate the server certificate presented by the external source. 
   - ssl_client_certificate (str) \[optional\] Contains a PEM encoded client certificate used for authentication. 
   - ssl_client_key (str) \[optional\] Contains a PEM encoded private key used for authentication. 
   - ssl_validation (bool) \[optional: defaults to True\]: If true, SSL peer validation must be performed. 
   - proxy_url (str) \[optional\] Contains the proxy URL. Format: scheme://user:password@host:port. 
   - username (str) \[optional\] Contains the username to be used for authentication when syncing. 
   - password (str) \[optional\] Contains the password to be used for authentication when syncing. 
   - <span style="color:orange;">download_policy (str) \[optional\] Contains the downloading policy name. This is a choice of three options:   
  \- immediate (default) - Downloading occurs during sync. The sync task does not complete until downloading is done.   
  \- background - Downloading is started by the sync but occurs in the background. The sync task completes before downloading is complete.   
  \- on-demand - The sync task records everything that would be downloaded but does not download content. Downloading occurs on demand as driven by client requests for content.</span> 
   - feed_url (str) \[optional\] Contains the URL of an external content source. This is optional. 
   - <span style="color:orange;">sync mode: (str) \[optional\]. It has two choices 'additive' and 'mirror':   
  \- additive (default) - all remote content is added to the local repository on sync. During sync no content is removed from the local repository.   
  \- mirror - the local content will mirror the remote content exactly, removing local content if not also present in the remote content.</span> 
   - name (str) \[required\] Contains the name. 
   - last_updated (datetime) \[read-only\] Contains the datetime of the last importer update. 
   - last_synced (datetime) \[read-only\] Contains the datetime of the last importer sync. 

 ## Publishers 

 note: Publisher attributes will commonly be available on publishers, but aren't guaranteed to be used by all publishers. 

 As an authenticated user, I can CRUD a publisher 

   - Create a publisher 
   - Read a publisher 
   - Update all mutable publisher fields 
   - Delete a publisher (asynchronous) 

 As an authenticated user, I have filters on the Publisher list: \[[3081](https://pulp.plan.io/issues/3081)\] 

   - id: (id_in_list) \# id equality is not necessary, objects are referenced by id 
   - name: (equality, name_in_list) 

 As an authenticated user I can configure the following attributes on a Publisher: 

   - relative_path (str) \[optional\] The (relative) path component of the published url. 
   - name - (str) \[required\] contains the name. 
   - last_published (datetime) \[read-only\] When the last successful publis%{color:red}h occurred. 
   - last_updated (datetime) \[read-only\] The datetime of the last publisher update. 
   - <span style="color:red;">auto_publish(bool) - ??? consider adding auto-publish feature to MVP</span> 

 ## <span style="color:red;">Distributions</span> 

 <span style="color:red;">As an authenticated user, I can CRUD Distributions:</span>   
 {color:red} \* Create a Distribution   
 {color:red} \* Read a Distribution   
 {color:red} \* Update all mutable Distribution fields   
 {color:red} \* Delete an Distribution (asynchronous?) 

 As an authenticated user, I have filters on the Distribution list: \[[3082](https://pulp.plan.io/issues/3082)\] 

   - id: (id_in_list) \# id equality is not necessary, objects are referenced by id 
   - name: (equality, name_in_list)  
   - base_path: (equality, substring, base_path_in_list) 

 ## <span style="color:red;">Publications</span> 

 {color:red} \* Read a Publication   
 {color:red} \* List all Publications   
 {color:red} \* Delete a Publication (asynchronous?) 

 {color:red} As an authenticated user, I have filters on the Publication list:   
 {color:red} \* id: (id_in_list) \# id equality is not necessary, objects are referenced by id 

 ## Sync and Publish 

 As an authenticated user, I can trigger an importer to sync. \[done\] 

   - I can follow the progress of all syncs. (Syncs are asynchronous.) 
   - I cannot pass "sync" options. 
   - Auto-publish is not included as an importer property. 

 As an authenticated user, I can trigger a publisher to publish. \[done\] 

   - I can follow the progress of all publishes. (Publishes are asynchronous.) 
   - I cannot pass "publish" options. 

 ## Content Manipulation 

 #### Uploading Artifacts 

 As an authenticated user, I can create an Artifact by uploading a file. \[done\] 

 As an authenticated user, I can specify a size and/or digest to validate the uploaded file. \[done\] 

 #### Creating Content Units 

 As an authenticated user, I can create a content unit by providing the content type (in the URL), references to Artifacts, and the metadata supplied in the POST body. \[done\] 

 #### Add / Remove Content from a Repository 

 As an authenticated user, I can add content (specific existing content) to a repository 

   - I can specify one or more content units by id/href 
   - I can follow the progress. (adding are asynchronous). 

 As an authenticated user, I can remove specified content from a repository 

   - I can specify one or more content units by id/href 
   - I can follow the progress. (removing are asynchronous). 

 #### Simple Copy 

 As an authenticated user, I can search (synchronous call) a repository's content using filtering. 

   - note: In a separate call (async), I can add this content to another repository. This is effectively a copy operation in two calls. 

 As an authenticated user, I can import all content from one repository into another repository in a single async call. (Clone use case) 

   - I can follow the progress. (adding are asynchronous). 

 #### Complex Copy 

 As a plugin writer I can provide a rich search features with abitrary viewsets. e.g. depsolving, verisoning, etc 

 Examples of specific plugin use cases motivating the above general viewset 

   - As an authenticated user, I can add an Errata from one repository to another repository along with packages mentioned in the Errata and all their dependencies that are present in the source repository. 
   - As an authenticated user, I can add bunch of dependencies and store n versions back for each RPM. 
   - As an authenticated user, I can use depsolving and versioning copy features together 
   - As an authenticated user, I can copy a puppet module and all of it's dependencies from one repository to another.  
   - As an authenticated user, I can depsolve units to be added to a destination repo based on an errata 

 #### Delete 

 FAs an authenticated user, I can delete a specific content unit%   
 <span style="color:red;">\* If the content unit is still in at least one repository the delete fails with a listing of all repositories the unit is part of.</span>   
 <span style="color:red;">\* Artifacts and associated files from the deleted unit are cleaned up</span> 

 <span style="color:red;">As an authenticated user, I can delete multiple content units with filtering</span>   
 <span style="color:red;">\* If a content unit is still in at least one repository the delete fails with a listing of all repositories the unit is part of.</span>   
 <span style="color:red;">\* Artifacts and associated files from deleted units are cleaned up</span> 

 #### Filtering 

 As an authenticated user, I can filter Content by repository information when plugin writers have provided such a filter 

 As an authenticated user, I can filter content by repository 

 As an authenticated user, I can filter content by type 

 As an authenticated user, I can filter content on plugin specific attributes when plugin writers have provided such a filter 

 ## <span style="color:red;">Versioned Repositories</span> 

 As an authenticated user, I can list the content in a particular repository version 

   - All fields are included 
   - Pagination is supported 
   - <span style="color:red;">Filtering support</span> 

 As an authenticated user, I can discover a URL to the latest version of a repository   
 <span class="resource repository the on attributes or endpoint, API dedicated a through \^ Is" style="color:red;"></span> 

 As an authenticated user, I can run a publisher without a repository version and have it default to the latest version. 

 <span style="color:red;">As an authenticated user, I can delete a repository version by specifying the version</span> 

 <span style="color:red;">As an authenticated user, I can upload multiple content(s?) and add create a single new version that adds all of them.</span> 

 ## Orphan Content Units and Artifacts 

 As an authenticated user, I can cause an action that cleans up **both** orphaned content units and orphaned artifacts. 

   - I cannot specify the units specifically (all types). 
   - I can follow the progress of all cleanups. (Cleanups are asynchronous.) 

 ## Task Management 

 As an authenticated user, I can list all tasks 

   - <span style="color:orange;">Filtering support on \['state', 'id', 'group'\]</span> 
   - <span style="color:orange;">This does not include associated progress reports</span> 

 As an authenticated user, I can see a detail view for a specific task \[done\] 

   - all attributes of a task 
   - all associated progress reports 

 As an authenticated user, I can cancel a task \[done\] 

   - don't dare to use the DELETE verb! 

 As an authenticated user, I can delete tasks. 

 As an authenticated user, I can filter tags by: \[[3144](https://pulp.plan.io/issues/3144)\] 

   - state (equality, state_in_list) 
   - started_at(started_in_range) 
   - finished_at(finished_in_range) 
   - worker (equality) 
   - parent (equality) 

 ## Task Group 

 <span style="color:red;">I can view a summary of the status of all tasks in a group</span> 

 ## Status 

 As an unauthenticated user I can view the status of Pulp workers, resource managers, and celerybeats. \[done\] 

 As an unauthenticated user I can view the status of the web server's connection to the database and message broker. \[done\] 

 As an unauthenticated user I can view the versions of core and each installed plugin. 

 ## Workers 

 As an authenticated user, I can filter workers by: \[[3143](https://pulp.plan.io/issues/3143)\] 

   - last_heartbeat (range) 
   - name (substring) 
   - gracefully_stopped (equality) 

 ## Plugin API 

 As a plugin writer, I have a plugin API that is semantically versioned at 0.x separate from the REST API \[done\] 

 As a plugin writer, I can report progress with a message and state \[done\] 

 As a plugin writer, I can report progress with an optional suffix \[done\] 

 As a plugin writer, I can report progress with a total count of things to do an the current count of things done \[done\] 

 As a plugin writer, non-fatal exceptions on the Task and are included in the Task detail. non_fatal exceptions do not cause the Task to be marked as failed, but may be interpreted by the user as not fully successful. \[done\] 

 As a plugin writer, the working directory is set before Task work is done and cleaned up afterwards. I should not need to interact with the file system outside of the working dir. \[done\] 

 <span style="color:red;">As a plugin writer, I can provide a subclassed Importer. The importer's responsibility is to synchronize the content of a Pulp repository with the content of a remote repository. (a circular import problem needs to be discussed and may cause this to change) \[done\]</span> 

 <span style="color:red;">As a plugin writer, I can provide a subclassed Publisher. The publisher's responsibility is to publish content. (a circular import problem needs to be discussed and may cause this to change) \[done\]</span> 

 As a plugin writer, I can define unit types by subclassing Content models to provide concrete content unit types to be manged by the platform. \[done\] 

 As a plugin writer, I can interact with and create Artifacts \[done\] 

 As a plugin writer, my app will be discovered by Pulp's app via an entry point provided by the plugin writer \[done\] 

 As a plugin writer, I can use the plugin API to query content units/artifacts associated with a repository. \[done\] 

 As a plugin writer, I can add and remove content units to and from a repository. \[done\] 

 As a plugin writer, I have documentation that shows how I can add filters to filter content responsibly. 

 As a plugin writer, I have documentation on how to write a filter for my Content that can use the RepositoryContent through table. 

   - note: This will allow users to filter content by repository information 

 ## Plugin Writer "live APIs" 

 As a plugin writer, I can provide a Live API with arbitrary views and viewsets.   
 As a plugin writer, I have documentation on what URLs I should -not use for my views and viewsets 

 Here are some concrete use cases driving the very Live API use cases above: 

 ~~~ 
 # Concrete user use cases: 
     As an authenticated user, I can use the puppet client to fetch content from Pulp using the Forge API 
     As an authenticated user I can use the docker client to fetch content from Pulp using the Docker v1 API 
     As an authenticated user I can use the docker client to fetch content from Pulp using the Docker v2 API 

 # Concrete plugin writer use cases 
     As a puppet plugin developer, I can provide a viewset which handles the server side of the puppet Forge v3 API 
     As a docker plugin developer, I can provide a viewset which handles the server side of the docker v1 API 
     As a docker plugin developer, I can provide a viewset which handles the server side of the docker v2 API 
 ~~~ 

 ## Webserver Deployment 

 As an authenticated user, I can deploy all Pulp webservices on one process 

 <span style="color:red;">As an authenticated user, I can deploy the Pulp REST API exclusively in one process</span>   
 <span style="color:red;">As an authenticated user, I can deploy the Pulp content serving view exclusively in one process</span>   
 <span style="color:red;">As an authenticated user, I can deploy the Pulp content serving view with reduced permissions.</span>   
 <span style="color:red;">As an authenticated user, I can deploy a Plugin's Live API exclusively in one process.</span> 

 ## CLI 

 We will use coreapi-cli to generate a one to one mapping of cli commands to rest api schema #3068   
 <span style="color:orange;">We will have a wrapper for coreapi-cli. This wrapper will handle parallel progress reporting</span> 

 ## Download API 

 As a plugin writer, I can download files via 

   - http:// 
   - https:// 
   - file:// 

 As a plugin writer, I can configure a downloader with: 

   - Basic Auth 
   - SSL Cert Client Auth 
   - Custom CAs will be configured via a "trust store" either on the system or similar. Pulp will not do anything to read/load/manage CAs directly. 

 As a plugin writer, I can provide arbitrary behaviors for customized downloaders 

   - For example token authentication in the docker plugin 

 As a plugin writer, I can have connection pooling/reuse 

 As a plugin writer, I have proxy settings 

   - proxy url (containing basic auth info) 

 As a plugin writer, I can have great logs 

 As an authenticated user, I have documentation about how to use something for bandwidth limiting 

 As a plugin writer, I can configure the validation mechanisms used at download time 

   - checksum validation - minimum (md5, sha1, sha256, sha512) 
   - size validation 

 <span style="color:red;">As a plugin writer, I expect units that are missing from the remote repository to not be created in Pulp when using the immediate download policy.</span> 

 <span style="color:red;">As a plugin writer, I expect units that are missing from the remote repository to be created in Pulp when using background or on_demand download policies.</span> 

 <span style="color:red;">As a plugin writer I can configure mirror lists and rotate between the mirrors</span>   
 <span style="color:red;">\* round robin</span>   
 <span style="color:red;">\* nearest mirror support</span> 

 As a plugin writer, the plugin API provides tooling whereby I can provide the content to be added and removed from the repository. This tooling supports both immediate and deferred downloading. 

 As a plugin writer I can manage the catalog by using ChangeSets 

 As a plugin writer, the plugin can participate in adding content for cases where the decision to add additional content is based content that has been downloaded. 

 As a plugin writer, I can fetch content myself (but I am not encouraged to do so) with code I write 

 As a plugin writer, I can CRUD content units 

 ## {color:red} Consumer Applicability 

 <span style="color:red;">Using consumer profiles and repo bindings I can compute applicability with 2.y parity   
 Performance needs to be awesome</span> 

 <span style="color:red;">\_(Is the Pulp Consumer going away in Pulp 3? If so, is this section still appropriate?)\_</span> 

 ## Migrations only involving Pulp 3 

 Users can run "pulp-manager migrate" to migrate the database and adjust state in other locations (filesystem, message broker, ...). \[done\] 

 ## Web Server Integration 

 <span style="color:red;">As a user, I can have content efficiently served to me by Apache by Pulp using the X-SEND response headers \[[2914](https://pulp.plan.io/issues/2914)\]</span>   
 <span style="color:red;">As a user, I can have content efficiently served to me by Nginx by Pulp using the X-Accel-Redirect response headers \[[2914](https://pulp.plan.io/issues/2914)\]</span> 

 <span style="color:red;">As a user, I can have an Ansible role to install Apache which enables Apache integration for Pulp and configures Apache to serve Pulp. \[[2921](https://pulp.plan.io/issues/2921)\]</span>   
 <span style="color:red;">As a user, I can have an Ansible role to install Nginx which enables Nginx integration for Pulp and configures Nginx to serve Pulp. \[[2922](https://pulp.plan.io/issues/2922)\]</span> 

 ## Glossary 

 Repository - A named collection of content. 

 Artifact - A file associated with one content (unit). Artifacts are not shared between content (units). Create a content unit using an uploaded file ID as the source for its metadata. Create Artifacts associated with the content unit using an uploaded file ID for each; commit as a single transaction. 

 Content (unit) - A single piece of content manged by Pulp. Each file associated with a content (unit) is called an Artifact. Each content (unit) may have zero or many Artifacts. 

 Orphan Content (unit): A content unit that is a member of 0 repositories 

 Orphan Artifact: An Artifact that is associated with 0 Content Units and 0 Publications 

 Add (Content Unit): An operation causing a repository to contain a content unit(s) 

 Remove (content unit): An operation causing a repository to not contain a content unit(s)