Project

Profile

Help

Pulp 3 Minimum Viable Product » History » Revision 166

Revision 165 (amacdona@redhat.com, 04/11/2018 05:22 PM) → Revision 166/167 (amacdona@redhat.com, 04/11/2018 05:27 PM)

# Pulp 3.0.0 Minimum Viable Product (MVP) 

 <span style="color:red;">Lines highlighted in red need more attention.</span> 

 ## Overall Guarantees 

   - This is not a direct replacement for Pulp 2. 
   - This is the minimum required for a 3.0.0 beta and GA. 
   - All REST API calls will update the DB using transactions as necessary to ensure data integrity. 

 ## Authentication 

 As an authenticated user I can manage user(s). \[done\] 

   - Add a user 
   - View user(s) 
   - Update any user detail 
   - Delete a user 

 As an API user, I can authenticate any API call with Basic auth \[done\] 

 As an authenticated user, I can filter users by: \[[3142](https://pulp.plan.io/issues/3142)\] 

   - username: (equality, username_in_list) 

 ## Repositories 

 As an authenticated user, I can list all repos. 

   - All fields are included \[done\] 
   - Pagination is supported \[done\] 

 As an authenticated user I can use filters on Repositories list: \[[3079](https://pulp.plan.io/issues/3079)\] 

   - name: (equality, name_in_list) 

 As an authenticated user, I can CRUD a repository 

   - Create a repo \[done\] 
   - Read a repo \[done\] 
   - Update all mutable repo fields (asynchronous) \[done\] 
   - Delete a repo (asynchronous) 

 As an authenticated user, when viewing a repository, I can discover a URL to the latest version of a repository. \[done\]\[[3235](https://pulp.plan.io/issues/3235)\] 

 ## Remotes 

 note: Remote attributes will commonly be available on remotes, but aren't guaranteed to be used by all remotes. 

 As an authenticated user, I can CRUD an remote 

   - Create an remote 
   - Read an remote 
   - Update all mutable remote fields (asynchronous) 
   - Delete an remote (asynchronous) 

 As an authenticated user, I have filters on the Remote list: \[[3080](https://pulp.plan.io/issues/3080)\] 

   - name: (equality, name_in_list) 

 As an authenticated user I can configure the following attributes on an Remote: \[done\] 

   - validate (bool) \[optional: defaults to True\]: If true, the plugin will validate imported content. 
   - ssl_ca_certificate (str) \[optional\] String containing a PEM encoded CA certificate used to validate the server certificate presented by the external source. 
   - ssl_client_certificate (str) \[optional\] Contains a PEM encoded client certificate used for authentication. 
   - ssl_client_key (str) \[optional\] Contains a PEM encoded private key used for authentication. 
   - ssl_validation (bool) \[optional: defaults to True\]: If true, SSL peer validation must be performed. 
   - proxy_url (str) \[optional\] Contains the proxy URL. Format: scheme://user:password@host:port. 
   - username (str) \[optional\] Contains the username to be used for authentication when syncing. 
   - password (str) \[optional\] Contains the password to be used for authentication when syncing. 
   - name (str) \[required\] Contains the name. 
   - last_updated (datetime) \[read-only\] Contains the datetime of the last remote update. 
   - last_synced (datetime) \[read-only\] Contains the datetime of the last remote sync. 

 ## Publishers 

 note: Publisher attributes will commonly be available on publishers, but aren't guaranteed to be used by all publishers. 

 As an authenticated user, I can CRUD a publisher 

   - Create a publisher 
   - Read a publisher 
   - Update all mutable publisher fields (asynchronous) 
   - Delete a publisher (asynchronous) 

 As an authenticated user, I have filters on the Publisher list: \[[3081](https://pulp.plan.io/issues/3081)\] 

   - name: (equality, name_in_list) 

 As an authenticated user I can configure the following attributes on a Publisher: 

   - relative_path (str) \[optional\] The (relative) path component of the published url. 
   - name (str) \[required\] contains the name. 
   - last_published (datetime) \[read-only\] When the last successful publis%{color:red}h occurred. 
   - last_updated (datetime) \[read-only\] The datetime of the last publisher update. 

 ## Distributions 

 As an authenticated user, I can CRUD Distributions: 

   - Create a Distribution. 
   - Read a Distribution 
   - List Distributions 
   - Update all mutable Distribution fields (synchronous)   
     \- base path 
   - Delete an Distribution (synchronous) 

 As a user, my distribution base paths don't conflict and my create/update is rejected identifying the conflicting distributions \[[2987](https://pulp.plan.io/issues/2987)\] 

 As an authenticated user, I can create or update a distribution that is not associated with any publication (NULL) 

 As an authenticated user, I can create or update a distribution that is not associated with any publisher/repository (NULL) 

   - publisher and repository are used together. If one is null, both must be null. 

 As a user, I can update a Distribution to distribute a specific Publication   
 As a user, I want a newly created publication to be automatically served by the content as defined by distributions. 

 As a user, I can see the full urls my base path is served at   
 As an authenticated user, I have filters on the Distribution list: \[[3082](https://pulp.plan.io/issues/3082)\]% 

   - name: (equality, name_in_list)  
   - base_path: (equality, substring, base_path_in_list) 

 ## Publications 

 As an authenticated user, when creating a Publication, I can post a repo version href to be published. \[[3221](https://pulp.plan.io/issues/3221)\] 

 As an authenticated user, I can publish a repository's latest version by posting a repository href to be published. \[[3223](https://pulp.plan.io/issues/3223)\] 

 As an authenticated user, I can view which repository version was used to create a particular publication. \[[3237](https://pulp.plan.io/issues/3237)\] 

 As an authenticated user, I can read Publication(s) 

   - Read a Publication - id, created datetime, list of distribution hrefs, repo version 
   - List all Publications - ordered by created datetime in descending order 

 As an authenticated user, I can delete publications.   
 \- asynchronously with a lock on the repository version.   
 \- prevented if associated with a distribution.   
 \- single object only. 

 As an authenticated user, I can list publications and I have enough information to choose which ones to delete. 

   - choose by created (older ones or perhaps latest) 
   - choose not associated to a distribution. 
   - does not imply filtering 

 As an authenticated user, I can list publications and i have enough information to select a publication to be associated with a distribution. 

   - choose by created (latest or just by publish date) 
       - manual promotion. "My rawhide publication has been tested and now I want to promote it to stable". 
       - rollback to an earlier publication. 
   - does not imply filtering 

 As an authenticated user, I can determine if and how a publication is distributed. 

 ## Exporters 

 <span style="color:black;">As a plugin writer, I can contribute an exporter that is discovered by core</span>   
 <span style="color:black;">As a plugin writer, I have docs on how to create a discoverable exporter</span>   
 <span style="color:black;">As a plugin writer, I can contribute an exporter that uses a publication.</span> 

 ## Artifacts 

 As an authenticated user, I can create an Artifact by uploading a file. \[done\] 

 As an authenticated user, I can optionally specify a size and/or digest to validate the uploaded file. \[done\] 

 ## Content Units 

 As an authenticated user, I can create a content unit. 

 As an authenticated user, I can read and list content units.   
 <span style="color:red;">\* As an authenticated user, I can filter content by repository version</span> 

 <span style="color:red;">As an authenticated user, I can delete a specific content unit if it is not in any repository version. (orphan)</span> 

   - Error message saying that the unit is in use by a repo version and a link to the filter to return all of the repo versions. 
   - As a user, I know that files (Artifacts) associated with the Content unit are not removed by this call (docs) 

 ## Versioned Repositories 

 #### CRD 

 As an authenticated user, I can list versions for a particular repository. \[done\] 

   - Pagination is supported 

 As an authenticated user, I can filter repository versions by: \[[3238](https://pulp.plan.io/issues/3238)\] 

   - number (equality, lt/lte, gt/gte) 
   - content id (equality, content_in_list) 
   - created datetime (range) 

 As an authenticated user, I can delete any repository version. \[[3219](https://pulp.plan.io/issues/3219)\] 

   - Latest versions are deleted simply. 
   - Content set changes are "squashed" into the following version when non-latest versions are deleted. 

 As an authenticated user, I can view content that was added or removed in a particular repository version (as compared to the previous version). \[done\] 

 As an authenticated user, I can see the number of content unit types with counts for each \[done\]\[[3059](https://pulp.plan.io/issues/3059)\] 

 #### Repository Version Content 

 As a user, I know content sets for repository versions are immutable. \[done\] 

 As an authenticated user, I can list the content in a particular repository version \[done\] 

   - All fields are included 
   - Pagination is supported 

 As an authenticated user, I can create a new version by adding or removing content to the latest version. \[[3234](https://pulp.plan.io/issues/3234)\] 

   - I can follow the progress. (adding/removing are asynchronous). 
   - can add and remove together in a single call 

 ## Orphan Content Units and Artifacts 

 As an authenticated user, I can cause an action that cleans up **both** orphaned content units and orphaned artifacts. 

   - I cannot specify the units specifically (all types). 
   - I can follow the progress of all cleanups. (Cleanups are asynchronous.) 

 ## Task Management 

 As an authenticated user, I can list all tasks 

   - This does not include associated progress reports 

 As an authenticated user, I can see a detail view for a specific task \[done\] 

   - all attributes of a task 
   - all associated progress reports 

 As an authenticated user, I can cancel a task \[done\] 

   - don't dare to use the DELETE verb! 

 As an authenticated user, I can delete tasks. 

 As an authenticated user, I can filter tasks by: \[[3144](https://pulp.plan.io/issues/3144)\] 

   - state (equality, state_in_list) 
   - started_at(started_in_range) 
   - finished_at(finished_in_range) 
   - worker (equality) 
   - parent (equality) 

 ## Status 

 As an unauthenticated user I can view the status of Pulp workers and resource managers. \[done\] 

 As an unauthenticated user I can view the status of the web server's connection to the database and message broker. \[done\] 

 As an unauthenticated user I can view the versions of core and each installed plugin. 

 ## Workers 

 As an authenticated user, I can filter workers by: \[[3143](https://pulp.plan.io/issues/3143)\] 

   - last_heartbeat (range) 
   - name (substring) 
   - gracefully_stopped (equality) 

 ## Plugin User Content Management stories 

 #### Simple Copy 

 As an authenticated user of a plugin, I can search (synchronous call) a repository version's content using filtering. 

   - note: In a separate call (async), I can add_content_units or remove_content_units to another repository's latest version. 

 #### Complex Copy 

 As a plugin writer I can provide a rich search features with arbitrary viewsets. e.g. depsolving, versioning, etc 

 Examples of specific plugin use cases motivating the above general viewset 

   - As an authenticated user, I can add an Errata from one repository to another repository along with packages mentioned in the Errata and all their dependencies that are present in the source repository. 
   - As an authenticated user, I can add bunch of dependencies and store n versions back for each RPM. 
   - As an authenticated user, I can use depsolving and versioning copy features together 
   - As an authenticated user, I can copy a puppet module and all of it's dependencies from one repository to another.  
   - As an authenticated user, I can depsolve units to be added to a destination repo based on an errata 

 ## Plugin API 

 As a plugin writer, I have a plugin API that is semantically versioned at 0.x separate from the REST API \[done\] 

 As a plugin writer, my app will be discovered by Pulp's app via an entry point provided by the plugin writer \[done\] 

 #### Task 

 As a plugin writer, I can report progress with a message and state \[done\] 

 As a plugin writer, I can report progress with an optional suffix \[done\] 

 As a plugin writer, I can report progress with a total count of things to do an the current count of things done \[done\] 

 As a plugin writer, non-fatal exceptions on the Task and are included in the Task detail. non_fatal exceptions do not cause the Task to be marked as failed, but may be interpreted by the user as not fully successful. \[done\] 

 As a plugin writer, the working directory is set before Task work is done and cleaned up afterwards. I should not need to interact with the file system outside of the working dir. \[done\] 

 #### Remote 

 As a plugin writer, I can provide a subclassed Remote. I can add custom fields to the subclassed Remote. 

 As a plugin writer, I can provide a UserFacingTask to perform a sync operation using information stored in a Remote. 

 As a plugin writer, I can provide a ViewSet for the subclassed Remote that has a 'sync' endpoint that dispatches the sync task with a reservation for Repository and Remote. 

 #### Publisher 

 As a plugin writer, I can provide a subclassed Publisher. I can add custom fields to the subclassed Publisher. 

 As a plugin writer, I can provide a UserFacingTask to perform a publish operation using information from the subclassed Publisher. 

 As a plugin writer, I can provide a ViewSet for the subclassed Publisher that has a 'publish' endpoint that dispatches the publish task with a reservation for Repository and Publisher. 

 #### Content 

 As a plugin writer, I can provide a subclassed Content unit. I can add custom fields to the subclassed Content. \[done\] 

 <span style="color:red;">As a plugin writer, I have documentation that shows how I can add filters to filter content responsibly.</span> 

 <span style="color:red;">As a plugin writer, I have documentation on how to write a filter for my Content that can use the RepositoryVersion manager.</span>   
 <span style="color:red;">\* note: This will allow users to filter content by repository version</span> 

 #### Content Management 

 As a plugin writer, I can interact with and create Artifacts \[done\] 

 As a plugin writer, I can query content units/artifacts associated with a repository. \[done\] 

 As a plugin writer, I can create a new repository version: \[done\] 

   - When I create the new version, it is incomplete. 
   - The new version starts with all the content from the previous version. 
   - I can add and remove content units to and from an incomplete repository version.  
   - When I am finished with the incomplete version, core finalizes the repository version for me.  
   - I have exclusive access to the new repository version while my code is running. 

 #### Publishing 

 As a plugin writer, I can create a new publication: \[done\] 

   - When I create the new publication, it is incomplete. 
   - When I am finished with the incomplete publication, core finalizes the publication for me.  
   - I have exclusive access to the repository version while my code is running. 
   - I have exclusive access to the new publication while my code is running. 

 #### "live APIs" 

 As a plugin writer, I can register views and viewsets to arbitrary endpoints. \[3360\]   
 As a plugin writer, I have documentation on what URLs I should not use for my views and viewsets \[3473\] 

 Here are some concrete use cases driving the very Live API use cases above: 

 ~~~ 
 # Concrete user use cases: 
     As an authenticated user, I can use the puppet client to fetch content from Pulp using the Forge API 
     As an authenticated user I can use the docker client to fetch content from Pulp using the Docker v1 API 
     As an authenticated user I can use the docker client to fetch content from Pulp using the Docker v2 API 

 # Concrete plugin writer use cases 
     As a puppet plugin developer, I can provide a viewset which handles the server side of the puppet Forge v3 API 
     As a docker plugin developer, I can provide a viewset which handles the server side of the docker v1 API 
     As a docker plugin developer, I can provide a viewset which handles the server side of the docker v2 API 
 ~~~ 

  
 h4. Storage 

 As a plugin writer, the plugin API provides an API that returns a fully qualified path to a shared and namespaced storage location used to store content. \["3182"https://pulp.plan.io/issues/3182\]   
 \*****\*****\*\*\*\* 

 ## Webserver Deployment 

 As a system administrator, I can deploy all Pulp web applications on one process   
 As a system administrator, I can deploy the Pulp REST API exclusively in one process   
 As a system administrator, I can deploy the Pulp content serving view exclusively in one process 

 As a system administrator, I can deploy all Pulp web applications inside a virtualenv.   
 As a system administrator, I can deploy all Pulp web applications without root permissions. 

 ## CLI 

 We will use coreapi-cli to generate a one to one mapping of cli commands to rest api schema #3068   
 <span style="color:red;">We will have a wrapper for coreapi-cli. This wrapper will handle parallel progress reporting</span> 

 ## Download API 

 As a plugin writer, I can download files via 

   - http:// 
   - https:// 
   - file:// 

 As a plugin writer, I can configure a downloader with: 

   - Basic Auth 
   - SSL Cert Client Auth 
   - Custom CAs will be configured via a "trust store" either on the system or similar. Pulp will not do anything to read/load/manage CAs directly. 

 As a plugin writer, I can provide arbitrary behaviors for customized downloaders 

   - For example token authentication in the docker plugin 

 As a plugin writer, I can have connection pooling/reuse 

 As a plugin writer, I have proxy settings 

   - proxy url (containing basic auth info) 

 <span style="color:red;">As a plugin writer, I can have great logs</span> 

 <span style="color:red;">As an authenticated user, I have documentation about how to use something for bandwidth limiting</span> 

 As a plugin writer, I can configure the validation mechanisms used at download time 

   - checksum validation - minimum (md5, sha1, sha256, sha512) 
   - size validation 

 <span style="color:red;">As a plugin writer, I expect units that are missing from the remote repository to not be created in Pulp when using the immediate download policy.</span> 

 <span style="color:red;">As a plugin writer, I expect units that are missing from the remote repository to be created in Pulp when using background or on_demand download policies.</span> 

 <span style="color:red;">As a plugin writer, the plugin API provides tooling whereby I can provide the content to be added and removed from the repository. This tooling supports both immediate and deferred downloading.</span> 

 <span style="color:red;">As a plugin writer I can manage the catalog by using ChangeSets</span> 

 <span style="color:red;">As a plugin writer, the plugin can participate in adding content for cases where the decision to add additional content is based on the content that has been downloaded.</span> 

 As a plugin writer, I can fetch content myself (but I am not encouraged to do so) with code I write 

 <span style="color:red;">As a plugin writer, I can CRUD content units</span> need a convention to handle multiple content units - see https://pulp.plan.io/issues/3472 

 ## Migrations only involving Pulp 3 

 Users can run "pulp-manager migrate" to migrate the database and adjust state in other locations (filesystem, message broker, ...). \[done\] 

 ## Web Server Integration 

 As a user, I can have content efficiently served to me by Apache by Pulp using the X-SEND response headers. \[done\]   
 As a user, I can have content efficiently served to me by Nginx by Pulp using the X-Accel-Redirect response headers. \[done\] 

 ## Glossary 

 Add (Content Unit): An operation causing a new repository version to contain a content unit(s) 

 Applicability - A plugin defined term meaning when a package update available in a repository is applicable to a given consumer as determined by the Consumer Profile. 

 Artifact - A file associated with one content (unit). Artifacts are not shared between content (units). Create a content unit using an uploaded file ID as the source for its metadata. Create Artifacts associated with the content unit using an uploaded file ID for each; commit as a single transaction. 

 Content (unit) - A single piece of content manged by Pulp. Each file associated with a content (unit) is called an Artifact. Each content (unit) may have zero or many Artifacts. 

 Distribution: Where and how the content app serves a Publication. i.e. http vs https and base path component of the URL. A Distribution defines: 

   - the base path of the repository publication (required) 
   - serving via http (default=False) 
   - serving via https (default=True) 
   - relationship w/ a Publisher for auto-distribution (should be allowed to be NULL) 
   - relationship with Publication (should be allowed to be NULL) 

 Exporter: An Exporter exports a Publication out of Pulp. e.g. rsync exporter exports content to a remote server 

 Live API: a viewset endpoint contributed by plugin. For examples see the [associated MVP section](https://pulp.plan.io/projects/pulp/wiki/Pulp_3\_Minimum_Viable_Product#Plugin-Writer-live-APIs) 

 Orphan Artifact: An Artifact that is associated with 0 Content Units and 0 Publications 

 Orphan Content (unit): A content unit that is a member of 0 repository versions 

 Remove (content unit): An operation causing a new repository version to not contain a content unit(s) 

 Repository - A named collection of repository versions. 

 Repository Version - An immutable set of content which is versioned by a sequential number.