Project

Profile

Help

Actions

Pulp 3 Minimum Viable Product » History » Revision 158

« Previous | Revision 158/167 (diff) | Next »
bmbouter, 04/10/2018 04:43 PM
Removing the legend since it's woefully out of date and not used anymore. The first line clearly calls out the meaning of red which we still use.


Pulp 3.0.0 Minimum Viable Product (MVP)

Lines highlighted in red need more attention.

Overall Guarantees

  • This is not a direct replacement for Pulp 2.
  • This is the minimum required for a 3.0.0 beta and GA.
  • All REST API calls will update the DB using transactions as necessary to ensure data integrity.

Authentication

As an authenticated user I can manage user(s). [done]

  • Add a user
  • View user(s)
  • Update any user detail
  • Delete a user

As an API user, I can authenticate any API call with Basic auth [done]

As an authenticated user, I can filter users by: [3142]

  • username: (equality, username_in_list)

Repositories

As an authenticated user, I can list all repos.

  • All fields are included [done]
  • Pagination is supported [done]

As an authenticated user I can use filters on Repositories list: [3079]

  • id: (id_in_list) # id equality is not necessary, v3/repositories/<UUID>/
  • name: (equality, name_in_list)

As an authenticated user, I can CRUD a repository

  • Create a repo [done]
  • Read a repo [done]
  • Update all mutable repo fields (asynchronous) [done]
  • Delete a repo (asynchronous)

As an authenticated user, when viewing a repository, I can discover a URL to the latest version of a repository. [done][3235]

Remotes

note: Remote attributes will commonly be available on remotes, but aren't guaranteed to be used by all remotes.

As an authenticated user, I can CRUD an remote

  • Create an remote
  • Read an remote
  • Update all mutable remote fields (asynchronous)
  • Delete an remote (asynchronous)

As an authenticated user, I have filters on the Remote list: [3080]

  • name: (equality, name_in_list)

As an authenticated user I can configure the following attributes on an Remote: [done]

  • validate (bool) [optional: defaults to True]: If true, the plugin will validate imported content.
  • ssl_ca_certificate (str) [optional] String containing a PEM encoded CA certificate used to validate the server certificate presented by the external source.
  • ssl_client_certificate (str) [optional] Contains a PEM encoded client certificate used for authentication.
  • ssl_client_key (str) [optional] Contains a PEM encoded private key used for authentication.
  • ssl_validation (bool) [optional: defaults to True]: If true, SSL peer validation must be performed.
  • proxy_url (str) [optional] Contains the proxy URL. Format: scheme://user:password@host:port.
  • username (str) [optional] Contains the username to be used for authentication when syncing.
  • password (str) [optional] Contains the password to be used for authentication when syncing.
  • name (str) [required] Contains the name.
  • last_updated (datetime) [read-only] Contains the datetime of the last remote update.
  • last_synced (datetime) [read-only] Contains the datetime of the last remote sync.

Publishers

note: Publisher attributes will commonly be available on publishers, but aren't guaranteed to be used by all publishers.

As an authenticated user, I can CRUD a publisher

  • Create a publisher
  • Read a publisher
  • Update all mutable publisher fields (asynchronous)
  • Delete a publisher (asynchronous)

As an authenticated user, I have filters on the Publisher list: [3081]

  • name: (equality, name_in_list)

As an authenticated user I can configure the following attributes on a Publisher:

  • relative_path (str) [optional] The (relative) path component of the published url.
  • name (str) [required] contains the name.
  • last_published (datetime) [read-only] When the last successful publis%{color:red}h occurred.
  • last_updated (datetime) [read-only] The datetime of the last publisher update.

Distributions

As an authenticated user, I can CRUD Distributions:

  • Create a Distribution.
  • Read a Distribution
  • List Distributions
  • Update all mutable Distribution fields (synchronous)
    - base path
  • Delete an Distribution (synchronous)

As a user, my distribution base paths don't conflict and my create/update is rejected identifying the conflicting distributions [2987]

As an authenticated user, I can create or update a distribution that is not associated with any publication (NULL)

As an authenticated user, I can create or update a distribution that is not associated with any publisher (NULL)

As a user, I can update a Distribution to distribute a specific Publication
As a user, I want a newly created publication to be automatically served by the content as defined by distributions.

As a user, I can see the full urls my base path is served at
As an authenticated user, I have filters on the Distribution list: [3082]%

  • name: (equality, name_in_list)
  • base_path: (equality, substring, base_path_in_list)

Publications

As an authenticated user, when creating a Publication, I can post a repo version href to be published. [3221]

As an authenticated user, I can publish a repository's latest version by posting a repository href to be published. [3223]

As an authenticated user, I can view which repository version was used to create a particular publication. [3237]

As an authenticated user, I can read Publication(s)

  • Read a Publication - id, created datetime, list of distribution hrefs, repo version
  • List all Publications - ordered by created datetime in descending order

As an authenticated user, I can delete publications.
- asynchronously with a lock on the repository version.
- prevented if associated with a distribution.
- single object only.

As an authenticated user, I can list publications and I have enough information to choose which ones to delete.

  • choose by created (older ones or perhaps latest)
  • choose not associated to a distribution.
  • does not imply filtering

As an authenticated user, I can list publications and i have enough information to select a publication to be associated with a distribution.

  • choose by created (latest or just by publish date)
    • manual promotion. "My rawhide publication has been tested and now I want to promote it to stable".
    • rollback to an earlier publication.
  • does not imply filtering

As an authenticated user, I can determine if and how a publication is distributed.

Exporters

As a plugin writer, I can contribute an exporter that is discovered by core
As a plugin writer, I have docs on how to create a discoverable exporter
As a plugin writer, I can contribute an exporter that uses a publication.

Artifacts

As an authenticated user, I can create an Artifact by uploading a file. [done]

As an authenticated user, I can specify a size and/or digest to validate the uploaded file. [done]

Content Units

As an authenticated user, I can create a content unit.

As an authenticated user, I can read and list content units.
* As an authenticated user, I can filter content by repository version

As an authenticated user, I can delete a specific content unit if it is not in any repository version. (orphan)

  • Error message saying that the unit is in use by a repo version and a link to the filter to return all of the repo versions.
  • As a user, I know that files (Artifacts) associated with the Content unit are not removed by this call (docs)

Versioned Repositories

CRD

As an authenticated user, I can list versions for a particular repository. [done]

  • Pagination is supported

As an authenticated user, I can filter repository versions by: [3238]

  • number (equality, lt/lte, gt/gte)
  • content id (equality, content_in_list)
  • created datetime (range)

As an authenticated user, I can delete any repository version. [3219]

  • Latest versions are deleted simply.
  • Content set changes are "squashed" into the following version when non-latest versions are deleted.

As an authenticated user, I can view content that was added or removed in a particular repository version (as compared to the previous version). [done]

As an authenticated user, I can see the number of content unit types with counts for each [done][3059]

Repository Version Content

As a user, I know content sets for repository versions are immutable. [done]

As an authenticated user, I can list the content in a particular repository version [done]

  • All fields are included
  • Pagination is supported

As an authenticated user, I can create a new version by adding or removing content to the latest version. [3234]

  • I can follow the progress. (adding/removing are asynchronous).
  • can add and remove together in a single call

Orphan Content Units and Artifacts

As an authenticated user, I can cause an action that cleans up both orphaned content units and orphaned artifacts.

  • I cannot specify the units specifically (all types).
  • I can follow the progress of all cleanups. (Cleanups are asynchronous.)

Task Management

As an authenticated user, I can list all tasks

  • This does not include associated progress reports

As an authenticated user, I can see a detail view for a specific task [done]

  • all attributes of a task
  • all associated progress reports

As an authenticated user, I can cancel a task [done]

  • don't dare to use the DELETE verb!

As an authenticated user, I can delete tasks.

As an authenticated user, I can filter tasks by: [3144]

  • state (equality, state_in_list)
  • started_at(started_in_range)
  • finished_at(finished_in_range)
  • worker (equality)
  • parent (equality)

Status

As an unauthenticated user I can view the status of Pulp workers and resource managers. [done]

As an unauthenticated user I can view the status of the web server's connection to the database and message broker. [done]

As an unauthenticated user I can view the versions of core and each installed plugin.

Workers

As an authenticated user, I can filter workers by: [3143]

  • last_heartbeat (range)
  • name (substring)
  • gracefully_stopped (equality)

h2.

Plugin User Content Management stories

Simple Copy

As an authenticated user of a plugin, I can search (synchronous call) a repository version's content using filtering.

  • note: In a separate call (async), I can add_content_units or remove_content_units to another repository's latest version.

Complex Copy

As a plugin writer I can provide a rich search features with arbitrary viewsets. e.g. depsolving, versioning, etc

Examples of specific plugin use cases motivating the above general viewset

  • As an authenticated user, I can add an Errata from one repository to another repository along with packages mentioned in the Errata and all their dependencies that are present in the source repository.
  • As an authenticated user, I can add bunch of dependencies and store n versions back for each RPM.
  • As an authenticated user, I can use depsolving and versioning copy features together
  • As an authenticated user, I can copy a puppet module and all of it's dependencies from one repository to another.
  • As an authenticated user, I can depsolve units to be added to a destination repo based on an errata

Plugin API

As a plugin writer, I have a plugin API that is semantically versioned at 0.x separate from the REST API [done]

As a plugin writer, my app will be discovered by Pulp's app via an entry point provided by the plugin writer [done]

Task

As a plugin writer, I can report progress with a message and state [done]

As a plugin writer, I can report progress with an optional suffix [done]

As a plugin writer, I can report progress with a total count of things to do an the current count of things done [done]

As a plugin writer, non-fatal exceptions on the Task and are included in the Task detail. non_fatal exceptions do not cause the Task to be marked as failed, but may be interpreted by the user as not fully successful. [done]

As a plugin writer, the working directory is set before Task work is done and cleaned up afterwards. I should not need to interact with the file system outside of the working dir. [done]

Remote

As a plugin writer, I can provide a subclassed Remote. I can add custom fields to the subclassed Remote.

As a plugin writer, I can provide a UserFacingTask to perform a sync operation using information stored in a Remote.

As a plugin writer, I can provide a ViewSet for the subclassed Remote that has a 'sync' endpoint that dispatches the sync task with a reservation for Repository and Remote.

Publisher

As a plugin writer, I can provide a subclassed Publisher. I can add custom fields to the subclassed Publisher.

As a plugin writer, I can provide a UserFacingTask to perform a publish operation using information from the subclassed Publisher.

As a plugin writer, I can provide a ViewSet for the subclassed Publisher that has a 'publish' endpoint that dispatches the publish task with a reservation for Repository and Publisher.

Content

As a plugin writer, I can provide a subclassed Content unit. I can add custom fields to the subclassed Content. [done]

As a plugin writer, I have documentation that shows how I can add filters to filter content responsibly.

As a plugin writer, I have documentation on how to write a filter for my Content that can use the RepositoryVersion manager.
* note: This will allow users to filter content by repository version

Content Management

As a plugin writer, I can interact with and create Artifacts [done]

As a plugin writer, I can query content units/artifacts associated with a repository. [done]

As a plugin writer, I can create a new repository version: [done]

  • When I create the new version, it is incomplete.
  • The new version starts with all the content from the previous version.
  • I can add and remove content units to and from an incomplete repository version.
  • When I am finished with the incomplete version, core finalizes the repository version for me.
  • I have exclusive access to the new repository version while my code is running.

Publishing

As a plugin writer, I can create a new publication: [done]

  • When I create the new publication, it is incomplete.
  • When I am finished with the incomplete publication, core finalizes the publication for me.
  • I have exclusive access to the repository version while my code is running.
  • I have exclusive access to the new publication while my code is running.

"live APIs"

As a plugin writer, I can provide a Live API with arbitrary views and viewsets [3473]
As a plugin writer, I have documentation on what URLs I should not use for my views and viewsets [3473]

Here are some concrete use cases driving the very Live API use cases above:

# Concrete user use cases:
    As an authenticated user, I can use the puppet client to fetch content from Pulp using the Forge API
    As an authenticated user I can use the docker client to fetch content from Pulp using the Docker v1 API
    As an authenticated user I can use the docker client to fetch content from Pulp using the Docker v2 API

# Concrete plugin writer use cases
    As a puppet plugin developer, I can provide a viewset which handles the server side of the puppet Forge v3 API
    As a docker plugin developer, I can provide a viewset which handles the server side of the docker v1 API
    As a docker plugin developer, I can provide a viewset which handles the server side of the docker v2 API

h4. Storage

As a plugin writer, the plugin API provides an API that returns a fully qualified path to a shared and namespaced storage location used to store content. ["3182"https://pulp.plan.io/issues/3182\]
******

Webserver Deployment

As a system administrator, I can deploy all Pulp web applications on one process
As a system administrator, I can deploy the Pulp REST API exclusively in one process
As a system administrator, I can deploy the Pulp content serving view exclusively in one process

As a system administrator, I can deploy all Pulp web applications inside a virtualenv.
As a system administrator, I can deploy all Pulp web applications without root permissions.

CLI

We will use coreapi-cli to generate a one to one mapping of cli commands to rest api schema #3068
We will have a wrapper for coreapi-cli. This wrapper will handle parallel progress reporting

Download API

As a plugin writer, I can download files via

  • http://
  • https://
  • file://

As a plugin writer, I can configure a downloader with:

  • Basic Auth
  • SSL Cert Client Auth
  • Custom CAs will be configured via a "trust store" either on the system or similar. Pulp will not do anything to read/load/manage CAs directly.

As a plugin writer, I can provide arbitrary behaviors for customized downloaders

  • For example token authentication in the docker plugin

As a plugin writer, I can have connection pooling/reuse

As a plugin writer, I have proxy settings

  • proxy url (containing basic auth info)

As a plugin writer, I can have great logs

As an authenticated user, I have documentation about how to use something for bandwidth limiting

As a plugin writer, I can configure the validation mechanisms used at download time

  • checksum validation - minimum (md5, sha1, sha256, sha512)
  • size validation

As a plugin writer, I expect units that are missing from the remote repository to not be created in Pulp when using the immediate download policy.

As a plugin writer, I expect units that are missing from the remote repository to be created in Pulp when using background or on_demand download policies.

As a plugin writer, the plugin API provides tooling whereby I can provide the content to be added and removed from the repository. This tooling supports both immediate and deferred downloading.

As a plugin writer I can manage the catalog by using ChangeSets

As a plugin writer, the plugin can participate in adding content for cases where the decision to add additional content is based on the content that has been downloaded.

As a plugin writer, I can fetch content myself (but I am not encouraged to do so) with code I write

As a plugin writer, I can CRUD content units need a convention to handle multiple content units - see https://pulp.plan.io/issues/3472

Migrations only involving Pulp 3

Users can run "pulp-manager migrate" to migrate the database and adjust state in other locations (filesystem, message broker, ...). [done]

Web Server Integration

As a user, I can have content efficiently served to me by Apache by Pulp using the X-SEND response headers. [done]
As a user, I can have content efficiently served to me by Nginx by Pulp using the X-Accel-Redirect response headers. [done]

As a user, I can have an Ansible role to install Apache which enables Apache integration for Pulp and configures Apache to serve Pulp. [2921]
As a user, I can have an Ansible role to install Nginx which enables Nginx integration for Pulp and configures Nginx to serve Pulp. [2922]

Glossary

Add (Content Unit): An operation causing a new repository version to contain a content unit(s)

Applicability - A plugin defined term meaning when a package update available in a repository is applicable to a given consumer as determined by the Consumer Profile.

Artifact - A file associated with one content (unit). Artifacts are not shared between content (units). Create a content unit using an uploaded file ID as the source for its metadata. Create Artifacts associated with the content unit using an uploaded file ID for each; commit as a single transaction.

Content (unit) - A single piece of content manged by Pulp. Each file associated with a content (unit) is called an Artifact. Each content (unit) may have zero or many Artifacts.

Distribution: Where and how the content app serves a Publication. i.e. http vs https and base path component of the URL. A Distribution defines:

  • the base path of the repository publication (required)
  • serving via http (default=False)
  • serving via https (default=True)
  • relationship w/ a Publisher for auto-distribution (should be allowed to be NULL)
  • relationship with Publication (should be allowed to be NULL)

Exporter: An Exporter exports a Publication out of Pulp. e.g. rsync exporter exports content to a remote server

Live API: a viewset endpoint contributed by plugin. For examples see the associated MVP section

Orphan Artifact: An Artifact that is associated with 0 Content Units and 0 Publications

Orphan Content (unit): A content unit that is a member of 0 repository versions

Remove (content unit): An operation causing a new repository version to not contain a content unit(s)

Repository - A named collection of repository versions.

Repository Version - An immutable set of content which is versioned by a sequential number.

Updated by bmbouter over 6 years ago · 158 revisions