Pulp 3.0.0 Minimum Viable Product (MVP)¶

Lines highlighted in red need more attention.

Overall Guarantees¶

• This is not a direct replacement for Pulp 2.
• This is the minimum required for a 3.0.0 beta and GA.
• All REST API calls will update the DB using transactions as necessary to ensure data integrity.

Legend¶

[done] means merged and documented
[in-progress] means started but not fully done
If there is no label the effort has not yet been started

Authentication¶

As an authenticated user I can manage user(s). [done]

• Add a user
• View user(s)
• Update any user detail
• Delete a user

As an API user, I can have documentation to generate a JSON Web Token (JWT) without the server being online. [done]

As an administrator, I can disable JWT token expiration. This configuration is in the settings file and is system-wide. [done]

As an administrator, I can configure the JWT tokens to expire after a configurable amount of time. This configuration is in the settings file and is system-wide. [done]

The JWT shall have a username identifier [done]

As an API user, I can authenticate any API call with a valid username and password [done]

As an API user, I can authenticate any API call with a valid JWT [3163]

As a JWT authenticated user, I can refresh my JWT token if Pulp is configured with JWT_ALLOW_REFRESH set to True (default is False) [3163]

As an API user, I can invalidate all existing JWT tokens for a given user. [done]

As an authenticated user, when deleting a user 'foo', all of user 'foo's existing JWTs are invalidated. [done]

As an un-authenticated user, I can obtain a JWT token by using a username and password. [done]

As an authenticated user, I can filter users by: [3142]

• username: (equality, username_in_list)

Repositories¶

As an authenticated user, I can list all repos.

• All fields are included [done]
• Pagination is supported [done]

As an authenticated user I can use filters on Repositories list: [3079]

• id: (id_in_list) # id equality is not necessary, v3/repositories/<UUID>/
• name: (equality, name_in_list)

As an authenticated user, I can CRUD a repository

• Create a repo [done]
• Read a repo [done]
• Update all mutable repo fields [done]
• Delete a repo (asynchronous)

As an authenticated user, I can list a repository's associated importers and publishers

• All fields are included [done]
• Pagination is supported [done]

As an authenticated user, I can see the number of content unit types with counts for each [done][3059]

As an authenticated user, when viewing a repository, I can discover a URL to the latest version of a repository.

Importers¶

note: Importer attributes will commonly be available on importers, but aren't guaranteed to be used by all importers.

As an authenticated user, I can CRUD an importer

• Create an importer
• Read an importer
• Update all mutable importer fields
• Delete an importer (asynchronous)

As an authenticated user, I have filters on the Importer list: [3080]

• id: (id_in_list) # id equality is not necessary, objects are referenced by id
• name: (equality, name_in_list)

As an authenticated user I can configure the following attributes on an Importer: [done]

• validate (bool) [optional: defaults to True]: If true, the plugin will validate imported content.
• ssl_ca_certificate (str) [optional] String containing a PEM encoded CA certificate used to validate the server certificate presented by the external source.
• ssl_client_certificate (str) [optional] Contains a PEM encoded client certificate used for authentication.
• ssl_client_key (str) [optional] Contains a PEM encoded private key used for authentication.
• ssl_validation (bool) [optional: defaults to True]: If true, SSL peer validation must be performed.
• proxy_url (str) [optional] Contains the proxy URL. Format: scheme://user:password@host:port.
• username (str) [optional] Contains the username to be used for authentication when syncing.
• password (str) [optional] Contains the password to be used for authentication when syncing.
• download_policy (str) [optional] Contains the downloading policy name. This is a choice of three options:
  - immediate (default) - Downloading occurs during sync. The sync task does not complete until downloading is done.
  - background - Downloading is started by the sync but occurs in the background. The sync task completes before downloading is complete.
  - on-demand - The sync task records everything that would be downloaded but does not download content. Downloading occurs on demand as driven by client requests for content.
• feed_url (str) [optional] Contains the URL of an external content source. This is optional.
• sync mode: (str) [optional]. It has two choices 'additive' and 'mirror':
  - additive (default) - all remote content is added to the base repo version on sync. During sync no content is removed from the base repo version.
  - mirror - the local content will mirror the remote content exactly, removing local content if not also present in the remote content.
• name (str) [required] Contains the name.
• last_updated (datetime) [read-only] Contains the datetime of the last importer update.
• last_synced (datetime) [read-only] Contains the datetime of the last importer sync.

As an authenticated user, I can define a default importer on a repo via href to be used whenever a new repo version is created.

Publishers¶

note: Publisher attributes will commonly be available on publishers, but aren't guaranteed to be used by all publishers.

As an authenticated user, I can CRUD a publisher

• Create a publisher
• Read a publisher
• Update all mutable publisher fields
• Delete a publisher (asynchronous)

As an authenticated user, I have filters on the Publisher list: [3081]

• id: (id_in_list) # id equality is not necessary, objects are referenced by id
• name: (equality, name_in_list)

As an authenticated user I can configure the following attributes on a Publisher:

• relative_path (str) [optional] The (relative) path component of the published url.
• name - (str) [required] contains the name.
• last_published (datetime) [read-only] When the last successful publis%{color:red}h occurred.
• last_updated (datetime) [read-only] The datetime of the last publisher update.
• auto_publish(bool) - ??? consider adding auto-publish feature to MVP

Distributions¶

As an authenticated user, I can CRUD Distributions:

• Create a Distribution.
• Read a Distribution
• List Distributions
• Update all mutable Distribution fields (synchronous)
  - base path
  - http
  - https
• Delete an Distribution (synchronous)

As a user, my distribution base paths don't conflict and my create/update is rejected identifying the conflicting distributions [2987]

As an authenticated user, I can create or update a distribution that is not associated with any publication (NULL)

As an authenticated user, I can create or update a distribution that is not associated with any publisher (NULL)

As a user, I can update a Distribution to distribute a specific Publication

As a user, I want a newly created publication to be automatically served by the content as defined by distributions.

As a user, I can see the full urls my base path is served at (one for http and one for https depending on what is on)

As an authenticated user, I have filters on the Distribution list: [3082]

• name: (equality, name_in_list)
• base_path: (equality, substring, base_path_in_list)

Publications¶

As an authenticated user, when creating a Publication, I can post a repo version href to be published.

As an authenticated user, I can publish a repository's latest version by posting a repository href to be published.

As an authenticated user, I can view which repository version was used to create a particular publication.

As an authenticated user, I can read Publication(s)

• Read a Publication - id, created datetime, list of distribution hrefs, repo version
• List all Publications - ordered by created datetime in descending order

As an authenticated user, I can delete publications.
- asynchronously with a lock on the repository version.
- prevented if associated with a distribution.
- single object only.

As an authenticated user, I can list publications and I have enough information to choose which ones to delete.

• choose by created (older ones or perhaps latest)
• choose not associated to a distribution.
• does not imply filtering

As an authenticated user, I can list publications and i have enough information to select a publication to be associated with a distribution.

• choose by created (latest or just by publish date)
  - manual promotion. "My rawhide publication has been tested and now I want to promote it to stable".
  - rollback to an earlier publication.
• does not imply filtering

As an authenticated user, I can determine if and how a publication is distributed.

Sync and Publish¶

As an authenticated user, I can trigger an importer to sync. [done]

• I can follow the progress of all syncs. (Syncs are asynchronous.)
• I cannot pass "sync" options.
• Auto-publish is not included as an importer property.

As an authenticated user, I can trigger a publisher to publish. [done]

• I can follow the progress of all publishes. (Publishes are asynchronous.)
• I cannot pass "publish" options.

Content Manipulation¶

Uploading Artifacts¶

As an authenticated user, I can create an Artifact by uploading a file. [done]

As an authenticated user, I can specify a size and/or digest to validate the uploaded file. [done]

Creating Content Units¶

As an authenticated user, I can create a content unit by providing the content type (in the URL), references to Artifacts, and the metadata supplied in the POST body. [done]

Simple Copy¶

As an authenticated user, I can search (synchronous call) a repository's content using filtering.
* note: In a separate call (async), I can add this content to another repository. This is effectively a copy operation in two calls.

As an authenticated user, I can import all content from one repository into another repository in a single async call. (Clone use case)

• I can follow the progress. (adding are asynchronous).

Complex Copy¶

As a plugin writer I can provide a rich search features with abitrary viewsets. e.g. depsolving, verisoning, etc

Examples of specific plugin use cases motivating the above general viewset

• As an authenticated user, I can add an Errata from one repository to another repository along with packages mentioned in the Errata and all their dependencies that are present in the source repository.
• As an authenticated user, I can add bunch of dependencies and store n versions back for each RPM.
• As an authenticated user, I can use depsolving and versioning copy features together
• As an authenticated user, I can copy a puppet module and all of it's dependencies from one repository to another.
• As an authenticated user, I can depsolve units to be added to a destination repo based on an errata

Delete¶

As an authenticated user, I can delete a specific content unit

• If the content unit is still in at least one repository version the delete fails
• Error message saying that the unit is in use by a repo version and a link to the filter to return all of the repo versions.
• Content unit deletion needs to be race condition free. For example: cannot delete a content unit during (for example)
  - sync that may be assuming the content exits.
  - copy operation

• As a user, I know that files (Artifacts) associated with the Content unit are not removed by this call (docs)

Filtering¶

As an authenticated user, I can filter Content by repository version information when plugin writers have provided such a filter

As an authenticated user, I can filter content by repository version

As an authenticated user, I can filter content by type

As an authenticated user, I can filter content on plugin specific attributes when plugin writers have provided such a filter

Versioned Repositories¶

CRD¶

As an authenticated user, I can create a new repository version. [3173]

As an authenticated user, I can list versions for a particular repository.

As an authenticated user, I can filter repository versions by:

• number (equality, lt/lte, gt/gte)
• content id (equality, content_in_list)

As an authenticated user, I can delete any repository version.

As an authenticated user, I can view attributes on a repository version including its base_version, importer, added/removed content ids.

Repository Version Content¶

As an authenticated user, I can retrieve read-only immutable content sets by href.

As an authenticated user, I can list the content in a particular repository version

• All fields are included
• Pagination is supported
• Filtering support

As an autheticated user, I can create a new version by adding or removing content to a particular base version (default is latest).

• I can follow the progress. (adding/removing are asynchronous).
• can add and remove in a single call

As an authenticated user, I can specify the base_version of content to be the basis for the new repository version. This defaults to the latest repo version associated with the repository.

Orphan Content Units and Artifacts¶

As an authenticated user, I can cause an action that cleans up both orphaned content units and orphaned artifacts.

• I cannot specify the units specifically (all types).
• I can follow the progress of all cleanups. (Cleanups are asynchronous.)

Task Management¶

As an authenticated user, I can list all tasks

• Filtering support on ['state', 'id', 'group']
• This does not include associated progress reports

As an authenticated user, I can see a detail view for a specific task [done]

• all attributes of a task
• all associated progress reports

As an authenticated user, I can cancel a task [done]

• don't dare to use the DELETE verb!

As an authenticated user, I can delete tasks.

As an authenticated user, I can filter tags by: [3144]

• state (equality, state_in_list)
• started_at(started_in_range)
• finished_at(finished_in_range)
• worker (equality)
• parent (equality)

Task Group¶

I can view a summary of the status of all tasks in a group

Status¶

As an unauthenticated user I can view the status of Pulp workers and resource managers. [done]

As an unauthenticated user I can view the status of the web server's connection to the database and message broker. [done]

As an unauthenticated user I can view the versions of core and each installed plugin.

Workers¶

As an authenticated user, I can filter workers by: [3143]

• last_heartbeat (range)
• name (substring)
• gracefully_stopped (equality)

Plugin API¶

As a plugin writer, I have a plugin API that is semantically versioned at 0.x separate from the REST API [done]

As a plugin writer, I can report progress with a message and state [done]

As a plugin writer, I can report progress with an optional suffix [done]

As a plugin writer, I can report progress with a total count of things to do an the current count of things done [done]

As a plugin writer, non-fatal exceptions on the Task and are included in the Task detail. non_fatal exceptions do not cause the Task to be marked as failed, but may be interpreted by the user as not fully successful. [done]

As a plugin writer, the working directory is set before Task work is done and cleaned up afterwards. I should not need to interact with the file system outside of the working dir. [done]

As a plugin writer, I can provide a subclassed Importer. The importer's responsibility is to synchronize the content of a Pulp repository with the content of a remote repository. (a circular import problem needs to be discussed and may cause this to change) [done]

As a plugin writer, I can provide a subclassed Publisher. The publisher's responsibility is to publish content. (a circular import problem needs to be discussed and may cause this to change) [done]

As a plugin writer, I can define unit types by subclassing Content models to provide concrete content unit types to be manged by the platform. [done]

As a plugin writer, I can interact with and create Artifacts [done]

As a plugin writer, my app will be discovered by Pulp's app via an entry point provided by the plugin writer [done]

As a plugin writer, I can use the plugin API to query content units/artifacts associated with a repository. [done]

As a plugin writer, I can add and remove content units to and from a repository. [done]

As a plugin writer, I have documentation that shows how I can add filters to filter content responsibly.

As a plugin writer, I have documentation on how to write a filter for my Content that can use the RepositoryContent through table.

• note: This will allow users to filter content by repository information

Plugin Writer "live APIs"¶

As a plugin writer, I can provide a Live API with arbitrary views and viewsets.
As a plugin writer, I have documentation on what URLs I should -not use for my views and viewsets

Here are some concrete use cases driving the very Live API use cases above:

# Concrete user use cases:
    As an authenticated user, I can use the puppet client to fetch content from Pulp using the Forge API
    As an authenticated user I can use the docker client to fetch content from Pulp using the Docker v1 API
    As an authenticated user I can use the docker client to fetch content from Pulp using the Docker v2 API

# Concrete plugin writer use cases
    As a puppet plugin developer, I can provide a viewset which handles the server side of the puppet Forge v3 API
    As a docker plugin developer, I can provide a viewset which handles the server side of the docker v1 API
    As a docker plugin developer, I can provide a viewset which handles the server side of the docker v2 API

Webserver Deployment¶

As an authenticated user, I can deploy all Pulp webservices on one process

As an authenticated user, I can deploy the Pulp REST API exclusively in one process
As an authenticated user, I can deploy the Pulp content serving view exclusively in one process
As an authenticated user, I can deploy the Pulp content serving view with reduced permissions.
As an authenticated user, I can deploy a Plugin's Live API exclusively in one process.

CLI¶

We will use coreapi-cli to generate a one to one mapping of cli commands to rest api schema #3068
We will have a wrapper for coreapi-cli. This wrapper will handle parallel progress reporting

Download API¶

As a plugin writer, I can download files via

• http://
• https://
• file://

As a plugin writer, I can configure a downloader with:

• Basic Auth
• SSL Cert Client Auth
• Custom CAs will be configured via a "trust store" either on the system or similar. Pulp will not do anything to read/load/manage CAs directly.

As a plugin writer, I can provide arbitrary behaviors for customized downloaders

• For example token authentication in the docker plugin

As a plugin writer, I can have connection pooling/reuse

As a plugin writer, I have proxy settings

• proxy url (containing basic auth info)

As a plugin writer, I can have great logs

As an authenticated user, I have documentation about how to use something for bandwidth limiting

As a plugin writer, I can configure the validation mechanisms used at download time

• checksum validation - minimum (md5, sha1, sha256, sha512)
• size validation

As a plugin writer, I expect units that are missing from the remote repository to not be created in Pulp when using the immediate download policy.

As a plugin writer, I expect units that are missing from the remote repository to be created in Pulp when using background or on_demand download policies.

As a plugin writer, the plugin API provides tooling whereby I can provide the content to be added and removed from the repository. This tooling supports both immediate and deferred downloading.

As a plugin writer I can manage the catalog by using ChangeSets

As a plugin writer, the plugin can participate in adding content for cases where the decision to add additional content is based content that has been downloaded.

As a plugin writer, I can fetch content myself (but I am not encouraged to do so) with code I write

As a plugin writer, I can CRUD content units

Consumer Profile Applicability¶

Using Consumer Profiles and repository bindings I can compute applicability with 2.y parity

• Performance needs to be awesome%

Migrations only involving Pulp 3¶

Users can run "pulp-manager migrate" to migrate the database and adjust state in other locations (filesystem, message broker, ...). [done]

Web Server Integration¶

As a user, I can have content efficiently served to me by Apache by Pulp using the X-SEND response headers. [done]
As a user, I can have content efficiently served to me by Nginx by Pulp using the X-Accel-Redirect response headers. [done]

As a user, I can have an Ansible role to install Apache which enables Apache integration for Pulp and configures Apache to serve Pulp. [2921]
As a user, I can have an Ansible role to install Nginx which enables Nginx integration for Pulp and configures Nginx to serve Pulp. [2922]

Glossary¶

Add (Content Unit): An operation causing a new repository version to contain a content unit(s)

Applicability - A plugin defined term meaning when a package update available in a repository is applicable to a given consumer as determined by the Consumer Profile.

Artifact - A file associated with one content (unit). Artifacts are not shared between content (units). Create a content unit using an uploaded file ID as the source for its metadata. Create Artifacts associated with the content unit using an uploaded file ID for each; commit as a single transaction.

Consumer Profile - A set of installed units on a specific machine. In Pulp3 this machine is not a "consumer" in the same sense as Pulp2. Pulp is not "managing" the machine anymore; Pulp3 only collects Profile information.

Content (unit) - A single piece of content manged by Pulp. Each file associated with a content (unit) is called an Artifact. Each content (unit) may have zero or many Artifacts.

Distribution: Where and how the content app serves a Publication. i.e. http vs https and base path component of the URL. A Distribution defines:

• the base path of the repository publication (required)
• serving via http (default=False)
• serving via https (default=True)
• relationship w/ a Publisher for auto-distribution (should be allowed to be NULL)
• relationship with Publication (should be allowed to be NULL)

Orphan Artifact: An Artifact that is associated with 0 Content Units and 0 Publications

Orphan Content (unit): A content unit that is a member of 0 repository versions

Remove (content unit): An operation causing a new repository version to not contain a content unit(s)

Repository - A named collection of repository versions.

Repository Version - An immutable set of content which is versioned by a sequential number.