Pulp 3.0.0 Minimum Viable Product (MVP)¶

Lines highlighted in red need more attention.

Overall Guarantees¶

• This is not a direct replacement for Pulp 2.
• This is the minimum required for a 3.0.0 beta and GA.
• All REST API calls will update the DB using transactions as necessary to ensure data integrity.

Legend¶

[done] means merged and documented
[in-progress] means started but not fully done
If there is no label the effort has not yet been started

Authentication¶

As an authenticated user I can manage user(s). [done]

• Add a user
• View user(s)
• Update any user detail
• Delete a user

As an API user, I can have documentation to generate a JSON Web Token (JWT) without the server being online. [done]

As an administrator, I can disable JWT token expiration. This configuration is in the settings file and is system-wide. [done]

As an administrator, I can configure the JWT tokens to expire after a configurable amount of time. This configuration is in the settings file and is system-wide. [done]

The JWT shall have a username identifier [done]

As an API user, I can authenticate any API call with a valid username and password [done]

As an API user, I can authenticate any API call with a valid JWT [3163]

As a JWT authenticated user, I can refresh my JWT token if Pulp is configured with JWT_ALLOW_REFRESH set to True (default is False) [3163]

As an API user, I can invalidate all existing JWT tokens for a given user. [done]

As an authenticated user, when deleting a user 'foo', all of user 'foo's existing JWTs are invalidated. [done]

As an un-authenticated user, I can obtain a JWT token by using a username and password. [done]

As an authenticated user, I can filter users by: [3142]

• username: (equality, username_in_list)

Repositories¶

As an authenticated user, I can list all repos.

• All fields are included [done]
• Pagination is supported [done]

As an authenticated user I can use filters on Repositories list: [3079]

• id: (id_in_list) # id equality is not necessary, v3/repositories/<UUID>/
• name: (equality, name_in_list)

As an authenticated user, I can CRUD a repository

• Create a repo [done]
• Read a repo [done]
• Update all mutable repo fields [done]
• Delete a repo (asynchronous) [done]

As an authenticated user, I can view content for a repository.

As an authenticated user, I can list a repository's associated importers and publishers

• All fields are included [done]
• Pagination is supported [done]

As an authenticated user, I can see the number of content unit types with counts for each [done][3059]

Importers¶

note: Importer attributes will commonly be available on importers, but aren't guaranteed to be used by all importers.

As an authenticated user, I can CRUD an importer

• Create an importer
• Read an importer
• Update all mutable importer fields
• Delete an importer (asynchronous)

As an authenticated user, I have filters on the Importer list: [3080]

• id: (id_in_list) # id equality is not necessary, objects are referenced by id
• name: (equality, name_in_list)

As an authenticated user I can configure the following attributes on an Importer: [done]

• validate (bool) [optional: defaults to True]: If true, the plugin will validate imported content.
• ssl_ca_certificate (str) [optional] String containing a PEM encoded CA certificate used to validate the server certificate presented by the external source.
• ssl_client_certificate (str) [optional] Contains a PEM encoded client certificate used for authentication.
• ssl_client_key (str) [optional] Contains a PEM encoded private key used for authentication.
• ssl_validation (bool) [optional: defaults to True]: If true, SSL peer validation must be performed.
• proxy_url (str) [optional] Contains the proxy URL. Format: scheme://user:password@host:port.
• username (str) [optional] Contains the username to be used for authentication when syncing.
• password (str) [optional] Contains the password to be used for authentication when syncing.
• download_policy (str) [optional] Contains the downloading policy name. This is a choice of three options:
  - immediate (default) - Downloading occurs during sync. The sync task does not complete until downloading is done.
  - background - Downloading is started by the sync but occurs in the background. The sync task completes before downloading is complete.
  - on-demand - The sync task records everything that would be downloaded but does not download content. Downloading occurs on demand as driven by client requests for content.
• feed_url (str) [optional] Contains the URL of an external content source. This is optional.
• sync mode: (str) [optional]. It has two choices 'additive' and 'mirror':
  - additive (default) - all remote content is added to the local repository on sync. During sync no content is removed from the local repository.
  - mirror - the local content will mirror the remote content exactly, removing local content if not also present in the remote content.
• name (str) [required] Contains the name.
• last_updated (datetime) [read-only] Contains the datetime of the last importer update.
• last_synced (datetime) [read-only] Contains the datetime of the last importer sync.

Publishers¶

note: Publisher attributes will commonly be available on publishers, but aren't guaranteed to be used by all publishers.

As an authenticated user, I can CRUD a publisher

• Create a publisher
• Read a publisher
• Update all mutable publisher fields
• Delete a publisher (asynchronous)

As an authenticated user, I have filters on the Publisher list: [3081]

• id: (id_in_list) # id equality is not necessary, objects are referenced by id
• name: (equality, name_in_list)

As an authenticated user I can configure the following attributes on a Publisher:

• relative_path (str) [optional] The (relative) path component of the published url.
• name - (str) [required] contains the name.
• last_published (datetime) [read-only] When the last successful publis%{color:red}h occurred.
• last_updated (datetime) [read-only] The datetime of the last publisher update.
• auto_publish(bool) - ??? consider adding auto-publish feature to MVP

Distributions¶

As an authenticated user, I can CRUD Distributions:

• Create a Distribution.
• Read a Distribution
• List Distributions
• Update all mutable Distribution fields (synchronous)
  - base path
  - http
  - https
• Delete an Distribution (synchronous)

As a user, my distribution base paths don't conflict and my create/update is rejected identifying the conflicting distributions [2987]

As an authenticated user, I can create or update a distribution that is not associated with any publication (NULL)

As an authenticated user, I can create or update a distribution that is not associated with any publisher (NULL)

As a user, I can update a Distribution to distribute a specific Publication

As a user, I want a newly created publication to be automatically served by the content as defined by distributions.

As a user, I can see the full urls my base path is served at (one for http and one for https depending on what is on)

As an authenticated user, I have filters on the Distribution list: [3082]

• name: (equality, name_in_list)
• base_path: (equality, substring, base_path_in_list)

Publications¶

As an authenticated user, I can CRD Publication(s)

• Create a publication which triggers a publish. This is asynchronous locking the repository.
• Read a Publication - id, created datetime, list of distribution hrefs
• List all Publications - ordered by created datetime in descending order
• Delete a Publication - asyncronously with a lock on the repository.
  - prevented if associated with a distribution.
  - single object only.

As an authenticated user, I can list publications and I have enough information to choose which ones to delete.

• choose by created (older ones or perhaps latest)
• choose not associated to a distribution.
• does not imply filtering

As an authenticated user, I can list publications and i have enough information to select a publication to be associated with a distribution.

• choose by created (latest or just by publish date)
  - manual promotion. "My rawhide publication has been tested and now I want to promote it to stable".
  - rollback to an earlier publication.
• does not imply filtering

As an authenticated user, I can determine if and how a publication is distributed.

Sync and Publish¶

As an authenticated user, I can trigger an importer to sync. [done]

• I can follow the progress of all syncs. (Syncs are asynchronous.)
• I cannot pass "sync" options.
• Auto-publish is not included as an importer property.

As an authenticated user, I can trigger a publisher to publish. [done]

• I can follow the progress of all publishes. (Publishes are asynchronous.)
• I cannot pass "publish" options.

Content Manipulation¶

Uploading Artifacts¶

As an authenticated user, I can create an Artifact by uploading a file. [done]

As an authenticated user, I can specify a size and/or digest to validate the uploaded file. [done]

Creating Content Units¶

As an authenticated user, I can create a content unit by providing the content type (in the URL), references to Artifacts, and the metadata supplied in the POST body. [done]

Add / Remove Content from a Repository¶

As an authenticated user, I can add content (specific existing content) to a repository

• I can specify one or more content units by id/href
• I can follow the progress. (adding are asynchronous).

As an authenticated user, I can remove specified content from a repository

• I can specify one or more content units by id/href
• I can follow the progress. (removing are asynchronous).

Simple Copy¶

As an authenticated user, I can search (synchronous call) a repository's content using filtering.
* note: In a separate call (async), I can add this content to another repository. This is effectively a copy operation in two calls.

As an authenticated user, I can import all content from one repository into another repository in a single async call. (Clone use case)

• I can follow the progress. (adding are asynchronous).

Complex Copy¶

As a plugin writer I can provide a rich search features with abitrary viewsets. e.g. depsolving, verisoning, etc

Examples of specific plugin use cases motivating the above general viewset

• As an authenticated user, I can add an Errata from one repository to another repository along with packages mentioned in the Errata and all their dependencies that are present in the source repository.
• As an authenticated user, I can add bunch of dependencies and store n versions back for each RPM.
• As an authenticated user, I can use depsolving and versioning copy features together
• As an authenticated user, I can copy a puppet module and all of it's dependencies from one repository to another.
• As an authenticated user, I can depsolve units to be added to a destination repo based on an errata

Delete¶

FAs an authenticated user, I can delete a specific content unit%
* If the content unit is still in at least one repository the delete fails with a listing of all repositories the unit is part of.
* Artifacts and associated files from the deleted unit are cleaned up

As an authenticated user, I can delete multiple content units with filtering
* If a content unit is still in at least one repository the delete fails with a listing of all repositories the unit is part of.
* Artifacts and associated files from deleted units are cleaned up

Filtering¶

As an authenticated user, I can filter Content by repository information when plugin writers have provided such a filter

As an authenticated user, I can filter content by repository

As an authenticated user, I can filter content by type

As an authenticated user, I can filter content on plugin specific attributes when plugin writers have provided such a filter

Versioned Repositories¶

As an authenticated user, I can list the content in a particular repository version

• All fields are included
• Pagination is supported
• Filtering support

As an authenticated user, I can discover a URL to the latest version of a repository

As an authenticated user, I can run a publisher without a repository version and have it default to the latest version.

As an authenticated user, I can delete a repository version by specifying the version

As an authenticated user, I can upload multiple content(s?) and add create a single new version that adds all of them.

Orphan Content Units and Artifacts¶

As an authenticated user, I can cause an action that cleans up both orphaned content units and orphaned artifacts.

• I cannot specify the units specifically (all types).
• I can follow the progress of all cleanups. (Cleanups are asynchronous.)

Task Management¶

As an authenticated user, I can list all tasks

• Filtering support on ['state', 'id', 'group']
• This does not include associated progress reports

As an authenticated user, I can see a detail view for a specific task [done]

• all attributes of a task
• all associated progress reports

As an authenticated user, I can cancel a task [done]

• don't dare to use the DELETE verb!

As an authenticated user, I can delete tasks.

As an authenticated user, I can filter tags by: [3144]

• state (equality, state_in_list)
• started_at(started_in_range)
• finished_at(finished_in_range)
• worker (equality)
• parent (equality)

Task Group¶

I can view a summary of the status of all tasks in a group

Status¶

As an unauthenticated user I can view the status of Pulp workers and resource managers. [done]

As an unauthenticated user I can view the status of the web server's connection to the database and message broker. [done]

As an unauthenticated user I can view the versions of core and each installed plugin.

Workers¶

As an authenticated user, I can filter workers by: [3143]

• last_heartbeat (range)
• name (substring)
• gracefully_stopped (equality)

Plugin API¶

As a plugin writer, I have a plugin API that is semantically versioned at 0.x separate from the REST API [done]

As a plugin writer, I can report progress with a message and state [done]

As a plugin writer, I can report progress with an optional suffix [done]

As a plugin writer, I can report progress with a total count of things to do an the current count of things done [done]

As a plugin writer, non-fatal exceptions on the Task and are included in the Task detail. non_fatal exceptions do not cause the Task to be marked as failed, but may be interpreted by the user as not fully successful. [done]

As a plugin writer, the working directory is set before Task work is done and cleaned up afterwards. I should not need to interact with the file system outside of the working dir. [done]

As a plugin writer, I can provide a subclassed Importer. The importer's responsibility is to synchronize the content of a Pulp repository with the content of a remote repository. (a circular import problem needs to be discussed and may cause this to change) [done]

As a plugin writer, I can provide a subclassed Publisher. The publisher's responsibility is to publish content. (a circular import problem needs to be discussed and may cause this to change) [done]

As a plugin writer, I can define unit types by subclassing Content models to provide concrete content unit types to be manged by the platform. [done]

As a plugin writer, I can interact with and create Artifacts [done]

As a plugin writer, my app will be discovered by Pulp's app via an entry point provided by the plugin writer [done]

As a plugin writer, I can use the plugin API to query content units/artifacts associated with a repository. [done]

As a plugin writer, I can add and remove content units to and from a repository. [done]

As a plugin writer, I have documentation that shows how I can add filters to filter content responsibly.

As a plugin writer, I have documentation on how to write a filter for my Content that can use the RepositoryContent through table.

• note: This will allow users to filter content by repository information

Plugin Writer "live APIs"¶

As a plugin writer, I can provide a Live API with arbitrary views and viewsets.
As a plugin writer, I have documentation on what URLs I should -not use for my views and viewsets

Here are some concrete use cases driving the very Live API use cases above:

# Concrete user use cases:
    As an authenticated user, I can use the puppet client to fetch content from Pulp using the Forge API
    As an authenticated user I can use the docker client to fetch content from Pulp using the Docker v1 API
    As an authenticated user I can use the docker client to fetch content from Pulp using the Docker v2 API

# Concrete plugin writer use cases
    As a puppet plugin developer, I can provide a viewset which handles the server side of the puppet Forge v3 API
    As a docker plugin developer, I can provide a viewset which handles the server side of the docker v1 API
    As a docker plugin developer, I can provide a viewset which handles the server side of the docker v2 API

Webserver Deployment¶

As an authenticated user, I can deploy all Pulp webservices on one process

As an authenticated user, I can deploy the Pulp REST API exclusively in one process
As an authenticated user, I can deploy the Pulp content serving view exclusively in one process
As an authenticated user, I can deploy the Pulp content serving view with reduced permissions.
As an authenticated user, I can deploy a Plugin's Live API exclusively in one process.

CLI¶

We will use coreapi-cli to generate a one to one mapping of cli commands to rest api schema #3068
We will have a wrapper for coreapi-cli. This wrapper will handle parallel progress reporting

Download API¶

As a plugin writer, I can download files via

• http://
• https://
• file://

As a plugin writer, I can configure a downloader with:

• Basic Auth
• SSL Cert Client Auth
• Custom CAs will be configured via a "trust store" either on the system or similar. Pulp will not do anything to read/load/manage CAs directly.

As a plugin writer, I can provide arbitrary behaviors for customized downloaders

• For example token authentication in the docker plugin

As a plugin writer, I can have connection pooling/reuse

As a plugin writer, I have proxy settings

• proxy url (containing basic auth info)

As a plugin writer, I can have great logs

As an authenticated user, I have documentation about how to use something for bandwidth limiting

As a plugin writer, I can configure the validation mechanisms used at download time

• checksum validation - minimum (md5, sha1, sha256, sha512)
• size validation

As a plugin writer, I expect units that are missing from the remote repository to not be created in Pulp when using the immediate download policy.

As a plugin writer, I expect units that are missing from the remote repository to be created in Pulp when using background or on_demand download policies.

As a plugin writer I can configure mirror lists and rotate between the mirrors
* round robin
* nearest mirror support

As a plugin writer, the plugin API provides tooling whereby I can provide the content to be added and removed from the repository. This tooling supports both immediate and deferred downloading.

As a plugin writer I can manage the catalog by using ChangeSets

As a plugin writer, the plugin can participate in adding content for cases where the decision to add additional content is based content that has been downloaded.

As a plugin writer, I can fetch content myself (but I am not encouraged to do so) with code I write

As a plugin writer, I can CRUD content units

Consumer Profile Applicability¶

Using Consumer Profiles and repository bindings I can compute applicability with 2.y parity

• Performance needs to be awesome%

Migrations only involving Pulp 3¶

Users can run "pulp-manager migrate" to migrate the database and adjust state in other locations (filesystem, message broker, ...). [done]

Web Server Integration¶

As a user, I can have content efficiently served to me by Apache by Pulp using the X-SEND response headers. [done]
As a user, I can have content efficiently served to me by Nginx by Pulp using the X-Accel-Redirect response headers. [done]

As a user, I can have an Ansible role to install Apache which enables Apache integration for Pulp and configures Apache to serve Pulp. [2921]
As a user, I can have an Ansible role to install Nginx which enables Nginx integration for Pulp and configures Nginx to serve Pulp. [2922]

Glossary¶

Repository - A named collection of content.

Artifact - A file associated with one content (unit). Artifacts are not shared between content (units). Create a content unit using an uploaded file ID as the source for its metadata. Create Artifacts associated with the content unit using an uploaded file ID for each; commit as a single transaction.

Content (unit) - A single piece of content manged by Pulp. Each file associated with a content (unit) is called an Artifact. Each content (unit) may have zero or many Artifacts.

Orphan Content (unit): A content unit that is a member of 0 repositories

Orphan Artifact: An Artifact that is associated with 0 Content Units and 0 Publications

Add (Content Unit): An operation causing a repository to contain a content unit(s)

Remove (content unit): An operation causing a repository to not contain a content unit(s)

Consumer Profile - A set of installed units on a specific machine. In Pulp3 this machine is not a "consumer" in the same sense as Pulp2. Pulp is not "managing" the machine anymore; Pulp3 only collects Profile information.

Applicability - A plugin defined term meaning when a package update available in a repository is applicable to a given consumer as determined by the Consumer Profile.

Distribution: Where and how the content app serves a Publication. i.e. http vs https and base path component of the URL. A Distribution defines:

• the base bath of the repository publication (required)
• serving via http (default=False)
• serving via https (default=True)
• relationship w/ a Publisher for auto-distribution (should be allowed to be NULL)
• relationship with Publication (should be allowed to be NULL)