Project

Profile

Help

Actions
History

Pulp 3.0.0 Minimum Viable Product (MVP)

Lines highlighted in red need more attention.

Overall Guarantees

  • This is not a direct replacement for Pulp 2.
  • This is the minimum required for a 3.0.0 beta and GA.
  • All REST API calls will update the DB using transactions as necessary to ensure data integrity.
    * All REST API responses will be paginated.

Authentication

As an authenticated user I can manage user(s). [done]

  • Add a user
  • View user(s)
  • Update any user detail
  • Delete a user

As an API user, I can authenticate any API call with Basic auth [done]

As an authenticated user, I can filter users by: [3142]

  • username: (equality, username_in_list)

Repositories

As an authenticated user, I can list all repos.

  • All fields are included [done]
  • Pagination is supported [done]

As an authenticated user I can use filters on Repositories list: [3079]

  • name: (equality, name_in_list)

As an authenticated user, I can CRUD a repository

  • Create a repo [done]
  • Read a repo [done]
  • Update all mutable repo fields (asynchronous) [done]
  • Delete a repo (asynchronous)

As an authenticated user, when viewing a repository, I can discover a URL to the latest version of a repository. [done][3235]

Remotes

note: Remote attributes will commonly be available on remotes, but aren't guaranteed to be used by all remotes.

As an authenticated user, I can CRUD an remote

  • Create an remote
  • Read an remote
  • Update all mutable remote fields (asynchronous)
  • Delete an remote (asynchronous)

As an authenticated user, I have filters on the Remote list: [3080]

  • name: (equality, name_in_list)

As an authenticated user I can configure the following attributes on an Remote: [done]

  • validate (bool) [optional: defaults to True]: If true, the plugin will validate imported content.
  • ssl_ca_certificate (str) [optional] String containing a PEM encoded CA certificate used to validate the server certificate presented by the external source.
  • ssl_client_certificate (str) [optional] Contains a PEM encoded client certificate used for authentication.
  • ssl_client_key (str) [optional] Contains a PEM encoded private key used for authentication.
  • ssl_validation (bool) [optional: defaults to True]: If true, SSL peer validation must be performed.
  • proxy_url (str) [optional] Contains the proxy URL. Format: scheme://user:password@host:port.
  • username (str) [optional] Contains the username to be used for authentication when syncing.
  • password (str) [optional] Contains the password to be used for authentication when syncing.
  • name (str) [required] Contains the name.
  • last_updated (datetime) [read-only] Contains the datetime of the last remote update.
  • last_synced (datetime) [read-only] Contains the datetime of the last remote sync.

Publishers

note: Publisher attributes will commonly be available on publishers, but aren't guaranteed to be used by all publishers.

As an authenticated user, I can CRUD a publisher

  • Create a publisher
  • Read a publisher
  • Update all mutable publisher fields (asynchronous)
  • Delete a publisher (asynchronous)

As an authenticated user, I have filters on the Publisher list: [3081]

  • name: (equality, name_in_list)

As an authenticated user I can configure the following attributes on a Publisher:

  • relative_path (str) [optional] The (relative) path component of the published url.
  • name (str) [required] contains the name.
  • last_published (datetime) [read-only] When the last successful publis%{color:red}h occurred.
  • last_updated (datetime) [read-only] The datetime of the last publisher update.

Distributions

As an authenticated user, I can CRUD Distributions:

  • Create a Distribution.
  • Read a Distribution
  • List Distributions
  • Update all mutable Distribution fields (synchronous)
    - base path
  • Delete an Distribution (synchronous)

As a user, my distribution base paths don't conflict and my create/update is rejected identifying the conflicting distributions [2987]

As an authenticated user, I can create or update a distribution that is not associated with any publication (NULL)

As an authenticated user, I can create or update a distribution that is not associated with any publisher/repository (NULL)

  • publisher and repository are used together. If one is null, both must be null.

As a user, I can update a Distribution to distribute a specific Publication
As a user, I want a newly created publication to be automatically served by the content as defined by distributions.

As a user, I can see the full urls my base path is served at
As an authenticated user, I have filters on the Distribution list: [3082]%

  • name: (equality, name_in_list)
  • base_path: (equality, substring, base_path_in_list)

Publications

As an authenticated user, when creating a Publication, I can post a repo version href to be published. [3221]

As an authenticated user, I can publish a repository's latest version by posting a repository href to be published. [3223]

As an authenticated user, I can view which repository version was used to create a particular publication. [3237]

As an authenticated user, I can read Publication(s)

  • Read a Publication - id, created datetime, list of distribution hrefs, repo version
  • List all Publications - ordered by created datetime in descending order

As an authenticated user, I can delete publications.
- asynchronously with a lock on the repository version.
- prevented if associated with a distribution.
- single object only.

As an authenticated user, I can list publications and I have enough information to choose which ones to delete.

  • choose by created (older ones or perhaps latest)
  • choose not associated to a distribution.
  • does not imply filtering

As an authenticated user, I can list publications and i have enough information to select a publication to be associated with a distribution.

  • choose by created (latest or just by publish date)
    • manual promotion. "My rawhide publication has been tested and now I want to promote it to stable".
    • rollback to an earlier publication.
  • does not imply filtering

As an authenticated user, I can determine if and how a publication is distributed.

Exporters

As a plugin writer, I can contribute an exporter that is discovered by core
As a plugin writer, I have docs on how to create a discoverable exporter
As a plugin writer, I can contribute an exporter that uses a publication.

Artifacts

As an authenticated user, I can create an Artifact by uploading a file. [done]

As an authenticated user, I can optionally specify a size and/or digest to validate the uploaded file. [done]

Content Units

As an authenticated user, I can create a content unit.

As an authenticated user, I can read and list content units.
* As an authenticated user, I can filter content by repository version

As an authenticated user, I can delete a specific content unit if it is not in any repository version. (orphan)

  • Error message saying that the unit is in use by a repo version and a link to the filter to return all of the repo versions.
  • As a user, I know that files (Artifacts) associated with the Content unit are not removed by this call (docs)

Versioned Repositories

CRD

As an authenticated user, I can list versions for a particular repository. [done]

  • Pagination is supported

As an authenticated user, I can filter repository versions by: [3238]

  • number (equality, lt/lte, gt/gte)
  • content id (equality, content_in_list)
  • created datetime (range)

As an authenticated user, I can delete any repository version. [3219]

  • Latest versions are deleted simply.
  • Content set changes are "squashed" into the following version when non-latest versions are deleted.

As an authenticated user, I can view content that was added or removed in a particular repository version (as compared to the previous version). [done]

As an authenticated user, I can see the number of content unit types with counts for each [done][3059]

Repository Version Content

As a user, I know content sets for repository versions are immutable. [done]

As an authenticated user, I can list the content in a particular repository version [done]

  • All fields are included
  • Pagination is supported

As an authenticated user, I can create a new version by adding or removing content to the latest version. [3234]

  • I can follow the progress. (adding/removing are asynchronous).
  • can add and remove together in a single call

Orphan Content Units and Artifacts

As an authenticated user, I can cause an action that cleans up both orphaned content units and orphaned artifacts.

  • I cannot specify the units specifically (all types).
  • I can follow the progress of all cleanups. (Cleanups are asynchronous.)

Task Management

As an authenticated user, I can list all tasks

  • This does not include associated progress reports

As an authenticated user, I can see a detail view for a specific task [done]

  • all attributes of a task
  • all associated progress reports

As an authenticated user, I can cancel a task [done]

  • don't dare to use the DELETE verb!

As an authenticated user, I can delete tasks.

As an authenticated user, I can filter tasks by: [3144]

  • state (equality, state_in_list)
  • started_at(started_in_range)
  • finished_at(finished_in_range)
  • worker (equality)
  • parent (equality)

Status

As an unauthenticated user I can view the status of Pulp workers and resource managers. [done]

As an unauthenticated user I can view the status of the web server's connection to the database and message broker. [done]

As an unauthenticated user I can view the versions of core and each installed plugin.

Workers

As an authenticated user, I can filter workers by: [3143]

  • last_heartbeat (range)
  • name (substring)
  • gracefully_stopped (equality)

Plugin User Content Management stories

Simple Copy

As an authenticated user of a plugin, I can search (synchronous call) a repository version's content using filtering.

  • note: In a separate call (async), I can add_content_units or remove_content_units to another repository's latest version.

Complex Copy

As a plugin writer I can provide a rich search features with arbitrary viewsets. e.g. depsolving, versioning, etc

Examples of specific plugin use cases motivating the above general viewset

  • As an authenticated user, I can add an Errata from one repository to another repository along with packages mentioned in the Errata and all their dependencies that are present in the source repository.
  • As an authenticated user, I can add bunch of dependencies and store n versions back for each RPM.
  • As an authenticated user, I can use depsolving and versioning copy features together
  • As an authenticated user, I can copy a puppet module and all of it's dependencies from one repository to another.
  • As an authenticated user, I can depsolve units to be added to a destination repo based on an errata

Plugin API

As a plugin writer, I have a plugin API that is semantically versioned at 0.x separate from the REST API [done]

As a plugin writer, my app will be discovered by Pulp's app via an entry point provided by the plugin writer [done]

Task

As a plugin writer, I can report progress with a message and state [done]

As a plugin writer, I can report progress with an optional suffix [done]

As a plugin writer, I can report progress with a total count of things to do an the current count of things done [done]

As a plugin writer, non-fatal exceptions on the Task and are included in the Task detail. non_fatal exceptions do not cause the Task to be marked as failed, but may be interpreted by the user as not fully successful. [done]

As a plugin writer, the working directory is set before Task work is done and cleaned up afterwards. I should not need to interact with the file system outside of the working dir. [done]

Remote

As a plugin writer, I can provide a subclassed Remote. I can add custom fields to the subclassed Remote.

As a plugin writer, I can provide a UserFacingTask to perform a sync operation using information stored in a Remote.

As a plugin writer, I can provide a ViewSet for the subclassed Remote that has a 'sync' endpoint that dispatches the sync task with a reservation for Repository and Remote.

Publisher

As a plugin writer, I can provide a subclassed Publisher. I can add custom fields to the subclassed Publisher.

As a plugin writer, I can provide a UserFacingTask to perform a publish operation using information from the subclassed Publisher.

As a plugin writer, I can provide a ViewSet for the subclassed Publisher that has a 'publish' endpoint that dispatches the publish task with a reservation for Repository and Publisher.

Content

As a plugin writer, I can provide a subclassed Content unit. I can add custom fields to the subclassed Content. [done]

As a plugin writer, I have documentation that shows how I can add filters to filter content responsibly.

As a plugin writer, I have documentation on how to write a filter for my Content that can use the RepositoryVersion manager.
* note: This will allow users to filter content by repository version

Content Management

As a plugin writer, I can interact with and create Artifacts [done]

As a plugin writer, I can query content units/artifacts associated with a repository. [done]

As a plugin writer, I can create a new repository version: [done]

  • When I create the new version, it is incomplete.
  • The new version starts with all the content from the previous version.
  • I can add and remove content units to and from an incomplete repository version.
  • When I am finished with the incomplete version, core finalizes the repository version for me.
  • I have exclusive access to the new repository version while my code is running.

Publishing

As a plugin writer, I can create a new publication: [done]

  • When I create the new publication, it is incomplete.
  • When I am finished with the incomplete publication, core finalizes the publication for me.
  • I have exclusive access to the repository version while my code is running.
  • I have exclusive access to the new publication while my code is running.

"live APIs"

As a plugin writer, I can register views and viewsets to arbitrary endpoints. [3360]
As a plugin writer, I have documentation on what URLs I should not use for my views and viewsets [3473]

Here are some concrete use cases driving the very Live API use cases above:

# Concrete user use cases:
    As an authenticated user, I can use the puppet client to fetch content from Pulp using the Forge API
    As an authenticated user I can use the docker client to fetch content from Pulp using the Docker v1 API
    As an authenticated user I can use the docker client to fetch content from Pulp using the Docker v2 API

# Concrete plugin writer use cases
    As a puppet plugin developer, I can provide a viewset which handles the server side of the puppet Forge v3 API
    As a docker plugin developer, I can provide a viewset which handles the server side of the docker v1 API
    As a docker plugin developer, I can provide a viewset which handles the server side of the docker v2 API

h4. Storage

As a plugin writer, the plugin API provides an API that returns a fully qualified path to a shared and namespaced storage location used to store content. ["3182"https://pulp.plan.io/issues/3182\]
******

Webserver Deployment

As a system administrator, I can deploy all Pulp web applications on one process
As a system administrator, I can deploy the Pulp REST API exclusively in one process
As a system administrator, I can deploy the Pulp content serving view exclusively in one process

As a system administrator, I can deploy all Pulp web applications inside a virtualenv.
As a system administrator, I can deploy all Pulp web applications without root permissions.

CLI

We will use coreapi-cli to generate a one to one mapping of cli commands to rest api schema #3068
We will have a wrapper for coreapi-cli. This wrapper will handle parallel progress reporting

Download API

As a plugin writer, I can download files via

  • http://
  • https://
  • file://

As a plugin writer, I can configure a downloader with:

  • Basic Auth
  • SSL Cert Client Auth
  • Custom CAs will be configured via a "trust store" either on the system or similar. Pulp will not do anything to read/load/manage CAs directly.

As a plugin writer, I can provide arbitrary behaviors for customized downloaders

  • For example token authentication in the docker plugin

As a plugin writer, I can have connection pooling/reuse

As a plugin writer, I have proxy settings

  • proxy url (containing basic auth info)

As a plugin writer, I can have great logs

As an authenticated user, I have documentation about how to use something for bandwidth limiting

As a plugin writer, I can configure the validation mechanisms used at download time

  • checksum validation - minimum (md5, sha1, sha256, sha512)
  • size validation

As a plugin writer I can manage the catalog by using ChangeSets

As a plugin writer, the plugin can participate in adding content for cases where the decision to add additional content is based on the content that has been downloaded.

As a plugin writer, I can fetch content myself (but I am not encouraged to do so) with code I write

As a plugin writer, I can CRUD content units need a convention to handle multiple content units - see https://pulp.plan.io/issues/3472

Migrations only involving Pulp 3

Users can run "pulp-manager migrate" to migrate the database and adjust state in other locations (filesystem, message broker, ...). [done]

Web Server Integration

As a user, I can have content efficiently served to me by Apache by Pulp using the X-SEND response headers. [done]
As a user, I can have content efficiently served to me by Nginx by Pulp using the X-Accel-Redirect response headers. [done]

Glossary

Add (Content Unit): An operation causing a new repository version to contain a content unit(s)

Applicability - A plugin defined term meaning when a package update available in a repository is applicable to a given consumer as determined by the Consumer Profile.

Artifact - A file associated with one content (unit). Artifacts are not shared between content (units). Create a content unit using an uploaded file ID as the source for its metadata. Create Artifacts associated with the content unit using an uploaded file ID for each; commit as a single transaction.

Content (unit) - A single piece of content manged by Pulp. Each file associated with a content (unit) is called an Artifact. Each content (unit) may have zero or many Artifacts.

Distribution: Where and how the content app serves a Publication. i.e. http vs https and base path component of the URL. A Distribution defines:

  • the base path of the repository publication (required)
  • serving via http (default=False)
  • serving via https (default=True)
  • relationship w/ a Publisher for auto-distribution (should be allowed to be NULL)
  • relationship with Publication (should be allowed to be NULL)

Exporter: An Exporter exports a Publication out of Pulp. e.g. rsync exporter exports content to a remote server

Live API: a viewset endpoint contributed by plugin. For examples see the associated MVP section

Orphan Artifact: An Artifact that is associated with 0 Content Units and 0 Publications

Orphan Content (unit): A content unit that is a member of 0 repository versions

Remove (content unit): An operation causing a new repository version to not contain a content unit(s)

Repository - A named collection of repository versions.

Repository Version - An immutable set of content which is versioned by a sequential number.

Updated by amacdona@redhat.com over 6 years ago ยท 167 revisions