Task #2868
Updated by jortel@redhat.com almost 7 years ago
h1. Overview The platform needs to support the composition and inventory of publications. Each _publication_ is a representation of a repository's content that can be consumed by a specific technology. An example of a technology is YUM. A publication is created by a _Publisher_ associated with the repository and is provided by a plugin. A repository may have multiple publishers. A publication is composed of two types of files. The first is a content Artifact. The second, is _metadata_ created by the publisher when the publication is created. Once created, a publication may be distributed for consumption in many ways. The platform will support common online distributing such as HTTP and HTTPS and offline distributing such as creating an ISO. Additional Goals * Clean separation between publishing and how a publication is distributed. * Eliminate the use of symlinks as the primary method of publishing. * Eliminate need for each plugin to provide an Apache .conf file for distributing via http/https. * Prevent orphaned content from being deleted while published. h1. Design h3. Tables <pre> [Publisher]---*[publication]---*[distribution] [Exporter] | * [Distributor] </pre> The _Publication_ table contains publications and is associated to the Publisher that created it. A Distribution defines a method for making publications available for consumption. Attributes currently modeled on _Publisher_ pertaining to distribution such as Auth would be removed. Auth would be handled by Apache and selected by user based Distribution.base_path. *Publication* * [pk] *id* - The primary key. * [fk] *publisher_id* - The publisher that created the publication. Constraint ensure it's deleted when the publication is deleted. * *created* - When the publication was created. The _PublishedArtifact_ table contains linkage to content Artifacts. *PublishedArtifact* * [pk] *id* - The primary key. * [fk] *publication_id* - A publication. Constraint ensure it's deleted when the publication is deleted. * [fk] *artifact_id* - An associated content artifact. * *relative_path* - The relative path component within the URL that is also relative to the Publication.base_path. The _PublishedMetadata_ table contains linkage to generated metadata files. *PublishedMetadata* * [pk] *id* - The primary key. * [fk] *publication_id* - A publication. Constraint ensure it's deleted when the publication is deleted. * *file* - An absolute path to a metadata file. Stored in: /var/lib/pulp/published/metadata/<id>/<name> * *relative_path* - The relative path component within the URL that is also relative to the Distribution.base_path. h1. Distribution A _Distribution_ maps a Publication of a distribution mechanism. It defines the URLs (paths) under which a publication is distributed, by which protocols and likely how access is authorized. Distributions also define which publications are _live_ (visible for consumption) for a publisher (and thus repository). *Distribution* * [pk] *id* - The primary key. * [fk] *repository_id* - An (optional) repository. When set, the publication is updated whenever a new publication is created by a publisher (for the specified repository). * [fk] *publisher_id* - An (optional) A publisher. When set, the publication is updated whenever a new publication is created by the publisher (for the specified repository). * [fk] *publication_id* - An (optional) publication (mutable). * *name* - The distribution name (Eg: rawhide, stable). * *base_path* - The base path for the publication. This is the root of the path component of URLs. * *auto_updated* - The publication_id is auto updated when a new publication is created. The value of (false) is designed to support promotion work flows. * *http* - HTTP is enabled (bool). The distribution is served by pulp using HTTP. * *https* - HTTPS is enabled (bool). The distribution is served by pulp using HTTPS. An _Exporter_ A _Distributor_ defines method of generating a static (external) file tree for a Distribution. An Exporter Some builtin and others contributed by plugins. Distributors are not needed for the HTTP/HTTPS distribution mechanism provided by the pulp platform. They are +only+ needed to generate an external representation of a Distribution. For example: rsync to static CDN or crane support. This is likely a master-detail pattern and only showing the master table here. *Exporter* *Distributor* * [pk] *id* - The primary key. * *name* [fk] *publisher_id* - The exporter name. A related (owner) publisher. * *last_export* *last_run* - The timestamp of last date/time the last successful export. distributor ran. * *autorun* - When (true), the distributor is _run_ automatically when a new publication is created. When (false), it must be _run_ explicitly. h1. Sample Data <pre> Publisher ------------------------------- publisher-1, ... </pre> <pre> Publication ------------------------------- publication-1, publisher-1, ... publication-2, publisher-1, ... </pre> <pre> PublishedMetadata ------------------------------- <id>, publication-1, /var/lib/pulp/published/../repodata/repomd.xml <id>, publication-1, /var/lib/pulp/published/../repodata/primary.xml </pre> <pre> PublishedArtifact ------------------------------- <id>, publication-1, artifact-1, packages/dog.rpm <id>, publication-1, artifact-2, packages/cat.rpm </pre> <pre> Distribution ------------------------------- <id>, publisher-1, publication-1, rawhide, f25/rawhide/x86_64, true <id>, publisher-1, publication-2, stable, f25/stable/x86_64, false </pre> h1. General Flows h2. Create A Repository <pre> 1. Create a repository. 2. Create a publisher associated with the repository. 3. Create desired distributions associated with the publisher. Each distribution will be configured with a base_path and http and/or https enabled as desired. </pre> h2. Publishing: _"The publisher will compose a publication"_ <pre> 1. Publisher creates a publication using the plugin API. 2. Publisher adds content artifacts to the publication. 3. Publisher generates some metadata files in the working dir. 4. Publisher adds the metadata files to the publication using the plugin API. 5. Publisher commits (publishes) the publication. The plugin API ensures this is atomic. 6. Distributions with auto_updated=true are updated with new publication_id. 7. Any distributors with autorun=true will be run. </pre> h2. Client makes a GET request for content (or metadata): <pre> 1. Request is routed to the content (WSGI) application (just like in pulp2 for RPM). 2. Query to get the Distribution. 3. Match the scheme against (http and https fields). 404 when scheme not enabled. 4. Query the PublishedArtifact and PublishedMetadata tables (in that order) by URL path component to get the artifact or the metadata. 5. forward the artifact storage path (or metadata path) to: <not stored locally> streamer <stored locally> x-send (or stream using django in dev environments) 6. Done. </pre> h1. Apache Configuration The platform will provide an /etc/httpd/conf.d/pulp.conf that configures support for HTTP and HTTPS. Published content would be consumed using URLs with a base of: <pre> /pulp/published/<path> </pre> where _path_ is the <Distribution.base_path>/(<PublishedArtifact.relative_path>|<PublishedMetadata.relative_path>) h1. Exporting An exporter is executed using the plugin defined endpoint. Example: /api/exporter/<type>/export/. The platform would provide a _resource_ for exporting a _Publication_ in different formats such as ISO and static file trees.
.