Project

Profile

Help

Task #3033

Task #2868: Platform support for publishing.

Add Publication API Endpoint.

Added by jortel@redhat.com about 2 years ago. Updated 6 months ago.

Status:
MODIFIED
Priority:
Normal
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Platform Release:
Blocks Release:
Backwards Incompatible:
No
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:
Sprint 29

Description

Add Publication API endpoint to support listing and deleting publications. The URL should be nested under the publisher. Publications are immutable but can be deleted when not referenced by a Distribution. This will be enforced by the integrity constraints.

Use Cases:
  • As a user, I want to create a publication.
  • The POST to publisher/<id>/publications/ will trigger a publish which will create a publication and return 202 and a task href.
  • On completion, the Task.created_resources will contain the created publication (unless the publisher fails).
  • As a user, I want to list publications created by a publisher to support:
  • Knowing what publications have been created.
  • Manually update a Distribution. (manual promotion).
  • Identify publications to be deleted.
  • As a user, I want to delete publications:
  • A Specific publication. URL something like: "../publishers/<id>/publications/<id>/"

The Distribution FK to Publication is currently on_delete=SET_NULL but thinking we should probably change it to on_delete=PROTECT to ensure that an active publication isn't ruined by mistake.

In the future, one-time publisher options will be stored on the Publication.

A new CreatedResources model:
  • task
  • resource (Generic FK)

The Task serializer will include the Task.created_resources

The publish task will create the Publication as defined by the POST document and passed to the plugin Publisher.publish(). On success, the created publication is added to CreatedResources. This should all be done in a single DB transaction to ensure that broken publications are not saved to the DB.


Checklist


Related issues

Blocked by Pulp - Task #3102: Make Distribution a top level resource in the API. MODIFIED Actions

Associated revisions

History

#1 Updated by dkliban@redhat.com about 2 years ago

Will this support performing a GET on a particular publication? Could you explain what you mean by "Inspect what got published" ?

#2 Updated by jortel@redhat.com about 2 years ago

  • Description updated (diff)

wrote:

Will this support performing a GET on a particular publication? Could you explain what you mean by "Inspect what got published" ?

It would be fine to support GET on a specific publication but doubt it would be necessary beyond ensuring that listed publications (resources) could be retrieved. Seems like good REST behavior.

What I had in mind for "Inspect what got published" was mainly to know about publications that a publisher has created. Eventually, I think it would be useful to support drilling down into published metadata and artifacts but don't have any use cases for that now.
I reworded in the description.

#3 Updated by dkliban@redhat.com about 2 years ago

What API endpoint will be used to DELETE "All publications not referenced by a Distribution."?

I think the scope of this work should be limited to just being able to retrieve a particular publication, list all publications for a publisher, delete a particular publication.

#4 Updated by jortel@redhat.com about 2 years ago

wrote:

What API endpoint will be used to DELETE "All publications not referenced by a Distribution."?

Publications will need regular purging that thought that something like /publisher/xx/publication?distribution_id=NULL would be useful.

I think the scope of this work should be limited to just being able to retrieve a particular publication, list all publications for a publisher, delete a particular publication.

This would certainly be the minimum to support the use case but seems unnecessarily unhelpful.

This should be consistent with the approach taken for orphaned content and artifact purging. Do you know how deleting those orphans is currently supported?

#5 Updated by dkliban@redhat.com about 2 years ago

I think the scope of this work should be limited to just being able to retrieve a particular publication, list all publications for a publisher, delete a particular publication.

This would certainly be the minimum to support the use case but seems unnecessarily unhelpful.

This should be consistent with the approach taken for orphaned content and artifact purging. Do you know how deleting those orphans is currently supported?

I don't think we currently support purging orphaned content or artifacts. That's one of the reasons I am hesitant to include this purge functionality as part of this story.

#6 Updated by mhrivnak about 2 years ago

A user will need to be able to see what publications exist for a publisher, uniquely identify each by URL (for reference when creating or updating a Distribution for example), and delete unwanted publications.

I like the idea of setting on_delete=PROTECT for the distributor's FK to a publication. That would make it simple for a client to delete publications with simplicity, not having to worry about a race condition with updating a distributor.

If we enable individual deletion of publications in that way, I think bulk deletion could reasonably start as a client-side activity.

Server-side bulk deletion could be done multiple ways, and I think that's worth addressing independently. A key question I think of is: how many publications can reasonably be deleted while waiting for an HTTP response? Should we decide that if a single action is deleting more than 1 publication, that should be a task? That decision aside, we could have a task that does bulk deletion either for a single publisher or at some wider scope. Or we could have a publisher setting for how many old publications to keep, and let it just delete old ones as it makes new ones. And there are probably other options worth considering.

A publication representation should include the created timestamp, a reference to the publisher, and references to any distributions that reference it. Eventually we'll want to add the ability to see what content is published, presumably via a reference to a repo version. Taking that reference from two publications should also allow use of a comparison feature to see what changed between the two versions, and thus what content is different in the two publications. A repo version reference will also enable incremental publishes.

We'll also likely want to expose the artifacts and metadata at some point, but I can't think of a specific use case that's worth prioritizing right now.

What will the "natural key" be for a publication? Or in other words, what will come in the path after "<publisher>/publications/" ?

#7 Updated by jortel@redhat.com about 2 years ago

I'm fine with this task being only about listing and deleting single (specific) publications so long was we all agree that we need to consider supporting DEL at the REST collection level for orphans at some point:
  • content
  • artifact
  • publication

Multiple deletes as a single DB operation will be fast and free of race conditions. I don't see how this should require tasking anything extraordinary in ViewSets.

As for the publication natural key .. there isn't one currently and can't think of anything that makes sense? You? It may need to be .../publishers/<id>/publications/<id>/

#8 Updated by mhrivnak about 2 years ago

Seems reasonable. You may be right that bulk deletes of publications, particularly if limited to a single publisher, could run as a single fast operation if that's all it does. So maybe that would not need to be in a task.

Artifacts could be more interesting if files on disk have to be deleted as well.

#9 Updated by jortel@redhat.com about 2 years ago

Publications can have associated metadata files on disk so should probably implement the same as artifacts.

#10 Updated by jortel@redhat.com about 2 years ago

  • Description updated (diff)

#11 Updated by dkliban@redhat.com about 2 years ago

  • Groomed changed from No to Yes

#12 Updated by amacdona@redhat.com about 2 years ago

  • Sprint Candidate changed from No to Yes

#13 Updated by jortel@redhat.com about 2 years ago

  • Sprint/Milestone set to 46

#14 Updated by jortel@redhat.com about 2 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to jortel@redhat.com

#15 Updated by jortel@redhat.com almost 2 years ago

  • Blocked by Task #3102: Make Distribution a top level resource in the API. added

#16 Updated by jortel@redhat.com almost 2 years ago

  • Description updated (diff)

#17 Updated by jortel@redhat.com almost 2 years ago

  • Description updated (diff)

#18 Updated by jortel@redhat.com almost 2 years ago

  • Description updated (diff)

#19 Updated by jortel@redhat.com almost 2 years ago

  • Description updated (diff)

#20 Updated by jortel@redhat.com almost 2 years ago

  • Description updated (diff)

#21 Updated by jortel@redhat.com almost 2 years ago

  • Description updated (diff)

#22 Updated by jortel@redhat.com almost 2 years ago

I updated the description to reflect the following decisions made on @pulp-dev:
  • A POST to /publications/ will create a publishing task which replaces publisher/<id>/publish/
  • In the future, one-time publisher options will be passed in the POST document and stored on the Publication as something generic like key-value pair.
  • The created publication will be stored in a new Task.created_resources.
  • Only successful publishing tasks will result in a Publication being stored in the DB and listed in Task.created_resources.

#23 Updated by mhrivnak almost 2 years ago

  • Sprint/Milestone changed from 46 to 47

#24 Updated by rchan almost 2 years ago

  • Sprint/Milestone changed from 47 to 48

#25 Updated by daviddavis almost 2 years ago

  • Tags Pulp 3 MVP added

#26 Updated by jortel@redhat.com almost 2 years ago

  • Status changed from ASSIGNED to MODIFIED
  • % Done changed from 0 to 100

#27 Updated by bmbouter over 1 year ago

  • Sprint set to Sprint 29

#28 Updated by bmbouter over 1 year ago

  • Sprint/Milestone deleted (48)

#29 Updated by dkliban@redhat.com over 1 year ago

  • Sprint/Milestone set to 3.0

#30 Updated by bmbouter 6 months ago

  • Tags deleted (Pulp 3, Pulp 3 MVP)

Please register to edit this issue

Also available in: Atom PDF