Project

Profile

Help

Story #3173

As a plugin author, I have documentation on how to implement a sync operation for my importer

Added by daviddavis almost 2 years ago. Updated 7 months ago.

Status:
CLOSED - DUPLICATE
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Platform Release:
Blocks Release:
Backwards Incompatible:
No
Groomed:
No
Sprint Candidate:
Yes
Tags:
Documentation
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:
Sprint 31

Description

The syncing with an importer portion of the plugin writer's guide needs to be updated. The guide needs to explain that it's the author's responsibility to

- define a UserFacingTask that creates a new RepositoryVersion of a repository by syncing with a specific importer.

- the task should create a repository version and add it to `task.created_resources` in a single transaction
- the task should add / remove content using the Importer to drive the download API
- the task should set the repository version's to complete field to true after all content manipulation is done

- define a detail route on the Importer ViewSet for 'sync' that accepts a 'repository' POST body parameter

- the route should perform validation and then dispatch the the sync task with reservations for the repository and the importer.


Related issues

Duplicated by Pulp - Issue #3228: As an authenticated user, I can create a new repository version CLOSED - DUPLICATE Actions
Duplicates Pulp - Task #3220: Expand plugin writer's guide to include RepositoryVersion creating actions (including sync) MODIFIED Actions

History

#1 Updated by dkliban@redhat.com almost 2 years ago

  • Parent task set to #3209

#2 Updated by dkliban@redhat.com almost 2 years ago

  • Description updated (diff)

#3 Updated by dkliban@redhat.com almost 2 years ago

  • Duplicated by Issue #3228: As an authenticated user, I can create a new repository version added

#4 Updated by daviddavis almost 2 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to daviddavis

#5 Updated by daviddavis almost 2 years ago

A couple items:

  • add_many and remove_many seem vague. I was thinking about something like add_content_units and remove_content_units.
  • Is the order of the actions (add, remove, then sync) correct? I thought we discussed sync, add, remove in the past.

#6 Updated by daviddavis almost 2 years ago

  • Blocks Story #3234: As an authenticated user, I can create a new version by adding or removing content to the latest version. added

#7 Updated by daviddavis almost 2 years ago

I talked a bit with @dkliban and it seems like the order should be sync, add, then remove. For instance, suppose I have a version that was synced that contains units A and B. If I want to sync again to get unit C but also remove unit B, then I can't actually remove B (because sync will just re-add it). I thinks users would be surprised if they called sync and remove B, only to find that B was still there because it was removed before sync and then sync re-added it.

#8 Updated by bmbouter almost 2 years ago

wrote:

I talked a bit with @dkliban and it seems like the order should be sync, add, then remove.

This makes sense to me. +1 to doing it in this way.

#9 Updated by dkliban@redhat.com almost 2 years ago

  • Description updated (diff)

#10 Updated by daviddavis almost 2 years ago

  • Subject changed from As an authenticated user, I can create a new repository version. to As an authenticated user, I can create a new repository version by POSTing a importer href.
  • Description updated (diff)
  • Status changed from ASSIGNED to NEW
  • Assignee deleted (daviddavis)

#11 Updated by dkliban@redhat.com almost 2 years ago

  • Groomed changed from No to Yes
  • Sprint Candidate changed from No to Yes

#12 Updated by dkliban@redhat.com almost 2 years ago

  • Blocks Task #3220: Expand plugin writer's guide to include RepositoryVersion creating actions (including sync) added

#13 Updated by amacdona@redhat.com almost 2 years ago

  • Blocks deleted (Story #3234: As an authenticated user, I can create a new version by adding or removing content to the latest version.)

#14 Updated by dkliban@redhat.com almost 2 years ago

  • Description updated (diff)
  • Tags Documentation added

#15 Updated by dkliban@redhat.com almost 2 years ago

  • Subject changed from As an authenticated user, I can create a new repository version by POSTing a importer href. to As a plugin author, I have documentation on how to implement a sync operation for my importer

#16 Updated by ttereshc almost 2 years ago

I would move - define a detail route on the Importer ViewSet for 'sync' that accepts a 'repository' POST body parameter one level up or it seems like it's a part of requirements for the sync task.

Resource reservation should probably also be on a repo version, what do you think?

s/task's created list/task.created_resources

#17 Updated by dkliban@redhat.com almost 2 years ago

If we are going to lock on the repository version being created, then the repository version needs to be created in the viewset. A lock on the repository should prevent a repository version from being deleted from under it. Please correct me if my understanding of the locking is wrong.

#18 Updated by bmbouter almost 2 years ago

To align w/ the locking that core would do as part of #3186. We need to tell them to lock on:
  • The repo version
  • The repository itself
  • The importer being used

The lock on the repository version is needed in addition to serialize any concurrent publishes that may also try to publish on that version. If that case is prevented due to the a publication not being allowed while complete=False then the lock would not be required.

#19 Updated by dkliban@redhat.com almost 2 years ago

We should definitely prevent publishing a repository version with complete=False. We would also need to lock on repository when deleting a repository version. Based on info in https://pulp.plan.io/issues/3219 we are planning to do that. With this in mind, I don't think we need to lock on a repository version during a sync.

#20 Updated by dkliban@redhat.com almost 2 years ago

  • Description updated (diff)

#21 Updated by amacdona@redhat.com almost 2 years ago

We do still need to lock on repo version. Otherwise, the version can be deleted, or the version before it could be deleted, which would modify the version in question.

#22 Updated by ttereshc almost 2 years ago

So how publish task will behave if it sees incomplete repo version?
Without lock on a repo version the publish task will run and fail? Or it won't be possible to create a publish task for incomplete repo version?

If sync task will lock on repo version then a publish task can be created and wait in a queue.

#23 Updated by dkliban@redhat.com almost 2 years ago

  • Sprint/Milestone set to 53

#24 Updated by dkliban@redhat.com almost 2 years ago

  • Blocks deleted (Task #3220: Expand plugin writer's guide to include RepositoryVersion creating actions (including sync))

#25 Updated by amacdona@redhat.com almost 2 years ago

  • Duplicates Task #3220: Expand plugin writer's guide to include RepositoryVersion creating actions (including sync) added

#26 Updated by amacdona@redhat.com almost 2 years ago

  • Groomed changed from Yes to No

#27 Updated by amacdona@redhat.com almost 2 years ago

  • Status changed from NEW to CLOSED - DUPLICATE

#28 Updated by amacdona@redhat.com almost 2 years ago

  • % Done changed from 0 to 100
  • Parent task deleted (#3209)

#29 Updated by bmbouter over 1 year ago

  • Sprint set to Sprint 31

#30 Updated by bmbouter over 1 year ago

  • Sprint/Milestone deleted (53)

#31 Updated by dkliban@redhat.com over 1 year ago

  • Sprint/Milestone set to 3.0

#32 Updated by bmbouter 7 months ago

  • Tags deleted (Pulp 3, Pulp 3 MVP)

Please register to edit this issue

Also available in: Atom PDF