Project

Profile

Help

Story #3173

closed

As a plugin author, I have documentation on how to implement a sync operation for my importer

Added by daviddavis about 7 years ago. Updated over 5 years ago.

Status:
CLOSED - DUPLICATE
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
Groomed:
No
Sprint Candidate:
Yes
Tags:
Documentation
Sprint:
Sprint 31
Quarter:

Description

The syncing with an importer portion of the plugin writer's guide needs to be updated. The guide needs to explain that it's the author's responsibility to

- define a UserFacingTask that creates a new RepositoryVersion of a repository by syncing with a specific importer.

- the task should create a repository version and add it to `task.created_resources` in a single transaction
- the task should add / remove content using the Importer to drive the download API
- the task should set the repository version's to complete field to true after all content manipulation is done

- define a detail route on the Importer ViewSet for 'sync' that accepts a 'repository' POST body parameter

- the route should perform validation and then dispatch the the sync task with reservations for the repository and the importer.


Related issues

Has duplicate Pulp - Issue #3228: As an authenticated user, I can create a new repository versionCLOSED - DUPLICATEActions
Is duplicate of Pulp - Task #3220: Expand plugin writer's guide to include RepositoryVersion creating actions (including sync)CLOSED - CURRENTRELEASEbizhang

Actions
Actions #1

Updated by dkliban@redhat.com about 7 years ago

  • Parent issue set to #3209
Actions #2

Updated by dkliban@redhat.com about 7 years ago

  • Description updated (diff)
Actions #3

Updated by dkliban@redhat.com about 7 years ago

  • Has duplicate Issue #3228: As an authenticated user, I can create a new repository version added
Actions #4

Updated by daviddavis about 7 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to daviddavis
Actions #5

Updated by daviddavis about 7 years ago

A couple items:

  • add_many and remove_many seem vague. I was thinking about something like add_content_units and remove_content_units.
  • Is the order of the actions (add, remove, then sync) correct? I thought we discussed sync, add, remove in the past.
Actions #6

Updated by daviddavis about 7 years ago

  • Blocks Story #3234: As an authenticated user, I can create a new version by adding or removing content to the latest version. added
Actions #7

Updated by daviddavis about 7 years ago

I talked a bit with @dkliban and it seems like the order should be sync, add, then remove. For instance, suppose I have a version that was synced that contains units A and B. If I want to sync again to get unit C but also remove unit B, then I can't actually remove B (because sync will just re-add it). I thinks users would be surprised if they called sync and remove B, only to find that B was still there because it was removed before sync and then sync re-added it.

Actions #8

Updated by bmbouter almost 7 years ago

wrote:

I talked a bit with @dkliban and it seems like the order should be sync, add, then remove.

This makes sense to me. +1 to doing it in this way.

Actions #9

Updated by dkliban@redhat.com almost 7 years ago

  • Description updated (diff)
Actions #10

Updated by daviddavis almost 7 years ago

  • Subject changed from As an authenticated user, I can create a new repository version. to As an authenticated user, I can create a new repository version by POSTing a importer href.
  • Description updated (diff)
  • Status changed from ASSIGNED to NEW
  • Assignee deleted (daviddavis)
Actions #11

Updated by dkliban@redhat.com almost 7 years ago

  • Groomed changed from No to Yes
  • Sprint Candidate changed from No to Yes
Actions #12

Updated by dkliban@redhat.com almost 7 years ago

  • Blocks Task #3220: Expand plugin writer's guide to include RepositoryVersion creating actions (including sync) added
Actions #13

Updated by amacdona@redhat.com almost 7 years ago

  • Blocks deleted (Story #3234: As an authenticated user, I can create a new version by adding or removing content to the latest version.)
Actions #14

Updated by dkliban@redhat.com almost 7 years ago

  • Description updated (diff)
  • Tags Documentation added
Actions #15

Updated by dkliban@redhat.com almost 7 years ago

  • Subject changed from As an authenticated user, I can create a new repository version by POSTing a importer href. to As a plugin author, I have documentation on how to implement a sync operation for my importer
Actions #16

Updated by ttereshc almost 7 years ago

I would move - define a detail route on the Importer ViewSet for 'sync' that accepts a 'repository' POST body parameter one level up or it seems like it's a part of requirements for the sync task.

Resource reservation should probably also be on a repo version, what do you think?

s/task's created list/task.created_resources

Actions #17

Updated by dkliban@redhat.com almost 7 years ago

If we are going to lock on the repository version being created, then the repository version needs to be created in the viewset. A lock on the repository should prevent a repository version from being deleted from under it. Please correct me if my understanding of the locking is wrong.

Actions #18

Updated by bmbouter almost 7 years ago

To align w/ the locking that core would do as part of #3186. We need to tell them to lock on:

  • The repo version
  • The repository itself
  • The importer being used

The lock on the repository version is needed in addition to serialize any concurrent publishes that may also try to publish on that version. If that case is prevented due to the a publication not being allowed while complete=False then the lock would not be required.

Actions #19

Updated by dkliban@redhat.com almost 7 years ago

We should definitely prevent publishing a repository version with complete=False. We would also need to lock on repository when deleting a repository version. Based on info in https://pulp.plan.io/issues/3219 we are planning to do that. With this in mind, I don't think we need to lock on a repository version during a sync.

Actions #20

Updated by dkliban@redhat.com almost 7 years ago

  • Description updated (diff)
Actions #21

Updated by amacdona@redhat.com almost 7 years ago

We do still need to lock on repo version. Otherwise, the version can be deleted, or the version before it could be deleted, which would modify the version in question.

Actions #22

Updated by ttereshc almost 7 years ago

So how publish task will behave if it sees incomplete repo version?
Without lock on a repo version the publish task will run and fail? Or it won't be possible to create a publish task for incomplete repo version?

If sync task will lock on repo version then a publish task can be created and wait in a queue.

Actions #23

Updated by dkliban@redhat.com almost 7 years ago

  • Sprint/Milestone set to 53
Actions #24

Updated by dkliban@redhat.com almost 7 years ago

  • Blocks deleted (Task #3220: Expand plugin writer's guide to include RepositoryVersion creating actions (including sync))
Actions #25

Updated by amacdona@redhat.com almost 7 years ago

  • Is duplicate of Task #3220: Expand plugin writer's guide to include RepositoryVersion creating actions (including sync) added
Actions #26

Updated by amacdona@redhat.com almost 7 years ago

  • Groomed changed from Yes to No
Actions #27

Updated by amacdona@redhat.com almost 7 years ago

  • Status changed from NEW to CLOSED - DUPLICATE
Actions #28

Updated by amacdona@redhat.com almost 7 years ago

  • % Done changed from 0 to 100
  • Parent issue deleted (#3209)
Actions #29

Updated by bmbouter almost 7 years ago

  • Sprint set to Sprint 31
Actions #30

Updated by bmbouter almost 7 years ago

  • Sprint/Milestone deleted (53)
Actions #31

Updated by dkliban@redhat.com over 6 years ago

  • Sprint/Milestone set to 3.0.0
Actions #32

Updated by bmbouter over 5 years ago

  • Tags deleted (Pulp 3, Pulp 3 MVP)

Also available in: Atom PDF