Project

Profile

Help

Story #7015

As a user, I can associate a remote with a repository

Added by wibbit 3 months ago. Updated about 2 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Sprint:
Quarter:

Description

Currently there is no way to link a repository with a remote, and as such, each time you wish to sync a repository against it's upstream link, you have to keep a record (or query for) the appropriate remote for that repo.

This can be done with careful consideration around naming conventions, however it would be nice if an optional link between a Repository and Remote was possible.

As such, once the Repository and Remote have been created an paired, a simply repository sync would be able to work out which remote to use, and sync.

Design

Model

Add a many-to-many foreign key between Repositories and Remotes. This association should be optional. For now Repositories can only have 0 or 1 remotes. Remotes can belong to any number of repositories.

Note: In the future we want to be able to support multiple remotes for repositories that can be synced at once. See https://pulp.plan.io/issues/7031. This is the purpose of having a many-to-many instead of one-to-many.

REMOTE_TYPES

Add an iterable to Repository similar to CONTENT_TYPES. This iterable will check that when a remote is added to a repo, it is of one of these types.

API

Repositories

Expose a remote hrefs field for repository endpoints. Users should be able to create/read/update this field. Validate the remote type against REMOTE_TYPES and that a user is only supplying 0..1 remote. Also, indicate in the help text that this field is optional.

Remotes

Expose a repository hrefs field for remotes. Users should be able to view this field to see which repos a remote belongs to. Also, indicate in the help text that this field is optional.

Syncing

Make the remote parameter on sync optional. If no remote is supplied, the sync task uses the remote field on Repository when syncing.

If a repository does not have a remote and no remote is supplied, raise an error.


Related issues

Copied to File Support - Story #7135: As a user, I can associate a remote with a repositoryCLOSED - CURRENTRELEASE

<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Copied to RPM Support - Story #7159: As a user, I can associate a remote with a repositoryCLOSED - CURRENTRELEASE

<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Copied to Ansible Plugin - Story #7194: As a user, I can associate a remote with a repositoryCLOSED - CURRENTRELEASE

<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision f31c7b61 View on GitHub
Added by daviddavis 2 months ago

Allow Remotes to be associated to Repositories

fixes #7015 https://pulp.plan.io/issues/7015

Required PR: https://github.com/pulp/pulp_file/pull/415

History

#1 Updated by fao89 3 months ago

  • Tracker changed from Issue to Story
  • % Done set to 0

#2 Updated by daviddavis 3 months ago

  • Description updated (diff)
  • Tags deleted (API Bindings)

#3 Updated by daviddavis 3 months ago

  • Description updated (diff)

#4 Updated by daviddavis 3 months ago

  • Subject changed from Ability to link Repository with Remote to As a user, I can associate a set of remotes with a repository

#5 Updated by bmbouter 3 months ago

Can we outline what validation means a bit more? Should the plugin developer specify on a typed Repository what Remote types can be paired with it? For example pulp_ansible has a AnsibleRepository and a CollectionRemote and RolesRemote. So in that case the Repository would specify both CollectionRemote and RolesRemore are allowed.

Also does a Repository only pair with a single Remote? Or are we allowing a Repository to pair with multiple remotes?

The model layer calls out the optional nature of the ManyToMany, but maybe the API should call out that these new href fields are optional as well (for clarity)?

#6 Updated by daviddavis 3 months ago

  • Description updated (diff)

Can we outline what validation means a bit more? Should the plugin developer specify on a typed Repository what Remote types can be paired with it? For example pulp_ansible has a AnsibleRepository and a CollectionRemote and RolesRemote. So in that case the Repository would specify both CollectionRemote and RolesRemore are allowed.

This sounds great. I wonder if this should be a constant like we did for Content or a method. Any thoughts?

Also does a Repository only pair with a single Remote? Or are we allowing a Repository to pair with multiple remotes?

I was thinking many-to-many so a repository could have multiple remotes and a remote could belong to multiple repos. I will update the description to be more clear.

The model layer calls out the optional nature of the ManyToMany, but maybe the API should call out that these new href fields are optional as well (for clarity)?

Good idea. Done.

#7 Updated by ipanova@redhat.com 3 months ago

Right now you can sync with one remote at time, the way it is described in the story - it would be possible to sync a repo wit X related remotes and as a result only one repo version will be created. Was that intentional/desirable?

How mirror on sync will work with multiple remotes?Or retain old count?

What if one the feed from one of X remotes is broken, or sync of one remote fails in the middle - do we fail all the rest remote syncing? What if one of the remotes is a mirrorlist?

#8 Updated by bmbouter 3 months ago

+1 to using an iterable as a constant on Repository to define validation.

Is there a way to launch this feature with just a single remote and add on N remotes later? If not, we'll have to discuss more about what multi-remote sync means. I'm ok with trying to figure that out if folks think we should. What do you think?

#9 Updated by daviddavis 3 months ago

Is there a way to launch this feature with just a single remote and add on N remotes later? If not, we'll have to discuss more about what multi-remote sync means. I'm ok with trying to figure that out if folks think we should. What do you think?

I'm ok with launching this with just supporting a single remote. However, I am not sure how we'd add support for N remotes later in a backwards compatible way?

#10 Updated by bmbouter 3 months ago

daviddavis wrote:

Is there a way to launch this feature with just a single remote and add on N remotes later? If not, we'll have to discuss more about what multi-remote sync means. I'm ok with trying to figure that out if folks think we should. What do you think?

I'm ok with launching this with just supporting a single remote. However, I am not sure how we'd add support for N remotes later in a backwards compatible way?

List of remotes and fail validation if len(remotes) > 1 for now?

#11 Updated by ipanova@redhat.com 3 months ago

bmbouter wrote:

daviddavis wrote:

Is there a way to launch this feature with just a single remote and add on N remotes later? If not, we'll have to discuss more about what multi-remote sync means. I'm ok with trying to figure that out if folks think we should. What do you think?

I'm ok with launching this with just supporting a single remote. However, I am not sure how we'd add support for N remotes later in a backwards compatible way?

List of remotes and fail validation if len(remotes) > 1 for now?

that's an option. And open a separate story for syncing form multiple remotes

#12 Updated by ttereshc 3 months ago

We can probably support multiple remotes if we require configuration parameters (apart from feed and credentials) to be the same.

A more flexible approach would require plugin input (which parameters have to be the same, which can't be used at all, etc.)

+1 to have a list of 1 remote for now.

#13 Updated by daviddavis 3 months ago

  • Subject changed from As a user, I can associate a set of remotes with a repository to As a user, I can associate a remote with a repository
  • Description updated (diff)

#14 Updated by daviddavis 3 months ago

  • Description updated (diff)

#15 Updated by daviddavis 3 months ago

  • Description updated (diff)

#16 Updated by daviddavis 3 months ago

  • Description updated (diff)

#17 Updated by ipanova@redhat.com 3 months ago

looks good to me, thanks!

#18 Updated by bmbouter 3 months ago

  • Description updated (diff)

#19 Updated by bmbouter 3 months ago

  • Groomed changed from No to Yes
  • Sprint Candidate changed from No to Yes

+1 to this story. Thanks!

#20 Updated by daviddavis 3 months ago

  • Copied to Story #7135: As a user, I can associate a remote with a repository added

#21 Updated by pulpbot 3 months ago

  • Status changed from NEW to POST

#22 Updated by daviddavis 3 months ago

  • Assignee set to daviddavis

#23 Updated by dalley 3 months ago

@Brian, is there a good place where I can read about why both CollectionRemote and RolesRemote are necessary and separate? Just curious since it's an atypical case to think about. I have some other questions about the PR that are kinda related to that question.

#24 Updated by daviddavis 2 months ago

  • Copied to Story #7159: As a user, I can associate a remote with a repository added

#25 Updated by daviddavis 2 months ago

  • Copied to Story #7194: As a user, I can associate a remote with a repository added

#26 Updated by daviddavis 2 months ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#27 Updated by dkliban@redhat.com about 2 months ago

  • Sprint/Milestone set to 3.6.0

#28 Updated by pulpbot about 2 months ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Please register to edit this issue

Also available in: Atom PDF