Project

Profile

Help

Story #7015

closed

As a user, I can associate a remote with a repository

Added by wibbit over 2 years ago. Updated about 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Sprint:
Quarter:

Description

Currently there is no way to link a repository with a remote, and as such, each time you wish to sync a repository against it's upstream link, you have to keep a record (or query for) the appropriate remote for that repo.

This can be done with careful consideration around naming conventions, however it would be nice if an optional link between a Repository and Remote was possible.

As such, once the Repository and Remote have been created an paired, a simply repository sync would be able to work out which remote to use, and sync.

Design

Model

Add a many-to-many foreign key between Repositories and Remotes. This association should be optional. For now Repositories can only have 0 or 1 remotes. Remotes can belong to any number of repositories.

Note: In the future we want to be able to support multiple remotes for repositories that can be synced at once. See https://pulp.plan.io/issues/7031. This is the purpose of having a many-to-many instead of one-to-many.

REMOTE_TYPES

Add an iterable to Repository similar to CONTENT_TYPES. This iterable will check that when a remote is added to a repo, it is of one of these types.

API

Repositories

Expose a remote hrefs field for repository endpoints. Users should be able to create/read/update this field. Validate the remote type against REMOTE_TYPES and that a user is only supplying 0..1 remote. Also, indicate in the help text that this field is optional.

Remotes

Expose a repository hrefs field for remotes. Users should be able to view this field to see which repos a remote belongs to. Also, indicate in the help text that this field is optional.

Syncing

Make the remote parameter on sync optional. If no remote is supplied, the sync task uses the remote field on Repository when syncing.

If a repository does not have a remote and no remote is supplied, raise an error.


Related issues

Copied to File Support - Story #7135: As a user, I can associate a remote with a repositoryCLOSED - CURRENTRELEASEdaviddavis

Actions
Copied to RPM Support - Story #7159: As a user, I can associate a remote with a repositoryCLOSED - CURRENTRELEASEdaviddavis

Actions
Copied to Ansible Plugin - Story #7194: As a user, I can associate a remote with a repositoryCLOSED - CURRENTRELEASEdaviddavis

Actions
Actions #1

Updated by fao89 over 2 years ago

  • Tracker changed from Issue to Story
  • % Done set to 0
Actions #2

Updated by daviddavis over 2 years ago

  • Description updated (diff)
  • Tags deleted (API Bindings)
Actions #3

Updated by daviddavis over 2 years ago

  • Description updated (diff)
Actions #4

Updated by daviddavis over 2 years ago

  • Subject changed from Ability to link Repository with Remote to As a user, I can associate a set of remotes with a repository
Actions #5

Updated by bmbouter over 2 years ago

Can we outline what validation means a bit more? Should the plugin developer specify on a typed Repository what Remote types can be paired with it? For example pulp_ansible has a AnsibleRepository and a CollectionRemote and RolesRemote. So in that case the Repository would specify both CollectionRemote and RolesRemore are allowed.

Also does a Repository only pair with a single Remote? Or are we allowing a Repository to pair with multiple remotes?

The model layer calls out the optional nature of the ManyToMany, but maybe the API should call out that these new href fields are optional as well (for clarity)?

Actions #6

Updated by daviddavis over 2 years ago

  • Description updated (diff)

Can we outline what validation means a bit more? Should the plugin developer specify on a typed Repository what Remote types can be paired with it? For example pulp_ansible has a AnsibleRepository and a CollectionRemote and RolesRemote. So in that case the Repository would specify both CollectionRemote and RolesRemore are allowed.

This sounds great. I wonder if this should be a constant like we did for Content or a method. Any thoughts?

Also does a Repository only pair with a single Remote? Or are we allowing a Repository to pair with multiple remotes?

I was thinking many-to-many so a repository could have multiple remotes and a remote could belong to multiple repos. I will update the description to be more clear.

The model layer calls out the optional nature of the ManyToMany, but maybe the API should call out that these new href fields are optional as well (for clarity)?

Good idea. Done.

Actions #7

Updated by ipanova@redhat.com over 2 years ago

Right now you can sync with one remote at time, the way it is described in the story - it would be possible to sync a repo wit X related remotes and as a result only one repo version will be created. Was that intentional/desirable?

How mirror on sync will work with multiple remotes?Or retain old count?

What if one the feed from one of X remotes is broken, or sync of one remote fails in the middle - do we fail all the rest remote syncing? What if one of the remotes is a mirrorlist?

Actions #8

Updated by bmbouter over 2 years ago

+1 to using an iterable as a constant on Repository to define validation.

Is there a way to launch this feature with just a single remote and add on N remotes later? If not, we'll have to discuss more about what multi-remote sync means. I'm ok with trying to figure that out if folks think we should. What do you think?

Actions #9

Updated by daviddavis over 2 years ago

Is there a way to launch this feature with just a single remote and add on N remotes later? If not, we'll have to discuss more about what multi-remote sync means. I'm ok with trying to figure that out if folks think we should. What do you think?

I'm ok with launching this with just supporting a single remote. However, I am not sure how we'd add support for N remotes later in a backwards compatible way?

Actions #10

Updated by bmbouter over 2 years ago

daviddavis wrote:

Is there a way to launch this feature with just a single remote and add on N remotes later? If not, we'll have to discuss more about what multi-remote sync means. I'm ok with trying to figure that out if folks think we should. What do you think?

I'm ok with launching this with just supporting a single remote. However, I am not sure how we'd add support for N remotes later in a backwards compatible way?

List of remotes and fail validation if len(remotes) > 1 for now?

Actions #11

Updated by ipanova@redhat.com over 2 years ago

bmbouter wrote:

daviddavis wrote:

Is there a way to launch this feature with just a single remote and add on N remotes later? If not, we'll have to discuss more about what multi-remote sync means. I'm ok with trying to figure that out if folks think we should. What do you think?

I'm ok with launching this with just supporting a single remote. However, I am not sure how we'd add support for N remotes later in a backwards compatible way?

List of remotes and fail validation if len(remotes) > 1 for now?

that's an option. And open a separate story for syncing form multiple remotes

Actions #12

Updated by ttereshc over 2 years ago

We can probably support multiple remotes if we require configuration parameters (apart from feed and credentials) to be the same.

A more flexible approach would require plugin input (which parameters have to be the same, which can't be used at all, etc.)

+1 to have a list of 1 remote for now.

Actions #13

Updated by daviddavis over 2 years ago

  • Subject changed from As a user, I can associate a set of remotes with a repository to As a user, I can associate a remote with a repository
  • Description updated (diff)
Actions #14

Updated by daviddavis over 2 years ago

  • Description updated (diff)
Actions #15

Updated by daviddavis over 2 years ago

  • Description updated (diff)
Actions #16

Updated by daviddavis over 2 years ago

  • Description updated (diff)
Actions #17

Updated by ipanova@redhat.com over 2 years ago

looks good to me, thanks!

Actions #18

Updated by bmbouter over 2 years ago

  • Description updated (diff)
Actions #19

Updated by bmbouter over 2 years ago

  • Groomed changed from No to Yes
  • Sprint Candidate changed from No to Yes

+1 to this story. Thanks!

Actions #20

Updated by daviddavis about 2 years ago

  • Copied to Story #7135: As a user, I can associate a remote with a repository added
Actions #21

Updated by pulpbot about 2 years ago

  • Status changed from NEW to POST
Actions #22

Updated by daviddavis about 2 years ago

  • Assignee set to daviddavis
Actions #23

Updated by dalley about 2 years ago

@Brian, is there a good place where I can read about why both CollectionRemote and RolesRemote are necessary and separate? Just curious since it's an atypical case to think about. I have some other questions about the PR that are kinda related to that question.

Actions #24

Updated by daviddavis about 2 years ago

  • Copied to Story #7159: As a user, I can associate a remote with a repository added
Actions #25

Updated by daviddavis about 2 years ago

  • Copied to Story #7194: As a user, I can associate a remote with a repository added

Added by daviddavis about 2 years ago

Revision f31c7b61

Allow Remotes to be associated to Repositories

fixes #7015 https://pulp.plan.io/issues/7015

Required PR: https://github.com/pulp/pulp_file/pull/415

Actions #26

Updated by daviddavis about 2 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100
Actions #27

Updated by dkliban@redhat.com about 2 years ago

  • Sprint/Milestone set to 3.6.0
Actions #28

Updated by pulpbot about 2 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF