RPM Support

The RPMDistribution will need to support users providing a repository or a repository version in addition to publications.

ideally we wouldn't have to generate a normal yum publication when going this route, as those are quite expensive to generate.

jsherril@redhat.com wrote:

The RPMDistribution will need to support users providing a repository or a repository version in addition to publications.

ideally we wouldn't have to generate a normal yum publication when going this route, as those are quite expensive to generate.

You would not need to create a publication. That's why we need to be able to serve the repository version directly.

This feature would be very welcome in RHUI as it would save us from regenerating the metadata every time the repo content is updated. So yes, it gets our votes!

Open question: Should the DeclarativeContent pipeline be extended to allow this functionality, or should it remain entirely within the plugin?

The latter might make more sense for the initial implementation, but if Debian wants to switch to this method we might want to be able to share the implementation.

This is a separate question from the invasive generic proposal.

dalley wrote:

Open question: Should the DeclarativeContent pipeline be extended to allow this functionality, or should it remain entirely within the plugin?

The latter might make more sense for the initial implementation, but if Debian wants to switch to this method we might want to be able to share the implementation.

I would suggest keeping the changes for now entirely in the plugin. Both RPM and Debian plugins are having a complex pipeline, would be good to first implement the proposal and then decouple what can be shared.

This is a separate question from the invasive generic proposal.

This could be accomplished by the following:

Have a way to distinguish between repositories with managed content and with the exact mirror (e.g. create a repository with exact_mirror=True or a new dedicated repository type, RpmMirrorRepository)

I think having a separate repo type will be a cleaner solution, we can disable endpoints we do not want to expose , for example /modify endpoint and also take control over what options to enable. I agree that this type of the repo should be immutable, meaning no content can be added to it or removed from it.

For such repos, create a publication at sync time (includes published artifacts and metadata).

I wonder how we would leave the room to the user to specify signing_service and gpg_check options?

For such repos, publish is no-op and always returns the existing publication for the requested repo version.

Apparently in this step we could allow user to re-publish the repo with signing_service and gpg_check options if needed, but definitely not allowing setting checksum_types

For such repos, no modifications are allowed except the sync in mirror mode.

I guess we should not allow skipping types.

I am wondering - has it been considered to add metadata as a separate content typse to the mirror repo type? This could allow us to distribute the repository right away without the need of creating the publication. On the other hand i would not know how we'd allow user setting a gpg_check option, for example. This idea is obviously far from being flawless, just throwing it on the table for discussion.

I wonder how we would leave the room to the user to specify signing_service and gpg_check options?

I'm not sure that we need to provide a way to sign repo metadata here. The idea is to have a pure mirror of the remote repo without any changes. But maybe I'm just not aware of a use case and customers will be interested in it. I'm open for feedback here.

I am wondering - has it been considered to add metadata as a separate content type to the mirror repo type? This could allow us to distribute the repository right away without the need of creating the publication. On the other hand i would not know how we'd allow user setting a gpg_check option, for example. This idea is obviously far from being flawless, just throwing it on the table for discussion.

I believe it was one of the initial ideas. One of the reason the current proposal is different because it's potentially a one step closer to a more generalised solution for the relative_path problem while the separate content type for metadata won't help in such case. Another potential concern is managing this content type in some special way... it's not a content type you want to show to a user and you need to disallow copy operation for it, aka not associate this content with any other repo, etc.

The initial proposal was to add some parameter to the repository, such as "mirror=True", and use this value to control how the remote behaves, or else to create an entire new model specifically about "mirroring".

I think we don't need such large changes to support this, and the less-invasive and lower-effort approach might actually provide better opportunities for improving Pulp workflows as a whole.

Instead of trying to implement this functionality at the data "model" layer, we should just provide a new viewset. We make no model changes whatsoever. We introduce a new RemoteRepositoryViewset which will provide a new primitive Pulp endpoint, /pulp/api/v3/remote_repositories/<plugin>/<type>/. This viewset will work with all repositories that have remote set, which is to say that the queryset would be Repository.objects.filter(remote__isnull=False).

This viewset would provide only one action endpoint, "refresh" (or "update"), which would perform a mirror-mode sync and publish with the internally-attached remote. The viewset would not provide the "modify" endpoint since mirrored repositories shouldn't be able to be changed, and it wouldn't provide the "sync" endpoint because there would not be any need to customize the sync in that way. This avoids needing to throw errors if a user attempts an operation they shouldn't do - because the new API provides no mechanism to do it - and also avoids creating a new model.

The old /repositories/<plugin>/<type>/ endpoint will remain exactly as it is currently, even at the code level, since all of the changes are in the new viewset. However we should discourage users from using this API with repositories they want to sync from external sources, and encourage using the new API instead, which has the benefit of being much easier to use for most if not all use cases.

The workflow:

1. Create a "remote repository", specifying name + description + any remote options, which will transparently create both the remote and repository.
2. "refresh" (or "update" or "sync") it - no parameters necessary because it uses the internal remote. syncs for most plugins would always "mirror"

If a user just wants local mirrors, they're already done. For use cases where a user wants to modify or combine repos, we would encourage the "Katello workflow" of just copying all content from their remote repo into the new one.

Compared to syncing from many remotes into one repo, this workflow:

• Is much more efficient (less network traffic, only parse metadata once, no need to query existing Content or Artifacts)
• Is more deterministic / reliable (the remote URLs could change between syncs, or the syncs could accidentally use different mirrors)

+1 to the plan outlined in comment 26. I think that takes us in the right direction and would be much for usable than having API endpoints which are available but throw errors when repositories don't have remotes, or when those that are only for mirroring must never use the /modify endpoint.

bmbouter wrote:

+1 to the plan outlined in comment 26. I think that takes us in the right direction and would be much for usable than having API endpoints which are available but throw errors when repositories don't have remotes, or when those that are only for mirroring must never use the /modify endpoint.

Concur with all of the above. While there may be some "what happens if a user does X" edge-cases to consider/close, this feels like a great approach!

dalley wrote:

The initial proposal was to add some parameter to the repository, such as "mirror=True", and use this value to control how the remote behaves, or else to create an entire new model specifically about "mirroring".

I think we don't need such large changes to support this, and the less-invasive and lower-effort approach might actually provide better opportunities for improving Pulp workflows as a whole.

Instead of trying to implement this functionality at the data "model" layer, we should just provide a new viewset. We make no model changes whatsoever. We introduce a new RemoteRepositoryViewset which will provide a new primitive Pulp endpoint, /pulp/api/v3/remote_repositories/<plugin>/<type>/. This viewset will work with all repositories that have remote set, which is to say that the queryset would be Repository.objects.filter(remote__isnull=False).

If other plugins would not take advantage of such workflows and it is not known of such as of today, my suggestion would be to use a pulp-rpm endpoint /pulp/api/v3/pulp_rpm/<mirrored-repos>/

My worry is that concepts like mirror/sync/remote repo/ are overused, and in plugins that don't have any repodata might create more confusion than desired.

This viewset would provide only one action endpoint, "refresh" (or "update"), which would perform a mirror-mode sync and publish with the internally-attached remote. The viewset would not provide the "modify" endpoint since mirrored repositories shouldn't be able to be changed, and it wouldn't provide the "sync" endpoint because there would not be any need to customize the sync in that way. This avoids needing to throw errors if a user attempts an operation they shouldn't do - because the new API provides no mechanism to do it - and also avoids creating a new model.

The old /repositories/<plugin>/<type>/ endpoint will remain exactly as it is currently, even at the code level, since all of the changes are in the new viewset. However we should discourage users from using this API with repositories they want to sync from external sources, and encourage using the new API instead, which has the benefit of being much easier to use for most if not all use cases.

The workflow:

1. Create a "remote repository", specifying name + description + any remote options, which will transparently create both the remote and repository.
2. "refresh" (or "update" or "sync") it - no parameters necessary because it uses the internal remote. syncs for most plugins would always "mirror"

If choosing between refresh/update/sync I would probably go with 'mirror' (or 'replicate'? too much, overkill?) Even though we have a sync mode option 'mirror' which does not alleviate the tautology.

If a user just wants local mirrors, they're already done. For use cases where a user wants to modify or combine repos, we would encourage the "Katello workflow" of just copying all content from their remote repo into the new one.

If one wants to get rid of 1 rpm copying the whole repo is a major inconvenience.

If understood correctly, whether I create a repo via the new endpoint or old one, if it has a remote it will appear in both endpoints. I know, what i am going to propose might be confusing but the user instead of copying content could just use the old API. It will require us to document in a very clear way, but why not leaving to the user both options where he could switch between new and old api based on the needs.

Compared to syncing from many remotes into one repo, this workflow:

• Is much more efficient (less network traffic, only parse metadata once, no need to query existing Content or Artifacts)
• Is more deterministic / reliable (the remote URLs could change between syncs, or the syncs could accidentally use different mirrors)

If other plugins would not take advantage of such workflows and it is not known of such as of today, my suggestion would be to use a pulp-rpm endpoint /pulp/api/v3/pulp_rpm//

My worry is that concepts like mirror/sync/remote repo/ are overused, and in plugins that don't have any repodata might create more confusion than desired.

This is definitely something that every plugin could take advantage of to make the workflow simpler, even if they don't use mirrored metadata. Instead of creating a remote + repo they would just create one remote_repository, and instead of having to keep track of the remote to pass it to /sync/ every time, they would just refresh (or whatever).

The fact that it fixes the metadata mirroring UX is a nice bonus.

If choosing between refresh/update/sync I would probably go with 'mirror' (or 'replicate'? too much, overkill?) Even though we have a sync mode option 'mirror' which does not alleviate the tautology.

Yeah, we can have a nice long bikeshed discussion about the names later on :)

If one wants to get rid of 1 rpm copying the whole repo is a major inconvenience.

If understood correctly, whether I create a repo via the new endpoint or old one, if it has a remote it will appear in both endpoints. I know, what i am going to propose might be confusing but the user instead of copying content could just use the old API. It will require us to document in a very clear way, but why not leaving to the user both options where he could switch between new and old api based on the needs.

This is a legitimate concern and the suggestion is definitely something the user could do if they wanted to, but I don't think we should encourage it. We can almost certainly come up with better ways to address the problem.

If a user is doing this, they're already running the risk of having the next sync wipe out the changes (package removals) they made, so it's an error-prone workflow to begin with. It's also not easy to tell what you've changed vs. what changed remotely because the history is all mixed together. Wheras with the copy API you can use "base_version=... remove_content_units=[...]" to exclude your couple of content units, and you can see what changes you made.**

**this might not be true if the repo version "added", "removed" aren't based on the "base_version" -- it's been a while since I checked. But in theory we could give the user more detailed information than they had previously with this workflow.

PR: https://github.com/pulp/pulp_rpm/pull/2007