Pulp

How much de-duplication do you expect on the remote filesystem? Say you have Repo A and Repo B both which have package 'foo' with the same NEVRA and checksum. Should 1 or 2 copies be stored on the filesystem? Not having to de-duplicate remotely makes this a lot easier, but also not as awesome. Overall I'd like to do something simple first.

What are you requirements?

Since our main usecase would be generating quarterly updates we'd be perfectly fine with no deduplication for now. All in all, it would surely be nice, but not required.

bmbouter wrote:

On Demand Content¶

This feature does not work for on-demand content because the content must be present locally for Pulp to write it to a filesystem.

I wonder if the remote content policy should really make a difference? I might get that wrong, but when the exporter requests the content "on_demand" content would be fetched from the remote or served from disk, "streamed" content would be streamed and "immediate" content would be served from disk, not?

fachleitner wrote:

I wonder if the remote content policy should really make a difference? I might get that wrong, but when the exporter requests the content "on_demand" content would be fetched from the remote or served from disk, "streamed" content would be streamed and "immediate" content would be served from disk, not?

These are great questions. This may be more info than you're interested in, but I want to share the info anyway.

Architecturally, the pulp content app and the pulp API are peer services that both face the user but not each other. Deploying Pulp in a way that the pulp content app also serves the Pulp API could be practically difficult as load balancers or network firewall and routing details begin to matter a lot. For example the services could be non-routable to each other.

Instead there is heavy code sharing between the pulp content app and the pulp api code in the form of the Pulp downloaders, downloader factory, etc. This means that for the exporter code to "get undownloaded content" it needs to make its own downloaders, download things, save them, etc, which means it effectively needs to cover all the same corner cases as a re-sync itself.

So if fetching on-demand content for export is equivalent to a re-sync only with policy='immediate' then we could either trigger one before export or ask the user to do the same. Auto-triggering a sync is practically complicated due to the plugin providing the sync endpoint, and also could be unexpected for a user who uses all policy='on_demand' or policy='streamed' to use this exporter and have to store a copy of all that in their Pulp in addition to their exported area (two copies).

For ^ reasons, I believe the least surprising thing to do is to halt an export that contains on-demand content and report to the user that they should re-sync the repository using policy to 'immediate' to acquire all content before trying to export it. Then they can run the export task again.

Feedback on this reasoning and other ideas are welcome.

This exporter should support accepting multiple repository versions.

bmbouter wrote:

For ^ reasons, I believe the least surprising thing to do is to halt an export that contains on-demand content and report to the user that they should re-sync the repository using policy to 'immediate' to acquire all content before trying to export it. Then they can run the export task again.

Feedback on this reasoning and other ideas are welcome.

Thanks, for the explanation. I think my question came from the assumption that the exporter would be a client of the content app, like reposync or yum/dnf. Do I understand correctly, that the exporter would be a seperate "service" (or "app"), parallel to the content server, and thus it needs the content on disk to be able to export it?

The user facing docs don't exist yet (this ticket to add it), but the Exporter is code that run in the Pulp tasking system (like all other Pulp tasks). The workflow is that it facilitates a "push" model where you can have pulp export content through a triggered command to Pulp.

The idea of having a "pull" tool to crawl the repository's contents as served from the Content App and save them as a concept for "export" would also be ok. In that case, to remain content type agnostic, tool would probably have to list content via the Pulp API and then fetch it from the content app one-by-one. That would work, and is more like a "pull" model executed from outside Pulp. That is a different idea than a Pulp Exporter though.

Per our discussion with bmbouter and dkliban, removing this from the sprint since we don't plan to work on it this sprint. This still should be up for consideration for 3.0 GA though.

I have a couple thoughts on this issue.

First, I think it might be better to start development in the plugins rather than core. I'm worried that we might make some incorrect assumptions about how plugins want to export their content. Then we'll be stuck with a solution that doesn't work for all plugins due to semver.

Secondly, I wonder if this needs to be a GA blocker. Do any stakeholders need this or does this work require backwards incompatible changes?

daviddavis wrote:

I have a couple thoughts on this issue.

First, I think it might be better to start development in the plugins rather than core. I'm worried that we might make some incorrect assumptions about how plugins want to export their content. Then we'll be stuck with a solution that doesn't work for all plugins due to semver.

From a risk perspective starting in the plugins is a strategy I can understand, but there are challenges with that approach here. If we're not shipping it in core what ships it? We could pick "a plugin" and put it there, but this code would be compatible with any Publication so then users who don't use "that plugin" would need to install it a plugin they don't want to receive this.

Secondly, I wonder if this needs to be a GA blocker. Do any stakeholders need this or does this work require backwards incompatible changes?

There are two reasons I think this should be a GA blocker, but lmk what you think.

1) We need to be sure that the Artifact storage format of a Publication can be fully represented on disk. Currently it cannot due to POSIX filesystem incompatibilities between a file and directory with the same names. We know this by thinking about the problem, but we don't know what other issues we'll run into fulfilling that requirement. Fixing at least this known problem is 1 backwards incompatible change. This export story is the proof point that we can do that (and there aren't more backwards incompatible changes)

2) We need to have at least 1 working example for GA of an exporter to be sure it works. Currently there is no code that uses the existing core machinery for exporters.

What do you think about ^?

This looks good. 2 questions I had:

1) Are we going to raise a validation error when the repo/repo_version or publication doesn't have all Artifacts locally? That would be good to me so we fail early in that case.

2) I had expected the two objects would be Detail classes based off of this existing master object