Pulp

Our current design allows the plugin writer to create publishers that can create publications that filter out some content from a repository version. This means users have two opportunities to compose a repository - at repository version creation time and when creating a publication. I would prefer to provide only one such opportunity at repository version creation time.

Another content type that does not require generating metadata at publish time is Maven. All the metadata is part of the content. So I can see a benefit for that plugin.

+1 from me. I think PublishedArtifact is a remnant from before repo versions. Being able to remove the table and the need to create a ton of records for every publication would be a big improvement and simplification.

Also, publication already has a FK to repo version so I think we're good there.

The purpose of the PublishedArtifact was to provide the publisher with the opportunity to publish each Artifact with a custom relative path.

For example: A (remote) DNF repository that looks like:

a.rpm
b.rpm
repodata/

The ContentArtifact.relative_path (and PublishedArtifact) for each artifact would be:

a.rpm
b.rpm

The PublishedArtifact provides for publishing in a different (custom) structure. Eg: to a packages/ directory.

packages/
    a.rpm
    b.rpm
repodata/

The PublishedArtifact.relative_path (for rpms) would be:

packages/a.rpm
packages/b.rpm

Perhaps we can support the publish-as-is use case by providing something like an as-is attribute to Publication since it already has a FK to the version. The content app could then resolve to ContentArtifact.relative_path as suggested when not matched to PublishedMetadata or PublishedArtifact. The core could provide base publishing that produces a PublishedVersion.

@jortel I see what you're saying about it providing that customization point, but is anyone using it? I was trying to think if anyone was, but all the plugins I found were using ContentArtifact.relative_path as-is. Even RPM uses it as-is: https://github.com/pulp/pulp_rpm/blob/9cd7b8237194bc79b5454c7e53eaba673a21077a/pulp_rpm/app/tasks/publishing.py#L194

Everything we add takes away from the other things so if this feature isn't being used by any plugins we should consider removing it until it's needed by someone (I think). Are plugin writers using this?

Doesn't this proposal mean that, on content creation, the plugin has to decide on the exact relative location of the artifact in every publication that will ever occur?

What if:
- The plugin writer wants to change the publication layout like done for RPM in Pulp 2.12 [0]
- The plugin writer wants to support user selectable layouts when publishing
- There is a new "v2" of the repo structure, which necessitates path changes. The plugin writer wants to support publishing existent artifacts using the new structure.

I am not saying that this is absolutely necessary, but we should know the consequences of this decision. We could say that these cases are use cases for a live API because it is more complex than what Pulp core is willing to handle.

The other idea I had is what @jortel just described. :-)

[0] https://pulpproject.org/2016/11/14/yum-repo-layout-changes/

bmbouter wrote:

Everything we add takes away from the other things so if this feature isn't being used by any plugins we should consider removing it until it's needed by someone (I think). Are plugin writers using this?

Yes, I am using this in pulp_cookbook. But only because it is convenient. Content units just have the name of the tar file as relative path (e.g. pulp-1.2.3.tar.gz). Publication is at cookbook_files/pulp/1_2_3/pulp-1.2.3.tar.gz. But I could use this path at content creation already.

I had not written this before but a main goal of this (for me) is to remove a step for the user! Currently you need to create a "publisher". I believe with this change they could just POST to create a publication w/ pass_through=True and the repository version and they could be done. One more call to a distributor and that content is live.

@jortel is ^ make sense to you? Can ^ be done as part of this piece of work?

bmbouter wrote:

I had not written this before but a main goal of this (for me) is to remove a step for the user! Currently you need to create a "publisher". I believe with this change they could just POST to create a publication w/ pass_through=True and the repository version and they could be done. One more call to a distributor and that content is live.

@jortel is ^ make sense to you? Can ^ be done as part of this piece of work?

Yes. I envisioned the same thing. Will include it.