RPM Support

Very-nice-to-have in 2.19.1 but not critical.

I've been looking into this for a while and the behavior here is definitely wrong in multiple different ways, which is making it extra tricky to figure out where the problems are coming from.

Recursive copy by name, stream filter

[vagrant@pulp2 ~]$ pulp-admin rpm repo copy module --str-eq="name=pteradactyl" --str-eq="stream=2" --from-repo-id=test --to-repo-id=test3 --recursive
This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Copied:
 modulemd:
  pteradactyl-1-20180730223407-deadbeef-noarch
  pteradactyl-1-20180730223408-deadbeef-noarch
  pteradactyl-2-20180730223407-deadbeef-noarch
  pteradactyl-2-20180730223408-deadbeef-noarch
  pteradactyl-3-20180730333407-deadbeef-noarch
  pteradactyl-3-20180730333408-deadbeef-noarch
  pteradactyl-4-20180740444407-deadbeef-noarch
  pteradactyl-4-20180740444408-deadbeef-noarch
 rpm:
  pteradactyl-2.0-1-noarch
  pteradactyl-2.1-1-noarch
  pteradactyl-4.1-1-noarch

Recursive copy by unit ID

[vagrant@pulp2 ~]$ pulp-admin rpm repo copy module --str-eq="_id=a0ed500f-9efe-4e23-a3c6-72c9e1a0f53c" --from-repo-id=test --to-repo-id=test3 --recursive
This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Copied:
 modulemd:
  pteradactyl-1-20180730223407-deadbeef-noarch
  pteradactyl-1-20180730223408-deadbeef-noarch
  pteradactyl-2-20180730223407-deadbeef-noarch
  pteradactyl-2-20180730223408-deadbeef-noarch
  pteradactyl-3-20180730333407-deadbeef-noarch
  pteradactyl-3-20180730333408-deadbeef-noarch
  pteradactyl-4-20180740444407-deadbeef-noarch
  pteradactyl-4-20180740444408-deadbeef-noarch
 rpm:
  pteradactyl-2.1-1-noarch
  pteradactyl-4.1-1-noarch

Conservative Recursive copy by name, stream filter

[vagrant@pulp2 ~]$ pulp-admin rpm repo copy module --str-eq="name=pteradactyl" --str-eq="stream=2" --from-repo-id=test --to-repo-id=test3 --recursive-conservative
This command may be exited via ctrl+c without affecting the request.

Running...

Copied:
 modulemd:
  pteradactyl-1-20180730223407-deadbeef-noarch
  pteradactyl-1-20180730223408-deadbeef-noarch
  pteradactyl-2-20180730223407-deadbeef-noarch
  pteradactyl-2-20180730223408-deadbeef-noarch
  pteradactyl-3-20180730333407-deadbeef-noarch
  pteradactyl-3-20180730333408-deadbeef-noarch
  pteradactyl-4-20180740444407-deadbeef-noarch
  pteradactyl-4-20180740444408-deadbeef-noarch
 rpm:
  pteradactyl-2.0-1-noarch
  pteradactyl-2.1-1-noarch

Conservative Recursive copy by unit ID

[vagrant@pulp2 ~]$ pulp-admin rpm repo copy module --str-eq="_id=a0ed500f-9efe-4e23-a3c6-72c9e1a0f53c" --from-repo-id=test --to-repo-id=test3 --recursive-conservative
This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Copied:
 modulemd:
  pteradactyl-1-20180730223407-deadbeef-noarch
  pteradactyl-1-20180730223408-deadbeef-noarch
  pteradactyl-2-20180730223407-deadbeef-noarch
  pteradactyl-2-20180730223408-deadbeef-noarch
  pteradactyl-3-20180730333407-deadbeef-noarch
  pteradactyl-3-20180730333408-deadbeef-noarch
  pteradactyl-4-20180740444407-deadbeef-noarch
  pteradactyl-4-20180740444408-deadbeef-noarch
 rpm:
  pteradactyl-2.1-1-noarch

So:

• Copying one module will copy all modules sharing the same name, regardless of the stream or version, and regardless of whether you filter by unit ID or using metadata fields.
• The actual RPMs being copied do actually seem to be the RPMs for the modules that we do want copied, but I'm not sure that's a good thing. The RPMs should ALWAYS be copied along with their modules, so even if the modules are copied incorrectly, we should see their RPMs also being copied so that at least the repository will always be complete and valid. But that isn't happening.
• Additionally in recursive (non-conservative) mode it's also copying the most recent RPM from any of the modules (4.1.1) in addition to the RPMs it should be copying, which should not be happening

Considering how messed up this is, I think I'm going to start working with QE to get these behaviors more rigorously tested before I do any more stepping through in a debugger. I'm also going to bump the priority and severity.

After further investigation I've found that some of the metadata on the pteradactyl RPMs used by this test repo is either invalid, or at least very atypical.

I'm going to try recreating this test repo in a more standard way and see if we still have any issues with Pulp.

Update: The form of "provides" in the pterodactyl test repo is indeed very uncommon, but it is apparently possible and valid. I was able to find a single module in the entire Fedora Module repo against which this bug could be reproduced.

(pulp) [vagrant@pulp2 devel]$ pulp-admin rpm repo copy module --str-eq="name=openmpi" --str-eq="stream=2.1" --from-repo-id=module1 --to-repo-id=test2 --recursive-conservative
This command may be exited via ctrl+c without affecting the request.

[-]
Running...

Copied:
  modulemd: 3
  rpm: 138

One module stream selected, 3 modules copied. With that said, it's probably OK to knock the priority of this back down.

Other bugs, maybe related, maybe not :(

(pulp) [vagrant@pulp2 devel]$ pulp-admin rpm repo copy module --str-eq="name=nodejs" --str-eq="stream=10" --from-repo-id=module1 --to-repo-id=test2 --recursive-conservative
This command may be exited via ctrl+c without affecting the request.

[-]
Running...

Copied:
 modulemd:
  nodejs-10-3020190424132100-a5b0195c-x86_64
  nodejs-12-3020190516000906-a5b0195c-x86_64
 rpm:
  c-ares-1.15.0-3.module_f30+3988+b4cb7ecc-x86_64
  http-parser-2.9.2-1.module_f30+4047+77c83006-x86_64
  http-parser-devel-2.9.2-1.module_f30+4047+77c83006-x86_64
  libnghttp2-1.38.0-1.module_f30+4047+77c83006-x86_64
  libnghttp2-devel-1.38.0-1.module_f30+4047+77c83006-x86_64
  libuv-1.28.0-1.module_f30+4047+77c83006-x86_64
  libuv-devel-1.28.0-1.module_f30+4047+77c83006-x86_64
  libuv-static-1.28.0-1.module_f30+4047+77c83006-x86_64
  nghttp2-1.38.0-1.module_f30+4047+77c83006-x86_64
  nodejs-10.15.3-2.module_f30+4047+77c83006-x86_64
  nodejs-devel-10.15.3-2.module_f30+4047+77c83006-x86_64
  nodejs-docs-10.15.3-2.module_f30+4047+77c83006-noarch
  nodejs-libs-10.15.3-2.module_f30+4047+77c83006-x86_64
  npm-6.4.1-1.10.15.3.2.module_f30+4047+77c83006-x86_64
  v8-devel-6.8.275.32-2.module_f30+4047+77c83006-x86_64

Note: NodeJS 10 was copied, only nodejs 10 RPMs were copied, but the nodejs 12 stream module was additionally copied. However there is also a nodejs 11 stream which was not copied. So unlike Pterodactyl, in this instance all of the modulemds were not copied regardless of stream.

Updated notes and observations here, it is unweildy to track it all in in the comments: https://etherpad.net/p/pulp-modularity-issues

I'm unsure whether we should split this issue into multiple ones or not because it's difficult to know what the boundaries between the different defects are without knowing exactly why they're happening. There are certainly multiple different problems though.

When using this PR, almost everything previously mentioned is fixed, and all the serious problems previously mentioned are fixed.

https://github.com/pulp/pulp_rpm/pull/1400

The remaining issue is that when copying one stream from pteradactyl, all streams get copied -- but unlike before, all of the artifacts for all streams are copied correctly, so an incomplete repo is not being created in that circumstance like it was previously. Additionally my "in the wild" example of this behavior now works fine.

Given that this example repo is atypical and created by a random tool off github, I think it's safe to de-prioritize for now, as long as you're OK with that Partha. Is this blocking anything?

dalley - You may want to copy https://etherpad.net/p/pulp-modularity-issues to a new location so we don't lose that when etherpad.net is shut down.
Also, the bz associated with this will be marked as still broken, so if this issue is largely addressed, we need to do some updating of the tracker trail to ensure the states of issues reflect the correct be. Do we need to keep this issue open at all?