Issue #4693
closedModule Streams not copying correctly with recursive and recursive_conservative
Added by paji@redhat.com over 5 years ago. Updated about 4 years ago.
Description
-
Create and sync the following yum repo (Source) -> https://partha.fedorapeople.org/test-repos/pteradactyl/
-
Create another repo Dest which will serve as the destination repo
-
Go to mongo and pick up a uuid for the pteradactly:2 module stream. This stream will be copied from source to dest .
-
run the following command
https://<fqdn>/pulp/api/v2/repositories/Dest/actions/associate/: {"source_repo_id":"Source","criteria":{"type_ids":["modulemd"],"filters":{"association":{"unit_id":{"$in":[<$MODULE UUID>]}}}},"override_config":{"recursive":true}}: {"content_type"=>"application/json", "accept"=>"application/json"}
-
pulp-admin rpm repo list. Check for the number of module mds copied over by the above call.
-
notice that with recursive set to true all the pteradactyl module streams gets copied over, instead of just pteradactly:2 and packages belonging to that
-
Behavior is similar for recursive conservative
Related issues
Updated by dkliban@redhat.com over 5 years ago
- Version set to 2.19.0
- Tags Pulp 2 added
Updated by ttereshc over 5 years ago
- Triaged changed from No to Yes
- Sprint set to Sprint 52
Updated by ttereshc over 5 years ago
Very-nice-to-have in 2.19.1 but not critical.
Updated by dalley over 5 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to dalley
Updated by dkliban@redhat.com over 5 years ago
- Sprint/Milestone deleted (
2.19.1) - Platform Release deleted (
2.19.1)
Updated by dalley over 5 years ago
I've been looking into this for a while and the behavior here is definitely wrong in multiple different ways, which is making it extra tricky to figure out where the problems are coming from.
Recursive copy by name, stream filter
[vagrant@pulp2 ~]$ pulp-admin rpm repo copy module --str-eq="name=pteradactyl" --str-eq="stream=2" --from-repo-id=test --to-repo-id=test3 --recursive
This command may be exited via ctrl+c without affecting the request.
[\]
Running...
Copied:
modulemd:
pteradactyl-1-20180730223407-deadbeef-noarch
pteradactyl-1-20180730223408-deadbeef-noarch
pteradactyl-2-20180730223407-deadbeef-noarch
pteradactyl-2-20180730223408-deadbeef-noarch
pteradactyl-3-20180730333407-deadbeef-noarch
pteradactyl-3-20180730333408-deadbeef-noarch
pteradactyl-4-20180740444407-deadbeef-noarch
pteradactyl-4-20180740444408-deadbeef-noarch
rpm:
pteradactyl-2.0-1-noarch
pteradactyl-2.1-1-noarch
pteradactyl-4.1-1-noarch
Recursive copy by unit ID
[vagrant@pulp2 ~]$ pulp-admin rpm repo copy module --str-eq="_id=a0ed500f-9efe-4e23-a3c6-72c9e1a0f53c" --from-repo-id=test --to-repo-id=test3 --recursive
This command may be exited via ctrl+c without affecting the request.
[\]
Running...
Copied:
modulemd:
pteradactyl-1-20180730223407-deadbeef-noarch
pteradactyl-1-20180730223408-deadbeef-noarch
pteradactyl-2-20180730223407-deadbeef-noarch
pteradactyl-2-20180730223408-deadbeef-noarch
pteradactyl-3-20180730333407-deadbeef-noarch
pteradactyl-3-20180730333408-deadbeef-noarch
pteradactyl-4-20180740444407-deadbeef-noarch
pteradactyl-4-20180740444408-deadbeef-noarch
rpm:
pteradactyl-2.1-1-noarch
pteradactyl-4.1-1-noarch
Conservative Recursive copy by name, stream filter
[vagrant@pulp2 ~]$ pulp-admin rpm repo copy module --str-eq="name=pteradactyl" --str-eq="stream=2" --from-repo-id=test --to-repo-id=test3 --recursive-conservative
This command may be exited via ctrl+c without affecting the request.
Running...
Copied:
modulemd:
pteradactyl-1-20180730223407-deadbeef-noarch
pteradactyl-1-20180730223408-deadbeef-noarch
pteradactyl-2-20180730223407-deadbeef-noarch
pteradactyl-2-20180730223408-deadbeef-noarch
pteradactyl-3-20180730333407-deadbeef-noarch
pteradactyl-3-20180730333408-deadbeef-noarch
pteradactyl-4-20180740444407-deadbeef-noarch
pteradactyl-4-20180740444408-deadbeef-noarch
rpm:
pteradactyl-2.0-1-noarch
pteradactyl-2.1-1-noarch
Conservative Recursive copy by unit ID
[vagrant@pulp2 ~]$ pulp-admin rpm repo copy module --str-eq="_id=a0ed500f-9efe-4e23-a3c6-72c9e1a0f53c" --from-repo-id=test --to-repo-id=test3 --recursive-conservative
This command may be exited via ctrl+c without affecting the request.
[\]
Running...
Copied:
modulemd:
pteradactyl-1-20180730223407-deadbeef-noarch
pteradactyl-1-20180730223408-deadbeef-noarch
pteradactyl-2-20180730223407-deadbeef-noarch
pteradactyl-2-20180730223408-deadbeef-noarch
pteradactyl-3-20180730333407-deadbeef-noarch
pteradactyl-3-20180730333408-deadbeef-noarch
pteradactyl-4-20180740444407-deadbeef-noarch
pteradactyl-4-20180740444408-deadbeef-noarch
rpm:
pteradactyl-2.1-1-noarch
So:
- Copying one module will copy all modules sharing the same name, regardless of the stream or version, and regardless of whether you filter by unit ID or using metadata fields.
- The actual RPMs being copied do actually seem to be the RPMs for the modules that we do want copied, but I'm not sure that's a good thing. The RPMs should ALWAYS be copied along with their modules, so even if the modules are copied incorrectly, we should see their RPMs also being copied so that at least the repository will always be complete and valid. But that isn't happening.
- Additionally in recursive (non-conservative) mode it's also copying the most recent RPM from any of the modules (4.1.1) in addition to the RPMs it should be copying, which should not be happening
Considering how messed up this is, I think I'm going to start working with QE to get these behaviors more rigorously tested before I do any more stepping through in a debugger. I'm also going to bump the priority and severity.
Updated by dalley over 5 years ago
- Priority changed from Normal to High
- Severity changed from 2. Medium to 3. High
Updated by amacdona@redhat.com over 5 years ago
- Sprint changed from Sprint 53 to Sprint 54
Updated by dalley over 5 years ago
After further investigation I've found that some of the metadata on the pteradactyl RPMs used by this test repo is either invalid, or at least very atypical.
I'm going to try recreating this test repo in a more standard way and see if we still have any issues with Pulp.
Updated by dalley over 5 years ago
Update: The form of "provides" in the pterodactyl test repo is indeed very uncommon, but it is apparently possible and valid. I was able to find a single module in the entire Fedora Module repo against which this bug could be reproduced.
(pulp) [vagrant@pulp2 devel]$ pulp-admin rpm repo copy module --str-eq="name=openmpi" --str-eq="stream=2.1" --from-repo-id=module1 --to-repo-id=test2 --recursive-conservative
This command may be exited via ctrl+c without affecting the request.
[-]
Running...
Copied:
modulemd: 3
rpm: 138
One module stream selected, 3 modules copied. With that said, it's probably OK to knock the priority of this back down.
Updated by dalley over 5 years ago
- Priority changed from High to Normal
- Severity changed from 3. High to 2. Medium
Updated by dalley over 5 years ago
Other bugs, maybe related, maybe not :(
(pulp) [vagrant@pulp2 devel]$ pulp-admin rpm repo copy module --str-eq="name=nodejs" --str-eq="stream=10" --from-repo-id=module1 --to-repo-id=test2 --recursive-conservative
This command may be exited via ctrl+c without affecting the request.
[-]
Running...
Copied:
modulemd:
nodejs-10-3020190424132100-a5b0195c-x86_64
nodejs-12-3020190516000906-a5b0195c-x86_64
rpm:
c-ares-1.15.0-3.module_f30+3988+b4cb7ecc-x86_64
http-parser-2.9.2-1.module_f30+4047+77c83006-x86_64
http-parser-devel-2.9.2-1.module_f30+4047+77c83006-x86_64
libnghttp2-1.38.0-1.module_f30+4047+77c83006-x86_64
libnghttp2-devel-1.38.0-1.module_f30+4047+77c83006-x86_64
libuv-1.28.0-1.module_f30+4047+77c83006-x86_64
libuv-devel-1.28.0-1.module_f30+4047+77c83006-x86_64
libuv-static-1.28.0-1.module_f30+4047+77c83006-x86_64
nghttp2-1.38.0-1.module_f30+4047+77c83006-x86_64
nodejs-10.15.3-2.module_f30+4047+77c83006-x86_64
nodejs-devel-10.15.3-2.module_f30+4047+77c83006-x86_64
nodejs-docs-10.15.3-2.module_f30+4047+77c83006-noarch
nodejs-libs-10.15.3-2.module_f30+4047+77c83006-x86_64
npm-6.4.1-1.10.15.3.2.module_f30+4047+77c83006-x86_64
v8-devel-6.8.275.32-2.module_f30+4047+77c83006-x86_64
Note: NodeJS 10 was copied, only nodejs 10 RPMs were copied, but the nodejs 12 stream module was additionally copied. However there is also a nodejs 11 stream which was not copied. So unlike Pterodactyl, in this instance all of the modulemds were not copied regardless of stream.
Updated by dalley over 5 years ago
- Priority changed from Normal to High
- Severity changed from 2. Medium to 3. High
Now that it's confirmed that these bugs are not confined to that one package, raising the priority again :angry_face:
Updated by bherring over 5 years ago
- Copied to Test #4922: Module Streams not copying correctly with recursive and recursive_conservative added
Updated by dalley over 5 years ago
Updated notes and observations here, it is unweildy to track it all in in the comments: https://etherpad.net/p/pulp-modularity-issues
I'm unsure whether we should split this issue into multiple ones or not because it's difficult to know what the boundaries between the different defects are without knowing exactly why they're happening. There are certainly multiple different problems though.
Updated by dalley over 5 years ago
- Related to Issue #4962: Modules are incorrectly copied when their artifacts shadow the names of non-modular RPM dependencies added
Updated by ttereshc over 5 years ago
- Sprint changed from Sprint 54 to Sprint 55
Updated by bherring over 5 years ago
- Copied to deleted (Test #4922: Module Streams not copying correctly with recursive and recursive_conservative )
Updated by ttereshc over 5 years ago
- Sprint/Milestone deleted (
2.20.0) - Platform Release deleted (
2.20.0)
Updated by dkliban@redhat.com over 5 years ago
- Sprint changed from Sprint 55 to Sprint 56
Updated by dalley over 5 years ago
When using this PR, almost everything previously mentioned is fixed, and all the serious problems previously mentioned are fixed.
https://github.com/pulp/pulp_rpm/pull/1400
The remaining issue is that when copying one stream from pteradactyl, all streams get copied -- but unlike before, all of the artifacts for all streams are copied correctly, so an incomplete repo is not being created in that circumstance like it was previously. Additionally my "in the wild" example of this behavior now works fine.
Given that this example repo is atypical and created by a random tool off github, I think it's safe to de-prioritize for now, as long as you're OK with that Partha. Is this blocking anything?
Updated by bherring over 5 years ago
- Copied to Test #5320: Module Streams not copying correctly with recursive and recursive_conservative added
Updated by dalley over 5 years ago
- Status changed from ASSIGNED to NEW
- Priority changed from High to Low
- Platform Release deleted (
2.21.0)
As far as I can tell, it is just this test fixture that has this problem, I am no longer able to reproduce it with any modules in an actual Fedora release even though I was able to with one of them previously.
As far as bugs go, since it's now copying (exclusively) too much and not too little, we can deprioritize this as it will not result in a broken repo if it does occur.
Updated by rchan about 5 years ago
dalley - You may want to copy https://etherpad.net/p/pulp-modularity-issues to a new location so we don't lose that when etherpad.net is shut down.
Also, the bz associated with this will be marked as still broken, so if this issue is largely addressed, we need to do some updating of the tracker trail to ensure the states of issues reflect the correct be. Do we need to keep this issue open at all?
Updated by ttereshc about 4 years ago
- Status changed from NEW to CLOSED - WONTFIX
Low impact, rare case. Pulp 2 is in maintenance mode.