Test #4364
closed
Test #4543: Test newly described definitions of recursive and recursive_conservative flags
Implement modularity content dependency solving
Description
Motivation¶
Pulp lacks the ability to perform dependency solving at a module level and to copy dependent modules and their artifacts during copy operation.
Pulp (@2.18) ignores module dependencies when copying modules recursively.
Pulp doesn't distinguish between modular and non-modular artifacts when recursively copying modules to a repository. Pulp however always copies all module artifacts, including the artifacts rpm dependencies.
A potential problem and a rare case:
- a mixture of modular and non-modular packages with the same NEVRA are present in a repo
- unlucky coincidence with versions
- as a result - a "bad" repo (client will likely have a problem with a module which doesn't have all its modular rpms in a repo, since some of the copied rpms were non-modular ones).
Proposed solution¶
It's important for modules to have all their artifacts present in a repo.
Regardless of the depsolving strategy for RPMs, all module artifacts and related modules should be copied to a target repo. E.g if the target repository already contains units newer than a module requires, the module-required units need to be copied in the older version, in addition to the already present units.
Modules may depend on other modules, either for the build- or the run-time. Pulp should perform dependency solving based on runtime dependencies only. They are already available on the Modulemd model.
To support these usecases, a fake libsolv solvable can be created for each module unit. This will allow tracking dependencies between the modules. To prevent non-modular content from satisfying dependencies in target repository, it seems exposing the pool->considered bitmap from the libsolv library to its Python binding will be required. This is how DNF deals with solving modular dependencies when installing and upgrading content. This effort is tracked in the sub-task #3741.
References¶
- the base pulp_modularity support tracker; Issue #3206
- the Modularity project docs page
- the depsolving library investigation #3528 POC
- libsolv
- modular errata
- A cross-team, modular-depsolving document
Files
Related issues
Updated by bherring almost 6 years ago
- Copied from Story #3740: Implement modularity content dependency solving added
Updated by bherring almost 6 years ago
Notes¶
Additional References:
- https://github.com/PulpQE/pulp-fixtures/blob/master/rpm/modules-updateinfo.patch
- https://github.com/PulpQE/Pulp-2-Tests/issues/152
- https://fedoramagazine.org/modularity-fedora-28-server-edition/
- https://help.sonatype.com/repomanager2/rpm-packages-and-yum-repositories
- https://docs.fedoraproject.org/en-US/modularity/using-modules/
- https://fedoramagazine.org/working-modules-fedora-28/
- https://docs.pagure.org/modularity/
- https://github.com/pulp/pulp_rpm/pull/1237/files
- https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-with-modules/
- https://docs.fedoraproject.org/en-US/modularity/
- https://docs.fedoraproject.org/en-US/modularity/making-modules/defining-modules/#_advanced_dependencies_optional
- https://fedoramagazine.org/fedora-classroom-session-fedora-modularity-101/
- https://computingforgeeks.com/how-to-use-fedora-29-modular-repository/
- https://docs.fedoraproject.org/en-US/modularity/making-modules/defining-modules/
- https://communityblog.fedoraproject.org/modularity-dead-long-live-modularity/
- https://docs.fedoraproject.org/en-US/modularity/using-modules/
Updated by bherring almost 6 years ago
Questions and Answers¶
recursive and recursive_conservative¶
- The recursive/recursive_conservative flags should not affect how modular artifacts are copied, correct? All module + dependencies for all versions should be copied always, right?
- not specifying 'recursive' should copy the module and all of it's artifacts
[13:47:25] <dkliban>when --recursive is specified the module, it's artifacts, and it's modular dependencies their artifacts should be copied
- not specifying 'recursive' should copy the module and all of it's artifacts
it's possible for modular RPMs to have non-modular dependencies¶
- Yes.
Notes¶
Should Test Items, Basic:¶
- With modular only for basic tests
- With Mixed-repo for basic tests (same results as modular)
- with recursive == False (only the module specified)
- with recursive == True (module + modular deps)
- With recursive_conservative == Either, should not change either above.
- Same, but with Mixed-repo (unsure if the non-modular units would be copied as well -- think so)
Should Test Edge Cases:¶
- Mixed-repo, NEVRA over-lap and copy content modules
- Verify what and how many are copied
- Believe the same as above, however initial setup has additional over-lapping NEVRA versions on A to copy to B
Updated by bherring almost 6 years ago
- File dalley_questions dalley_questions added
- File dkliban_questions dkliban_questions added
Notes¶
Added the Q/A with 2 of 3 of devs on the proper operation of modular repos (fully modular) without RPM deps on Modules.
Tanya was not up-to-speed or available at the time the questions were broached.
Note 7 above is really the questions that came out of that discussion.
#4371 is the ticket where we are asking for clarification and verification on:
- Which test-fixtures are interesting
- rpm-with-modularity
- rpm-pure-modular
- modular-with-modular-deps-that-have-RPM-deps
From there, all the behavioral questions should be addressed.
At the time of this writing, I am assuming that EVERYTHING should always be copied. However that sounds strange. I can currently not test that hypothesis.
Working on:
- updating the test_rich_weak_dependencies.py to address everything I believe should be there
- Creating test-fixtures that should create the 2 additional scenarios required for test
Updated by bherring almost 6 years ago
- Related to Story #4162: As a user, I have dependency solving when copying modules added
Updated by bherring almost 6 years ago
- File dev_baseline_meeting.log dev_baseline_meeting.log added
Update¶
Had a meeting, with Daniel, Dennis, Tanya and Kersom about questions related to testing
Main output was:
- review the fixtures we have
- Ensure that there are at least two levels of deps of modular artifacts and rpms to ensure that the correct recursive actions are taking place for both RPM and MODULAR artifacts
- work around provided to get 2.19 installed and functioning
Actions¶
- Review rpm-with-modules to see if this is rich enough for our case
- Review test_copy, test_rich_weak_dependencies, test_modularity to ensure all permutations are covered
- Get a functioning 2.19 system
- Update any test logic to include missing permutation tests.
Updated by bherring almost 6 years ago
Other notes¶
Other
Any package in an enabled module takes priority over any ursine RPM with the same package name, regardless of which repositories those modules and packages may be in.
If there’s an ursine or modular RPM with a higher NVR than the hotfix one, the ursine or modular one wins. This is to ensure new proper updates override the temporary hotfix RPMs.
-- https://docs.fedoraproject.org/en-US/modularity/architecture/consuming/dnf-behavior/
Updated by ipanova@redhat.com almost 6 years ago
- Status changed from NEW to MODIFIED
Applied in changeset pulp_docker:pulp_docker|27ae9fc663f6587227479b59addf945fc6ea1e30.
Updated by ipanova@redhat.com over 5 years ago
- Status changed from MODIFIED to NEW
Updated by bherring over 5 years ago
On hold.¶
I think there is already a useful dependency chain:
elephant (RPM) --> duck (modular deps) --> walrus (modular deps) --> whale (RPM)
Can't test this with the current libsolv issue, easily that is linked as blocked above #4405
Going to block for equally important tickets while waiting for the package solve.
Updated by bherring over 5 years ago
- Blocked by Issue #4405: Pulp 2.19master nightly ci regression at test_modularity.py added
Updated by ipanova@redhat.com over 5 years ago
- Status changed from NEW to MODIFIED
Applied in changeset pulp_docker:27ae9fc663f6587227479b59addf945fc6ea1e30.
Updated by bherring over 5 years ago
- Related to Test #4543: Test newly described definitions of recursive and recursive_conservative flags added
Updated by bherring over 5 years ago
- Related to deleted (Test #4543: Test newly described definitions of recursive and recursive_conservative flags)
Updated by bherring over 5 years ago
Notes¶
Moved update test_rich_weak_deps with new assumptions and test criteria into its own ticket under the Parent.
Added by ipanova@redhat.com over 5 years ago
Updated by ipanova@redhat.com over 5 years ago
- Status changed from ASSIGNED to MODIFIED
Applied in changeset pulp_docker:pulp_docker|a06efd7eb05f6c3d1662531714779e64640aa504.
Added by bherring over 5 years ago
Revision d53734c1 | View on GitHub
Updating CopyModulesTestCase to include latest dependency test
Documentation added from #4371 highlighted that using the --recursive option,
the latest RPM dependencies should be copied to he B repo.
Ticket #4543 updates tests with the inclusion of logic from #4371.
Additional cases where walrus-0.71 RPM is already on B included.
Through multiple permutations of module_defaults and modulemd with override_config,
copying the walrus module will copy the correct streams of the walrus module and dependent RPMs
from A to B.
See:
closes #4364
Updated by bherring over 5 years ago
Applied in changeset pulp:pulp-2-tests|d53734c142edf6c0dbda8bd085e0967d799cd9c2.
Updated by bherring over 5 years ago
- Status changed from MODIFIED to CLOSED - COMPLETE
.
Update apache confing to return signed schema1 mediatype.
closes #4364 https://pulp.plan.io/issues/4384
(cherry picked from commit 27ae9fc663f6587227479b59addf945fc6ea1e30)