Project

Profile

Help

Test #4364

Test #4543: Test newly described definitions of recursive and recursive_conservative flags

Implement modularity content dependency solving

Added by bherring 9 months ago. Updated 5 months ago.

Status:
CLOSED - COMPLETE
Priority:
High
Assignee:
Category:
-
Sprint/Milestone:
-
Version:
Platform Release:
Tags:
Pulp 2
Sprint:

Description

Motivation

Pulp lacks the ability to perform dependency solving at a module level and to copy dependent modules and their artifacts during copy operation.

Pulp (@2.18) ignores module dependencies when copying modules recursively.
Pulp doesn't distinguish between modular and non-modular artifacts when recursively copying modules to a repository. Pulp however always copies all module artifacts, including the artifacts rpm dependencies.
A potential problem and a rare case:
- a mixture of modular and non-modular packages with the same NEVRA are present in a repo
- unlucky coincidence with versions
- as a result - a "bad" repo (client will likely have a problem with a module which doesn't have all its modular rpms in a repo, since some of the copied rpms were non-modular ones).

Proposed solution

It's important for modules to have all their artifacts present in a repo.
Regardless of the depsolving strategy for RPMs, all module artifacts and related modules should be copied to a target repo. E.g if the target repository already contains units newer than a module requires, the module-required units need to be copied in the older version, in addition to the already present units.

Modules may depend on other modules, either for the build- or the run-time. Pulp should perform dependency solving based on runtime dependencies only. They are already available on the Modulemd model.

To support these usecases, a fake libsolv solvable can be created for each module unit. This will allow tracking dependencies between the modules. To prevent non-modular content from satisfying dependencies in target repository, it seems exposing the pool->considered bitmap from the libsolv library to its Python binding will be required. This is how DNF deals with solving modular dependencies when installing and upgrading content. This effort is tracked in the sub-task #3741.

References

dkliban_questions (2.3 KB) dkliban QA on modular recursive copy options bherring, 02/18/2019 07:31 PM dkliban_questions
dalley_questions (3.49 KB) dalley QA on modular recursive copy options bherring, 02/18/2019 07:31 PM dalley_questions
dev_baseline_meeting.log (2.53 KB) Baseline questions addressed bherring, 02/19/2019 04:39 PM dev_baseline_meeting.log

Checklist


Related issues

Related to RPM Support - Story #4162: As a user, I have RPM dependency solving when copying modules NEW Actions
Blocked by Pulp - Issue #4405: Pulp 2.19master nightly ci regression at test_modularity.py CLOSED - CURRENTRELEASE Actions
Copied from RPM Support - Story #3740: Implement modularity content dependency solving CLOSED - CURRENTRELEASE Actions

Associated revisions

Revision a06efd7e View on GitHub
Added by ipanova@redhat.com 7 months ago

Update apache confing to return signed schema1 mediatype.

closes #4364
https://pulp.plan.io/issues/4384

(cherry picked from commit 27ae9fc663f6587227479b59addf945fc6ea1e30)

Revision d53734c1 View on GitHub
Added by bherring 7 months ago

Updating CopyModulesTestCase to include latest dependency test

Documentation added from #4371 highlighted that using the `--recursive` option,
the latest RPM dependencies should be copied to he B repo.

Ticket #4543 updates tests with the inclusion of logic from #4371.

Additional cases where `walrus-0.71` RPM is already on B included.

Through multiple permutations of `module_defaults` and `modulemd` with `override_config`,
copying the walrus module will copy the correct streams of the walrus module and dependent RPMs
from A to B.

See:

- https://pulp.plan.io/issues/4371
- https://pulp.plan.io/issues/4543

closes #4364

History

#1 Updated by bherring 9 months ago

  • Copied from Story #3740: Implement modularity content dependency solving added

#3 Updated by bherring 8 months ago

Questions and Answers

recursive and recursive_conservative

  • The recursive/recursive_conservative flags should not affect how modular artifacts are copied, correct? All module + dependencies for all versions should be copied always, right?
    • not specifying 'recursive' should copy the module and all of it's artifacts
      [13:47:25] <dkliban>when --recursive is specified the module, it's artifacts, and it's modular dependencies their artifacts should be copied

it's possible for modular RPMs to have non-modular dependencies

  • Yes.

Notes

Should Test Items, Basic:

  • With modular only for basic tests
  • With Mixed-repo for basic tests (same results as modular)
  • with recursive False (only the module specified)
  • with recursive True (module + modular deps)
  • With recursive_conservative == Either, should not change either above.
  • Same, but with Mixed-repo (unsure if the non-modular units would be copied as well -- think so)

Should Test Edge Cases:

  • Mixed-repo, NEVRA over-lap and copy content modules
    • Verify what and how many are copied
    • Believe the same as above, however initial setup has additional over-lapping NEVRA versions on A to copy to B

#4 Updated by bherring 8 months ago

  • Checklist item deleted (associating a module content unit supports the recursive behavior)
  • Checklist item deleted (associating a module always considers all installation profiles)
  • Checklist item deleted (associating a module considers other modular dependencies)
  • Checklist item deleted (file a test issue to automate recursive copy of modules and their dependencies)
  • Checklist item create purely modular dep fixture added
  • Checklist item create modular<-->modular dep fixture where modular dep ALSO has RPM deps added
  • Checklist item update test_rich_weak_deps with new assumptions and test criteria added
  • File dalley_questions dalley_questions added
  • File dkliban_questions dkliban_questions added

Notes

Added the Q/A with 2 of 3 of devs on the proper operation of modular repos (fully modular) without RPM deps on Modules.

Tanya was not up-to-speed or available at the time the questions were broached.

Note 7 above is really the questions that came out of that discussion.

#4371 is the ticket where we are asking for clarification and verification on:
  • Which test-fixtures are interesting
    • rpm-with-modularity
    • rpm-pure-modular
    • modular-with-modular-deps-that-have-RPM-deps

From there, all the behavioral questions should be addressed.

At the time of this writing, I am assuming that EVERYTHING should always be copied. However that sounds strange. I can currently not test that hypothesis.

Working on:
  • updating the test_rich_weak_dependencies.py to address everything I believe should be there
  • Creating test-fixtures that should create the 2 additional scenarios required for test

#5 Updated by bherring 8 months ago

  • Related to Story #4162: As a user, I have RPM dependency solving when copying modules added

#6 Updated by bherring 8 months ago

Update

Had a meeting, with Daniel, Dennis, Tanya and Kersom about questions related to testing

Main output was:
  • review the fixtures we have
    • Ensure that there are at least two levels of deps of modular artifacts and rpms to ensure that the correct recursive actions are taking place for both RPM and MODULAR artifacts
  • work around provided to get 2.19 installed and functioning

Actions

  • Review rpm-with-modules to see if this is rich enough for our case
  • Review test_copy, test_rich_weak_dependencies, test_modularity to ensure all permutations are covered
  • Get a functioning 2.19 system
  • Update any test logic to include missing permutation tests.

#7 Updated by bherring 8 months ago

Other notes

Other
Any package in an enabled module takes priority over any ursine RPM with the same package name, regardless of which repositories those modules and packages may be in.

If there’s an ursine or modular RPM with a higher NVR than the hotfix one, the ursine or modular one wins. This is to ensure new proper updates override the temporary hotfix RPMs.

-- https://docs.fedoraproject.org/en-US/modularity/architecture/consuming/dnf-behavior/

#8 Updated by kersom 8 months ago

  • Assignee set to bherring

#9 Updated by ipanova@redhat.com 8 months ago

  • Status changed from NEW to MODIFIED

#10 Updated by ipanova@redhat.com 8 months ago

  • Status changed from MODIFIED to NEW

#11 Updated by bherring 8 months ago

On hold.

I think there is already a useful dependency chain:

elephant (RPM) --> duck (modular deps) --> walrus (modular deps) --> whale (RPM)

Can't test this with the current libsolv issue, easily that is linked as blocked above #4405

Going to block for equally important tickets while waiting for the package solve.

#12 Updated by bherring 8 months ago

  • Blocked by Issue #4405: Pulp 2.19master nightly ci regression at test_modularity.py added

#13 Updated by ipanova@redhat.com 8 months ago

  • Status changed from NEW to MODIFIED

#14 Updated by bherring 7 months ago

  • Related to Test #4543: Test newly described definitions of recursive and recursive_conservative flags added

#15 Updated by bherring 7 months ago

  • Checklist item create purely modular dep fixture set to Done
  • Checklist item create modular<-->modular dep fixture where modular dep ALSO has RPM deps set to Done

#16 Updated by bherring 7 months ago

  • Status changed from MODIFIED to ASSIGNED

#17 Updated by bherring 7 months ago

  • Priority changed from Normal to High

#18 Updated by bherring 7 months ago

  • Related to deleted (Test #4543: Test newly described definitions of recursive and recursive_conservative flags)

#19 Updated by bherring 7 months ago

  • Parent task set to #4543

#20 Updated by bherring 7 months ago

Notes

Moved update test_rich_weak_deps with new assumptions and test criteria into its own ticket under the Parent.

#21 Updated by bherring 7 months ago

  • Checklist item deleted (update test_rich_weak_deps with new assumptions and test criteria)

#22 Updated by ipanova@redhat.com 7 months ago

  • Status changed from ASSIGNED to MODIFIED

#23 Updated by bherring 7 months ago

  • Sprint/Milestone set to 2.19.0

#24 Updated by bherring 7 months ago

  • Sprint/Milestone deleted (2.19.0)

#26 Updated by bmbouter 6 months ago

  • Tags Pulp 2 added

#27 Updated by bherring 5 months ago

  • Status changed from MODIFIED to CLOSED - COMPLETE

Please register to edit this issue

Also available in: Atom PDF