Project

Profile

Help

Test #4364

closed

Test #4543: Test newly described definitions of recursive and recursive_conservative flags

Implement modularity content dependency solving

Added by bherring about 5 years ago. Updated almost 5 years ago.

Status:
CLOSED - COMPLETE
Priority:
High
Assignee:
Sprint/Milestone:
-
Version:
Platform Release:
Tags:
Pulp 2
Sprint:
Quarter:

Description

Motivation

Pulp lacks the ability to perform dependency solving at a module level and to copy dependent modules and their artifacts during copy operation.

Pulp (@2.18) ignores module dependencies when copying modules recursively.
Pulp doesn't distinguish between modular and non-modular artifacts when recursively copying modules to a repository. Pulp however always copies all module artifacts, including the artifacts rpm dependencies.
A potential problem and a rare case:
- a mixture of modular and non-modular packages with the same NEVRA are present in a repo
- unlucky coincidence with versions
- as a result - a "bad" repo (client will likely have a problem with a module which doesn't have all its modular rpms in a repo, since some of the copied rpms were non-modular ones).

Proposed solution

It's important for modules to have all their artifacts present in a repo.
Regardless of the depsolving strategy for RPMs, all module artifacts and related modules should be copied to a target repo. E.g if the target repository already contains units newer than a module requires, the module-required units need to be copied in the older version, in addition to the already present units.

Modules may depend on other modules, either for the build- or the run-time. Pulp should perform dependency solving based on runtime dependencies only. They are already available on the Modulemd model.

To support these usecases, a fake libsolv solvable can be created for each module unit. This will allow tracking dependencies between the modules. To prevent non-modular content from satisfying dependencies in target repository, it seems exposing the pool->considered bitmap from the libsolv library to its Python binding will be required. This is how DNF deals with solving modular dependencies when installing and upgrading content. This effort is tracked in the sub-task #3741.

References


Files

dkliban_questions (2.3 KB) dkliban_questions dkliban QA on modular recursive copy options bherring, 02/18/2019 07:31 PM
dalley_questions (3.49 KB) dalley_questions dalley QA on modular recursive copy options bherring, 02/18/2019 07:31 PM
dev_baseline_meeting.log (2.53 KB) dev_baseline_meeting.log Baseline questions addressed bherring, 02/19/2019 04:39 PM

Related issues

Related to RPM Support - Story #4162: As a user, I have dependency solving when copying modulesCLOSED - CURRENTRELEASEdalley

Actions
Blocked by Pulp - Issue #4405: Pulp 2.19master nightly ci regression at test_modularity.pyCLOSED - CURRENTRELEASEpcreechActions
Copied from RPM Support - Story #3740: Implement modularity content dependency solvingCLOSED - CURRENTRELEASEmilan

Actions
Actions #1

Updated by bherring about 5 years ago

  • Copied from Story #3740: Implement modularity content dependency solving added
Actions #3

Updated by bherring about 5 years ago

Questions and Answers

recursive and recursive_conservative

  • The recursive/recursive_conservative flags should not affect how modular artifacts are copied, correct? All module + dependencies for all versions should be copied always, right?
    • not specifying 'recursive' should copy the module and all of it's artifacts
      [13:47:25] <dkliban>when --recursive is specified the module, it's artifacts, and it's modular dependencies their artifacts should be copied

it's possible for modular RPMs to have non-modular dependencies

  • Yes.

Notes

Should Test Items, Basic:

  • With modular only for basic tests
  • With Mixed-repo for basic tests (same results as modular)
  • with recursive == False (only the module specified)
  • with recursive == True (module + modular deps)
  • With recursive_conservative == Either, should not change either above.
  • Same, but with Mixed-repo (unsure if the non-modular units would be copied as well -- think so)

Should Test Edge Cases:

  • Mixed-repo, NEVRA over-lap and copy content modules
    • Verify what and how many are copied
    • Believe the same as above, however initial setup has additional over-lapping NEVRA versions on A to copy to B
Actions #4

Updated by bherring about 5 years ago

Notes

Added the Q/A with 2 of 3 of devs on the proper operation of modular repos (fully modular) without RPM deps on Modules.

Tanya was not up-to-speed or available at the time the questions were broached.

Note 7 above is really the questions that came out of that discussion.

#4371 is the ticket where we are asking for clarification and verification on:

  • Which test-fixtures are interesting
    • rpm-with-modularity
    • rpm-pure-modular
    • modular-with-modular-deps-that-have-RPM-deps

From there, all the behavioral questions should be addressed.

At the time of this writing, I am assuming that EVERYTHING should always be copied. However that sounds strange. I can currently not test that hypothesis.

Working on:

  • updating the test_rich_weak_dependencies.py to address everything I believe should be there
  • Creating test-fixtures that should create the 2 additional scenarios required for test
Actions #5

Updated by bherring about 5 years ago

  • Related to Story #4162: As a user, I have dependency solving when copying modules added
Actions #6

Updated by bherring about 5 years ago

Update

Had a meeting, with Daniel, Dennis, Tanya and Kersom about questions related to testing

Main output was:

  • review the fixtures we have
    • Ensure that there are at least two levels of deps of modular artifacts and rpms to ensure that the correct recursive actions are taking place for both RPM and MODULAR artifacts
  • work around provided to get 2.19 installed and functioning

Actions

  • Review rpm-with-modules to see if this is rich enough for our case
  • Review test_copy, test_rich_weak_dependencies, test_modularity to ensure all permutations are covered
  • Get a functioning 2.19 system
  • Update any test logic to include missing permutation tests.
Actions #7

Updated by bherring about 5 years ago

Other notes

Other
Any package in an enabled module takes priority over any ursine RPM with the same package name, regardless of which repositories those modules and packages may be in.

If there’s an ursine or modular RPM with a higher NVR than the hotfix one, the ursine or modular one wins. This is to ensure new proper updates override the temporary hotfix RPMs.

-- https://docs.fedoraproject.org/en-US/modularity/architecture/consuming/dnf-behavior/

Actions #8

Updated by kersom about 5 years ago

  • Assignee set to bherring
Actions #9

Updated by ipanova@redhat.com about 5 years ago

  • Status changed from NEW to MODIFIED
Actions #10

Updated by ipanova@redhat.com about 5 years ago

  • Status changed from MODIFIED to NEW
Actions #11

Updated by bherring about 5 years ago

On hold.

I think there is already a useful dependency chain:

elephant (RPM) --> duck (modular deps) --> walrus (modular deps) --> whale (RPM)

Can't test this with the current libsolv issue, easily that is linked as blocked above #4405

Going to block for equally important tickets while waiting for the package solve.

Actions #12

Updated by bherring about 5 years ago

  • Blocked by Issue #4405: Pulp 2.19master nightly ci regression at test_modularity.py added
Actions #13

Updated by ipanova@redhat.com about 5 years ago

  • Status changed from NEW to MODIFIED
Actions #14

Updated by bherring about 5 years ago

  • Related to Test #4543: Test newly described definitions of recursive and recursive_conservative flags added
Actions #15

Updated by bherring about 5 years ago

Actions #16

Updated by bherring about 5 years ago

  • Status changed from MODIFIED to ASSIGNED
Actions #17

Updated by bherring about 5 years ago

  • Priority changed from Normal to High
Actions #18

Updated by bherring about 5 years ago

  • Related to deleted (Test #4543: Test newly described definitions of recursive and recursive_conservative flags)
Actions #19

Updated by bherring about 5 years ago

  • Parent issue set to #4543
Actions #20

Updated by bherring about 5 years ago

Notes

Moved update test_rich_weak_deps with new assumptions and test criteria into its own ticket under the Parent.

Actions #21

Updated by bherring about 5 years ago

Added by ipanova@redhat.com almost 5 years ago

Revision a06efd7e | View on GitHub

Update apache confing to return signed schema1 mediatype.

closes #4364 https://pulp.plan.io/issues/4384

(cherry picked from commit 27ae9fc663f6587227479b59addf945fc6ea1e30)

Actions #22

Updated by ipanova@redhat.com almost 5 years ago

  • Status changed from ASSIGNED to MODIFIED
Actions #23

Updated by bherring almost 5 years ago

  • Sprint/Milestone set to 2.19.0
Actions #24

Updated by bherring almost 5 years ago

  • Sprint/Milestone deleted (2.19.0)

Added by bherring almost 5 years ago

Revision d53734c1 | View on GitHub

Updating CopyModulesTestCase to include latest dependency test

Documentation added from #4371 highlighted that using the --recursive option, the latest RPM dependencies should be copied to he B repo.

Ticket #4543 updates tests with the inclusion of logic from #4371.

Additional cases where walrus-0.71 RPM is already on B included.

Through multiple permutations of module_defaults and modulemd with override_config, copying the walrus module will copy the correct streams of the walrus module and dependent RPMs from A to B.

See:

closes #4364

Actions #26

Updated by bmbouter almost 5 years ago

  • Tags Pulp 2 added
Actions #27

Updated by bherring almost 5 years ago

  • Status changed from MODIFIED to CLOSED - COMPLETE

Also available in: Atom PDF