Project

Profile

Help

Story #4049

closed

As a user, I can know if an RPM is modular or not

Added by ttereshc about 6 years ago. Updated over 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
2.18.0
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Pulp 2
Sprint:
Sprint 44
Quarter:

Description

All modular RPMs have DISTTAG tag (not to be confused with the %{dist} tag) set to the module they have been built for.

>>> headers[rpm.RPMTAG_DISTTAG]
'module(nodejs:10:20180813130636:9edba152)'

Presence of 'module(...)' indicates that an RPM is a modular one.
Detailed info (NSVC) about a module the RPM was built for is not reliable and can't be used to identify the module the RPM belongs to. The NSVC indicates which module that RPM was built for originally, but the same RPM can potentially be used in different modules and the DISTTAG won't be updated.

Pulp needs this information to perform reliable filtering of modular RPMs.
This is required for applicability calculation.
It can also be helpful for:

  • checking module consistency
  • finding modular RPMs
  • upload of modular RPMs if/when Pulp needs to create a reference to a module

Suggested solution:
Add a new field "modular" to the RPM model which will indicate if RPM is modular or not.

In case of on_demand sync, RPM headers can't be analysed, so the "modular" field can be set only by analyzing modules metadata in modules.yaml file. For all its artifacts, "modular" field should be set to True. We will rely on the repodata information provided in the modules.yaml during sync whether policy is immediate or on_demand.

In case of upload, a DISTTAG tag from header can be used as described above.

Migration is needed. "modular" filed can be set by analysing existing modules in Pulp. In case there are modular RPMs in Pulp which don't belong to any module, the "modular" flag would be set incorrectly. At this moment the likelihood of that is low:

  • there are no production bits for modularity content at the moment, only F29 beta content.
  • if module is removed from Pulp, its artifacts/RPMs are removed as well.

Related issues

Related to Pulp - Test #4146: Test is_modular flag in RPM unitsCLOSED - COMPLETEragbalakActions
Actions #1

Updated by ttereshc about 6 years ago

  • Description updated (diff)
Actions #2

Updated by ipanova@redhat.com about 6 years ago

  • Description updated (diff)
Actions #3

Updated by ipanova@redhat.com about 6 years ago

during the migration - do we plan to look just into modules information?in case we do so, then the rest of non-modular rpms will miss the 'modular' field, right?
how do we plan to identify ursine modular rpms? analysing each header for rpm ( which is present on fs) does not seem like a good approach.
we could add a validation later to figure out any ursine content

Actions #4

Updated by ttereshc about 6 years ago

I suggest for migration to rely on modulemd content only and set a new field for every RPM either to True or to False.
We can have false negatives but at the moment the probability is very low + we can always verify or fix it with a validation tool.
So +1 to ignore that problem.
+1 for a validation tool later.
For huge repos with an immediate policy it could be too expensive migration if we look at every RPM on a FS.

Any thoughts from someone else?

Actions #5

Updated by milan about 6 years ago

LGTM;

FWIW I was able to check 150k unit sizes in about 8min with pulp-integrity, checking checksums took about 45 minutes.
Reading the rpm header could take longer indeed than checking the unit sizes; the stats are cached in-memory thru i-nodes but to figure out the flag, each rpm unit header would have to be read and parsed by the rpm lib but will probably take less than the checksums calculation as only the rpm headers need processing. We could still experiment and measure this way of migration, if not in too much hurry, the code might be simpler too (reusing the upload bits).

Actions #6

Updated by ttereshc about 6 years ago

  • Sprint/Milestone set to 2.18.0
Actions #7

Updated by jortel@redhat.com about 6 years ago

Sounds like a reasonable approach/plan to me. The migration should be a best effort using the manifests in the DB. Further migration or correctness should be provided by running a CLI tool at customer request.

Actions #8

Updated by jortel@redhat.com about 6 years ago

  • Groomed changed from No to Yes
  • Sprint Candidate changed from No to Yes
Actions #9

Updated by ttereshc about 6 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ttereshc
  • Sprint set to Sprint 44
Actions #11

Updated by milan about 6 years ago

Just a thought: do we want a new check function (plugin) for pulp-integrity for the migration check?

Added by ttereshc about 6 years ago

Revision 56b7bd07 | View on GitHub

Mark modular RPMs during sync and upload

  • Add new field is_modular for Rpm model
  • Inspect headers to mark RPMs as modular during upload
  • Mark RPMs as modular during sync based on artifacts info in modules.yaml
  • Inspect headers to mark RPMs as modular ones upon successful download
  • Migrate all RPMs, set is_modular to True based on data in Modulemd units

closes #4049 https://pulp.plan.io/issues/4049

Added by ttereshc about 6 years ago

Revision 28f4d726 | View on GitHub

Update progress with a custom number of processed items

re #4049 https://pulp.plan.io/issues/4049

Actions #12

Updated by ttereshc about 6 years ago

Probably not. We no longer can reliably do any checks. Not all modular RPMs are with modular metadata in a repo. Not all modular RPMs have DISTTAG set :( E.g. F28 modular repo.

Actions #13

Updated by ttereshc about 6 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100
Actions #14

Updated by kersom about 6 years ago

  • Related to Test #4146: Test is_modular flag in RPM units added
Actions #15

Updated by ttereshc about 6 years ago

  • Platform Release set to 2.18.0
Actions #16

Updated by ttereshc about 6 years ago

  • Status changed from MODIFIED to 5

Added by kersom about 6 years ago

Revision 82b9e28a | View on GitHub

Add pulp version required to run CheckIsModularFlagTestCase

Add the minimum Pulp version required in order to run is_modular test case. Pulp 2.18.

#4049 https://pulp.plan.io/issues/4049

Actions #17

Updated by ttereshc almost 6 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE
Actions #18

Updated by bmbouter over 5 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF