Project

Profile

Help

Story #3919

closed

As a user, I can manage modular Errata content

Added by ttereshc over 4 years ago. Updated almost 4 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Estimated time:
(Total: 0:00 h)
Platform Release:
2.18.0
Groomed:
Yes
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 43
Quarter:

Description

If an erratum is released for modular content, it will contain information about a module the RPMs are relevant to.
Information about a module is added to a collection in a pkglist.
Each collection corresponds to one module or to non-modular content.
There can be multiple collections in one pkglist (this is just for information, no changes here, it's this way now).
The module element is optional. Absence of the module element means that the collection is for normal non-modular RPMs and should be processed as usual.

Example of the modular errata content can be found in updateinfo.xml in the fixtures repo:
https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-with-modules/

Here is the snippet of the new format:

<pkglist>
    <collection short="">
        <name>coll_name1</name>
        <module name="kangaroo" stream="0" version="20180730223407" context="deadbeef" arch="noarch"/>
        <package arch="noarch" name="kangaroo" release="1" src="http://www.fedoraproject.org" version="0.3">
            <filename>kangaroo-0.3-1.noarch.rpm</filename>
        </package>
        <package ... >
             ...
         </package>
    </collection>
    <collection short="">
        <name>coll_name2</name>
        ...
    </collection>
</pkglist>

For upload, JSON is usually used, so the format will be:

{   
    "issued": "2015-03-05 05:42:53 UTC",
    "pkglist": [{
                   "name": "coll_name1",
                   "module": {
                                 "name": "duck",
                                 "stream": "0",
                                 "version": "20180730233102",
                                 "context": "deadbeef",
                                 "arch": "noarch"
                             },
                    "packages": [{
                                    "arch": "noarch",
                                    "filename": "duck-0.7-1.noarch.rpm",
                                    "name": "duck",
                                    "release": "1",
                                    "sum": ["sha256", "ceb0f0bb58be244393cc565e8ee5ef0ad36884d8ba8eec74542ff47d299a34c1"],
                                    "version": "0.7",
                                }, { 
                                    ...
                                }],
                },
                {
                   "name": "coll_name2",
                   ...

                }],
        ...
}

Deliverables

  • Sync should recognize module element in the erratum and save it on the model
  • no changes to the model itself are needed (aka migration is not needed) due to changes in the nested structure, inside a pkglist.
  • Publish should respect the module element and publish info about the module in the errata if this info is available
  • see the fixture repo
  • publish should filter by module being available in the repo; modular errata rpms not attached to a module present in the repo should be excluded from publishing
  • publish should continue filtering by rpms being available in the repo
  • if no rpms are found for/associated with an erratum but module information is specified, such erratum has to be published. It's possible that just module metadata changed and no rpms has been updated, e.g an rpm was removed from the artifacts to fix the module issue.
  • document that filtering of errata being published is performed according to rpms and modules present in a repo
  • make sure collection names are unique per erratum (Pulp takes care of that and regenerates collection names to make them unique. I think at the moment coll_name = repo_id + idx, the suggestion is to make it, repo_id + module info + idx)
  • Upload should support the module element
  • see the JSON format in the description above
  • Copy
  • simple copy is expected to just work
  • copy with --recursive flag should respect the module info in the erratum
  • a whole module should be copied in this case: all its RPMs, module itself but related module_defaults should not be copied.
  • Search API
  • probably no changes needed, it should just work

Out of scope: Applicability (it will be implemented as a separate story).

Note

Mind that the module and rpms listed in the erratum are the fixed ones (the ones present in the repo)


Subtasks 2 (0 open2 closed)

Story #3959: Modular errata sync and publishCLOSED - CURRENTRELEASEmilan

Actions
Issue #4002: pulp-admin doesn't support uploading modular errataCLOSED - WONTFIXActions
Actions #2

Updated by ttereshc over 4 years ago

  • Description updated (diff)
Actions #3

Updated by ttereshc over 4 years ago

  • Description updated (diff)
Actions #4

Updated by ttereshc over 4 years ago

  • Description updated (diff)
Actions #5

Updated by milan over 4 years ago

  • Description updated (diff)
Actions #6

Updated by milan over 4 years ago

  • Description updated (diff)
Actions #7

Updated by dkliban@redhat.com over 4 years ago

  • Sprint/Milestone set to 2.18.0
Actions #8

Updated by daviddavis over 4 years ago

  • Groomed changed from No to Yes
  • Sprint set to Sprint 41
Actions #9

Updated by ipanova@redhat.com over 4 years ago

from publish part:

publish should continue filtering by rpms being available in the repo
publish should filter by module being available in the repo

I suggest we explicitly first filter by module being available in the repo and based on result:
a) positive, continue with filtering by rpms being available in the repo
b) negative, to do not publish the collection at all( module name + packages)and do not proceed with filtering by rpms avaialable in the repo. We do not want to end up to publish in updateinfo.xml module which is not present in the repo, but for some reason rpms are present in the repo)

Actions #10

Updated by rchan over 4 years ago

  • Sprint changed from Sprint 41 to Sprint 42
Actions #11

Updated by milan over 4 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to milan
Actions #12

Updated by milan over 4 years ago

  • Description updated (diff)

wrote:

from publish part:
[...]

I suggest we explicitly first filter by module being available in the repo and based on result:
a) positive, continue with filtering by rpms being available in the repo
b) negative, to do not publish the collection at all( module name + packages)and do not proceed with filtering by rpms avaialable in the repo. We do not want to end up to publish in updateinfo.xml module which is not present in the repo, but for some reason rpms are present in the repo)

+1 makes sense; I'm updating the description accordingly

  • if no rpms are found for/associated with an erratum but module information is specified, such erratum has to be published. It's possible that just module metadata changed and no rpms has been updated, e.g an rpm was removed from the artifacts to fix the module issue.

I guess we need to distinguish between a situation where all the RPMs were filtered out by the (module) repo presence and a genuine, metadata-only modular erratum; a consequence of neglect in the former case would be an unwanted, "empty" update on the consumer side.
The genuine, metadata-only erratum of a module might have to, in case of (modular) dependency updates, pull-in additional content into the repository to avoid dependency issues on the consumer side.

Probably we could address both these concerns by calculating implied dependencies of modular errata:

  • publish should filter by module being available in the repo; modular errata rpms not attached to a module present in the repo should be excluded from publishing
  • publish should continue filtering by rpms being available in the repo
  • skip the module erratum publish altogether if there are unsatisfied (modular) dependencies in the repository
  • if no rpms are found for/associated with an erratum but module information is specified, such erratum has to be published. It's possible that just module metadata changed and no rpms has been updated, e.g an rpm was removed from the artifacts to fix the module issue.
Actions #13

Updated by ipanova@redhat.com over 4 years ago

i think whenever erratum provides some changes/updates it would provide the updated content as well. So if there is some dependency change in the module metadata i would expect this dependency provided in the erratum. do we check on the erratum integrity or we rely on the ET correctness?

Actions #14

Updated by milan over 4 years ago

wrote:

i think whenever erratum provides some changes/updates it would provide the updated content as well. So if there is some dependency change in the module metadata i would expect this dependency provided in the erratum. do we check on the erratum integrity or we rely on the ET correctness?

I'm afraid it is also about the case when we filter all the rpms from an erratum so we make it metadata-only even tho it wasn't supposed to be.
Anyways, I guess I'm being too cautious about the edge cases here; we can always bugfix later or wave in case nobody reports issues with that.

Actions #15

Updated by milan over 4 years ago

I just realized we might have to store the relation to the modulemd affected by the erratum; once parsed, the <module name=".." /> tag is lost and since we don't store the original erratum XML (because of the publish-time adjustments), we need to record the erratum--module relationship in the DB otherwise we won't be able to publish it later. I guess the most appropriate model is ErratumPkgList and I suppose the field should be a foreign key linking back to the Modulemd collection

Thoughts?

Actions #16

Updated by milan over 4 years ago

milan wrote:

I just realized we might have to store the relation to the modulemd affected by the erratum; once parsed, the <module name=".." /> tag is lost and since we don't store the original erratum XML (because of the publish-time adjustments), we need to record the erratum--module relationship in the DB otherwise we won't be able to publish it later. I guess the most appropriate model is ErratumPkgList and I suppose the field should be a foreign key linking back to the Modulemd collection

Thoughts?

Actually, I'm wrong \o/ :)

db.erratum_pkglists.find({'repo_id': 'modulerr'})[4]
{
        "_id" : ObjectId("5b87f4c030f252463b42b28a"),
        "errata_id" : "RHEA-2012:0059",
        "repo_id" : "modulerr",
        "collections" : [
                {
                        "packages" : [
                                {
                                        "src" : "http://www.fedoraproject.org",
                                        "name" : "kangaroo",
                                        "sum" : null,
                                        "filename" : "kangaroo-0.3-1.noarch.rpm",
                                        "epoch" : null,
                                        "version" : "0.3",
                                        "release" : "1",
                                        "arch" : "noarch"
                                }
                        ],
                        "name" : "coll_name1",
                        "module" : {
                                "stream" : "0",
                                "version" : "20180730223407",
                                "arch" : "noarch",
                                "name" : "kangaroo",
                                "context" : "deadbeef"
                        },
                        "short" : ""
                },
                // -------->%-------------
            ]
}
Actions #17

Updated by milan over 4 years ago

  • Status changed from ASSIGNED to POST
Actions #18

Updated by milan over 4 years ago

  • Status changed from POST to NEW
  • Assignee deleted (milan)
Actions #19

Updated by dkliban@redhat.com over 4 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to milan
Actions #20

Updated by milan over 4 years ago

Actions #21

Updated by milan over 4 years ago

The upload works out of the box with the sync&publish PR applied.

Actions #22

Updated by milan over 4 years ago

Just checked that the simple copy works OK

[vagrant@pulp2 pulp_rpm]$ http POST https://admin:admin@pulp2.dev/pulp/api/v2/repositor
ies/modulerr_copy/actions/associate/ source_repo_id=modulerr criteria:='{"type_ids": ["
erratum"], "filters": {"unit": {"errata_id": "RHEA-2018:0042"}}}'
HTTP/1.1 202 Accepted
Connection: Keep-Alive
Content-Length: 172
Content-Type: application/json; charset=utf-8
Date: Mon, 10 Sep 2018 15:30:43 GMT
Keep-Alive: timeout=5, max=10000
Server: Apache/2.4.33 (Fedora) OpenSSL/1.1.0h-fips mod_wsgi/4.5.15 Python/2.7

{
    "error": null,
    "result": null,
    "spawned_tasks": [
        {
            "_href": "/pulp/api/v2/tasks/e9e5b53c-ce69-465a-9f24-1a236964ca21/",
            "task_id": "e9e5b53c-ce69-465a-9f24-1a236964ca21"
        }
    ]
}

[vagrant@pulp2 pulp_rpm]$ http https://admin:admin@pulp2.dev/pulp/api/v2/tasks/e9e5b53c-ce69-465a-9f24-1a236964ca21/
HTTP/1.1 200 OK
Connection: Keep-Alive
Content-Length: 832
Content-Type: application/json; charset=utf-8
Date: Mon, 10 Sep 2018 15:31:01 GMT
ETag: "c75cc3612b1cbf4073d5602e0eab95ef"
Keep-Alive: timeout=5, max=10000
Server: Apache/2.4.33 (Fedora) OpenSSL/1.1.0h-fips mod_wsgi/4.5.15 Python/2.7

{
    "_href": "/pulp/api/v2/tasks/e9e5b53c-ce69-465a-9f24-1a236964ca21/",
    "_id": {
        "$oid": "5b968e237f513442b097f791"
    },
    "_ns": "task_status",
    "error": null,
    "exception": null,
    "finish_time": "2018-09-10T15:30:43Z",
    "id": "5b968e237f513442b097f791",
    "progress_report": {},
    "queue": "reserved_resource_worker-4@pulp2.dev.dq2",
    "result": {
        "units_failed_signature_filter": [],
        "units_successful": [
            {
                "type_id": "erratum",
                "unit_key": {
                    "id": "RHEA-2018:0042"
                }
            }
        ]
    },
    "spawned_tasks": [],
    "start_time": "2018-09-10T15:30:43Z",
    "state": "finished",
    "tags": [
        "pulp:repository:modulerr_copy",
        "pulp:repository:modulerr",
        "pulp:action:associate"
    ],
    "task_id": "e9e5b53c-ce69-465a-9f24-1a236964ca21",
    "task_type": "pulp.server.managers.repo.unit_association.associate_from_repo",
    "traceback": null,
    "worker_name": "reserved_resource_worker-4@pulp2.dev"
}

[vagrant@pulp2 pulp_rpm]$

Double checking Mongo:

> db.repo_content_units.findOne({"repo_id": "modulerr_copy", "unit_type_id": "erratum"})
{
        "_id" : ObjectId("5b966c0c7f513442b096c9e7"),
        "repo_id" : "modulerr_copy",
        "unit_id" : "01f04ca4-0788-4e1c-84e8-9b2f21efab1f",
        "unit_type_id" : "erratum",
        "updated" : "2018-09-10T15:39:05Z",
        "created" : "2018-09-10T13:05:16Z"
}
> db.units_erratum.findOne({"_id": "01f04ca4-0788-4e1c-84e8-9b2f21efab1f"})
{
        "_id" : "01f04ca4-0788-4e1c-84e8-9b2f21efab1f",
        "pulp_user_metadata" : {

        },
        "_last_updated" : 1536332456,
        "errata_id" : "RHEA-2018:0042",
        "status" : "stable",
        "updated" : "2018-09-05 11:23:42",
        "description" : "Walrus Erratum; metadata-only",
        "issued" : "2002-02-20 00:22:00",
        "pushcount" : "1",
        "references" : [ ],
        "reboot_suggested" : false,
        "relogin_suggested" : false,
        "restart_suggested" : false,
        "from" : "betelgeuze",
        "version" : "1",
        "release" : "1",
        "type" : "enhancement",
        "pkglist" : [ ],
        "title" : "Walrus",
        "_ns" : "units_erratum",
        "_content_type_id" : "erratum"
}
>
Actions #23

Updated by milan over 4 years ago

Recursive copy will require a fix


[vagrant@pulp2 pulp_rpm]$ pulp-admin rpm repo create --repo-id modulerr_copy
Successfully created repository [modulerr_copy]

[vagrant@pulp2 pulp_rpm]$ pulp-admin rpm repo copy errata -f modulerr -t modulerr_copy
--str-eq 'errata_id=RHEA-2012:0058' --recursive
This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Copied:
 erratum:
  RHEA-2012:0058
 rpm:
  gorilla-0.62-1-noarch

[vagrant@pulp2 pulp_rpm]$ pulp-admin rpm repo copy errata -f modulerr -t modulerr_copy
--str-eq 'errata_id=RHEA-2018:0042' --recursive
This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Copied:
  RHEA-2018:0042

[vagrant@pulp2 pulp_rpm]$
Actions #24

Updated by milan over 4 years ago

Actions #25

Updated by milan over 4 years ago

Actions #27

Updated by rchan over 4 years ago

  • Sprint changed from Sprint 42 to Sprint 43

Added by milan over 4 years ago

Revision 7bb8f881

Introduce module--erratum recursive copy

Previously, the modulemd units would be skipped during the recursive copy.

Fixes: #3919 https://pulp.plan.io/issues/3919

Actions #28

Updated by milan over 4 years ago

  • Status changed from ASSIGNED to MODIFIED
  • % Done changed from 50 to 100
Actions #29

Updated by ttereshc over 4 years ago

  • Platform Release set to 2.18.0
Actions #30

Updated by kersom about 4 years ago

This feature was already automated. See Pulp 2 Test Case

Actions #31

Updated by rchan about 4 years ago

kersom wrote:

This feature was already automated. See Pulp 2 Test Case

kersom - do we not list the Smash Test # linked to an story anymore? I'm glad to have this note here linking this story to the Smash Test, but curious why it's a note and not the link to the story?

Actions #32

Updated by kersom about 4 years ago

We already have automation in place for this one. ManageModularErrataTestCase

Actions #33

Updated by ttereshc about 4 years ago

  • Status changed from MODIFIED to 5
Actions #34

Updated by ttereshc about 4 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE
Actions #35

Updated by bmbouter almost 4 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF