Project

Profile

Help

Issue #6568

closed

Pulp 3 Sync of CentOS 8 Base OS repo, also ends up including AppStream

Added by mped over 2 years ago. Updated about 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
RHEL 7
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 77
Quarter:

Description

Hi

I have been using Pulp 3.2.1 and have found that when Synching the following CentOS 8 repo: http://mirror.centos.org/centos/8.1.1911/BaseOS/x86_64/os/, when it is Published and Distributed it ends up including RPMs that are also in http://mirror.centos.org/centos/8.1.1911/AppStream/x86_64/os/Packages/.

A sample of how the repo ends up being Published is as follows:

.treeinfo
389-ds-base-1.4.1.3-7.module_el8.1.0+234+96aec258.x86_64.rpm
389-ds-base-devel-1.4.1.3-7.module_el8.1.0+234+96aec258.x86_64.rpm
389-ds-base-legacy-tools-1.4.1.3-7.module_el8.1.0+234+96aec258.x86_64.rpm
389-ds-base-libs-1.4.1.3-7.module_el8.1.0+234+96aec258.x86_64.rpm
389-ds-base-snmp-1.4.1.3-7.module_el8.1.0+234+96aec258.x86_64.rpm
389-ds1.48010020191119214651eb48df33x86_64snippet
AppStream/
CUnit-2.1.3-17.el8.i686.rpm
CUnit-2.1.3-17.el8.x86_64.rpm
GConf2-3.2.6-22.el8.i686.rpm
GConf2-3.2.6-22.el8.x86_64.rpm
Judy-1.0.5-18.module_el8.1.0+217+4d875839.x86_64.rpm

You will also see that there is an AppStream folder that has been created, which also contains its own repo data.

I recently tried out the same thing Pulp 3.3.0 today, and again see the same results. However due to the recent change in Pulp-RPM publication, when viewing the distributed content in a browser you see

.treeinfo
AppStream/
Packages/
images/
repodata/

Again AppStream contains it's own repodata, and then when you browse through the packages folders you will then see package belonging to AppStream e.g. adobe-mappings-cmap-20171205-3.el8.noarch.rpm

I have been running this on RHEL 7.7, using python 3.6

Matt


Files

Actions #1

Updated by mped over 2 years ago

Sorry I meant to raise this in pulp-rpm

Actions #2

Updated by fao89 over 2 years ago

  • Project changed from Pulp to RPM Support
Actions #3

Updated by ttereshc over 2 years ago

It's correct that AppStream repo is also synced. This CentOS repo contains a distribution tree (or kickstart tree) which one can use for provisioning. The details are described in the .treeinfo file http://mirror.centos.org/centos/8.1.1911/BaseOS/x86_64/os/.treeinfo There are 2 variant there, BaseOS and Appstream. One needs both for provisioning, so Pulp syncs both of them.

The fact that it ended up in the root directory is questionable and what we need check that. Could you post a content of the .treeinfo file which you distribute with Pulp?

Do I understand correctly that Appstream/ contains only repodata and no packages, while the Packages/ directory in the root of the repo has packages from both BaseOS and Appstream?

Actions #5

Updated by ttereshc over 2 years ago

  • Priority changed from Normal to High
  • Triaged changed from No to Yes
  • Sprint set to Sprint 71

Thanks for confirmation! It's definitely a bug.

Actions #6

Updated by ipanova@redhat.com over 2 years ago

Problematic area seems to be in publish. Number of packages in the synced repo version match number in the upstream primary.xmkl of BaseOs repo. And i also cannot find the adobe-mappings-cmap in the repo version as a proof.

$ http GET $BASE_ADDR/pulp/api/v3/repositories/rpm/rpm/51730a45-5ee3-4124-8a10-2aeb8eb6e2ec/versions/1/
HTTP/1.1 200 OK
Allow: GET, DELETE, HEAD, OPTIONS
Connection: close
Content-Length: 2125
Content-Type: application/json
Date: Tue, 28 Apr 2020 12:41:45 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "base_version": null,
    "content_summary": {
        "added": {
            "rpm.distribution_tree": {
                "count": 1,
                "href": "/pulp/api/v3/content/rpm/distribution_trees/?repository_version_added=/pulp/api/v3/repositories/rpm/rpm/51730a45-5ee3-4124-8a10-2aeb8eb6e2ec/versions/1/"
            },
            "rpm.package": {
                "count": 1663,
                "href": "/pulp/api/v3/content/rpm/packages/?repository_version_added=/pulp/api/v3/repositories/rpm/rpm/51730a45-5ee3-4124-8a10-2aeb8eb6e2ec/versions/1/"
            },
            "rpm.packagecategory": {
                "count": 4,
                "href": "/pulp/api/v3/content/rpm/packagecategories/?repository_version_added=/pulp/api/v3/repositories/rpm/rpm/51730a45-5ee3-4124-8a10-2aeb8eb6e2ec/versions/1/"
            },
            "rpm.packageenvironment": {
                "count": 3,
                "href": "/pulp/api/v3/content/rpm/packageenvironments/?repository_version_added=/pulp/api/v3/repositories/rpm/rpm/51730a45-5ee3-4124-8a10-2aeb8eb6e2ec/versions/1/"
            },
            "rpm.packagegroup": {
                "count": 35,
                "href": "/pulp/api/v3/content/rpm/packagegroups/?repository_version_added=/pulp/api/v3/repositories/rpm/rpm/51730a45-5ee3-4124-8a10-2aeb8eb6e2ec/versions/1/"
            }
        },
        "present": {
            "rpm.distribution_tree": {
                "count": 1,
                "href": "/pulp/api/v3/content/rpm/distribution_trees/?repository_version=/pulp/api/v3/repositories/rpm/rpm/51730a45-5ee3-4124-8a10-2aeb8eb6e2ec/versions/1/"
            },
            "rpm.package": {
                "count": 1663,
                "href": "/pulp/api/v3/content/rpm/packages/?repository_version=/pulp/api/v3/repositories/rpm/rpm/51730a45-5ee3-4124-8a10-2aeb8eb6e2ec/versions/1/"
            },
            "rpm.packagecategory": {
                "count": 4,
                "href": "/pulp/api/v3/content/rpm/packagecategories/?repository_version=/pulp/api/v3/repositories/rpm/rpm/51730a45-5ee3-4124-8a10-2aeb8eb6e2ec/versions/1/"
            },
            "rpm.packageenvironment": {
                "count": 3,
                "href": "/pulp/api/v3/content/rpm/packageenvironments/?repository_version=/pulp/api/v3/repositories/rpm/rpm/51730a45-5ee3-4124-8a10-2aeb8eb6e2ec/versions/1/"
            },
            "rpm.packagegroup": {
                "count": 35,
                "href": "/pulp/api/v3/content/rpm/packagegroups/?repository_version=/pulp/api/v3/repositories/rpm/rpm/51730a45-5ee3-4124-8a10-2aeb8eb6e2ec/versions/1/"
            }
        },
        "removed": {}
    },
    "number": 1,
    "pulp_created": "2020-04-28T11:03:02.297245Z",
    "pulp_href": "/pulp/api/v3/repositories/rpm/rpm/51730a45-5ee3-4124-8a10-2aeb8eb6e2ec/versions/1/"
}

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 ~]$ http GET $BASE_ADDR'/pulp/api/v3/content/rpm/packages/?name=adobe-mappings-cmap&repository_version=/pulp/api/v3/repositories/rpm/rpm/51730a45-5ee3-4124-8a10-2aeb8eb6e2ec/versions/1/'
HTTP/1.1 200 OK
Allow: GET, POST, HEAD, OPTIONS
Connection: close
Content-Length: 52
Content-Type: application/json
Date: Tue, 28 Apr 2020 12:45:20 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "count": 0,
    "next": null,
    "previous": null,
    "results": []
}

Actions #7

Updated by dkliban@redhat.com over 2 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dkliban@redhat.com

There are two problems:

When pulp_rpm saves the .treeinfo file it saves the exact value of 'packages' and 'repository' of a variant and addon. These should instead be values that make sense to Pulp. The 'repository' should be the name of the variant or addon. The 'packages' should be /Packages. 

When pulp_rpm is publishing a Distribution Tree, it simply takes the content that belongs to the subrepo and creates published artifacts from it. Published artifacts for subrepos need to have their relative_path modified to include the name of the 'variant' or 'addon'.

Actions #8

Updated by rchan over 2 years ago

  • Sprint changed from Sprint 71 to Sprint 72
Actions #9

Updated by rchan about 2 years ago

  • Sprint changed from Sprint 72 to Sprint 73
Actions #10

Updated by rchan about 2 years ago

  • Sprint changed from Sprint 73 to Sprint 74
Actions #11

Updated by rchan about 2 years ago

  • Sprint changed from Sprint 74 to Sprint 75
Actions #12

Updated by pulpbot about 2 years ago

  • Status changed from ASSIGNED to POST
Actions #13

Updated by rchan about 2 years ago

  • Sprint changed from Sprint 75 to Sprint 76
Actions #14

Updated by rchan about 2 years ago

  • Sprint changed from Sprint 76 to Sprint 77

Added by dkliban@redhat.com about 2 years ago

Revision 4b0ad21d

Fixes how DistributionTrees (kickstart repos) are published.

This patch does the following:

  • fixes the layout of sub-repos in publications.
  • generates a TreeInfo file that contains repository and packages paths that represent the publish structure of an RPM Publication that contains a DistributionTree and creates PublishedArtifact using this file.
  • refactors the TreeInfoData object. The variants and addons properties have been converted from List to Dict. This enables easier access to their properties when generating the new TreeInfo file.
  • Extends the CentOS 8 performance test to check that Distribution Tree is published correctly

fixes: #6568 https://pulp.plan.io/issues/6568

[nocoverage]

Actions #15

Updated by dkliban@redhat.com about 2 years ago

  • Status changed from POST to MODIFIED

Added by dkliban@redhat.com about 2 years ago

Revision f325cd78

Fixes how DistributionTrees (kickstart repos) are published.

This patch does the following:

  • fixes the layout of sub-repos in publications.
  • generates a TreeInfo file that contains repository and packages paths that represent the publish structure of an RPM Publication that contains a DistributionTree and creates PublishedArtifact using this file.
  • refactors the TreeInfoData object. The variants and addons properties have been converted from List to Dict. This enables easier access to their properties when generating the new TreeInfo file.
  • Extends the CentOS 8 performance test to check that Distribution Tree is published correctly

fixes: #6568 https://pulp.plan.io/issues/6568

[nocoverage]

(cherry picked from commit 4b0ad21d2ec51b030ce6c3bb17c21adbc02400c0)

Actions #17

Updated by ttereshc about 2 years ago

  • Sprint/Milestone set to Pulp RPM 3.4.2
Actions #18

Updated by ttereshc about 2 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF