Issue #6568
closedPulp 3 Sync of CentOS 8 Base OS repo, also ends up including AppStream
Description
Hi
I have been using Pulp 3.2.1 and have found that when Synching the following CentOS 8 repo: http://mirror.centos.org/centos/8.1.1911/BaseOS/x86_64/os/, when it is Published and Distributed it ends up including RPMs that are also in http://mirror.centos.org/centos/8.1.1911/AppStream/x86_64/os/Packages/.
A sample of how the repo ends up being Published is as follows:
.treeinfo
389-ds-base-1.4.1.3-7.module_el8.1.0+234+96aec258.x86_64.rpm
389-ds-base-devel-1.4.1.3-7.module_el8.1.0+234+96aec258.x86_64.rpm
389-ds-base-legacy-tools-1.4.1.3-7.module_el8.1.0+234+96aec258.x86_64.rpm
389-ds-base-libs-1.4.1.3-7.module_el8.1.0+234+96aec258.x86_64.rpm
389-ds-base-snmp-1.4.1.3-7.module_el8.1.0+234+96aec258.x86_64.rpm
389-ds1.48010020191119214651eb48df33x86_64snippet
AppStream/
CUnit-2.1.3-17.el8.i686.rpm
CUnit-2.1.3-17.el8.x86_64.rpm
GConf2-3.2.6-22.el8.i686.rpm
GConf2-3.2.6-22.el8.x86_64.rpm
Judy-1.0.5-18.module_el8.1.0+217+4d875839.x86_64.rpm
You will also see that there is an AppStream folder that has been created, which also contains its own repo data.
I recently tried out the same thing Pulp 3.3.0 today, and again see the same results. However due to the recent change in Pulp-RPM publication, when viewing the distributed content in a browser you see
.treeinfo
AppStream/
Packages/
images/
repodata/
Again AppStream contains it's own repodata, and then when you browse through the packages folders you will then see package belonging to AppStream e.g. adobe-mappings-cmap-20171205-3.el8.noarch.rpm
I have been running this on RHEL 7.7, using python 3.6
Matt
Files
Updated by ttereshc over 4 years ago
It's correct that AppStream repo is also synced. This CentOS repo contains a distribution tree (or kickstart tree) which one can use for provisioning. The details are described in the .treeinfo file http://mirror.centos.org/centos/8.1.1911/BaseOS/x86_64/os/.treeinfo There are 2 variant there, BaseOS and Appstream. One needs both for provisioning, so Pulp syncs both of them.
The fact that it ended up in the root directory is questionable and what we need check that. Could you post a content of the .treeinfo file which you distribute with Pulp?
Do I understand correctly that Appstream/ contains only repodata and no packages, while the Packages/ directory in the root of the repo has packages from both BaseOS and Appstream?
Updated by ipanova@redhat.com over 4 years ago
Updated by ttereshc over 4 years ago
- Priority changed from Normal to High
- Triaged changed from No to Yes
- Sprint set to Sprint 71
Thanks for confirmation! It's definitely a bug.
Updated by ipanova@redhat.com over 4 years ago
Problematic area seems to be in publish. Number of packages in the synced repo version match number in the upstream primary.xmkl of BaseOs repo. And i also cannot find the adobe-mappings-cmap
in the repo version as a proof.
$ http GET $BASE_ADDR/pulp/api/v3/repositories/rpm/rpm/51730a45-5ee3-4124-8a10-2aeb8eb6e2ec/versions/1/
HTTP/1.1 200 OK
Allow: GET, DELETE, HEAD, OPTIONS
Connection: close
Content-Length: 2125
Content-Type: application/json
Date: Tue, 28 Apr 2020 12:41:45 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN
{
"base_version": null,
"content_summary": {
"added": {
"rpm.distribution_tree": {
"count": 1,
"href": "/pulp/api/v3/content/rpm/distribution_trees/?repository_version_added=/pulp/api/v3/repositories/rpm/rpm/51730a45-5ee3-4124-8a10-2aeb8eb6e2ec/versions/1/"
},
"rpm.package": {
"count": 1663,
"href": "/pulp/api/v3/content/rpm/packages/?repository_version_added=/pulp/api/v3/repositories/rpm/rpm/51730a45-5ee3-4124-8a10-2aeb8eb6e2ec/versions/1/"
},
"rpm.packagecategory": {
"count": 4,
"href": "/pulp/api/v3/content/rpm/packagecategories/?repository_version_added=/pulp/api/v3/repositories/rpm/rpm/51730a45-5ee3-4124-8a10-2aeb8eb6e2ec/versions/1/"
},
"rpm.packageenvironment": {
"count": 3,
"href": "/pulp/api/v3/content/rpm/packageenvironments/?repository_version_added=/pulp/api/v3/repositories/rpm/rpm/51730a45-5ee3-4124-8a10-2aeb8eb6e2ec/versions/1/"
},
"rpm.packagegroup": {
"count": 35,
"href": "/pulp/api/v3/content/rpm/packagegroups/?repository_version_added=/pulp/api/v3/repositories/rpm/rpm/51730a45-5ee3-4124-8a10-2aeb8eb6e2ec/versions/1/"
}
},
"present": {
"rpm.distribution_tree": {
"count": 1,
"href": "/pulp/api/v3/content/rpm/distribution_trees/?repository_version=/pulp/api/v3/repositories/rpm/rpm/51730a45-5ee3-4124-8a10-2aeb8eb6e2ec/versions/1/"
},
"rpm.package": {
"count": 1663,
"href": "/pulp/api/v3/content/rpm/packages/?repository_version=/pulp/api/v3/repositories/rpm/rpm/51730a45-5ee3-4124-8a10-2aeb8eb6e2ec/versions/1/"
},
"rpm.packagecategory": {
"count": 4,
"href": "/pulp/api/v3/content/rpm/packagecategories/?repository_version=/pulp/api/v3/repositories/rpm/rpm/51730a45-5ee3-4124-8a10-2aeb8eb6e2ec/versions/1/"
},
"rpm.packageenvironment": {
"count": 3,
"href": "/pulp/api/v3/content/rpm/packageenvironments/?repository_version=/pulp/api/v3/repositories/rpm/rpm/51730a45-5ee3-4124-8a10-2aeb8eb6e2ec/versions/1/"
},
"rpm.packagegroup": {
"count": 35,
"href": "/pulp/api/v3/content/rpm/packagegroups/?repository_version=/pulp/api/v3/repositories/rpm/rpm/51730a45-5ee3-4124-8a10-2aeb8eb6e2ec/versions/1/"
}
},
"removed": {}
},
"number": 1,
"pulp_created": "2020-04-28T11:03:02.297245Z",
"pulp_href": "/pulp/api/v3/repositories/rpm/rpm/51730a45-5ee3-4124-8a10-2aeb8eb6e2ec/versions/1/"
}
(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 ~]$ http GET $BASE_ADDR'/pulp/api/v3/content/rpm/packages/?name=adobe-mappings-cmap&repository_version=/pulp/api/v3/repositories/rpm/rpm/51730a45-5ee3-4124-8a10-2aeb8eb6e2ec/versions/1/'
HTTP/1.1 200 OK
Allow: GET, POST, HEAD, OPTIONS
Connection: close
Content-Length: 52
Content-Type: application/json
Date: Tue, 28 Apr 2020 12:45:20 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN
{
"count": 0,
"next": null,
"previous": null,
"results": []
}
Updated by dkliban@redhat.com over 4 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to dkliban@redhat.com
There are two problems:
When pulp_rpm saves the .treeinfo file it saves the exact value of 'packages' and 'repository' of a variant and addon. These should instead be values that make sense to Pulp. The 'repository' should be the name of the variant or addon. The 'packages' should be /Packages.
When pulp_rpm is publishing a Distribution Tree, it simply takes the content that belongs to the subrepo and creates published artifacts from it. Published artifacts for subrepos need to have their relative_path modified to include the name of the 'variant' or 'addon'.
Updated by pulpbot over 4 years ago
- Status changed from ASSIGNED to POST
Added by dkliban@redhat.com over 4 years ago
Updated by dkliban@redhat.com over 4 years ago
- Status changed from POST to MODIFIED
Applied in changeset 4b0ad21d2ec51b030ce6c3bb17c21adbc02400c0.
Added by dkliban@redhat.com over 4 years ago
Revision f325cd78 | View on GitHub
Fixes how DistributionTrees (kickstart repos) are published.
This patch does the following:
- fixes the layout of sub-repos in publications.
- generates a TreeInfo file that contains repository and packages paths that represent the publish structure of an RPM Publication that contains a DistributionTree and creates PublishedArtifact using this file.
- refactors the TreeInfoData object. The variants and addons properties have been converted from List to Dict. This enables easier access to their properties when generating the new TreeInfo file.
- Extends the CentOS 8 performance test to check that Distribution Tree is published correctly
fixes: #6568 https://pulp.plan.io/issues/6568
[nocoverage]
(cherry picked from commit 4b0ad21d2ec51b030ce6c3bb17c21adbc02400c0)
Updated by dkliban@redhat.com over 4 years ago
Applied in changeset f325cd787d704798378668ce3150969cfb14c9c6.
Updated by ttereshc over 4 years ago
- Status changed from MODIFIED to CLOSED - CURRENTRELEASE
Fixes how DistributionTrees (kickstart repos) are published.
This patch does the following:
fixes: #6568 https://pulp.plan.io/issues/6568
[nocoverage]