Project

Profile

Help

Actions
Send by e-mail Copy link

Test #4542

closed

[pulp_rpm] - Repository publishing duplicates RPM files under Packages and Packages/<LETTER> structure

Added by bherring about 5 years ago. Updated almost 4 years ago.

Status:
CLOSED - COMPLETE
Priority:
High
Assignee:
kersom
Category:
-
Sprint/Milestone:
-
Version:
Platform Release:
Tags:
Pulp 2
Sprint:
Quarter:

Description

Some repositories are getting content duplicated on its repodata. For example, the CentoOS 7 Base repository, gets duplicated symlinks as follows:

[root@pulp ~]# pulp-admin  rpm repo list --details --repo-id centos-7-base
+----------------------------------------------------------------------+
                            RPM Repositories
+----------------------------------------------------------------------+

Id:                   centos-7-base
Display Name:         None
Description:          None
Content Unit Counts:  
  Distribution:        1
  Package Category:    11
  Package Environment: 10
  Package Group:       88
  Package Langpacks:   1
  Rpm:                 10019
Notes:                
Scratchpad:           
  Checksum Type: sha256
Importers:            
  Config:               
    Feed: http://centos.mirror.constant.com/7/os/x86_64/
  Id:                   yum_importer
  Importer Type Id:     yum_importer
  Last Override Config: 
  Last Sync:            2019-03-13T19:58:14Z
  Last Updated:         2019-03-13T19:54:40Z
  Repo Id:              centos-7-base
  Scratchpad:           
    Repomd Revision: 1543161601
Distributors:         
  Auto Publish:         True
  Config:               
    Http:         True
    Https:        True
    Relative URL: 7/os/x86_64/
  Distributor Type Id:  yum_distributor
  Id:                   yum_distributor
  Last Override Config: 
  Last Publish:         2019-03-14T04:17:40Z
  Last Updated:         2019-03-13T19:54:40Z
  Repo Id:              centos-7-base
  Scratchpad:           
  Auto Publish:         False
  Config:               
    Http:         True
    Https:        True
    Relative URL: 7/os/x86_64/
  Distributor Type Id:  export_distributor
  Id:                   export_distributor
  Last Override Config: 
  Last Publish:         None
  Last Updated:         2019-03-13T19:54:40Z
  Repo Id:              centos-7-base
  Scratchpad:

Once the repository is published, it ended up having duplicated RPMs

pulp-admin  rpm repo  publish run  --repo-id df05f52b-431e-483d-9878-9ffef41d70c8 --force-full

○ → tree /var/lib/pulp/published/yum/https/repos/7/os/x86_64/Packages/  | grep 389-ds-base
│   ├── 389-ds-base-1.3.8.4-15.el7.x86_64.rpm -> /var/lib/pulp/content/units/rpm/07/cb4bf5d6199cdc7192cbdbbf283cccdef38f18751c53660107b14232d2d349/389-ds-base-1.3.8.4-15.el7.x86_64.rpm
│   ├── 389-ds-base-devel-1.3.8.4-15.el7.x86_64.rpm -> /var/lib/pulp/content/units/rpm/ab/e6f96db9ad2c0f7f097f71b506b25a0d83a4a9926e904100f006361ef66d1f/389-ds-base-devel-1.3.8.4-15.el7.x86_64.rpm
│   ├── 389-ds-base-libs-1.3.8.4-15.el7.x86_64.rpm -> /var/lib/pulp/content/units/rpm/94/5011e802dbf9d470522eac943442a8718e08ba321dc0649a55a0d7f130e363/389-ds-base-libs-1.3.8.4-15.el7.x86_64.rpm
│   └── 389-ds-base-snmp-1.3.8.4-15.el7.x86_64.rpm -> /var/lib/pulp/content/units/rpm/e9/b6b32a0421e87ab83a6ea7346f51e3929746e10eeb2c9b5f05e8278c9e7b61/389-ds-base-snmp-1.3.8.4-15.el7.x86_64.rpm
├── 389-ds-base-1.3.8.4-15.el7.x86_64.rpm -> /var/lib/pulp/content/units/rpm/07/cb4bf5d6199cdc7192cbdbbf283cccdef38f18751c53660107b14232d2d349/389-ds-base-1.3.8.4-15.el7.x86_64.rpm
├── 389-ds-base-devel-1.3.8.4-15.el7.x86_64.rpm -> /var/lib/pulp/content/units/rpm/ab/e6f96db9ad2c0f7f097f71b506b25a0d83a4a9926e904100f006361ef66d1f/389-ds-base-devel-1.3.8.4-15.el7.x86_64.rpm
├── 389-ds-base-libs-1.3.8.4-15.el7.x86_64.rpm -> /var/lib/pulp/content/units/rpm/94/5011e802dbf9d470522eac943442a8718e08ba321dc0649a55a0d7f130e363/389-ds-base-libs-1.3.8.4-15.el7.x86_64.rpm
├── 389-ds-base-snmp-1.3.8.4-15.el7.x86_64.rpm -> /var/lib/pulp/content/units/rpm/e9/b6b32a0421e87ab83a6ea7346f51e3929746e10eeb2c9b5f05e8278c9e7b61/389-ds-base-snmp-1.3.8.4-15.el7.x86_64.rpm

That creates a big problem whenever exporting the content via ISO , as the size of the repo will be twice big.

The repodata is being created correctly as it only references the files under the new strucuture Packages/<first_letter>/

○ → zcat  /var/lib/pulp/published/yum/https/repos/7/os/x86_64/Packages/repodata/a3f98f3b850725e0ac52472653efd309be6ba436296e66b1b997ca92c95fcdf1-primary.xml.gz  | grep 'href' | grep 389-ds-base
  <location href="Packages/3/389-ds-base-libs-1.3.8.4-15.el7.x86_64.rpm"/>
  <location href="Packages/3/389-ds-base-1.3.8.4-15.el7.x86_64.rpm"/>
  <location href="Packages/3/389-ds-base-snmp-1.3.8.4-15.el7.x86_64.rpm"/>
  <location href="Packages/3/389-ds-base-devel-1.3.8.4-15.el7.x86_64.rpm"/>

Related issues

Copied from RPM Support - Issue #4541: Repository publishing duplicates RPM files under Packages and Packages/<LETTER> structureCLOSED - CURRENTRELEASEdkliban@redhat.comActions
Actions
Copy link
 #1

Updated by bherring about 5 years ago

  • Copied from Issue #4541: Repository publishing duplicates RPM files under Packages and Packages/<LETTER> structure added
Actions
Copy link
 #2

Updated by bherring about 5 years ago

  • Priority changed from Normal to High
Actions
Copy link
 #3

Updated by bherring about 5 years ago

  • Sprint/Milestone set to 2.19.0
Actions
Copy link
 #4

Updated by bherring about 5 years ago

  • Sprint/Milestone deleted (2.19.0)
Actions
Copy link
 #5

Updated by kersom about 5 years ago

  • Assignee set to kersom
Actions
Copy link
 #6

Updated by kersom about 5 years ago

  • Status changed from NEW to CLOSED - COMPLETE

Manually tested. Not suitable for automation.

Pulp Version

[root@localhost ~]# rpm -qa | grep pulp | sort
pulp-admin-client-2.19.0-0.1.rc.el7.noarch
pulp-deb-admin-extensions-1.9.0-0.2.rc.git.110.c3056d7.el7.noarch
pulp-deb-plugins-1.9.0-0.2.rc.git.110.c3056d7.el7.noarch
pulp-docker-admin-extensions-3.2.3-0.1.rc.el7.noarch
pulp-docker-plugins-3.2.3-0.1.rc.el7.noarch
pulp-ostree-admin-extensions-1.4.0-1.el7.noarch
pulp-ostree-plugins-1.4.0-1.el7.noarch
pulp-puppet-admin-extensions-2.19.0-0.1.rc.el7.noarch
pulp-puppet-plugins-2.19.0-0.1.rc.el7.noarch
pulp-puppet-tools-2.19.0-0.1.rc.el7.noarch
pulp-python-admin-extensions-2.0.3-1.el7.noarch
pulp-python-plugins-2.0.3-1.el7.noarch
pulp-rpm-admin-extensions-2.19.0-0.1.rc.el7.noarch
pulp-rpm-plugins-2.19.0-0.1.rc.el7.noarch
pulp-selinux-2.19.0-0.1.rc.el7.noarch
pulp-server-2.19.0-0.1.rc.el7.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
python-pulp-bindings-2.19.0-0.1.rc.el7.noarch
python-pulp-client-lib-2.19.0-0.1.rc.el7.noarch
python-pulp-common-2.19.0-0.1.rc.el7.noarch
python-pulp-deb-common-1.9.0-0.2.rc.git.110.c3056d7.el7.noarch
python-pulp-docker-common-3.2.3-0.1.rc.el7.noarch
python-pulp-oid_validation-2.19.0-0.1.rc.el7.noarch
python-pulp-ostree-common-1.4.0-1.el7.noarch
python-pulp-puppet-common-2.19.0-0.1.rc.el7.noarch
python-pulp-python-common-2.0.3-1.el7.noarch
python-pulp-repoauth-2.19.0-0.1.rc.el7.noarch
python-pulp-rpm-common-2.19.0-0.1.rc.el7.noarch
python-pulp-streamer-2.19.0-0.1.rc.el7.noarch

commands:


pulp-admin login -u admin -p admin
pulp-admin rpm repo create --repo-id foo --feed http://centos.mirror.constant.com/7/os/x86_64/
pulp-admin rpm repo sync run --repo-id foo
pulp-admin rpm repo publish run --repo-id foo
pulp-admin  rpm repo list --details --repo-id foo
pulp-admin  rpm repo  publish run  --repo-id foo --force-full

[root@localhost ~]# pulp-admin  rpm repo list --details --repo-id foo
+----------------------------------------------------------------------+
                            RPM Repositories
+----------------------------------------------------------------------+

Id:                   foo
Display Name:         None
Description:          None
Content Unit Counts:  
  Distribution:        1
  Package Category:    11
  Package Environment: 10
  Package Group:       88
  Package Langpacks:   1
  Rpm:                 10016
Notes:                
Scratchpad:           
  Checksum Type: sha256
Importers:            
  Config:               
    Feed: http://centos.mirror.constant.com/7/os/x86_64/
  Id:                   yum_importer
  Importer Type Id:     yum_importer
  Last Override Config: 
  Last Sync:            2019-01-29T22:45:35Z
  Last Updated:         2019-01-29T22:30:38Z
  Repo Id:              foo
  Scratchpad:           None
Distributors:         
  Auto Publish:         True
  Config:               
    Http:         False
    Https:        True
    Relative URL: 7/os/x86_64/
  Distributor Type Id:  yum_distributor
  Id:                   yum_distributor
  Last Override Config: 
  Last Publish:         2019-01-29T22:49:52Z
  Last Updated:         2019-01-29T22:30:38Z
  Repo Id:              foo
  Scratchpad:           
  Auto Publish:         False
  Config:               
    Http:         False
    Https:        True
    Relative URL: 7/os/x86_64/
  Distributor Type Id:  export_distributor
  Id:                   export_distributor
  Last Override Config: 
  Last Publish:         None
  Last Updated:         2019-01-29T22:30:38Z
  Repo Id:              foo
  Scratchpad:

No duplicate RPMs present.

[root@localhost ~]# tree /var/lib/pulp/published/yum/https/repos/7/os/x86_64/Packages/  | grep 389-ds-base
│   ├── 389-ds-base-1.3.8.4-15.el7.x86_64.rpm -> /var/lib/pulp/content/units/rpm/07/cb4bf5d6199cdc7192cbdbbf283cccdef38f18751c53660107b14232d2d349/389-ds-base-1.3.8.4-15.el7.x86_64.rpm
│   ├── 389-ds-base-devel-1.3.8.4-15.el7.x86_64.rpm -> /var/lib/pulp/content/units/rpm/ab/e6f96db9ad2c0f7f097f71b506b25a0d83a4a9926e904100f006361ef66d1f/389-ds-base-devel-1.3.8.4-15.el7.x86_64.rpm
│   ├── 389-ds-base-libs-1.3.8.4-15.el7.x86_64.rpm -> /var/lib/pulp/content/units/rpm/94/5011e802dbf9d470522eac943442a8718e08ba321dc0649a55a0d7f130e363/389-ds-base-libs-1.3.8.4-15.el7.x86_64.rpm
│   └── 389-ds-base-snmp-1.3.8.4-15.el7.x86_64.rpm -> /var/lib/pulp/content/units/rpm/e9/b6b32a0421e87ab83a6ea7346f51e3929746e10eeb2c9b5f05e8278c9e7b61/389-ds-base-snmp-1.3.8.4-15.el7.x86_64.rpm
Actions
Copy link
 #7

Updated by bmbouter about 5 years ago

  • Tags Pulp 2 added
Actions
Copy link
 #8

Updated by bmbouter almost 4 years ago

  • Category deleted (14)

We are removing the 'API' category per open floor discussion June 16, 2020.

Actions
Copy link
 #9

Updated by bmbouter almost 4 years ago

  • Tags deleted (Release Engineering)
Actions
Send by e-mail Copy link

Also available in: Atom PDF