Project

Profile

Help

Task #2668

Test Pulp Syncing Several Repos with Large Metadata

Added by bmbouter over 4 years ago. Updated over 2 years ago.

Status:
CLOSED - WONTFIX
Priority:
Normal
Assignee:
-
Sprint/Milestone:
-
Start date:
Due date:
% Done:

0%

Estimated time:
Platform Release:
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

A user reported that Pulp may not sync some repos due to large metadata even with the fix in https://github.com/pulp/pulp_rpm/pull/1018

Katello ticket 2487 has a link to the repo that failed to sync:
https://fedorapeople.org/groups/katello/releases/yum/foreman/1.13/el6/x86_64/

Jboss package from ticket 723 should be available from brew:
https://brewweb.engineering.redhat.com/brew/packageinfo?packageID=36912

You can also try to sync full JBEAP latest compose (I do not have pulp installed anywhere right now, but I think this should also fail):
http://download-node-02.eng.bos.redhat.com/devel/candidates/JBEAP/composing/latest-JBEAP-7.0-RHEL-7/compose/Server/x86_64/os/

History

#1 Updated by pthomas@redhat.com over 4 years ago


[root@ibm-x3550m3-13 ~]# rpm -qa |grep pulp |sort
pulp-admin-client-2.12.2-0.1.beta.el7.noarch
pulp-docker-admin-extensions-2.3.0-1.el7.noarch
pulp-docker-plugins-2.3.0-1.el7.noarch
pulp-ostree-admin-extensions-1.2.1-0.1.beta.el7.noarch
pulp-ostree-plugins-1.2.1-0.1.beta.el7.noarch
pulp-puppet-admin-extensions-2.12.2-0.1.beta.el7.noarch
pulp-puppet-plugins-2.12.2-0.1.beta.el7.noarch
pulp-python-admin-extensions-1.1.3-1.el7.noarch
pulp-python-plugins-1.1.3-1.el7.noarch
pulp-rpm-admin-extensions-2.12.2-0.1.beta.el7.noarch
pulp-rpm-plugins-2.12.2-0.1.beta.el7.noarch
pulp-selinux-2.12.2-0.1.beta.el7.noarch
pulp-server-2.12.2-0.1.beta.el7.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
python-kombu-3.0.33-6.pulp.el7.noarch
python-pulp-bindings-2.12.2-0.1.beta.el7.noarch
python-pulp-client-lib-2.12.2-0.1.beta.el7.noarch
python-pulp-common-2.12.2-0.1.beta.el7.noarch
python-pulp-docker-common-2.3.0-1.el7.noarch
python-pulp-oid_validation-2.12.2-0.1.beta.el7.noarch
python-pulp-ostree-common-1.2.1-0.1.beta.el7.noarch
python-pulp-puppet-common-2.12.2-0.1.beta.el7.noarch
python-pulp-python-common-1.1.3-1.el7.noarch
python-pulp-repoauth-2.12.2-0.1.beta.el7.noarch
python-pulp-rpm-common-2.12.2-0.1.beta.el7.noarch
python-pulp-streamer-2.12.2-0.1.beta.el7.noarch
[root@ibm-x3550m3-13 ~]# 

[root@ibm-x3550m3-13 ~]# pulp-admin rpm repo create --repo-id foreman --feed https://fedorapeople.org/groups/katello/releases/yum/foreman/1.13/el6/x86_64/
Successfully created repository [foreman]

[root@ibm-x3550m3-13 ~]# pulp-admin rpm repo sync run --repo-id foreman
+----------------------------------------------------------------------+
                   Synchronizing Repository [foreman]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[|]
... completed

Downloading repository content...
[-]
[==================================================] 100%
RPMs:       344/344 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items
... completed

Importing errata...
[-]
... completed

Importing package groups/categories...
[-]
... completed

Cleaning duplicate packages...
[-]
... completed

Task Succeeded

Initializing repo metadata
[-]
... completed

Publishing Distribution files
[-]
... completed

Publishing RPMs
[==================================================] 100%
344 of 344 items
... completed

Publishing Delta RPMs
... skipped

Publishing Errata
[\]
... completed

Publishing Comps file
[-]
... completed

Publishing Metadata.
[-]
... completed

Closing repo metadata
[-]
... completed

Generating sqlite files
... skipped

Generating HTML files
... skipped

Publishing files to web
[-]
... completed

Writing Listings File
[-]
... completed

Task Succeeded

[root@ibm-x3550m3-13 ~]# pulp-admin rpm repo create --repo-id issue2668 --feed http://download-node-02.eng.bos.redhat.com/devel/candidates/JBEAP/composing/latest-JBEAP-7.0-RHEL-7/compose/Server/x86_64/os/
Successfully created repository [issue2668]

[root@ibm-x3550m3-13 ~]# pulp-admin repo sync run --repo-id issue2668 
Usage: pulp-admin [SUB_SECTION, ..] COMMAND
Description: list repositories and manage repo groups

Available Sections:
  group   - repository group commands
  history - show sync and publish history
  tasks   - list and cancel tasks related to a specific repository

Available Commands:
  download - queues a full download of all missing units in a repository that is using a
             "background" or "on_demand" download policy.
  list     - lists repositories on the Pulp server
[root@ibm-x3550m3-13 ~]# pulp-admin rpm repo sync run --repo-id issue2668 
+----------------------------------------------------------------------+
                  Synchronizing Repository [issue2668]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[|]
... completed

Downloading repository content...
[-]
[==================================================] 100%
RPMs:       296/296 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items
... completed

Importing errata...
[-]
... completed

Importing package groups/categories...
[-]
... completed

Cleaning duplicate packages...
[-]
... completed

Task Succeeded

Initializing repo metadata
[-]
... completed

Publishing Distribution files
[-]
... completed

Publishing RPMs
[==================================================] 100%
296 of 296 items
... completed

Publishing Delta RPMs
... skipped

Publishing Errata
[-]
... completed

Publishing Comps file
[==================================================] 100%
2 of 2 items
... completed

Publishing Metadata.
[-]
... completed

Closing repo metadata
[-]
... completed

Generating sqlite files
... skipped

Generating HTML files
... skipped

Publishing files to web
[\]
... completed

Writing Listings File
[-]
... completed

Task Succeeded

#2 Updated by bmbouter over 4 years ago

Thank you for testing this.

Can a test also be done that uploads a few of the jbossas-javadocs rpms? Those are from here: https://brewweb.engineering.redhat.com/brew/packageinfo?packageID=36912

With that third piece I would consider this done.

#3 Updated by pthomas@redhat.com over 4 years ago

[root@hp-dl380pgen8-02-vm-12 ~]# pulp-admin rpm repo uploads rpm --repo-id upload --dir test/ -vvv
+----------------------------------------------------------------------+
                              Unit Upload
+----------------------------------------------------------------------+

Extracting necessary metadata for each request...
[==================================================] 100%
Analyzing: jbossas-javadocs-7.5.14-2.Final_redhat_2.1.ep6.el7.noarch.rpm
... completed

Files to be uploaded:
  jbossas-javadocs-7.5.14-1.Final_redhat_1.1.ep6.el5.noarch.rpm
  jbossas-javadocs-7.5.14-2.Final_redhat_2.1.ep6.el6.noarch.rpm
  jbossas-javadocs-7.5.14-2.Final_redhat_2.1.ep6.el7.noarch.rpm

Creating upload requests on the server...
[==================================================] 100%
Initializing: jbossas-javadocs-7.5.14-2.Final_redhat_2.1.ep6.el7.noarch.rpm
... completed

Starting upload of selected units. If this process is stopped through ctrl+c,
the uploads will be paused and may be resumed later using the resume command or
canceled entirely using the cancel command.

Uploading: jbossas-javadocs-7.5.14-1.Final_redhat_1.1.ep6.el5.noarch.rpm
[==================================================] 100%
49558304/49558304 bytes
... completed

Importing into the repository...
This command may be exited via ctrl+c without affecting the request.

[/]
Running...

Task Succeeded

Deleting the upload request...
... completed

Uploading: jbossas-javadocs-7.5.14-2.Final_redhat_2.1.ep6.el6.noarch.rpm
[==================================================] 100%
36352104/36352104 bytes
... completed

Importing into the repository...
This command may be exited via ctrl+c without affecting the request.

[/]
Running...

Task Succeeded

Deleting the upload request...
... completed

Uploading: jbossas-javadocs-7.5.14-2.Final_redhat_2.1.ep6.el7.noarch.rpm
[==================================================] 100%
31275636/31275636 bytes
... completed

Importing into the repository...
This command may be exited via ctrl+c without affecting the request.

[|]
Running...

Task Succeeded

Deleting the upload request...
... completed

#4 Updated by bmbouter over 2 years ago

  • Status changed from NEW to CLOSED - WONTFIX

Pulp 2 is approaching maintenance mode, and this Pulp 2 ticket is not being actively worked on. As such, it is being closed as WONTFIX. Pulp 2 is still accepting contributions though, so if you want to contribute a fix for this ticket, please reopen or comment on it. If you don't have permissions to reopen this ticket, or you want to discuss an issue, please reach out via the developer mailing list.

#5 Updated by bmbouter over 2 years ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF