Project

Profile

Help

Issue #1846

closed

pulp does not sync puppetforge correctly

Added by jokroepke about 6 years ago. Updated about 3 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
2.8.0
Platform Release:
2.13.1
OS:
CentOS 7
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 1
Quarter:

Description

Hi,

i have some problems with syncing the puppet forge.

How to reproduce (for me):

[root@pulp1 ~]#  pulp-admin  puppet repo sync run --repo-id=forge
+----------------------------------------------------------------------+
                    Synchronizing Repository [forge]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[==================================================] 100%
Metadata Query: 1/1 items
... completed

Downloading new modules...
[==================================================] 100%
Module: 40/40 items
... completed

Could not import the following modules:

    passenger-0.5.0: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/marcel-passenger-0.5.0.tar.gz']
    vmwaretools-2014.04.22: FileRetrievalException: Bad Request
    gitlist-0.1.2: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/eshamow-gitlist-0.1.2.tar.gz']
    simpleid-0.1.0: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/eshamow-simpleid-0.1.0.tar.gz']
    denyhosts-0.1.0: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/puppetlabs-denyhosts-0.1.0.tar.gz']
    riemann-0.0.2: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/jamtur01-riemann-0.0.2.tar.gz']
    spamassassin-1.0.5: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/BillWeiss-spamassassin-1.0.5.tar.gz']
    icecast-0.1.0: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/nwolfe-icecast-0.1.0.tar.gz']
    stack_wordpress-0.1.3: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/jaronson-stack_wordpress-0.1.3.tar.gz']
    ec2legacy-0.1.1: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/mrzarquon-ec2legacy-0.1.1.tar.gz']
    puppet_hipchat-0.6.0: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/jamtur01-puppet_hipchat-0.6.0.tar.gz']
    curator-1.0.1: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/TubeMogul-curator-1.0.1.tar.gz']
    xhgui-0.1.0: 'utf8' codec can't decode byte 0xbd in position 13: invalid
start byte
    puppet_server-0.0.1: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/ajroetker-puppet_server-0.0.1.tar.gz']
    puppet_config-0.1.0: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/vjanelle-puppet_config-0.1.0.tar.gz']
    resolv_conf-3.0.5: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/zanloy-resolv_conf-3.0.5.tar.gz']
    pagerduty-0.0.2: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/jamtur01-pagerduty-0.0.2.tar.gz']
    strongswan-0.0.2: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/Nextdoor-strongswan-0.0.2.tar.gz']
    figlet-1.0.1: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/zanloy-figlet-1.0.1.tar.gz']
    xmpp-0.0.2: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/jamtur01-xmpp-0.0.2.tar.gz']

Task Succeeded

Publishing modules...
[==================================================] 100%
Module: 1/1 items
... completed

Generating repository metadata...
[-]
... completed

Publishing repository over HTTP...
... completed

Publishing repository over HTTPS...
... skipped

Task Succeeded

Actually Result:

Sync 40 Modules

Expected Result:

Sync all (~1800) modules.

Best Regards,
Jan-Otto Kröpke
noris network AG


Related issues

Related to Puppet Support - Issue #4140: Puppet fails to find metadata file if there are multiple top-level files/dirsCLOSED - CURRENTRELEASEdaviddavisActions
Actions #1

Updated by mhrivnak about 6 years ago

  • Priority changed from Normal to High
  • Severity changed from 2. Medium to 3. High
  • Platform Release set to 2.8.3
  • Triaged changed from No to Yes
Actions #2

Updated by jokroepke about 6 years ago

Hi,

after drop the units_puppet_module collections and reinstall all pulp-puppet* packages, its looks like the sync works again.

A reinstall of all pulp-packages does not help. pulp-manage-db does not help do.

I run mongodump on the collection, before I drop it.

Actions #3

Updated by pthomas@redhat.com about 6 years ago


[root@qe-blade-03 ~]# rpm -qa pulp-server
pulp-server-2.8.3-0.1.beta.git.44.70215c3.el7.noarch
[root@qe-blade-03 ~]# 
[root@qe-blade-03 ~]# pulp-admin puppet repo create --repo-id=forge --feed=http://forge.puppetlabs.com
Successfully created repository [forge]

[root@qe-blade-03 ~]#  pulp-admin puppet repo sync run --repo-id=forge
+----------------------------------------------------------------------+
                    Synchronizing Repository [forge]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[==================================================] 100%
Metadata Query: 1/1 items
... completed

Downloading new modules...
[==================================================] 100%
Module: 4128/4128 items
... completed

Could not import the following modules:

    icecast-0.1.0: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/nwolfe-icecast-0.1.0.
tar.gz']
    windows_xmltask-0.0.3: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/noma4i-windows_xmltas
k-0.0.3.tar.gz']
    xmpp-0.0.2: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/jamtur01-xmpp-0.0.2.t
ar.gz']
    resolv_conf-3.0.5: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/zanloy-resolv_conf-3.
0.5.tar.gz']
    puppet_config-0.1.0: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/vjanelle-puppet_confi
g-0.1.0.tar.gz']
    denyhosts-0.1.0: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/puppetlabs-denyhosts-
0.1.0.tar.gz']
    demo-0.1.1: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/tobiashochguertel-dem
o-0.1.1.tar.gz']
    irc-0.0.7: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/jamtur01-irc-0.0.7.ta
r.gz']
    apachemodules-0.0.2: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/jamtur01-apachemodule
s-0.0.2.tar.gz']
    ssfe-0.1.5: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/solarflare-ssfe-0.1.5
.tar.gz']
    curator-1.0.1: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/TubeMogul-curator-1.0
.1.tar.gz']
    simmons-0.3.3: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/nwolfe-simmons-0.3.3.
tar.gz']
    apparix-0.1.2: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/zanloy-apparix-0.1.2.
tar.gz']
    hugs-1.0.1: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/kitchen-hugs-1.0.1.ta
r.gz']
    vmwaretools-2014.04.22: FileRetrievalException: Bad Request
    papertrail-0.2.0: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/jschilperoord-papertr
ail-0.2.0.tar.gz']
    xhgui-0.1.0: 'utf8' codec can't decode byte 0xbd in position 13: invalid
start byte
    passenger-0.5.0: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/marcel-passenger-0.5.
0.tar.gz']
    httpauth-0.0.3: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/jamtur01-httpauth-0.0
.3.tar.gz']
    ec2legacy-0.1.1: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/mrzarquon-ec2legacy-0
.1.1.tar.gz']

Task Succeeded

Publishing modules...
[==================================================] 100%
Module: 4089/4089 items
... completed

Generating repository metadata...
[/]
Publishing repository over HTTP...
... completed

Publishing repository over HTTPS...
[-]
... completed

Task Succeeded

[root@qe-blade-03 ~]# 
[root@qe-blade-03 ~]# pulp-admin puppet repo list
+----------------------------------------------------------------------+
                          Puppet Repositories
+----------------------------------------------------------------------+

Id:                  forge
Display Name:        None
Description:         None
Content Unit Counts: 
  Puppet Module: 4089

[root@qe-blade-03 ~]# 
Actions #4

Updated by Anonymous about 6 years ago

  • Sprint/Milestone set to 19
Actions #5

Updated by semyers about 6 years ago

  • Platform Release changed from 2.8.3 to 2.8.4
Actions #6

Updated by mhrivnak about 6 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to mhrivnak
Actions #7

Updated by mhrivnak about 6 years ago

  • Status changed from ASSIGNED to CLOSED - WORKSFORME
  • Platform Release deleted (2.8.4)
  • Triaged changed from Yes to No

Thank you for submitting this report. Unfortunately (or fortunately?) we haven't been able to reproduce the problem. If you are able to add more information to help us reproduce it, please feel free to re-open the issue.

Actions #8

Updated by lzap@redhat.com about 5 years ago

We have experienced this at customer with Satellite 6.2:

https://bugzilla.redhat.com/show_bug.cgi?id=1445625

Actions #9

Updated by daviddavis about 5 years ago

  • Status changed from CLOSED - WORKSFORME to NEW

It looks like the MacOS files are causing this. I looked at the extracted file lists for these modules and they're all of the format:

-rwxr-xr-x   1  501 user  311 Mar 27  2015 ._BillWeiss-spamassassin-1.0.5
drwxr-xr-x   7  501 user 4.0K Mar 27  2015 BillWeiss-spamassassin-1.0.5

The Pulp code is simply picking the first entry [1]. We could change it to be smarter and pick the first non-file entry.

Also worth noting is that I couldn't reproduce this in my pulp dev environment. I had to use a Satellite 6.2 box for whatever reason. I'm guessing maybe the ordering of the files is nondeterministic as sometimes the directory is listed first.

(Pdb) os.listdir(extraction_dir)
['marcel-passenger-0.5.0', '._marcel-passenger-0.5.0']

[1] https://git.io/v9OHs

Actions #10

Updated by daviddavis about 5 years ago

  • Status changed from NEW to POST
Actions #11

Updated by daviddavis about 5 years ago

  • Status changed from POST to ASSIGNED
  • Assignee changed from mhrivnak to daviddavis
Actions #12

Updated by daviddavis about 5 years ago

  • Status changed from ASSIGNED to POST

Added by daviddavis about 5 years ago

Revision e9d24ad3

Fixing bug where modules had extraneous files

Pulp was unable to find metadata in certain modules that had MacOS content because it was simply selecting the first file OR dir. Changing this to filter the list of entries to only dirs.

fixes #1846 https://pulp.plan.io/issues/1846

Actions #14

Updated by daviddavis about 5 years ago

  • Status changed from POST to MODIFIED
Actions #15

Updated by bizhang about 5 years ago

  • Platform Release set to 2.13.1
Actions #16

Updated by bizhang about 5 years ago

  • Status changed from MODIFIED to 5
Actions #17

Updated by pthomas@redhat.com about 5 years ago

[root@qe-blade-12 ~]# rpm -qa |grep pulp |sort
pulp-admin-client-2.13.1-0.1.beta.el7.noarch
pulp-docker-admin-extensions-2.4.0-0.2.rc.el7.noarch
pulp-docker-plugins-2.4.0-0.2.rc.el7.noarch
pulp-ostree-admin-extensions-1.2.1-1.2.rc.el7.noarch
pulp-ostree-plugins-1.2.1-1.2.rc.el7.noarch
pulp-puppet-admin-extensions-2.13.1-0.1.beta.el7.noarch
pulp-puppet-plugins-2.13.1-0.1.beta.el7.noarch
pulp-python-admin-extensions-2.0.0-0.3.rc.el7.noarch
pulp-python-plugins-2.0.0-0.3.rc.el7.noarch
pulp-rpm-admin-extensions-2.13.1-0.1.beta.el7.noarch
pulp-rpm-plugins-2.13.1-0.1.beta.el7.noarch
pulp-selinux-2.13.1-0.1.beta.el7.noarch
pulp-server-2.13.1-0.1.beta.el7.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
python-kombu-3.0.33-6.pulp.el7.noarch
python-pulp-bindings-2.13.1-0.1.beta.el7.noarch
python-pulp-client-lib-2.13.1-0.1.beta.el7.noarch
python-pulp-common-2.13.1-0.1.beta.el7.noarch
python-pulp-docker-common-2.4.0-0.2.rc.el7.noarch
python-pulp-oid_validation-2.13.1-0.1.beta.el7.noarch
python-pulp-ostree-common-1.2.1-1.2.rc.el7.noarch
python-pulp-puppet-common-2.13.1-0.1.beta.el7.noarch
python-pulp-python-common-2.0.0-0.3.rc.el7.noarch
python-pulp-repoauth-2.13.1-0.1.beta.el7.noarch
python-pulp-rpm-common-2.13.1-0.1.beta.el7.noarch
python-pulp-streamer-2.13.1-0.1.beta.el7.n

[root@qe-blade-12 ~]# pulp-admin puppet repo uploads  upload --repo-id issue1846 -f marcel-passenger-0.5.0.tar.gz -v
+----------------------------------------------------------------------+
                              Unit Upload
+----------------------------------------------------------------------+

Extracting necessary metadata for each request...
[==================================================] 100%
Analyzing: marcel-passenger-0.5.0.tar.gz
... completed

Files to be uploaded:
  marcel-passenger-0.5.0.tar.gz

Creating upload requests on the server...
[==================================================] 100%
Initializing: marcel-passenger-0.5.0.tar.gz
... completed

Starting upload of selected units. If this process is stopped through ctrl+c,
the uploads will be paused and may be resumed later using the resume command or
canceled entirely using the cancel command.

Uploading: marcel-passenger-0.5.0.tar.gz
[==================================================] 100%
8033/8033 bytes
... completed

Importing into the repository...
This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Task Succeeded

Deleting the upload request...
... completed

[root@qe-blade-12 ~]# pulp-admin puppet repo list
+----------------------------------------------------------------------+
                          Puppet Repositories
+----------------------------------------------------------------------+

Id:                  issue1846
Display Name:        None
Description:         None
Content Unit Counts: 
  Puppet Module: 1

[root@qe-blade-12 ~]# pulp-admin puppet repo publish run --repo-id issue1846
+----------------------------------------------------------------------+
                   Publishing Repository [issue1846]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Publishing modules...
[==================================================] 100%
Module: 1/1 items
... completed

Generating repository metadata...
[-]
... completed

Publishing repository over HTTP...
... completed

Publishing repository over HTTPS...
... skipped

Task Succeeded

[root@qe-blade-12 ~]# 
Actions #18

Updated by bizhang about 5 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE
Actions #19

Updated by bmbouter about 4 years ago

  • Sprint set to Sprint 1
Actions #20

Updated by bmbouter about 4 years ago

  • Sprint/Milestone deleted (19)
Actions #21

Updated by daviddavis over 3 years ago

  • Related to Issue #4140: Puppet fails to find metadata file if there are multiple top-level files/dirs added
Actions #22

Updated by bmbouter about 3 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF