Project

Profile

Help

Issue #1846

pulp does not sync puppetforge correctly

Added by jokroepke almost 5 years ago. Updated almost 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
2.8.0
Platform Release:
2.13.1
OS:
CentOS 7
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 1
Quarter:

Description

Hi,

i have some problems with syncing the puppet forge.

How to reproduce (for me):

[root@pulp1 ~]#  pulp-admin  puppet repo sync run --repo-id=forge
+----------------------------------------------------------------------+
                    Synchronizing Repository [forge]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[==================================================] 100%
Metadata Query: 1/1 items
... completed

Downloading new modules...
[==================================================] 100%
Module: 40/40 items
... completed

Could not import the following modules:

    passenger-0.5.0: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/marcel-passenger-0.5.0.tar.gz']
    vmwaretools-2014.04.22: FileRetrievalException: Bad Request
    gitlist-0.1.2: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/eshamow-gitlist-0.1.2.tar.gz']
    simpleid-0.1.0: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/eshamow-simpleid-0.1.0.tar.gz']
    denyhosts-0.1.0: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/puppetlabs-denyhosts-0.1.0.tar.gz']
    riemann-0.0.2: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/jamtur01-riemann-0.0.2.tar.gz']
    spamassassin-1.0.5: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/BillWeiss-spamassassin-1.0.5.tar.gz']
    icecast-0.1.0: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/nwolfe-icecast-0.1.0.tar.gz']
    stack_wordpress-0.1.3: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/jaronson-stack_wordpress-0.1.3.tar.gz']
    ec2legacy-0.1.1: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/mrzarquon-ec2legacy-0.1.1.tar.gz']
    puppet_hipchat-0.6.0: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/jamtur01-puppet_hipchat-0.6.0.tar.gz']
    curator-1.0.1: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/TubeMogul-curator-1.0.1.tar.gz']
    xhgui-0.1.0: 'utf8' codec can't decode byte 0xbd in position 13: invalid
start byte
    puppet_server-0.0.1: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/ajroetker-puppet_server-0.0.1.tar.gz']
    puppet_config-0.1.0: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/vjanelle-puppet_config-0.1.0.tar.gz']
    resolv_conf-3.0.5: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/zanloy-resolv_conf-3.0.5.tar.gz']
    pagerduty-0.0.2: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/jamtur01-pagerduty-0.0.2.tar.gz']
    strongswan-0.0.2: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/Nextdoor-strongswan-0.0.2.tar.gz']
    figlet-1.0.1: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/zanloy-figlet-1.0.1.tar.gz']
    xmpp-0.0.2: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/jamtur01-xmpp-0.0.2.tar.gz']

Task Succeeded

Publishing modules...
[==================================================] 100%
Module: 1/1 items
... completed

Generating repository metadata...
[-]
... completed

Publishing repository over HTTP...
... completed

Publishing repository over HTTPS...
... skipped

Task Succeeded

Actually Result:

Sync 40 Modules

Expected Result:

Sync all (~1800) modules.

Best Regards,
Jan-Otto Kröpke
noris network AG


Related issues

Related to Puppet Support - Issue #4140: Puppet fails to find metadata file if there are multiple top-level files/dirsCLOSED - CURRENTRELEASE<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision e9d24ad3 View on GitHub
Added by daviddavis over 3 years ago

Fixing bug where modules had extraneous files

Pulp was unable to find metadata in certain modules that had MacOS content because it was simply selecting the first file OR dir. Changing this to filter the list of entries to only dirs.

fixes #1846 https://pulp.plan.io/issues/1846

History

#1 Updated by mhrivnak almost 5 years ago

  • Priority changed from Normal to High
  • Severity changed from 2. Medium to 3. High
  • Platform Release set to 2.8.3
  • Triaged changed from No to Yes

#2 Updated by jokroepke almost 5 years ago

Hi,

after drop the units_puppet_module collections and reinstall all pulp-puppet* packages, its looks like the sync works again.

A reinstall of all pulp-packages does not help. pulp-manage-db does not help do.

I run mongodump on the collection, before I drop it.

#3 Updated by pthomas@redhat.com almost 5 years ago


[root@qe-blade-03 ~]# rpm -qa pulp-server
pulp-server-2.8.3-0.1.beta.git.44.70215c3.el7.noarch
[root@qe-blade-03 ~]# 
[root@qe-blade-03 ~]# pulp-admin puppet repo create --repo-id=forge --feed=http://forge.puppetlabs.com
Successfully created repository [forge]

[root@qe-blade-03 ~]#  pulp-admin puppet repo sync run --repo-id=forge
+----------------------------------------------------------------------+
                    Synchronizing Repository [forge]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[==================================================] 100%
Metadata Query: 1/1 items
... completed

Downloading new modules...
[==================================================] 100%
Module: 4128/4128 items
... completed

Could not import the following modules:

    icecast-0.1.0: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/nwolfe-icecast-0.1.0.
tar.gz']
    windows_xmltask-0.0.3: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/noma4i-windows_xmltas
k-0.0.3.tar.gz']
    xmpp-0.0.2: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/jamtur01-xmpp-0.0.2.t
ar.gz']
    resolv_conf-3.0.5: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/zanloy-resolv_conf-3.
0.5.tar.gz']
    puppet_config-0.1.0: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/vjanelle-puppet_confi
g-0.1.0.tar.gz']
    denyhosts-0.1.0: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/puppetlabs-denyhosts-
0.1.0.tar.gz']
    demo-0.1.1: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/tobiashochguertel-dem
o-0.1.1.tar.gz']
    irc-0.0.7: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/jamtur01-irc-0.0.7.ta
r.gz']
    apachemodules-0.0.2: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/jamtur01-apachemodule
s-0.0.2.tar.gz']
    ssfe-0.1.5: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/solarflare-ssfe-0.1.5
.tar.gz']
    curator-1.0.1: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/TubeMogul-curator-1.0
.1.tar.gz']
    simmons-0.3.3: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/nwolfe-simmons-0.3.3.
tar.gz']
    apparix-0.1.2: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/zanloy-apparix-0.1.2.
tar.gz']
    hugs-1.0.1: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/kitchen-hugs-1.0.1.ta
r.gz']
    vmwaretools-2014.04.22: FileRetrievalException: Bad Request
    papertrail-0.2.0: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/jschilperoord-papertr
ail-0.2.0.tar.gz']
    xhgui-0.1.0: 'utf8' codec can't decode byte 0xbd in position 13: invalid
start byte
    passenger-0.5.0: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/marcel-passenger-0.5.
0.tar.gz']
    httpauth-0.0.3: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/jamtur01-httpauth-0.0
.3.tar.gz']
    ec2legacy-0.1.1: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/mrzarquon-ec2legacy-0
.1.1.tar.gz']

Task Succeeded

Publishing modules...
[==================================================] 100%
Module: 4089/4089 items
... completed

Generating repository metadata...
[/]
Publishing repository over HTTP...
... completed

Publishing repository over HTTPS...
[-]
... completed

Task Succeeded

[root@qe-blade-03 ~]# 
[root@qe-blade-03 ~]# pulp-admin puppet repo list
+----------------------------------------------------------------------+
                          Puppet Repositories
+----------------------------------------------------------------------+

Id:                  forge
Display Name:        None
Description:         None
Content Unit Counts: 
  Puppet Module: 4089

[root@qe-blade-03 ~]# 

#4 Updated by Anonymous over 4 years ago

  • Sprint/Milestone set to 19

#5 Updated by semyers over 4 years ago

  • Platform Release changed from 2.8.3 to 2.8.4

#6 Updated by mhrivnak over 4 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to mhrivnak

#7 Updated by mhrivnak over 4 years ago

  • Status changed from ASSIGNED to CLOSED - WORKSFORME
  • Platform Release deleted (2.8.4)
  • Triaged changed from Yes to No

Thank you for submitting this report. Unfortunately (or fortunately?) we haven't been able to reproduce the problem. If you are able to add more information to help us reproduce it, please feel free to re-open the issue.

#8 Updated by lzap@redhat.com over 3 years ago

We have experienced this at customer with Satellite 6.2:

https://bugzilla.redhat.com/show_bug.cgi?id=1445625

#9 Updated by daviddavis over 3 years ago

  • Status changed from CLOSED - WORKSFORME to NEW

It looks like the MacOS files are causing this. I looked at the extracted file lists for these modules and they're all of the format:

-rwxr-xr-x   1  501 user  311 Mar 27  2015 ._BillWeiss-spamassassin-1.0.5
drwxr-xr-x   7  501 user 4.0K Mar 27  2015 BillWeiss-spamassassin-1.0.5

The Pulp code is simply picking the first entry [1]. We could change it to be smarter and pick the first non-file entry.

Also worth noting is that I couldn't reproduce this in my pulp dev environment. I had to use a Satellite 6.2 box for whatever reason. I'm guessing maybe the ordering of the files is nondeterministic as sometimes the directory is listed first.

(Pdb) os.listdir(extraction_dir)
['marcel-passenger-0.5.0', '._marcel-passenger-0.5.0']

[1] https://git.io/v9OHs

#10 Updated by daviddavis over 3 years ago

  • Status changed from NEW to POST

#11 Updated by daviddavis over 3 years ago

  • Status changed from POST to ASSIGNED
  • Assignee changed from mhrivnak to daviddavis

#12 Updated by daviddavis over 3 years ago

  • Status changed from ASSIGNED to POST

#14 Updated by daviddavis over 3 years ago

  • Status changed from POST to MODIFIED

#15 Updated by bizhang over 3 years ago

  • Platform Release set to 2.13.1

#16 Updated by bizhang over 3 years ago

  • Status changed from MODIFIED to 5

#17 Updated by pthomas@redhat.com over 3 years ago

[root@qe-blade-12 ~]# rpm -qa |grep pulp |sort
pulp-admin-client-2.13.1-0.1.beta.el7.noarch
pulp-docker-admin-extensions-2.4.0-0.2.rc.el7.noarch
pulp-docker-plugins-2.4.0-0.2.rc.el7.noarch
pulp-ostree-admin-extensions-1.2.1-1.2.rc.el7.noarch
pulp-ostree-plugins-1.2.1-1.2.rc.el7.noarch
pulp-puppet-admin-extensions-2.13.1-0.1.beta.el7.noarch
pulp-puppet-plugins-2.13.1-0.1.beta.el7.noarch
pulp-python-admin-extensions-2.0.0-0.3.rc.el7.noarch
pulp-python-plugins-2.0.0-0.3.rc.el7.noarch
pulp-rpm-admin-extensions-2.13.1-0.1.beta.el7.noarch
pulp-rpm-plugins-2.13.1-0.1.beta.el7.noarch
pulp-selinux-2.13.1-0.1.beta.el7.noarch
pulp-server-2.13.1-0.1.beta.el7.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
python-kombu-3.0.33-6.pulp.el7.noarch
python-pulp-bindings-2.13.1-0.1.beta.el7.noarch
python-pulp-client-lib-2.13.1-0.1.beta.el7.noarch
python-pulp-common-2.13.1-0.1.beta.el7.noarch
python-pulp-docker-common-2.4.0-0.2.rc.el7.noarch
python-pulp-oid_validation-2.13.1-0.1.beta.el7.noarch
python-pulp-ostree-common-1.2.1-1.2.rc.el7.noarch
python-pulp-puppet-common-2.13.1-0.1.beta.el7.noarch
python-pulp-python-common-2.0.0-0.3.rc.el7.noarch
python-pulp-repoauth-2.13.1-0.1.beta.el7.noarch
python-pulp-rpm-common-2.13.1-0.1.beta.el7.noarch
python-pulp-streamer-2.13.1-0.1.beta.el7.n

[root@qe-blade-12 ~]# pulp-admin puppet repo uploads  upload --repo-id issue1846 -f marcel-passenger-0.5.0.tar.gz -v
+----------------------------------------------------------------------+
                              Unit Upload
+----------------------------------------------------------------------+

Extracting necessary metadata for each request...
[==================================================] 100%
Analyzing: marcel-passenger-0.5.0.tar.gz
... completed

Files to be uploaded:
  marcel-passenger-0.5.0.tar.gz

Creating upload requests on the server...
[==================================================] 100%
Initializing: marcel-passenger-0.5.0.tar.gz
... completed

Starting upload of selected units. If this process is stopped through ctrl+c,
the uploads will be paused and may be resumed later using the resume command or
canceled entirely using the cancel command.

Uploading: marcel-passenger-0.5.0.tar.gz
[==================================================] 100%
8033/8033 bytes
... completed

Importing into the repository...
This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Task Succeeded

Deleting the upload request...
... completed

[root@qe-blade-12 ~]# pulp-admin puppet repo list
+----------------------------------------------------------------------+
                          Puppet Repositories
+----------------------------------------------------------------------+

Id:                  issue1846
Display Name:        None
Description:         None
Content Unit Counts: 
  Puppet Module: 1

[root@qe-blade-12 ~]# pulp-admin puppet repo publish run --repo-id issue1846
+----------------------------------------------------------------------+
                   Publishing Repository [issue1846]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Publishing modules...
[==================================================] 100%
Module: 1/1 items
... completed

Generating repository metadata...
[-]
... completed

Publishing repository over HTTP...
... completed

Publishing repository over HTTPS...
... skipped

Task Succeeded

[root@qe-blade-12 ~]# 

#18 Updated by bizhang over 3 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE

#19 Updated by bmbouter almost 3 years ago

  • Sprint set to Sprint 1

#20 Updated by bmbouter almost 3 years ago

  • Sprint/Milestone deleted (19)

#21 Updated by daviddavis about 2 years ago

  • Related to Issue #4140: Puppet fails to find metadata file if there are multiple top-level files/dirs added

#22 Updated by bmbouter almost 2 years ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF