Issue #1846
closed
pulp does not sync puppetforge correctly
Status:
CLOSED - CURRENTRELEASE
Description
Hi,
i have some problems with syncing the puppet forge.
How to reproduce (for me):
[root@pulp1 ~]# pulp-admin puppet repo sync run --repo-id=forge
+----------------------------------------------------------------------+
Synchronizing Repository [forge]
+----------------------------------------------------------------------+
This command may be exited via ctrl+c without affecting the request.
Downloading metadata...
[==================================================] 100%
Metadata Query: 1/1 items
... completed
Downloading new modules...
[==================================================] 100%
Module: 40/40 items
... completed
Could not import the following modules:
passenger-0.5.0: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/marcel-passenger-0.5.0.tar.gz']
vmwaretools-2014.04.22: FileRetrievalException: Bad Request
gitlist-0.1.2: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/eshamow-gitlist-0.1.2.tar.gz']
simpleid-0.1.0: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/eshamow-simpleid-0.1.0.tar.gz']
denyhosts-0.1.0: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/puppetlabs-denyhosts-0.1.0.tar.gz']
riemann-0.0.2: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/jamtur01-riemann-0.0.2.tar.gz']
spamassassin-1.0.5: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/BillWeiss-spamassassin-1.0.5.tar.gz']
icecast-0.1.0: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/nwolfe-icecast-0.1.0.tar.gz']
stack_wordpress-0.1.3: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/jaronson-stack_wordpress-0.1.3.tar.gz']
ec2legacy-0.1.1: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/mrzarquon-ec2legacy-0.1.1.tar.gz']
puppet_hipchat-0.6.0: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/jamtur01-puppet_hipchat-0.6.0.tar.gz']
curator-1.0.1: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/TubeMogul-curator-1.0.1.tar.gz']
xhgui-0.1.0: 'utf8' codec can't decode byte 0xbd in position 13: invalid
start byte
puppet_server-0.0.1: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/ajroetker-puppet_server-0.0.1.tar.gz']
puppet_config-0.1.0: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/vjanelle-puppet_config-0.1.0.tar.gz']
resolv_conf-3.0.5: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/zanloy-resolv_conf-3.0.5.tar.gz']
pagerduty-0.0.2: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/jamtur01-pagerduty-0.0.2.tar.gz']
strongswan-0.0.2: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/Nextdoor-strongswan-0.0.2.tar.gz']
figlet-1.0.1: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/zanloy-figlet-1.0.1.tar.gz']
xmpp-0.0.2: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-0@pulp1.mgmt.tb.noris.de/c4e2aa1b-24
b5-47f4-8382-1a75a4ad9c40/http-downloads/jamtur01-xmpp-0.0.2.tar.gz']
Task Succeeded
Publishing modules...
[==================================================] 100%
Module: 1/1 items
... completed
Generating repository metadata...
[-]
... completed
Publishing repository over HTTP...
... completed
Publishing repository over HTTPS...
... skipped
Task Succeeded
Actually Result:
Sync 40 Modules
Expected Result:
Sync all (~1800) modules.
Best Regards,
Jan-Otto Kröpke
noris network AG
- Priority changed from Normal to High
- Severity changed from 2. Medium to 3. High
- Platform Release set to 2.8.3
- Triaged changed from No to Yes
Hi,
after drop the units_puppet_module collections and reinstall all pulp-puppet* packages, its looks like the sync works again.
A reinstall of all pulp-packages does not help. pulp-manage-db does not help do.
I run mongodump on the collection, before I drop it.
[root@qe-blade-03 ~]# rpm -qa pulp-server
pulp-server-2.8.3-0.1.beta.git.44.70215c3.el7.noarch
[root@qe-blade-03 ~]#
[root@qe-blade-03 ~]# pulp-admin puppet repo create --repo-id=forge --feed=http://forge.puppetlabs.com
Successfully created repository [forge]
[root@qe-blade-03 ~]# pulp-admin puppet repo sync run --repo-id=forge
+----------------------------------------------------------------------+
Synchronizing Repository [forge]
+----------------------------------------------------------------------+
This command may be exited via ctrl+c without affecting the request.
Downloading metadata...
[==================================================] 100%
Metadata Query: 1/1 items
... completed
Downloading new modules...
[==================================================] 100%
Module: 4128/4128 items
... completed
Could not import the following modules:
icecast-0.1.0: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/nwolfe-icecast-0.1.0.
tar.gz']
windows_xmltask-0.0.3: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/noma4i-windows_xmltas
k-0.0.3.tar.gz']
xmpp-0.0.2: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/jamtur01-xmpp-0.0.2.t
ar.gz']
resolv_conf-3.0.5: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/zanloy-resolv_conf-3.
0.5.tar.gz']
puppet_config-0.1.0: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/vjanelle-puppet_confi
g-0.1.0.tar.gz']
denyhosts-0.1.0: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/puppetlabs-denyhosts-
0.1.0.tar.gz']
demo-0.1.1: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/tobiashochguertel-dem
o-0.1.1.tar.gz']
irc-0.0.7: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/jamtur01-irc-0.0.7.ta
r.gz']
apachemodules-0.0.2: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/jamtur01-apachemodule
s-0.0.2.tar.gz']
ssfe-0.1.5: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/solarflare-ssfe-0.1.5
.tar.gz']
curator-1.0.1: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/TubeMogul-curator-1.0
.1.tar.gz']
simmons-0.3.3: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/nwolfe-simmons-0.3.3.
tar.gz']
apparix-0.1.2: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/zanloy-apparix-0.1.2.
tar.gz']
hugs-1.0.1: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/kitchen-hugs-1.0.1.ta
r.gz']
vmwaretools-2014.04.22: FileRetrievalException: Bad Request
papertrail-0.2.0: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/jschilperoord-papertr
ail-0.2.0.tar.gz']
xhgui-0.1.0: 'utf8' codec can't decode byte 0xbd in position 13: invalid
start byte
passenger-0.5.0: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/marcel-passenger-0.5.
0.tar.gz']
httpauth-0.0.3: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/jamtur01-httpauth-0.0
.3.tar.gz']
ec2legacy-0.1.1: Invalid properties:
[u'/var/cache/pulp/reserved_resource_worker-7@qe-blade-03.idmqe.lab.eng.bos.redh
at.com/677a74c5-7c89-411a-8790-8ac1d2417c3f/http-downloads/mrzarquon-ec2legacy-0
.1.1.tar.gz']
Task Succeeded
Publishing modules...
[==================================================] 100%
Module: 4089/4089 items
... completed
Generating repository metadata...
[/]
Publishing repository over HTTP...
... completed
Publishing repository over HTTPS...
[-]
... completed
Task Succeeded
[root@qe-blade-03 ~]#
[root@qe-blade-03 ~]# pulp-admin puppet repo list
+----------------------------------------------------------------------+
Puppet Repositories
+----------------------------------------------------------------------+
Id: forge
Display Name: None
Description: None
Content Unit Counts:
Puppet Module: 4089
[root@qe-blade-03 ~]#
- Sprint/Milestone set to 19
- Platform Release changed from 2.8.3 to 2.8.4
- Status changed from NEW to ASSIGNED
- Assignee set to mhrivnak
- Status changed from ASSIGNED to CLOSED - WORKSFORME
- Platform Release deleted (
2.8.4)
- Triaged changed from Yes to No
Thank you for submitting this report. Unfortunately (or fortunately?) we haven't been able to reproduce the problem. If you are able to add more information to help us reproduce it, please feel free to re-open the issue.
- Status changed from CLOSED - WORKSFORME to NEW
It looks like the MacOS files are causing this. I looked at the extracted file lists for these modules and they're all of the format:
-rwxr-xr-x 1 501 user 311 Mar 27 2015 ._BillWeiss-spamassassin-1.0.5
drwxr-xr-x 7 501 user 4.0K Mar 27 2015 BillWeiss-spamassassin-1.0.5
The Pulp code is simply picking the first entry [1]. We could change it to be smarter and pick the first non-file entry.
Also worth noting is that I couldn't reproduce this in my pulp dev environment. I had to use a Satellite 6.2 box for whatever reason. I'm guessing maybe the ordering of the files is nondeterministic as sometimes the directory is listed first.
(Pdb) os.listdir(extraction_dir)
['marcel-passenger-0.5.0', '._marcel-passenger-0.5.0']
[1] https://git.io/v9OHs
- Status changed from NEW to POST
- Status changed from POST to ASSIGNED
- Assignee changed from mhrivnak to daviddavis
- Status changed from ASSIGNED to POST
- Status changed from POST to MODIFIED
- Platform Release set to 2.13.1
- Status changed from MODIFIED to 5
[root@qe-blade-12 ~]# rpm -qa |grep pulp |sort
pulp-admin-client-2.13.1-0.1.beta.el7.noarch
pulp-docker-admin-extensions-2.4.0-0.2.rc.el7.noarch
pulp-docker-plugins-2.4.0-0.2.rc.el7.noarch
pulp-ostree-admin-extensions-1.2.1-1.2.rc.el7.noarch
pulp-ostree-plugins-1.2.1-1.2.rc.el7.noarch
pulp-puppet-admin-extensions-2.13.1-0.1.beta.el7.noarch
pulp-puppet-plugins-2.13.1-0.1.beta.el7.noarch
pulp-python-admin-extensions-2.0.0-0.3.rc.el7.noarch
pulp-python-plugins-2.0.0-0.3.rc.el7.noarch
pulp-rpm-admin-extensions-2.13.1-0.1.beta.el7.noarch
pulp-rpm-plugins-2.13.1-0.1.beta.el7.noarch
pulp-selinux-2.13.1-0.1.beta.el7.noarch
pulp-server-2.13.1-0.1.beta.el7.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
python-kombu-3.0.33-6.pulp.el7.noarch
python-pulp-bindings-2.13.1-0.1.beta.el7.noarch
python-pulp-client-lib-2.13.1-0.1.beta.el7.noarch
python-pulp-common-2.13.1-0.1.beta.el7.noarch
python-pulp-docker-common-2.4.0-0.2.rc.el7.noarch
python-pulp-oid_validation-2.13.1-0.1.beta.el7.noarch
python-pulp-ostree-common-1.2.1-1.2.rc.el7.noarch
python-pulp-puppet-common-2.13.1-0.1.beta.el7.noarch
python-pulp-python-common-2.0.0-0.3.rc.el7.noarch
python-pulp-repoauth-2.13.1-0.1.beta.el7.noarch
python-pulp-rpm-common-2.13.1-0.1.beta.el7.noarch
python-pulp-streamer-2.13.1-0.1.beta.el7.n
[root@qe-blade-12 ~]# pulp-admin puppet repo uploads upload --repo-id issue1846 -f marcel-passenger-0.5.0.tar.gz -v
+----------------------------------------------------------------------+
Unit Upload
+----------------------------------------------------------------------+
Extracting necessary metadata for each request...
[==================================================] 100%
Analyzing: marcel-passenger-0.5.0.tar.gz
... completed
Files to be uploaded:
marcel-passenger-0.5.0.tar.gz
Creating upload requests on the server...
[==================================================] 100%
Initializing: marcel-passenger-0.5.0.tar.gz
... completed
Starting upload of selected units. If this process is stopped through ctrl+c,
the uploads will be paused and may be resumed later using the resume command or
canceled entirely using the cancel command.
Uploading: marcel-passenger-0.5.0.tar.gz
[==================================================] 100%
8033/8033 bytes
... completed
Importing into the repository...
This command may be exited via ctrl+c without affecting the request.
[\]
Running...
Task Succeeded
Deleting the upload request...
... completed
[root@qe-blade-12 ~]# pulp-admin puppet repo list
+----------------------------------------------------------------------+
Puppet Repositories
+----------------------------------------------------------------------+
Id: issue1846
Display Name: None
Description: None
Content Unit Counts:
Puppet Module: 1
[root@qe-blade-12 ~]# pulp-admin puppet repo publish run --repo-id issue1846
+----------------------------------------------------------------------+
Publishing Repository [issue1846]
+----------------------------------------------------------------------+
This command may be exited via ctrl+c without affecting the request.
Publishing modules...
[==================================================] 100%
Module: 1/1 items
... completed
Generating repository metadata...
[-]
... completed
Publishing repository over HTTP...
... completed
Publishing repository over HTTPS...
... skipped
Task Succeeded
[root@qe-blade-12 ~]#
- Status changed from 5 to CLOSED - CURRENTRELEASE
- Sprint/Milestone deleted (
19)
- Related to Issue #4140: Puppet fails to find metadata file if there are multiple top-level files/dirs added
Also available in: Atom
PDF
.
Fixing bug where modules had extraneous files
Pulp was unable to find metadata in certain modules that had MacOS content because it was simply selecting the first file OR dir. Changing this to filter the list of entries to only dirs.
fixes #1846 https://pulp.plan.io/issues/1846