Project

Profile

Help

Issue #1388

pulp forge sends wrong dependencies for modules

Added by cristi.falcas@gmail.com almost 5 years ago. Updated over 1 year ago.

Status:
CLOSED - NOTABUG
Priority:
High
Assignee:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
2.7.0
Platform Release:
2.8.1
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

When querying the pulp forge for module dependencies, it also sends dependencies from old modules.

Let me explain:
- have moduleA, version 1.0 that depends on moduleB
- have moduleA, version 2.0 that does not depend on moduleB
- have moduleC, that depends on moduleA, version 2.0
- pulp forge will list moduleB as a dependency for moduleC

How to test:

  1. get an old module of rabbitmq (this one depends on puppetlabs-prosvc_repo) and import it in pulp
    wget https://forge.puppetlabs.com/v3/files/puppetlabs-rabbitmq-1.0.4.tar.gz
    pulp-admin puppet repo uploads upload --file ./puppetlabs-rabbitmq-1.0.4.tar.gz --repo-id test
  1. get a module that depends on newer version of rabbitmq
    puppet module install stackforge-openstacklib --target-dir /tmp/test --config /dev/null
  2. import it and all dependencies
    cd /tmp/test; ls | while read module;do puppet module build ./$module; pulp-admin puppet repo uploads upload --dir ./$module/pkg/ --repo-id test;done
  1. publish repo
    pulp-admin puppet repo publish run --repo-id test
  1. try to install module from local repo
    puppet module --module_repository http://repository:test@localhost install stackforge-openstacklib --target-dir /tmp/test1 --config /dev/null
    Notice: Preparing to install into /tmp/test1 ...
    Notice: Created target directory /tmp/test1
    Notice: Downloading from http://repository:test@localhost ...
    Error: Request to Puppet Forge failed.
    The server being queried was http://repository:test@localhost/v3/releases?module=puppetlabs-prosvc_repo
    The HTTP response we received was '404 NOT FOUND'
  1. delete old version of rabbitmq
    pulp-admin puppet repo remove --repo-id test --match "name=rabbitmq*" --match "version=1.0.4*"
    pulp-admin puppet repo publish run --repo-id test
  1. install now works
    puppet module --module_repository http://repository:test@localhost install stackforge-openstacklib --target-dir /tmp/test1 --config /dev/null

History

#1 Updated by jortel@redhat.com almost 5 years ago

  • Priority changed from Normal to High
  • Severity changed from 2. Medium to 3. High
  • Triaged changed from No to Yes

#2 Updated by dkliban@redhat.com over 4 years ago

  • Platform Release set to 2.8.1

#3 Updated by mhrivnak over 4 years ago

  • Project changed from Pulp to Puppet Support

#4 Updated by dkliban@redhat.com over 4 years ago

  • Status changed from NEW to CLOSED - NOTABUG

After doing some investigation it looks like Pulp is doing exactly what Puppet Forge /v3/releases API endpoint does. This is more of a bug with how the Puppet client handles dependency resolution. Below is the debug output of installing the same openstacklib moduel from Puppet Forge. You will notice that in this case the client also makes a request for "https://forgeapi.puppetlabs.com/v3/releases?module=puppetlabs-prosvc_repo". This request is successful because Puppet Forge database is aware ofall the dependencies for the older version of rabbitmq. The Pulp repo created in the example above does not know about all those dependencies.

[vagrant@dev ~]$ puppet module install stackforge-openstacklib --target-dir /tmp/test --config /dev/null --debug
Debug: Runtime environment: puppet_version=4.2.1, ruby_version=2.2.4, run_mode=user, default_encoding=UTF-8
Notice: Preparing to install into /tmp/test ...
Notice: Downloading from https://forgeapi.puppetlabs.com ...
Debug: HTTP GET https://forgeapi.puppetlabs.com/v3/releases?module=stackforge-openstacklib
Debug: Evicting cache entry for environment 'production'
Debug: Caching environment 'production' (ttl = 0 sec)
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: HTTP GET https://forgeapi.puppetlabs.com/v3/releases?module=aimonb-aviator
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: HTTP GET https://forgeapi.puppetlabs.com/v3/releases?module=puppetlabs-mysql
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: HTTP GET https://forgeapi.puppetlabs.com/v3/releases?module=puppetlabs-mysql&limit=20&offset=20
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: HTTP GET https://forgeapi.puppetlabs.com/v3/releases?module=puppetlabs-stdlib
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: HTTP GET https://forgeapi.puppetlabs.com/v3/releases?module=puppetlabs-stdlib&limit=20&offset=20
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: HTTP GET https://forgeapi.puppetlabs.com/v3/releases?module=puppetlabs-stdlib&limit=20&offset=40
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: HTTP GET https://forgeapi.puppetlabs.com/v3/releases?module=nanliu-staging
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: HTTP GET https://forgeapi.puppetlabs.com/v3/releases?module=bodepd-create_resources
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: HTTP GET https://forgeapi.puppetlabs.com/v3/releases?module=puppetlabs-rabbitmq
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: HTTP GET https://forgeapi.puppetlabs.com/v3/releases?module=puppetlabs-apt
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: HTTP GET https://forgeapi.puppetlabs.com/v3/releases?module=puppetlabs-apt&limit=20&offset=20
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: HTTP GET https://forgeapi.puppetlabs.com/v3/releases?module=garethr-erlang
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: HTTP GET https://forgeapi.puppetlabs.com/v3/releases?module=stahnma-epel
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: HTTP GET https://forgeapi.puppetlabs.com/v3/releases?module=puppetlabs-prosvc_repo
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Info: Resolving dependencies ...
Info: Preparing to install ...
Debug: HTTP GET https://forgeapi.puppetlabs.com/v3/files/stackforge-openstacklib-5.1.0.tar.gz
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: Executing: 'gzip -dc /home/vagrant/.puppet/cache/puppet-module/cache/stackforge-openstacklib20160311-15147-1hsicwx | tar xof
 -'
Debug: Executing: 'find . -type d -exec chmod 755 {} +'
Debug: Executing: 'find . -type f -exec chmod u+rw,g+r,a-st {} +'
Debug: Executing: 'chown -R 1001:1001 .'
Debug: HTTP GET https://forgeapi.puppetlabs.com/v3/files/aimonb-aviator-0.5.1.tar.gz
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: Executing: 'gzip -dc /home/vagrant/.puppet/cache/puppet-module/cache/aimonb-aviator20160311-15147-1dqrxin | tar xof -'
Debug: Executing: 'find . -type d -exec chmod 755 {} +'
Debug: Executing: 'find . -type f -exec chmod u+rw,g+r,a-st {} +'
Debug: Executing: 'chown -R 1001:1001 .'
Debug: HTTP GET https://forgeapi.puppetlabs.com/v3/files/puppetlabs-mysql-2.3.1.tar.gz
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: Executing: 'gzip -dc /home/vagrant/.puppet/cache/puppet-module/cache/puppetlabs-mysql20160311-15147-hztryd | tar xof -'
Debug: Executing: 'find . -type d -exec chmod 755 {} +'
Debug: Executing: 'find . -type f -exec chmod u+rw,g+r,a-st {} +'
Debug: Executing: 'chown -R 1001:1001 .'
Debug: HTTP GET https://forgeapi.puppetlabs.com/v3/files/puppetlabs-rabbitmq-3.1.0.tar.gz
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: Executing: 'gzip -dc /home/vagrant/.puppet/cache/puppet-module/cache/puppetlabs-rabbitmq20160311-15147-nqmkn7 | tar xof -'
Debug: Executing: 'find . -type d -exec chmod 755 {} +'
Debug: Executing: 'find . -type f -exec chmod u+rw,g+r,a-st {} +'
Debug: Executing: 'chown -R 1001:1001 .'
Debug: HTTP GET https://forgeapi.puppetlabs.com/v3/files/garethr-erlang-0.3.0.tar.gz
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: Executing: 'gzip -dc /home/vagrant/.puppet/cache/puppet-module/cache/garethr-erlang20160311-15147-1bw66wi | tar xof -'
Debug: Executing: 'find . -type d -exec chmod 755 {} +'
Debug: Executing: 'find . -type f -exec chmod u+rw,g+r,a-st {} +'
Debug: Executing: 'chown -R 1001:1001 .'
Debug: HTTP GET https://forgeapi.puppetlabs.com/v3/files/puppetlabs-apt-2.2.2.tar.gz
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: Executing: 'gzip -dc /home/vagrant/.puppet/cache/puppet-module/cache/puppetlabs-apt20160311-15147-7orkij | tar xof -'
Debug: Executing: 'find . -type d -exec chmod 755 {} +'
Debug: Executing: 'find . -type f -exec chmod u+rw,g+r,a-st {} +'
Debug: Executing: 'chown -R 1001:1001 .'
Debug: HTTP GET https://forgeapi.puppetlabs.com/v3/files/puppetlabs-stdlib-4.11.0.tar.gz
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: Executing: 'gzip -dc /home/vagrant/.puppet/cache/puppet-module/cache/puppetlabs-stdlib20160311-15147-vnmpjw | tar xof -'
Debug: Executing: 'find . -type d -exec chmod 755 {} +'
Debug: Executing: 'find . -type f -exec chmod u+rw,g+r,a-st {} +'
Debug: Executing: 'chown -R 1001:1001 .'
Debug: HTTP GET https://forgeapi.puppetlabs.com/v3/files/stahnma-epel-1.2.2.tar.gz
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: Executing: 'gzip -dc /home/vagrant/.puppet/cache/puppet-module/cache/stahnma-epel20160311-15147-jj65v0 | tar xof -'
Debug: Executing: 'find . -type d -exec chmod 755 {} +'
Debug: Executing: 'find . -type f -exec chmod u+rw,g+r,a-st {} +'
Debug: Executing: 'chown -R 1001:1001 .'
Notice: Installing -- do not interrupt ...
/tmp/test
└─┬ stackforge-openstacklib (v5.1.0)
  ├── aimonb-aviator (v0.5.1)
  ├─┬ puppetlabs-mysql (v2.3.1)
  │ └── puppetlabs-stdlib (v4.11.0)
  └─┬ puppetlabs-rabbitmq (v3.1.0)
    ├─┬ garethr-erlang (v0.3.0)
    │ └── stahnma-epel (v1.2.2)
    └── puppetlabs-apt (v2.2.2)

When I attempted to reproduce the bug, I got the following output:

[vagrant@dev test]$ puppet module --module_repository http://repository:test@localhost install stackforge-openstacklib --target-dir /tmp/test1 --config /dev/null --debug
Debug: Runtime environment: puppet_version=4.2.1, ruby_version=2.2.4, run_mode=user, default_encoding=UTF-8
Notice: Preparing to install into /tmp/test1 ...
Notice: Downloading from http://repository:test@localhost ...
Debug: HTTP GET http://repository:test@localhost/v3/releases?module=stackforge-openstacklib
Debug: Evicting cache entry for environment 'production'
Debug: Caching environment 'production' (ttl = 0 sec)
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: HTTP GET http://repository:test@localhost/v3/releases?module=aimonb-aviator
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: HTTP GET http://repository:test@localhost/v3/releases?module=puppetlabs-mysql
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: HTTP GET http://repository:test@localhost/v3/releases?module=puppetlabs-stdlib
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: HTTP GET http://repository:test@localhost/v3/releases?module=puppetlabs-rabbitmq
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: HTTP GET http://repository:test@localhost/v3/releases?module=puppetlabs-apt
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: HTTP GET http://repository:test@localhost/v3/releases?module=garethr-erlang
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: HTTP GET http://repository:test@localhost/v3/releases?module=stahnma-epel
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: HTTP GET http://repository:test@localhost/v3/releases?module=puppetlabs-prosvc_repo
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Error: Request to Puppet Forge failed.
  The server being queried was http://repository:test@localhost/v3/releases?module=puppetlabs-prosvc_repo
  The HTTP response we received was '404 NOT FOUND'

The output is the same as when installing from Puppet Forge until you get to the request that fails. The failing request is for information about puppetlabs-prosvc_repo module. This module is a dependency for the older rabbitmq 1.0.4 module. rabbitmq 1.0.4 is not a dependency of the openstacklib module, but a new version of it is. If you look at the successful output, you will notice that information about it is requested, but it is actually never installed. My theory is that the client just queries for information about all the versions of the dependencies and then decides which ones it actually needs to download.

Because puppetlabs-prosvc_repo is no longer available I could not verify that uploading it to the Pulp repo would resolve the problem. However, the metadata for this module does seem to still be available. Though you cannot download the module itself from the Puppet Forge.

#5 Updated by bmbouter over 1 year ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF