Project

Profile

Help

Story #1536

As a user I would like support for v3/files API so that r10k may work

Added by tarrantmarshall over 3 years ago. Updated about 1 month ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
% Done:

100%

Platform Release:
2.8.0
Blocks Release:
Backwards Incompatible:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:

Description

The r10k system for puppet relies on a lot more of the v3 forge API to download modules
Ultimately it needs a list of releases, dependencies and file URI's - utilising what Forge would call their search API. Knowing Search isn't supported yet in the pulp puppet plugin, we could do some partial support to get r10k over the line which is commonly used.

From my discovery so far a new files endpoint, and release needs an extension.

v3/files
v3/releases/<author>-<name>-<version>

r10k makes these requests

GET /v3/files/ceh-nagios-1.2.0.tar.gz
GET /v3/modules/ceh-nagios
GET /v3/releases/ceh-nagios-1.2.0  #Currently gives a 400 error as it expects ?module=. Th

/v3/releases/ceh-nagios-1.2.0 currently gives a 400 error, and the output is different to what /v3/releases?modulename=ceh-nagios would be as the former is for a specific version.

Requests are made by the PuppetForge.gem used by r10k

I have hacked together some of the v3/files API response already to gauge how much of a task this would be and so far r10k seems happy with the mocked/partial response.
https://github.com/tarrantmarshall/pulp_puppet/blob/master/pulp_puppet_plugins/pulp_puppet/forge/views/files.py

Associated revisions

Revision ba207e5a View on GitHub
Added by Eric Parton over 3 years ago

Added support for the /releases/{user}-{module}-{version} endpoint of the Puppet Forge v3 API

https://pulp.plan.io/issues/1536
closes #1536

History

#1 Updated by lastmikoi over 3 years ago

I second this request as Librarian-puppet, which uses the /v3/modules API endpoint, is currently unable to use a pulp_puppet-powered forge.

This issue, combined with https://pulp.plan.io/issues/945 effectively makes it impossible to use pulp_puppet in a production environment where multiple module versions are to be used from a same repository.

#2 Updated by etparton over 3 years ago

I’ve been looking into this issue as well, and it turns out that the “/v3/files/whatever.tar.gz” endpoint isn’t a part of the forge api; it’s just the location that the official Puppetlabs puppet forge hosts their module archive files. The json response from the releases endpoint contains a “file_uri” property which the PuppetForge gem can use to download the file:

{
    "file_md5": "74b0f2e2282c35c02b91d40911b7835e",
    "file_uri": "/pulp/puppet/test/system/releases/c/camptocamp/camptocamp-postfix-1.2.14.tar.gz",
    "metadata": {
        "dependencies": [
            {
                "name": "puppetlabs/stdlib",
                "version_requirement": ">=3.2.0 <5.0.0" 
            },
            {
                "name": "camptocamp/augeas",
                "version_requirement": ">=1.0.0 <2.0.0" 
            }
        ],
        "name": "camptocamp/postfix",
        "version": "1.2.14" 
    }
}

I submitted a pull request (https://github.com/puppetlabs/forge-ruby/pull/24) to have the PuppetForge gem use the file_uri property instead of the hardcoded “/v3/files/” uri and one of their developers just merged it and cut a new release which will be automatically included in the current version of r10k.

I've also submitted a pull request (https://github.com/pulp/pulp_puppet/pull/207) for an implementation of the "/v3/releases/user-module-version” endpoint to the pulp-puppet repo. I’ve been testing the latest version of r10k with my Pulp development server that has the code from the pull request installed and it all works together perfectly.

The only caveat is that r10k requires the module api to download modules whose versions have not been not specified at all (it defaults to the latest version in this case). And as lastmikoi said earlier, librarian-puppet requires the module api in order to download anything.

#3 Updated by tarrantmarshall over 3 years ago

etparton wrote:

I submitted a pull request (https://github.com/puppetlabs/forge-ruby/pull/24) to have the PuppetForge gem use the file_uri property instead of the hardcoded “/v3/files/” uri and one of their developers just merged it and cut a new release which will be automatically included in the current version of r10k.

Ah. I coded a work around on my branch to basically return a HTTP REDIRECT to the actual file. PuppetForge gem follows it correctly.

I've also submitted a pull request (https://github.com/pulp/pulp_puppet/pull/207) for an implementation of the "/v3/releases/user-module-version” endpoint to the pulp-puppet repo. I’ve been testing the latest version of r10k with my Pulp development server that has the code from the pull request installed and it all works together perfectly.

Awesome! I was yet to code this part.

The only caveat is that r10k requires the module api to download modules whose versions have not been not specified at all (it defaults to the latest version in this case). And as lastmikoi said earlier, librarian-puppet requires the module api in order to download anything.

I have, for the most part, done the /modules part of the API and started on some unit tests. I'm not sure i'll have time in the next 10 days to complete it though.

https://github.com/aeria/pulp_puppet/commit/b509c05b2cdca26ed91347116d9ca9f6637eed3a

#4 Updated by Anonymous about 3 years ago

  • Status changed from NEW to MODIFIED
  • % Done changed from 0 to 100

#5 Updated by dkliban@redhat.com about 3 years ago

  • Status changed from MODIFIED to ON_QA
  • Platform Release set to 2.8.0

#6 Updated by dkliban@redhat.com about 3 years ago

  • Status changed from ON_QA to CLOSED - CURRENTRELEASE

#7 Updated by lastmikoi about 3 years ago

I believe this issue should be re-opened, as https://github.com/pulp/pulp_puppet/pull/207 implemented the `/v3/releases/module-name` endpoint but both the `/v3/files` (apparently used by r10k) and `/v3/modules` (used in my setup of librarian-puppet) are still missing from the API.

I don't know if r10k is able to use pulp_puppet as of now, but my librarian-puppet setup is failing because of the missing `/v3/modules` endpoint.

#8 Updated by etparton about 3 years ago

lastmikoi wrote:

I believe this issue should be re-opened, as https://github.com/pulp/pulp_puppet/pull/207 implemented the `/v3/releases/module-name` endpoint but both the `/v3/files` (apparently used by r10k) and `/v3/modules` (used in my setup of librarian-puppet) are still missing from the API.

I don't know if r10k is able to use pulp_puppet as of now, but my librarian-puppet setup is failing because of the missing `/v3/modules` endpoint.

Fortunately, r10k no longer requires the unofficial files api to download modules, so it is able to work with the version of pulp-puppet that includes the pull request that you mentioned. I totally agree that the modules api should be implemented, but it might make sense to open a new issue with a more relevant name.

#9 Updated by lastmikoi about 3 years ago

etparton wrote:

Fortunately, r10k no longer requires the unofficial files api to download modules, so it is able to work with the version of pulp-puppet that includes the pull request that you mentioned. I totally agree that the modules api should be implemented, but it might make sense to open a new issue with a more relevant name.

Fair enough, I'll file a new feature request for the `/v3/modules` API endpoint then.

Thanks for the clarification over r10k.

#10 Updated by lastmikoi about 3 years ago

lastmikoi wrote:

Fair enough, I'll file a new feature request for the `/v3/modules` API endpoint then.

I filed https://pulp.plan.io/issues/1848 accordingly.

#11 Updated by bmbouter about 1 month ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF