RPM Support

@cityofships could you describe what goes wrong? Error messages, incorrect metadata generated, etc? The pastebin link linked on the original issue is expired.

This issue was just filed, it could be the same - although I don't know if you're seeing different symptoms. https://pulp.plan.io/issues/9328

dalley - this sequence fails:

pulp rpm remote create --name test --url "http://mirrorlist.centos.org/?arch=x86_64&release=8&repo=opstools-collectd-5"
pulp rpm repository create --name test --remote test
pulp rpm repository sync --name test
pulp rpm repository sync --name test
Started background task /pulp/api/v3/tasks/b1f34e98-7ea2-4994-84b5-58ea3ee4b1e5/
.Error: Task /pulp/api/v3/tasks/b1f34e98-7ea2-4994-84b5-58ea3ee4b1e5/ failed: '404, message='Not Found', url=URL('http://mirrorlist.centos.org/%3Farch=x86_64&release=8&repo=opstools-collectd-5')'

This sequence works fine:

pulp rpm remote create --name fmirror --url "https://mirrors.fedoraproject.org/mirrorlist?repo=epel-modular-8&arch=x86_64&infra=stock&content=centos"
pulp rpm repository create --name fmirror --remote fmirror
pulp rpm repository sync --name fmirror
pulp rpm repository sync --name fmirror
Started background task /pulp/api/v3/tasks/ef10b45f-0dda-4a4e-b2c3-51e48fe2e11e/
.....................................Done.

(The URL above is what gets used in our test pulp_rpm.tests.functional.api.test_sync.BasicSyncTestCase.test_sync_from_valid_mirror_list_feed_with_params , which passes)

But this sequence fails:

pulp rpm remote create --name fmirror2 --url "https://mirrors.fedoraproject.org/mirrorlist/?repo=epel-modular-8&arch=x86_64&infra=stock&content=centos"
pulp rpm repository create --name fmirror2 --remote fmirror2
pulp rpm repository sync --name fmirror2
Started background task /pulp/api/v3/tasks/83b29ef6-e5d1-430c-bfd6-a187ee86576c/
.Error: Task /pulp/api/v3/tasks/83b29ef6-e5d1-430c-bfd6-a187ee86576c/ failed: '404, message='Not Found', url=URL('https://mirrors.fedoraproject.org/mirrorlist/?repo=epel-modular-8&arch=x86_64&infra=stock&content=centos')'

Note the only difference between 2 and 3 is "https.../mirrorlist?..." and "https.../mirrorlist/?..." . Clearly, tripping over a(nother) URL-parsing-subtlety.

Discussion on matrix here: https://riot.im/app/#/room/#pulp-rpm:matrix.org/$YdlA42DKgjCpJF2Ngz7k1y07ZrZeLSnPJgf4g1F2ipY

And here: https://riot.im/app/#/room/#pulp-rpm:matrix.org/$8b98jRZv1khE2Ps3Q_gWcgdeNRpMt_ykyFlbDlCXC9E

ggainey, example 3 doesn't seem to work even from a browser, or from httpie, so I'm not sure that's a Pulp issue.

(pulp) [vagrant@pulp3-source-fedora34 devel]$ http "https://mirrors.fedoraproject.org/mirrorlist/?repo=epel-modular-8&arch=x86_64&infra=stock&content=centos"
HTTP/1.1 404 Not Found

Example 1 however is a legit bug.

PR: https://github.com/pulp/pulp_rpm/pull/2120

@cityofships FYI we'll push a new release as soon as we fix this second mirrorlist-related issue: https://pulp.plan.io/issues/9328

We have issue with something similar:

FAILED - RETRYING: Sync vendor-product-develop-8 repository from remote (15 retries left).Result was: {
    "attempts": 1,
    "changed": false,
    "invocation": {
        "module_args": {
            "mirror": true,
            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "pulp_url": "http://localhost:24817",
            "refresh_api_cache": false,
            "remote": "vendor-product-develop-8-remote",
            "repository": "vendor-product-develop-8",
            "username": "admin",
            "validate_certs": true
        }
    },
    "msg": "Task failed to complete. (failed; https://username_with_underscores-and-dashes:somepasswordstring%40downloads.vendor.url.com/repo/product/rhel/8/x86_64/current/repodata/repomd.xml)",
    "retries": 16
}

Is it related or should I open separate issue? Seems that escaping @ to %40 breaks the URL and repo sync.

@martin Which version of pulp_rpm? If it is the same bug, then it should be fixed in 3.14.4

It's possible that it's a separate bug however.

dalley, it is happening on latest pulpcore 3.14.6 and pulp_rpm 3.14.4

: /

@martin Could you please provide your remote configuration? You can use "pulp rpm remote show --name $remote_name"

Not sure if this is the correct place to ask but will 3.15.x get a similar fix?

Well, if it is still broken in some situations even with this patch, then we'll need new releases for everyone.

Probably we will not do a 3.15.z because 3.16 will be releasing in a week or so. And most users are either using Katello RPMs, which are on the 3.14 branch, or the container images, which use the latest upstream releases available. But I'm open to hearing out anyone that has a particular need for a 3.15.z release.

I'm attempting to use the container release as described https://pulpproject.org/pulp-in-one-container/

docker.io/pulp/pulp:latest which ships with 3.15.0

{"versions":[{"component":"core","version":"3.15.2"},{"component":"rpm","version":"3.15.0"}.....

The 3.14 tag is a month old so presumably doesn't include this fix either.

You're right about the 3.14 tag being out of date, I just pushed a new one (and also probably document this as something we need to automate or trigger more frequently).

But if it turns out it is still broken then it won't help. I'm trying to reproduce that currently, I have not yet been able to do so.

As for the "latest" tag, it should just use the newest version, whether that is pulp_rpm 3.15.1, or pulp_rpm 3.16.0. So it won't matter too much what type of release we do so long as we do one soon.

Ok thanks, I'll await 3.16.0 then.

I've partially applied the fix locally in the meantime so I can sync the repo I was having trouble with.

@martin Please file a separate issue