Issue #9329
closedRegression: Syncing mirrolist based remotes do fail again since 3.14.1
Description
Very much related to: https://pulp.plan.io/issues/8981
Affected versions are at least 3.14.1, 3.14.2 & 3.14.3
Related issues
Updated by dalley over 3 years ago
- Priority changed from Normal to High
- Severity changed from 2. Medium to 3. High
- Sprint set to Sprint 104
Updated by dalley over 3 years ago
@cityofships could you describe what goes wrong? Error messages, incorrect metadata generated, etc? The pastebin link linked on the original issue is expired.
This issue was just filed, it could be the same - although I don't know if you're seeing different symptoms. https://pulp.plan.io/issues/9328
Updated by dalley over 3 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to dalley
Updated by ggainey over 3 years ago
dalley - this sequence fails:
pulp rpm remote create --name test --url "http://mirrorlist.centos.org/?arch=x86_64&release=8&repo=opstools-collectd-5"
pulp rpm repository create --name test --remote test
pulp rpm repository sync --name test
pulp rpm repository sync --name test
Started background task /pulp/api/v3/tasks/b1f34e98-7ea2-4994-84b5-58ea3ee4b1e5/
.Error: Task /pulp/api/v3/tasks/b1f34e98-7ea2-4994-84b5-58ea3ee4b1e5/ failed: '404, message='Not Found', url=URL('http://mirrorlist.centos.org/%3Farch=x86_64&release=8&repo=opstools-collectd-5')'
This sequence works fine:
pulp rpm remote create --name fmirror --url "https://mirrors.fedoraproject.org/mirrorlist?repo=epel-modular-8&arch=x86_64&infra=stock&content=centos"
pulp rpm repository create --name fmirror --remote fmirror
pulp rpm repository sync --name fmirror
pulp rpm repository sync --name fmirror
Started background task /pulp/api/v3/tasks/ef10b45f-0dda-4a4e-b2c3-51e48fe2e11e/
.....................................Done.
(The URL above is what gets used in our test pulp_rpm.tests.functional.api.test_sync.BasicSyncTestCase.test_sync_from_valid_mirror_list_feed_with_params
, which passes)
But this sequence fails:
pulp rpm remote create --name fmirror2 --url "https://mirrors.fedoraproject.org/mirrorlist/?repo=epel-modular-8&arch=x86_64&infra=stock&content=centos"
pulp rpm repository create --name fmirror2 --remote fmirror2
pulp rpm repository sync --name fmirror2
Started background task /pulp/api/v3/tasks/83b29ef6-e5d1-430c-bfd6-a187ee86576c/
.Error: Task /pulp/api/v3/tasks/83b29ef6-e5d1-430c-bfd6-a187ee86576c/ failed: '404, message='Not Found', url=URL('https://mirrors.fedoraproject.org/mirrorlist/?repo=epel-modular-8&arch=x86_64&infra=stock&content=centos')'
Note the only difference between 2 and 3 is "https.../mirrorlist?..." and "https.../mirrorlist/?..." . Clearly, tripping over a(nother) URL-parsing-subtlety.
Updated by dalley over 3 years ago
- Status changed from ASSIGNED to NEW
- Assignee deleted (
dalley)
Updated by dalley over 3 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to dalley
Updated by dalley over 3 years ago
Updated by dalley over 3 years ago
ggainey, example 3 doesn't seem to work even from a browser, or from httpie, so I'm not sure that's a Pulp issue.
(pulp) [vagrant@pulp3-source-fedora34 devel]$ http "https://mirrors.fedoraproject.org/mirrorlist/?repo=epel-modular-8&arch=x86_64&infra=stock&content=centos"
HTTP/1.1 404 Not Found
Example 1 however is a legit bug.
Updated by pulpbot over 3 years ago
Updated by dalley over 3 years ago
@cityofships FYI we'll push a new release as soon as we fix this second mirrorlist-related issue: https://pulp.plan.io/issues/9328
Updated by dalley about 3 years ago
- Sprint/Milestone changed from 3.14.4 to 3.16.0
Updated by dalley about 3 years ago
- Copied to Backport #9394: Backport #9329 "Regression: Syncing mirrolist based remotes do fail again since 3.14.1" to 3.14.z added
Added by dalley about 3 years ago
Updated by dalley about 3 years ago
- Status changed from POST to MODIFIED
Applied in changeset d328e0ab975ea654950c050c3c642c3002c946ed.
Updated by martin about 3 years ago
We have issue with something similar:
FAILED - RETRYING: Sync vendor-product-develop-8 repository from remote (15 retries left).Result was: {
"attempts": 1,
"changed": false,
"invocation": {
"module_args": {
"mirror": true,
"password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
"pulp_url": "http://localhost:24817",
"refresh_api_cache": false,
"remote": "vendor-product-develop-8-remote",
"repository": "vendor-product-develop-8",
"username": "admin",
"validate_certs": true
}
},
"msg": "Task failed to complete. (failed; https://username_with_underscores-and-dashes:somepasswordstring%40downloads.vendor.url.com/repo/product/rhel/8/x86_64/current/repodata/repomd.xml)",
"retries": 16
}
Is it related or should I open separate issue? Seems that escaping @ to %40 breaks the URL and repo sync.
Updated by dalley about 3 years ago
@martin Which version of pulp_rpm? If it is the same bug, then it should be fixed in 3.14.4
It's possible that it's a separate bug however.
Updated by martin about 3 years ago
dalley, it is happening on latest pulpcore 3.14.6 and pulp_rpm 3.14.4
Updated by dalley about 3 years ago
: /
@martin Could you please provide your remote configuration? You can use "pulp rpm remote show --name $remote_name"
Updated by johng about 3 years ago
Not sure if this is the correct place to ask but will 3.15.x get a similar fix?
Updated by dalley about 3 years ago
Well, if it is still broken in some situations even with this patch, then we'll need new releases for everyone.
Probably we will not do a 3.15.z because 3.16 will be releasing in a week or so. And most users are either using Katello RPMs, which are on the 3.14 branch, or the container images, which use the latest upstream releases available. But I'm open to hearing out anyone that has a particular need for a 3.15.z release.
Updated by johng about 3 years ago
I'm attempting to use the container release as described https://pulpproject.org/pulp-in-one-container/
docker.io/pulp/pulp:latest which ships with 3.15.0
{"versions":[{"component":"core","version":"3.15.2"},{"component":"rpm","version":"3.15.0"}.....
The 3.14 tag is a month old so presumably doesn't include this fix either.
Updated by dalley about 3 years ago
You're right about the 3.14 tag being out of date, I just pushed a new one (and also probably document this as something we need to automate or trigger more frequently).
But if it turns out it is still broken then it won't help. I'm trying to reproduce that currently, I have not yet been able to do so.
As for the "latest" tag, it should just use the newest version, whether that is pulp_rpm 3.15.1, or pulp_rpm 3.16.0. So it won't matter too much what type of release we do so long as we do one soon.
Updated by johng about 3 years ago
Ok thanks, I'll await 3.16.0 then.
I've partially applied the fix locally in the meantime so I can sync the repo I was having trouble with.
Updated by martin about 3 years ago
- Copied to Issue #9464: Regression: Syncing username+password authenticated remotes fails added
Updated by pulpbot about 3 years ago
- Status changed from MODIFIED to CLOSED - CURRENTRELEASE
Fix a regression in mirrorlist sync support
closes: #9329 https://pulp.plan.io/issues/9329