Project

Profile

Help

Issue #9329

Regression: Syncing mirrolist based remotes do fail again since 3.14.1

Added by cityofships about 2 months ago. Updated 2 days ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 105
Quarter:

Description

Very much related to: https://pulp.plan.io/issues/8981

Affected versions are at least 3.14.1, 3.14.2 & 3.14.3


Related issues

Copied to RPM Support - Backport #9394: Backport #9329 "Regression: Syncing mirrolist based remotes do fail again since 3.14.1" to 3.14.zCLOSED - CURRENTRELEASE

<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>
Copied to RPM Support - Issue #9464: Regression: Syncing username+password authenticated remotes failsCLOSED - CURRENTRELEASE<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision d328e0ab View on GitHub
Added by dalley 30 days ago

Fix a regression in mirrorlist sync support

closes: #9329 https://pulp.plan.io/issues/9329

History

#1 Updated by dalley about 2 months ago

  • Priority changed from Normal to High
  • Severity changed from 2. Medium to 3. High
  • Sprint set to Sprint 104

#2 Updated by dalley about 2 months ago

@cityofships could you describe what goes wrong? Error messages, incorrect metadata generated, etc? The pastebin link linked on the original issue is expired.

This issue was just filed, it could be the same - although I don't know if you're seeing different symptoms. https://pulp.plan.io/issues/9328

#3 Updated by dalley about 2 months ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dalley

#4 Updated by ggainey about 2 months ago

dalley - this sequence fails:

pulp rpm remote create --name test --url "http://mirrorlist.centos.org/?arch=x86_64&release=8&repo=opstools-collectd-5"
pulp rpm repository create --name test --remote test
pulp rpm repository sync --name test
pulp rpm repository sync --name test
Started background task /pulp/api/v3/tasks/b1f34e98-7ea2-4994-84b5-58ea3ee4b1e5/
.Error: Task /pulp/api/v3/tasks/b1f34e98-7ea2-4994-84b5-58ea3ee4b1e5/ failed: '404, message='Not Found', url=URL('http://mirrorlist.centos.org/%3Farch=x86_64&release=8&repo=opstools-collectd-5')'

This sequence works fine:

pulp rpm remote create --name fmirror --url "https://mirrors.fedoraproject.org/mirrorlist?repo=epel-modular-8&arch=x86_64&infra=stock&content=centos"
pulp rpm repository create --name fmirror --remote fmirror
pulp rpm repository sync --name fmirror
pulp rpm repository sync --name fmirror
Started background task /pulp/api/v3/tasks/ef10b45f-0dda-4a4e-b2c3-51e48fe2e11e/
.....................................Done.

(The URL above is what gets used in our test pulp_rpm.tests.functional.api.test_sync.BasicSyncTestCase.test_sync_from_valid_mirror_list_feed_with_params , which passes)

But this sequence fails:

pulp rpm remote create --name fmirror2 --url "https://mirrors.fedoraproject.org/mirrorlist/?repo=epel-modular-8&arch=x86_64&infra=stock&content=centos"
pulp rpm repository create --name fmirror2 --remote fmirror2
pulp rpm repository sync --name fmirror2
Started background task /pulp/api/v3/tasks/83b29ef6-e5d1-430c-bfd6-a187ee86576c/
.Error: Task /pulp/api/v3/tasks/83b29ef6-e5d1-430c-bfd6-a187ee86576c/ failed: '404, message='Not Found', url=URL('https://mirrors.fedoraproject.org/mirrorlist/?repo=epel-modular-8&arch=x86_64&infra=stock&content=centos')'

Note the only difference between 2 and 3 is "https.../mirrorlist?..." and "https.../mirrorlist/?..." . Clearly, tripping over a(nother) URL-parsing-subtlety.

#5 Updated by dalley about 2 months ago

  • Status changed from ASSIGNED to NEW
  • Assignee deleted (dalley)

#6 Updated by dalley about 2 months ago

  • Triaged changed from No to Yes

#8 Updated by dalley about 1 month ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dalley

#10 Updated by rchan about 1 month ago

  • Sprint changed from Sprint 104 to Sprint 105

#11 Updated by dalley about 1 month ago

ggainey, example 3 doesn't seem to work even from a browser, or from httpie, so I'm not sure that's a Pulp issue.

(pulp) [vagrant@pulp3-source-fedora34 devel]$ http "https://mirrors.fedoraproject.org/mirrorlist/?repo=epel-modular-8&arch=x86_64&infra=stock&content=centos"
HTTP/1.1 404 Not Found

Example 1 however is a legit bug.

#12 Updated by dalley about 1 month ago

  • Status changed from ASSIGNED to POST

#14 Updated by dalley about 1 month ago

@cityofships FYI we'll push a new release as soon as we fix this second mirrorlist-related issue: https://pulp.plan.io/issues/9328

#15 Updated by dalley about 1 month ago

  • Sprint/Milestone set to 3.14.4

#16 Updated by dalley about 1 month ago

  • Sprint/Milestone changed from 3.14.4 to 3.16.0

#17 Updated by dalley about 1 month ago

  • Copied to Backport #9394: Backport #9329 "Regression: Syncing mirrolist based remotes do fail again since 3.14.1" to 3.14.z added

#18 Updated by dalley 30 days ago

  • Status changed from POST to MODIFIED

#19 Updated by martin 28 days ago

We have issue with something similar:

FAILED - RETRYING: Sync vendor-product-develop-8 repository from remote (15 retries left).Result was: {
    "attempts": 1,
    "changed": false,
    "invocation": {
        "module_args": {
            "mirror": true,
            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "pulp_url": "http://localhost:24817",
            "refresh_api_cache": false,
            "remote": "vendor-product-develop-8-remote",
            "repository": "vendor-product-develop-8",
            "username": "admin",
            "validate_certs": true
        }
    },
    "msg": "Task failed to complete. (failed; https://username_with_underscores-and-dashes:somepasswordstring%40downloads.vendor.url.com/repo/product/rhel/8/x86_64/current/repodata/repomd.xml)",
    "retries": 16
}

Is it related or should I open separate issue? Seems that escaping @ to %40 breaks the URL and repo sync.

#20 Updated by dalley 28 days ago

@martin Which version of pulp_rpm? If it is the same bug, then it should be fixed in 3.14.4

It's possible that it's a separate bug however.

#21 Updated by martin 25 days ago

dalley, it is happening on latest pulpcore 3.14.6 and pulp_rpm 3.14.4

#22 Updated by dalley 25 days ago

: /

@martin Could you please provide your remote configuration? You can use "pulp rpm remote show --name $remote_name"

#23 Updated by johng 24 days ago

Not sure if this is the correct place to ask but will 3.15.x get a similar fix?

#24 Updated by dalley 24 days ago

Well, if it is still broken in some situations even with this patch, then we'll need new releases for everyone.

Probably we will not do a 3.15.z because 3.16 will be releasing in a week or so. And most users are either using Katello RPMs, which are on the 3.14 branch, or the container images, which use the latest upstream releases available. But I'm open to hearing out anyone that has a particular need for a 3.15.z release.

#25 Updated by johng 24 days ago

I'm attempting to use the container release as described https://pulpproject.org/pulp-in-one-container/

docker.io/pulp/pulp:latest which ships with 3.15.0

{"versions":[{"component":"core","version":"3.15.2"},{"component":"rpm","version":"3.15.0"}.....

The 3.14 tag is a month old so presumably doesn't include this fix either.

#26 Updated by dalley 24 days ago

You're right about the 3.14 tag being out of date, I just pushed a new one (and also probably document this as something we need to automate or trigger more frequently).

But if it turns out it is still broken then it won't help. I'm trying to reproduce that currently, I have not yet been able to do so.

As for the "latest" tag, it should just use the newest version, whether that is pulp_rpm 3.15.1, or pulp_rpm 3.16.0. So it won't matter too much what type of release we do so long as we do one soon.

#27 Updated by johng 24 days ago

Ok thanks, I'll await 3.16.0 then.

I've partially applied the fix locally in the meantime so I can sync the repo I was having trouble with.

#28 Updated by dalley 24 days ago

@martin Please file a separate issue

#29 Updated by martin 23 days ago

  • Copied to Issue #9464: Regression: Syncing username+password authenticated remotes fails added

#30 Updated by pulpbot 2 days ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Please register to edit this issue

Also available in: Atom PDF