Project

Profile

Help

Issue #9329

closed

Regression: Syncing mirrolist based remotes do fail again since 3.14.1

Added by cityofships over 3 years ago. Updated about 3 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 105
Quarter:

Description

Very much related to: https://pulp.plan.io/issues/8981

Affected versions are at least 3.14.1, 3.14.2 & 3.14.3


Related issues

Copied to RPM Support - Backport #9394: Backport #9329 "Regression: Syncing mirrolist based remotes do fail again since 3.14.1" to 3.14.zCLOSED - CURRENTRELEASEdalley

Actions
Copied to RPM Support - Issue #9464: Regression: Syncing username+password authenticated remotes failsCLOSED - CURRENTRELEASEggaineyActions
Actions #1

Updated by dalley over 3 years ago

  • Priority changed from Normal to High
  • Severity changed from 2. Medium to 3. High
  • Sprint set to Sprint 104
Actions #2

Updated by dalley over 3 years ago

@cityofships could you describe what goes wrong? Error messages, incorrect metadata generated, etc? The pastebin link linked on the original issue is expired.

This issue was just filed, it could be the same - although I don't know if you're seeing different symptoms. https://pulp.plan.io/issues/9328

Actions #3

Updated by dalley over 3 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dalley
Actions #4

Updated by ggainey over 3 years ago

dalley - this sequence fails:

pulp rpm remote create --name test --url "http://mirrorlist.centos.org/?arch=x86_64&release=8&repo=opstools-collectd-5"
pulp rpm repository create --name test --remote test
pulp rpm repository sync --name test
pulp rpm repository sync --name test
Started background task /pulp/api/v3/tasks/b1f34e98-7ea2-4994-84b5-58ea3ee4b1e5/
.Error: Task /pulp/api/v3/tasks/b1f34e98-7ea2-4994-84b5-58ea3ee4b1e5/ failed: '404, message='Not Found', url=URL('http://mirrorlist.centos.org/%3Farch=x86_64&release=8&repo=opstools-collectd-5')'

This sequence works fine:

pulp rpm remote create --name fmirror --url "https://mirrors.fedoraproject.org/mirrorlist?repo=epel-modular-8&arch=x86_64&infra=stock&content=centos"
pulp rpm repository create --name fmirror --remote fmirror
pulp rpm repository sync --name fmirror
pulp rpm repository sync --name fmirror
Started background task /pulp/api/v3/tasks/ef10b45f-0dda-4a4e-b2c3-51e48fe2e11e/
.....................................Done.

(The URL above is what gets used in our test pulp_rpm.tests.functional.api.test_sync.BasicSyncTestCase.test_sync_from_valid_mirror_list_feed_with_params , which passes)

But this sequence fails:

pulp rpm remote create --name fmirror2 --url "https://mirrors.fedoraproject.org/mirrorlist/?repo=epel-modular-8&arch=x86_64&infra=stock&content=centos"
pulp rpm repository create --name fmirror2 --remote fmirror2
pulp rpm repository sync --name fmirror2
Started background task /pulp/api/v3/tasks/83b29ef6-e5d1-430c-bfd6-a187ee86576c/
.Error: Task /pulp/api/v3/tasks/83b29ef6-e5d1-430c-bfd6-a187ee86576c/ failed: '404, message='Not Found', url=URL('https://mirrors.fedoraproject.org/mirrorlist/?repo=epel-modular-8&arch=x86_64&infra=stock&content=centos')'

Note the only difference between 2 and 3 is "https.../mirrorlist?..." and "https.../mirrorlist/?..." . Clearly, tripping over a(nother) URL-parsing-subtlety.

Actions #5

Updated by dalley over 3 years ago

  • Status changed from ASSIGNED to NEW
  • Assignee deleted (dalley)
Actions #6

Updated by dalley over 3 years ago

  • Triaged changed from No to Yes
Actions #8

Updated by dalley over 3 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dalley
Actions #10

Updated by rchan over 3 years ago

  • Sprint changed from Sprint 104 to Sprint 105
Actions #11

Updated by dalley over 3 years ago

ggainey, example 3 doesn't seem to work even from a browser, or from httpie, so I'm not sure that's a Pulp issue.

(pulp) [vagrant@pulp3-source-fedora34 devel]$ http "https://mirrors.fedoraproject.org/mirrorlist/?repo=epel-modular-8&arch=x86_64&infra=stock&content=centos"
HTTP/1.1 404 Not Found

Example 1 however is a legit bug.

Actions #12

Updated by dalley over 3 years ago

  • Status changed from ASSIGNED to POST
Actions #14

Updated by dalley over 3 years ago

@cityofships FYI we'll push a new release as soon as we fix this second mirrorlist-related issue: https://pulp.plan.io/issues/9328

Actions #15

Updated by dalley about 3 years ago

  • Sprint/Milestone set to 3.14.4
Actions #16

Updated by dalley about 3 years ago

  • Sprint/Milestone changed from 3.14.4 to 3.16.0
Actions #17

Updated by dalley about 3 years ago

  • Copied to Backport #9394: Backport #9329 "Regression: Syncing mirrolist based remotes do fail again since 3.14.1" to 3.14.z added

Added by dalley about 3 years ago

Revision d328e0ab | View on GitHub

Fix a regression in mirrorlist sync support

closes: #9329 https://pulp.plan.io/issues/9329

Actions #18

Updated by dalley about 3 years ago

  • Status changed from POST to MODIFIED
Actions #19

Updated by martin about 3 years ago

We have issue with something similar:

FAILED - RETRYING: Sync vendor-product-develop-8 repository from remote (15 retries left).Result was: {
    "attempts": 1,
    "changed": false,
    "invocation": {
        "module_args": {
            "mirror": true,
            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "pulp_url": "http://localhost:24817",
            "refresh_api_cache": false,
            "remote": "vendor-product-develop-8-remote",
            "repository": "vendor-product-develop-8",
            "username": "admin",
            "validate_certs": true
        }
    },
    "msg": "Task failed to complete. (failed; https://username_with_underscores-and-dashes:somepasswordstring%40downloads.vendor.url.com/repo/product/rhel/8/x86_64/current/repodata/repomd.xml)",
    "retries": 16
}

Is it related or should I open separate issue? Seems that escaping @ to %40 breaks the URL and repo sync.

Actions #20

Updated by dalley about 3 years ago

@martin Which version of pulp_rpm? If it is the same bug, then it should be fixed in 3.14.4

It's possible that it's a separate bug however.

Actions #21

Updated by martin about 3 years ago

dalley, it is happening on latest pulpcore 3.14.6 and pulp_rpm 3.14.4

Actions #22

Updated by dalley about 3 years ago

: /

@martin Could you please provide your remote configuration? You can use "pulp rpm remote show --name $remote_name"

Actions #23

Updated by johng about 3 years ago

Not sure if this is the correct place to ask but will 3.15.x get a similar fix?

Actions #24

Updated by dalley about 3 years ago

Well, if it is still broken in some situations even with this patch, then we'll need new releases for everyone.

Probably we will not do a 3.15.z because 3.16 will be releasing in a week or so. And most users are either using Katello RPMs, which are on the 3.14 branch, or the container images, which use the latest upstream releases available. But I'm open to hearing out anyone that has a particular need for a 3.15.z release.

Actions #25

Updated by johng about 3 years ago

I'm attempting to use the container release as described https://pulpproject.org/pulp-in-one-container/

docker.io/pulp/pulp:latest which ships with 3.15.0

{"versions":[{"component":"core","version":"3.15.2"},{"component":"rpm","version":"3.15.0"}.....

The 3.14 tag is a month old so presumably doesn't include this fix either.

Actions #26

Updated by dalley about 3 years ago

You're right about the 3.14 tag being out of date, I just pushed a new one (and also probably document this as something we need to automate or trigger more frequently).

But if it turns out it is still broken then it won't help. I'm trying to reproduce that currently, I have not yet been able to do so.

As for the "latest" tag, it should just use the newest version, whether that is pulp_rpm 3.15.1, or pulp_rpm 3.16.0. So it won't matter too much what type of release we do so long as we do one soon.

Actions #27

Updated by johng about 3 years ago

Ok thanks, I'll await 3.16.0 then.

I've partially applied the fix locally in the meantime so I can sync the repo I was having trouble with.

Actions #28

Updated by dalley about 3 years ago

@martin Please file a separate issue

Actions #29

Updated by martin about 3 years ago

  • Copied to Issue #9464: Regression: Syncing username+password authenticated remotes fails added
Actions #30

Updated by pulpbot about 3 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF