Project

Profile

Help

Issue #4725

closed

Debian lazy sync virt-install fails w/ a 404 error, yet it's on the remote's CDN

Added by bmbouter almost 5 years ago. Updated almost 4 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Quarter:

Description

mdellweg and I both are experiencing this issue with the latest source checkouts of pulpcore, pulpcore-plugin and the pulp_deb. Using pulp_lift you can easily get into this setup.

Then sync+publish w/ a script like:

# Sync Repository
export REPOSITORY=$(http POST :24817/pulp/api/v3/repositories/ name=debian_stretch | jq -r '._href')
export REMOTE=$(http POST :24817/pulp/api/v3/remotes/deb/apt/ name=debian_stretch policy=streamed url="http://ftp.de.debian.org/debian/" distributions=stretch components=main architectures=amd64 sync_udebs=True sync_installer=True | jq -r '._href')
export SYNC_TASK=$(http POST :24817${REMOTE}sync/ repository=${REPOSITORY} | jq -r '.task')
# wait for task to finish
export VERSION=$(http :24817${SYNC_TASK} | jq -r '.created_resources[0]')

# Publish Repository
export PUBLISHER=$(http POST :24817/pulp/api/v3/publishers/deb/verbatim/ name=verbatim_stretch | jq -r '._href')
export PUBLISH_TASK=$(http POST :24817${PUBLISHER}publish/ repository=${REPOSITORY} | jq -r '.task')
# wait for task to finish
export PUBLICATION=$(http :24817${PUBLISH_TASK} | jq -r '.created_resources[0]')
http POST :24817/pulp/api/v3/distributions/ name=verbatim_stretch base_path=verbatim_stretch publication=${PUBLICATION}

You'll need your local ports forwarding from all local interfaces to the pulplift VM for ports 24816 and 24817. You can do this with something like:

vagrant ssh pulp3-source-fedora29 -- -R 29437:localhost:29437 -R 29438:localhost:29438 -L *:24816:127.0.0.1:24816 -L *:24817:127.0.0.1:24817

Then virt-install with a command like:

virt-install     \
    --name deb-pulp-install \
    --disk size=10 \
    --memory 1024 \
    --connect qemu:///system \
    --os-variant debian9 \
    --location http://192.168.122.1:24816/pulp/content/verbatim_stretch/dists/stretch/main/installer-amd64/ \
    --initrd-inject preseed.cfg

Eventually your system will fail with an error that looks like:

It's requesting the URL:

http://192.168.122.1:24816/pulp/content/verbatim_stretch/pool/main/b/busybox/busybox_1.22.0-19+b3_amd64.deb

which should exist because this URL exists:

http://ftp.de.debian.org/debian/pool/main/b/busybox/busybox_1.22.0-19+b3_amd64.deb


Files

deb_install-1.png (15.7 KB) deb_install-1.png bmbouter, 04/23/2019 06:57 PM
Actions #1

Updated by bmbouter almost 5 years ago

In debugging the pulp-streamer the request that shows up here on the call that returns the 404 it is showing up as: 'verbatim_stretch/pool/main/b/busybox/busybox_1.22.0-19%2Bb3_amd64.deb'

That is strange because the console output shows it having the '+' on it. Because it's not unquoted it's failing to match on the ContentArtifact.relative_path here

Actions #2

Updated by bmbouter almost 5 years ago

I confirmed that this PR fix does resolve the issue, but I'm still labelling it as a WIP: https://github.com/pulp/pulpcore/pull/96

I'm going to look around a bit more though because I don't think the unquote should be necessary...

Actions #3

Updated by bmbouter almost 5 years ago

  • Status changed from ASSIGNED to POST

I enabled access logging and reran the virt-install and I can see the URL that is failing is being requested as:

GET /pulp/content/verbatim_stretch/pool/main/b/busybox/busybox_1.22.0-19%2Bb3_amd64.deb

In debugging I can see that both request.path and request.raw_path are both /pulp/content/verbatim_stretch/pool/main/b/busybox/busybox_1.22.0-19%2Bb3_amd64.deb which means aiohttp is failing to unquote like it should. I filed this bug https://github.com/aio-libs/aiohttp/issues/3713 to resolve that.

I moved this to POST because we can merge the unquote workaround now and I also filed https://pulp.plan.io/issues/4735 to remove it before the GA release. The PR is here: https://github.com/pulp/pulpcore/pull/96

Actions #4

Updated by amacdona@redhat.com almost 5 years ago

  • Triaged changed from No to Yes
Actions #5

Updated by bmbouter over 4 years ago

  • Status changed from POST to NEW
Actions #6

Updated by fao89 over 4 years ago

  • Sprint/Milestone set to 3.0.0
Actions #7

Updated by fao89 over 4 years ago

I also got this error when I was testing kickstarts installing CentOS 7

Actions #8

Updated by bmbouter over 4 years ago

  • Assignee deleted (bmbouter)
Actions #9

Updated by bmbouter over 4 years ago

  • Sprint/Milestone deleted (3.0.0)

Removing from 3.0 blocker because there won't be a code change Pulp can make to fix it. Also yarl has fixes it, but they have not yet released that fix.

Actions #10

Updated by bmbouter almost 4 years ago

  • Status changed from NEW to CLOSED - CURRENTRELEASE

This has been resolved by yarl 1.4.0 which is released and shows the fix in its changelog here: https://github.com/aio-libs/yarl/blob/master/CHANGES.rst#140-2019-11-29. "Fix quoting of plus in path by pure python version (#339)"

Also available in: Atom PDF