Issue #4725
closedDebian lazy sync virt-install fails w/ a 404 error, yet it's on the remote's CDN
Description
mdellweg and I both are experiencing this issue with the latest source checkouts of pulpcore, pulpcore-plugin and the pulp_deb. Using pulp_lift you can easily get into this setup.
Then sync+publish w/ a script like:
# Sync Repository
export REPOSITORY=$(http POST :24817/pulp/api/v3/repositories/ name=debian_stretch | jq -r '._href')
export REMOTE=$(http POST :24817/pulp/api/v3/remotes/deb/apt/ name=debian_stretch policy=streamed url="http://ftp.de.debian.org/debian/" distributions=stretch components=main architectures=amd64 sync_udebs=True sync_installer=True | jq -r '._href')
export SYNC_TASK=$(http POST :24817${REMOTE}sync/ repository=${REPOSITORY} | jq -r '.task')
# wait for task to finish
export VERSION=$(http :24817${SYNC_TASK} | jq -r '.created_resources[0]')
# Publish Repository
export PUBLISHER=$(http POST :24817/pulp/api/v3/publishers/deb/verbatim/ name=verbatim_stretch | jq -r '._href')
export PUBLISH_TASK=$(http POST :24817${PUBLISHER}publish/ repository=${REPOSITORY} | jq -r '.task')
# wait for task to finish
export PUBLICATION=$(http :24817${PUBLISH_TASK} | jq -r '.created_resources[0]')
http POST :24817/pulp/api/v3/distributions/ name=verbatim_stretch base_path=verbatim_stretch publication=${PUBLICATION}
You'll need your local ports forwarding from all local interfaces to the pulplift VM for ports 24816 and 24817. You can do this with something like:
vagrant ssh pulp3-source-fedora29 -- -R 29437:localhost:29437 -R 29438:localhost:29438 -L *:24816:127.0.0.1:24816 -L *:24817:127.0.0.1:24817
Then virt-install with a command like:
virt-install \
--name deb-pulp-install \
--disk size=10 \
--memory 1024 \
--connect qemu:///system \
--os-variant debian9 \
--location http://192.168.122.1:24816/pulp/content/verbatim_stretch/dists/stretch/main/installer-amd64/ \
--initrd-inject preseed.cfg
Eventually your system will fail with an error that looks like:
It's requesting the URL:
which should exist because this URL exists:
http://ftp.de.debian.org/debian/pool/main/b/busybox/busybox_1.22.0-19+b3_amd64.deb
Files
Updated by bmbouter almost 5 years ago
In debugging the pulp-streamer the request that shows up here on the call that returns the 404 it is showing up as: 'verbatim_stretch/pool/main/b/busybox/busybox_1.22.0-19%2Bb3_amd64.deb'
That is strange because the console output shows it having the '+' on it. Because it's not unquoted it's failing to match on the ContentArtifact.relative_path here
Updated by bmbouter almost 5 years ago
I confirmed that this PR fix does resolve the issue, but I'm still labelling it as a WIP: https://github.com/pulp/pulpcore/pull/96
I'm going to look around a bit more though because I don't think the unquote should be necessary...
Updated by bmbouter almost 5 years ago
- Status changed from ASSIGNED to POST
I enabled access logging and reran the virt-install and I can see the URL that is failing is being requested as:
GET /pulp/content/verbatim_stretch/pool/main/b/busybox/busybox_1.22.0-19%2Bb3_amd64.deb
In debugging I can see that both request.path and request.raw_path are both /pulp/content/verbatim_stretch/pool/main/b/busybox/busybox_1.22.0-19%2Bb3_amd64.deb which means aiohttp is failing to unquote like it should. I filed this bug https://github.com/aio-libs/aiohttp/issues/3713 to resolve that.
I moved this to POST because we can merge the unquote workaround now and I also filed https://pulp.plan.io/issues/4735 to remove it before the GA release. The PR is here: https://github.com/pulp/pulpcore/pull/96
Updated by amacdona@redhat.com almost 5 years ago
- Triaged changed from No to Yes
Updated by fao89 over 4 years ago
I also got this error when I was testing kickstarts installing CentOS 7
Updated by bmbouter over 4 years ago
- Sprint/Milestone deleted (
3.0.0)
Removing from 3.0 blocker because there won't be a code change Pulp can make to fix it. Also yarl has fixes it, but they have not yet released that fix.
Updated by bmbouter almost 4 years ago
- Status changed from NEW to CLOSED - CURRENTRELEASE
This has been resolved by yarl 1.4.0 which is released and shows the fix in its changelog here: https://github.com/aio-libs/yarl/blob/master/CHANGES.rst#140-2019-11-29. "Fix quoting of plus in path by pure python version (#339)"