Project

Profile

Help

Issue #4725

Debian lazy sync virt-install fails w/ a 404 error, yet it's on the remote's CDN

Added by bmbouter 6 months ago. Updated 5 days ago.

Status:
NEW
Priority:
High
Assignee:
-
Category:
-
Sprint/Milestone:
Start date:
Due date:
Severity:
2. Medium
Version:
Platform Release:
Blocks Release:
OS:
Backwards Incompatible:
No
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:

Description

@mdellweg and I both are experiencing this issue with the latest source checkouts of pulpcore, pulpcore-plugin and the pulp_deb. Using pulp_lift you can easily get into this setup.

Then sync+publish w/ a script like:

# Sync Repository
export REPOSITORY=$(http POST :24817/pulp/api/v3/repositories/ name=debian_stretch | jq -r '._href')
export REMOTE=$(http POST :24817/pulp/api/v3/remotes/deb/apt/ name=debian_stretch policy=streamed url="http://ftp.de.debian.org/debian/" distributions=stretch components=main architectures=amd64 sync_udebs=True sync_installer=True | jq -r '._href')
export SYNC_TASK=$(http POST :24817${REMOTE}sync/ repository=${REPOSITORY} | jq -r '.task')
# wait for task to finish
export VERSION=$(http :24817${SYNC_TASK} | jq -r '.created_resources[0]')

# Publish Repository
export PUBLISHER=$(http POST :24817/pulp/api/v3/publishers/deb/verbatim/ name=verbatim_stretch | jq -r '._href')
export PUBLISH_TASK=$(http POST :24817${PUBLISHER}publish/ repository=${REPOSITORY} | jq -r '.task')
# wait for task to finish
export PUBLICATION=$(http :24817${PUBLISH_TASK} | jq -r '.created_resources[0]')
http POST :24817/pulp/api/v3/distributions/ name=verbatim_stretch base_path=verbatim_stretch publication=${PUBLICATION}

You'll need your local ports forwarding from all local interfaces to the pulplift VM for ports 24816 and 24817. You can do this with something like:

vagrant ssh pulp3-source-fedora29 -- -R 29437:localhost:29437 -R 29438:localhost:29438 -L *:24816:127.0.0.1:24816 -L *:24817:127.0.0.1:24817

Then virt-install with a command like:

virt-install     \
    --name deb-pulp-install \
    --disk size=10 \
    --memory 1024 \
    --connect qemu:///system \
    --os-variant debian9 \
    --location http://192.168.122.1:24816/pulp/content/verbatim_stretch/dists/stretch/main/installer-amd64/ \
    --initrd-inject preseed.cfg

Eventually your system will fail with an error that looks like:

It's requesting the URL:

http://192.168.122.1:24816/pulp/content/verbatim_stretch/pool/main/b/busybox/busybox_1.22.0-19+b3_amd64.deb

which should exist because this URL exists:

http://ftp.de.debian.org/debian/pool/main/b/busybox/busybox_1.22.0-19+b3_amd64.deb

deb_install-1.png (15.7 KB) bmbouter, 04/23/2019 06:57 PM deb_install-1.png
250

History

#1 Updated by bmbouter 6 months ago

In debugging the pulp-streamer the request that shows up here on the call that returns the 404 it is showing up as: 'verbatim_stretch/pool/main/b/busybox/busybox_1.22.0-19%2Bb3_amd64.deb'

That is strange because the console output shows it having the '+' on it. Because it's not unquoted it's failing to match on the ContentArtifact.relative_path here

#2 Updated by bmbouter 6 months ago

I confirmed that this PR fix does resolve the issue, but I'm still labelling it as a WIP: https://github.com/pulp/pulpcore/pull/96

I'm going to look around a bit more though because I don't think the unquote should be necessary...

#3 Updated by bmbouter 6 months ago

  • Status changed from ASSIGNED to POST

I enabled access logging and reran the virt-install and I can see the URL that is failing is being requested as:

GET /pulp/content/verbatim_stretch/pool/main/b/busybox/busybox_1.22.0-19%2Bb3_amd64.deb

In debugging I can see that both request.path and request.raw_path are both /pulp/content/verbatim_stretch/pool/main/b/busybox/busybox_1.22.0-19%2Bb3_amd64.deb which means aiohttp is failing to unquote like it should. I filed this bug https://github.com/aio-libs/aiohttp/issues/3713 to resolve that.

I moved this to POST because we can merge the unquote workaround now and I also filed https://pulp.plan.io/issues/4735 to remove it before the GA release. The PR is here: https://github.com/pulp/pulpcore/pull/96

#4 Updated by amacdona@redhat.com 6 months ago

  • Triaged changed from No to Yes

#5 Updated by bmbouter about 1 month ago

  • Status changed from POST to NEW

#6 Updated by fabricio.aguiar 25 days ago

  • Sprint/Milestone set to 3.0

#7 Updated by fabricio.aguiar 25 days ago

I also got this error when I was testing kickstarts installing CentOS 7

#8 Updated by bmbouter 5 days ago

  • Assignee deleted (bmbouter)

Please register to edit this issue

Also available in: Atom PDF