Project

Profile

Help

Issue #2587

on_demand downloads aren't being cached by Squid

Added by Ichimonji10 about 3 years ago. Updated 11 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Severity:
2. Medium
Version:
Platform Release:
2.12.1
Blocks Release:
OS:
Backwards Incompatible:
No
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
QA Contact:
Complexity:
Smash Test:
Verified:
Yes
Verification Required:
No
Sprint:
Sprint 15

Description

The on_demand download policy lets Pulp defer the download of a content unit until it's requested. When a content unit is requested for the first time, Pulp downloads the file and streams it to the requesting client as the content becomes available. When the same content unit is requested in the future, Squid serves the file directly. For whatever reason, Squid is not serving cached copies of content units.

The tests cases that demonstrate this issue are:

  • pulp_smash.tests.rpm.api_v2.test_download_policies.OnDemandTestCase.test_rpm_cache_control_header
  • pulp_smash.tests.rpm.api_v2.test_download_policies.OnDemandTestCase.test_same_rpm_cache_header
  • pulp_smash.tests.rpm.api_v2.test_download_policies.SwitchPoliciesTestCase.test_background_to_on_demand
  • pulp_smash.tests.rpm.api_v2.test_download_policies.SwitchPoliciesTestCase.test_immediate_to_on_demand

Attached is the output produced by Pulp Smash when these tests are executed (pending this pr). The test failures clearly describe cache misses occurring. I can reproduce this issue on at least Fedora 24 and Fedora 25 with Pulp 2.12 and Pulp 2.13. Manual investigation of all these systems shows that:

  • Squid is running.
  • All services are OK, according to systemctl status.
  • No error messages are logged, according to journalctl.

Here's some of the packages installed on my Fedora 24 system with Pulp 2.12 installed:

[root@fedora-24-pulp-2-12 ~]# rpm -qa | grep squid | sort
squid-3.5.20-1.fc24.x86_64
[root@fedora-24-pulp-2-12 ~]# rpm -qa | grep -i pulp | sort
pulp-admin-client-2.12.1-0.1.alpha.git.67.22eb720.fc24.noarch
pulp-docker-admin-extensions-2.3.1-0.1.alpha.git.1.ad83c84.fc24.noarch
pulp-docker-plugins-2.3.1-0.1.alpha.git.1.ad83c84.fc24.noarch
pulp-ostree-admin-extensions-1.2.1-0.1.alpha.git.22.19760f7.fc24.noarch
pulp-ostree-plugins-1.2.1-0.1.alpha.git.22.19760f7.fc24.noarch
pulp-puppet-admin-extensions-2.12.1-0.1.alpha.git.1.d407541.fc24.noarch
pulp-puppet-plugins-2.12.1-0.1.alpha.git.1.d407541.fc24.noarch
pulp-python-admin-extensions-2.0.1-0.1.alpha.git.2.be7d11e.fc24.noarch
pulp-python-plugins-2.0.1-0.1.alpha.git.2.be7d11e.fc24.noarch
pulp-rpm-admin-extensions-2.12.1-0.1.alpha.git.23.f90340d.fc24.noarch
pulp-rpm-plugins-2.12.1-0.1.alpha.git.23.f90340d.fc24.noarch
pulp-selinux-2.12.1-0.1.alpha.git.67.22eb720.fc24.noarch
pulp-server-2.12.1-0.1.alpha.git.67.22eb720.fc24.noarch
python-kombu-3.0.33-6.pulp.fc24.noarch
python-pulp-bindings-2.12.1-0.1.alpha.git.67.22eb720.fc24.noarch
python-pulp-client-lib-2.12.1-0.1.alpha.git.67.22eb720.fc24.noarch
python-pulp-common-2.12.1-0.1.alpha.git.67.22eb720.fc24.noarch
python-pulp-docker-common-2.3.1-0.1.alpha.git.1.ad83c84.fc24.noarch
python-pulp-oid_validation-2.12.1-0.1.alpha.git.67.22eb720.fc24.noarch
python-pulp-ostree-common-1.2.1-0.1.alpha.git.22.19760f7.fc24.noarch
python-pulp-puppet-common-2.12.1-0.1.alpha.git.1.d407541.fc24.noarch
python-pulp-python-common-2.0.1-0.1.alpha.git.2.be7d11e.fc24.noarch
python-pulp-repoauth-2.12.1-0.1.alpha.git.67.22eb720.fc24.noarch
python-pulp-rpm-common-2.12.1-0.1.alpha.git.23.f90340d.fc24.noarch
python-pulp-streamer-2.12.1-0.1.alpha.git.67.22eb720.fc24.noarch
pulp smash output.txt (4.47 KB) Ichimonji10, 02/14/2017 09:26 PM pulp smash output.txt

Related issues

Duplicated by Pulp - Issue #3140: pulp_streamer doesnt set HTTP response headers, causing squid cache is not used CLOSED - DUPLICATE Actions

Associated revisions

Revision 0ced6570 View on GitHub
Added by jortel@redhat.com about 3 years ago

Fix streamer http header management. closes #2587

Revision 0ced6570 View on GitHub
Added by jortel@redhat.com about 3 years ago

Fix streamer http header management. closes #2587

History

#1 Updated by semyers about 3 years ago

The version numbers are for a different build than the 2.12.1 beta, but these failures show up in the builds done against the 2.12.1 beta and do apply to that release.

#2 Updated by mhrivnak about 3 years ago

Is this behavior the same in 2.12.0? Or does it appear to be a regression only in 2.12.1?

#3 Updated by mhrivnak about 3 years ago

Seeing the system log and apache request logs from during this test case's run would be helpful.

#4 Updated by mhrivnak about 3 years ago

Wildly speculating, here is our top suspect for introducing the regression:

https://github.com/pulp/pulp/pull/2935

but this is also worth considering:

https://github.com/pulp/pulp/pull/2932

#5 Updated by mhrivnak about 3 years ago

  • Status changed from NEW to POST
  • Assignee set to jortel@redhat.com
  • Sprint/Milestone set to 33

#7 Updated by jortel@redhat.com about 3 years ago

  • Project changed from RPM Support to Pulp

#8 Updated by jortel@redhat.com about 3 years ago

  • Status changed from POST to MODIFIED

#9 Updated by bizhang about 3 years ago

  • Platform Release set to 2.12.1
  • Blocks Release deleted (2.12.z)

#10 Updated by bizhang about 3 years ago

  • Status changed from MODIFIED to ON_QA

#11 Updated by bizhang about 3 years ago

  • Version set to 2.12.1

#12 Updated by bizhang about 3 years ago

  • Version deleted (2.12.1)

#13 Updated by Ichimonji10 about 3 years ago

  • Verified changed from No to Yes

This issue is fixed in the current 2.12.1 and 2.13.0 nightlies.

#14 Updated by bizhang about 3 years ago

  • Status changed from ON_QA to CLOSED - CURRENTRELEASE

#15 Updated by dkliban@redhat.com about 2 years ago

  • Duplicated by Issue #3140: pulp_streamer doesnt set HTTP response headers, causing squid cache is not used added

#17 Updated by bmbouter almost 2 years ago

  • Sprint set to Sprint 15

#18 Updated by bmbouter almost 2 years ago

  • Sprint/Milestone deleted (33)

#19 Updated by bmbouter 11 months ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF