Issue #7958
closed
Last Modified header not updated on repodata updates
Description
For a repo, any content that is changed or updated after an action like repo sync, the repo metadata file gets updated . ( repomd.xml ) But on trying to access it, the response received does not have the Last Modified header updated to match the time stamp that the repomd.xml was actually modified.
It continues to send out a response with the Last Modified time set to an old updates timestamp This is causing issues where the browser or any other caching proxy is getting a 304 Not modified response and hece the cache is not getting invalidated and the new updated version is not received.
This may not be the case with only repomd.xml but with all content serverd by pulp. This is noticed in repomd.xml because its the only one that's modified frequently and the others are all mostly new content/new rpm or iso versions.
# ls -l /var/lib/pulp/published/yum/http/repos/centos/7/x86_64/updates/repodata/repomd.xml
-rw-r--r-- 1 apache apache 2171 Dec 8 21:30 /var/lib/pulp/published/yum/http/repos/centos/7/x86_64/updates/repodata/repomd.xml
The file timestamp modification time is 21:30 Dec 8 PDT (Wed, 09 Dec 2020 05:30 GMT) Below is the response header :
------
Request Method: GET
Status Code: 304 Not Modified
Remote Address: 10.**.**.**:80
Referrer Policy: strict-origin-when-cross-origin
Content-Length: 2171
Content-Type: text/xml
Date: Wed, 09 Dec 2020 05:41:15 GMT
ETag: "d41d8cd98f00b204e9800998ecf8427e"
Last-Modified: Wed, 09 Dec 2020 05:29:07 GMT
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_wsgi/3.4 Python/2.7.5
----------
On clearing the cache it gets a fresh version which is the latest. But it fails to invalidate out-dated caches due to this issue with Last-Modified not getting an update and the 304 response from the pulp server.
Updated by deepthireddy21 over 3 years ago
Found some more details.
This seems to be an issue with the ETag . Even when the content changes , the Etag header does not change. This is causing the 304 Not modified response and not the Last Modified timestamp .
Updated by ttereshc over 3 years ago
- Description updated (diff)
- Tags Pulp 2 added
Updated by ggainey over 3 years ago
Having done some investigation, it appears that a Pulp2 instalklation returns the same ETag for everything. "d41d8cd98f00b204e9800998ecf8427e" is the MD5 checksum of an empty string.
So far I don't see us doing anything explicit with ETag.
A workaround is to turn off the ETag header completely:
$ sudo vi /etc/httpd/conf.d/pulp_content.conf
Header unset ETag
Anything that cares, will then rely on the Last-Modified header.
Investigation continues.
Updated by ggainey over 3 years ago
Ok, more data - #c3 is slightly in error, as I intended the workaround to be
$ sudo vi /etc/httpd/conf.d/pulp_content.conf
Header unset ETag
FileETagNone
As it turns out, if you just "Header unset ETag" you start getting correct ETags:
[vagrant@pulp2-nightly-pulp3-source-centos7 ~]$ wget -v -S --no-check-certificate https://pulp2-nightly-pulp3-source-centos7/pulp/repos/rpm-unsigned/repodata/repomd.xml
--2020-12-09 15:51:44-- https://pulp2-nightly-pulp3-source-centos7/pulp/repos/rpm-unsigned/repodata/repomd.xml
Resolving pulp2-nightly-pulp3-source-centos7 (pulp2-nightly-pulp3-source-centos7)... 127.0.0.1
Connecting to pulp2-nightly-pulp3-source-centos7 (pulp2-nightly-pulp3-source-centos7)|127.0.0.1|:443... connected.
HTTP request sent, awaiting response...
HTTP/1.1 200 OK
Date: Wed, 09 Dec 2020 15:51:44 GMT
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_wsgi/3.4 Python/2.7.5
Last-Modified: Wed, 09 Dec 2020 15:08:12 GMT
ETag: "869-5b6096d916e72"
Content-Length: 2153
Keep-Alive: timeout=5, max=10000
Connection: Keep-Alive
Content-Type: text/xml
Length: 2153 (2.1K) [text/xml]
Saving to: ‘repomd.xml.19’
So the workaround in #c3 is better than I thought, because it clears the way for 'correct' ETags to be generated.
Note: don't forget to restart services after changing config-files:
$ sudo systemctl restart httpd squid
Updated by ggainey over 3 years ago
- Status changed from NEW to CLOSED - WONTFIX
Pulp2 is in critical-fixes-only mode at this point in its lifecycle, and the settings-workaround in #note-3 fixes the issue. Closing as WONTFIX
.