Pulp

The best values here are Content-Type of application/x-gzip, and no Content-Encoding.

mod_mime_magic on RHEL and related distributions ships with a default configuration that (in some opinions, mine included) incorrectly assumes that all gzipped files should have their Content-Encoding header set to x-gzip. Unfortunately, that's lead to misconceptions about the correct use of these headers. This bit of RFC 7231 nicely summarizes why it's incorrect:

   If the media type includes an inherent encoding, such as a data
   format that is always compressed, then that encoding would not be
   restated in Content-Encoding even if it happens to be the same
   algorithm as one of the content codings.  Such a content coding would
   only be listed if, for some bizarre reason, it is applied a second
   time to form the representation.

https://tools.ietf.org/html/rfc7231#section-3.1.2.2

The implication seems that my suggestion of Content-Type: text/xml, Content-Encoding: gzip for an .xml.gz file is incorrect, but I can't agree...

I think this part of the RFC is trying to say that it would be incorrect to specify the encoding in Content-Encoding if the encoding is already intrinsic to the Content-Type. For example, for a file compressed by gzip, it would be wrong to respond with Content-Type: application/x-gzip, Content-Encoding: gzip. Unless the file were compressed by gzip twice ("for some bizarre reason" in the RFC's words).

This isn't a problem for reporting Content-type: text/xml, Content-Encoding: gzip, since no inherent encoding from text/xml is being restated in Content-Encoding.

The point is that, if the client applies the decoding corresponding to the server's reported encoding, then the resulting content should be of the type reported in Content-Type. This is true when reporting Content-Type: text/xml, Content-Encoding: gzip, for an .xml.gz file. It would allow browsers to directly open and view the files.

Here is a helpful question to consider when trying to understand the difference between Content-Type and Content-Encoding; when would it be appropriate for a response to have the Content-Type header set to "application/x-gzip"?

The answer usually boils down to: when you want to use HTTP to retrieve a file that happens to be gzip compressed, but you do not want the HTTP layer to automatically decompress it or treat it differently than any other file.

Or to think of it another way, in the case where the Content-Type header is "application/x-gzip", that is by definition a "media type includes an inherent encoding, such as a data format that is always compressed", as described in the RFC.

In the case of yum metadata files, an application reasonably wants to receive the original gzipped file, put it somewhere, and later think about opening it and unzipping it. That is certainly pulp's expectation. A file might be retrieved via http, ftp, or from a local filesystem; I want each of those mechanisms to just give me the file unmodified.

Many HTTP libraries, such as python-requests, will automatically unzip the message body when Content-Encoding is set to gzip. That illustrates the different expected use cases. As mentioned in my first comment here, there are some servers on the internet putting misleading Content-Encoding headers on responses. Pulp has encountered this problem, and even has a work-around to ignore the content-encoding for any file that ends in .gz, to prevent python-requests from auto-unzipping it: https://github.com/pulp/nectar/blob/python-nectar-1.5.1-1/nectar/downloaders/threaded.py#L321

Putting "gzip" in either the Content-Type or Content-Encoding header depends on the use case. Do you want the zipped blob? Or do you want the http layer to be aware of the encoding and remove it on the client side, so you can immediately start reading unzipped bytes? For pulp's purposes, I think we want to be serving zipped blobs that clients will write to disk, likely cache for a period of time, and then open for reading as they see fit.

jcline@redhat.com wrote:

We're pretty clearly violating the RFC and the most correct value would be Content-Type: text/xml, Content-Encoding: gzip. I was not aware that mod_xsendfile tosses out Content-Encoding (although we're not even setting the Content-Encoding in pulp.server.content.web.views).

This would be correct if we wanted to deliver an XML file, which I don't think is what we want. We want to deliver whatever was requested, which in this case is a gzip file. What's inside that gzip file is a separate concern, to be handled by the client after receiving the requested file.

While that's my opinion, it seemed like a good idea to ask "How does fedora do it?"

Here's how:

curl -v http://mirrors.rit.edu/fedora/fedora/linux/releases/23/Everything/x86_64/os/repodata/0fa09bb5f82e4a04890b91255f4b34360e38ede964fe8328f7377e36f06bad27-primary.xml.gz >/dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 129.21.171.72...
* Connected to mirrors.rit.edu (129.21.171.72) port 80 (#0)
> GET /fedora/fedora/linux/releases/23/Everything/x86_64/os/repodata/0fa09bb5f82e4a04890b91255f4b34360e38ede964fe8328f7377e36f06bad27-primary.xml.gz HTTP/1.1
> Host: mirrors.rit.edu
> User-Agent: curl/7.43.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-Type: application/x-gzip
< Accept-Ranges: bytes
< ETag: "146301521"
< Last-Modified: Sat, 31 Oct 2015 17:30:17 GMT
< Content-Length: 12106437
< Date: Mon, 04 Apr 2016 22:04:18 GMT
< Server: lighttpd/1.4.39

If we did want to mess around with Content-Encoding: gzip, the filenames should have the appropriate extension for the encoded content type. So, if we really do want to send Content-Type: xml, Content-Encoding: gzip, all the metadata XML files should end in .xml, not .gz.

In my opinion, this is a bug.

To fix this, when responding to a request for a gzipped file, like the yum metadata, pulp should include "Content-Type: application/x-gzip" for gzipped files, and not use the "Content-Encoding" header.

I'm not entirely sure that #1868 is a dupe of this (but I'm pretty sure...). Anyone tackling this issue should take a look in there, though, due to the excellent amount of additional detail recorded there.

While investigating this on EL7 using pulp 2.8.3, I found some interesting behavior.

Created a YUM repository in pulp using the https://repos.fedorapeople.org/repos/pulp/pulp/fixtures/rpm/ repository named "zoo"

Sync and publish the "zoo" repository.

Created an ISO repository in pulp named "files".

Then, used curl to GET the published "updateinfo.xml.gz" and noted the Content-Type: text/xml.

[jortel@el7u ~]$ curl -v GET --insecure https://el7u.redhat.com/pulp/repos/repos/pulp/pulp/fixtures/rpm/repodata/da5a83a4af0e670b1a0a582743555f20cad88da4071f61f20b5c6ab4e3b16df8-updateinfo.xml.gz
* Could not resolve host: GET; Name or service not known
* Closing connection 0
curl: (6) Could not resolve host: GET; Name or service not known
* About to connect() to el7u.redhat.com port 443 (#1)
*   Trying 192.168.122.113...
* Connected to el7u.redhat.com (192.168.122.113) port 443 (#1)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*     subject: CN=el7u.redhat.com,O=Default Company Ltd,L=Default City,C=XX
*     start date: Sep 22 17:57:57 2016 GMT
*     expire date: Sep 22 17:57:57 2017 GMT
*     common name: el7u.redhat.com
*     issuer: CN=el7u.redhat.com,O=Default Company Ltd,L=Default City,C=XX
> GET /pulp/repos/repos/pulp/pulp/fixtures/rpm/repodata/da5a83a4af0e670b1a0a582743555f20cad88da4071f61f20b5c6ab4e3b16df8-updateinfo.xml.gz HTTP/1.1
> User-Agent: curl/7.29.0
> Host: el7u.redhat.com
> Accept: */*
> 
* skipping SSL peer certificate verification
* NSS: client certificate not found (nickname not specified)
* skipping SSL peer certificate verification
< HTTP/1.1 200 OK
< Date: Wed, 18 Jan 2017 14:48:30 GMT
< Server: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.1e-fips mod_fcgid/2.3.9 mod_wsgi/3.4 Python/2.7.5
< Last-Modified: Mon, 16 Jan 2017 21:14:55 GMT
< ETag: "281-5463cacda5029"
< Content-Length: 641
< Content-Type: text/xml

I uploaded the (previously downloaded from "zoo") updateinfo.xml.gz into the "files" repo.

Then, used curl to GET the uploaded "updateinfo.xml.gz" and noted the Content-Type: application/x-gzip.

[jortel@el7u ~]$ curl -v GET --insecure https://el7u.redhat.com/pulp/isos/iso/da5a83a4af0e670b1a0a582743555f20cad88da4071f61f20b5c6ab4e3b16df8-updateinfo.xml.gz
* Could not resolve host: GET; Name or service not known
* Closing connection 0
curl: (6) Could not resolve host: GET; Name or service not known
* About to connect() to el7u.redhat.com port 443 (#1)
*   Trying 192.168.122.113...
* Connected to el7u.redhat.com (192.168.122.113) port 443 (#1)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*     subject: CN=el7u.redhat.com,O=Default Company Ltd,L=Default City,C=XX
*     start date: Sep 22 17:57:57 2016 GMT
*     expire date: Sep 22 17:57:57 2017 GMT
*     common name: el7u.redhat.com
*     issuer: CN=el7u.redhat.com,O=Default Company Ltd,L=Default City,C=XX
> GET /pulp/isos/iso/da5a83a4af0e670b1a0a582743555f20cad88da4071f61f20b5c6ab4e3b16df8-updateinfo.xml.gz HTTP/1.1
> User-Agent: curl/7.29.0
> Host: el7u.redhat.com
> Accept: */*
> 
* skipping SSL peer certificate verification
* NSS: client certificate not found (nickname not specified)
< HTTP/1.1 200 OK
< Date: Wed, 18 Jan 2017 14:45:39 GMT
< Server: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.1e-fips mod_fcgid/2.3.9 mod_wsgi/3.4 Python/2.7.5
< Last-Modified: Wed, 18 Jan 2017 14:41:51 GMT
< ETag: "281-5465f6ad18d92"
< Accept-Ranges: bytes
< Content-Length: 641
< Content-Type: application/x-gzip
< 

I added tried the work around in comment 15:

MimeMagicFile /dev/null

to every where I could think of:

• At the end of httpd.conf (global)
• In ssl.conf in <VirtualHost default:443/>
• At the top of pulp_rpm.conf (global)

Note, "MimeMagicFile NEVER_EVER_USE" already exists globally in pulp_docker.conf

Also, tried adding (global) to the top of pulp_rpm.conf:

AddType application/x-gzip .gz 

NOTHING helped.

After completely denuding httpd of any modules that work with content-type, I can confirm that this is all related to xsendfile throwing away the guessed Content-Encoding:
https://github.com/pulp/pulp/blob/master/server/pulp/server/content/web/views.py#L79

I created three files: foo, foo.xml, foo.xml.gz, and then I ran mimetypes.guess_type on them. foo reported (None, None). foo.xml reported ('text/xml', None), foo.xml.gz reported ('text/xml', 'gzip'). For fun, I also ran it against foo.gzip, which reported (None, 'gzip'), which seems especially annoying.

guess_type ultimately uses posixpath.splitext to split the extension off, so it will never see '.xml.gz' as a file's extension. If it does see '.gz' at the end of a file, it quickly shorts out, having at least set Content-Encoding to 'gzip', which we then ignore. It does splitext the file again to try to find the content type, which is how we end up with ('text/xml', 'gzip'). The encoding is decided by the contents of the mimetypes.encodings_map dictionary.

Easy solution, low risk:
The mimetypes module is a builtin module. You can mutate it all you want, but probably don't want to since other libs might be depending on it behaving a certain way. Looking at the source, you can see that all of the module-level functions pass back to a _db object, which is itself just an instance of mimetypes.MimeTypes. So, for the content serving views, since xsendfile throws away Content-Encoding, we should use our own instance of mimetypes.MimeTypes with an empty encodings_map, which will cause anything.whatever.gz to receive content-type application/gzip.

I'll implement and try to test this tomorrow, and also look into application/gzip vs. application/x-gzip. /etc/mime.types on my system has application/gzip, so I expect I won't have to do anything with or about the Content-Type value, and can instead just let the local system do what it thinks is right via mimetypes.MimeTypes().guess_type, after clearing its encodings map.