Pulp

I'm not able to reproduce this on latest master, I will try with 3.11

In [1]: Artifact.objects.all() 
Out[1]: <QuerySet [<Artifact: pk=864fb941-43c8-4ff6-b747-0c8e755881c4>]>

In [2]: exit


## A completely different file from ^^ one, this one has never been downloaded before

(pulp) [vagrant@pulp3-source-centos7 pulpcore]$ curl -k http://pulp3-source-centos7.localhost.example.com/pulp/content/fixture/duck-0.8-1.noarch.rpm   -H "Range: bytes=1-200" > duck-0.8-1.noarch.rpm                                                                                                                        
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   200  100   200    0     0    464      0 --:--:-- --:--:-- --:--:--   464
(pulp) [vagrant@pulp3-source-centos7 pulpcore]$ ls -al
total 240
... snip ...
-rw-rw-r--. 1 vagrant vagrant   200 Jun  4 16:16 duck-0.8-1.noarch.rpm
-rw-rw-r--. 1 vagrant vagrant   200 Jun  4 16:09 fox-1.1-2.noarch.rpm
... snip ...

(pulp) [vagrant@pulp3-source-centos7 pulpcore]$ python manage.py shell_plus                                          
... snip ...

In [1]: Artifact.objects.all()
Out[1]: <QuerySet [<Artifact: pk=864fb941-43c8-4ff6-b747-0c8e755881c4>, <Artifact: pk=47e95aca-8bf2-4268-8a74-2238df24eeb7>]>

Both files were streamed but both provided 200 bytes back as requested by curl.

I'm not entirely sure if ^^ was a fluke or not, and maybe it is reproducible on other versions, but in any case I did verify this behavior on 3.11

I was also able to reproduce this (on 3.11), by

1. syncing an on_demand repo
2. changeing the repo to immediate it
3. syncing it again

The behavior was exactly the same, which surprised me

It was a fluke, I can reproduce this on master.

https://github.com/pulp/pulpcore/pull/1399

PR: https://github.com/pulp/pulpcore/pull/1399

So what's the recommended behavior here?

1. Have it stream and save all of the file even though the client asked for a portion of it (and serve just that to the client)?
2. Have it fetch what the client is asking for every time (basically making ignoring policy=immediate)?

IMO, #1 is the better option. But then, maybe it depends on why exactly Anaconda is making the range requests. Does it make range requests against every RPM to read the headers, or just ones that it thinks it may need to install?

I think it only fetches the ones it thinks it may need to install. I agree that option 1) is probably preferred, but i don't think that option 2) is that terrible (assuming the header request will use the on-disk rpm if available, that was a little unclear).

Most likely the header will be requested and then the rpm later on.

I'm going to pursue option 1 as it will result in fewer requests to external servers over time.

Also I think we need to get the response headers right, so I'm going to mimic what is responded by an official centos mirror for example:

$ curl -i https://packages.oit.ncsu.edu/centos/7/os/x86_64/Packages/sg3_utils-1.37-19.el7.x86_64.rpm  -H "Range: bytes=1384-44339"
HTTP/1.1 206 Partial Content
Date: Thu, 08 Jul 2021 14:35:58 GMT
Server: Apache
Last-Modified: Fri, 03 Apr 2020 21:08:05 GMT
ETag: "a16b8-5a269504aa76b"
Accept-Ranges: bytes
Content-Length: 42956
Content-Range: bytes 1384-44339/661176
Content-Type: application/x-rpm

Warning: Binary output can mess up your terminal. Use "--output -" to tell 
Warning: curl to output it to your terminal anyway, or consider "--output 
Warning: <FILE>" to save to a file.