Task #1195
Updated by bmbouter over 9 years ago
For lazy-sync to work Squid needs to have another software component act as a client presenting the correct certificate to the feed URL. That streaming component is called the "pulp-streamer" for now.
h2. Configuration
The configuration will come from the server.conf file. This is appropriate because the streamer will need many things from server.conf (ie: database).
* have the streamer use an entry named 'streamer_port' in the [lazy] section of server.conf to use its port. This needs to have a default too. I'll suggest 8751 which is unused according to an IANA page I looked at.
* have the streamer use an entry named 'streamer_interface' in the [lazy] section of server.conf. This field should default to 'lo' which will cause it to listen on localhost only by default. The field accepts a comma separated list which can be used to limit which interfaces it should listen on.
* The streamer will use a header to tell squid how long to cache content it is delivering to squid. This should be configurable using the streamer_cache_timeout setting in the [lazy] section. This is expressed in seconds and will default to 86400 (the number of seconds in 1 day).
h2. Requirements
* Use the "unit catalog" to determine which hostname and URL the incoming request coming in should be translated to
* Make a request to the URL determined by the "unit catalog" to present the correct SSL client certificate corresponding with that request
* Have the streamer verify the identity of the server side of the SSL connection consistent with Pulp's existing functionality
* Pass through the headers from the server as-is. This will require #1179 to be fixed first
* Overwrite and set the "Cache Control" header to the streamer_cache_timeout setting specified by server.conf. It also should append "public". The "public" part is not configurable.
* Headers must be delivered to the client before any data.
* Stream the data to the client as the streamer receives it. This is not a store-and-forward software it should stream in chunks.
* Concurrently handle multiple downloads at a time efficiently.
h2. Alternate Content sources
The streamer needs to be integrated to allow alternate content sources. In this usage when Pulp has an alternate content source configured a lazy repo can receive content from the alternate content source by the streamer reading the bits from disk instead of the upstream --feed location.
A proof of concept stream based was developed (see attachment), use that as a starting point. This story does not do any rpm packaging, init script work, or systemd unit work; that is all part of another story.
.