Project

Profile

Help

Task #1195

Updated by bmbouter over 8 years ago

For lazy-sync to work Squid needs to have another software component act as a client presenting the correct certificate to the feed URL. That streaming component is called the "pulp-streamer" for now. 

 h2. Configuration 

 The configuration will come from the server.conf file. This is appropriate because the streamer will need many things from server.conf (ie: database). 

 * have the streamer use an entry named 'streamer_port' in the [lazy] section of server.conf to use its port. This needs to have a default too. I'll suggest 8751 which is unused according to an IANA page I looked at. 
 * have the streamer use an entry named 'streamer_interface' in the [lazy] section of server.conf. This field should default to 'lo' which will cause it to listen on localhost only by default. The field accepts a comma separated list which can be used to limit which interfaces it should listen on. 
 * The streamer will use a header to tell squid how long to cache content it is delivering to squid. This should be configurable using the streamer_cache_timeout setting in the [lazy] section. This is expressed in seconds and will default to 86400 (the number of seconds in 1 day). 

 h2. Requirements 

 * Use the "unit catalog" to determine which hostname and URL the incoming request coming in should be translated to 
 * Make a request to the URL determined by the "unit catalog" to present the correct SSL client certificate corresponding with that request 
 * Have the streamer verify the identity of the server side of the SSL connection consistent with Pulp's existing functionality 
 * Pass through the headers from the server as-is. This will require #1179 to be fixed first 
 * Overwrite and set the "Cache Control" header to the streamer_cache_timeout setting specified by server.conf. It also should append "public". The "public" part is not configurable. 
 * Headers must be delivered to the client before any data. 
 * Stream the data to the client as the streamer receives it. This is not a store-and-forward software it should stream in chunks. 
 * Concurrently handle multiple downloads at a time efficiently. 

 h2. Alternate Content sources 

 The streamer needs to be integrated to allow alternate content sources. In this usage when Pulp has an alternate content source configured a lazy repo can receive content from the alternate content source by the streamer reading the bits from disk instead of the upstream --feed location. 

 A proof of concept stream based was developed (see attachment), use that as a starting point. This story does not do any rpm packaging, init script work, or systemd unit work; that is all part of another story.

Back