Story #4181
closedStory #3693: Lazy for Pulp3
As a streamer user, I have content protection like in Pulp itself
100%
Description
Problem¶
Content that should be protected could be fetched by unauthorized clients via the streamer which could either (a) fetch the content fresh and hand it to the client or (b) serve up an already-saved Artifact from Pulp's filesystem.
One Solution: Signed URLs¶
This is how we did it in Pulp 2. The client authorization for that content is checked in the content app. Then when Pulp redirects the client it signs it with a time-limited signature. Then that redirect is validated by a webserver, e.g. apache, nginx, gunicorn, etc. If the url is valid, the url is reverse proxy sent to either squid or the streamer itself.
The downsides here are that each webserver will need to handle this differently so Pulp remaining webserver agnostic is unlikely.
Also it makes the architecture a lot more complicated introducing additional dependencies, crypto calls, and an additional webserver everywhere the streamer is run.
Another Solution: Don't Use Squid Temporarily¶
Add the content protection to the streamer app the same way it was added to the content app. This would cause squid to never be able to be used though because content cached in front of the streamer couldn't be protected and could be fetched by another client.
Not using squid mainly affects repos with policy='cache_only' which serves the content and then forgets it. This means there is no caching w/ that setting.
Note that with policy='on_demand' the only downsides are multiple requests arriving at the streamer for the same file before the first one completes would not be de-duplicated like squid would have done. Once the first request saves the Artifact, additional requests are de-duplicated.
Updated by bmbouter about 6 years ago
- Description updated (diff)
- Status changed from ASSIGNED to NEW
- Assignee deleted (
bmbouter)
Updated by bmbouter about 6 years ago
- Description updated (diff)
I'm advocating for dropping squid in the call flow for now and adding content protection directly into the aiohttp streamer. We can add the streamer and maybe signed URLs in later releases.
The developer time and user complexity benefits of not shipping squid integration right away I think are a lot relative to the amount that caching is benefiting users. Also since it can be added later I don't think we need to focus on getting squid integrated right now. So with that reasoning I recommend not doing squid integration now, and resolving content protection by adding it to the streamer itself.
Updated by jortel@redhat.com about 6 years ago
I like the idea of the streamer implementing content protection identically to the content app. The streamer would delegate authorization to the ContentGuard. This seems simpler than signed URLs and l don't see any significant performance differences. Best I can tell, this decision supports making squid optional which I like.
Updated by bmbouter about 6 years ago
- Status changed from NEW to POST
- Assignee set to bmbouter
@jortel, I implemented the content protection in the streamer code at github.com/bmbouter/pulp_streamer/ master, so I'm moving to POST.
We can move to MODIFIED once the repo is moved to the Pulp organization pending a bigger review/test.
Updated by bmbouter about 6 years ago
PR available here: https://github.com/pulp/pulp/pull/3779
Added by bmbouter about 6 years ago
Added by bmbouter about 6 years ago
Revision 8219efd5 | View on GitHub
Adds the content app to pulpcore.content
The Settings can be a lot simpler now that the architecture no longer required redirection. It does need to account for the content being served on a different HOST. This PR handles that.
Since the settings are now single values, they are much easier to override using Dynaconf, which is great.
The content app can be removed now that the streamer code is handling it all.
This PR is a prototype and has no docs. Use Pulp as normal , but run the streamer from github.com/bmbouter/pulp_streamer/ and run the streamer with:
gunicorn pulpcore.streamer:server --bind localhost:8080 --worker-class aiohttp.GunicornWebWorker -w 2
This PR needs pulp-smash changes from pulp_file so it requires the PR below:
Required PR: https://github.com/pulp/pulp_file/pull/140
https://pulp.plan.io/issues/4239 closes #4239
https://pulp.plan.io/issues/3698 closes #3698
https://pulp.plan.io/issues/3699 closes #3699
Updated by bmbouter about 6 years ago
- Status changed from POST to MODIFIED
- % Done changed from 0 to 100
Applied in changeset pulp|8219efd59098fc5e883ea831d4fb38b7ae15422d.
Updated by bmbouter about 5 years ago
- Status changed from MODIFIED to CLOSED - CURRENTRELEASE
Adds the content app to pulpcore.content
The Settings can be a lot simpler now that the architecture no longer required redirection. It does need to account for the content being served on a different HOST. This PR handles that.
Since the settings are now single values, they are much easier to override using Dynaconf, which is great.
The content app can be removed now that the streamer code is handling it all.
This PR is a prototype and has no docs. Use Pulp as normal , but run the streamer from github.com/bmbouter/pulp_streamer/ and run the streamer with:
gunicorn pulpcore.streamer:server --bind localhost:8080 --worker-class aiohttp.GunicornWebWorker -w 2
This PR needs pulp-smash changes from pulp_file so it requires the PR below:
Required PR: https://github.com/pulp/pulp_file/pull/140
https://pulp.plan.io/issues/4239 closes #4239
https://pulp.plan.io/issues/3698 closes #3698
https://pulp.plan.io/issues/3699 closes #3699
https://pulp.plan.io/issues/4181 closes #4181
https://pulp.plan.io/issues/4243 closes #4243