Project

Profile

Help

Story #4181

closed

Story #3693: Lazy for Pulp3

As a streamer user, I have content protection like in Pulp itself

Added by bmbouter over 5 years ago. Updated over 4 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Quarter:

Description

Problem

Content that should be protected could be fetched by unauthorized clients via the streamer which could either (a) fetch the content fresh and hand it to the client or (b) serve up an already-saved Artifact from Pulp's filesystem.

One Solution: Signed URLs

This is how we did it in Pulp 2. The client authorization for that content is checked in the content app. Then when Pulp redirects the client it signs it with a time-limited signature. Then that redirect is validated by a webserver, e.g. apache, nginx, gunicorn, etc. If the url is valid, the url is reverse proxy sent to either squid or the streamer itself.

The downsides here are that each webserver will need to handle this differently so Pulp remaining webserver agnostic is unlikely.

Also it makes the architecture a lot more complicated introducing additional dependencies, crypto calls, and an additional webserver everywhere the streamer is run.

Another Solution: Don't Use Squid Temporarily

Add the content protection to the streamer app the same way it was added to the content app. This would cause squid to never be able to be used though because content cached in front of the streamer couldn't be protected and could be fetched by another client.

Not using squid mainly affects repos with policy='cache_only' which serves the content and then forgets it. This means there is no caching w/ that setting.

Note that with policy='on_demand' the only downsides are multiple requests arriving at the streamer for the same file before the first one completes would not be de-duplicated like squid would have done. Once the first request saves the Artifact, additional requests are de-duplicated.

Actions #1

Updated by bmbouter over 5 years ago

  • Parent issue set to #3693

Adding to the lazy epic.

Actions #2

Updated by bmbouter over 5 years ago

  • Description updated (diff)
  • Status changed from ASSIGNED to NEW
  • Assignee deleted (bmbouter)
Actions #3

Updated by bmbouter over 5 years ago

  • Description updated (diff)

I'm advocating for dropping squid in the call flow for now and adding content protection directly into the aiohttp streamer. We can add the streamer and maybe signed URLs in later releases.

The developer time and user complexity benefits of not shipping squid integration right away I think are a lot relative to the amount that caching is benefiting users. Also since it can be added later I don't think we need to focus on getting squid integrated right now. So with that reasoning I recommend not doing squid integration now, and resolving content protection by adding it to the streamer itself.

Actions #4

Updated by bmbouter over 5 years ago

  • Description updated (diff)
Actions #5

Updated by jortel@redhat.com over 5 years ago

I like the idea of the streamer implementing content protection identically to the content app. The streamer would delegate authorization to the ContentGuard. This seems simpler than signed URLs and l don't see any significant performance differences. Best I can tell, this decision supports making squid optional which I like.

Actions #6

Updated by bmbouter over 5 years ago

  • Status changed from NEW to POST
  • Assignee set to bmbouter

@jortel, I implemented the content protection in the streamer code at github.com/bmbouter/pulp_streamer/ master, so I'm moving to POST.

We can move to MODIFIED once the repo is moved to the Pulp organization pending a bigger review/test.

Actions #7

Updated by bmbouter over 5 years ago

Added by bmbouter over 5 years ago

Revision 8219efd5 | View on GitHub

Adds the content app to pulpcore.content

The Settings can be a lot simpler now that the architecture no longer required redirection. It does need to account for the content being served on a different HOST. This PR handles that.

Since the settings are now single values, they are much easier to override using Dynaconf, which is great.

The content app can be removed now that the streamer code is handling it all.

This PR is a prototype and has no docs. Use Pulp as normal , but run the streamer from github.com/bmbouter/pulp_streamer/ and run the streamer with:

gunicorn pulpcore.streamer:server --bind localhost:8080 --worker-class aiohttp.GunicornWebWorker -w 2

This PR needs pulp-smash changes from pulp_file so it requires the PR below:

Required PR: https://github.com/pulp/pulp_file/pull/140

https://pulp.plan.io/issues/4239 closes #4239

https://pulp.plan.io/issues/3698 closes #3698

https://pulp.plan.io/issues/3699 closes #3699

https://pulp.plan.io/issues/4181 closes #4181

https://pulp.plan.io/issues/4243 closes #4243

Added by bmbouter over 5 years ago

Revision 8219efd5 | View on GitHub

Adds the content app to pulpcore.content

The Settings can be a lot simpler now that the architecture no longer required redirection. It does need to account for the content being served on a different HOST. This PR handles that.

Since the settings are now single values, they are much easier to override using Dynaconf, which is great.

The content app can be removed now that the streamer code is handling it all.

This PR is a prototype and has no docs. Use Pulp as normal , but run the streamer from github.com/bmbouter/pulp_streamer/ and run the streamer with:

gunicorn pulpcore.streamer:server --bind localhost:8080 --worker-class aiohttp.GunicornWebWorker -w 2

This PR needs pulp-smash changes from pulp_file so it requires the PR below:

Required PR: https://github.com/pulp/pulp_file/pull/140

https://pulp.plan.io/issues/4239 closes #4239

https://pulp.plan.io/issues/3698 closes #3698

https://pulp.plan.io/issues/3699 closes #3699

https://pulp.plan.io/issues/4181 closes #4181

https://pulp.plan.io/issues/4243 closes #4243

Actions #8

Updated by bmbouter over 5 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100
Actions #9

Updated by daviddavis almost 5 years ago

  • Sprint/Milestone set to 3.0.0
Actions #10

Updated by bmbouter almost 5 years ago

  • Tags deleted (Pulp 3)
Actions #11

Updated by bmbouter over 4 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF