Project

Profile

Help

Story #4181

Story #3693: Lazy for Pulp3

As a streamer user, I have content protection like in Pulp itself

Added by bmbouter 11 months ago. Updated 6 months ago.

Status:
MODIFIED
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Platform Release:
Blocks Release:
Backwards Incompatible:
No
Groomed:
No
Sprint Candidate:
No
Tags:
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:

Description

Problem

Content that should be protected could be fetched by unauthorized clients via the streamer which could either (a) fetch the content fresh and hand it to the client or (b) serve up an already-saved Artifact from Pulp's filesystem.

One Solution: Signed URLs

This is how we did it in Pulp 2. The client authorization for that content is checked in the content app. Then when Pulp redirects the client it signs it with a time-limited signature. Then that redirect is validated by a webserver, e.g. apache, nginx, gunicorn, etc. If the url is valid, the url is reverse proxy sent to either squid or the streamer itself.

The downsides here are that each webserver will need to handle this differently so Pulp remaining webserver agnostic is unlikely.

Also it makes the architecture a lot more complicated introducing additional dependencies, crypto calls, and an additional webserver everywhere the streamer is run.

Another Solution: Don't Use Squid Temporarily

Add the content protection to the streamer app the same way it was added to the content app. This would cause squid to never be able to be used though because content cached in front of the streamer couldn't be protected and could be fetched by another client.

Not using squid mainly affects repos with policy='cache_only' which serves the content and then forgets it. This means there is no caching w/ that setting.

Note that with policy='on_demand' the only downsides are multiple requests arriving at the streamer for the same file before the first one completes would not be de-duplicated like squid would have done. Once the first request saves the Artifact, additional requests are de-duplicated.

Associated revisions

Revision 8219efd5 View on GitHub
Added by bmbouter 10 months ago

Adds the content app to pulpcore.content

The Settings can be a lot simpler now that the architecture no longer
required redirection. It does need to account for the content being
served on a different HOST. This PR handles that.

Since the settings are now single values, they are much easier to
override using Dynaconf, which is great.

The content app can be removed now that the streamer code is handling it
all.

This PR is a prototype and has no docs. Use Pulp as normal , but run the
streamer from github.com/bmbouter/pulp_streamer/ and run the streamer
with:

`gunicorn pulpcore.streamer:server --bind localhost:8080 --worker-class aiohttp.GunicornWebWorker -w 2`

This PR needs pulp-smash changes from pulp_file so it requires the PR
below:

Required PR: https://github.com/pulp/pulp_file/pull/140

https://pulp.plan.io/issues/4239
closes #4239

https://pulp.plan.io/issues/3698
closes #3698

https://pulp.plan.io/issues/3699
closes #3699

https://pulp.plan.io/issues/4181
closes #4181

https://pulp.plan.io/issues/4243
closes #4243

Revision 8219efd5 View on GitHub
Added by bmbouter 10 months ago

Adds the content app to pulpcore.content

The Settings can be a lot simpler now that the architecture no longer
required redirection. It does need to account for the content being
served on a different HOST. This PR handles that.

Since the settings are now single values, they are much easier to
override using Dynaconf, which is great.

The content app can be removed now that the streamer code is handling it
all.

This PR is a prototype and has no docs. Use Pulp as normal , but run the
streamer from github.com/bmbouter/pulp_streamer/ and run the streamer
with:

`gunicorn pulpcore.streamer:server --bind localhost:8080 --worker-class aiohttp.GunicornWebWorker -w 2`

This PR needs pulp-smash changes from pulp_file so it requires the PR
below:

Required PR: https://github.com/pulp/pulp_file/pull/140

https://pulp.plan.io/issues/4239
closes #4239

https://pulp.plan.io/issues/3698
closes #3698

https://pulp.plan.io/issues/3699
closes #3699

https://pulp.plan.io/issues/4181
closes #4181

https://pulp.plan.io/issues/4243
closes #4243

History

#1 Updated by bmbouter 11 months ago

  • Parent task set to #3693

Adding to the lazy epic.

#2 Updated by bmbouter 11 months ago

  • Description updated (diff)
  • Status changed from ASSIGNED to NEW
  • Assignee deleted (bmbouter)

#3 Updated by bmbouter 11 months ago

  • Description updated (diff)

I'm advocating for dropping squid in the call flow for now and adding content protection directly into the aiohttp streamer. We can add the streamer and maybe signed URLs in later releases.

The developer time and user complexity benefits of not shipping squid integration right away I think are a lot relative to the amount that caching is benefiting users. Also since it can be added later I don't think we need to focus on getting squid integrated right now. So with that reasoning I recommend not doing squid integration now, and resolving content protection by adding it to the streamer itself.

#4 Updated by bmbouter 11 months ago

  • Description updated (diff)

#5 Updated by jortel@redhat.com 11 months ago

I like the idea of the streamer implementing content protection identically to the content app. The streamer would delegate authorization to the ContentGuard. This seems simpler than signed URLs and l don't see any significant performance differences. Best I can tell, this decision supports making squid optional which I like.

#6 Updated by bmbouter 11 months ago

  • Status changed from NEW to POST
  • Assignee set to bmbouter

@jortel, I implemented the content protection in the streamer code at github.com/bmbouter/pulp_streamer/ master, so I'm moving to POST.

We can move to MODIFIED once the repo is moved to the Pulp organization pending a bigger review/test.

#7 Updated by bmbouter 10 months ago

#8 Updated by bmbouter 10 months ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#9 Updated by daviddavis 6 months ago

  • Sprint/Milestone set to 3.0

#10 Updated by bmbouter 6 months ago

  • Tags deleted (Pulp 3)

Please register to edit this issue

Also available in: Atom PDF