Project

Profile

Help

Task #2895

Task #2868: Platform support for publishing.

As a user, I can have content served to me by Pulp

Added by jortel@redhat.com over 2 years ago. Updated 6 months ago.

Status:
MODIFIED
Priority:
High
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

0%

Platform Release:
Blocks Release:
Backwards Incompatible:
No
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:
Sprint 25

Description

Add content handling urls to serve conent. Some can be cannibalized from pulp2. Mainly parts of the ContentView. [0]

The handling URLs should be able to:
  • Based on incoming request URL, determine the distribution
  • Match the scheme against (http and https fields). 404 when scheme not enabled.
  • Query the PublishedArtifact and PublishedMetadata tables (in that order) by URL path component to get the artifact or the metadata.
  • respond with contents of the corresponding file

For this task the request and response should be handled with the normal WSGI application. There is a task to do enable this with apache later [1].

The content urls can be broken out separately later with magic in the settings.py and/or separate .wsgi files.

[0] https://github.com/pulp/pulp/tree/91a1e28c9e7d3dee418d5c7680dbf25c3e7adc63/server/pulp/server/content/web/views.py

[1] https://pulp.plan.io/issues/2914

History

#1 Updated by jortel@redhat.com over 2 years ago

  • Tags Pulp 3, Pulp 3 Plugin Writer Alpha added

What is involved with getting the test web server to route content URLs to the content app?

#2 Updated by mhrivnak over 2 years ago

wrote:

What is involved with getting the test web server to route content URLs to the content app?

Assuming it's a separate django app, which I think it appropriate, we would just need to add it to INSTALLED_APPS in settings.py. It would be responsible for having its own urls.py and setting up its own routing there.

#3 Updated by bmbouter over 2 years ago

Isn't this more than a django app installed to the pulp API's WSGI app, but it actually is it's own, distinct WSGI app? The content serving in pulp2 was its own WSGI app to not mix the content requests with the pulp API requests.

If that's the case it would get it's own wsgi entry point, and have its own urls.py and very few handlers. It would be called with runserver by specifying that settings.py application

#4 Updated by mhrivnak over 2 years ago

I think it would be a good idea to add more detail to this task about exactly what functionality the app needs to have. I know the whole picture is spelled out in the other issue, but it's good to think through and write down what exactly this individual piece needs to do. Quick first thoughts:

  • Based on incoming request URL, determine what publication is being accessed
  • Using the publication, further determine which artifact or metadata file is being accessed
  • respond with contents of the corresponding file
  • or send through the streamer workflow (is this a refactor, rewrite, or mostly-stay-the-same situation?)
  • somehow respond to requests for "listing" files (definitely not required for plugin API alpha)

#5 Updated by mhrivnak over 2 years ago

bmbouter wrote:

Isn't this more than a django app installed to the pulp API's WSGI app, but it actually is it's own, distinct WSGI app? The content serving in pulp2 was its own WSGI app to not mix the content requests with the pulp API requests.

If that's the case it would get it's own wsgi entry point, and have its own urls.py and very few handlers. It would be called with runserver by specifying that settings.py application

Yes that's all correct. But the question was about using the test (dev) server, and for that I'm not sure how much it matters. We could require the developer to run two different dev servers: one for the rest api, and one for content. Or we could try to do it all in one process. I'm sure there are pros and cons of each, but neither seems completely objectionable just yet.

#6 Updated by bmbouter over 2 years ago

@mhrivnak that list looks great.

Also for the "respond with the contents of the file" that should actually stream the file using Django which allows for a streaming response type. I think we should file an add-on ticket that introduces a setting which would allow the view to return a mod xsendheader instead of streaming the actual bytes from the Django request handler. If enabled it responds with that xSendFile header set and an empty body so that Apache can stream it efficiently. I can file that if you want, just let me know.

We could do it in the same processes that is true. It certainly keeps it simple. The downside is that content data and api requests are mixed, but there is a nice upside of reduced memory usage and fewer qpid+database connections. The more I think about it the more I like it since apache will be serving production instances anyway.

#7 Updated by jortel@redhat.com over 2 years ago

bmbouter wrote:

@mhrivnak that list looks great.

Also for the "respond with the contents of the file" that should actually stream the file using Django which allows for a streaming response type. I think we should file an add-on ticket that introduces a setting which would allow the view to return a mod xsendheader instead of streaming the actual bytes from the Django request handler. If enabled it responds with that xSendFile header set and an empty body so that Apache can stream it efficiently. I can file that if you want, just let me know.

We could do it in the same processes that is true. It certainly keeps it simple. The downside is that content data and api requests are mixed, but there is a nice upside of reduced memory usage and fewer qpid+database connections. The more I think about it the more I like it since apache will be serving production instances anyway.

In pulp2 the content app is a separate WSGI app1. One advantage to keep it a separate app is that it can be deployed on a separate server and configured with different resource allocations in Apache.

Is the new xsend setting something as simple as just adding:

XSEND = (true/false)

to settings.py.

[1] https://github.com/pulp/pulp/tree/master/server/pulp/server/content/web

#8 Updated by jortel@redhat.com over 2 years ago

mhrivnak wrote:

I think it would be a good idea to add more detail to this task about exactly what functionality the app needs to have. I know the whole picture is spelled out in the other issue, but it's good to think through and write down what exactly this individual piece needs to do. Quick first thoughts:

  • Based on incoming request URL, determine what publication is being accessed
  • Using the publication, further determine which artifact or metadata file is being accessed
  • respond with contents of the corresponding file
  • or send through the streamer workflow (is this a refactor, rewrite, or mostly-stay-the-same situation?)

There isn't much logic in the pulp2 ContentView but most of it can be generally reused.

  • somehow respond to requests for "listing" files (definitely not required for plugin API alpha)

This list looks fine.

#9 Updated by jortel@redhat.com over 2 years ago

After talking through this, it seems like the content app can be just a View in the platform/app. We can deploy separately with magic in the settings.py and/or separate .wsgi files.

#10 Updated by bmbouter over 2 years ago

Yes I think that is fine for today and may be fine forever even. Production installs shouldn't use Django to stream bits so they shouldn't experience issues from content requests streaming large data back which would tie up the WSGI processes and could queue/block API requests.

#11 Updated by amacdona@redhat.com over 2 years ago

  • Subject changed from Port content WSGI applicaiton from pulp2 to pulp3. to Port content WSGI application from pulp2 to pulp3.

#12 Updated by bizhang over 2 years ago

@bizhang will groom

#13 Updated by bmbouter over 2 years ago

Note this is to be split into two tasks: https://pulp.plan.io/issues/2868#note-22

#14 Updated by jortel@redhat.com over 2 years ago

  • Description updated (diff)

#15 Updated by jortel@redhat.com over 2 years ago

  • Subject changed from Port content WSGI application from pulp2 to pulp3. to Add Content WSGI application to server content.
  • Description updated (diff)

#16 Updated by bizhang over 2 years ago

  • Description updated (diff)

#17 Updated by bizhang over 2 years ago

  • Groomed changed from No to Yes
  • Sprint Candidate changed from No to Yes

#18 Updated by bmbouter over 2 years ago

  • Subject changed from Add Content WSGI application to server content. to As a user, I can have content served to me by Pulp
  • Description updated (diff)

I retitled and edited it to reflect the comments where we decided to use the existing WSGI application instead of adding a new one.

#19 Updated by mhrivnak about 2 years ago

  • Priority changed from Normal to High

#20 Updated by mhrivnak about 2 years ago

  • Sprint/Milestone set to 42

#21 Updated by jortel@redhat.com about 2 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to jortel@redhat.com

#22 Updated by jortel@redhat.com about 2 years ago

  • Status changed from ASSIGNED to POST

#23 Updated by mhrivnak about 2 years ago

  • Sprint/Milestone changed from 42 to 43

#24 Updated by jortel@redhat.com about 2 years ago

  • Sprint/Milestone changed from 43 to 44

#25 Updated by jortel@redhat.com about 2 years ago

  • Status changed from POST to MODIFIED

#26 Updated by bmbouter almost 2 years ago

  • Tags deleted (Pulp 3 Plugin Writer Alpha)

Cleaning up Redmine tags

#27 Updated by pthomas@redhat.com over 1 year ago

  • Smash Test set to 872

#28 Updated by bmbouter over 1 year ago

  • Sprint set to Sprint 25

#29 Updated by bmbouter over 1 year ago

  • Sprint/Milestone deleted (44)

#30 Updated by daviddavis 6 months ago

  • Sprint/Milestone set to 3.0

#31 Updated by bmbouter 6 months ago

  • Tags deleted (Pulp 3)

Please register to edit this issue

Also available in: Atom PDF