Pulp

jortel@redhat.com wrote:

What is involved with getting the test web server to route content URLs to the content app?

Assuming it's a separate django app, which I think it appropriate, we would just need to add it to INSTALLED_APPS in settings.py. It would be responsible for having its own urls.py and setting up its own routing there.

Isn't this more than a django app installed to the pulp API's WSGI app, but it actually is it's own, distinct WSGI app? The content serving in pulp2 was its own WSGI app to not mix the content requests with the pulp API requests.

If that's the case it would get it's own wsgi entry point, and have its own urls.py and very few handlers. It would be called with runserver by specifying that settings.py application

I think it would be a good idea to add more detail to this task about exactly what functionality the app needs to have. I know the whole picture is spelled out in the other issue, but it's good to think through and write down what exactly this individual piece needs to do. Quick first thoughts:

• Based on incoming request URL, determine what publication is being accessed
• Using the publication, further determine which artifact or metadata file is being accessed
• respond with contents of the corresponding file
• or send through the streamer workflow (is this a refactor, rewrite, or mostly-stay-the-same situation?)
• somehow respond to requests for "listing" files (definitely not required for plugin API alpha)

bmbouter wrote:

Isn't this more than a django app installed to the pulp API's WSGI app, but it actually is it's own, distinct WSGI app? The content serving in pulp2 was its own WSGI app to not mix the content requests with the pulp API requests.

If that's the case it would get it's own wsgi entry point, and have its own urls.py and very few handlers. It would be called with runserver by specifying that settings.py application

Yes that's all correct. But the question was about using the test (dev) server, and for that I'm not sure how much it matters. We could require the developer to run two different dev servers: one for the rest api, and one for content. Or we could try to do it all in one process. I'm sure there are pros and cons of each, but neither seems completely objectionable just yet.

mhrivnak that list looks great.

Also for the "respond with the contents of the file" that should actually stream the file using Django which allows for a streaming response type. I think we should file an add-on ticket that introduces a setting which would allow the view to return a mod xsendheader instead of streaming the actual bytes from the Django request handler. If enabled it responds with that xSendFile header set and an empty body so that Apache can stream it efficiently. I can file that if you want, just let me know.

We could do it in the same processes that is true. It certainly keeps it simple. The downside is that content data and api requests are mixed, but there is a nice upside of reduced memory usage and fewer qpid+database connections. The more I think about it the more I like it since apache will be serving production instances anyway.

bmbouter wrote:

mhrivnak that list looks great.

Also for the "respond with the contents of the file" that should actually stream the file using Django which allows for a streaming response type. I think we should file an add-on ticket that introduces a setting which would allow the view to return a mod xsendheader instead of streaming the actual bytes from the Django request handler. If enabled it responds with that xSendFile header set and an empty body so that Apache can stream it efficiently. I can file that if you want, just let me know.

We could do it in the same processes that is true. It certainly keeps it simple. The downside is that content data and api requests are mixed, but there is a nice upside of reduced memory usage and fewer qpid+database connections. The more I think about it the more I like it since apache will be serving production instances anyway.

In pulp2 the content app is a separate WSGI app[1]. One advantage to keep it a separate app is that it can be deployed on a separate server and configured with different resource allocations in Apache.

Is the new xsend setting something as simple as just adding:

XSEND = (true/false)

to settings.py.

[1] https://github.com/pulp/pulp/tree/master/server/pulp/server/content/web

mhrivnak wrote:

I think it would be a good idea to add more detail to this task about exactly what functionality the app needs to have. I know the whole picture is spelled out in the other issue, but it's good to think through and write down what exactly this individual piece needs to do. Quick first thoughts:

• Based on incoming request URL, determine what publication is being accessed
• Using the publication, further determine which artifact or metadata file is being accessed
• respond with contents of the corresponding file
• or send through the streamer workflow (is this a refactor, rewrite, or mostly-stay-the-same situation?)

There isn't much logic in the pulp2 ContentView but most of it can be generally reused.

• somehow respond to requests for "listing" files (definitely not required for plugin API alpha)

This list looks fine.

After talking through this, it seems like the content app can be just a View in the platform/app. We can deploy separately with magic in the settings.py and/or separate .wsgi files.

Yes I think that is fine for today and may be fine forever even. Production installs shouldn't use Django to stream bits so they shouldn't experience issues from content requests streaming large data back which would tie up the WSGI processes and could queue/block API requests.

bizhang will groom

Note this is to be split into two tasks: https://pulp.plan.io/issues/2868#note-22