As a (plug-in) developer I have the option to install Pulp without a message broker
- an optional deployment style without requiring multiple
systemdservices running (no worker processes)
- pulling less dependencies for restricted environments e.g running Pulp on one's laptop from a
venvinstead of a VM
- simpler proofs of concept with a single service to e.g put in a container, esp. now that we support SQLite3
- functional testing of core--plug-in API with the ability to mock&assert selected codepaths, all being able to run from
within a test case
- functional testing of Pulp worker API in a similar fashion
- possibly avoid remote debugging of workers
The Kombu library has an in-memory transport that I hope we could utilize for this purpose.
This of course would still require some investigation work and probably couple of tweaks here and there to make it work properly.
- pulp development installation utilities
- pulp core in case issues are spotted once the pulp worker processes turn into threads
- the story subject in mind is a developer performing daily duties: creating proofs of concept, debugging issues, testing ideas, implementing new (plug-in) features, running (unit/functional) tests
#1 Updated by milan about 2 years ago
- Subject changed from As a (plug-in) developer I have the option to install Pulp with SQLite3 as the default DB to As a (plug-in) developer I have the option to install Pulp without the message broker
- Tags Dev Environment added
Based on the Plugin Writer's Coding Workshop Feedback, the desire for simpler things, folks would like to have the option to deploy Pulp without a message broker for the (plug-in) development purposes.
#3 Updated by bmbouter about 2 years ago
The in-memory option is maybe ok, but I'm concerned that it would be so significantly different from how Pulp is supposed to run that it could create more issues than benefits. For example webservers timeout and will randomly kill celery tasks that are executing in-memory during the httpd call. Also when asynchronous work becomes synchronous it's possible code runs in different orders. For example the actual celery tasking code would be completed before the _queue_reserved_resource task actually completes. This can happen during normal execution due to race conditions, but it's rare. These aren't blockers, just concerns that it's meaningfully different.
I recognize (and share) the desire to not install the message bus. It's a barrier to using Pulp. That being said I think since we are 100% build on top of Celery we can't avoid doing it that way. Feel free to continue to argue the case here in the comments. If not, then we should close as WONTFIX.
#4 Updated by milan about 2 years ago
indeed the system behavior might, in the corner cases, be different from the target production environment. As you however already mentioned, the main use of this story is to allow a plug-in developer/tester, that doesn't care right away about those corner cases, getting started small.
Another use of this story might be to make it easier to implement a test plug-in, asserting the Pulp plug-in API specification in a semi-functional environment; a worker would have to be a thread within the (test) app.
I guess more investigation is necessary here (in both the use cases) when it comes to the "walking the path" part and some stuff might get hairy indeed but my feeling is enabling this won't be a waste of resources esp. when it comes to discovering possible race conditions sooner than in production.
#5 Updated by bmbouter about 2 years ago
The in-memory db won't work correctly due to concurrency problems that in-memory introduces that aren't present in the normal usage due to the tasking system. Specifically a sync and a publish received by two different webserver processes will execute concurrently when celery configured to use in-memory. Having them run together is unsafe from a correctness perspective.
The normal Celery execution handles this correctly because the tasks are routed into a queue which hands them to be run by a worker serially.
So I think the in-memory option is unfortunately not viable. It is unfortunate because users would like to not install a broker and I also can appreciate that.
Is in-memory viable and I'm not thinking about this right?
Is there another option?
#6 Updated by milan about 2 years ago
The Kombu in-memory transport seems it does use a queue so the correctness might not be an issue but of course, if multiple worker processes are concerned, in-memory transport wouldn't be possible; we'd have to use worker threads instead what might not be worth the effort of refactoring. Hard to tell without further investigation.
#7 Updated by bmbouter about 2 years ago
I was assuming we would still have it be multi-process, and work would still be run by the Celery workers. Running a multi-process application as a single process is doable, but it could be risky. Relative to the benefit I don't think it's worth the risk. More input on the tradeoff could be good.
#11 Updated by daviddavis almost 2 years ago
Can I ask who the target audience is? Is it core developers, plugin developers, or pulp end users? Or maybe some combination of all three? I see a lot of value—it's always nice to simplify environments and give users options—but I also some risks as well. I'm a bit torn.
By the way, thanks for leading this effort @milan.
#12 Updated by milan almost 2 years ago
- Description updated (diff)
The audience got blurred with the motivation, I guess.
I't would be developers creating proofs of concept, debugging, testing new ideas, implementing plugins, or running semi-functional tests on PRs.
Not suitable for production, I'd say.
Though, someone might like crazy ideas as run Pulp over NFS behind HAProxy or what not ;)
I added the target user in the scope.
#14 Updated by dalley almost 2 years ago
Now that we're using Redis + RQ instead of Rabbit / Qpid + Celery + Kombu, we can re-evaluate this issue. But the only thing like this I've seen is "FakeRedis" which is meant to be used exclusively for unit tests, and probably shouldn't be used even for a developer installation even if it might work.
My two cents, I don't think it's a good idea to promote environments that are easy but potentially wrong, because no matter how many warnings we put on it, someone will try to deploy it that way in production.
#15 Updated by bmbouter almost 2 years ago
- Status changed from NEW to CLOSED - NOTABUG
Pulp3 no longer uses message brokers so I think we can close this as NOTABUG. I could see a similar desire with Redis to not be required for a dev environment, but I'm not sure how that is possible. RQ has a 'Bypassing Workers' section in its docs but even though a task will be run synchronously it says it still needs a connection to Redis.
If you learn about how to have RQ run in-memory and not via Redis let's try to get that documented for users.
Please register to edit this issue