Pulp

I did some preliminary research and it might be possible to modify our developer installation to optionally allow an in-memory broker

The in-memory option is maybe ok, but I'm concerned that it would be so significantly different from how Pulp is supposed to run that it could create more issues than benefits. For example webservers timeout and will randomly kill celery tasks that are executing in-memory during the httpd call. Also when asynchronous work becomes synchronous it's possible code runs in different orders. For example the actual celery tasking code would be completed before the _queue_reserved_resource task actually completes. This can happen during normal execution due to race conditions, but it's rare. These aren't blockers, just concerns that it's meaningfully different.

I recognize (and share) the desire to not install the message bus. It's a barrier to using Pulp. That being said I think since we are 100% build on top of Celery we can't avoid doing it that way. Feel free to continue to argue the case here in the comments. If not, then we should close as WONTFIX.

Brian,

indeed the system behavior might, in the corner cases, be different from the target production environment. As you however already mentioned, the main use of this story is to allow a plug-in developer/tester, that doesn't care right away about those corner cases, getting started small.

Another use of this story might be to make it easier to implement a test plug-in, asserting the Pulp plug-in API specification in a semi-functional environment; a worker would have to be a thread within the (test) app.

I guess more investigation is necessary here (in both the use cases) when it comes to the "walking the path" part and some stuff might get hairy indeed but my feeling is enabling this won't be a waste of resources esp. when it comes to discovering possible race conditions sooner than in production.

--
milan

The in-memory db won't work correctly due to concurrency problems that in-memory introduces that aren't present in the normal usage due to the tasking system. Specifically a sync and a publish received by two different webserver processes will execute concurrently when celery configured to use in-memory. Having them run together is unsafe from a correctness perspective.

The normal Celery execution handles this correctly because the tasks are routed into a queue which hands them to be run by a worker serially.

So I think the in-memory option is unfortunately not viable. It is unfortunate because users would like to not install a broker and I also can appreciate that.

Is in-memory viable and I'm not thinking about this right?
Is there another option?

The Kombu in-memory transport seems it does use a queue so the correctness might not be an issue but of course, if multiple worker processes are concerned, in-memory transport wouldn't be possible; we'd have to use worker threads instead what might not be worth the effort of refactoring. Hard to tell without further investigation.

I was assuming we would still have it be multi-process, and work would still be run by the Celery workers. Running a multi-process application as a single process is doable, but it could be risky. Relative to the benefit I don't think it's worth the risk. More input on the tradeoff could be good.

Can I ask who the target audience is? Is it core developers, plugin developers, or pulp end users? Or maybe some combination of all three? I see a lot of value—it's always nice to simplify environments and give users options—but I also some risks as well. I'm a bit torn.

By the way, thanks for leading this effort milan.

Now that we're using Redis + RQ instead of Rabbit / Qpid + Celery + Kombu, we can re-evaluate this issue. But the only thing like this I've seen is "FakeRedis" which is meant to be used exclusively for unit tests, and probably shouldn't be used even for a developer installation even if it might work.

https://github.com/jamesls/fakeredis

My two cents, I don't think it's a good idea to promote environments that are easy but potentially wrong, because no matter how many warnings we put on it, someone will try to deploy it that way in production.