Pulp

On a multi-machine deployment, this could be difficult to guarantee, but pulp-manage-db could make a best effort.

Pulp is a clustered software. We have to assume that there are multiple machines with pulp workers running on them. Can you elaborate on what a best-effort solution would look like?

A best effort could, for example, check the local process list for anything (besides itself) containing the word "pulp", and if found, ask the user if they're sure they want to proceed. Slightly fancier would be to check the database for any known workers or any tasks listed in a running state.

Of course that would not be a guarantee or catch all cases, but even the simple process check would have caught all of the recent cases we've been seeing where people ran pulp-manage-db on a single-system deployment while pulp was running. Those cases lead to database corruption.

I had a discussion with Brian regarding this, and here are the pros and cons of each opinion:

1.) Check DB listing

• Would work in a clustered environment
• status.get_workers() will have celery_beat worker left over even when all processes have stopped
  • We can try to check the worker timestamp but can't tell for sure until 390s have elapsed
  • there are some failure scenarios where processes could be running but the timestamps have stopped being recorded
  • if we proceed with this changes need to made to scheduler.py for a sigterm handler to mitigate the above concerns

2.) Check local process listing

• Can tell if there are any workers left real time
• does not support clustered upgrade

Personally I believe that 1 is the correct way to go, and will make the changes in scheduler.py to support this.

I see a lot of usable value in option 1 also. Thanks bizhang!

To correct a point I made from IRC, I think you can use a failover time of 300 seconds. The 390 number is the maximum delay until recover from failover, but if all records are over 300 seconds old then we can continue with the migrations.

Also if the database is down we should check that pulp-manage-db handles it gracefully. It wouldn't be good to have a traceback occur because this database check can't connect. I'm not sure what it does today or where that code lives.

https://github.com/pulp/pulp/pull/2883 is the PR that reverted this feature. I've merged the revert forward to master, so evidence of this feature no longer exists from 2.11-dev forward; you ca use the referenced PR to find easy commit hashes for restoring it when we pick this work back up.