Pulp

++ This bug was initially created as a clone of Bugzilla Bug #1104320 ++

Description of problem:

Created attachment 901871
Screen log of immediate repo sync call

Description of problem:
Currently, pulp allows requests even though the pulp_worker-*.services are not
fully operational.
A related scenario is a repo sync request right after a pulp_worker restart, where pulp_database.available queues entries get lost from mongo and stale reservation appears in the pulp_database.reserved_resources collection.

Version-Release number of selected component (if applicable):
- Fedora 20 (systemd)
- pulp-server-2.4.0-0.19.beta.fc20.noarch

How reproducible:
~100% (on my system)

Steps to Reproduce:
1. create a rpm repo (via pulp-admin)
pulp-admin -u admin -p admin rpm repo create --repo-id zoo --feed \
http://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/zoo/
2. systemctl restart pulp_worker-{0..8}.service ; \
pulp-admin -u admin -p admin rpm repo sync run --repo-id zoo

Actual results:
- a task queue lost even though the both the pulp_worker processes are up and running
- stale resource reservation item

[root@ec2-79-125-97-56 ~]# fg
mongo
> db.available_queues.find()
{ "_id" : "reserved_resource_worker-0@ec2-79-125-97-56.eu-west-1.compute.amazonaws.com", "last_heartbeat" : ISODate("2014-06-03T17:07:26.060Z"), "num_reservations" : 0 }
> db.reserved_resources.find()
{ "_id" : "repository:zoo", "assigned_queue" : null, "num_reservations" : 1 }
> db.task_status.find()
> db.available_queues.find()
{ "_id" : "reserved_resource_worker-0@ec2-79-125-97-56.eu-west-1.compute.amazonaws.com", "last_heartbeat" : ISODate("2014-06-03T17:08:08.281Z"), "num_reservations" : 0 }
>

Expected results:
503 Service Unavailable Response until the pulp_worker-*.service prepare all necessary resources
i.e. Pulp server (wsgi) doesn't allow requests while pulp_worker-*.service aren't fully operational

Additional info:

--- Additional comment from mkovacik@redhat.com at 06/03/2014 18:31:42 ---

Created attachment 901873
journalctl screen log of the scenario

--- Additional comment from mkovacik@redhat.com at 06/04/2014 12:02:04 ---

Created attachment 902159
reproducer bash script