Pulp

Hello guys,

I took the troubleshooting a little further and it does it seems we are not passing the right connection string to the Redis.
I added some troubleshooting code to the worker.py within rq and it is receiving localhost by default.

1) Starting a worker

$ rq worker -n 'resource_manager@%h' -w 'pulpcore.tasking.worker.PulpWorker
pdb is running on 127.0.0.1:4444

2) Checking with rpdb

bash-4.4# telnet localhost 4444
> /usr/local/lib/python3.6/site-packages/rq/worker.py(108)all()
-> if queue:
(Pdb) l
103          @classmethod
104          def all(cls, connection=None, job_class=None, queue_class=None, queue=None):
105              """Returns an iterable of all Workers.
106              """
107              import rpdb; rpdb.set_trace()
108  ->            if queue:
109                  connection = queue.connection
110              elif connection is None:
111                  connection = get_current_connection()
112      
113              worker_keys = get_keys(queue=queue, connection=connection)

(Pdb) connection
StrictRedis<ConnectionPool<Connection<host=localhost,port=6379,db=0>>>

If we look at the rq code, it will point to the localhost:

    def __init__(self, host='localhost', port=6379,
                 db=0, password=None, socket_timeout=None,
                 socket_connect_timeout=None,
                 socket_keepalive=None, socket_keepalive_options=None,
                 connection_pool=None, unix_socket_path=None,
                 encoding='utf-8', encoding_errors='strict',
                 charset=None, errors=None,
                 decode_responses=False, retry_on_timeout=False,
                 ssl=False, ssl_keyfile=None, ssl_certfile=None,
                 ssl_cert_reqs=None, ssl_ca_certs=None,
                 max_connections=None):

Now to test, if we change the redis port on the server.yaml if still does not change the redis string:

bash-4.4# cat pulp/pulpcore/pulpcore/etc/pulp/server.yaml | grep REDIS -A 3 
REDIS:
  HOST: redis
  PORT: 6378
  PASSWORD: 

bash-4.4# telnet localhost 4444
> /usr/local/lib/python3.6/site-packages/rq/worker.py(108)all()
-> if queue:
(Pdb) l
103          @classmethod
104          def all(cls, connection=None, job_class=None, queue_class=None, queue=None):
105              """Returns an iterable of all Workers.
106              """
107              import rpdb; rpdb.set_trace()
108  ->            if queue:
109                  connection = queue.connection
110              elif connection is None:
111                  connection = get_current_connection()
112      
113              worker_keys = get_keys(queue=queue, connection=connection)
(Pdb) connection
StrictRedis<ConnectionPool<Connection<host=localhost,port=6379,db=0>>>

So it seems we are not passing any options to the rq when calling it.

Still looking

Alright, so if we pass the --url to the RQ worker then it works.

rq worker --url "redis://redis:6379/0"  -n 'resource_manager@%h' -w 'pulpcore.tasking.worker.PulpWorker'

Checking the postgresql database, it then got its status registered:

bash-4.4# psql  -h pgsql -U pulp
Password for user pulp: 
psql (10.4)
Type "help" for help.

pulp=# select * from pulp_app_worker ;
                  id                  |            created            |         last_updated          |             name              |        last_heartbeat         | gracefully_stopped | cleaned_up 
--------------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+--------------------+------------
 711f762b-24c1-4518-9557-5d2debd9840a | 2018-07-17 21:44:04.153992+00 | 2018-07-17 21:55:36.149917+00 | resource_manager@3891802b8d5c | 2018-07-17 21:56:06.255469+00 | f                  | f
(1 row)

So it seems we have to pass that directly to the worker manually or have it globally somewhere in Pulp.

PR submitted https://github.com/pulp/pulp/pull/3555

Tested and worked fine.