Project

Profile

Help

Issue #3861

closed

A remote Redis server is being ignored when configured at server.yaml

Added by tchellomello over 5 years ago. Updated over 4 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Quarter:

Description

A remote Redis server is being ignored when configured at server.yaml.

Regardless of the configuration placed for Redis, it will always point back to localhost.

Note: Remote database is working as expected

#server.yaml
bash-4.4# cat server.yaml  | grep -v ^$  | grep -v ^#
REDIS:
  HOST: redis
  PORT: 6379
  PASSWORD: 

DATABASES:
  default:
    CONN_MAX_AGE: 0
    ENGINE: django.db.backends.postgresql_psycopg2
    NAME: pulp
    USER: pulp
    PASSWORD: pulp
    HOST: pgsql
    PORT: 5432

SECRET_KEY: lrr7on0ossjj993vq8o6bbpnk5x+5*zl6n(+$8a0#982_$v^gx
DEBUG: True

connectivity works

bash-4.4# ping redis
PING redis (172.18.0.3): 56 data bytes
64 bytes from 172.18.0.3: seq=0 ttl=64 time=0.100 ms

Troubleshooting with the rpdb we can see the configured is being merged by https://github.com/pulp/pulp/blob/master/pulpcore/pulpcore/app/settings.py#L259-L272


bash-4.4# rq worker -n 'resource_manager@%h' -w 'pulpcore.tasking.worker.PulpWorker'
pdb is running on 127.0.0.1:4444
bash-4.4# telnet localhost 4444
> /nightly/pulp/pulpcore/pulpcore/app/settings.py(273)load_settings()
-> return settings
(Pdb) l
268      
269          for setting_name, setting_value in settings.items():
270              setattr(sys.modules[__name__], setting_name.upper(), setting_value)
271      
272          import rpdb; rpdb.set_trace()
273  ->        return settings
274      
275      
276      # Read PULP_SETTINGS environment variable to find the location of server.yaml,
277      # defaults to /etc/pulp/server.yaml
278      PULP_SETTINGS = os.getenv('PULP_SETTINGS', '/etc/pulp/server.yaml')
(Pdb) settings
{'DATABASES': {'default': {'ENGINE': 'django.db.backends.postgresql_psycopg2', 'NAME': 'pulp', 'USER': 'pulp', 'CONN_MAX_AGE': 0, 'PASSWORD': 'pulp', 'HOST': 'pgsql', 'PORT': 5432}}, 'logging': {'version': 1, 'disable_existing_loggers': False, 'formatters': {'simple': {'format': 'pulp: %(name)s:%(levelname)s: %(message)s'}}, 'handlers': {'console': {'class': 'logging.StreamHandler', 'formatter': 'simple'}, 'syslog': {'address': '/dev/log', 'class': 'logging.handlers.SysLogHandler', 'formatter': 'simple'}}, 'loggers': {'': {'handlers': ['syslog'], 'level': 'INFO'}}}, 'SERVER': {'WORKING_DIRECTORY': '/var/lib/pulp/tmp'}, 'CONTENT': {'WEB_SERVER': 'django', 'HOST': None}, 'REDIS': {'HOST': 'redis', 'PORT': 6379, 'PASSWORD': None}, 'PROFILING': {'ENABLED': False, 'DIRECTORY': '/var/lib/pulp/c_profiles'}, 'SECRET_KEY': 'lrr7on0ossjj993vq8o6bbpnk5x+5*zl6n(+$8a0#982_$v^gx', 'DEBUG': True}
(Pdb) settings.get('REDIS')    
{'HOST': 'redis', 'PORT': 6379, 'PASSWORD': None}

However, the worker will try to connect locally:

bash-4.4# rq worker -n 'resource_manager@%h' -w 'pulpcore.tasking.worker.PulpWorker'
pdb is running on 127.0.0.1:4444
Error 99 connecting to localhost:6379. Address not available.
Actions #1

Updated by dkliban@redhat.com over 5 years ago

  • Project changed from 31 to Pulp
  • Tags Pulp 3 added
Actions #2

Updated by tchellomello over 5 years ago

Hello guys,

I took the troubleshooting a little further and it does it seems we are not passing the right connection string to the Redis.
I added some troubleshooting code to the worker.py within rq and it is receiving localhost by default.

1) Starting a worker

$ rq worker -n 'resource_manager@%h' -w 'pulpcore.tasking.worker.PulpWorker
pdb is running on 127.0.0.1:4444

2) Checking with rpdb

bash-4.4# telnet localhost 4444
> /usr/local/lib/python3.6/site-packages/rq/worker.py(108)all()
-> if queue:
(Pdb) l
103          @classmethod
104          def all(cls, connection=None, job_class=None, queue_class=None, queue=None):
105              """Returns an iterable of all Workers.
106              """
107              import rpdb; rpdb.set_trace()
108  ->            if queue:
109                  connection = queue.connection
110              elif connection is None:
111                  connection = get_current_connection()
112      
113              worker_keys = get_keys(queue=queue, connection=connection)

(Pdb) connection
StrictRedis<ConnectionPool<Connection<host=localhost,port=6379,db=0>>>

If we look at the rq code, it will point to the localhost:

    def __init__(self, host='localhost', port=6379,
                 db=0, password=None, socket_timeout=None,
                 socket_connect_timeout=None,
                 socket_keepalive=None, socket_keepalive_options=None,
                 connection_pool=None, unix_socket_path=None,
                 encoding='utf-8', encoding_errors='strict',
                 charset=None, errors=None,
                 decode_responses=False, retry_on_timeout=False,
                 ssl=False, ssl_keyfile=None, ssl_certfile=None,
                 ssl_cert_reqs=None, ssl_ca_certs=None,
                 max_connections=None):

Now to test, if we change the redis port on the server.yaml if still does not change the redis string:

bash-4.4# cat pulp/pulpcore/pulpcore/etc/pulp/server.yaml | grep REDIS -A 3 
REDIS:
  HOST: redis
  PORT: 6378
  PASSWORD: 
bash-4.4# telnet localhost 4444
> /usr/local/lib/python3.6/site-packages/rq/worker.py(108)all()
-> if queue:
(Pdb) l
103          @classmethod
104          def all(cls, connection=None, job_class=None, queue_class=None, queue=None):
105              """Returns an iterable of all Workers.
106              """
107              import rpdb; rpdb.set_trace()
108  ->            if queue:
109                  connection = queue.connection
110              elif connection is None:
111                  connection = get_current_connection()
112      
113              worker_keys = get_keys(queue=queue, connection=connection)
(Pdb) connection
StrictRedis<ConnectionPool<Connection<host=localhost,port=6379,db=0>>>

So it seems we are not passing any options to the rq when calling it.

Still looking

Actions #3

Updated by tchellomello over 5 years ago

Alright, so if we pass the --url to the RQ worker then it works.

rq worker --url "redis://redis:6379/0"  -n 'resource_manager@%h' -w 'pulpcore.tasking.worker.PulpWorker'

Checking the postgresql database, it then got its status registered:

bash-4.4# psql  -h pgsql -U pulp
Password for user pulp: 
psql (10.4)
Type "help" for help.

pulp=# select * from pulp_app_worker ;
                  id                  |            created            |         last_updated          |             name              |        last_heartbeat         | gracefully_stopped | cleaned_up 
--------------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+--------------------+------------
 711f762b-24c1-4518-9557-5d2debd9840a | 2018-07-17 21:44:04.153992+00 | 2018-07-17 21:55:36.149917+00 | resource_manager@3891802b8d5c | 2018-07-17 21:56:06.255469+00 | f                  | f
(1 row)

So it seems we have to pass that directly to the worker manually or have it globally somewhere in Pulp.

Actions #4

Updated by tchellomello over 5 years ago

PR submitted https://github.com/pulp/pulp/pull/3555

Tested and worked fine.

Added by tchellomello over 5 years ago

Revision 259bc4ca | View on GitHub

Make sure the REDIS connection string present in server.yaml is inherited by workers and resource_manager.

fixes #3861 https://pulp.plan.io/issues/3861

Added by tchellomello over 5 years ago

Revision 259bc4ca | View on GitHub

Make sure the REDIS connection string present in server.yaml is inherited by workers and resource_manager.

fixes #3861 https://pulp.plan.io/issues/3861

Actions #5

Updated by dkliban@redhat.com over 5 years ago

  • Status changed from NEW to POST
  • Assignee set to tchellomello
Actions #6

Updated by tchellomello over 5 years ago

  • Status changed from POST to MODIFIED
Actions #7

Updated by daviddavis almost 5 years ago

  • Sprint/Milestone set to 3.0.0
Actions #8

Updated by bmbouter almost 5 years ago

  • Tags deleted (Pulp 3)
Actions #9

Updated by bmbouter over 4 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF