Project

Profile

Help

Issue #4732

pulp workers only use localhost for redis

Added by Anonymous over 1 year ago. Updated 5 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
Installer
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Quarter:

Description

I'm trying to use pulp with a postgres db in rds on aws and an aws redis backend.

echo REDIS_HOST=fqdn >> /etc/pulp/settings.py

rq worker -n 'reserved-resource-worker-1@%h' -w 'pulpcore.tasking.worker.PulpWorker' 2>&1 | less

2019-04-24:13:11:10,491 ERROR [redis_loader.py:58 - load] Error 111 connecting to localhost:6379. Connection refused.

Further.. this could be an error on my part... but the env vars for REDIS seem fine. The /etc/pulp/settings.py file seems to be ignored.


(.venv) ubuntu@buildbox:~$ echo $DJANGO_SETTINGS_MODULE
pulpcore.app.settings
(.venv) ubuntu@buildbox:~$ ipython
Python 3.6.7 (default, Oct 22 2018, 11:32:17) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.4.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from django.conf import settings

In [2]: print(settings.REDIS_HOST)
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-2-2807dc9204c5> in <module>
----> 1 print(settings.REDIS_HOST)

~/.venv/lib/python3.6/site-packages/django/conf/__init__.py in __getattr__(self, name)
     78         if self._wrapped is empty:
     79             self._setup(name)
---> 80         val = getattr(self._wrapped, name)
     81         self.__dict__[name] = val
     82         return val

AttributeError: 'Settings' object has no attribute 'REDIS_HOST'

Associated revisions

Revision b9ade2a2 View on GitHub
Added by bmbouter over 1 year ago

Start workers with the -c option

Without the -c option the systemd files won't read the RQ settings from Pulp. This causes defualts like localhost to be the only options you can received.

https://pulp.plan.io/issues/4732 re #4732

Revision b9ade2a2 View on GitHub
Added by Brian Bouterse over 1 year ago

Start workers with the -c option

Without the -c option the systemd files won't read the RQ settings from Pulp. This causes defualts like localhost to be the only options you can received.

https://pulp.plan.io/issues/4732 re #4732

Revision a37f9abb View on GitHub
Added by bmbouter over 1 year ago

Adds -c option for RQ workers in docs

The docs did not tell users much about how to run an RQ worker. This tries to clarify that some.

https://pulp.plan.io/issues/4732 closes #4732

Revision d391f206 View on GitHub
Added by bmbouter over 1 year ago

Including fix for systemd template options

An important bugfix landed in ansible-pulp that adds the -c option to RQ workers for systemd profiles. This PR will include that fix in pulplift too.

https://pulp.plan.io/issues/4732 re #4732

History

#1 Updated by bmbouter over 1 year ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to bmbouter
  • Tags Pulp 3 installer added

The issue is that the pulp_resource_manager and pulp_workers are being started without the -c 'pulpcore.rqconfig' option. The docs link to the installer templates for these things so fixing the installer template should resolve the issue.

The docs used to contain this

#2 Updated by Anonymous over 1 year ago

My commentary about the redis settings not working: if I use master instead of rc1 the env vars work. As do the settings.py vars.

REDIS vars seem working in
commit 1ec10543ee2dcdb4c317a73c94005412f9d9dd04 (HEAD -> master)

Installed via:
git clone https://github.com/pulp/pulpcore.git
pip install -e ./pulpcore[postgres]
git clone https://github.com/pulp/pulpcore-plugin.git
pip install -e ./pulpcore-plugin

I blame: https://docs.pulpproject.org/en/3.0/nightly/installation/instructions.html

#3 Updated by Anonymous over 1 year ago

(.venv) ubuntu@buildbox:~$ rq worker -n -c pulpcore.rqconfig 'resource-manager@%h' -w 'pulpcore.tasking.worker.PulpWorker'
2019-04-24:13:26:24,811 ERROR [redis_loader.py:58 - load] Error 111 connecting to localhost:6379. Connection refused.
Error 111 connecting to localhost:6379. Connection refused.
(.venv) ubuntu@buildbox:~$ grep REDIS /etc/pulp/settings.py
REDIS_ENABLED_FOR_DYNACONF=True
REDIS_HOST='jps-pulp-test.k5xwqi.ng.0001.use1.cache.amazonaws.com'
REDIS_PORT= 6379

#4 Updated by Anonymous over 1 year ago

More context. I was stepping thru pulpcore/pulpcore/rqconfig.py and notice it was coughing an error about redis:

This is with DJANGO_SETTINGS_MODULE set.

In [1]: import sys

In [2]: from pulpcore.app.settings import settings
2019-04-24:14:05:30,274 ERROR    [redis_loader.py:58 - load] Error 111 connecting to localhost:6379. Connection refused.

#5 Updated by bmbouter over 1 year ago

  • Status changed from ASSIGNED to POST

#6 Updated by Anonymous over 1 year ago

Chasing the bug deeper, I ended up in redis_loader.py line 58 in dynaconf v2.0.1

The line in question is in an except Exception block that just swallows and logs everything. Which explains the ERROR observed above.

Further debugging using the powerful print() function showed I was hitting the else block of the try/catch.

Removing the try/catch block


def load(obj, env=None, silent=True, key=None):
    """Reads and loads in to "settings" a single key or all keys from redis

    :param obj: the settings instance
    :param env: settings env default='DYNACONF'
    :param silent: if errors should raise
    :param key: if defined load a single key, else load all in env
    :return: None
    """
    redis = StrictRedis(**obj.get("REDIS_FOR_DYNACONF"))
    holder = obj.get("ENVVAR_PREFIX_FOR_DYNACONF")
    try:
        if key:
            value = redis.hget(holder.upper(), key)
            if value:
                obj.logger.debug(
                    "redis_loader: loading by key: %s:%s (%s:%s)",
                    key,
                    value,
                    IDENTIFIER,
                    holder,
                )
            if value:
                parsed_value = parse_conf_data(value, tomlfy=True)
                if parsed_value:
                    obj.set(key, parsed_value)
        else:
            data = {
                key: parse_conf_data(value, tomlfy=True)
                for key, value in redis.hgetall(holder.upper()).items()
            }
            if data:
                obj.logger.debug(
                    "redis_loader: loading: %s (%s:%s)",
                    data,
                    IDENTIFIER,
                    holder,
                )
                obj.update(data, loader_identifier=IDENTIFIER)
    except Exception as e:
        if silent:
            if hasattr(obj, "logger"):
                obj.logger.error(str(e))
            return False
        raise

I removed the try/catch and realized i was mis-reading a list comprehension and the root error is it's still trying to connect to localhost.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "x.py", line 1, in <module>
    from pulpcore.app.settings import settings
  File "/home/ubuntu/pulpcore/pulpcore/app/settings.py", line 218, in <module>
    ENVVAR_FOR_DYNACONF='PULP_SETTINGS',
  File "/home/ubuntu/.venv/lib/python3.6/site-packages/dynaconf/contrib/django_dynaconf_v2.py", line 67, in load
    lazy_settings.populate_obj(django_settings_module)
  File "/home/ubuntu/.venv/lib/python3.6/site-packages/dynaconf/base.py", line 90, in __getattr__
    self._setup()
  File "/home/ubuntu/.venv/lib/python3.6/site-packages/dynaconf/base.py", line 121, in _setup
    settings_module=settings_module, **self._kwargs
  File "/home/ubuntu/.venv/lib/python3.6/site-packages/dynaconf/base.py", line 184, in __init__
    self.execute_loaders()
  File "/home/ubuntu/.venv/lib/python3.6/site-packages/dynaconf/base.py", line 736, in execute_loaders
    loader.load(self, env, silent=silent, key=key)
  File "/home/ubuntu/.venv/lib/python3.6/site-packages/dynaconf/loaders/redis_loader.py", line 44, in load
    for key, value in redis.hgetall(holder.upper()).items()
  File "/home/ubuntu/.venv/lib/python3.6/site-packages/redis/client.py", line 2649, in hgetall
    return self.execute_command('HGETALL', name)
  File "/home/ubuntu/.venv/lib/python3.6/site-packages/redis/client.py", line 774, in execute_command
    connection.send_command(*args)
  File "/home/ubuntu/.venv/lib/python3.6/site-packages/redis/connection.py", line 619, in send_command
    self.send_packed_command(self.pack_command(*args))
  File "/home/ubuntu/.venv/lib/python3.6/site-packages/redis/connection.py", line 594, in send_packed_command
    self.connect()
  File "/home/ubuntu/.venv/lib/python3.6/site-packages/redis/connection.py", line 498, in connect
    raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 111 connecting to localhost:6379. Connection refused.

#7 Updated by bmbouter over 1 year ago

update for pulplift is here: https://github.com/pulp/pulplift/pull/28

#8 Updated by bmbouter over 1 year ago

  • Status changed from POST to MODIFIED

#9 Updated by bmbouter 10 months ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

#10 Updated by bmbouter 5 months ago

  • Category set to Installer
  • Tags deleted (Pulp 3 installer)

Please register to edit this issue

Also available in: Atom PDF