https://pulp.plan.io/https://pulp.plan.io/favicon.ico2019-04-24T13:24:59ZPulpPulp - Issue #4732: pulp workers only use localhost for redishttps://pulp.plan.io/issues/4732?journal_id=411192019-04-24T13:24:59Zbmbouterbmbouter@redhat.com
<ul><li><strong>Status</strong> changed from <i>NEW</i> to <i>ASSIGNED</i></li><li><strong>Assignee</strong> set to <i>bmbouter</i></li><li><strong>Tags</strong> <i>Pulp 3 installer</i> added</li></ul><p>The issue is that the pulp_resource_manager and pulp_workers are being started without the <code>-c 'pulpcore.rqconfig'</code> option. The docs link to the installer templates for these things so fixing the installer template should resolve the issue.</p>
<p>The docs <a href="https://github.com/pulp/pulpcore/blob/c07d5983b53986cefc4e43148d9ba892a727e7b2/docs/installation/instructions.rst#systemd" class="external">used to contain this</a></p> Pulp - Issue #4732: pulp workers only use localhost for redishttps://pulp.plan.io/issues/4732?journal_id=411212019-04-24T13:25:25ZAnonymous
<ul></ul><p>My commentary about the redis settings not working: if I use master instead of rc1 the env vars work. As do the settings.py vars.</p>
<p>REDIS vars seem working in<br>
commit 1ec10543ee2dcdb4c317a73c94005412f9d9dd04 (HEAD -> master)</p>
<p>Installed via:<br>
git clone <a href="https://github.com/pulp/pulpcore.git" class="external">https://github.com/pulp/pulpcore.git</a><br>
pip install -e ./pulpcore[postgres]<br>
git clone <a href="https://github.com/pulp/pulpcore-plugin.git" class="external">https://github.com/pulp/pulpcore-plugin.git</a><br>
pip install -e ./pulpcore-plugin</p>
<p>I blame: <a href="https://docs.pulpproject.org/en/3.0/nightly/installation/instructions.html" class="external">https://docs.pulpproject.org/en/3.0/nightly/installation/instructions.html</a></p> Pulp - Issue #4732: pulp workers only use localhost for redishttps://pulp.plan.io/issues/4732?journal_id=411222019-04-24T13:27:03ZAnonymous
<ul></ul><p>(.venv) ubuntu@buildbox:~$ rq worker -n -c pulpcore.rqconfig 'resource-manager@%h' -w 'pulpcore.tasking.worker.PulpWorker'<br>
2019-04-24:13:26:24,811 ERROR [redis_loader.py:58 - load] Error 111 connecting to localhost:6379. Connection refused.<br>
Error 111 connecting to localhost:6379. Connection refused.<br>
(.venv) ubuntu@buildbox:~$ grep REDIS /etc/pulp/settings.py<br>
REDIS_ENABLED_FOR_DYNACONF=True<br>
REDIS_HOST='jps-pulp-test.k5xwqi.ng.0001.use1.cache.amazonaws.com'<br>
REDIS_PORT= 6379</p> Pulp - Issue #4732: pulp workers only use localhost for redishttps://pulp.plan.io/issues/4732?journal_id=411252019-04-24T14:09:26ZAnonymous
<ul></ul><p>More context. I was stepping thru pulpcore/pulpcore/rqconfig.py and notice it was coughing an error about redis:</p>
<p>This is with DJANGO_SETTINGS_MODULE set.</p>
<pre><code>In [1]: import sys
In [2]: from pulpcore.app.settings import settings
2019-04-24:14:05:30,274 ERROR [redis_loader.py:58 - load] Error 111 connecting to localhost:6379. Connection refused.
</code></pre> Pulp - Issue #4732: pulp workers only use localhost for redishttps://pulp.plan.io/issues/4732?journal_id=411262019-04-24T14:21:09Zbmbouterbmbouter@redhat.com
<ul><li><strong>Status</strong> changed from <i>ASSIGNED</i> to <i>POST</i></li></ul><p>PRs available at:</p>
<p><a href="https://github.com/pulp/ansible-pulp/pull/108" class="external">https://github.com/pulp/ansible-pulp/pull/108</a><br>
<a href="https://github.com/pulp/pulpcore/pull/98" class="external">https://github.com/pulp/pulpcore/pull/98</a></p> Pulp - Issue #4732: pulp workers only use localhost for redishttps://pulp.plan.io/issues/4732?journal_id=411272019-04-24T14:36:58ZAnonymous
<ul></ul><p>Chasing the bug deeper, I ended up in redis_loader.py line 58 in dynaconf v2.0.1</p>
<p>The line in question is in an except Exception block that just swallows and logs everything. Which explains the ERROR observed above.</p>
<p>Further debugging using the powerful print() function showed I was hitting the else block of the try/catch.</p>
<p>Removing the try/catch block</p>
<pre><code>
def load(obj, env=None, silent=True, key=None):
"""Reads and loads in to "settings" a single key or all keys from redis
:param obj: the settings instance
:param env: settings env default='DYNACONF'
:param silent: if errors should raise
:param key: if defined load a single key, else load all in env
:return: None
"""
redis = StrictRedis(**obj.get("REDIS_FOR_DYNACONF"))
holder = obj.get("ENVVAR_PREFIX_FOR_DYNACONF")
try:
if key:
value = redis.hget(holder.upper(), key)
if value:
obj.logger.debug(
"redis_loader: loading by key: %s:%s (%s:%s)",
key,
value,
IDENTIFIER,
holder,
)
if value:
parsed_value = parse_conf_data(value, tomlfy=True)
if parsed_value:
obj.set(key, parsed_value)
else:
data = {
key: parse_conf_data(value, tomlfy=True)
for key, value in redis.hgetall(holder.upper()).items()
}
if data:
obj.logger.debug(
"redis_loader: loading: %s (%s:%s)",
data,
IDENTIFIER,
holder,
)
obj.update(data, loader_identifier=IDENTIFIER)
except Exception as e:
if silent:
if hasattr(obj, "logger"):
obj.logger.error(str(e))
return False
raise
</code></pre>
<p>I removed the try/catch and realized i was mis-reading a list comprehension and the root error is it's still trying to connect to localhost.</p>
<pre><code>During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "x.py", line 1, in <module>
from pulpcore.app.settings import settings
File "/home/ubuntu/pulpcore/pulpcore/app/settings.py", line 218, in <module>
ENVVAR_FOR_DYNACONF='PULP_SETTINGS',
File "/home/ubuntu/.venv/lib/python3.6/site-packages/dynaconf/contrib/django_dynaconf_v2.py", line 67, in load
lazy_settings.populate_obj(django_settings_module)
File "/home/ubuntu/.venv/lib/python3.6/site-packages/dynaconf/base.py", line 90, in __getattr__
self._setup()
File "/home/ubuntu/.venv/lib/python3.6/site-packages/dynaconf/base.py", line 121, in _setup
settings_module=settings_module, **self._kwargs
File "/home/ubuntu/.venv/lib/python3.6/site-packages/dynaconf/base.py", line 184, in __init__
self.execute_loaders()
File "/home/ubuntu/.venv/lib/python3.6/site-packages/dynaconf/base.py", line 736, in execute_loaders
loader.load(self, env, silent=silent, key=key)
File "/home/ubuntu/.venv/lib/python3.6/site-packages/dynaconf/loaders/redis_loader.py", line 44, in load
for key, value in redis.hgetall(holder.upper()).items()
File "/home/ubuntu/.venv/lib/python3.6/site-packages/redis/client.py", line 2649, in hgetall
return self.execute_command('HGETALL', name)
File "/home/ubuntu/.venv/lib/python3.6/site-packages/redis/client.py", line 774, in execute_command
connection.send_command(*args)
File "/home/ubuntu/.venv/lib/python3.6/site-packages/redis/connection.py", line 619, in send_command
self.send_packed_command(self.pack_command(*args))
File "/home/ubuntu/.venv/lib/python3.6/site-packages/redis/connection.py", line 594, in send_packed_command
self.connect()
File "/home/ubuntu/.venv/lib/python3.6/site-packages/redis/connection.py", line 498, in connect
raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 111 connecting to localhost:6379. Connection refused.
</code></pre> Pulp - Issue #4732: pulp workers only use localhost for redishttps://pulp.plan.io/issues/4732?journal_id=411322019-04-24T14:51:56Zbmbouterbmbouter@redhat.com
<ul></ul><p>update for pulplift is here: <a href="https://github.com/pulp/pulplift/pull/28" class="external">https://github.com/pulp/pulplift/pull/28</a></p> Pulp - Issue #4732: pulp workers only use localhost for redishttps://pulp.plan.io/issues/4732?journal_id=411662019-04-24T21:39:11Zbmbouterbmbouter@redhat.com
<ul><li><strong>Status</strong> changed from <i>POST</i> to <i>MODIFIED</i></li></ul><p>Applied in changeset <a class="changeset" title="Adds -c option for RQ workers in docs The docs did not tell users much about how to run an RQ wo..." href="https://pulp.plan.io/projects/pulp/repository/pulpcore/revisions/a37f9abb2bc0d39dfeb54f4a1d4d16147b479acb">pulpcore|a37f9abb2bc0d39dfeb54f4a1d4d16147b479acb</a>.</p> Pulp - Issue #4732: pulp workers only use localhost for redishttps://pulp.plan.io/issues/4732?journal_id=509342019-12-13T17:28:47Zbmbouterbmbouter@redhat.com
<ul><li><strong>Status</strong> changed from <i>MODIFIED</i> to <i>CLOSED - CURRENTRELEASE</i></li></ul> Pulp - Issue #4732: pulp workers only use localhost for redishttps://pulp.plan.io/issues/4732?journal_id=563952020-05-13T20:28:47Zbmbouterbmbouter@redhat.com
<ul><li><strong>Category</strong> set to <i>Installer - Moved to GitHub issues</i></li><li><strong>Tags</strong> deleted (<del><i>Pulp 3 installer</i></del>)</li></ul>