Issue #4679: Custom settings not propogating - Pulp

Actions

Send by e-mail Copy link

Issue #4679

closed

Custom settings not propogating

Added by tustvold over 5 years ago. Updated over 5 years ago.

Status:

CLOSED - WORKSFORME

Priority:

Normal

Assignee:

rochacbruno

Category:

Sprint/Milestone:

3.0.0

Start date:

Due date:

Estimated time:

Severity:

3. High

Version:

Platform Release:

OS:

Triaged:

Yes

Groomed:

Sprint Candidate:

Tags:

Sprint:

Sprint 51

Quarter:

Description

I'm trying to configure the S3 storage backend, however, am running into an issue with settings not propogating correctly.

I have set DJANGO_SETTINGS_MODULE to pulpcore.app.settings as an environment variable, and additionally have set PULP_SETTINGS to /var/lib/pulp/settings.yaml which is where I have set the variables. The templated contents of which are below:

default:
  # Django
  SECRET_KEY: '${PULP_SECRET}'
  DEBUG: True

  CONTENT_HOST: '${CONTENT_HOST}'

  # Redis
  REDIS_HOST: '${REDIS_HOST}'
  REDIS_PORT: ${REDIS_PORT}
  REDIS_PASSWORD: '${REDIS_PASSWORD}'

  # Storage
  AWS_ACCESS_KEY_ID: '${S3_ACCESS_KEY_ID}'
  AWS_SECRET_ACCESS_KEY: '${S3_SECRET_ACCESS_KEY}'
  AWS_STORAGE_BUCKET_NAME: '${S3_STORAGE_BUCKET_NAME}'
  AWS_DEFAULT_ACL:
  AWS_AUTO_CREATE_BUCKET: False

  DEFAULT_FILE_STORAGE: storages.backends.s3boto3.S3Boto3Storage
  MEDIA_ROOT:  ''

  # Postgres
  DATABASES:
    default:
      CONN_MAX_AGE: 0
      ENGINE: django.db.backends.postgresql_psycopg2
      NAME: '${PGSQL_DB}'
      USER: '${PGSQL_USER}'
      PASSWORD: '${PGSQL_PASSWORD}'
      HOST: '${PGSQL_HOST}'
      PORT: ${PGSQL_PORT}

The problem appears to be that pulpcore.app.settings is following the explicit approach as described here but is missing the crucial settings.populate_obj(sys.modules[__name__]). This means that only the settings explicitly mentioned in the file are propogated.

Caveat: I am new to django and dynaconf so might be missing something incredibly obvious

Actions

Copy link

Updated by tustvold over 5 years ago

For reference the pip packages are

aiofiles==0.4.0
aiohttp==3.5.4
async-timeout==3.0.1
attrs==19.1.0
backoff==1.8.0
boto3==1.9.130
botocore==1.12.130
certifi==2019.3.9
chardet==3.0.4
Click==7.0
coreapi==2.3.3
coreschema==0.0.4
Django==2.1.7
django-filter==2.1.0
django-storages==1.7.1
djangorestframework==3.9.2
djangorestframework-queryfields==1.0.0
docutils==0.14
drf-chunked-upload==0.4.2
drf-nested-routers==0.91
drf-yasg==1.14.0
dynaconf==1.2.1
gunicorn==19.9.0
idna==2.8
inflection==0.3.1
itypes==1.1.0
Jinja2==2.10
jmespath==0.9.4
MarkupSafe==1.1.1
multidict==4.5.2
psycopg2-binary==2.7.7
pulp-file==0.0.1b10
pulpcore==3.0.0rc1
pulpcore-plugin==0.1.0rc1
python-box==3.2.4
python-dateutil==2.8.0
python-dotenv==0.10.1
pytz==2018.9
PyYAML==3.13
redis==3.1.0
requests==2.21.0
rq==0.13.0
ruamel.yaml==0.15.89
s3transfer==0.2.0
six==1.12.0
toml==0.10.0
uritemplate==3.0.0
urllib3==1.24.1
whitenoise==4.1.2
yarl==1.3.0

Actions

Copy link

Updated by amacdona@redhat.com over 5 years ago

Status changed from NEW to ASSIGNED
Assignee set to rochacbruno
Sprint/Milestone set to 3.0.0
Triaged changed from No to Yes
Sprint set to Sprint 51

Actions

Copy link

Updated by rochacbruno over 5 years ago

Status changed from ASSIGNED to MODIFIED

This issue was also reported in: https://github.com/rochacbruno/dynaconf/issues/89

And it is now fixed by: https://github.com/pulp/pulpcore/pull/72 (pending review/merge)

I used exactly the same settings you provided in the same location and

(pulp) django-admin print_settings DATABASES
DATABASES                                = <Box: {'default': {'CONN_MAX_AGE': 0, 'ENGINE': 'django.db.backends.postgresql_psycopg2', 'NAME': '${PGSQL_DB}', 'USER': '${PGSQL_USER}', 'PASSWORD': '${PGSQL_PASSWORD}', 'HOST': '${PGSQL_HOST}', 'PORT': '${PGSQL_PORT}'}}>

(pulp) django-admin print_settings REDIS_HOST
REDIS_HOST                               = '${REDIS_HOST}'

(pulp) dynaconf list -k DATABASES
Django app detected
Working in development environment 
DATABASES: {'default': {'CONN_MAX_AGE': 0,
             'ENGINE': 'django.db.backends.postgresql_psycopg2',
             'HOST': '${PGSQL_HOST}',
             'NAME': '${PGSQL_DB}',
             'PASSWORD': '${PGSQL_PASSWORD}',
             'PORT': '${PGSQL_PORT}',
             'USER': '${PGSQL_USER}'}}

(pulp) dynaconf list -k REDIS_HOST
Django app detected
Working in development environment 
REDIS_HOST: '${REDIS_HOST}'

(pulp) dynaconf list -k SETTINGS
Django app detected
Working in development environment 
SETTINGS: '/var/lib/pulp/settings.yaml'

Note that dynaconf does not support template substitutions yet (there is this issue https://github.com/rochacbruno/dynaconf/issues/145 to track and make the use of `${EXISTING_VAR}` available.

Actions

Copy link

Updated by tustvold over 5 years ago

That is fantastic news, I look forward to being able to remove the below from my Dockerfile

RUN echo "import sys; settings.populate_obj(sys.modules[__name__])" >> $(python -c "import pulpcore.app.settings; print (pulpcore.app.settings.__file__)")

Also the templating is being done by terraform prior to the yaml file being deployed as a kubernetes secret, so the dynaconf issue isn't a problem for me.

Actions

Copy link