Story #3943
closedIssue #3981: Switch Pulp3 settings to Dynaconf
As a user, I can configure Pulp using environment variables
100%
Description
Problem Statement¶
While trying to get Pulp3 running on Openshift I ran into an issue where I needed to set a setting in the settings.yaml file from an environment variable. For example the IP of the postgresql instance is only known to openshift and available via an enviornment variable.
This is a blocker on deploying Pulp3 into Openshift.
Solution¶
Make our settings file a python module instead with the config available at as a module attribute named 'PULP_CONFIG', e.g.
$ cat settings.py
# get lots of data from ENVIRONMENT variables
PULP_SETTINGS = {} # This would be the toplevel dictionary that would overwrite _DEFAULT_PULP_SETTINGS
# _DEFAULT_PULP_SETTINGS is here: https://github.com/pulp/pulp/blob/master/pulpcore/pulpcore/app/settings.py#L163
How will the conf file be imported?¶
Same way as today, first an environment variable is checked which contains a full path to the python module. If that environment variable isn't set it defaults to look to /etc/pulp/settings.py
The only change is /etc/pulp/server.yaml to /etc/pulp/settings.py as the default location accessed.
What happens if Pulp finds no settings file or no PULP_SETTINGS attribute in there. It has to fail because Pulp requires the settings file to read the value of SECRET_KEY which we can never ship.¶
Updated by mhrivnak over 6 years ago
Most services that run on kubernetes try to not require a config file at all, or at least make it optional. They'll accept settings both as command-line arguments and as environment variables, with the command-line args usually winning in case both are used. Here are a couple of options that would be reasonable and fit convention.
1. Somewhere pulp is parsing a config file and storing the values it finds in a structure that gets used in code. That point of parsing is a reasonable place to also look for an environment variable and let it override any setting found in the file. Here is a popular go library that takes such an approach, which may be interesting as an example: https://github.com/spf13/viper
2. Run a small entrypoint script in your container that reads environment variables and uses them to create the yaml file on the spot, then exec pulp's process. Such scripts are very common for doing environmental setup of that variety.
3. Store the whole config file in a kubernetes Secret, mount it into the pod as a file, and symlink from the path where Pulp expects it to be found to wherever you mounted it. One advantage here is that you can make just one secret and mount it into all the pods. That's the approach I took here: https://github.com/mhrivnak/pulp-k8s
Hopefully that's helpful. Feel free to ping me if you'd like to chat more about any of it.
Updated by daviddavis over 6 years ago
- Tracker changed from Issue to Story
- Subject changed from Cannot drive conf file from environment variables to As a user, I can configure Pulp using environment variables
- % Done set to 0
Updated by bmbouter over 6 years ago
For my near-term needs I used a bash script to read environment variables and write the necessary yaml config. For the future I think switching to a Python settings file would resolve several issues including this one.
Updated by bmbouter over 6 years ago
- Sprint deleted (
Sprint 43)
This will be resolved by the epic 3981.
Updated by dkliban@redhat.com about 6 years ago
- Status changed from NEW to POST
- Assignee set to dkliban@redhat.com
Added by Bruno Rocha about 6 years ago
Added by Bruno Rocha about 6 years ago
Revision e1fcdda1 | View on GitHub
Problem: Pulp can't be configured using environment variables
Solution: use dynaconf to manage all of Pulp configuration settings
This patch also adds pulpcore.rqconfig module that is dynamically generated from the Django settings generated by dynaconf. The rq workers use this module for their configuration.
The default location of the config file is now at /etc/pulp/settings.py
. Pulp can operate without by relying
on environment variables.
closes: #3981 https://pulp.plan.io/issues/3981
closes: #3879 https://pulp.plan.io/issues/3879
closes: #3980 https://pulp.plan.io/issues/3980
Updated by Anonymous about 6 years ago
- Status changed from POST to MODIFIED
- % Done changed from 0 to 100
Applied in changeset pulp|e1fcdda1d56f388bc8daba20f13d6a25217f5c25.
Updated by bmbouter about 5 years ago
- Status changed from MODIFIED to CLOSED - CURRENTRELEASE
Problem: Pulp can't be configured using environment variables
Solution: use dynaconf to manage all of Pulp configuration settings
This patch also adds pulpcore.rqconfig module that is dynamically generated from the Django settings generated by dynaconf. The rq workers use this module for their configuration.
The default location of the config file is now at
/etc/pulp/settings.py
. Pulp can operate without by relying on environment variables.closes: #3981 https://pulp.plan.io/issues/3981
closes: #3879 https://pulp.plan.io/issues/3879
closes: #3980 https://pulp.plan.io/issues/3980
closes #3618 https://pulp.plan.io/issues/3618
closes: #3943 https://pulp.plan.io/issues/3943