Project

Profile

Help

Story #3943

closed

Issue #3981: Switch Pulp3 settings to Dynaconf

As a user, I can configure Pulp using environment variables

Added by bmbouter over 5 years ago. Updated over 4 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Quarter:

Description

Problem Statement

While trying to get Pulp3 running on Openshift I ran into an issue where I needed to set a setting in the settings.yaml file from an environment variable. For example the IP of the postgresql instance is only known to openshift and available via an enviornment variable.

This is a blocker on deploying Pulp3 into Openshift.

Solution

Make our settings file a python module instead with the config available at as a module attribute named 'PULP_CONFIG', e.g.

$ cat settings.py

# get lots of data from ENVIRONMENT variables
PULP_SETTINGS = {}  # This would be the toplevel dictionary that would overwrite _DEFAULT_PULP_SETTINGS
# _DEFAULT_PULP_SETTINGS is here:  https://github.com/pulp/pulp/blob/master/pulpcore/pulpcore/app/settings.py#L163

How will the conf file be imported?

Same way as today, first an environment variable is checked which contains a full path to the python module. If that environment variable isn't set it defaults to look to /etc/pulp/settings.py

The only change is /etc/pulp/server.yaml to /etc/pulp/settings.py as the default location accessed.

What happens if Pulp finds no settings file or no PULP_SETTINGS attribute in there. It has to fail because Pulp requires the settings file to read the value of SECRET_KEY which we can never ship.

Actions #1

Updated by bmbouter over 5 years ago

  • Description updated (diff)
Actions #2

Updated by bmbouter over 5 years ago

  • Description updated (diff)
Actions #3

Updated by mhrivnak over 5 years ago

Most services that run on kubernetes try to not require a config file at all, or at least make it optional. They'll accept settings both as command-line arguments and as environment variables, with the command-line args usually winning in case both are used. Here are a couple of options that would be reasonable and fit convention.

1. Somewhere pulp is parsing a config file and storing the values it finds in a structure that gets used in code. That point of parsing is a reasonable place to also look for an environment variable and let it override any setting found in the file. Here is a popular go library that takes such an approach, which may be interesting as an example: https://github.com/spf13/viper

2. Run a small entrypoint script in your container that reads environment variables and uses them to create the yaml file on the spot, then exec pulp's process. Such scripts are very common for doing environmental setup of that variety.

3. Store the whole config file in a kubernetes Secret, mount it into the pod as a file, and symlink from the path where Pulp expects it to be found to wherever you mounted it. One advantage here is that you can make just one secret and mount it into all the pods. That's the approach I took here: https://github.com/mhrivnak/pulp-k8s

Hopefully that's helpful. Feel free to ping me if you'd like to chat more about any of it.

Actions #4

Updated by daviddavis over 5 years ago

  • Tracker changed from Issue to Story
  • Subject changed from Cannot drive conf file from environment variables to As a user, I can configure Pulp using environment variables
  • % Done set to 0
Actions #5

Updated by bmbouter over 5 years ago

  • Parent issue set to #3981
Actions #6

Updated by bmbouter over 5 years ago

For my near-term needs I used a bash script to read environment variables and write the necessary yaml config. For the future I think switching to a Python settings file would resolve several issues including this one.

Actions #7

Updated by amacdona@redhat.com over 5 years ago

  • Sprint set to Sprint 43
Actions #8

Updated by bmbouter over 5 years ago

  • Sprint deleted (Sprint 43)

This will be resolved by the epic 3981.

Actions #9

Updated by dkliban@redhat.com over 5 years ago

  • Status changed from NEW to POST
  • Assignee set to dkliban@redhat.com

Added by Bruno Rocha over 5 years ago

Revision e1fcdda1 | View on GitHub

Problem: Pulp can't be configured using environment variables

Solution: use dynaconf to manage all of Pulp configuration settings

This patch also adds pulpcore.rqconfig module that is dynamically generated from the Django settings generated by dynaconf. The rq workers use this module for their configuration.

The default location of the config file is now at /etc/pulp/settings.py. Pulp can operate without by relying on environment variables.

closes: #3981 https://pulp.plan.io/issues/3981

closes: #3879 https://pulp.plan.io/issues/3879

closes: #3980 https://pulp.plan.io/issues/3980

closes #3618 https://pulp.plan.io/issues/3618

closes: #3943 https://pulp.plan.io/issues/3943

Added by Bruno Rocha over 5 years ago

Revision e1fcdda1 | View on GitHub

Problem: Pulp can't be configured using environment variables

Solution: use dynaconf to manage all of Pulp configuration settings

This patch also adds pulpcore.rqconfig module that is dynamically generated from the Django settings generated by dynaconf. The rq workers use this module for their configuration.

The default location of the config file is now at /etc/pulp/settings.py. Pulp can operate without by relying on environment variables.

closes: #3981 https://pulp.plan.io/issues/3981

closes: #3879 https://pulp.plan.io/issues/3879

closes: #3980 https://pulp.plan.io/issues/3980

closes #3618 https://pulp.plan.io/issues/3618

closes: #3943 https://pulp.plan.io/issues/3943

Actions #10

Updated by Anonymous over 5 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100
Actions #11

Updated by daviddavis about 5 years ago

  • Sprint/Milestone set to 3.0.0
Actions #12

Updated by bmbouter about 5 years ago

  • Tags deleted (Pulp 3)
Actions #13

Updated by bmbouter over 4 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF