Project

Profile

Help

Story #3943

Issue #3981: Switch Pulp3 settings to Dynaconf

As a user, I can configure Pulp using environment variables

Added by bmbouter over 1 year ago. Updated 8 months ago.

Status:
MODIFIED
Priority:
Normal
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Platform Release:
Blocks Release:
Backwards Incompatible:
No
Groomed:
No
Sprint Candidate:
No
Tags:
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:

Description

Problem Statement

While trying to get Pulp3 running on Openshift I ran into an issue where I needed to set a setting in the settings.yaml file from an environment variable. For example the IP of the postgresql instance is only known to openshift and available via an enviornment variable.

This is a blocker on deploying Pulp3 into Openshift.

Solution

Make our settings file a python module instead with the config available at as a module attribute named 'PULP_CONFIG', e.g.

$ cat settings.py

# get lots of data from ENVIRONMENT variables
PULP_SETTINGS = {}  # This would be the toplevel dictionary that would overwrite _DEFAULT_PULP_SETTINGS
# _DEFAULT_PULP_SETTINGS is here:  https://github.com/pulp/pulp/blob/master/pulpcore/pulpcore/app/settings.py#L163

How will the conf file be imported?

Same way as today, first an environment variable is checked which contains a full path to the python module. If that environment variable isn't set it defaults to look to /etc/pulp/settings.py

The only change is /etc/pulp/server.yaml to /etc/pulp/settings.py as the default location accessed.

What happens if Pulp finds no settings file or no PULP_SETTINGS attribute in there. It has to fail because Pulp requires the settings file to read the value of SECRET_KEY which we can never ship.

Associated revisions

Revision e1fcdda1 View on GitHub
Added by Bruno Rocha about 1 year ago

Problem: Pulp can't be configured using environment variables

Solution: use dynaconf to manage all of Pulp configuration settings

This patch also adds pulpcore.rqconfig module that is dynamically generated from the Django settings
generated by dynaconf. The rq workers use this module for their configuration.

The default location of the config file is now at `/etc/pulp/settings.py`. Pulp can operate without by relying
on environment variables.

closes: #3981
https://pulp.plan.io/issues/3981

closes: #3879
https://pulp.plan.io/issues/3879

closes: #3980
https://pulp.plan.io/issues/3980

closes #3618
https://pulp.plan.io/issues/3618

closes: #3943
https://pulp.plan.io/issues/3943

Revision e1fcdda1 View on GitHub
Added by Bruno Rocha about 1 year ago

Problem: Pulp can't be configured using environment variables

Solution: use dynaconf to manage all of Pulp configuration settings

This patch also adds pulpcore.rqconfig module that is dynamically generated from the Django settings
generated by dynaconf. The rq workers use this module for their configuration.

The default location of the config file is now at `/etc/pulp/settings.py`. Pulp can operate without by relying
on environment variables.

closes: #3981
https://pulp.plan.io/issues/3981

closes: #3879
https://pulp.plan.io/issues/3879

closes: #3980
https://pulp.plan.io/issues/3980

closes #3618
https://pulp.plan.io/issues/3618

closes: #3943
https://pulp.plan.io/issues/3943

History

#1 Updated by bmbouter over 1 year ago

  • Description updated (diff)

#2 Updated by bmbouter over 1 year ago

  • Description updated (diff)

#3 Updated by mhrivnak over 1 year ago

Most services that run on kubernetes try to not require a config file at all, or at least make it optional. They'll accept settings both as command-line arguments and as environment variables, with the command-line args usually winning in case both are used. Here are a couple of options that would be reasonable and fit convention.

1. Somewhere pulp is parsing a config file and storing the values it finds in a structure that gets used in code. That point of parsing is a reasonable place to also look for an environment variable and let it override any setting found in the file. Here is a popular go library that takes such an approach, which may be interesting as an example: https://github.com/spf13/viper

2. Run a small entrypoint script in your container that reads environment variables and uses them to create the yaml file on the spot, then exec pulp's process. Such scripts are very common for doing environmental setup of that variety.

3. Store the whole config file in a kubernetes Secret, mount it into the pod as a file, and symlink from the path where Pulp expects it to be found to wherever you mounted it. One advantage here is that you can make just one secret and mount it into all the pods. That's the approach I took here: https://github.com/mhrivnak/pulp-k8s

Hopefully that's helpful. Feel free to ping me if you'd like to chat more about any of it.

#4 Updated by daviddavis over 1 year ago

  • Tracker changed from Issue to Story
  • Subject changed from Cannot drive conf file from environment variables to As a user, I can configure Pulp using environment variables
  • % Done set to 0

#5 Updated by bmbouter over 1 year ago

  • Parent task set to #3981

#6 Updated by bmbouter over 1 year ago

For my near-term needs I used a bash script to read environment variables and write the necessary yaml config. For the future I think switching to a Python settings file would resolve several issues including this one.

#7 Updated by amacdona@redhat.com about 1 year ago

  • Sprint set to Sprint 43

#8 Updated by bmbouter about 1 year ago

  • Sprint deleted (Sprint 43)

This will be resolved by the epic 3981.

#9 Updated by dkliban@redhat.com about 1 year ago

  • Status changed from NEW to POST
  • Assignee set to dkliban@redhat.com

#10 Updated by Anonymous about 1 year ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#11 Updated by daviddavis 8 months ago

  • Sprint/Milestone set to 3.0

#12 Updated by bmbouter 8 months ago

  • Tags deleted (Pulp 3)

Please register to edit this issue

Also available in: Atom PDF