Project

Profile

Help

Task #5031

closed

Consolidate Pulp settings into a single role variable

Added by bmbouter almost 5 years ago. Updated almost 4 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Category:
Installer - Moved to GitHub issues
Sprint/Milestone:
-
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
Groomed:
Yes
Sprint Candidate:
No
Tags:
Sprint:
Sprint 55
Quarter:

Description

Problem

The majority of Pulp settings cannot be set by the Ansible installer. The reason is because the current approach is to have a 1-to-1 mapping of role-variables to Pulp settings, and we only have maybe 5 of the settings mirrored in the installer role variables. The first problem is the gap in the user not being able to set these.

This approach creates a significant maintenance burden though as new settings will have to be specially added to the Installer which is even more work.

Also settings that would be set in Pulp's settings file (e.g. unrelated Django settings) also can't be specified with this approach either. This is probably the largest issue.

Solution

I asked in #ansible and here is what they recommended:

1. Have a single role variable named 'pulp_settings' and make this a dictionary that takes arbitrary key/values.
2. Have the jinja template that builds settings.py unroll all top-level keys into the file.

Details

We need to determine how this works with the minimum required variables like SECRET_KEY...

Actions #1

Updated by mdepaulo@redhat.com almost 5 years ago

I agree that this probably the least undesirable solution.

However:
1. We are going to have document these settings in pulpcore, but tell users to specify them when running ansible-pulp.
2. Users are going to have to specify them in a .yml file. Because specifying them as command-line args to ansible-playbook is too messy. We already do this by default for variables, but now it is infeasible to specify them by hand on the command-line.
3. In which git repo are we going to include an example or default dictionary file?

Actions #2

Updated by bmbouter almost 5 years ago

wrote:

I agree that this probably the least undesirable solution.

However:
1. We are going to have document these settings in pulpcore, but tell users to specify them when running ansible-pulp.

Each setting will need to be documented in the pulpcore docs. For the installer though, I imagined the description of the pulp_settings variable would allow the user to know how to build the dictionary.

2. Users are going to have to specify them in a .yml file. Because specifying them as command-line args to ansible-playbook is too messy. We already do this by default for variables, but now it is infeasible to specify them by hand on the command-line.

I see what you're saying about making it easy to express them. Will there be problems in taking a yaml settings file and making it Python? Since we auto-choose Python as the format for the settings file the installer is producing, maybe the user should just provide the settings.py file itself?

3. In which git repo are we going to include an example or default dictionary file?

We struggled with settings file defaults in Pulp2 because the actual defaults in the code and the documented defaults were regularly out of date. From that experience my main takeaway is that rather than having the defaults "supplied" at runtime, just have them in Pulp itself and have users only think about how they want to change those defaults. This also avoids problems from newer setting in Pulp and having to merge new 'default' files with your already customized version over and over again. Does this make sense?

Actions #3

Updated by cassell almost 5 years ago

I'd recommend against a large dictionary as it makes it hard to override just one setting. Can you point me in the code where this is being done? (I'll try to suggest something better)

Actions #4

Updated by mdepaulo@redhat.com almost 5 years ago

bmbouter wrote:

wrote:

I agree that this probably the least undesirable solution.

However:
1. We are going to have document these settings in pulpcore, but tell users to specify them when running ansible-pulp.

Each setting will need to be documented in the pulpcore docs. For the installer though, I imagined the description of the pulp_settings variable would allow the user to know how to build the dictionary.

They'd probably want to modify a file with it instead of using the default/example file.

2. Users are going to have to specify them in a .yml file. Because specifying them as command-line args to ansible-playbook is too messy. We already do this by default for variables, but now it is infeasible to specify them by hand on the command-line.

I see what you're saying about making it easy to express them. Will there be problems in taking a yaml settings file and making it Python?

No, this isn't the problem.

Since we auto-choose Python as the format for the settings file the installer is producing, maybe the user should just provide the settings.py file itself?

That might be simpler, yeah.

3. In which git repo are we going to include an example or default dictionary file?

We struggled with settings file defaults in Pulp2 because the actual defaults in the code and the documented defaults were regularly out of date. From that experience my main takeaway is that rather than having the defaults "supplied" at runtime, just have them in Pulp itself and have users only think about how they want to change those defaults. This also avoids problems from newer setting in Pulp and having to merge new 'default' files with your already customized version over and over again. Does this make sense?

Ahh, right, I forgot about that.

I guess if by default Pulp users never override these vars, then that makes sense.

So here's what your proposal will look like if a user wanted to specify 3 settings:
you would see the file:
https://github.com/pulp/ansible-pulp/blob/master/example-source/group_vars/all
And at the end, the user would add a dictionary variable like:

pulp_app_settings:
  setting1: value1
  setting2: value2
  setting3: value3

Now, there are features of Ansible to put this variable in another .yml file rather than this one. Probably one that just looks like:

setting1: value1
setting2: value2
setting3: value3

However, say the user wants to specify it on the command-line.

Instead of specifying:

ansible-playbook  example-use/playbook.yml  -e setting1="value1"  -e setting2="value2"  -e setting3="value3"

You would have to override the entire dictionary, with proper JSON (or YML) syntax:

ansible-playbook  example-use/playbook.yml  -e "pulp_app_settings":{ "setting1": "value1", "setting2":"value2", "setting3":"value3" }'

Those double quotes may not all be necessary, but still.

I literally forgot a double-quote as I originally wrote that command.

Actions #5

Updated by bmbouter almost 5 years ago

@mikedep333 those examples all look great to me. I can include those examples formats in the README section on this variable to provide clear examples.

I want to finalize the details around pulp_secret_key. That is the only required setting Pulp users must set. The installer has it today and uses it. So what is better?

1. Keep pulp_secret_key separate and have custom jinja logic that ensures pulp_secret_key is used even if 'secret_key' is specified via 'pulp_settings'?

2. Don't do anything special and make pulp_settings a required setting with at a minimum the 'secret_key' value?

I think I lean towards option 2 because it's "less custom logic so that would be less-surprising for users.

Actions #6

Updated by amacdona@redhat.com almost 5 years ago

+1 to Option 2. I like the idea of all settings.py vars in 1 place.

Actions #7

Updated by mdepaulo@redhat.com almost 5 years ago

wrote:

+1 to Option 2. I like the idea of all settings.py vars in 1 place.

I don't think there are 2 options.

I think the 2-item-list is the 2 parts of the same solution.

Actions #8

Updated by amacdona@redhat.com almost 5 years ago

Mike, oops, didn't mean to be ambiguous. Both parts of the solution in the description +1.

My comment about option 2 was how to deal with the pulp secret. https://pulp.plan.io/issues/5031#note-5

Actions #9

Updated by amacdona@redhat.com almost 5 years ago

  • Groomed changed from No to Yes
  • Sprint set to Sprint 55
Actions #10

Updated by amacdona@redhat.com almost 5 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to amacdona@redhat.com
Actions #12

Updated by amacdona@redhat.com almost 5 years ago

  • Status changed from ASSIGNED to POST

Added by amacdona@redhat.com almost 5 years ago

Revision 3df36099 | View on GitHub

Combine all pulp django settings into single var

By creating a dictionary pulp_settings we can allow users to add or update arbitrary values which will go into their local settings.py. This also creates a clear scope separation between ansible variables and pulp settings variables.

fixes #5031

Added by amacdona@redhat.com almost 5 years ago

Revision 3df36099 | View on GitHub

Combine all pulp django settings into single var

By creating a dictionary pulp_settings we can allow users to add or update arbitrary values which will go into their local settings.py. This also creates a clear scope separation between ansible variables and pulp settings variables.

fixes #5031

Actions #13

Updated by amacdona@redhat.com almost 5 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

Added by amacdona@redhat.com almost 5 years ago

Revision 8f85185c | View on GitHub

Use new ansible commit with pulp_settings dict

And add the settings dict to the example use yaml.

https://pulp.plan.io/issues/5031 re #5031

Added by amacdona@redhat.com almost 5 years ago

Revision 36ebc33b | View on GitHub

Update for latest ansible-pulp settings

Required PR: https://github.com/pulp/ansible-pulp/pull/126 re #5031

Added by amacdona@redhat.com almost 5 years ago

Revision 84dd0aa2 | View on GitHub

update travis for consolidated settings dict

re #5031

Added by amacdona@redhat.com almost 5 years ago

Revision 05c45ecc | View on GitHub

Update travis to use ansible pulp_settings dict

re #5031

Added by Mike DePaulo over 4 years ago

Revision bed43995 | View on GitHub

Combine all pulp django settings into single var

By creating a dictionary pulp_settings we can allow users to add or update arbitrary values which will go into their local settings.py. This also creates a clear scope separation between ansible variables and pulp settings variables.

re #5031

[noissue]

Actions #14

Updated by bmbouter over 4 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE
Actions #15

Updated by bmbouter almost 4 years ago

  • Category set to Installer - Moved to GitHub issues
  • Tags deleted (Pulp 3 installer)

Also available in: Atom PDF