This is a blog with random OpenShift, Kubernetes, OpenStack and Linux related notes so I don't forget things. If you find something inaccurate or that could be fixed, please file a bug report here.

View on GitHub

Back to home

23 August 2018

Configuration files in TripleO

by Juan Antonio Osorio Robles

There have been several ocasions where I’ve been asked where a file gets generated, or where does it come from in a TripleO deployment. Ideally this would be quite an easy thing to answer. However, more often than not, there is quite a bit of stuff going on in the deployment steps that makes this task non-trivial.

There are ways to go forward and look. The aim of this blog post is to try to put in words a systematic way of doing this.

Where do I begin?

It all begins in tripleo-heat-templates, which we will refer from now on as t-h-t. Note that it’s relevant to know how to write Heat templates in order to understand how this all works.

In these templates we define a lot of things about the service’s deployment. The services themselves are defined in the */services/ folder.

There are three main folders currently in t-h-t:

Why was this relevant?

These are the places were you will look for information, these folders tell you how the service was deployed, and the templates inside these folders contain the information about how the service is being configured. Most of the information is set as the output for these templates, which are actually Heat stacks. The output itself is grouped in a common output called role_data, which will contain several sections that are used in different ways.

Digging in the templates

There’s a lot of information in the templates, but, relevant to our search are the following:

Note that not all of the aforementioned sections are mandatory, but it’s relevant to understand them to know what’s going on in the deployment.

With all this information in hand, we can now know how the configuration was made and figure out where to look. For most services, chances are that the next step is to look at the puppet manifest that configures this service. So lets do that.


As mentioned before, most services are still configured by puppet. The way to know what repository to look at is to check the value of step_config, then take the first word in the include statement, and prepend “puppet-” to it.

Most of the services, however, have a wrapper manifest in the puppet-tripleo repository. The way to get which puppet file to look at also relies on the include invocation we saw on step_config. Lets figure this out with an example:

For keystone, we can see that the value of step_config is ::tripleo::profile::base::keystone. From this we can tell that it was configured from a manifest in puppet-tripleo, because of the “tripleo” keyword in the beginning of the include. Inside puppet-tripleo, we can take the rest of the keywords in that invocation to figure out the manifest. All of the manifests are inside the manifests/ directory, and inside there, we go deeper with each keyword. So for ::tripleo::profile::base::keystone we’ll find the manifest in manifests/profile/base/keystone.pp.

With this in mind, we can go as deep as needed to find where configurations are done.

We can see that the keystone.pp manifest calls the “keystone” class:

    class { '::keystone':
      sync_db                    => $sync_db,
      enable_bootstrap           => $sync_db,

So, to see what that class does, we need to follow the same logic as we did before. We prepend the “puppet-” keyword in order to know what repository contains what we need. In this case, we need to look in puppet-keystone.

If the class or the include statement doesn’t contain a keyword after the one we used to derive the repository, this means that the init.pp manifest was used, which is also in the manifests/ directory of the repository.

It is here that we’ll start seeing signs of where the actual configuration parameters are set.

Lets look at the following statement in the init.pp file for puppet-keystone:

  keystone_config {
    'token/provider':              value => $token_provider;
    'DEFAULT/max_token_size':      value => $max_token_size;
    'DEFAULT/notification_format': value => $notification_format;

Here we can explicitly see how token-related options are configured for keystone. It’s fairly simple to map these options to keystone.conf. Lets take the first option here as an example: token/provider. Here, token is the group for the configuration option, and provider is the actual option. In keystone.conf this will look as:

provider = <some value>

But, how do we know that this keystone_config statement configures /etc/keystone/keystone.conf?

Well, for this, you need to know some puppet. But the main thing is that puppet allows you to define “providers”, which are pieces of code that allow you to extend puppet’s functionality to do a certain task. There are providers to create the keystone endpoints, to modify configuration files, and all sorts of things. To find them, we’ll need to look in the following directory: lib/puppet/provider/. Here we can see the different providers that the specific pupet module enables.

Going back to the keystone example, as we can see, the aforementioned keystone_config definition is here. The full path to it would be lib/puppet/provider/keystone_config/ini_setting.rb. Here, we can explicitly see that this module configures the configuration file:

  :parent => Puppet::Type.type(:openstack_config).provider(:ini_setting)
) do

  def self.file_path


So… What does this mean in the actual deployment?

When looking in the configuration files, if a service was deployed on bare-metal, the configuration files will be exactly where you expect. However, when dealing with containerized services, TripleO executes puppet inside a container and persists the configurations in the following directory: **/var/lib/config-data/puppet-generated/**. Inside the aforementioned directory you'll find only the files that puppet manages and modifies, with the file structure you would expect in the system. So, if you're looking for **/etc/keystone/keystone.conf** you'll find it in **/var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf**

But… The file I was looking for wasn’t managed by puppet

It could be the case that the service is managed by puppet, however, the specific file you were looking for isn’t managed by puppet. While this might be a bit confusing, don’t worry, there is no black magic here :) (or is it?).

Chances are the specific file you were looking for was already part of the container. Which means, it comes from a package.

Packages themselves are maintained by a group of awesome folks, the RDO community.

They have their own Gerrit instance, on which they host the rpm spec definitions. You’ll normally be able to find the spec definition, and other relevant files in a special repository with the -distgit suffix. So, to find the packaging for Keystone, we need to look at the keystone-distgit repository.

The .spec file will show us all the files that are included for that package, and some extra files will be also part of the distgit repository, such as the logrotate configuration.

How do I know what packages are included in the container?

TripleO uses Kolla to build containers. So, the place to look at is the kolla repository. In this repository, and under the docker/ directory, we can find the list of projects they support. Here we have the Dockerfile definitions used to build the containers, which are set as jinja templates in order to allow folks to extend them. The hierarchy will usually be **docker//**, although some projects consist of only one container, in which case, you'll find the Dockerflie definition under **docker/**.

Looking at the dockerfile, you’ll find the packages installed are defined by the following keyword: <container name>_packages. For instance, for barbican, we’ll find the following in the barbican-api definition:


       {% set barbican_api_packages = [
       ] %}


For TripleO, we might have some extra needs for the container though. So, for things that we are not useful for the Kolla community and are TripleO-specific, we’ll need to look at the tripleo-common directory. Any overrides we do to the kolla container images, we’ll find under container-images/tripleo_kolla_template_overrides.j2. Package additions can be found to be set with the <container name>_packages_append variable.

tags: tripleo - openstack

Back to home