Cooking with Chef

I wanted to take some time to explain one of my favorite configuration management tools. That is Chef. I've been utilizing it pretty heavily over the passed few months and thought I'd share some of what I learned. I'm by no means an expert, but I've been managing my own chef-server as well as writing plenty of my own recipes.

Part 1: Cookbooks and Recipes

It is important to understand what Recipes and Cookbooks really are. A cookbook is a collection of Recipes, Attributes, Templates, Files, and possibly even Resources and Providers. On the file system, it looks like this:

dpadmin@chef-east:~/cookbooks$ ls haproxy/
attributes  metadata.json  metadata.rb  README.rdoc  recipes  templates

Each piece of your cookbook will be under the sub-directory related to what that piece represents. So for example, your template files will go under templates, recipes under recipes, etc. The main pieces of your cookbooks will be the Recipes. This is where you define what will happen when applied to a node. Let's start with something basic, like installing and starting HAproxy:

package "haproxy" do
  action :install
end

service "haproxy" do
  supports :restart => true, :status => true, :reload => true
  action [:enable, :start]
end

More information can be found on Opscode's Wiki for Resources, which you should get familiar with because it really helps you configure individual resources and understand what is going on. In the above example, you have two Resources: a Package and a Service resource. The package resource does nothing more than install a particular package. The service resource defines a service that we want to control in our recipes. In this case it is haproxy (a load balancer). We tell chef that haproxy supports the restart, status, and reload actions (think init scripts). Then we tell the service to enable itself (chkconfig, update-rc.d, etc), then start the service.

Now that we have the required package installed and service configured, we want to start using templates for the configuration files. Here's a template resource for haproxy.cfg:

template "/etc/haproxy/haproxy.cfg" do
  source "haproxy.cfg.erb"
  owner "root"
  group "root"
  mode "644"
  variables(:servers => server_list)
  notifies :reload, resources(:service => "haproxy")
end

In this resource definition, you can see we specify the full path to the file we will be templating (/etc/haproxy/haproxy.cfg) and the source of the template (haproxy.cfg.erb). Here's a great explanation of template location specificity, but for now we will focus more on the Recipe aspect rather than Templates. The owner, group, and mode directives here specify the permissions the template will have. The variables are exactly what they are called, variables. These variables are passed to the Erubis template engine to be parsed. In this case we are passing a list of servers called :servers and defining it as the contents of server_list (an array, for those familiar with programming concepts). Finally, since the template is the configuration for a service, we need to tell our haproxy service to reload its configuration. The notifies directive can be used in any Resource definition and allows you to tell another resource to perform some sort of action. Here we tell haproxy to reload (/etc/init.d/haproxy reload).

Part 2: Templates

Templates are part of Cookbooks, but really need their own section because they can be confusing if you don't know how they are parsed. Take a look at this template for haproxy.cfg:

dpadmin@chef-east:~/cookbooks$ cat haproxy/templates/default/haproxy.cfg.erb 
global
        log 127.0.0.1   local0
        log 127.0.0.1   local1 notice
        #log loghost    local0 info
        maxconn 4096
        #debug
        #quiet
        user haproxy
        group haproxy

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        option  forwardfor
        retries 3
        option  redispatch
        maxconn 2000
        contimeout      5000
        clitimeout      50000
        srvtimeout      50000

# Set up application listeners here.
listen application *:80
        mode http
        balance roundrobin
        stats enable
        stats auth myuser:mypass
        <% @servers.each do |name,ip| -%>
        server <%= name %> <%= ip %> check
        <% end -%>

For the most part this configuration is pretty static. The only rendered piece is at the bottom:

        <% @servers.each do |name,ip| -%>
        server <%= name %> <%= ip %> check
        <% end -%>

As you saw in the template definition in the previous section, we pass the variable :servers, which in the template file is called an instance variable. In Ruby, instance variables are prefixed by an @. The rest is Ruby syntax, but I will do my best to explain what's going on here. A hash (key->value store), has various methods (functions). In this case we are calling the each method, which will loop through each key->value pair in the hash. The |name,ip| part gets assigned the key and hash values, then we output these values by calling them directly inside the loop. This allows us to generate a final product that looks like this:

        server web2.example.com 10.204.39.249 check
        server web1.example.com 10.203.83.213 check

As you can see, our @servers hash had two elements. The values of each element were as follows:

{
  "web2.example.com" => "10.204.39.249",
  "web1.example.com" => "10.203.83.213"
}

So to wrap it up, the Templates get passed variables from Recipes. These variables can be anything including hashes, arrays, strings, and integers. Then the ERB template engine is in charge of parsing the data to come up with our final result.

You may be wondering where we got server_list to begin with. I left it out of the previous examples as it's necessary to understand the guts of Chef before moving towards where data is stored. The real power of Chef is being able to separate all of your data from the actual working parts. This allows easy re-use of the same recipes across any number of different data stores. In this case, we would be talking about each config environment having its own data to be used with any pre-written recipes.

Part 3: Understanding where data is stored

Chef stores a wealth of information that is indexed and search-able from within recipes. When you are writing your recipes, you don't want to keep variables that are statically assigned. This makes them less re-usable and way too specific. So, continuing with the HAproxy example, I will show you where server_list (the hash of servers to generate our config) came from. In a Data Bag Item, we have the following JSON data:

...
  "haproxy": {
    "eip": "204.236.219.111",
    "production_role": "prod-web"
  },
  "common_names": {
    "lb1": "lb1.example.com",
    "web1": "web1.example.com",
    "web2": "web2.example.com"
  },
...

Then, within our recipe file we have the following logic to generate the server_list hash:

# get our config data bag item
config_info = data_bag_item("configs", node.default[:cid])

# pull out the common_names hash for later use
common_names = config_info['common_names']

# assign a new hash that we will store hostnames to ip addresses
server_list = Hash.new

# basic logging
Chef::Log.debug("Searching for role #{config_info['haproxy']['production_role']}")

# search for nodes that have the "config_info['haproxy']['production_role']" role applied to them,
# loop through each one and assign any matches to the server variable
search(:node, "role:#{config_info['haproxy']['production_role']}").each do |server|
  Chef::Log.debug("Found a server: #{server} IP: #{server.cloud.private_ips.first}")

  # add to our hash. use the common name for the server as key and the private ip as the value
  server_list[common_names[server.name]] = server.cloud.private_ips.first
  Chef::Log.info("Added #{common_names[server.name]} => #{server_list[common_names[server.name]]}")
end

In this case we retrieve some data from a Data Bag Item. The Data Bag being "configs" and the item being node.default[:cid]. Here we access a custom attribute called :cid. So when running this recipe against a group of nodes with the config ID of "example," the Data Bag Item is configs[example]. Now, inside that data bag item, we have a bunch of key->value pairs (think hashes). We access the information stored under 'common_names', 'haproxy', and whatever else we want to store. All of this is just being pulled from an arbitrary JSON data store that Chef refers to as a Data Bag.

When it comes down to it, what we are doing is searching for any nodes that have the role "prod-web" applied to them. I created this role for any production web servers that example.com will be load balancing between. So when our HAproxy recipe is applied to a different node, it generates the configuration file based off of which servers have that role applied to them. Data Bags are not the only place information can be stored. There are three main places we can store data:

  • Data Bag Items
  • Role Attributes
  • Node Attributes

In the HAproxy example, we make use of all three of the above. The "common_names" for each host in a solution and which role to search for are stored in a Data Bag Item. The config identifier (cid) is being stored in a Role Attribute, and our IP address information comes from Node Attributes. All of this data is easily viewable using the knife command line tool.

Data Bags

For Chef, there's two parts to this. There's the Data Bag, then the Data Bag Item. A data bag can store multiple items. Each Item is nothing more than an arbitrary JSON data store. This is a great place to store information that will be used in various recipes. I've been using it as a central configuration for various services. What this means is, I have the configs data bag, and each Item is called by the configs ID. Inside each data store, I keep configuration variables and general data for any recipes that will be applied to nodes in that config's environment. To give you a brief idea, here's what example.com has:

{
  "ldap": {
    "bindpw": "xxxxxxxxxxxxxxxxxxx",
    "binddn": "cn=proxy,ou=services,dc=example,dc=cid",
  },
  "common_names": {
    "lb1": "lb1.example.com",
    "web1": "web1.example.com",
    "web2": "web2.example.com"
  },
  "hosts": {
    "us-east-1": [
      "10.204.39.249 web2.example.com web2",
      "10.203.83.213 web1.example.com web1",
      "10.203.29.220 lb1.example.com lb1"
    ]
  },
  "id": "example",
  "aws": {
    "aws_access_key_id": "XXXXXXXXXXXXXXXXXXXX",
    "aws_secret_access_key_id": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
  },
  "apache": {
    "vhosts": {
      "www.example.com": {
        "DocumentRoot": "/ebs/sites/example.com"
      },
      ...
    }
  },
  "haproxy": {
    "eip": "204.236.219.111",
    "production_role": "prod-web"
  }
}

You sort of have to think of this data store as a central place to manipulate the entire environment. Rather than thinking about individual nodes, think of configuring everything in one place. Accessing data from a data bag in a recipe, requires that you use:

data = data_bag_item('configs', 'example')
puts data['haproxy']['production_role'] # prints prod-web to stdout

From there you will have a nice hash with all of the JSON data in that data bag item.

Role Attributes

Again stored in JSON, Role Attributes come from your Role definitions. When you define a role, it will look like this:

{
    "name": "base",
    "default_attributes": {
      "cid": "example"
    },
    "json_class": "Chef::Role",
    "run_list": [
      "recipe[ldap::auth]",
      "recipe[ldap::sudoers]",
      "recipe[sshd]",
      "recipe[zabbix]",
      "recipe[clients::hosts]",
      "recipe[clients::hostname]",
      "recipe[postfix]"
    ],
    "description": "",
    "chef_type": "role",
    "override_attributes": {
    }
}

Don't let the JSON make you think creating new roles is difficult. Creating a role with a properly configured knife client is pretty straight forward:

# knife role create <some_role>

This will give you a basic role with no run_list and no default_attributes. All I do for this role (base), is define what recipes to run, and set the config ID that we use in various recipes. Accessing the data from these attributes is done the same way as the Node Attributes. Really we can just call them Attributes, but because they are technically stored in different places I wanted to cover them separately.

Node Attributes

If you're thinking that this is stored using JSON, you are correct. Each node, upon being bootstrapped, gets populated with a large data store. This contains everything from the contents of /etc/passwd, to the amount of bytes transmitted on eth0. Each time a node checks in with the server, this data gets updated. You can modify any data for a node directly using knife, and to give you an example of the extensive information stored by Chef, here's what the ec2 section looks like:

      "ec2": {
        "public_hostname": "ec2-174-129-1.compute-1.amazonaws.com",
        "placement_availability_zone": "us-east-1a",
        "block_device_mapping_root": "/dev/sda1",
        "instance_id": "i-d1535fbb",
        "instance_type": "m1.large",
        "local_ipv4": "10.204.39.249",
        "block_device_mapping_ephemeral0": "/dev/sdb",
        "reservation_id": "r-940419ff",
        "public_ipv4": "174.129.1.1",
        "local_hostname": "ip-10-204-39-249.ec2.internal",
        "kernel_id": "aki-2407f24d",
        "hostname": "ip-10-204-39-249.ec2.internal",
        "ami_id": "ami-6006f309",
        "userdata": null,
        "security_groups": [
          "web"
        ],
        "block_device_mapping_ami": "/dev/sda1",
        "ami_manifest_path": "(unknown)",
        "ami_launch_index": "0"
      },

Accessing any of these attributes from recipes can be done in the following manner:

# the node's name, ex: web1
node.name

# the AMI a node is running, ex: ami-6006f309
node.ec2.ami_id

# node's uptime, ex: "16 days 16 hours 35 minutes 23 seconds"
node.uptime

# distro, ex: "ubuntu"
node.platform

# version of php on the instance, ex: "5.3.2-1ubuntu4.5"
node.languages.php.version

# gcc version
node.languages.c.gcc.version

# number of packets sent/received 
node.network.interfaces.eth0.tx.packets
node.network.interfaces.eth0.rx.packets

As long as you have a node object, you can access any of this data via the object's methods. When you are in a recipe, you can use the variable node as it is defined automatically by Chef.

Further Reading

Tags: