Chef Server Deployment, Architectures, and Administration

Jacob Bills

June 06, 2020

In this tutorial you will learn how to build various chef server topologies and determine which one is best suited for your environment. Chef server can be deployed as standalone server, tiered with multiple frontends and single backend, or highly available with multiple frontends and multiple backends. The various topologies are available so that organizations with specific requirements like availability, scalability, and data security can be satisfied.

This blog is focused for the chef administrator and does not go into detail regarding development of cookbooks. We'll use the learn_chef_httpd cookbook to cover the fundamentals of administering and deploying cookbooks. If you have gone through the chef learning modules (which I highly recommend) you may already be familiar with this cookbook.

I'll be running the installations on CentOS 7. The commands and configs that are used will be the same regardless of the platform you are running.


In this blog


Overview of chef topologies

Capacity planning is important for electing an architecture to deploy. You should have a good idea of how many CCR/min (chef client runs) your deployment will be supporting when choosing your deployment model. Most deployments can be satisfied with a single chef server.

Standalone deployments
This style is best for development or testing environments where availability and scale is not typically a concern.
Tiered and Highly Available deployments
These deployments are good for production environments. Both models consist of a frontend and backend server(s) which some organizations may require. Generally, you should consider using these models for large scale deployments, for restricting access to data components to meet PCI DSS or HIPAA security requirements and regulations, or when you need to strategically deploy hosts across multiple physical networks.

Use the following sizes for your frontend and backend servers

Standalone deployment

Standalone deployments are by far the easiest to manage and deploy. 
#!/bin/bash
# download chef infra
wget https://packages.chef.io/files/stable/chef-server/13.1.13/el/7/chef-server-core-13.1.13-1.el7.x86_64.rpm
# install package
rpm -Uvh chef-server-core-13.1.13-1.el7.x86_64.rpm
# configure and accept license
chef-server-ctl reconfigure
# configure chef host firewall
firewall-cmd --permanent --add-service=http
firewall-cmd --permanent --add-service=https
firewall-cmd --reload
Use the health check url to verify your chef server is ready to use. 
[root@chef-server ~]# curl https://chef.example.com/_status -k
{"status":"pong","upstreams":{"chef_solr":"pong","chef_sql":"pong","chef_index":"pong","oc_chef_action":"pong","oc_chef_authz":"pong"},"keygen":{"keys":10,"max":10,"max_workers":1,"cur_max_workers":1,"inflight":0,"avail_workers":1,"start_size":0},"indexing":{"mode":"rabbitmq","indexer_message_queue_length":0},"analytics_queue":{"queue_at_capacity":false,"dropped_since_last_check":0,"max_length":10000,"last_recorded_length":0,"total_dropped":0,"check_count":146,"mailbox_length":0}}

Tiered deployment

This topology consists of a single backend server and any number of frontend servers. Start by installing chef on the backend server 
#!/bin/bash
# download chef infra
wget https://packages.chef.io/files/stable/chef-server/13.1.13/el/7/chef-server-core-13.1.13-1.el7.x86_64.rpm
# install package
rpm -Uvh chef-server-core-13.1.13-1.el7.x86_64.rpm
Next, open /etc/opscode/chef-server.rb and apply the following configurations. Make sure to change the device name to which the virtual ip address will bind to. This is typically the public interface of the server 
topology "tier"

server "chef-backend-01.example.com",
  :ipaddress => "X.X.X.X",
  :role => "backend",
  :bootstrap => true

backend_vip "chef-backend.example.com",
  :ipaddress => "Y.Y.Y.Y",
  :device => "eth0"

server "chef-frontend-01.example.com",
  :ipaddress => "Z.Z.Z.Z",
  :role => "frontend"

#for each frontend add the following to config
#server "chef-frontend-02.example.com",
#  :ipaddress => "Z.Z.Z.Z",
#  :role => "frontend"

#frontend VIP
api_fqdn "chef.example.com"
Apply the configuration to chef backend server and configure host firewall 
#!/bin/bash
# apply configuration
chef-server-ctl reconfigure
# configure chef backend host firewall
firewall-cmd --permanent --add-port=15672/tcp
firewall-cmd --permanent --add-port=5432/tcp
firewall-cmd --permanent --add-port=443/tcp
firewall-cmd --permanent --add-port=16379/tcp
firewall-cmd --permanent --add-port=4321/tcp
firewall-cmd --permanent --add-port=5672/tcp
firewall-cmd --permanent --add-port=9680/tcp
firewall-cmd --permanent --add-port=80/tcp
firewall-cmd --permanent --add-port=9683/tcp
firewall-cmd --permanent --add-port=8983/tcp
firewall-cmd --reload
Now its time to configure the frontend server(s). Copy all of the contents of /etc/opscode from the backend server to /etc/opscode on each frontend server. Apply the configurations to each frontend server. 
#!/bin/bash
# apply configuration
chef-server-ctl reconfigure
# configure chef frontend host firewall
firewall-cmd --permanent --add-service=http
firewall-cmd --permanent --add-service=https
firewall-cmd --reload
Use the health check url to verify your chef server is ready to use. 
[root@chef-frontend-01 ~]# curl https://chef.example.com/_status -k
{"status":"pong","upstreams":{"chef_solr":"pong","chef_sql":"pong","chef_index":"pong","oc_chef_action":"pong","oc_chef_authz":"pong"},"keygen":{"keys":10,"max":10,"max_workers":1,"cur_max_workers":1,"inflight":0,"avail_workers":1,"start_size":0},"indexing":{"mode":"rabbitmq","indexer_message_queue_length":0},"analytics_queue":{"queue_at_capacity":false,"dropped_since_last_check":0,"max_length":10000,"last_recorded_length":0,"total_dropped":0,"check_count":146,"mailbox_length":0}}

Highly available deployment

This topology consists of 3 backend servers and any number of frontend servers. Start by installing chef backend on each backend server and configuring the backend host firewall. 
#!/bin/bash
# download chef backend
wget https://packages.chef.io/files/stable/chef-backend/2.1.0/el/7/chef-backend-2.1.0-1.el7.x86_64.rpm
# install package
rpm -Uvh chef-backend-2.1.0-1.el7.x86_64.rpm
# configure host firewall for backend
firewall-cmd --permanent --add-port=2379/tcp
firewall-cmd --permanent --add-port=2380/tcp
firewall-cmd --permanent --add-port=5432/tcp
firewall-cmd --permanent --add-port=7331/tcp
firewall-cmd --permanent --add-port=9200-9400/tcp
firewall-cmd --reload
Next, open /etc/chef-backend/chef-backend.rb and apply the following configuration to each backend server 
publish_address 'external_IP_address_of_backend_server'
If any of the backends or frontends are in different networks you will also need to add all the network addresses in CIDR notation that are participating in the chef cluster. 
postgresql.md5_auth_cidr_addresses = ["samehost", "samenet", "{{NET-1_IN_CIDR}}", ..., "{{NET-N_IN_CIDR}}"]
Apply the configuration to one of the backend servers in the cluster 
#!/bin/bash
# create the backend cluster
chef-backend-ctl create-cluster
Now copy /etc/chef-backend/chef-backend-secrets.json from this server to all the other backend servers. This file is required for joining the other backend servers to the cluster. You should carefully handle this file as it contains secure credentials and delete it from all the other backend servers once finished joining them to the cluster. 
#!/bin/bash
# copy the secrets file to other backend servers
scp /etc/chef-backend/chef-backend-secrets.json user@backend-host-x:/home/user
Instruct the other backend nodes to join the cluster. 
#!/bin/bash
chef-backend-ctl join-cluster {{IP_FIRST_BACKEND_SERVER}} -s /home/user/chef-backend-secrets.json
You can check the status of the backend cluster from any backend node with the following commands. If the cluster is setup properly you should see output similar to the following example. 
[root@chef-backend-01 ~]# chef-backend-ctl status
Service        Local Status        Time in State  Distributed Node Status
leaderl        running (pid 3533)  9d 3h 19m 4s   leader: 1; waiting: 0; follower: 2; total: 3
epmd           running (pid 1065)  9d 3h 31m 2s   status: local-only
etcd           running (pid 1056)  9d 3h 31m 2s   health: green; healthy nodes: 3/3
postgresql     running (pid 3596)  9d 3h 19m 3s   leader: 1; offline: 0; syncing: 0; synced: 2
elasticsearch  running (pid 1062)  9d 3h 31m 3s   state: green; nodes online: 3/3

System  Local Status                                          Distributed Node Status
disks   /var/log/chef-backend: OK; /var/opt/chef-backend: OK  health: green; healthy nodes: 3/3


[root@chef-backend-01 ~]# chef-backend-ctl cluster-status
Name             IP              GUID                              Role      PG        ES          Blocked      Eligible
chef-backend-01  192.168.56.101  5c0742fa952550634ce47777824b6589  leader    leader    master      not_blocked  true
chef-backend-02  192.168.56.104  9d8ddf6c2b347f497bcfd15154c03a9a  follower  follower  not_master  not_blocked  true
chef-backend-03  192.168.56.103  6dc8f50f58c0e82799d842f04c8e54fb  follower  follower  not_master  not_blocked  true
After you have verified the backend cluster is successfully up and running, from any backend server use chef-backend-ctl to generate configs for each frontend server. 
#!/bin/bash
chef-backend-ctl gen-server-config chef-frontend-01.example.com -f chef-server.rb.FE1
Copy the generated configurations to each of their respective frontend servers 
#!/bin/bash
scp chef-server.rb.FE1 user@chef-frontend-01:/home/user
Now install chef and configure the host firewall for each frontend server 
#!/bin/bash
# download chef infra
wget https://packages.chef.io/files/stable/chef-server/13.1.13/el/7/chef-server-core-13.1.13-1.el7.x86_64.rpm
# install package
rpm -Uvh chef-server-core-13.1.13-1.el7.x86_64.rpm
# configure chef host firewall
firewall-cmd --permanent --add-service=http
firewall-cmd --permanent --add-service=https
firewall-cmd --reload
Apply the configuration to the first frontend server 
#!/bin/bash
# rename and copy the configuration to /etc/opscode
cp chef-server.rb.FE1 /etc/opscode/chef-server.rb
# apply the configuration
chef-server-ctl reconfigure
Copy /etc/opscode/private-chef-secrets.json from the first frontend server to each subsequent frontend servers /etc/opscode directory. It's worth mentioning here that when you upgrade a frontend server it creates a file /var/opt/opscode/upgrades/migration-level that you must also copy to other frontends before running the configuration. 
#!/bin/bash
# create the upgrades directory on new frontends.
mkdir -p /var/opt/opscode/upgrades/
# create an empty bootstrapped file
touch /var/opt/opscode/bootstrapped
# apply configuration
chef-server-ctl reconfigure
Use the health check url to verify your chef server is ready to use. Note that the health check response is slightly different from a standalone or tiered deployment. 
[root@chef-frontend-01 ~]# curl https://chef.example.com/_status -k
{"status":"pong","upstreams":{"chef_elasticsearch":"pong","chef_sql":"pong","chef_index":"pong","oc_chef_authz":"pong","data_collector":"pong"},"keygen":{"keys":10,"max":10,"max_workers":1,"cur_max_workers":1,"inflight":0,"avail_workers":1,"start_size":0},"indexing":{"mode":"batch"}}

Command line chef basics

When your chef installation is complete, the first thing you will want to do is create an administrative user and associate that user to an organization. Run these commands from your standalone chef instance or one of your frontend servers. 
#!/bin/bash
# create a new user
chef-server-ctl user-create jbills Jacob Bills jbills@bustedware.com 'password' --filename jbills.pem
# create a new organization and associate the user
chef-server-ctl org-create bwllc 'Bustedware LLC' --association_user jbills --filename bwllc-validator.pem
The knife command line tool is available in the ChefDK package and will allow you to manage: You will need the users private key file on the workstation administering the chef installation. By default knife will look for a config.rb (previously known as knife.rb in versions < 12) configuration in the .chef folder of your current working directory. It will recursively search up directories for .chef if one is not available in your current working directory.

First, install ChefDK on your workstation 
#!/bin/bash
# download chefdk
wget https://packages.chef.io/files/stable/chefdk/4.7.73/el/7/chefdk-4.7.73-1.el7.x86_64.rpm
# install chefdk
rpm -Uvh chefdk-4.7.73-1.el7.x86_64.rpm
# create .chef configuration directory
mkdir .chef
# create cookbooks directory
mkdir cookbooks
# download the learn_chef_httpd cookbook
cd cookbooks
git clone https://github.com/learn-chef/learn_chef_httpd
Copy the users private key and create config.rb in the .chef folder. \#{current_dir} refers to the location of .chef folder when running knife. 
current_dir = File.dirname(__FILE__)
log_level                 :info
log_location              STDOUT
node_name                 "jbills"
client_key                "\#{current_dir}/jbills.pem"
chef_server_url           "https://chef.example.com/organizations/bwllc"
cookbook_path             ["\#{current_dir}/../cookbooks"]
Before interacting with the chef server for the first time you will need to fetch the servers SSL certificate and add it to knifes trusted ssl certs directory. You can do this easily with the following command 
#!/bin/bash
knife ssl fetch
Bootstrapping a node installs the chef client on the targeted server which is used for running cookbooks. You can use ssh key authentication or username/password authentication when bootstrapping nodes. Run the following commands to bootstrap a server and upload a cookbook 
#!/bin/bash
# bootstrap with ssh key authentication
knife bootstrap chef-target.example.com -N chef-target
# bootstrap with username/password authentication
knife bootstrap chef-target.example.com -U USERNAME -P PASSWORD -N chef-target
# upload the learn_chef_httpd cookbook
knife cookbook upload learn_chef_httpd
Familiarize yourself with some knife commands. Here's a few of the most common commands to get you started 
#!/bin/bash
# check ssl certificate is in trusted certs
knife ssl check
# get list of cookbooks
knife cookbook list
# get list of nodes
knife node list
Lets add the learn_chef_httpd cookbook to the run list for the node that we just bootstrapped. Once the run list is configured you can either execute chef client directly from that node or use knife ssh to execute it remotely from your workstation. 
#!/bin/bash
# configure the nodes run list
knife node run_list add chef-target.example.com 'recipe[learn_chef_httpd]'
# execute chef client on that node with knife ssh
knife ssh 'name:chef-target.example.com' 'sudo chef-client'
The simple web server should now be running on the target server.

Chef manage web interface

Chef manage has been officially deprecated and is no longer under active development. It's still worth mentioning though because it's an easy to install management interface that can interact with your chef server from your web browser. Chef manage is particularly helpful for beginners that are still learning how to use their chef server. Advanced chef administrators can interact with chef server directly using ChefDK from their terminal.

The installation takes place on standalone instances and frontend instances of your chef server and uses the same series of commands regardless of the type of installation you are running. 
#!/bin/bash
# install with internet
chef-server-ctl install chef-manage

# isntall without internet
# wget https://packages.chef.io/files/stable/chef-manage/2.5.16/el/7/chef-manage-2.5.16-1.el7.x86_64.rpm
# rpm -Uvh chef-manage-2.5.16-1.el7.x86_64.rpm

# reconfigure chef server
chef-server-ctl reconfigure
# apply initial configuration for chef manage
chef-manage-ctl reconfigure
Chef manage uses email for password resets, new user invitations, failover notifications, and failed job notifications. Configuring a local mail transfer agent is out of scope for this blog, therefore, you should already have a new user created from the command line to access your chef manage web interface.

VLOG