About Policyfiles

Why Policyfiles?

Policyfiles make it easier to test and promote code safely with a simpler interface. Using a Policyfile improves the user experience and resolves real-world problems that some workflows built around Chef Infra must deal with. The following sections discuss in more detail some of the good reasons to use Policyfile, including:

• Focus the workflow on the entire system
• Safer development workflows
• Less expensive computation
• Code visibility
• Role mutability
• Cookbook mutability
• Replaces Berkshelf and the environment cookbook pattern

Focused System Workflows

The knife command line tool maps closely to the Chef Infra Server API and the objects defined by it: roles, environments, run-lists, cookbooks, data bags, nodes, and so on. Chef Infra Client assembles these pieces at run-time and configures a host to do useful work.

Policyfile focuses that workflow onto the entire system, rather than the individual components. For example, Policyfile describes whole systems, whereas each individual revision of the Policyfile.lock.json file uploaded to the Chef Infra Server describes a part of that system, inclusive of roles, environments, cookbooks, and the other Chef Infra Server objects necessary to configure that part of the system.

Safer Workflows

Policyfile encourages safer workflows by making it easier to publish development versions of cookbooks to the Chef Infra Server without the risk of mutating the production versions and without requiring a complicated versioning scheme to work around cookbook mutability issues. Roles are mutable and those changes are applied only to the nodes specified by the policy. Policyfile does not require any changes to your normal workflows. Use the same repositories you are already using, the same cookbooks, and workflows. Policyfile will prevent an updated cookbook or role from being applied immediately to all machines.

Code Visibility

When running Chef Infra without a Policyfile, the exact set of cookbooks that are applied to a node is defined by:

• The node’s run_list property
• Any roles that are present in the node’s run-list or recursively included by those roles
• The environment, which may restrict the set of valid cookbook versions for a node based on a variety of constraint operators
• Dependencies, as defined by each cookbook’s metadata
• Dependency resolution picks the “best” set of cookbooks that meet dependency and environment criteria

These conditions are re-evaluated every time Chef Infra Client runs, which can make it harder to know which cookbooks will be run by Chef Infra Client or what the effects of updating a role or uploading a new cookbook will be.

Policyfile simplifies this behavior by computing the cookbook set on the workstation, and then producing a readable document of that solution: a Policyfile.lock.json file. This pre-computed file is uploaded to the Chef Infra Server, and is then used in each Chef Infra Client run that is managed by that particular policy name and policy group.

Less Expensive Computation

When running Chef Infra without Policyfile, the Chef Infra Server loads dependency data for all known versions of all known cookbooks, and then runs an expensive computation to determine the correct set.

Policyfile moves this computation to the workstation, where it is done less frequently.

Role and Environment Mutability

When running Chef Infra without Policyfile roles and environments are global objects. Changes to roles and environments are applied immediately to any node that contains that role in its run-list or belong to a particular environment. This can make it hard to update roles and environments and in some use cases discourages using them at all.

Policyfile effectively replaces roles and environments. Policyfile files are versioned automatically and new versions are applied to systems only when promoted.

Cookbook Mutability

When running Chef without Policyfile, existing versions of cookbooks are mutable. This is convenient for many use cases, especially if users upload in-development cookbook revisions to the Chef Infra Server. But this sometimes creates issues that are similar to role mutability by allowing those cookbook changes to be applied immediately to nodes that use that cookbook. Users account for this by rigorous testing processes to ensure that only fully integrated cookbooks are ever published. This process does enforce good development habits, but at the same time it should not be a required part of a workflow that ends with publishing an in-development cookbook to the Chef Infra Server for testing against real nodes Policyfile solves this issue by using a cookbook publishing API for the Chef Infra Server that does not provide cookbook mutability. Name collisions are prevented by storing cookbooks by name and an opaque identifier that is computed from the content of the cookbook itself.

For example, name/version collisions can occur when users temporarily fork an upstream cookbook. Even if the user contributes their change and the maintainer is responsive, there may be a period of time where the user needs their fork to make progress. This situation presents a versioning dilemma: if the user does not update their own version, they must overwrite the existing copy of that cookbook on the Chef Infra Server, wheres if they do update the version number it might conflict with the version number of the future release of that upstream cookbook.

Opaque IDs

The opaque identifier that is computed from the content of a cookbook is the only place where an opaque identifier is necessary. When working with Policyfile, be sure to:

• Use the same names and version constraints as normal in the Policyfile.rb file
• Use the same references to cookbooks pulled from Chef Supermarket
• Use the same branch, tag, and revision patterns for cookbooks pulled from git
• Use the same paths for cookbooks pulled from disk

Extra metadata about the cookbook is stored and included in Chef Infra Server API responses and in the Policyfile.lock.json file, including the source of a cookbook (Chef Supermarket, git, local disk, etc.), as well as any upstream identifiers, such as git revisions. For cookbooks that are loaded from the local disk that are in a git repo, the upstream URL, current revision ID, and the state of the repo are stored also.

The opaque identifier is mostly behind the scenes and is only visible once published to the Chef Infra Server. Cookbooks that are uploaded to the Chef Infra Server may have extended version numbers such as 1.0.0-dev.

Environment Cookbooks

Policyfile replaces the environment cookbook pattern that is often required by Berkshelf, along with a dependency solver and fetcher. That said, Policyfile does not replace all Berkshelf scenarios.

Knife Commands

The following knife commands used to set the policy group and policy name on the Chef Infra Server. For example:

knife node policy set test-node 'test-policy-group-name' 'test-policy-name'

Policyfile.rb

Syntax

A Policyfile.rb is a Ruby file in which run-list and cookbook locations are specified. The syntax is as follows:

name "name"
run_list "ITEM", "ITEM", ...
default_source :SOURCE_TYPE, *args
cookbook "NAME" [, "VERSION_CONSTRAINT"] [, SOURCE_OPTIONS]
ATTRIBUTE_TYPE['attribute'] = 'value'

Settings

A Policyfile.rb file may contain the following settings:

name "name"

Required. The name of the policy. Use a name that reflects the purpose of the machines against which the policy will run, such as “application server”, “chat server”, “load balancer”, and so on.

run_list "ITEM", "ITEM", ...

Required. The run-list Chef Infra Client will use to apply the policy to one (or more) nodes.

default_source :SOURCE_TYPE, *args

The location in which any cookbooks not specified by cookbook are located.

Possible values for :SOURCE_TYPE are:

• :artifactory
• :chef_repo
• :chef_server
• :supermarket

:artifactory

Pulls cookbooks from an Artifactory server. Requires either artifactory_api_key to be set in config.rb or ARTIFACTORY_API_KEY to be set in your environment.

For example, default_source :artifactory, "https://artifactory.example/api/chef/my-supermarket".

:chef_repo

Pulls cookbooks from a monolithic cookbook repository. This may be a path to the top-level of a cookbook repository or to the /cookbooks directory within that repository.

For example, default_source :chef_repo, "path/to/repo".

:chef_server

Pulls cookbooks from the Chef Infra Server.

For example, default_source :chef_server, "https://chef-server.example/organizations/example".

:supermarket

Pulls cookbooks from the public Chef Supermarket or a private Chef Supermarket.

By default :supermarket pulls cookbooks from the public Chef Supermarket. For example, default_source :supermarket.

Specify the Supermarket URL to pull cookbooks from a private Supermarket. For example, default_source :supermarket, "https://supermarket-name.example".

You can specify multiple cookbook sources. For example from the public Chef Supermarket and a monolithic repository:

default_source :supermarket
default_source :chef_repo, 'path/to/repo'

or from both a public and private Chef Supermarket:

default_source :supermarket
default_source :supermarket, 'https://supermarket.example'

Note

If a run-list or any dependencies require a cookbook that is present in more than one source, be explicit about which source is preferred. This will ensure that a cookbook is always pulled from an expected source. For example, an internally-developed cookbook named chef-client will conflict with the public chef-client cookbook that is maintained by Chef. To specify a named source for a cookbook:

default_source :supermarket
default_source :supermarket, 'https://supermarket.example' do |s|
  s.preferred_for 'chef-client'
end

List multiple cookbooks on the same line:

default_source :supermarket
default_source :supermarket, 'https://supermarket.example' do |s|
  s.preferred_for 'chef-client', 'nginx', 'mysql'
end

cookbook "NAME" [, "VERSION_CONSTRAINT"] [, SOURCE_OPTIONS]

Add cookbooks to the policy, specify a version constraint, or specify an alternate source location, such as Chef Supermarket. For example, add a cookbook:

cookbook 'apache2'

Specify a version constraint:

run_list 'jenkins::master'

# Restrict the jenkins cookbook to version 2.x, greater than 2.1
cookbook 'jenkins', '~> 2.1'

Specify an alternate source:

cookbook 'my_app', path: 'cookbooks/my_app'

or:

cookbook 'mysql', github: 'opscode-cookbooks/mysql', branch: 'master'

or:

cookbook 'chef-ingredient', git: 'https://github.com/chef-cookbooks/chef-ingredient.git', tag: 'v0.12.0'

named_run_list "NAME", "ITEM1", "ITEM2", ...

Specify a named run-list to be used as an alternative to the override run-list. This setting should be used carefully and for specific use cases, like running a small set of recipes to quickly converge configuration for a single application on a host or for one-time setup tasks. For example:

named_run_list :update_app, 'my_app_cookbook::default'

include_policy "NAME", *args

Specify a policyfile lock to be merged with this policy. Chef Workstation supports pulling this lock from a local or remote file, from a Chef Infra Server, or from a git repository. When the policyfile lock is included, its run-list will appear before the current policyfile’s run-list. This setting requires that the solved cookbooks appear as-is from the included policyfile lock. If conflicting attributes or cookbooks are provided, an error will be presented. See RFC097 for the full specifications of this feature.

Pull the policyfile lock from ./NAME.lock.json:

include_policy 'NAME', path: '.'

Pull the policyfile lock from ./foo.lock.json.

include_policy 'NAME', path: './foo.lock.json'

Pull the policyfile lock foo.lock.json from the example/foo Git repository on the git.example.com Git server.

include_policy 'NAME', git: 'https://git.example.com/example/foo', path: 'foo.lock.json'

Pull the policyfile lock from ./bar.lock.json with revision ID ‘revision1’.

include_policy 'NAME', policy_revision_id: 'revision1', path: './bar.lock.json'

Pull the policyfile lock from a remote server https://internal.example.com/foo.lock.json.

include_policy 'NAME', remote: 'https://internal.example.com/foo.lock.json'

Pull the policyfile lock from a remote server https://internal.example.com/bar.lock.json and with revision ID ‘revision1’.

include_policy 'NAME', policy_revision_id: 'revision1', remote: 'https://internal.example.com/foo.lock.json'

Pull the policy NAME with revision ID revision1 from the http://chef-server.example Chef Infra Server:

include_policy 'NAME', policy_revision_id: 'revision1', server: 'http://chef-server.example'

Pull the policy foo with revision ID revision1:

include_policy 'NAME', policy_name: 'foo', policy_revision_id: 'revision1', server: 'http://chef-server.example'

Pull and lock the current revision for policy foo in policy group prod:

include_policy 'NAME', policy_name: 'foo', policy_group: 'prod', server: 'http://chef-server.example'

ATTRIBUTE_TYPE['attribute'] = 'value'

Specify one or more attributes to be included with the policy. This is similar to defining attributes using roles.

Possible values for ATTRIBUTE_TYPE are:

• default
• override

default

A default attribute is automatically reset at the start of every Chef Infra Client run and has the lowest attribute precedence.

For example:

default['attribute'] = 'value'
default['attribute']['level2'] = 'another_value'

override

An override attribute is automatically reset at the start of every Chef Infra Client run and has a higher attribute precedence than a default attribute.

override['attribute'] = 'value'
override['attribute']['level2'] = 'another_value'

Attribute hoisting allows you to define attributes by policy group.

Use the following syntax to define policy group-specific attributes:

ATTRIBUTE_TYPE['POLICY_GROUP']['attribute'] = 'value'

where:

• ATTRIBUTE_TYPE is either default or override as described above.
• POLICY_GROUP is a user-defined policy group, such as “dev”, “test” “staging”, or “production”.

In the following example, the value of default['attribute'] is set to either dev_value or prod_value depending on the policy group.

default['dev']['attribute'] = 'dev_value'
default['prod']['attribute'] = 'prod_value'

Example

For example:

name 'jenkins-master'
run_list 'java', 'jenkins::master', 'recipe[policyfile_demo]'
default_source :supermarket, 'https://mysupermarket.example'
cookbook 'policyfile_demo', path: 'cookbooks/policyfile_demo'
cookbook 'jenkins', '~> 8.2'
cookbook 'mysql', github: 'sous-chefs/mysql', branch: 'master'
default['stage']['mysql']['install_s3'] = 'https://s3-eu-west-1.amazonaws.com/example/stage/file.rpm'
default['prod']['mysql']['install_s3'] = 'https://s3-eu-west-1.amazonaws.com/example/prod/file.rpm'

client.rb Settings

The following settings may be configured in the client.rb file for use with Policyfile:

named_run_list

The run-list associated with a policy file.

policy_group

The name of a policy group that exists on the Chef Infra Server. policy_name must also be specified.

policy_name

The name of a policy, as identified by the name setting in a Policyfile.rb file. policy_group must also be specified.

use_policyfile

Chef Infra Client automatically checks the configuration, node JSON, and the stored node on the Chef Infra Server to determine if Policyfile files are being used, and then automatically updates this flag. Default value: false.

knife bootstrap

A node may be bootstrapped to use Policyfile files. Use the following options as part of the bootstrap command:

--policy-group POLICY_GROUP

The name of a policy group that exists on the Chef Infra Server.

--policy-name POLICY_NAME

The name of a policy, as identified by the name setting in a Policyfile.rb file.

For a customized bootstrap process, add policy_name and policy_group to the first-boot JSON file that is passed to Chef Infra Client.

knife search

The policy_name and policy_group settings for a node are stored as searchable attributes and as such are available when using a fuzzy matching search pattern. For example: knife search dev will return nodes that are part of the dev policy group.

Test w/Kitchen

Kitchen may be used to test Policyfile files. Add the following to kitchen.yml:

provisioner:
  name: chef_zero

A named run-list may be used for each suite:

suites:
  - name: client
    provisioner:
      named_run_list: test_client_recipe
  - name: server
    provisioner:
      named_run_list: test_server_recipe

or globally:

provisioner:
  name: chef_zero
  named_run_list: integration_test_run_list

or testing with policies for each suite, once the Policyfile files are available in your repo:

suites:
   - name: defaultmega
      provisioner:
         policyfile: policies/default.rb
   - name: defaultultra
      provisioner:
         policyfile: policies/defaulttwo.rb

chef Commands

chef clean-policy-cookbooks

Use the chef clean-policy-cookbooks subcommand to delete cookbooks that are not used by Policyfile files. Cookbooks are considered unused when they are not referenced by any policy revisions on the Chef Infra Server.

Syntax

This subcommand has the following syntax:

chef clean-policy-cookbooks (options)

Options

This subcommand has the following options:

-c CONFIG_FILE, --config CONFIG_FILE

The path to the knife configuration file.

-D, --debug

Enable stack traces and other debug output. Default value: false.

-h, --help

Show help for the command.

-v, --version

The Chef Infra Client version.

chef clean-policy-revisions

Syntax

This subcommand has the following syntax:

chef clean-policy-revisions (options)

Options

This subcommand has the following options:

-c CONFIG_FILE, --config CONFIG_FILE

The path to the knife configuration file.

-D, --debug

Enable stack traces and other debug output. Default value: false.

-h, --help

Show help for the command.

-v, --version

The Chef Infra Client version.

chef delete-policy

Syntax

This subcommand has the following syntax:

chef delete-policy POLICY_NAME (options)

Options

This subcommand has the following options:

-c CONFIG_FILE, --config CONFIG_FILE

The path to the knife configuration file.

-D, --debug

Enable stack traces and other debug output. Default value: false.

-h, --help

Show help for the command.

-v, --version

The Chef Infra Client version.

chef delete-policy-group

Syntax

This subcommand has the following syntax:

chef delete-policy-group POLICY_GROUP (options)

Options

This subcommand has the following options:

-c CONFIG_FILE, --config CONFIG_FILE

The path to the knife configuration file.

-D, --debug

Enable stack traces and other debug output. Default value: false.

-h, --help

Show help for the command.

-v, --version

The Chef Infra Client version.

chef diff

Syntax

This subcommand has the following syntax:

chef diff POLICY_FILE --head | --git POLICY_GROUP | POLICY_GROUP...POLICY_GROUP (options)

Options

This subcommand has the following options:

-c CONFIG_FILE, --config CONFIG_FILE

The path to the knife configuration file.

-D, --debug

Enable stack traces and other debug output. Default value: false.

-g GIT_REF, --git GIT_REF

Compare the specified git reference against the current revision of a Policyfile.lock.json file or against another git reference.

-h, --help

Show help for the command.

--head

A shortcut for chef diff --git HEAD. When a git-specific flag is not provided, the on-disk Policyfile.lock.json file is compared to one on the Chef Infra Server or (if a Policyfile.lock.json file is not present on-disk) two Policyfile.lock.json files in the specified policy group on the Chef Infra Server are compared.

--[no-]pager

Use --pager to enable paged output for a Policyfile.lock.json file. Default value: --pager.

-v, --version

The Chef Infra Client version.

Examples

Compare current lock to latest commit on latest branch

chef diff --git HEAD

Compare current lock with latest commit on master branch

chef diff --git master

Compare current lock to specified revision

chef diff --git v1.0.0

Compare lock on master branch to lock on revision

chef diff --git master...dev

Compare lock for version with latest commit on master branch

chef diff --git v1.0.0...master

Compare current lock with latest lock for policy group

chef diff staging

Compare locks for two policy group

chef diff production...staging

chef export

Syntax

This subcommand has the following syntax:

chef export POLICY_FILE DIRECTORY (options)

Options

This subcommand has the following options:

-a, --archive

Export an archive as a tarball, instead as a directory. Default value: false.

-D, --debug

Enable stack traces and other debug output. Default value: false.

-f, --force

Remove the contents of the destination directory if that directory is not empty. Default value: false.

-h, --help

Show help for the command.

-v, --version

The Chef Infra Client version.

chef generate policyfile

Syntax

This subcommand has the following syntax:

chef generate policyfile POLICY_NAME (options)

Options

This subcommand has the following options:

-h, --help

Show help for the command.

-v, --version

The Chef Infra Client version.

chef generate repo

Syntax

This subcommand has the following syntax:

chef generate repo REPO_NAME (options)

Options

This subcommand has the following options:

-h, --help

Show help for the command.

-p, --policy-only

Create a repository that does not store cookbook files, only Policyfile files.

-P, --policy

Use Policyfile instead of Berkshelf.

-r, --roles

Create directories for /roles and /environments instead of creating directories for Policyfile.

-v, --version

The Chef Infra Client version.

chef install

Syntax

This subcommand has the following syntax:

chef install POLICY_FILE (options)

Options

This subcommand has the following options:

-D, --debug

Enable stack traces and other debug output.

-h, --help

Show help for the command.

-v, --version

The Chef Infra Client version.

Policyfile.lock.json

When the chef install command is run, Chef Workstation caches any necessary cookbooks and emits a Policyfile.lock.json file that describes:

• The versions of cookbooks in use
• A hash of cookbook content
• The source for all cookbooks
• Attributes included with the Policyfile

A Policyfile.lock.json file is associated with a specific policy group, which means it is associated with one (or more) nodes that use the same revision of a given policy.

A Policyfile.lock.json file is similar to:

{
  "revision_id": "288ed244f8db8bff3caf58147e840bbe079f76e0",
  "name": "jenkins",
  "run_list": [
    "recipe[java::default]",
    "recipe[jenkins::master]",
    "recipe[policyfile_demo::default]"
  ],
  "cookbook_locks": {
    "policyfile_demo": {
      "version": "0.1.0",
      "identifier": "f04cc40faf628253fe7d9566d66a1733fb1afbe9",
      "dotted_decimal_identifier": "67638399371010690.23642238397896298.25512023620585",
      "source": "cookbooks/policyfile_demo",
      "cache_key": null,
      "scm_info": null,
      "source_options": {
        "path": "cookbooks/policyfile_demo"
      }
    },
  "java": {
    "version": "1.24.0",
    "identifier": "4c24ae46a6633e424925c24e683e0f43786236a3",
    "dotted_decimal_identifier": "21432429158228798.18657774985439294.16782456927907",
    "cache_key": "java-1.24.0-supermarket.chef.io",
    "origin": "https://supermarket.chef.io/api/v1/cookbooks/java/versions/1.24.0/download",
    "source_options": {
      "artifactserver": "https://supermarket.chef.io/api/v1/cookbooks/java/versions/1.24.0/download",
      "version": "1.24.0"
    }
  "default_attributes": {
    "audit": {
      "reporter": [
        "chef-server-automate",
        "cli"
      ]
    }
  },
  "override_attributes": {

  },

chef push

Syntax

This subcommand has the following syntax:

chef push POLICY_GROUP POLICY_FILE (options)

Options

This subcommand has the following options:

-c CONFIG_FILE, --config CONFIG_FILE

The path to the knife configuration file.

-D, --debug

Enable stack traces and other debug output.

-h, --help

Show help for the command.

-v, --version

The Chef Infra Client version.

chef push-archive

Syntax

This subcommand has the following syntax:

chef push-archive POLICY_GROUP ARCHIVE_FILE (options)

Options

This subcommand has the following options:

-c CONFIG_FILE, --config CONFIG_FILE

The path to the knife configuration file.

-D, --debug

Enable stack traces and other debug output. Default value: false.

-h, --help

Show help for the command.

-v, --version

The Chef Infra Client version.

chef show-policy

Syntax

This subcommand has the following syntax:

chef show-policy POLICY_NAME POLICY_GROUP (options)

Options

This subcommand has the following options:

-c CONFIG_FILE, --config CONFIG_FILE

The path to the knife configuration file.

-D, --debug

Enable stack traces and other debug output. Default value: false.

-h, --help

Show help for the command.

-o, --orphans

Show policy revisions that are not currently assigned to any policy group.

--[no-]pager

Use --pager to enable paged output for a Policyfile.lock.json file. Default value: --pager.

-v, --version

The Chef Infra Client version.

chef undelete

Use the chef undelete subcommand to recover a deleted policy or policy group. This command:

• Does not detect conflicts. If a deleted item has been recreated, running this command will overwrite it
• Does not include cookbooks that may be referenced by Policyfiles; cookbooks that are cleaned after running this command may not be fully restorable to their previous state
• Does not store access control data

Syntax

This subcommand has the following syntax:

chef undelete (options)

When run with no arguments, returns a list of available operations.

Options

This subcommand has the following options:

-c CONFIG_FILE, --config CONFIG_FILE

The path to the knife configuration file.

-D, --debug

Enable stack traces and other debug output.

-h, --help

Show help for the command.

-i ID, --id ID

Undo the delete operation specified by ID.

-l, --last

Undo the most recent delete operation.

--list

Default. Return a list of available operations.

-v, --version

The Chef Infra Client version.

chef update

Syntax

This subcommand has the following syntax:

chef update POLICY_FILE (options)

Options

This subcommand has the following options:

-a, --attributes

Update attributes. Default value: false.

-D, --debug

Enable stack traces and other debug output. Default value: false.

-h, --help

Show help for the command.

-v, --version

The Chef Infra Client version.