Skip to main content
SaltStack Support

Event-based Orchestration: An Overview

In this tutorial, you will be introduced to the basics of creating flexible, event-driven orchestration workflows with Salt.

Today is Not going to be Awesome

This is all FAKE NEWS!

Remember the Alamo!

Roses are Red, Violets are Blue
This is Our Finest Hour

And this is a lowly block of text that repeats itself over and over again. And this is a lowly block of text that repeats itself over and over again. And this is a lowly block of text that repeats itself over and over again. And this is a lowly block of text that repeats itself over and over again. And this is a lowly block of text that repeats itself over and over again. And this is a lowly block of text that repeats itself over and over again. 

OBJECTIVES

You will learn to:

  • Configure the “Salt-API” service (rest_cherrypy) to accept calls (and POST data) from external systems
  • Create an Orchestration State that accepts inline pillar data to modify runtime behavior

  • leverages execution calls in Jinja to access external data

    • Create a Reactor State to pass POST data as inline pillar data to Orchestration States

    • Connect each of these components to demonstrate an end-to-end provisioning example

Part 1: Configuring Salt API (rest_cherrypy)
USE CASES FOR API CALLS

Salt includes a RESTful API called “rest_cherrypy” that enables you to interact fully with your Salt masters.  All the familiar tasks that you perform with Salt are available through the API interface.  The available interfaces include:

  • Local Client (publishing message to minions via the event bus)
  • Runner Client (invoking modules, such as orchestration, on the Salt master)
  • Wheel Client (managing keys on the Salt master)

Also, SaltStack Enterprise customers have access to an enhanced API, which offers even more functionality and integration with the Role Based Access Control (RBAC) features of SaltStack Enterprise.  To learn more, visit:

#Python Code

import whois

data = raw_input("Enter a domain: ")
w = whois.whois(data)

print w

https://saltstack.com/saltstack-enterprise/

INSTALLATION AND CONFIGURATION OF rest_cherrypy

To install and configure the rest_cherrypy interface, please perform the following steps.

Install Salt API

yum –y install salt-api

Create a User for Authenticating to the API

     useradd saltapi

     passwd saltapi

Use “saltapi” as the password for this lab exercise.

Please use a strong password in your environment.

Generate a TLS Certificate

salt-call --local tls.create_self_signed_cert

Please use a trusted certificate in your production environment.


Update Master Configuration (external_auth)

vi /etc/salt/master.d/external_auth.conf
pam:
    saltapi:
      - '@runner'
      - '@wheel'
      - .

Please remember to restrict permissions, as needed, for the security of your environment.  For more information, please visit:

https://docs.saltstack.com/en/latest/topics/eauth/

Update Master Configuration (rest_cherrypy)

vi /etc/salt/master.d/rest_cherrypy.conf
rest_cherrypy:
  port: 8000
  ssl_crt: /etc/pki/tls/certs/localhost.crt
  ssl_key: /etc/pki/tls/certs/localhost.key

Restart the Salt Master and Salt API Services

systemctl restart salt-master salt-api
USING rest_cherrypy

Now that the Salt API package is installed and configured, let’s start making calls through the rest_cherrypy interface.

Login to the Salt API

curl -sSk https://localhost:8000/login \
     -H 'Accept: application/x-yaml' \
     -d username=saltapi \
     -d password=saltapi \
     -d eauth=pam \
     -c /tmp/saltapi_cookie.txt

Notice the return data, showing you information about your session and your privileges within the interface.

Run a Remote Execution Function

curl -sSk https://localhost:8000 \
     -b /tmp/saltapi_cookie.txt \
     -H 'Accept: application/x-yaml' \
     -d client=local \
     -d tgt='*' \
     -d fun=test.version

Notice the response, indicating the installed version of the Salt minion.  This command, invoked by the API, would be just like running the following remote execution call:

thisRun a Remote Execution Function with an Argument

curl -sSk https://localhost:8000 \
     -b /tmp/saltapi_cookie.txt \
     -H 'Accept: application/x-yaml' \
     -d client=local \
     -d tgt='*' \
     -d fun='grains.item' \
     -d arg='os'

Notice the response, indicating the Operating System of the Salt minion.  This command, invoked by the API, would be just like running the following remote execution call:

re

Run a Remote Execution Function with an Argument (Returned as JSON)

curl -sSk https://localhost:8000 \
     -b /tmp/saltapi_cookie.txt \
     -H 'Accept: application/json' \
     -d client=local \
     -d tgt='*' \
     -d fun='grains.item' \
     -d arg='os'

Functionally, this call is identical to the previous call, however, the output has been returned as JSON.  This output would likely be much easier to parse for the calling interface.  This command, invoked by the API, would be just like running the following remote execution call:

salt \* grains.item os --out json
Part 2: Create and Test a Simple Orchestration State
AN OVERVIEW OF ORCHESTRATION STATES

Orchestration states are very similar to the configuration states that are executed on Salt minions.  They can call execution functions, initiate the execution of states (or highstates) on sets of targeted minions, and execute various runners.  A key difference between configuration states and orchestration states is that orchestration states are executed on the Salt master.  They often serve as a point of coordination for complex orchestration workflows and they can leverage all the same requisites you are used to from your configuration states.

CREATING A SIMPLE ORCHESTRATION STATE

Let’s create a very basic orchestration state.  This orchestration state will simply append some text to a log file.  We will call this state “hooktest” because we are going to add a webhook in the Salt API configuration to trigger the orchestration state.

Create a Directory for Orchestration States

mkdir /srv/salt/orch

Create the Hooktest Orchestration State

     vi /srv/salt/orch/hooktest.sls

hooktest_log:

  cmd.run:

    - name: date >> /tmp/hooktest.log

Test the Orchestration State

     salt-run state.orch orch.hooktest

     cat /tmp/hooktest.log

Notice that the current date has been appended to the log file.

CREATE A REACTOR TO TRIGGER THE ORCHESTRATION STATE

In this section, we will add a simple reactor state to trigger our hooktest orchestration state.  A call to the “webhook” interface will be used to trigger the reactor state.

Create a Directory for Reactor States

     mkdir /srv/reactor

Create the Hooktest Reactor State

vi /srv/reactor/hooktest.sls

hooktest_runner:

  runner.state.orch:

    - mods: orch.hooktest

Configure the Salt Master to Use the Reactor

vi /etc/salt/master.d/reactor.conf

reactor:

  - 'salt/netapi/hook/hooktest':

    - /srv/reactor/hooktest.sls

Update the rest_cherrypy Configuration to Disable Webhook Authentication

vi /etc/salt/master.d/rest_cherrypy.conf

rest_cherrypy:

  port: 8000

  ssl_crt: /etc/pki/tls/certs/localhost.crt

  ssl_key: /etc/pki/tls/certs/localhost.key

  webhook_disable_auth: True

Reason for this example: Not all calling interfaces allow specification of the required authentication parameters for Salt API calls.  To handle this, one may disable authentication and then pass an application key that they have generated.  This key may be validated against pillar data to authorize the action that is being triggered.

Additional Considerations: It is prudent to limit what devices may access the port that rest_cherry is listening on.  Also, hosting rest_cherrypy behind a traditional web server, like nginx or Apache, will offer finer controls (such as which SSL Ciphers may be used or other web server configuration options).

Restart the Salt Master and Salt API Services

     systemctl restart salt-master salt-api

Test the Webhook

curl -sSk https://localhost:8000/hook/hooktest \

     -X POST

cat /tmp/hooktest.log

Notice that the current date has been appended to the log file.  This is just like before, however this time it was initiated by a web-based call to Salt’s API interface.  With just this amount of knowledge about reactors and rest_cherrypy, you could easily imagine how various potential calling interfaces (such as monitoring systems or web-based self-service portals) could be leveraged to interact with your Salt environment.

MODIFY ORCHESTRATION STATE BEHAVIOR WITH INLINE PILLAR DATA

You may have a use case where you need to change the behavior of an orchestration state with parameters that are only known at runtime.  Salt configuration states and orchestration states can address this with inline pillar data. 

An example of this could be use of Salt in a Continuous Integration / Continuous Delivery (CICD) pipeline where a new version of a software application has been tagged.  A post-commit git hook then initiates a call to Salt’s API to automatically provision infrastructure and launch a test suite.

In this section, we will modify our orchestration state and our reactor state to leverage data that is provided at runtime.

Revise the Orchestration State to Accept Inline Pillar Data

vi /srv/salt/orch/hooktest.sls

{% set release = salt['pillar.get']('release', 'latest') %}

hooktest_log:

  cmd.run:

    - name: echo

UndefinedNameError: reference to undefined name 'release' (click for details)
Callstack:
    at (Enterprise_Solutions/Automation_and_Orchestration/Event-based_Orchestration:_An_Overview), /content/body/div[3]/div[7]/p[39]/span/span, line 1, column 2
>> /tmp/hooktest.log

Test the Revised Orchestration State

     salt-run state.orch orch.hooktest \

pillar='{"release": "v1.0"}'

     cat /tmp/hooktest.log

Notice that the orchestration state runs as before, but now the value of the “release” variable is being passed into the hooktest.log file.

Revise the Reactor State to Pass POST Data as Inline Pillar Data

vi /srv/reactor/hooktest.sls

{% set post_data = data.get('post', {}) %}

{% set release = post_data.get('RELEASE', 'latest') %}

hooktest_runner:

  runner.state.orch:

    - mods: orch.hooktest

    - kwarg:

        pillar:

          release:

UndefinedNameError: reference to undefined name 'release' (click for details)
Callstack:
    at (Enterprise_Solutions/Automation_and_Orchestration/Event-based_Orchestration:_An_Overview), /content/body/div[3]/div[7]/p[54]/span/span, line 1, column 2

Test the Revised Webhook

curl -sSk https://localhost:8000/hook/hooktest \

     -X POST \

     -d RELEASE=v1.1

cat /tmp/hooktest.log

Notice the following:

  • we are passing POST data through a web-based call to Salt’s API
  • the POST data is being captured with Jinja variables
  • the Jinja variables are being passed as inline pillar data to the orchestration state

Food for thought…

How could you leverage this approach in your environment to automate activities such as provisioning, notification, or self-healing via Salt?

Part 3: Create an Orchestration State
to Provision a Node in AWS with Inline Pillar Data

USE CASES FOR AUTOMATED PROVISIONING VIA ORCHESTRATION STATES

In our Professional Services work at SaltStack, we are often asked for guidance on how to automate provisioning actions across cloud environments.  Often, customers will have a Self-Service Portal that results in a System Administrator “automatically” getting an email to go provision a virtual machine.

By leveraging the techniques described above, we can truly automate this workflow by passing inline pillar data to an orchestration state, which provisions a virtual machine using the values provided at runtime.

Create the Orchestration State

vi /srv/salt/orch/create_vm.sls

 

{# Sample Invocation:salt-run state.orch orch.create_vm pillar='{"vm": {"hostname": "minion01", "role": 

["apache", "salt-minion"], "os": "rhel", "provider": "ec2_us_west_1", "az": "us-west-1b"}}' #}

{# Set minion_id of Salt Master with External Data Gathered during the Jinja Rendering Phase #}{% set saltmaster = salt['http.query']('http://169.254.169.254/latest/meta-data/public-ipv4')['body'] %}

{# Define variables for tracking #}{% set vm = salt['pillar.get']('vm', {}) %}

{# Provide sane defaults for optional values #}{% if not "role" in vm %}{% do vm.update({'role': ['salt-minion', ]}) %}{% endif %}

{# Provide Stateful Check #}{% set errors = [] %}
{% if "hostname" not in vm %}{% do errors.append("Please provide a hostname") %}{% endif %}
{% if "os" not in vm %}{% do errors.append("Please identify an OS") %}{% endif %}
{% if "provider" not in vm %}{% do errors.append("Please identify a provider to use") %}{% endif %}
{% if "az" not in vm %}{% do errors.append("Please identify an availability zone") %}{% endif %}

{# Preflight Check #}
{% if errors|length > 0 %}preflight_check:  test.configurable_test_state:    - name: Additional information required...    - changes: False    - result: False    - comment: "{{ errors|join(', ') }}"{% else %}preflight_check:  test.succeed_without_changes:    - name: "All parameters accepted"{% endif %}

{# Provision VM #}{% if opts['test'] == True %}provision_vm:  test.succeed_without_changes:    - name: "VM {{ vm.get('hostname') }} would have been provisioned"{% else %}provision_vm:  salt.runner:    - name: cloud.profile    - prof: {{ vm.get('provider') }}_{{ vm.get('os') }}    - instances:      - {{ vm.get('hostname') }}    - vm_overrides:        availability_zone: {{ vm.get('az') }}        minion:          master: {{ saltmaster }}        grains:          role:            {% for role in vm.get('role') %}            - {{ role }}            {% endfor %}    - require:      - test: preflight_check{% endif %}

Test the Orchestration State

salt-run state.orch orch.create_vm

pillar='{"vm": {"hostname": "minion01",

"role": ["apache", "salt-minion"], "os": "rhel",

"provider": "ec2_us_west_1", "az": "us-west-1b"}}'

Note: the command above is meant to be entered on a single line
(the wrapped text is for the readability of this guide).

Observe that the orchestration state calls the cloud runner to provision an AWS instance, using the parameters that you provided as vm_overrides.  Also note that execution modules may be called from within Jinja to gather additional data to be leveraged by the orchestration state.

For reference, the Salt Cloud Profile used for this example is:

vi /etc/salt/cloud.profiles.d/ec2_profiles.conf

ec2_us_west_1_rhel:

  provider: ec2_us_west_1

  image: ami-66eec506

  size: t2.micro

  ssh_interface: public_ips

  ssh_username: ec2-user

  script_args: stable 2017.7.2

Part 4: Create a Reactor State
to Invoke the Orchestration State
to Provision a Node in AWS with Inline Pillar Data

TYING IT ALL TOGETHER

In this last section, we will bring all of the pieces together to:

  • see a web-based call trigger a reactor
  • which passes POST data as inline pillar data
  • to an orchestration state
  • that passes vm_overrides to the cloud runner
  • to provision a minion with parameters provided at runtime

Create the Reactor State

vi /srv/reactor/create_vm.sls

{# Assign POST Data to Jinja Variables #}

{% set post_data = data.get('post', {}) %}

{% set hostname = post_data.get('HOSTNAME', '') %}

{% set os = post_data.get('OS', '') %}

{% set provider = post_data.get('PROVIDER', '') %}

{% set az = post_data.get('AZ', '') %}

{% set role = post_data.get('ROLE', '') %}

{# Convert comma separated string to a sorted list #}

{% set role = role.split(',')|sort %}

{# Initiate the Orchestration State #}

create_vm_runner:

  runner.state.orch:

    - mods: orch.create_vm

    - kwarg:

        pillar:

          vm:

            hostname:

UndefinedNameError: reference to undefined name 'hostname' (click for details)
Callstack:
    at (Enterprise_Solutions/Automation_and_Orchestration/Event-based_Orchestration:_An_Overview), /content/body/div[3]/div[7]/p[104]/span/span, line 1, column 2

            os:

UndefinedNameError: reference to undefined name 'os' (click for details)
Callstack:
    at (Enterprise_Solutions/Automation_and_Orchestration/Event-based_Orchestration:_An_Overview), /content/body/div[3]/div[7]/p[105]/span/span, line 1, column 2

            provider:

UndefinedNameError: reference to undefined name 'provider' (click for details)
Callstack:
    at (Enterprise_Solutions/Automation_and_Orchestration/Event-based_Orchestration:_An_Overview), /content/body/div[3]/div[7]/p[106]/span/span, line 1, column 2

            az:

UndefinedNameError: reference to undefined name 'az' (click for details)
Callstack:
    at (Enterprise_Solutions/Automation_and_Orchestration/Event-based_Orchestration:_An_Overview), /content/body/div[3]/div[7]/p[107]/span/span, line 1, column 2

            role:

UndefinedNameError: reference to undefined name 'role' (click for details)
Callstack:
    at (Enterprise_Solutions/Automation_and_Orchestration/Event-based_Orchestration:_An_Overview), /content/body/div[3]/div[7]/p[108]/span/span, line 1, column 2

Revise the Reactor Configuration

vi /etc/salt/master.d/reactor.conf

reactor:

  - 'salt/netapi/hook/hooktest':

    - /srv/reactor/hooktest.sls

  - 'salt/netapi/hook/create_vm':

    - /srv/reactor/create_vm.sls

Restart the Salt Master and Salt API Services

     systemctl restart salt-master salt-api

Watch the Event Bus in Another Terminal Window

     salt-run state.event pretty=True

Note: Please remember to do this in a 2nd Terminal Window

Test the Creation of a New Virtual Machine via a Webhook

curl -sSk https://localhost:8000/hook/create_vm \

   -X POST \

   -d HOSTNAME="minion02" \

   -d OS="rhel" \

   -d PROVIDER="ec2_us_west_1" \

   -d AZ="us-west-1b" \

   -d ROLE="salt-minion,apache"

NEXT STEPS

In this guide, you may have noticed that “role” is one of the attributes that has been provided.  As an independent exercise, do the following:

  • Create a simple state to install and configure Apache
  • Create a top file that targets minions according to their role
    • If the minion has the role of “apache”,
      then invoke your Apache state during the highstate
  • Create a reactor to initiate the highstate
  • Use the webhook to create a third minion to be automatically provisioned as an Apache server, with no interaction from you beyond the initial curl command

FURTHER READING

Salt Documentation: rest_cherrypy

https://docs.saltstack.com/en/latest/ref/netapi/all/salt.netapi.rest_cherrypy.html

Salt Documentation: Orchestration Runner

https://docs.saltstack.com/en/latest/ref/runners/all/salt.runners.state.html

Salt Documentation: Events

https://docs.saltstack.com/en/latest/topics/event/events.html

Salt Documentation: Reactors

https://docs.saltstack.com/en/latest/topics/reactor/

THANK YOU

We hope that this tutorial has been valuable to you!

  • Was this article helpful?