Skip to main content
SaltStack Support

Troubleshooting pillar information in reactor/orchestration states

 

This article demonstrates a quick way of troubleshooting a case in which an orchestration (triggered by reactor) is not working, suspecting on wrong or missed pillar data. A very common issue is that required pillar information is not received or received with wrong values.

First, understand how the orchestration works from start to end:
#1 reactor configuration:

cat /etc/salt/master.d/reactor.conf (or similar)
reactor:
  # Send keys email
  - CSE/portal/luks/sendkeys:      # Event tag that initiates the reactor
    - /srv/reactor/send_keys.sls    # Triggered Reactor state 

Comments:
  - How is this event generated (usually external call)
  - What arguments are passed together with the event tag (these arguments are handled as pillar data)


#2 reactor state

cat /srv/reactor/send_keys.sls
send_luks_keys:
  runner.state.orch:
# ORIGINAL
#    - mods: orch.send_keys
#      - pillar:
#        minion_id: {{ data.data['Minionid'] }}
#        sender_id: {{ data['id'] }}
# Mod: calling a duplicate for testing
    - mods: orch.test_send_keys
    - pillar:
        minion_id: {{ data.data['Minionid'] }}
        sender_id: {{ data['id'] }}

Comments:
  - If possible, comment out existing orchestration and call a duplicate one for testing
  - Observe events arguments are passed to orchestration state as pillar data

 

#3 (if possible) Make a duplicate of the orch state as you defined in reactor state

cp /srv/salt/orch/send_keys.sls /srv/salt/orch/test_send_keys.sls



#4 Alter orchestration state
Usually, the orchestration state will begin with some pillar.get statements and -maybe- some sort of conditional evaluation
Something similar to this:

{% set minion_id = salt['pillar.get']('minion_id') %}
{% set sender_id = salt['pillar.get']('sender_id') %}
{% if minion_id == 'master' %}
{% if sender_id in salt['pillar.get']('lamp:approved_key_senders') %}
...
...
{% endif %}
{% endif %} 

If minion_id and sender_id are not received from pillar or have unexpected values, the state won't execute as desired.
Even worst, there's not default values in the set statements that could be used to prevent unexpected results

So first thing, stop that faulty code by adding a {# Jinja comment #}

{# JINJA COMMENT START
{% set minion_id = salt['pillar.get']('minion_id') %}
{% set sender_id = salt['pillar.get']('sender_id') %}
{% if minion_id == 'master' %}
{% if sender_id in salt['pillar.get']('lamp:approved_key_senders') %}
...
...
{% endif %}
{% endif %}
JINJA COMMENT END#} 

 

 

Then add the following code on top in the orch state (adapt to your received pillar items), you will be checking that required pillar items are present, showing the pillar values and saving to a file:

# Code to check pillar values are received
# Check all the required pillar items
test_check_pillar:
  test.check_pillar:
    - present:
      - sender_id
      - minion_id
# Receive and give default value to pillar if requested pillar is not present
{% set sender_id = salt['pillar.get']('sender_id','PILLAR NOT FOUND!') %}
{% set minion_id = salt['pillar.get']('minion_id', 'PILLAR NOT FOUND!') %}
test_show_pillar_values:
  test.configurable_test_state:
    - name: Show Pillar values
    - changes: false
    - result: true
    - comment: |
        # Double spacing - 4 spaces is required
        PILLAR VALUES
        1-sender_id: {{ sender_id }}
        2-minion_id: {{ minion_id }}
# OPTIONAL
#    - require:
#      - test: test_check_pillar # Will not execute this state if test_check_pillar fails
# SAVE PILLAR VALUES TO FILE
test_save_pillar_2file:
  file.managed:
    - name: /tmp/pillar_info.txt
    - contents: |
        PILLAR VALUES
        1-sender_id: {{ sender_id }}
        2-minion_id: {{ minion_id }} 

 

 

#5 Test orchestration passing pillar information

-- OPTIONAL: You may want to check pillar items received by the minion, maybe remove all pillar items for the given minion tweaking the pillar top file if it gets confusing and refreshing pillar on the minion

# Test missing pillar information sender_id

salt-run state.orch orch.test_send_keys pillar='{"minion_id": "some_minion_id"}'
[ERROR   ] Pillar key "sender_id" is missing.
Those keys passed the check:
- minion_id (None)
ip-172-31-46-65.us-east-2.compute.internal_master:
----------
          ID: test_check_pillar
    Function: test.check_pillar
      Result: False
     Comment: Pillar key "sender_id" is missing.
              Those keys passed the check:
              - minion_id (None)
     Started: 19:42:37.747194
    Duration: 1.06 ms
     Changes:
----------
          ID: test_show_pillar_values
    Function: test.configurable_test_state
        Name: Show Pillar values
      Result: True
     Comment: # Double spacing - 4 spaces is required
              PILLAR VALUES
              1-sender_id: PILLAR NOT FOUND!
              2-minion_id: some_minion_id
     Started: 19:42:37.748445
    Duration: 0.347 ms
     Changes:
----------
          ID: test_save_pillar_2file
    Function: file.managed
        Name: /tmp/pillar_info.txt
      Result: True
     Comment: File /tmp/pillar_info.txt updated
     Started: 19:42:37.751222
    Duration: 5.375 ms
     Changes:
              ----------
              diff:
                  New file
Summary for ip-172-31-46-65.us-east-2.compute.internal_master
------------
Succeeded: 2 (changed=1)
Failed:    1
------------
Total states run:     3
Total run time:   6.782 ms


# Test with complete pillar information

salt-run state.orch orch.test_send_keys pillar='{"minion_id": "some_minion_id", "sender_id": "some_sender_id"}'
ip-172-31-46-65.us-east-2.compute.internal_master:
----------
          ID: test_check_pillar
    Function: test.check_pillar
      Result: True
     Comment: Those keys passed the check:
              - sender_id (None)
              - minion_id (None)
     Started: 19:45:56.938944
    Duration: 0.692 ms
     Changes:
----------
          ID: test_show_pillar_values
    Function: test.configurable_test_state
        Name: Show Pillar values
      Result: True
     Comment: # Double spacing - 4 spaces is required
              PILLAR VALUES
              1-sender_id: some_sender_id
              2-minion_id: some_minion_id
     Started: 19:45:56.939807
    Duration: 0.344 ms
     Changes:
----------
          ID: test_save_pillar_2file
    Function: file.managed
        Name: /tmp/pillar_info.txt
      Result: True
     Comment: File /tmp/pillar_info.txt updated
     Started: 19:45:56.942512
    Duration: 5.416 ms
     Changes:
              ----------
              diff:
                  New file
Summary for ip-172-31-46-65.us-east-2.compute.internal_master
------------
Succeeded: 3 (changed=1)
Failed:    0
------------
Total states run:     3
Total run time:   6.452 ms




cat /tmp/pillar_info.txt
PILLAR VALUES
1-sender_id: some_sender_id
2-minion_id: some_minion_id 

 

 


#6 Trigger reactor


Now that the orchestration test works fine, let's trigger orchestration using reactor
We need to send an event with the expected tag and arguments to trigger reactor

If possible, start two new connections to salt-master
Session 1: Run salt-master in foreground

service salt-master stop
salt-master -l debug


Session 2: Watch the event bus and save events to a log file /tmp/events.log  temporarily

salt-run state.event pretty=true | tee /tmp/events.log 

 

 

Then send an event with the required tag and expected arguments

salt-call event.send 'CSE/portal/luks/sendkeys' '{"Minionid": "some_minion_id"}'
local:
    True


You should see live action on the event bus and master log screens.

Once done...

Review pillar_info.txt

# cat /tmp/pillar_info.txt
PILLAR VALUES
1-sender_id: CSE
2-minion_id: some_minion_id

Follow execution trace in the events.log file, observe the pillar data and flow of items

cat /tmp/events.log
# TRIGGER EVENT
CSE/portal/luks/sendkeys    {
    "_stamp": "2019-07-19T19:56:33.447762",
    "cmd": "_minion_event",
    "data": {
        "Minionid": "some_minion_id",
        "__pub_fun": "event.send",
        "__pub_jid": "20190719195633436543",
        "__pub_pid": 22839,
        "__pub_tgt": "salt-call"
    },
    "id": "CSE",
    "tag": "CSE/portal/luks/sendkeys"
}
# RETURN FOR event.send
salt/job/20190719195633460956/ret/CSE    {
    "_stamp": "2019-07-19T19:56:33.462203",
    "arg": [
        "CSE/portal/luks/sendkeys",
        "{\"Minionid\": \"some_minion_id\"}"
    ],
    "cmd": "_return",
    "fun": "event.send",
    "fun_args": [
        "CSE/portal/luks/sendkeys",
        "{\"Minionid\": \"some_minion_id\"}"
    ],
    "id": "CSE",
    "jid": "20190719195633460956",
    "retcode": 0,
    "return": true,
    "tgt": "CSE",
    "tgt_type": "glob"
}
# NEW ORCHESTRATION JOB (MEANING REACTOR IS WORKING)
salt/run/20190719195634097590/new    {
    "_stamp": "2019-07-19T19:56:34.100982",
    "fun": "runner.state.orch",
    "fun_args": [
        "orch.test_send_keys",
        {
            "exclude": null,
            "orchestration_jid": null,
            "pillar": {
                "minion_id": "some_minion_id",
                "sender_id": "CSE"
            },
            "pillar_enc": null,
            "pillarenv": null,
            "saltenv": "base",
            "test": null
        }
    ],
    "jid": "20190719195634097590",
    "user": "Reactor"
}
# ORCHESTRATION orch.test_send_keys RETURN
salt/run/20190719195634097590/ret    {
    "_stamp": "2019-07-19T19:56:34.933333",
    "fun": "runner.state.orch",
    "fun_args": [
        "orch.test_send_keys",
        {
            "exclude": null,
            "orchestration_jid": null,
            "pillar": {
                "minion_id": "some_minion_id",
                "sender_id": "CSE"
            },
            "pillar_enc": null,
            "pillarenv": null,
            "saltenv": "base",
            "test": null
        }
    ],
    "jid": "20190719195634097590",
    "return": {
        "data": {
            "ip-172-31-46-65.us-east-2.compute.internal_master": {
                "file_|-test_save_pillar_2file_|-/tmp/pillar_info.txt_|-managed": {
                    "__id__": "test_save_pillar_2file",
                    "__run_num__": 2,
                    "__sls__": "orch.test_send_keys",
                    "changes": {
                        "diff": "New file"
                    },
                    "comment": "File /tmp/pillar_info.txt updated",
                    "duration": 6.384,
                    "name": "/tmp/pillar_info.txt",
                    "pchanges": {},
                    "result": true,
                    "start_time": "19:56:34.923077"
                },
                "test_|-test_check_pillar_|-test_check_pillar_|-check_pillar": {
                    "__id__": "test_check_pillar",
                    "__run_num__": 0,
                    "__sls__": "orch.test_send_keys",
                    "changes": {},
                    "comment": "Those keys passed the check:\n- sender_id (None)\n- minion_id (None)\n",
                    "duration": 1.076,
                    "name": "test_check_pillar",
                    "result": true,
                    "start_time": "19:56:34.918016"
                },
                "test_|-test_show_pillar_values_|-Show Pillar values_|-configurable_test_state": {
                    "__id__": "test_show_pillar_values",
                    "__run_num__": 1,
                    "__sls__": "orch.test_send_keys",
                    "changes": {},
                    "comment": "# Double spacing - 4 spaces is required\nPILLAR VALUES\n1-sender_id: CSE\n2-minion_id: some_minion_id\n",
                    "duration": 0.869,
                    "name": "Show Pillar values",
                    "result": true,
                    "start_time": "19:56:34.919411"
                }
            }
        },
        "outputter": "highstate",
        "retcode": 0
    },
    "success": true,
    "user": "Reactor"
}
 



Follow execution trace in master log screen:

(REDACTED)
In the master log, you will see all the pillar and state rendering and the orchestration result

[DEBUG   ] Sending event: tag = salt/run/20190719195634097590/ret; data = {u'fun_args': [u'orch.test_send_keys', {'pillarenv': None, 'test': None, 'exclude': None, 'pillar': OrderedDict([(u'minion_id', u'some_minion_id'), (u'sender_id', u'CSE')]), 'saltenv': u'base', 'orchestration_jid': None, 'pillar_enc': None}], u'jid': u'20190719195634097590', u'return': {u'outputter': u'highstate', u'data': {u'ip-172-31-46-65.us-east-2.compute.internal_master': {u'file_|-test_save_pillar_2file_|-/tmp/pillar_info.txt_|-managed': {u'comment': u'File /tmp/pillar_info.txt updated', u'pchanges': {}, u'name': u'/tmp/pillar_info.txt', u'start_time': '19:56:34.923077', u'result': True, u'duration': 6.384, u'__run_num__': 2, u'__sls__': u'orch.test_send_keys', u'changes': {u'diff': u'New file'}, u'__id__': u'test_save_pillar_2file'}, u'test_|-test_check_pillar_|-test_check_pillar_|-check_pillar': {u'comment': u'Those keys passed the check:\n- sender_id (None)\n- minion_id (None)\n', u'name': u'test_check_pillar', u'start_time': '19:56:34.918016', u'result': True, u'duration': 1.076, u'__run_num__': 0, u'__sls__': u'orch.test_send_keys', u'changes': {}, u'__id__': u'test_check_pillar'}, u'test_|-test_show_pillar_values_|-Show Pillar values_|-configurable_test_state': {u'comment': u'# Double spacing - 4 spaces is required\nPILLAR VALUES\n1-sender_id: CSE\n2-minion_id: some_minion_id\n', u'name': u'Show Pillar values', u'start_time': '19:56:34.919411', u'result': True, u'duration': 0.869, u'__run_num__': 1, u'__sls__': u'orch.test_send_keys', u'changes': {}, u'__id__': u'test_show_pillar_values'}}}, u'retcode': 0}, u'success': True, u'_stamp': '2019-07-19T19:56:34.933333', u'user': u'Reactor', u'fun': u'runner.state.orch'}
...
...
ip-172-31-46-65.us-east-2.compute.internal_master:
----------
          ID: test_check_pillar
    Function: test.check_pillar
      Result: True
     Comment: Those keys passed the check:
              - sender_id (None)
              - minion_id (None)
     Started: 19:56:34.918016
    Duration: 1.076 ms
     Changes:
----------
          ID: test_show_pillar_values
    Function: test.configurable_test_state
        Name: Show Pillar values
      Result: True
     Comment: # Double spacing - 4 spaces is required
              PILLAR VALUES
              1-sender_id: CSE
              2-minion_id: some_minion_id
     Started: 19:56:34.919411
    Duration: 0.869 ms
     Changes:
----------
          ID: test_save_pillar_2file
    Function: file.managed
        Name: /tmp/pillar_info.txt
      Result: True
     Comment: File /tmp/pillar_info.txt updated
     Started: 19:56:34.923077
    Duration: 6.384 ms
     Changes:
              ----------
              diff:
                  New file
Summary for ip-172-31-46-65.us-east-2.compute.internal_master
------------
Succeeded: 3 (changed=1)
Failed:    0
------------
Total states run:     3
Total run time:   8.329 ms
[INFO    ] Runner completed: 20190719195634097590 

 

Finally, you can query the job cache results

salt-run jobs.print_job 20190719195634097590
20190719195634097590:
    ----------
    Result:
        ----------
        ip-172-31-46-65.us-east-2.compute.internal_master:
            ----------
            return:
                ----------
                _stamp:
                    2019-07-19T19:56:34.100982
                fun:
                    runner.state.orch
                fun_args:
                    - orch.test_send_keys
                    |_
                      ----------
                      exclude:
                          None
                      orchestration_jid:
                          None
                      pillar:
                          ----------
                          minion_id:
                              some_minion_id
                          sender_id:
                              CSE
                      pillar_enc:
                          None
                      pillarenv:
                          None
                      saltenv:
                          base
                      test:
                          None
                jid:
                    20190719195634097590
                return:
                    ----------
                    data:
                        ----------
                        ip-172-31-46-65.us-east-2.compute.internal_master:
                            ----------
                            file_|-test_save_pillar_2file_|-/tmp/pillar_info.txt_|-managed:
                                ----------
                                __id__:
                                    test_save_pillar_2file
                                __run_num__:
                                    2
                                __sls__:
                                    orch.test_send_keys
                                changes:
                                    ----------
                                    diff:
                                        New file
                                comment:
                                    File /tmp/pillar_info.txt updated
                                duration:
                                    6.384
                                name:
                                    /tmp/pillar_info.txt
                                pchanges:
                                    ----------
                                result:
                                    True
                                start_time:
                                    19:56:34.923077
                            test_|-test_check_pillar_|-test_check_pillar_|-check_pillar:
                                ----------
                                __id__:
                                    test_check_pillar
                                __run_num__:
                                    0
                                __sls__:
                                    orch.test_send_keys
                                changes:
                                    ----------
                                comment:
                                    Those keys passed the check:
                                    - sender_id (None)
                                    - minion_id (None)
                                duration:
                                    1.076
                                name:
                                    test_check_pillar
                                result:
                                    True
                                start_time:
                                    19:56:34.918016
                            test_|-test_show_pillar_values_|-Show Pillar values_|-configurable_test_state:
                                ----------
                                __id__:
                                    test_show_pillar_values
                                __run_num__:
                                    1
                                __sls__:
                                    orch.test_send_keys
                                changes:
                                    ----------
                                comment:
                                    # Double spacing - 4 spaces is required
                                    PILLAR VALUES
                                    1-sender_id: CSE
                                    2-minion_id: some_minion_id
                                duration:
                                    0.869
                                name:
                                    Show Pillar values
                                result:
                                    True
                                start_time:
                                    19:56:34.919411
                    outputter:
                        highstate
                    retcode:
                        0
                success:
                    True
                user:
                    Reactor
    StartTime:
        2019, Jul 19 19:56:34.097590 

 

 

SUMMARY: Now you have a clear orchestration workflow with proper pillar information received and executed, adapt your current orchestration as needed or review the system making the external call that triggers reactors to make sure the proper arguments are passed to the reactor.

 
  • Was this article helpful?