Skip to main content
SaltStack Support

Background Worker Options

Overview

SaltStack Enterprise includes new background worker options to optimize performance for a range of deployment scenarios, since version 5.5. Background workers handle processes that run in the background, such as running jobs according to a given schedule, or syncing with an LDAP server.

New options for background workers are particularly useful for high-availability deployments. You can customize background worker settings to share background processes across multiple servers, as demonstrated in the following diagram.

High Availability Diagram.png

How to configure options for background workers

To define background worker options, modify the raas configuration file when you install or upgrade SaltStack Enterprise, or any time you want to improve the performance of your Enterprise deployment.

See the Enterprise download and installation page for full installation instructions.

See Commands for a list of commands that are useful for troubleshooting.

 

Background Worker options

The following table describes available background worker options.

For recommended background worker settings, see Examples of background worker settings.

Option Description
combined_process Boolean to define whether both web server processes and background workers run on the server.
  • If True, both types of processes run on the server.
  • If False, only one type of process runs on the server.
concurrency Number of worker processes. One unit represents one CPU core.
  • If 0, automatically calculate one process per core, divided evenly between web server and background worker processes.
max_tasks Number of tasks a worker can complete until it restarts.
max_memory Amount of memory in kB a worker can consume until it restarts.
  • If 0, define automatically based on available memory.
  • If None, allow the maximum memory consumption possible, up to 1GB.

 

Commands

Use the following commands to assist with troubleshooting. The -l flag indicates debug mode.

Command Description
raas -l info Run both web server processes and background worker processes.

To run only web server processes, add the following to etc/raas/raas before running the command.

background_workers:
 combined_process: False
raas -l info worker Run only worker processes.
raas -l info beat Run only the beat process. The beat process initiates other processes at a set interval, distributing periodic tasks to workers.
raas celery purge Purge the task queue.

 

Examples of background worker settings

You can customize background worker settings to improve performance for a range of deployment scenarios. Multiple scenarios are described below, with recommended settings for each.

Note: In the following examples, num_processes represents the number of web server processes.


Scenario 1: High availability

The following example is a shared system where both web processes and background worker processes run concurrently on each server, supported by a load balancer. Each server includes 10 CPU cores. Processing power is distributed evenly across both types of processes.

Background Worker Scenario 1.png/etc/raas/raas

On each server:

num_processes: 5
background_workers:
 combined_process: True
 concurrency: 5
 max_tasks: 100000
 max_memory: 1048576

Scenario 2: Scaling web processes

The following example is a deployment that experiences a large amount of web server processes and relatively few background processes.

This scenario might be found, for example, in an environment that has many minions, and does not rely heavily on schedules. 

Background Worker Scenario 2.png

/etc/raas/raas

On one server:

num_processes: 5
background_workers:
 combined_process: True
 concurrency: 5
 max_tasks: 100000
 max_memory: 1048576

On a second server:

num_processes: 10
background_workers:
 combined_process: False

Scenario 3: Scaling schedules

The following example is a deployment that needs to support a large amount of background processes, and relatively few web server processes.

This scenario might be found, for example, in an environment that relies heavily on schedules. Each server includes 10 CPU cores.

Background Worker Scenario 3.png

/etc/raas/raas

On one server:

num_processes: 5
background_workers:
 combined_process: True
 concurrency: 5
 max_tasks: 100000
 max_memory: 1048576

On a second server:

background_workers:
 combined_process: False
 concurrency: 10

Scenario 4: Scaling schedules

The following example is a multi-node system where web processes and background worker processes run on separate servers. Each server includes 10 CPU cores.

This configuration provides a simple framework for scaling SaltStack Enterprise without a load balancer, by moving your background work to a second server.

Background Worker Scenario 4.png

/etc/raas/raas

On one server:

num_processes: 10
background_workers:
 combined_process: False

On a second server:

background_workers:
 combined_process: False
 concurrency: 10
  • Was this article helpful?