A simple example of a mco command can be seen below:
In this example the ping sub-command is referred to as an
application. Mcollective provides many applications, for a list of
them, type mco help. You can also create your own application to plug
into the framework. The help sub-command will show you something like
You can request help for a specific application using either mco help
application or mco application —help. Shown below is part of the
help for the rpc application:
The help first shows a basic overview of the command line syntax
followed by options specific to this command. Following that you will
see some Common Options and Host Filters that generally apply to
Making RPC Requests
Overview of a Request
The rpc application is the main application used to make requests to
your servers. It is capable of interacting with any standard Remote
Procedure Call (RPC) agent. Below is an example that shows an attempt to
start a webserver on several machines:
The order of events in this process are:
Perform discovery against the network and discover 10 servers
Send the request and then show a progress bar of the replies
Show any results that were out of the ordinary
Show some statistics
Mcollective client applications aim to only provide the most relevant
information. In this case, the application is not showing verbose
information about the nine OK results, since the most important issue
is the one Failure. Keep this in mind when viewing the results of
Anatomy of a Request
MCollective agents are broken up into actions and each action can take
This shows the basic make-up of an RPC command. In this case we are:
using the rpc application - a generic application that can interact with any agent
directing our request to machines with the service agent
sending a request to the stop action of the service agent
supplying a value, httpd, to the service argument of the stop action
The same command has a longer form as well:
These two commands are functionally identical.
Discovering Available Agents
The above command showed you how to interact with the service agent,
but how can you find out that this agent even exists? On a correctly
installed MCollective system you can use the plugin application to get
The first part of this list shows all the agents this computer is aware
of. In order to show up on this list, an agent must have a DDL file
and be installed locally.
To find out the actions, inputs and outputs for a specific agent
use the plugin application again:
This shows a truncated example of the auto-generated help for the
service agent. First shown is metadata such as version, author and
license. This is followed by the list of actions available, in this case
the restart, start, status and stop actions.
Further information is shown about each action. For example, you can see
that the status action requires an input called service which is a
string, has a maximum length of 30, etc. You can also see you will
receive one output called status
With this information, you can request the status for a specific
Unlike the previous example, in this case specific information is
returned on the success of the action. This is because this specific
action is meant to retrieve information and so mcollective assumes you
would like to see complete, thorough data regardless of success or
Note that this output displays Service Status as shown in the mco
plugin doc service help page. Any time you need more information about
a display name, the doc for the associated agent will have a
Description section for every input and output.
Selecting Request Targets Using Filters
A key capability of mcollective is fast discovery of network resources.
Discovery rules are written using filters. For example:
This shows a filter rule that limits the RPC request to being run on
machines that are either in the Puppet environment development or
belong to the Customer acme.
Filtering can be based on facts, the presence of a Configuration
Management Class on the node, the node’s Identity, or installed
Agents on the node.
Here are a number of examples of this with short descriptions of each
As you can see, you can filter by Agent, Class and/or Fact, and you can
use regular expressions almost anywhere. You can also combine filters
additively in a command so that all the criteria have to be matched.
Note: You can use a shortcut to combine Class and Fact filters:
Complex Compound or Select Queries
While the above examples are easy to enter, they are limited in that
they can only combine filters additively. If you want to create searches
with more complex boolean logic use the -S switch. For example:
The above example shows a scenario where the development environment is
usually labeled development but one customer has chosen to use
staging. You want to find all machines in those customer’s
environments that match the class apache. This search would be
impossible using the previously shown methods, but the above command
uses -S to allow the use of boolean operators such as and and or
so you can easily build the logic of the search.
The -S switch also allows for negative matches using not or !:
Filtering Using Data Plugins
As of version 2.1.0, custom data plugins can also be used to create
This will search for the md5 hash of a specific file with matches
restricted to the development environment. Note that as before,
regular expressions can also be used
Structured facts require the use of data plugins.
As with agents, you can also discover which plugins are available for
For information on the input these plugins take and output they provide
use the mco plugin doc fstat command.
Currently, each data function can only accept one input while matches
are restricted to a single output field per invocation.
Chaining RPC Requests
The rpc application can chain commands one after the other. The
example below uses the package agent to find machines with a specific
version of mcollective and then schedules Puppet runs on those machines:
Mcollective results can also be filtered using the opensource gem,
jgrep. Mcollective data output is fully compatible with jgrep.
Using with PuppetDB
Recent versions of PuppetDB has a built in query language called
Puppet Query Language that you use via the puppet query command.
Much like the above example of chaining RPC requests MCollective supports
reading results from Puppet Query:
This will run Puppet on all CentOS machines
Seeing the Raw Data
By default the rpc application will try to show human-readable data.
To see the actual raw data, add the -v flag to disable the display
This data can also be returned in JSON format:
When an application encounters an error, it returns an explanatory
By default only an abbreviated error string is shown that provides some
insight into the nature of the problem. For more details, add the -v
flag to show a full stack trace:
When mco finishes, it generates an exit code. The returned exit code depends on the nature of the issue:
0: If nodes were discovered and all passed.
0: If no discovery was performed, at least 1 response was received, and all responses were OK.
1: If no nodes were discovered, or if mco encountered an issue not listed here.
2: If nodes were discovered but some RPC requests failed.
3: If nodes were discovered, but no responses were received.
4: If no discovery was performed, and no responses were received.
The rpc application should suit most needs. However, sometimes the
data being returned calls for customization such as custom aggregation,
summarising or complete custom display.
In such cases, a custom application may be useful For example, the
package application provides concluding summaries and provides some
basic safe guards for its use. The agent also provides the commonly
required data. Typical package output looks like this:
Notice how this application recognises that you are acting on all
possible machines, an action which might have a big impact on your YUM
servers. Consequently, package prompts for confirmation and, at the
end of processing, displays a brief summary of the network status.
While the behaviors of custom applications are not always consistent
with each other, in general they accept the standard discovery flags.
For details of which flags are accepted in a given application, use the
mco help appname command.
To discover which custom applications are available, run mco or mco