Published on 30 March 2011 by

Check out Part 1 of this post here.

Using MCollective to Query Your Nodes

MCollective is an awesome solution for accessing information from all of your nodes very quickly. MCollective has a facter plugin that can pull fact information out of facter, but there's a method that Jordan Sissel devised for exporting all of your facter facts (as well as all of the variables in the puppet scope) to a YAML file (specifically, /etc/mcollective/facts.yaml). Setting this up is pretty trivial (it's a single-line inline template), and the benefits are exponential; not only is this MUCH faster than querying facter, but, as I mentioned, it pulls all of your variables out of the puppet scope in which you declared the /etc/mcollective/facts.yaml file. Follow the instructions in that link and let's setup the /etc/mcollective/facts.yaml file.

Once you've set this up, let's make sure MCollective is able to read from the file. This can be set in your /etc/mcollective/server.cfg config file with the following:

        factsource = yaml
	plugin.yaml = /etc/mcollective/facts.yaml

Next, we need to install a plugin so MCollective can handle YAML files as a factsource. The yaml_facts.rb file is included as a plugin with the MCollective source, but it can also be downloaded from the PuppetLabs Github site. Drop this file into your MCollective library's facts folder (usually /usr/libexec/mcollective/mcollective/facts), ensure that the factsource is set in your server.cfg config file, and restart MCollective to test it out. You can execute "mc-inventory (hostname)" to test and make sure that facts are being reported properly.

With MCollective up and running and reading facts directly from a YAML file, let's run a sample query with "mc-facts warranty_out" and see what we get:

        Report for fact: warranty_out

	        No                                      found 38 times
	        Yes                                     found 75 times

	Finished processing 113 / 113 hosts in 1047.24 ms

If you remember, this fact tells us if our machine's warranty has expired. In this test with 113 nodes, you can see that 75 nodes have expired warranties and 38 nodes are currently under warranty. Next, let's look at the expiration date for our warranties with "mc-facts warranty_end":

        Report for fact: warranty_end

	        2011.02.23                              found 4 times
	        2011.04.30                              found 1 times
	        2011.07.14                              found 23 times
	        2011.07.15                              found 3 times
	        2011.07.16                              found 2 times
	        2011.07.17                              found 2 times
	        2011.07.19                              found 16 times
	        2011.09.16                              found 2 times
	        2012.09.15                              found 5 times
	        Expired                                 found 123 times

	Finished processing 183 / 183 hosts in 3098.34 ms

Here's a larger sampling of nodes with several warranty expiration dates. This is all great information, but wouldn't it be nice to see exactly WHICH nodes are corresponding to these dates. You can do that by running "mc-facts warranty_end -v". The -v argument will list nodes corresponding to each fact underneath that fact in your output:

        2012.09.15                              found 4 times

	            berry-shawstaff-shel
	            deville-staff-wes
	            kluding-staff-shel
	            solomon-staff-shel

You can repeat this command with ANY fact. Since I am using the YAML solution as a factsource, it's also exporting variables from my puppet scope. I have several variables associated with packages that are to be installed, and these package reside in the same scope as the /etc/mcollective/facts.yaml file declaration, so I can also query these variables with "mc-facts (variablename)"

        Report for fact: firefox

	        firefox3.6.11.dmg                       found 218 times

	Finished processing 247 / 247 hosts in 3027.37 ms

NOTE: Pay attention to the scope where you declare the /etc/mcollective/facts.yaml file. If this file is declared within the scope of its own class, you won't see variables OUTSIDE its scope. You can do creative things to change the scope of this file if you wish (possibly using puppet-concat), but I'll leave that up to you.

What About Offline Nodes?

MCollective works great for nodes that are online and connecting to your middleware solution (ActiveMQ, RabbitMQ, etc.), but what about machines such as laptops (or machines that have hardware failures) that might not be online? Fortunately, your puppet master server keeps a YAML store of fact data for all nodes that have ever connected to it. Located in your puppet master's $vardir/yaml/facts directory, there's a YAML file (identified by $certname) for every node. Writing a script to traverse the directory and query the nodes is pretty trivial—you can even create an MCollective Agent to do the task for you. Let's look at a ruby script I created to search the YAML store:

        #!/usr/bin/env ruby
	#
	# File:			yaml_store_search.rb
	#
	# Description:	A script to search through your puppet master's YAML store
	#				 for a specific node's data.

	require 'getoptlong'
	require 'puppet'

	Puppet.settings.parse

	if not ARGV[0]
	  puts "You must have an argument.  Use --help for more information."
	  exit(1)
	end

	# Parse the arguments here and print the help prompt if any incorrect arguments
	#  are being used.
	opts = GetoptLong.new(
	    [ '--hostname', '-n', GetoptLong::REQUIRED_ARGUMENT],
	    [ '--certname', '-c', GetoptLong::REQUIRED_ARGUMENT],
	    [ '--help', '-h', GetoptLong::NO_ARGUMENT]
	)

	begin
	  opts.each do |opt, arg|
	      case opt
	          when '--hostname'
	              $hostname = arg
	          when '--certname'
	              $certname = arg
	          when '--help'
	              puts <<-EOF
	              yaml_store_search.rb [OPTION] value

	              -n, --hostname:
	                Search for a specific hostname in the YAML store. Requires a hostname argument

	              -c, --certname:
	                Search for a specific certname in the YAML store. Requires a certname argument

	              -h, --help:
	                Display this help text

	                EOF
	          end
	        end
	rescue GetoptLong::InvalidOption, GetoptLong::AmbigousOption
	  puts <<-EOF   

	  -n, --hostname:
	    Search for a specific hostname in the YAML store. Requires a hostname argument

	  -c, --certname:
	    Search for a specific certname in the YAML store. Requires a certname argument

	  -h, --help:
	    Display this help text

	    EOF
	  exit(1)
	end

	# Determine whether we're doing a lookup via hostname or certname
	searchfield = 'hostname' and arg = $hostname if $hostname
	searchfield = 'certname' and arg = $certname if $certname

	# Iterate through your puppet YAML store using the $vardir from
	#  your puppet.conf settings. If we find the YAML file with a
	#  matching certname or hostname, break out of the loop.
	Dir.glob("#{Puppet[:vardir]}/yaml/facts/*") {|file|
	  $tempfile = YAML::load_file(file).values
	  if $tempfile[searchfield] == arg
	    $found_file = true
	    break
	  end
	}

	# Output the list of facts.
	if $tempfile
	  $tempfile.each_pair {|key, value|
	    puts "#{key} = #{value}"
	  }
	end

This script will gather the value of your $vardir and will search the $vardir/yaml/facts directory in search of a node by way of its hostname or certname. If the node was found, it will output every fact that it has kept about that node. This script will ONLY work on puppet masters as they're the only machines who keep this YAML store.

Since we're talking about MCollective, let's look at an MCollective Agent that will execute this code on your puppet master. Here's the sample agent I created:

        module MCollective
	    module Agent
	        class Yaml_store "yaml_store.rb",
	                          :description => "A conduit to search your puppet master's YAML store.",
	                          :author      => "Gary Larizza ",
	                          :license     => "Apache License, Version 2.0",
	                          :version     => "1.0",
	                          :url         => "http://glarizza.posterous.com",
	                          :timeout     => 3

	              # Search action:  This action will check for the YAML file of a specified certname or hostname.
	              # =>               If the file exists, it will output the contents of the YAML file. If it
	              # =>               doesn't exist, you will receive an error message.
	              # Variables:
	              # =>              fact  => The fact for which we're checking a value
	              # =>              value => The value for which we're searching
	              # Calling:
	              # =>              Run the fact with 'mc-rpc --agent yaml_store --action search --arg hostname=lab01-hsimaclab-hhs -v'
	              # =>               or with 'mc-rpc --agent yaml_store --action search --arg certname=lab01-hsimaclab-hhs -v'
	              # =>               or with 'mc-rpc --agent yaml_store --action search --arg certname=lab01-hsimaclab-hhs --arg fact=warranty_end -v'
	              # 

	              action "search" do

	                if request.include?(:hostname)
	                  if request.include?(:fact)
	                    searchfield = 'hostname'
	                    arg = request[:hostname]
	                    reply[:facts] = get_specific_fact(searchfield, arg, request[:fact])
	                  else
	                    searchfield = 'hostname'
	                    arg = request[:hostname]
	                    reply[:facts] = get_all_facts(searchfield, arg)
	                  end
	                elsif request.include?(:certname)
	                  if request.include?(:fact)
	                    searchfield = 'certname'
	                    arg = request[:certname]
	                    reply[:facts] = get_specific_fact(searchfield, arg, request[:fact])
	                  else
	                    searchfield = 'certname'
	                    arg = request[:certname]
	                    reply[:facts] = get_all_facts(searchfield, arg)
	                  end
	                end

	              end # Action end

	                def get_all_facts(searchfield, arg)
	                  Dir.glob("#{Puppet[:vardir]}/yaml/facts/*") {|file|
	                    $tempfile = YAML::load_file(file).values
	                    if $tempfile[searchfield] == arg
	                      $found_file = true
	                      break
	                    end
	                  }

	                  if $found_file
	                    $tempfile.each_pair{|key, value|
	                      puts "#{key} = #{value}"
	                    }
	                  else
	                    puts "That #{searchfield} was not found."
	                  end

	                end # Def end

	                def get_specific_fact(searchfield, arg, thefact)
	                  Dir.glob("#{Puppet[:vardir]}/yaml/facts/*") {|file|
	                    $tempfile = YAML::load_file(file).values
	                    if $tempfile[searchfield] == arg
	                      $found_file = true
	                      break
	                    end
	                  }

	                  if $found_file
	                    if $tempfile[thefact]
	                      puts "#{thefact} = #{$tempfile[thefact]}"
	                    else
	                      puts "#{thefact} is not a valid fact, or wasn't found."
	                    end
	                  else
	                    puts "That #{searchfield} was not found."
	                  end

	                end # Def end

	        end # Class Etc_facts end

	  end # Module Agent end

	end # Module MCollective end

This agent will accept an argument of a node's hostname or certname and also, optionally, a specific fact whose value we would want to inspect. I've also created a DDL file that will parse the output of this agent properly. Here's that DDL file:

        metadata                :name        => "Yaml Store",
				:description => "A conduit to search your puppet master's YAML store",
				:author      => "Gary Larizza ",
				:license     => "Apache License, Version 2.0",
				:version     => "1.0",
				:url         => "http://glarizza.posterous.com",
				:timeout     => 60

	action "search", :description => "Retrieves Facter facts for specified nodes" do
	    display :always

	    input :hostname,
	          :prompt      => "Hostname",
	          :description => "The hostname of the node for which we're getting facts",
	          :type        => :string,
	          :optional    => true,
			  :validation  => '(.*?)',
	          :maxlength   => 230

		input :certname,
	          :prompt      => "The Puppet Certname Variable",
	          :description => "The certname of the node for which we're getting facts",
	          :type        => :string,
	          :optional    => true,
			  :validation  => '(.*?)',
	          :maxlength   => 230

		input :fact,
	          :prompt      => "A Searchable Facter Fact",
	          :description => "The facter fact for which we want a value",
	          :type        => :string,
	          :optional    => true,
	 	      :validation  => '(.*?)',
	          :maxlength   => 230

	    output 'facts',
	          :description => "The Facter facts you want displayed",
	          :display_as  => "Fact Information"
	end

In order for this agent to work, your puppet master must also be running MCollective and the agent and DDL file must be located in your MCollective library's agent folder (typically /usr/libexec/mcollective/mcollective/agent). Once you've dropped these files into the agent folder, you'll need to restart MCollective for it to work. Here's the syntax you will use to run it:

        mc-rpc --agent yaml_store --action search --arg hostname=machine_hostname
        mc-rpc --agent yaml_store --action search --arg certname=machine_certname
	mc-rpc --agent yaml_store --action search --arg hostname=machine_hostname --arg fact=facter_fact
	mc-rpc --agent yaml_store --action search --arg certname=machine_hostname --arg fact=facter_fact

        # Or Simplified:
        mc-rpc yaml_store search hostname=machine_hostname
        mc-rpc yaml_store search certname=machine_certname fact=facter_fact

You can run this command from ANY MCollective client so long as the agent is installed on your puppet master server. The catch is that if you want the client to have the output parsed correctly, they will need to have ONLY the DDL file dropped into their MCollective library's agent folder (the agent file itself is only needed on the puppet master server). If the client DOESN'T have the DDL file installed, they can simply append "-v" at the end of the above commands to see output from the reply hash that is used to communicate with the DDL file.

This agent will return either the entire list of facts for your specified hostname/certname or, optionally, the value of a specific fact. This is useful if your machine has malfunctioned (or is offline) and you quickly want to access whether the warranty has expired or not. Being able to access it from an MCollective client who has access to mc-rpc is handy if you're not currently on your work computer. This access, however, is entirely up to you.

In Conclusion

If you're familiar with MCollective and Puppet (and you have a functional understanding of Ruby), it's pretty easy to write custom Facter facts and MCollective agents to automate simple tasks. The process of maintaining an inventory can be particularly troublesome, but these scripts should give you near-immediate access to some of the most important facts about your Macs. Like I mentioned in the opening, most vendors will provide you with a conduit to access similar information to facts I've presented here for my Macs. All it will take is a bit of tweaking to adapt my code for your use. Keep pulling those strings!

Gary Larizza is the Director of Technology for a small K-12 School district in Northern Ohio where he routinely breaks systems in the name of advancement. When he's not trying to automate himself out of a job through the use of Puppet and Ruby, he enjoys seeking out awesome open source tools that will make Mac Management easier. He's been using Puppet in the Education sector for the past two years on hardware that would make Linux admins cry (namely, G5 Xserves and an iMac) and loves to share the knowledge he's gained with anyone who buys him a wheat beer (Weihenstephaner FTW). All of his notes are on his Posterous blog, and he will be speaking at the 2011 Penn State University Macadmins Conference if you'd prefer to chat with him in-person.

Share via:
Posted in:

Add new comment

The content of this field is kept private and will not be shown publicly.

Restricted HTML

  • Allowed HTML tags: <a href hreflang> <em> <strong> <cite> <blockquote cite> <code> <ul type> <ol start type> <li> <dl> <dt> <dd> <h2 id> <h3 id> <h4 id> <h5 id> <h6 id>
  • Lines and paragraphs break automatically.
  • Web page addresses and email addresses turn into links automatically.