The Marionette Collective, also known as MCollective, is a framework for building server orchestration or parallel job-execution systems. Most users programmatically execute administrative tasks on clusters of servers.
Deprecation Note: As of Puppet agent 5.5.4, MCollective is deprecated and will be removed in a future version of Puppet agent. If you use MCollective with Puppet Enterprise, consider moving from MCollective to Puppet orchestrator. If you use MCollective with open source Puppet, consider migrating MCollective agents and filters using tools like Bolt and PuppetDB’s Puppet Query Language.
MCollective has some unique strengths for working with large numbers of servers:
- Instead of relying on a static list of hosts to command, it uses metadata-based discovery and filtering. It can use a rich data source like PuppetDB, or can perform real-time discovery across the network.
- Instead of directly connecting to each host (which can be resource-intensive and slow), it uses publish/subscribe middleware to communicate in parallel with many hosts at once.
To get an immediate feel for what this means, check out the videos on the Screencasts page. Then, keep reading below for further information and links.
We’ve also created a Vagrant-based demo, where you can easily experiment with MCollective.
What is MCollective, and What Does It Allow You to Do?
- Interact with clusters of servers, whether in small groups or very large deployments.
- Use a broadcast paradigm to distribute requests. All servers receive all requests at the same time, requests have filters attached, and only servers matching the filter will act on requests. There is no central asset database to go out of sync, because the network is the only true source.
- Break free from identifying devices through complex host-naming conventions, and instead use a rich set of metadata provided by each machine — from Puppet, Facter, or other sources — to address them.
- Use simple command-line tools to call remote agents.
- Write custom reports about your infrastructure.
- Use agent plugins to manage packages, services, and other common components created by the community.
- Write simple RPC style agents, clients, and web UIs in Ruby.
- Extremely pluggable and adaptable to local needs.
- Leverage rich authentication and authorization models in middleware systems as a first line of control.
- Include fine-grained authentication using SSL or RSA, authorization, and request auditing. For more information, see the Security Overview.
- Re-use middleware features for clustering, routing, and network isolation to realize secure and scalable configurations.
MCollective provides a stable core framework that you can build into a system to meet your own needs. MCollective is pluggable in the following areas:
- Replace our STOMP-compliant middleware with your own, such as something AMQP-based.
- Replace our authorization system with one that suits your local needs.
- Replace our Ruby Marshal and YAML-based serialization with your own, such as cross-language JSON Schema.
- Add data sources. It’s easy to use Puppet’s data, and you can configure or build plugins for other data sources.
- Create a central inventory of services leveraging MCollective as transport that can regularly run and distribute inventory data.
MCollective is licensed under the Apache License, Version 2.0.
Next Steps and Further Reading
Introductory and Tutorial Pages
Internal References and Developer Docs
- Finding it hard to do something complex with Simple RPC? See Writing Agents for what lies underneath.
- Role based security, authentication and authorization using ActiveMQ.
- Structure of Request and Reply messages.