Puppet Deployment: MCollective in Puppet 3.x

Puppet master/agent deployment approach is a pull-based mode (by default) fetching the master node controlled catalog to target agent nodes and apply to target agent nodes periodically (30 minutes, by default).

Puppet 2.7 Out-Of-Box puppet kick (Remotely control puppet agent) has been deprecated in Puppet 3.0.

The Marionette Collective (MCollective for short) is a tool for system administration. It can run commands on large numbers of servers in parallel, and uses a broadcast architecture so that you can administer a large network without the need for a central master server or asset database.

Each node runs an MCollective daemon that listens for requests, and can execute commands locally or return information about the server. This can be used to filter the list of target nodes. So, for example, you could use MCollective to execute a given command on all nodes that match certain criteria. You can think of MCollective as a complement to Puppet.

For example, your provisioning process for a new node might require firewall changes on other machines, permissions granted on a database server, and so on, which is not very easy to do with Puppet. Although you could automate specific jobs using shell scripts and SSH, MCollective provides a powerful and flexible way to solve this general problem.

In order to have a Puppet master/agent deployment infrastructure with a push-based mode and deploy the master nodes controlled catalog into target agent nodes on demand. 

Here is the puppet 3.0.2 + mcollective configuration example to achieve a Puppet master/agent deployment infrastructure in a push-mode mode:
1. Within a Puppet master/agent deployment infrastrucutre with cert sign (make sure all of puppet nodes sync with Network Time Protocol otherwise there will be some difficulties to do puppet cert sign)
2. On puppet agent nodes, configured puppet agent service to non-daemonized
3. Installation of RPMs:

1. yum install activemq activemq-info-provider on a middle-ware node
2. yum install mcollective mcollective-client mcollective-puppet-agent mcollective-puppet-client on mcollective client node where the requests come from 
3. yum install mcollective mcollective-puppet-agent on puppet agent nodes

4. Configuration

1. make mcollective daemon is up running on all of nodes
2. example of /etc/mcollective/server.cfg

cat /etc/mcollective/server.cfg
# main config
libdir = /usr/libexec/mcollective
logfile = /var/log/mcollective.log
loglevel = info
daemonize = 1

# connector plugin config
connector = activemq
plugin.activemq.pool.size = 1
plugin.activemq.pool.1.host = <activemq_middle-ware_node>
plugin.activemq.pool.1.port = 61613
plugin.activemq.pool.1.user = mcollective
plugin.activemq.pool.1.password = secret

# facts
factsource = yaml
plugin.yaml = /etc/mcollective/facts.yaml

# Plugins
securityprovider = psk
plugin.psk = unset

# puppet
plugin.puppet.command = puppet agent
plugin.puppet.splay = true
plugin.puppet.splaylimit = 30
plugin.puppet.config = /etc/puppet/puppet.conf
plugin.puppet.resource_allow_managed_resources = true
5. mcollective command with puppet plugin 
[root@linux64-client ~]#  mco puppet runonce -v
Discovering hosts using the mc method for 2 second(s) .... 2

* [ ============================================================> ] 2 / 2

linux64-node2                               : OK
    {:summary=>      "Started a background Puppet run using the 'puppet agent --onetime --daemonize --color=false --splay --splaylimit 30' command"}

linux64-client                               : OK
    {:summary=>      "Started a background Puppet run using the 'puppet agent --onetime --daemonize --color=false --splay --splaylimit 30' command"}



---- rpc stats ----
           Nodes: 2 / 2
     Pass / Fail: 2 / 0
      Start Time: Tue May 14 21:58:51 -0400 2013
  Discovery Time: 2015.99ms
      Agent Time: 1148.63ms
      Total Time: 3164.62ms
[root@linux64-client ~]# mco puppet status linux64-node2 -v
Discovering hosts using the mc method for 2 second(s) .... 2

* [ ============================================================> ] 2 / 2

   linux64-client: Currently stopped; last completed run 12 seconds ago
   linux64-node2: Currently applying a catalog; last completed run 9 minutes 32 seconds ago

Summary of Applying:

    true = 1
   false = 1

Summary of Daemon Running:

   running = 1
   stopped = 1

Summary of Enabled:

   enabled = 2

Summary of Idling:

   false = 2

Summary of Status:

              stopped = 1
   applying a catalog = 1


---- rpc stats ----
           Nodes: 2 / 2
     Pass / Fail: 2 / 0
      Start Time: Tue May 14 21:59:31 -0400 2013
  Discovery Time: 2015.43ms
      Agent Time: 50.95ms

      Total Time: 2066.38ms