In this section we will discuss two different ways to enhance the security of your middleware configuration. Both of these options use Transport Layer Security (TLS) which is an enhanced version of Secure Sockets Layer (SSL).
TLS protects traffic by encrypting it with a pre-arranged symmetric key. This key is used to encrypt the traffic flowing between the two sides. Each side of the TLS connection can (optionally) validate the far side’s X.509 certificate. This asymmetric cryptography can assure that the far side with whom they are communicating is valid prior to sending any data.
As an example, when you connect to your bank’s website your browser does a cryptographic validation that the website is really your bank’s site. It does this by ensuring that the bank’s public key was signed (in an X.509 certificate) by an authority that the browser recognizes and trusts.
The bank does not usually require your browser to provide a certificate back to it proving who you are, although this is a valid TLS configuration.
If you wish to implement TLS encryption or Trusted TLS authentication it is essential that you understand these configuration choices.
Trusted TLS Security provides not only encrypted transport, but also cryptographic authentication between the MCollective clients and servers and the middleware. This configuration requires that every MCollective client and server have a pre-signed TLS certificate to access the middleware. This ensures the most extensive security for the middleware.
There are two parts of enabling Trusted TLS Security - Trusted Servers and Trusted Clients, each of which have their own section below.
Let’s go through how to configure these two options now.
Middleware security options only control the ability to connect to the middleware. Which queues and topics a node can read and write from is controlled by the authorizationEntry
configuration documented in Authentication and Authorization.
MCollective has its own Authorization system which controls whether or not a given MCollective request is allowed on a server. So this layer of security only controls whether or not a node can connect, and whether or not the communication is encrypted.
Anonymous TLS Security is the easier to configure option to encrypt traffic between MCollective and the Middleware. This option uses TLS encryption to protects the connection from snooping of usernames, passwords, and MCollective request data. The clients continue to authenticate to the middleware using the usernames and passwords we configured in Configuring ActiveMQ.
This is a good security model if you trust your firewall, dns, etc infrastructure to ensure that no unauthorized systems can reach your middleware systems. You should avoid using this security model when using insecure DNS or Internet transit.
mcollective
.registration.agent
topic
If you are using the puppet module provided with this book, you need only the following Hiera data and all three of the following steps will be done for you. No other changes are required. Once every node has accepted the changes, everything will be working over SSL.
mcollective::connector_ssl : true
mcollective::connector_ssl_type : anonymous
mcollective::middleware::keystore_password: openssl rand -base64 32
Or if you use old school expressive Puppet policies:
nodesomething-every-node-inherits
{ class { 'mcollective': connector_ssl => true, connector_ssl_type => 'anonymous', } class { 'mcollective::middleware': keystore_password => 'openssl rand -base64 20
', } }
Next we want to set up the SSL keys used for negotiating the encrypted connection. If you want to setup the keystore by hand, you can use any existing SSL keypair. The following example uses the Puppet keypair to create the Java keystore.
$mkdir /etc/activemq/ssl
$cd /etc/activemq/ssl
$sudo puppet agent --configprint ssldir
/var/lib/puppet/ssl $export HOSTNAME=$( hostname -f )
$sudo cat /var/lib/puppet/ssl/certs/$HOSTNAME.pem /var/lib/puppet/ssl/private_keys/$HOSTNAME.pem > $HOSTNAME-combined.pem
$openssl pkcs12 -export -in $HOSTNAME-combined.pem -out $HOSTNAME.p12 -name $HOSTNAME
Enter Export Password:secret
Verifying - Enter Export Password:secret
$sudo keytool -importkeystore -storetype JKS -destkeystore keystore.jks -srcstoretype PKCS12 -srckeystore $HOSTNAME.p12 -alias $HOSTNAME
Enter destination keystore password:write down this password
Re-enter new password:use the same as last time
Enter source keystore password:secret
$rm $HOSTNAME.p12 $HOSTNAME-combined.pem
$sudo chown activemq keystore.jks
$sudo chmod 0400 keystore.jks
The first password can be junk like secret because you are removing the file immediately after creating the keystore. You’ll need to save the keystore password as it will be used in the next section. It is best to use a completely random string like openssl rand -base64 20
and dropping the final character which is always an equals = sign.
Alternate ways of creating Java keystores can be found at MCollective >> Deploy >> Middleware >> ActiveMQ Keystores
Now in the /etc/activemq/activemq.xml let’s set up the SSL connector and the keystore definition.
<transportConnectors>
<transportConnector
name=
"stomp+ssl"
uri=
"stomp+ssl://0.0.0.0:61614"
/>
</transportConnectors>
<sslContext>
<sslContext
keyStore=
"ssl/keystore.jks"
keyStorePassword=
"
password from creation above
" /></sslContext>
Add the following lines to both the mcollective server and client configuration. The fallback parameter is required to tell MCollective to connect without its own private key information.
mcollective/server.cfg and mcollective/client.cfg.
plugin
.
activemq
.
pool
.
1.
port
=
61614
plugin
.
activemq
.
pool
.
1.
ssl
=
true
plugin
.
activemq
.
pool
.
1.
ssl
.
fallback
=
true
Once you have completed these steps, validate that the keystore matches up with your original key:
$keytool -list -keystore
Enter keystore password: Keystore type: JKS Keystore provider: SUN Your keystore contains 1 entry geode.netconsonance.com, Feb 16, 2014, PrivateKeyEntry, Certificate fingerprint (MD5): 61:EA:B9:63:BE:C9:AB:D7:C4:D2:2F:F3:D4:66:E2:43 $/etc/activemq/ssl/keystore.jks
cd
$/var/lib/puppet/ssl
/certssudo openssl x509 -in $( hostname ).pem -fingerprint -md5 | head -1
MD5 Fingerprint=61:EA:B9:63:BE:C9:AB:D7:C4:D2:2F:F3:D4:66:E2:43
After you enable TLS, test in the exact same manner as described in Testing Your Installation. You may also want to examine the packets in Wireshark
to confirm that SSL/TLS encryption was used.
A more complete security improvement is to authenticate connections between MCollective nodes and the Middleware using TLS keys and certificates. This option ensures that communications with your middleware is both encrypted and limited to access by pre-authorized client certificates. Access to the middleware will require both:
truststore
This is a good security model if you want to ensure that every host which connects to the middleware has been pre-approved for this connection. It has the added setup cost of creating and signing keys for each server or client which is connecting.
There are two different choices for doing TLS authentication. The first and easiest choice is to use the existing key infrastructure of Puppet. This requires very little extra configuration, and takes advantage of the existing Puppet Certificate Authority.
The second choice is to create or use your own Certificate Authority, and to manually sign each server and client key. This is more labor intensive.
We document both of these choices below.
In this section we will use a Puppet Certificate Authority and the existing Puppet SSL keys and certificates to secure access to the Middleware.
If you are using the puppet module provided with this book, you need only the following Hiera data and all three of the following steps will be done for you. No other changes are required. Once every node has accepted the changes, everything will be working over SSL.
mcollective::connector_ssl : true mcollective::connector_ssl_type : trusted mcollective::middleware::keystore_password :use a random string, right?
mcollective::middleware::truststore_password:another random string
Or if you are into old school expressive policies:
nodesomething-every-node-inherits
{ class { 'mcollective': connector_ssl => true, connector_ssl_type => 'trusted', } class { 'mcollective::middleware': keystore_password => 'use a random string, right?
', truststore_password => 'another random string
', } }
The Java trustStore will be created based on the Puppet server’s public key. We cannot use a public CA, as that would allow anyone signed by the public CA to connect to our system. One of the best sources of this information is the existing key infrastructure provided by Puppet. Here’s an example of creating the truststore using the Puppet CA.
$sudo mkdir /etc/activemq/ssl
$cd /etc/activemq/ssl
$sudo puppet agent --configprint ssldir
/var/lib/puppet/ssl $keytool -importcert -alias "
Enter keystore password:puppet.example.net
" -file/var/lib/puppet/ssl
/certs/ca.pem -keystore truststore.jks -storetype JKSwrite down this pass#1
Owner: CN=Puppet CA: puppet.example.net Issuer: CN=Puppet CA: puppet.example.net Serial number: 1 Valid from: Sat Jan 05 14:32:01 PST 2013 until: Fri Jan 05 14:32:01 PST 2018 Certificate fingerprints: MD5: F2:F8:C7:F7:45:92:DF:9A:BB:E0:0E:E9:F0:55:C6:B0 SHA1: 84:5C:ED:3F:03:C2:19:DC:F9:95:A8:0E:32:65:9D:0E:B5:A4:81:AC Signature algorithm name: SHA256withRSA Version: 3 ...blah Extensions blah ObjectId blah blah Identifier blah... Trust this certificate? [no]:yes
Certificate was added to keystore $sudo chown activemq truststore.jks
$sudo chmod 0400 truststore.jks
You’ll need to save this truststore password as it will be used in the middleware configuration files.
Next we want to set up the SSL keys used for negotiating the encrypted connection. The following example uses the existing Puppet keypair to create the Java keystore.
$mkdir /etc/activemq/ssl
$cd /etc/activemq/ssl
$sudo puppet agent --configprint ssldir
/var/lib/puppet/ssl $export HOSTNAME=$( hostname -f )
$sudo cat /var/lib/puppet/ssl/certs/$HOSTNAME.pem /var/lib/puppet/ssl/private_keys/$HOSTNAME.pem > $HOSTNAME-combined.pem
$openssl pkcs12 -export -in $HOSTNAME-combined.pem -out $HOSTNAME.p12 -name $HOSTNAME
Enter Export Password:secret
Verifying - Enter Export Password:secret
$sudo keytool -importkeystore -storetype JKS -destkeystore keystore.jks -srcstoretype PKCS12 -srckeystore $HOSTNAME.p12 -alias $HOSTNAME
Enter destination keystore password:write down this password
Re-enter new password:use the same as last time
Enter source keystore password:secret
$rm $HOSTNAME.p12 $HOSTNAME-combined.pem
$sudo chown activemq keystore.jks
$sudo chmod 0400 keystore.jks
The first password can be junk like secret because you are removing the file immediately after creating the keystore. You’ll need to save the keystore password as it will be used in the next section. It is best to use a completely random string like openssl rand -base64 20
and dropping the final character which is always an equals = sign.
Here we configure ActiveMQ to use the Java Truststore and Keystore we’ve created. You’ll need to put the passwords you used above in the appropriate places. You would add the following lines to the ActiveMQ configuration file /etc/activemq/activemq.xml:
<transportConnectors>
<transportConnector
name=
"stomp+ssl"
uri=
"stomp+ssl://0.0.0.0:61614?needClientAuth=true"
/>
</transportConnectors>
<sslContext
keyStore=
"ssl/keystore.jks"
keyStorePassword=
"
password #2
" trustStore="ssl/truststore.jks" trustStorePassword="password #1
" /></sslContext>
You would add the following lines to the mcollective server configuration server.cfg:
plugin.activemq.pool.1.ssl = true plugin.activemq.pool.1.port = 61614 plugin.activemq.pool.1.ssl.ca = /var/lib/puppet/ssl/certs/ca.pem plugin.activemq.pool.1.ssl.cert = /var/lib/puppet/ssl/certs/hostname
.pem plugin.activemq.pool.1.ssl.key = /var/lib/puppet/ssl/private_keys/hostname
.pem
If an existing Puppet Certificate Authority is not available or not appropriate, this section provides a process for creating a new CA, or using your existing CA, to secure access to the middleware.
Creating and managing SSL keys is a complex topic beyond the scope of this book. We have found the following commands to work when testing, but Your Mileage May Vary.
If you do not have an existing Certificate Authority to use, the following steps will create one:
$openssl genrsa -out CA_key.pem 2048
Generating RSA private key, 2048 bit long modulus ............+++ ...........................................................................+++ e is 65537 (0x10001) $openssl req -x509 -new -nodes -key CA_key.pem -days 10240 -out CA_cert.pem
You are about to be asked to enter information that will be incorporated into your certificate request. ...snip
Hide that key away in a dark place. It is now -literally- the key to access for your middleware. If that key is compromised you will have a very annoying task of replacing it and recreating each and every client and node certificate.
Next we create the Java Truststore from the public key of our Certificate Authority. We cannot use a public CA, as that would allow anyone signed by the public CA to connect to our system. Following is the process of creating the trustStore from the CA’s public key.
$keytool -importcert -alias "
Enter keystore password:MyCA
" -file CA_cert.pem -keystore truststore.jks -storetype JKSwrite down this pass#1
Owner: CN=My CA: myca.example.net Issuer: CN=My CA: myca.example.net Serial number: 1 Valid from: Sat Jan 05 14:32:01 PST 2013 until: Fri Jan 05 14:32:01 PST 2018 Certificate fingerprints: MD5: F2:F8:C7:F7:45:92:DF:9A:BB:E0:0E:E9:F0:55:C6:B0 SHA1: 84:5C:ED:3F:03:C2:19:DC:F9:95:A8:0E:32:65:9D:0E:B5:A4:81:AC Signature algorithm name: SHA256withRSA Version: 3 ...blah Extensions blah ObjectId blah blah Identifier blah... Trust this certificate? [no]:yes
Certificate was added to keystore $sudo chown activemq truststore.jks
$sudo chmod 0400 truststore.jks
You’ll need to save this trustStore password as it will be used in the middleware configuration files.
Next, create a new key for each host and sign it with your Certificate Authority. Be sure to avoid making the certificate expiration longer than the Certificate Authority’s lifetime.
$openssl genrsa -out
............+++ ...........................................................................+++ e is 65537 (0x10001) $node1
_key.pem 2048openssl req -new -key
You are about to be asked to enter information that will be incorporated into your certificate request. ...snip $node1
_key.pem -outnode1
.csropenssl x509 -req -days 5120 -set_serial 01 -CA CA_cert.pem -CAkey CA_key.pem -in
Signature ok subject=/C=node1
.csr -outnode1
_cert.pemUS
/ST=California
/L=San Jose
/O=Example
/CN=Node1
Getting CA Private Key
For each middleware broker you will need to create a Java keyStore. Simply combine the PEM files to generate a PKCS12 file, and create the keyStore from that.
$cd /etc/activemq/ssl
$sudo cat
$/etc/ssl/private/hostname
_key.pem/etc/ssl/certs/hostname
_cert.pem > combined.pemopenssl pkcs12 -export -in combined.pem -out combined.p12 -name $( hostname -f )
Enter Export Password:secret
Verifying - Enter Export Password:secret
$sudo keytool -importkeystore -storetype JKS -destkeystore keystore.jks -srcstoretype PKCS12 -srckeystore $combined.p12 -alias $( hostname -f )
Enter destination keystore password:write down pass#2
Re-enter new password:same as last time
Enter source keystore password:secret
$rm combined.p12 combined.pem
$sudo chown activemq truststore.jks
$sudo chmod 0400 truststore.jks
You’ll need to save the keystore password as it will be used in the next section.
Here we configure ActiveMQ to use the Java Truststore and Keystore we’ve created. You’ll need to put the passwords you used above in the appropriate places. You would add the following lines to the ActiveMQ configuration file /etc/activemq/activemq.xml:
<transportConnectors>
<transportConnector
name=
"stomp+ssl"
uri=
"stomp+ssl://0.0.0.0:61614?needClientAuth=true"
/>
</transportConnectors>
<sslContext>
<sslContext
keyStore=
"ssl/keystore.jks"
keyStorePassword=
"
password #2
" trustStore="ssl/truststore.jks" trustStorePassword="password #1
" /></sslContext>
You would add the following lines to the mcollective server configuration server.cfg:
plugin.activemq.pool.1.ssl = true plugin.activemq.pool.1.port = 61614 plugin.activemq.pool.1.ssl.ca =/etc/ssl/certs/
CA_cert.pem plugin.activemq.pool.1.ssl.cert =/etc/ssl/certs/node1
_cert.pem plugin.activemq.pool.1.ssl.key =/etc/ssl/private/node1
_key.pem
No matter which type of CA you chose to use, stop and validate that the keystore matches up with your original key. In the examples below we are referring to the Puppet cert locations, but the same commands work wherever you stored the certificate.
$keytool -list -keystore
Enter keystore password: Keystore type: JKS Keystore provider: SUN Your keystore contains 1 entry geode.netconsonance.com, Feb 16, 2014, PrivateKeyEntry, Certificate fingerprint (MD5): 61:EA:B9:63:BE:C9:AB:D7:C4:D2:2F:F3:D4:66:E2:43 $/etc/activemq/ssl/keystore.jks
sudo openssl x509 -in
MD5 Fingerprint=61:EA:B9:63:BE:C9:AB:D7:C4:D2:2F:F3:D4:66:E2:43/var/lib/puppet/ssl/certs/geode.netconsonance.com.pem
-fingerprint -md5 | head -1
Then validate that the trustStore was imported correctly:
$keytool -list -keystore
Enter keystore password: Keystore type: JKS Keystore provider: SUN Your keystore contains 1 entry puppet.example.net, Feb 16, 2014, PrivateKeyEntry, Certificate fingerprint (MD5): F2:F8:C7:F7:45:92:DF:9A:BB:E0:0E:E9:F0:55:C6:B0 $/etc/activemq/ssl/truststore.jks
sudo openssl x509 -in
MD5 Fingerprint=F2:F8:C7:F7:45:92:DF:9A:BB:E0:0E:E9:F0:55:C6:B0/var/lib/puppet/ssl/certs/ca.pem
-fingerprint -md5 | head -1
Each client requires their own certificate in order to connect. A common way would be to make each user have their own certificate. A perhaps less cumbersome way is to create certificates for each team, and ensure that only that team has access to the private key. You can likely think of other ways to break this out. Either way, you will need to create an SSL certificates for each unique entity you wish to validate.
Although this will be cumbersome, you will be able to re-use these certificates for Authorization as described in Chapter 11 as long as you name the certificate file the name you wish to use for authentication followed by .pem
.
One way to create client certificates is to generate them on a Puppet Master with puppet cert generate username
, and then copy them to the desired system. Unfortunately this requires each user generating a keypair to have root access on a puppetmaster. Moving the generated keys back and forth can also be problematic.
The process as documented at Configure MCollective Clients works well if you have shared network-mounted home directories.
I have found the following process much easier to perform for any logged in user.
client$puppet agent --certname
Info: Creating a new SSL key forusername
--serverpuppetmaster
--testusername
Info: Caching certificate for ca Info: csr_attributes file loading from /home/username
/.puppet/csr_attributes.yaml Info: Creating a new SSL certificate request forusername
Info: Certificate Request fingerprint (SHA256): 83:26:59:C9:A2:A4:93:97:79:... Info: Caching certificate for ca Exiting; no certificate found and waitforcert is disabled
At this point an admin on the puppetmaster need only issue one command:
puppetmaster$sudo puppet cert sign
Notice: Signed certificate request forusername
username
Notice: Removing file Puppet::SSL::CertificateRequestusername
at '/var/lib/puppet-server/ssl/ca/requests/username
.pem'
The user can run the following command and have their own key, cert, and ca as necessary for their config.
client$puppet agent --certname
Info: Caching certificate forusername
--serverpuppetmaster
--no-daemonize --no-client --verboseusername
Info: Caching certificate forusername
Notice: Starting Puppet client version 3.5.1 Error: Could not run: Daemons must have an agent, server, or both
That error is the exact response we want to get. The --no-daemonize --no-client
options at the end of the second command are necessary to prevent puppet from trying to run. If you don’t supply these options, Puppet agent will try to run a default catalog on your system. Logged in as a normal user that may not cause any problems, but it’s best to be avoided.
If you are considering using autosign
to avoid having to sign each user’s certificate, realize that this removes the trusted nature of this configuration. It would provide no more security than the anonymous
configuration and yet require so much more effort on your part.
The good news is that you can re-use these certificates for MCollective Authorization as described in Chapter 11.
Add the following lines to the {role="filename"]~/.mcollective file for the user, or any other configuration file specified with -c config
on the command line.
plugin.activemq.pool.1.port = 61614 plugin.activemq.pool.1.ssl = true plugin.activemq.pool.1.ssl.ca = /home/user
/.puppet/ssl/certs/ca.pem plugin.activemq.pool.1.ssl.cert = /home/user
/.puppet/ssl/certs/user
.pem plugin.activemq.pool.1.ssl.key = /home/user
/.puppet/ssl/private_keys/user
.pem
How about a sneaky little trick? Well, if a user has root
access on a puppetized machine, they will be able to access the middleware using the node’s puppet certificate. It’s a perfectly valid certificate signed by the same Puppet CA. This is one more reason that MCollective’s Authorization is so important.
I don’t generally feel that this is a security problem. If a node is part of the puppet framework, allowing it to connect to the middleware only makes sense. If they have root the server
login to the middleware is visible to them anyway. This is one of the many reasons that we ensure that the client passwords and permissions are distinct.
You’ll need to follow the instructions in this section if you don’t have or won’t be using the Puppet Certificate authority to sign client certificates.
Each user who wants to connect will need to generate a new keypair and then submit a signing request.
$mkdir -p .mcollective.d/certs .mcollective.d/private_keys
$cd .mcollective.d
$openssl genrsa -out private_keys/
............+++ ...........................................................................+++ e is 65537 (0x10001) $username
.pem 2048openssl req -new -key private_keys/
You are about to be asked to enter information that will be incorporated into your certificate request. ...snipusername
.pem -outusername
.csr
The user then submits the resulting CSR file to be signed. An administrator with access to that all-important private key will use the following commands to sign the user’s request with the Certificate Authority. Be sure to avoid making the certificate expiration longer than the Certificate Authority’s lifetime.
$openssl x509 -req -days 5120 -set_serial 01 -CA CA_cert.pem -CAkey CA_key.pem -in
Signature ok subject=/C=username
.csr -outusername
.pemUS
/ST=California
/L=San Jose
/O=Jo Rhett
/CN=jorhett
Getting CA Private Key
If you are considering allowing anyone to access and sign keys in order to reduce help requests, realize that this removes the trusted nature of this configuration. It would provide no more security than the anonymous
configuration and yet require more effort on your part. The ability to sign keys must be limited to people you can trust.
The good news is that you can re-use these certificates for MCollective Authorization as described in Chapter 11.
The resulting PEM file and the certificate CA_cert.pem should both be returned to the user. They should be placed in the directories as identified below. Add the following lines to the {role="filename"]~/.mcollective file for the user, or any other configuration file specified with -c config
on the command line.
plugin.activemq.pool.1.port = 61614 plugin.activemq.pool.1.ssl = true plugin.activemq.pool.1.ssl.ca = /home/user
/.mcollective.d/certs/CA_cert.pem plugin.activemq.pool.1.ssl.cert = /home/user
/.mcollective.d/certs/user
.pem plugin.activemq.pool.1.ssl.key = /home/user
/.mcollective.d/private_keys/user
.pem
There are two ways to secure the nodes connections to the ActiveMQ broker.
Both of these choices will protect the login credentials and the MCollective data from being sniffed on the wire.