Anthony Critelli

Anthony Critelli

https://www.github.com/acritelli https://www.linkedin.com/in/anthonycritelli

Using Vault as a CA for Graylog

February 25, 2019

Overview

Graylog is a pretty sweet log management solution that allows you to quickly get up and running with centralized log collection and analysis. One common way to get your logs into Graylog is to use Filebeat, which can be further secured using TLS. Graylog even includes a handy Collector Sidecar for handling configuration.

Vault is an excellent secrets management tool created by Hashicorp. It includes the ability to easily set up a public key infrastructure, right out of the box. The Vault focus is on programmability: everything can be done via a reasonably straightforward and documented API. By focusing on programmatic access to the PKI, it becomes easily to automatically renew certificates and avoid certificates with excessively long lifetimes (a potential security risk, depending on your environment).

The goal of this article is to demonstrate the use of Vault as a CA for Graylog Filebeat inputs. We’ll set up Vault and create identities for each server in our environment so that they can submit their own CSRs and sign them, but only for their FQDN. This can allow us to create short lived certificates that are programmatically refreshed on a regular schedule. The focus will be on how this can be accomplished using the various bits and pieces of Vault, but we won’t actually write any of the automated pieces right now. We’ll do things manually to demonstrate the concepts, and it’s assumed that a reader with a Vault and Graylog environment is capable enough to write some scripts in the language of their choice to call the API endpoints that we’ll use and discuss.

Let’s start by looking at the topology:

Graylog and Vault Topology

The topology above consists of 3 servers:

  • graylog01: A single server running the full Graylog stack
  • vault01: A single server running Vault with a filesystem backend.
  • webtest01: An Apache webserver with the Graylog Collector Sidecar installed and shipping logs to graylog01

Prerequisites and House Keeping

Before we get started, there’s a few things about this article to be aware of:

This article assumes that you have a working Graylog, Vault, and web server environment with the Graylog Collector Sidecar installed and properly configured. Basic configuration of these services is outside the scope of this article, which will be solely dedicated to securing the Filebeat channel using Vault issued certificates. Likewise, it’s assumed that you’re comfortable working with Graylog and the Vault CLI/API.

Not much is done or explained in the area of tuning secret or certificate lifetimes. If you were implementing these concepts in production, then you absolutely want to understand the lifetimes of the things that you’re signing. It’s likely that a programmatically managed environment will benefit from very short certificate lifetimes.

You’ll also notice that, for the sake of clarity, I usually provide all of the command output for each step. Never, ever, ever reveal sensitive information (such as Vault tokens, passwords, etc.) in a production environment.

While we’re on the topic of security, you’ll notice that the Vault API endpoint is served over HTTP. Despite this being an article about certificates, I was in too much of a hurry to set up HTTPS for the actual Vault API. 0/10, you should set up TLS for Vault in production (or even dev).

Graylog Configuration Overview

First, let’s take a quick look at the way Graylog handles TLS Filebeat inputs for mutual authentication.

On the server side, you need to have a trusted CA certificate, a signed certificate for the local listener (i.e. for graylog01.acritelli.com), and a directory (or single file) that contains all certificates for clients that are allowed to connect. If you follow the Graylog documentation for securing Beats, then this is likely /etc/graylog/server/trusted_clients

The server side configuration for the Filebeat input looks something like this (taken from System > Inputs):

Graylog Filebeat Input

Note that I’ve truncated the screenshot a bit so that only the relevant SSL configuration is present.

On the collector side, you need to have the trusted CA certificate and the certificate for the collector (i.e. webtest01.acritelli.com). In my test environment, this is accomplished on the Graylog side in the Beats Output section of the Collector Configuration (taken from System > Collectors):

Graylog Filebeat Output

The client then automatically pulls down the appropriate configuration from Graylog.

With this configuration, the Graylog server (more specifically, the Filebeat input) and the Collector Sidecar Filebeat service will mutually authenticate. The collector will verify that the certificate presented by the Graylog server is signed by the trusted CA, and the Graylog server will verify that the certificate presented by the collector is contained in the /etc/graylog/server/trusted_clients directory. If either of these things fail, then the collector will be unable to send Filebeat logs to the server. Client authentication is particularly useful because, without it, anyone could send logs to Graylog.

Using Vault as the CA

Vault has a useful PKI secrets engine that can be used to issue certificates. Using the Vault CLI (or web API) can greatly simplify the process of requesting and generating certificates. In a simple setup, you could:

  1. Use Vault as your CA
  2. Add the CA certificate to the Graylog server and Collector Sidecar as trusted
  3. Manually generate certificates for the Graylog server and for each client
  4. Manually copy the client certificates to both the Collector Sidecar and to the Graylog server (in /etc/graylog/server/trusted_clients)
  5. Profit

Sweet. Aside from the ease of initial Vault CA setup, this is really no different than using your own CA with different tools, such as the set of scripts in the Graylog labs repository. The power of Vault really comes from the ease of automation, and the granular control that is provided both by Vault policies and by the PKI secrets engine roles. Let’s consider an easier, more automated workflow for client certificates using Vault (specifically, let’s consider allowing webtest01.acritelli.com to generate its own certificates):

  1. Use Vault as your CA
  2. Add the CA certificate to the Graylog server and Collector Sidecar as trusted
  3. Create a PKI role at /pki/roles/webtest01 that allows the collector sidecar to request certificates only for webtest01.acritelli.com
  4. Create a Vault Identity for the Collector Sidecar with an entity name of webtest01
  5. Create a set of Vault credentials and associate them with the Identity
  6. Create a policy that allows the Identity to write to /pki/sign/{{identity.entity.name}}
  7. Give the credentials to the Collector Sidecar
  8. The Collector Sidecar can generate CSRs and sign them, all by itself. Perhaps this runs as a regular cron job to refresh certificates.

The idea behind using both an Identity and a set of Vault credentials is to allow us to write one policy, using variable replacement for identity.entity.name. All of the steps below could be accomplished using some type of automation tool. However, this article will focus on the basic configuration. We’ll show the steps necessary to accomplish this configuration and leave it up to the reader to implement it in their language of choice. Just remember that anything typed at the command line could also be accomplished in your preferred automation tool.

The steps above allow the Collector Sidecar to generate new client certificates. But how do we get those into /etc/graylog/server/trusted_clients? Luckily, the PKI API has methods to both list and read certificates. We can grant these permissions to the Graylog Vault user, and then it can periodically list and pull down all of the certificates stored by Vault. Again, this might be accomplished by an occasional cron script.

Now that we have the background information and a good idea about our goals, we can start with the implementation.

Create the Root and Intermediate CA

We’ll start by creating the first two links in the chain of trust. While most “documentation” on the web will have you just create a root CA and directly sign certificates for this sort of thing, we’re going to try to do this using best practices with an intermediate CA. Ideally, you’ll also want to store your root private key offline. These steps will closely follow the Vault tutorial on building a CA. Also, as an important reminder: I’m not paying close attention to lifetimes in this tutorial, since it’s irrelevant for a temporary lab environment. If you decide to use something like this in production, be sure to understand your lifetimes.

First, we need to enable the PKI secrets engine (if it isn’t already enabled). I’m going to enable this at a particular path, since this PKI infrastructure will be used exclusively for Graylog:

root@vault01:~# vault secrets enable -path=graylog pki
Success! Enabled the pki secrets engine at: graylog/

Next, create the root CA (note that I’m skipping the creation of a CRL for this tutorial):

# Set PKI to issue a max of 365 days
root@vault01:~# vault secrets tune -max-lease-ttl=8760h graylog
Success! Tuned the secrets engine at: graylog/

# Create the CA cert with lifetime of 365 days
root@vault01:~# vault write -field=certificate graylog/root/generate/internal common_name="acritelli.com" ttl=8760h > CA_cert.crt

The generated CA certificate needs to be placed in /etc/graylog/collector-sidecar/ssl/ca.crt, or whatever location you have specified in your collector’s Filebeat configuration, on graylog01.

Next, we can create an intermediate CA that will actually sign our certificates:

# Enable the PKI engine on another path for the intermediate
root@vault01:~# vault secrets enable -path=graylog_int pki
Success! Enabled the pki secrets engine at: graylog_int/

# Let the intermediate sign certificates for a max of 4380 hours
root@vault01:~# vault secrets tune -max-lease-ttl=4380h graylog_int
Success! Tuned the secrets engine at: graylog_int/

# Generate an intermediate certificate
root@vault01:~# vault write -format=json graylog_int/intermediate/generate/internal \
 common_name="acritelli.com Intermediate Authority" ttl="4380h" \
         | jq -r '.data.csr' > pki_intermediate.csr

# Sign the intermediate with the root
root@vault01:~# vault write -format=json graylog/root/sign-intermediate csr=@pki_intermediate.csr \
         format=pem_bundle \
         | jq -r '.data.certificate' > intermediate.cert.pem

# Import the signed certificate
root@vault01:~# vault write graylog_int/intermediate/set-signed certificate=@intermediate.cert.pem
Success! Data written to: graylog_int/intermediate/set-signed

Generate a Certificate for Graylog

With a root and intermediate created, we can start signing certificates. Instead of manually signing certs, we’re going to give clients the ability to sign their own CSRs once they’ve authenticated to Vault. We’ll start with graylog01.

First, we’ll create a PKI role that only permits the Graylog server to request 30 day certificates for its FQDN:

root@vault01:~# vault write graylog_int/roles/graylog01 \
         allowed_domains="graylog01.acritelli.com" \
         max_ttl="720h" \
         allow_bare_domains="true"
Success! Data written to: graylog_int/roles/graylog01

Next, we can create a policy that allows Graylog to do a few things:

  • Sign CSRs only for its entity name
  • List all certificates
  • Read any certificate

The last two permissions will be needed when we start working with client certificates for mutual authentication. By allowing Graylog to pull down all certificates, it can automatically add devices to the trusted_clients directory.

# Create the policy
root@vault01:~# vault policy write graylog-server-policy graylog-server-policy.hcl 
Success! Uploaded policy: graylog-server-policy

# Contents of graylog-server-policy.hcl
root@vault01:~# cat graylog-server-policy.hcl 
path "graylog_int/sign/{{identity.entity.name}}" {
  capabilities = ["create", "update"]
}

path "graylog_int/certs" {
  capabilities = ["list"]
}

path "graylog_int/cert/*" {
  capabilities = ["read"]
}

With a policy created, we can add in an identity for graylog01 associated with the graylog-server-policy policy:

root@vault01:~# vault write /identity/entity name="graylog01" policies="graylog-server-policy"
Key        Value
---        -----
aliases    <nil>
id         b5446423-6dec-3ef7-edfa-10fcb028f255

I’m going to set this up with username and password authentication for the sake of simplicity. I tried getting this working with AppRoles, but they just don’t seem to work in this use case. As soon as I associated the app role with an alias, it completely broke the ability of the AppRole to log in. It could just be something that I was doing, but after 2 hours of frustration I gave up and decided to use simple user/password authentication.

# Create a username/password auth for graylog
root@vault01:~# vault write auth/userpass/users/graylog01 password="graylog01pass"
'Success! Data written to: auth/userpass/users/graylog01

# Get the mount accessor for the userpass authentication method
root@vault01:~# vault auth list -detailed
Path         Plugin      Accessor                  Default TTL    Max TTL    Token Type         Replication    Seal Wrap    Options    Description
----         ------      --------                  -----------    -------    ----------         -----------    ---------    -------    -----------
token/       token       auth_token_c03c3561       system         system     default-service    replicated     false        map[]      token based credentials
userpass/    userpass    auth_userpass_d48a56dd    system         system     default-service    replicated     false        map[]      n/a

# Create an entity alias. The canonical ID is taken from the entity that was previously created
root@vault01:~# vault write identity/entity-alias name=graylog01 canonical_id=b5446423-6dec-3ef7-edfa-10fcb028f255 mount_accessor=auth_userpass_d48a56dd                                                           
Key             Value
---             -----
canonical_id    b5446423-6dec-3ef7-edfa-10fcb028f255
id              c86aad88-1a8f-bd5c-1162-028cfb26d083

OK, our Graylog server should now have all of the permissions necessary to sign CSRs for itself. Let’s give this a try. First, we’ll create an OpenSSL config file with all of the information that we need:

root@graylog01:/etc/graylog/server/ssl# cat graylog01.cfg 
prompt                  = no

[ req ]
default_bits            = 2048
default_keyfile         = graylog01.key
distinguished_name      = req_distinguished_name
string_mask = utf8only

[ req_distinguished_name ]
countryName                     = US
stateOrProvinceName             = New York
localityName                    = Rochester
0.organizationName              = Some Great Company
0.organizationalUnitName        = Infrastructure Ops
commonName                      = graylog01.acritelli.com
emailAddress                    = email@example.com

We’ll then use this to generate a CSR:

root@graylog01:/etc/graylog/server/ssl# openssl req -new -nodes -config graylog01.cfg -out graylog01.csr
Generating a 2048 bit RSA private key
.........+++
.....................................................................................................+++
writing new private key to 'graylog01.key'
-----

root@graylog01:/etc/graylog/server/ssl# cat graylog01.csr 
-----BEGIN CERTIFICATE REQUEST-----
MIIC+DCCAeACAQAwgbIxCzAJBgNVBAYTAlVTMREwDwYDVQQIDAhOZXcgWW9yazES
MBAGA1UEBwwJUm9jaGVzdGVyMRswGQYDVQQKDBJTb21lIEdyZWF0IENvbXBhbnkx
GzAZBgNVBAsMEkluZnJhc3RydWN0dXJlIE9wczEgMB4GA1UEAwwXZ3JheWxvZzAx
LmFjcml0ZWxsaS5jb20xIDAeBgkqhkiG9w0BCQEWEWVtYWlsQGV4YW1wbGUuY29t
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA0aEENM7onHyiuQuc9oGk
Ye6aUcKxoNmQPuqaaG/Oz68tmlCTuGn7FRjRhwM32KFUBf0kZcDBUEVoiP6HPPy3
rMumRQcas73DIkScz+l3mY3rJy+AHFOV6u1x/lSru6cY+hroNCHBHRg2/zfGQnat
qrnBzkTdJyaxMe4aE+sODCR+Q6rterkR3OCCejoUkDhMuOP+Dsunq/HhDvA9us5B
3ZBQuxu0F9LJqkY5ECMK4nk6r24wjkBery6S5ApkdOjZY8/vfpnmMzY8FnSgGZ3u
X2HGu1pe53dds59yChAa7AznjOw8NEp0rVzmNMi0POfvicNOAVsU97SRoTbxZtfJ
wwIDAQABoAAwDQYJKoZIhvcNAQELBQADggEBAKQzk520s6aoU88ruwT6Ex5/rL09
oWWSUl6V6LavzBjnffTCZo4ouChvEUp22m/Bh505GiHRzzyNEijmXBiJjYPcng6T
3/rJUuhTLqEfe4LOods01voSbBPxc7wCE59i+ONRkf3vu5zW+vf8r5Z/DSz8fLuX
zGPHbc9D8u1FbuNXoD7Us+Ju6B4NLc1bwmA7Q9uvN47kVzznj+zVF5T+YYOG0cOb
Zzjvxu6wWY2UXVGtR+hYoC8B98R5ZlXaz2clkWyl3j7RTbRgAFpjFg6FbjAfqzMR
XDHLwkpLpeai2DDYHekPN86bZEU8sb9FKKmhSSk4xmGqPQozhqNQsg4j+eM=
-----END CERTIFICATE REQUEST-----
root@graylog01:/etc/graylog/server/ssl#

Finally, we can authenticate to our vault server, submit this CSR, and get back a certificate. Instead of trying to deal with the complete dumpster fire of trying to convert a CSR to JSON content for cURL, I opted to just install the vault CLI on the client for this tutorial. If you’re using a real scripting language, such as Python, you should have built in utilities to handle JSON serialization and passing it in requests.

# First we'll log in
root@graylog01:/opt# vault login -method=userpass username=graylog01 password=graylog01pass
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                    Value
---                    -----
token                  s.so6v19OUepvuD4WdxfmL2umc
token_accessor         pIQdkoh4NTFtr2XjOPtWpIau
token_duration         768h
token_renewable        true
token_policies         ["default"]
identity_policies      ["default" "graylog-server-policy"]
policies               ["default" "graylog-server-policy"]
token_meta_username    graylog01

# And then we can use our permissions to sign our CSR and get back the certificate
root@graylog01:/etc/graylog/server/ssl# vault write graylog_int/sign/graylog01 csr=@graylog01.csr
Key              Value
---              -----
ca_chain         [-----BEGIN CERTIFICATE-----

# --- Output snipped --- #

The returned certificate chain (certificate and issuing certificate) should be concatenated together and placed into graylog01.crt.

This takes care of obtaining a certificate on the Graylog server. Next, we’ll basically repeat this configuration to create a user and grant the appropriate permissions to the test webserver.

Generate a Certificate for webtest01

Our goal now is to allow the webserver to obtain certificates for its Collector Sidecar. This will allow the webserver to occasionally regenerate a certificate to use for mutual authentication with the Graylog server. Many of these steps will be repetitive based on what we accomplished previously.

First, we’ll create the PKI role. The webserver should only be allowed to request certificates for its FQDN:

root@vault01:~# vault write graylog_int/roles/webtest01 \
          allowed_domains="webtest01.acritelli.com" \
          max_ttl="720h" \
          allow_bare_domains="true"
Success! Data written to: graylog_int/roles/webtest01

Next, we’ll use the flexibility of Vault policies to create a generic policy for all sidecars. This policy will only allow the sidecars to do one thing: sign CSRs for their entity name. Unlike the Graylog server, the servers with sidecars on them don’t need the ability to list or read other certificates. Note that, when servers are added in the future, they won’t need to have an entirely separate policy written for them. Just the PKI role, identity association, and a set of credentials.

# Create the policy
root@vault01:~# vault policy write sidecar-policy sidecar-policy.hcl 
Success! Uploaded policy: sidecar-policy

# Contents of sidecar-policy.hcl
root@vault01:~# cat sidecar-policy.hcl 
path "graylog_int/sign/{{identity.entity.name}}" {
  capabilities = ["create", "update"]
}

With this policy created, we can add in an identity for webtest01 and associate it with the sidecar-policy policy:

root@vault01:~# vault write /identity/entity name="webtest01" policies="sidecar-policy"
Key        Value
---        -----
aliases    <nil>
id         8f369724-aa40-be54-64b7-3c9eb1ffda5a

Just like with the Graylog server, I’m going to use username and password authentication and associate it with the identity:

# Create the username/password authentication
root@vault01:~# vault write auth/userpass/users/webtest01 password="webtest01pass"
Success! Data written to: auth/userpass/users/webtest01

# Associate the username with the identity. Remember that canonical_id comes from the created identity in the previous step
# The mount accessor is also unchanged
root@vault01:~# vault write identity/entity-alias name=webtest01 canonical_id=8f369724-aa40-be54-64b7-3c9eb1ffda5a mount_accessor=auth_userpass_d48a56dd
Key             Value
---             -----
canonical_id    8f369724-aa40-be54-64b7-3c9eb1ffda5a
id              271ca33e-d4e7-0bfc-165c-e3ebe0aa6737

Now that all of the vault configuration is done, the webserver should now be able to sign CSRs for itself. First, we’ll create an OpenSSL config file similar to the one for the Graylog server:

# With the exception of the DN and CN, this is identical to the graylog config file
root@webtest01:/etc/graylog/collector-sidecar/ssl# cat webtest01.cfg 
prompt                  = no

[ req ]
default_bits            = 2048
default_keyfile         = webtest01.key
distinguished_name      = req_distinguished_name
string_mask = utf8only

[ req_distinguished_name ]
countryName                     = US
stateOrProvinceName             = New York
localityName                    = Rochester
0.organizationName              = Some Great Company
0.organizationalUnitName        = Infrastructure Ops
commonName                      = webtest01.acritelli.com
emailAddress                    = email@example.com

We’ll then generate the CSR:

root@webtest01:/etc/graylog/collector-sidecar/ssl# openssl req -new -nodes -config webtest01.cfg -out webtest01.csr
Generating a 2048 bit RSA private key
..............................................................+++
.................................................................................................+++
writing new private key to 'webtest01.key'
-----

root@webtest01:/etc/graylog/collector-sidecar/ssl# cat webtest01.csr 
-----BEGIN CERTIFICATE REQUEST-----
MIIC+DCCAeACAQAwgbIxCzAJBgNVBAYTAlVTMREwDwYDVQQIDAhOZXcgWW9yazES
MBAGA1UEBwwJUm9jaGVzdGVyMRswGQYDVQQKDBJTb21lIEdyZWF0IENvbXBhbnkx
GzAZBgNVBAsMEkluZnJhc3RydWN0dXJlIE9wczEgMB4GA1UEAwwXd2VidGVzdDAx
LmFjcml0ZWxsaS5jb20xIDAeBgkqhkiG9w0BCQEWEWVtYWlsQGV4YW1wbGUuY29t
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA37mrgtKMv2Jct0js3yOv
dG5T2c6jpIE/SJ+o4oyQEaWrcDWf6e+OuVofvf+JxseGUHTKuBpOqfxfdCDqYYdK
+Dp78yoagJHGmdP0hugF7jJY9oYoG3LS/wkNeuIzQZH140g6YSNmJzXIqTBJbSr/
7Jw1aXBVkVyjEHG4BjDmBH+OnI9oojpungJkzdqbTkO465x9ngywB8XnJRb2aZZA
UD9D2WM9Cy9uLTcbHMwYuDa5AcrNhxgeCS5OZUE4UY1t7U+W6leAeqyBJKoN3NCC
vmsvogSwOTzzj3AdniO1ObO516ybaeEWtroUHKOhJYlq3IDBN4O/3H4CLBZR0CQt
wwIDAQABoAAwDQYJKoZIhvcNAQELBQADggEBAM3gFdX9uEqBRAhE0SwJOAWR/m1z
/xi1+8Vebn8D6YHZEvmQ+JlWEsog37YSrheXy4+Vnc4i1lRDXR/krHqihterCH6O
+Ce/+wbgw0ZlmGjtgpVlx/sjPgaW7E/w0E2RCyjjqNqayQMZnR13gLwRTwaSb+0V
GIdZl1O7rQlz4yeoXQFDzmMuj4moH3SgSkKbrIF3tZdchRNBTukRE7XwieR8qhpm
OTTNNthgnlc7Aco2NWndRRU4GQSZXC+MOksqPwxHTDt4ODcY1RNTYE9k7l2d5HeH
wgHP3iAgPdrOrpIeIxZgiohQY6okPdTPhZrOZd+UxIjefergvxM8c6F8plw=
-----END CERTIFICATE REQUEST-----

Finally, we’ll authenticate to Vault, submit the CSR, and get back our certificate. Again, I’ve opted to just use the Vault CLI on the client.

root@webtest01:/etc/graylog/collector-sidecar/ssl# vault login -method=userpass username=webtest01 password=webtest01pass
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                    Value
---                    -----
token                  s.hlQoThs2Hy8y91dAqMHT1Vsc
token_accessor         5vzdiyYIWb50NneUoBBIZCTe
token_duration         768h
token_renewable        true
token_policies         ["default"]
identity_policies      ["sidecar-policy"]
policies               ["default" "sidecar-policy"]
token_meta_username    webtest01

root@webtest01:/etc/graylog/collector-sidecar/ssl# vault write graylog_int/sign/webtest01 csr=@webtest01.csr
Key              Value
---              -----
ca_chain         [-----BEGIN CERTIFICATE-----

# --- Output snipped --- #

The returned certificate and issuing certificate should be concatenated together and placed into /etc/graylog/collector-sidecar/ssl/webtest01.crt.

Getting the CA Certificate on webtest01

Recall that the Collector Sidecar also needs to know the certificate for the CA. This is actually kind of a pain, because you can’t get the CA certificate from the Vault CLI. For this tutorial, I’m just going to grab it via cURL. However, since you’d normally be doing this programmatically, this isn’t a big deal anyway.

# Authenticate to Vault and get a token
root@webtest01:~# curl --request POST --data '{"password": "webtest01pass"}' http://vault01.acritelli.com:8200/v1/auth/userpass/login/webtest01

{"request_id":"35bc5ba1-6de9-b2ce-42eb-7ed6f3b64165","lease_id":"","renewable":false,"lease_duration":0,"data":null,"wrap_info":null,"warnings":null,"auth":{"client_token":"s.sCpC8jqlmHsDXFzS1cE7pkIo","accessor":"gwXCUihqG3Glimpsu2Hetrb0","policies":["default","sidecar-policy"],"token_policies":["default"],"identity_policies":["sidecar-policy"],"metadata":{"username":"webtest01"},"lease_duration":2764800,"renewable":true,"entity_id":"8f369724-aa40-be54-64b7-3c9eb1ffda5a","token_type":"service"}}

# Use the token to get the PEM encoded CA certificate
root@webtest01:~# curl --request GET \
> --header "X-Vault-Token: s.sCpC8jqlmHsDXFzS1cE7pkIo" \
> http://vault01.acritelli.com:8200/v1/graylog/ca/pem
-----BEGIN CERTIFICATE-----
MIIDOzCCAiOgAwIBAgIUYNJAnB1Yv5YwXJEDYxtnJO0mmIwwDQYJKoZIhvcNAQEL
BQAwGDEWMBQGA1UEAxMNYWNyaXRlbGxpLmNvbTAeFw0xOTAxMjkwMjQ2MTFaFw0y
MDAxMjkwMjQ2NDFaMBgxFjAUBgNVBAMTDWFjcml0ZWxsaS5jb20wggEiMA0GCSqG
SIb3DQEBAQUAA4IBDwAwggEKAoIBAQD5M6yOUB6ylXIK86ig7Mh1dsXuZ9KBhgmG
11WRT/opRd8/+8humlQqVRNuJpcjmH6RNP/twrExtnhjc7UNCCLBaUUWzjrLAl3u
JQv1F5WgZqKeI8XxVzMtR4+WM0fv61U92h1nZLEHkc87OMsY5mn4j8U6rdEuSNhA
EigjDOX09M/8ywDRwEy4t5X3eaP43++VlFpLooMluWQLmadiW3dOnMiNHBu4puQ0
L0zWfH15aBRVxh+qT71uGjXwWjQAMhB5DlUY7Z1+39p84tH3+6X4LHvVqsfut15Z
lXk/cMQBqHSqh/iLxu5hm1ITaCRc7/78hqA0pcXB5vNVxwoPeBFBAgMBAAGjfTB7
MA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBSH9enn
MQrr3TXO8E8BIkD0jCLDcDAfBgNVHSMEGDAWgBSH9ennMQrr3TXO8E8BIkD0jCLD
cDAYBgNVHREEETAPgg1hY3JpdGVsbGkuY29tMA0GCSqGSIb3DQEBCwUAA4IBAQAv
asMz09lPBBtRSmQSY+E2neqvsaIqMX2JjCghghpPlJltzB0WtF/dg+TADZ7rHlJL
QxzRv1Yhc6C6mqtiGnAZyNHVct7YIKh+0CcqKb0Vb87uktOm9rmWsmqnCYO3kK3i
2RCBRVfdOmR3iK+6R8U4ofYSUUyrMacy8vbAHMy1cIH4n9riUlPC8Q7jV7fCGH1h
J/uNyl8tdPRUKYQUd5jmdXSlvqViXQGR8MGCj4GovsPoP7QdJXXKA0ruAMcJVml/
UuRw0dpGEDloms76LhPWTZNV29npU4Cw/E6FdwyXxgvivroi9OYBCbwAOw6KfSGD
k3Fl7bFQyrYl5I4/jyd0
-----END CERTIFICATE-----

The certificate should be placed in /etc/graylog/collector-sidecar/ssl/ca.crt per the configuration seen at the beginning of this article.

Downloading the Client Certificates on the Graylog Server

Our Graylog input configuration specifies that a client’s certificate must be present in the trusted_clients directory for two-way authentication to work. Recall that the Vault policy for our Graylog server allows for listing and reading all of the certificates in Vault. This allows the Graylog user to grab all of the certificates for clients and add them into the trusted_clients directory. Let’s give that a shot.

First, we’ll get a list of all certificates (note that I already logged in via vault login):

root@graylog01:/etc/graylog/server/ssl# vault list graylog_int/certs
Keys
----
0d-32-75-40-06-b3-bf-6d-4a-34-92-3e-07-40-7f-23-74-20-60-a7
1d-77-0b-ed-8f-60-b7-b5-0b-5d-ec-06-13-96-44-cc-41-9c-9d-01
2a-86-6a-ed-d1-1f-5b-ef-12-f2-31-9d-fc-09-82-8f-82-94-45-63
2c-72-86-5d-7d-8b-d3-3a-40-04-66-fd-8f-ad-91-96-8a-f1-99-d1
51-bc-79-26-26-bb-23-2d-17-69-78-37-9e-9f-23-94-a8-00-6f-9a
64-e5-49-c9-cb-fd-d6-fa-ee-5d-15-d3-57-5e-5a-dd-2e-6a-8f-b3
67-ff-54-3c-36-5b-07-77-15-06-62-20-3f-32-d0-a6-ca-c4-39-ae

There are obviously a bunch of certificates, listed by their serial numbers. If you’re following along at home, you’ll likely see fewer (several are just from my experimentation). In an end implementation, we would programmatically iterate over all of the serial numbers and grab the corresponding certificate. Instead, I already know the serial number for webtest01, so I’m just going to grab that particular cert.

If you want to know how to grab the serial number of the certificate, then that’s pretty easy. On webtest01:

root@webtest01:/etc/graylog/collector-sidecar/ssl# openssl x509 -in webtest01.crt -noout -text | grep -A 1 -i serial
        Serial Number:
            51:bc:79:26:26:bb:23:2d:17:69:78:37:9e:9f:23:94:a8:00:6f:9a

And then we can just grab the individual certificate with that serial number:

root@graylog01:/etc/graylog/server/ssl# vault read graylog_int/cert/51-bc-79-26-26-bb-23-2d-17-69-78-37-9e-9f-23-94-a8-00-6f-9a
Key                Value
---                -----
certificate        -----BEGIN CERTIFICATE-----
MIIDdjCCAl6gAwIBAgIUUbx5Jia7Iy0XaXg3np8jlKgAb5owDQYJKoZIhvcNAQEL
BQAwLzEtMCsGA1UEAxMkYWNyaXRlbGxpLmNvbSBJbnRlcm1lZGlhdGUgQXV0aG9y
aXR5MB4XDTE5MDIyNDIxMDAxMFoXDTE5MDMyNjIxMDA0MFowIjEgMB4GA1UEAxMX
d2VidGVzdDAxLmFjcml0ZWxsaS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw
ggEKAoIBAQDfuauC0oy/Yly3SOzfI690blPZzqOkgT9In6jijJARpatwNZ/p7465
Wh+9/4nGx4ZQdMq4Gk6p/F90IOphh0r4OnvzKhqAkcaZ0/SG6AXuMlj2higbctL/
CQ164jNBkfXjSDphI2YnNcipMEltKv/snDVpcFWRXKMQcbgGMOYEf46cj2iiOm6e
AmTN2ptOQ7jrnH2eDLAHxeclFvZplkBQP0PZYz0LL24tNxsczBi4NrkBys2HGB4J
Lk5lQThRjW3tT5bqV4B6rIEkqg3c0IK+ay+iBLA5PPOPcB2eI7U5s7nXrJtp4Ra2
uhQco6EliWrcgME3g7/cfgIsFlHQJC3DAgMBAAGjgZYwgZMwDgYDVR0PAQH/BAQD
AgOoMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAdBgNVHQ4EFgQUVmkW
OfEq0L4btP1dlysbrgoqtoIwHwYDVR0jBBgwFoAUT4OlhNoE1dusNYZPpkwIVM0U
4ucwIgYDVR0RBBswGYIXd2VidGVzdDAxLmFjcml0ZWxsaS5jb20wDQYJKoZIhvcN
AQELBQADggEBAH7Ucgp5cMiwZk+ZhorFLo19xEhW5cS3B8akhOARYvgEAifeo0Df
MWQnOFQa7+B8ERWkRQdnYU03+CLaQ+DDY7zIipoY7iwK9R+XLnxCL3XfmgBNIfP/
ZZImoCoEr6c1XJG++pr3eQ2fi1FBojlqQ9anO8suUZYf77zEsjHUBu5Gw9lKOhVQ
QpP7Z9tb+svSdfNqRYRMzDdlXXKAXln5RD9UPchLAC5jEuGfBs/3ZCh/5W3/jJTq
VH+gd/xWE+eLXlkuTa4/UXEqG3FFIoZNfjBny4mpaiaB4RL00Y+4tBzXypQg+4+n
ICl/Mh6VXeMJEHKd4ZjMf9EBSVQAYiNBnbI=
-----END CERTIFICATE-----
revocation_time    0

That certificate should be placed into a file in /etc/graylog/server/ssl/trusted_clients. It doesn’t matter what you call the file, but I’ll call it webtest01.crt.

It’s also necessary to grab the certificate for the intermediate CA (who signs all of these certificates). Again, I happen to have noted down the serial number, so I’ll just grab this and throw it into /etc/graylog/server/ssl/trusted_clients/intermediate.crt:

root@graylog01:/etc/graylog/server/ssl# vault read graylog_int/cert/64-e5-49-c9-cb-fd-d6-fa-ee-5d-15-d3-57-5e-5a-dd-2e-6a-8f-b3
Key                Value
---                -----
certificate        -----BEGIN CERTIFICATE-----
MIIDODCCAiCgAwIBAgIUZOVJycv91vruXRXTV15a3S5qj7MwDQYJKoZIhvcNAQEL
BQAwGDEWMBQGA1UEAxMNYWNyaXRlbGxpLmNvbTAeFw0xOTAyMjQyMDU5NDlaFw0x
OTAzMjgyMTAwMTlaMC8xLTArBgNVBAMTJGFjcml0ZWxsaS5jb20gSW50ZXJtZWRp
YXRlIEF1dGhvcml0eTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEl
oelApVkZmbjuvTp2O3XP/PLyfeWfLPtefyYJiUXzk8y2io7vVroUy9EbAoEjpf9x
Il8fulGpQlXnQq+QfAbHUg8HNxOOyZDgGOPl0/wPNrcdNeCdBwcf1KAPkRjHqdps
dF4CTx4CjIQBjURVIlgTAM4IFZ747attV8uewcphOk+YCVim3++bmhwpspOapqpH
za2IwNwc4IKRfL8rc2jYp0fnbAj/TWRYrYgRZYYr8bnTBWVzoOypPs4CRnEgwMkt
rQ2nhxcbf2n9fzASXeRlj43NDwOrR6PXBw79K7bj0K1M6Ow6yyKnGcwQ8cTrSSiU
6kQQnDUxhsehpXxQq0cCAwEAAaNjMGEwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB
/wQFMAMBAf8wHQYDVR0OBBYEFE+DpYTaBNXbrDWGT6ZMCFTNFOLnMB8GA1UdIwQY
MBaAFIf16ecxCuvdNc7wTwEiQPSMIsNwMA0GCSqGSIb3DQEBCwUAA4IBAQAZ8VwE
oXTJe7cB5aRq/H5+yPa9XSQJNWa4mz9hPuJ3aEawP5tngo4nkBD704EC5GhQ+ckb
5IouDkNuS6Cg2JnjAOZwqk7ZN3aGjwmmHV91iMpKahIMV3B5kvN0XGzMx6baq/M3
5FGLZ+OIi/Nt1uX1QJO2h1dtmUAeLrepwN36lgzrVhaBASIigPsl5kTBZTrWKOrf
T1Qo/mH+051SpoUaXvX1PRb1rzkoC3tuSZUv6meRhas9gD1SaxRxmNsNV1RvJdVP
au0jIkpvFktT4X7ygquh/0eV6ImcQ+NHCQrTNixp6sMDbynG01u08OFaULYp551c
tIN+mFrmff/FkT/V
-----END CERTIFICATE-----
revocation_time    0

Again, I want to emphasize: an end implementation of this would perform these steps programmatically and pull down all of the certificates into this directory.

Wrapping Up

At this point, everything is in place for Graylog and the Collector Sidecar on webtest01 to start talking to each other. If try navigating to a few nonexistent URLs on webtest01, then we should see the 404 messages show up in Graylog:

Graylog Log Messages

If you’re following along and don’t see your log messages showing up, then the Graylog log files are usually pretty helpful:

  • On the server side: /var/log/graylog-server/server.log
  • On the Collector Sidecar side: /var/log/graylog/collector-sidecar/filebeat and /var/log/graylog/collector-sidecar/collector_sidecar.log
    • Note that if you tail -f these files while reloading the service, you may need to break out of the tail and re-tail the log file.

If you really want to verify that everything is working over TLS, then you can take a quick packet capture and confirm that you see the TLS handshake and encrypted traffic.

And that’s about it! Vault provides a pretty great interface for handling PKI functions. You can use this to programmatically sign certificates for your Graylog deployment, allowing for more frequent certificate rotation and granular signing permissions. Hopefully some variation of this article might be helpful in your environment (especially if you take the time to actually script out your interactions with the Vault API).

Previous article: Monitoring VMware with Icinga

Next article: Automating Vault and Consul Template Management