Keeping Secrets Secure and Away from Your VCS Using Vault

This post is for those who find that their system’s secrets are misplaced or not well hidden.

Keeping your secrets well hidden and accessible to your applications can be hard. Sometimes we have them accessible, maybe we have them stored in our VCS, and other times we have them well hidden in the configuration files deployed on servers (and not backed up anywhere).

Having them stored in our VCS means that everyone with access to the repository will see them, which may not be the ideal case for you. On the opposite side, having secrets live only in the configuration files deployed onto the servers can mean that if the file is corrupted or deleted or if you lose access to the server, you won’t be able to get it back.

This should not be a zero-sum game but a positive-sum one. We should be able to have our secrets well hidden and accessible by whoever needs them, be it an application or a user.

Hashicorp’s Vault comes in handy with this process, providing a centralized place with encryption and auditing where we can store our secrets and manage their access. This will also help a lot when managing distributed systems, as you will have a service where all secrets are stored and fetched from.

Let me share a simplistic view of how to keep your secrets secure. I hope you have an easier time grasping these concepts after reading this!

I am responsible for the infrastructure of a small project to which I dedicate some of my time outside my real job. It has been a great experience.

Our tools for today are:

• Terraform
• Ansible
• Vault

This can be applied to any web application that needs to access a data store, locally or remotely. To do this, the application will need a token or a username/password combination to access it.

For the sake of this post, let’s pretend we have a Laravel app that needs access to a database.
Those database access secrets, in this case, a username and a password, which should be a secret to everyone, even us. (Okay, you will always be able to access them if you need, but with Vault, you can audit its access)

And that’s what we will be dealing with, how to keep your database credentials managed by these applications rather than doing it manually.

• There is a Vault already running at vault.ops.yourdomain.com:8200 that is reachable by the machine where we run Terraform and Ansible.
• Terraform and Ansible’s users have permissions to read/write/update secrets under Vault’s /secret/database path.

• Terraform generates username/password for the database.
• Terraform imports the secrets into Vault.
• Ansible fetches secrets from Vault.
• Ansible inserts secrets into .env file in the server.

You can check here for detailed documentation, but you just need to set Vault’s address and token, either by environment variables or by injecting them in the CLI command.

I chose the latter approach for simplicity:

After that, we need to generate the random strings which will be our username and password.

For this, we will use Terraform’s random_string resource.

Inserting values into Vault is pretty straight forward. It’s like inserting a json file into a path.

After this, we know there are 2 values inserted into Vault’s KV storage, which we didn’t need to manually generate or even know what they are (You can access them by the UI or the CLI).

Now let’s go to the Ansible part!

A nice approach is to have the variables attribution to fetch the credentials from Vault and then use the variables for its reference.

First, we are setting them with the hashi_vault lookup plugin, as well as the .env file path which is based on the application root directory.

Note: vault_domain is set as a global variable to all playbooks and vault_token is encrypted by ansible-vault.

Here we are telling Ansible to use the lineinfile module to ensure the DB_PASSWORD and DB_USERNAME keys are set with the correct values in a .env file.

After this, you should have a .env file populated with the fields coming directly from an encrypted, secure, and audited Vault.

How have you been hiding your secrets? Share with me your own way of doing it by leaving a comment below.

Also, if you have any questions or suggestions for improvements, please let us know!
Thanks a lot for reading 🙂

– – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – –