Config you know plus some love

# Config you know [+]() some Love

### by [@tolitius](http://twitter.com/tolitius)

---

---

![define configuration](img/def-configuration.png)
--
<br/><br/><br/>
.center[_"most beautiful molecules are exquisitely configured"_]

---

![dna config](img/dna-config.png)

---
class: middle,center, inverse
# Simulation Argument

---
class: middle
<h2 style="text-align:center">Simulation Argument</h2>

In 2003, philosopher Nick Bostrom proposed a trilemma that he called "the simulation argument"

At least [one]() of the following three propositions [is almost certainly true]()

* the human species is very likely to go extinct before reaching a "posthuman" stage

* any "posthuman" civilization is extremely unlikely to run a significant number of simulations

* we are almost certainly living in a computer simulation

<p style="text-align:center; margin-top:40px; font-size:0.6em">
  <em>(April 2003): "
      <a href="http://www.simulation-argument.com/simulation.pdf" style="color:rgba(23, 125, 210, 0.97);">Are You Living in a Computer Simulation?</a>
      ". Philosophical Quarterly. 53 (211): 243–255
  </em>
</p>

---
class: center
<h2 style="margin: 60px 0px 0px 0px">1984 Configs</h2>

If we are to [simulate reality](), _the whole universe_, or multiples of it, of, say

the year of

<p style="text-align:center">
  <img src="img/1984.png" style="width:400px"/>
</p>
<br/>
It will need to be ... [configured]()

---
class: middle, center
## "[pre]() posthuman" configuration

is:

> "_port numbers, host names and some versions_"

> "_so let's make it flat, key value and strings only_"

--
```
¯\_(ツ)_/¯
```

---
class: middle,center,inverse
### _let's talk about_
# Twelve Factor Config

---
class: middle,center
## Twelve Factor Config

> "_The twelve-factor app stores config in environment variables_"

Take the above "[1984 world simulation]()" in different environments:

<div style="margin-left: 33%">
  <ul style="text-align: left">
      <li>Earth</li>
      <li>Mars</li>
      <li>Horsehead Nebula</li>
  </ul>
</div>

Controlled by [ENV]() variables? :)

---
class: center
<h2 style="margin: 64px 0px 0px">Twelve Factor Config</h2>

ENV variables [security]() and [isolation]() story in Docker:

<p style="text-align:center; margin-top:40px; font-size:0.6em">
  <em>docker/docker, open issue:
      "<a href="https://github.com/docker/docker/issues/13490#issue-81134963" style="color:rgba(23, 125, 210, 0.97);">Secrets: write-up best practices, do's and don'ts, roadmap</a>"
  </em>
</p>
---
<h2 style="text-align: center;margin:18% 0px 0px;">Twelve Factor Config</h2>
<h5 style="color:rgba(15, 129, 15, 0.73);
           text-align: center;
           margin: 8px 0px 40px 0px;"><em>the good</em></h5>
.center[[don't]() keep creds in sources]
<div style="margin: 0 auto;
            margin-top: 20px;
            max-width: 59%;
            border: solid 1px rgba(22, 22, 22, 0.05);"></div>
<div style="margin: 20px 0px 0px 53px">
    <em style="color: rgba(18, 18, 18, 0.73); margin-left: 20px">
        "A litmus test for whether an app has all config correctly factored out of the code is whether the codebase could be made open source at any moment, without compromising any credentials."
    </em>
</div>

---
<h2 style="text-align: center;margin:18% 0px 0px;">Twelve Factor Config</h2>
<h5 style="color:rgba(15, 129, 15, 0.73);
           text-align: center;
           margin: 8px 0px 40px 0px;"><em>the good</em></h5>
.center[[don't]() create "profile"/"env" aware configs]
<div style="margin: 0 auto;
            margin-top: 20px;
            max-width: 59%;
            border: solid 1px rgba(22, 22, 22, 0.05);"></div>
<div style="margin: 20px 0px 0px 53px">
    <em style="color: rgba(18, 18, 18, 0.73); margin-left: 20px">
        i.e. do not store profile / env specific values in the <strong>core</strong> app configs
    </em>
</div>

---
<h2 style="text-align: center;margin:18% 0px 0px;">Twelve Factor Config</h2>
<h5 style="color:rgba(15, 129, 15, 0.73);
           text-align: center;
           margin: 8px 0px 40px 0px;"><em>the good</em></h5>
.center[[don't]() create mutiple config files for different "profiles"/"env"]
<div style="margin: 0 auto;
            margin-top: 20px;
            max-width: 83%;
            border: solid 1px rgba(22, 22, 22, 0.05);"></div>
<div style="margin: 20px 0px 0px 53px; color: rgba(18, 18, 18, 0.73);">
    <p style="margin-top: 20px"><em>"This method does not scale cleanly:</em></p>
        
    <p style="margin-left: 20px"><em>as more deploys of the app are created, new environment names are necessary, such as "<span style="color: rgba(32, 133, 208, 0.72);">staging</span>" or "<span style="color: rgba(32, 133, 208, 0.72);">qa</span>".</em></p>

<p style="margin-left: 20px"><em>As the project grows further, developers may add their own special environments like "<span style="color: rgba(32, 133, 208, 0.72);">joes-staging</span>", resulting in a combinatorial explosion of config which makes managing deploys of the app very brittle."</em></p>
</div>

---
class: middle, center, inverse
# Configuration Whereabouts

---
class: middle
<h2 style="text-align: center">Configuration Whereabouts</h2>

* Local Files

* System Properties

* ENV Variables

* Remote Tools (i.e. Zookeeper, Consul, etcd, etc.)

* Secure Stores (Vault, encrypted in DB, keystores, secure fs mounts, etc.)

* Database.. of 1984 configs

---
class: middle, center, inverse
# Config is Data

---
class: middle, center
## Config is Data

from "`port=42`" to the whole "virtual word" configuration

<div style="text-align: left;
            margin: 30px 0px 0px 150px;
            padding-left: 20px;
            border-left: solid rgba(76, 216, 16, 0.68) 5px">
    <p>If a model of the Universe is <em><a href="#">hierarchical</a></em></p>
    <p>so should be the data that configures the model </p>
</div>

---
class: middle, center
## Config used Efficiently

<div style="text-align: left;
            margin: 0px 0px 0px 80px;
            padding-left: 20px;">
    <p>In computer science, a <a href="#">data structure</a> is a particular way of</p>
    <p>organizing <em>data</em> in a computer so that it can be used efficiently</p>
</div>

---
<h2 style="text-align:center; margin: 145px 0px 5px">Don't introduce Yet Another Language in configs</h2>

<div style="text-align: left;
            margin: 30px 0px 30px 150px;
            padding-left: 20px;">
    <p style="margin-bottom: 20px">Say "<em>no</em>" to DSL for configs</p>
    <p style="margin-top: 0px">Say "<em>yes</em>" to <a href="#">just data</a> (structures) for configs</p>
</div>

<ul style="font-size: 0.9em; line-height: 35px; margin-left: 90px">
    <li>more people are able to collaborate</li>
    <li>more tools are able to process</li>
    <li>DSLs are harder to validate</li>
    <li>data structures have well defined language level API</li>
    <li>"just data" is better understood by operation/devops teams</li>
</ul>

---
class: middle,center
## Easy to Override

If configuration is a Clojure [data structure]()<br/>it can be overriden with Clojure APIs.

It's simple.

---
<h2 style="text-align:center; margin: 60px 0px 20px">Portability</h2>

<div style="text-align:center; line-height: 10px; margin-bottom: 32px">
    <p>If configuration is <a href="#">data structure</a> it can be ported to any format</p>
    <p>(transit, protobufs, json, etc.) using standard libs</p>
</div>

```clojure
=> (def config {:backoff {:initial-wait-ms 1000
                          :max-wait-ms 120000
                          :wait-fn '(fn [ms] (* ms 2))}})
```

```clojure
=> (transit-out config)

#object["[B" 0x3cc03a51 "[B@3cc03a51"]
```

```clojure
=> (-> config
       transit-out 
       transit-in)

{:backoff {:initial-wait-ms 1000,
           :max-wait-ms 120000,
           :wait-fn (fn [ms] (* ms 2))}}
```

---
class: center, middle
## Config Should be Queryable

with Clojure, Specter, Odin, etc.
 
---
class: middle
### "What is a web server port?"

```clojure
{:hubble {:server {:port 4242}
          :store {:url "spacecraft://tape"}
          :camera {:mode "mono"}
          :mission {:target "Eagle Nebula"}

:log {:enabled false
                :hazelcast {:hosts "OVERRIDE ME"
                            :group-name "OVERRIDE ME"
                            :group-password "OVERRIDE ME"
                            :retry-ms 5000
                            :retry-max 720000}}}}
```

Simple Clojure APIs handle a large number of queries:

```clojure
=> (get-in config [:hubble :server :port])

4242
```

---
class: middle
### "What data stores are we connected to?"

```clojure
;; contrived example

{:nrepl {:port 4242},
 :web-server {:port 8080, :threads 42},
 :store
 {:bank
  {:url "jdbc:postgresql://user:pass@localhost:5432/open-source-bank"},
  :audit
  {:url
   "datomic:sql://iremember?jdbc:oracle:thin:datomic/datomic@10.20.30.42:1521:zurg"},
  :customers
  {:url
   "jdbc:mysql://localhost:8889/db_name?user=boss&password=secret"}}}
```

Specter is great at querying:

```clojure
;; Specter
=> (select [MAP-VALS MAP-VALS] (config :store))

["jdbc:postgresql://user:pass@localhost:5432/open-source-bank"
 "datomic:sql://iremember?jdbc:oracle:thin:datomic/datomic@10.20.30.42:1521:zurg"
 "jdbc:mysql://localhost:8889/db_name?user=boss&password=secret"]
 ```

---
class: middle
### "What ports does the application bind to?"

```clojure
{:nrepl {:port 4242},
 :web-server {:port 8080, :threads 42},
 :store
 {:hazelcast {:hosts ["192.168.0.12" "192.168.0.13"], :port 5701}}}
```

Specter is really great at querying:

```clojure
;; Specter
(select [(walker :port) :port] m)
```

```clojure
[4242 8080 5701]
```

---
class: middle,center
## Validation

Confings like any other respectful [data structure]() need to be validated

```clojure
(s/conform specification config)
```

Want to "make sure"? Spec it.

---
class: middle
<h2 style="text-align:center">Filter Exposed Configuration</h2>

filter, slice, hide certain parts of the configuration (i.e. `/config`  endpoint).

In case configuration is just a [data structure](), Clojure API or standard query libs could be used to do that.

```clojure
=> (require '[cprop.tools :refer [cloak]])

=> (def config {:foo {:bar {:auth-token "0891c7f2-03aa-4f37-a0b0-3a5f14b773b9"}}
                :baz {:dont-expose-me "I should not be seen"}
                :other-things {:a 42}})
```

```clojure
=> (cloak config [:foo :bar :auth-token]
                 [:baz :dont-expose-me])

{:foo {:bar {:auth-token "*******"}},
 :baz {:dont-expose-me "*******"},
 :other-things {:a 42}}
```

---
class: middle
<h2 style="text-align:center">Config: "Allow me to be Functional"</h2>

_from "[Programming in Lua](https://www.lua.org/pil/25.2.html)":_

> A great strength of Lua is that a configuration file can define functions to be called by the application.

The "enterprise reaction":

> no, no and no. It's a huge security risk.

---
class: middle
<h2 style="text-align:center">Config: "Allow me to be Functional"</h2>

But it does not have to be:

```lua
-- make (sandbox) environment
-- add functions you know are safe here:
local env = { tonumber = tonumber, 
              tostring = tostring,
              -- ...               
              }

-- run in a sandbox environment
local function run(untrusted_code)
  local untrusted_function, message = load(untrusted_code,
                                           nil,
                                           't',
                                           env)
  if not untrusted_function then return nil, message end
  return pcall(untrusted_function)
end
```

---
class: middle
<h2 style="text-align:center">Config: "Allow me to be Functional"</h2>

Depending on your use case / environment / language, adding functions to your configuration _could_ definitely be a security risk.

If it is not, then functions are beautiful beings:

```clojure
{:backoff {:initial-wait-ms 1000
           :max-wait-ms 120000
           :wait-fn '(fn [ms] (* ms 2))}}
```

---
class: middle
<h2 style="text-align:center">Declarative vs. Programmable Configuration</h2>

Leiningen

> `project.clj` _(declarative)_

Boot

> `build.boot` _([programmable]())_

An example where "build _configuration_" is mostly _functions_ and it's beautiful

---
class: middle
## Config: "Version me"

<div style="margin: 0 auto;
            margin: 20px 0px;
            border: solid 1px rgba(22, 22, 22, 0.05);"></div>
we version code with version control tools (i.e. git)

configuration should not be any different

---
class: middle
<h2 style="text-align:center">Config: "Version me"</h2>

### core configuration

> lives and is versioned with the source code

### overrides

> dev/qa/stage may or may not live in source code<br/>
> but _should also be versioned_

> production overrides should live _outside_ of the source code<br/>
> but _should also be versioned_ (i.e. [git2consul](https://github.com/Cimpress-MCP/git2consul))

---
class: middle, center
## Config: "Let me explain"

Should have comments (i.e. `JSON` does not)

---
class: middle, center
## Config: "Change me over (run) time"

<p style="text-align:left; margin-left: 175px">
- "..and make sure there is no downtime."</br>
- "done."
</p>

![nginx config reload](img/nginx-config-reload.png)

---
class: middle
<h2 style="text-align:center">Coordination</h2>
<img style="width:225px" src="img/zoo-cluster.png">
<img style="width:245px;margin: 0px 10px 0px 25px" src="img/consul-cluster.png">
<img style="width:225px" src="img/etcd-cluster.png">

---
class: middle
<h2 style="text-align:center">Security</h2>

---
class: middle, center
<h1 style="text-align:center">Hubble Space Mission Securely Configured</h1>

---
class: middle
<h2 style="text-align:center">cprop: (load-confg)</h2>

* loads an EDN config from a classpath and/or file system

* merges it with system properties and ENV variables + the optional merge from sources (file, db, mqtt, http, etc.)

* returns an (immutable) map

* while keeping no internal state => different configs could be used within the same app, i.e. for app sub modules

<p style="margin-top: 40px; text-align:center">
    <a href="https://github.com/tolitius/cprop">github.com/tolitius/cprop</a>
</p>

---
class: middle
<h2 style="text-align:center">cprop: (load-confg)</h2>

There are a lot more options to choose from:

```clojure
(load-config :resource "path/within/classpath/to.edn"
             :file "/path/to/some.edn"
             :merge [{:datomic {:url "foo.bar"}} 
                     (from-file "/path/to/another.edn")
                     (from-resource "path/within/classpath/to-another.edn")
                     (parse-runtime-args ...)
                     (from-props-file "/path/to/some.properties")
                     (from-system-props)
                     (from-env)])
```

It can get as creative as needed, but.. _this should cover most cases_:

```clojure
(load-config)
```
<p style="margin-top: 30px; text-align:center">
    <a href="https://github.com/tolitius/cprop/blob/master/README.md">github.com/tolitius/cprop/README.md</a>
</p>

---
class: middle
<h2 style="text-align:center; margin-bottom: 0px">envoy: (watch-path)</h2>

<p style="margin: 14px 0px 40px 160px;"><em>a gentle touch of Clojure to Hashicorp's Consul</em></p>

```clojure
                       (watch-path path fun)
```

![hubble store](img/hubble-store.png)

```clojure
(envoy/watch-path "http://localhost:8500/v1/kv/hubble/store"
                  #(println "watcher says:" %)))
```

<p style="margin-top: 30px; text-align:center">
    <a href="https://github.com/tolitius/envoy">github.com/tolitius/envoy</a>
</p>
---
<h2 style="text-align:center; margin-top: 80px">Hubble Config</h2>

```clojure
{:hubble {:server {:port 4242}
          :store {:url "spacecraft://tape"}
          :camera {:mode "mono"}
          :mission {:target "Eagle Nebula"}
 
          :log {:enabled false              ;; can be overridden at startup / runtime / consul, etc.
                :auth-token "OVERRIDE ME"
                :name "hubble-log"
                :hazelcast {:hosts "OVERRIDE ME"
                            :group-name "OVERRIDE ME"
                            :group-password "OVERRIDE ME"
                            :retry-ms 5000
                            :retry-max 720000}}
 
          :vault {:url "OVERRIDE ME"}}}
```

---
<h2 style="text-align:center; margin-top: 60px">Loading Config</h2>

loads config and overrides it form Consul:

```clojure
(defn create-config [consul-path]
  (->> (cprop/load-config)
       (envoy/merge-with-consul consul-path)))
```

by default `(load-config)` would also override anything _matching_ from ENV:

```bash
$ export HUBBLE__LOG__AUTH_TOKEN=eda33881-5f34-cc34-806d-3e7da3906230
```

would override

```clojure
{:hubble {:log {:auth-token "OVERRIDE ME"}}}
```

---
<h2 style="text-align:center; margin-top: 90px">Keeping it on the hush hush</h2>

overrides a piece of config with creds from Vault:
```clojure
(defn with-creds [conf at token]
  (-> (vault/merge-config conf {:at at
                                :vhost [:hubble :vault :url]
                                :token token})
      (get-in at)))
```

feeds Hazelcast client with its config + creds from Vault:
```clojure
(defn connect [config]
  (hz/client-instance (env/with-creds config
                                      [:hubble :log :hazelcast]
                                      [:hubble :log :auth-token])))
```
---
class: middle, center, inverse
# Rock & Roll

<h4 style="margin: -20px 0px 20px 0px"><em>(demo)</em></h4>
<img src="img/hubble-mission.jpg" style="width: 600px"></img>
<p style="margin: 20px 0px 0px; font-size: 0.8em;"><a href="https://github.com/tolitius/hubble">github.com/tolitius/hubble</a></p>
<h3 style="margin: -5px 0px 0px 0px; color: white;">+</h3>
<p style="margin: 0px; font-size: 0.8em;"><a href="https://github.com/tolitius/cault">github.com/tolitius/cault</a></p>

---
class: middle, center, inverse
# Thank you,
### [@tolitius](http://github.com/tolitius)