path: root/doc/markdown/automation-stack-architecture.md

# Far-Edge Automation Stack Deployment Guide

The Far-Edge automation stack consists of nine Ubuntu 18.04 hosts:

- Two load balancers
- Two middleware application hosts
- Two ansible hosts
- Three database hosts

The provisioning of these hosts in openstack is outside the scope of
this document. There will be a separate document to address this.

There is an additional host in the production stack to handle LDAP
dual-stack proxying to reach USWIN. This host does not live in the
automation stack tenant space. It is a standalone VM managed by VCPe.

## OpenStack tenant space

In both production and MTCE the far-edge automation stack runs in an
OpenStack tenant space. (Production runs in VCPe.) Creating the
OpenStack objects (neutron networks, cinder volumes, instances) is
beyond the scope of this document and will be documented separately.

Secgroup rules must be created to allow network traffic. Use a script
like the following to create the secgroup rules:

    #!/bin/bash

    private_network_cidr="2001:4888:a21:3102:245:29::/112"
    public_network_cidrs="2001:4888::/32"

    for name in default-grp; do
        openstack security group show ${name} || openstack security group create --description "${name}" ${name}
        for protocol in tcp udp; do
            if [ "openstack security group show ${name} -f shell -c rules | grep -F 'ingress' | grep -F \"port_range_min='1'\" | grep -F \"port_range_max='65535'\" | grep -F \"${protocol}\" | grep -F 'IPv6'" == "" ]; then
                openstack security group rule create --egress --protocol ${protocol} --src-ip ::/0 --dst-port 1:65535 --ethertype IPv6 ${name}
            fi
        done
        for cidr in ${private_network_cidr} ${public_network_cidrs}; do
            if [ "$(openstack security group show ${name} -f shell -c rules | grep -F \"icmp\" | grep -F \"remote_ip_prefix='${cidr}'\")" == "" ]; then
                openstack security group rule create --ingress --protocol icmp --src-ip ${cidr} --ethertype IPv6 ${name}
            fi
        done
        for port in 22; do
            for cidr in ${private_network_cidr} ${public_network_cidrs}; do
                if [ "$(openstack security group show ${name} -f shell -c rules | grep -F \"port_range_min='${port}'\" | grep -F \"remote_ip_prefix='${cidr}'\")" == "" ]; then
                    openstack security group rule create --ingress --protocol tcp --dst-port ${port} --src-ip ${cidr} --ethertype IPv6 ${name}
                fi
            done
        done
    done

    for name in icinga; do
        openstack security group show ${name} || openstack security group create --description "${name}" ${name}
        for port in 5665; do
            for cidr in ${private_network_cidr}; do
                if [ "$(openstack security group show ${name} -f shell -c rules | grep -F \"port_range_min='${port}'\" | grep -F \"remote_ip_prefix='${cidr}'\")" == "" ]; then
                    openstack security group rule create --ingress --protocol tcp --dst-port ${port} --src-ip ${cidr} --ethertype IPv6 ${name}
                fi
            done
        done
    done

    for name in ldap; do
        openstack security group show ${name} || openstack security group create --description "${name}" ${name}
        for port in 636; do
            for cidr in ${private_network_cidr}; do
                if [ "$(openstack security group show ${name} -f shell -c rules | grep -F \"port_range_min='${port}'\" | grep -F \"remote_ip_prefix='${cidr}'\")" == "" ]; then
                    openstack security group rule create --ingress --protocol tcp --dst-port ${port} --src-ip ${cidr} --ethertype IPv6 ${name}
                fi
            done
        done
    done

    for name in web; do
        openstack security group show ${name} || openstack security group create --description "${name}" ${name}
        for port in 443 3000; do
            for cidr in ${private_network_cidr} ${public_network_cidrs}; do
                if [ "$(openstack security group show ${name} -f shell -c rules | grep -F \"port_range_min='${port}'\" | grep -F \"remote_ip_prefix='${cidr}'\")" == "" ]; then
                    openstack security group rule create --ingress --protocol tcp --dst-port ${port} --src-ip ${cidr} --ethertype IPv6 ${name}
                fi
            done
        done
        for port in 80 3000 3080 3128:3130 8000; do
            for cidr in ${private_network_cidr}; do
                if [ "$(openstack security group show ${name} -f shell -c rules | grep -F \"port_range_min='${port}'\" | grep -F \"remote_ip_prefix='${cidr}'\")" == "" ]; then
                    openstack security group rule create --ingress --protocol tcp --dst-port ${port} --src-ip ${cidr} --ethertype IPv6 ${name}
                fi
            done
        done
    done

    for name in galera; do
        openstack security group show ${name} || openstack security group create --description "${name}" ${name}
        for port in 3306 4444 4567 9200 13306; do
            for cidr in ${private_network_cidr}; do
                if [ "$(openstack security group show ${name} -f shell -c rules | grep -F \"port_range_min='${port}'\" | grep -F \"remote_ip_prefix='${cidr}'\")" == "" ]; then
                    openstack security group rule create --ingress --protocol tcp --dst-port ${port} --src-ip ${cidr} --ethertype IPv6 ${name}
                fi
            done
        done
    done

    for name in zmq; do
        openstack security group show ${name} || openstack security group create --description "${name}" ${name}
        for port in 5555:5559; do
            for cidr in ${private_network_cidr}; do
                if [ "$(openstack security group show ${name} -f shell -c rules | grep -F \"port_range_min='${port}'\" | grep -F \"remote_ip_prefix='${cidr}'\")" == "" ]; then
                    openstack security group rule create --ingress --protocol tcp --dst-port ${port} --src-ip ${cidr} --ethertype IPv6 ${name}
                fi
            done
        done
    done

    exit 0

## Database

```far_edge_ops_api``` and icinga2 require a database backend. We
currently use postgresql. The goal is to migrate to
galera/mariadb. For now we are using only one database host, but when
we switch to galera we will use all three. The galera solution is
dockerized. It is a mature and well-tested product of the VCP Metering
project.

### Installation

Install postgresql on the first database host per the typical
procedure that can be found on the internet. TODO: provide a link to
an internet HOWTO that best fits our needs.

#### postgresql.conf:

    data_directory = '/var/lib/postgresql/12/data'
    listen_addresses = '*'
    port = 5432
    max_connections = 2000
    password_encryption = md5
    ssl = off
    shared_buffers = 2000MB
    work_mem = 4MB
    maintenance_work_mem = 256MB
    dynamic_shared_memory_type = posix
    wal_buffers = 2MB
    checkpoint_timeout = 15min
    max_wal_size = 1GB
    min_wal_size = 80MB
    checkpoint_completion_target = 0.9
    logging_collector = on
    log_directory = '/var/lib/postgresql/12_log/'
    log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log'
    log_rotation_size = 50MB
    log_min_duration_statement = 5
    log_checkpoints = on
    log_connections = on
    log_disconnections = on
    log_duration = on
    log_error_verbosity = default
    log_line_prefix = '%m, %d, %a. %r, %s, %x, %p '
    log_timezone = 'GMT'
    autovacuum = on
    idle_in_transaction_session_timeout = 43200000
    datestyle = 'iso, dmy'
    timezone = 'GMT'
    lc_messages = 'en_US.utf8'
    lc_monetary = 'en_US.utf8'
    lc_numeric = 'en_US.utf8'
    lc_time = 'en_US.utf8'
    default_text_search_config = 'pg_catalog.simple'
    plperl.on_init = 'use utf8; use re; package utf8; require "utf8_heavy.pl";'

#### pg_hba.conf:

    local   icinga          icinga                                  md5
    local   all             all                                     password
    local   replication     postgres                                trust
    host    icinga          icinga          127.0.0.1/32            md5
    host    icinga          icinga          ::1/128                 md5
    host    all             all             127.0.0.1/32            password
    host    all             all             ::1/128                 password
    host    icinga          icinga          2001:4888:a21:3102::/32 md5
    host    all             all             2001:4888:a21:3102::/32 password

#### sysctl.conf:

Configure ```kernel.shmmax``` and ```kernel.shmall``` to be 25\% of
available RAM. For example on a 16GB host:

    kernel.shmmax=4199411712
    kernel.shmall=4199411712

### User and database creation

As the postgres user:

    # createuser -l -P -e faredge
    # createdb -e -O faredge faredge

We do not need to populate the database with a schema. Django will do
this automatically.

## Load Balancers

The services which run on the loadbalancer hosts are as follows:

- apache
- haproxy
- icinga2

### Apache

#### SSL Certificates:

A prerequisite for configuring apache is an SSL x509 certificate that
answers to the following names:

    middleware.<fqdn>
    icinga.<fqdn>
    dns-admin.<fqdn>

Place the certificate in ```/etc/ssl/certs/``` and the key in
```/etc/ssl/private```.

#### Modules:

Apache needs the following modules enabled:

    - access_compat.load
    - alias.conf
    - alias.load
    - auth_basic.load
    - authn_core.load
    - authn_file.load
    - authz_host.load
    - authz_user.load
    - authz_core.load
    - autoindex.conf
    - autoindex.load
    - deflate.conf
    - deflate.load
    - dir.conf
    - dir.load
    - env.load
    - filter.load
    - http2.load
    - mime.conf
    - mime.load
    - mpm_prefork.conf
    - mpm_prefork.load
    - negotiation.conf
    - negotiation.load
    - php7.2.conf
    - php7.2.load
    - proxy.conf
    - proxy.load
    - proxy_http.load
    - proxy_http2.load
    - proxy_wstunnel.load
    - reqtimeout.conf
    - reqtimeout.load
    - rewrite.load
    - setenvif.conf
    - setenvif.load
    - socache_shmcb.load
    - ssl.conf
    - ssl.load
    - status.conf
    - status.load

#### VirtualHosts:

Apache needs the following sites enabled:

    - dns-admin.conf
    - far_edge_ops_api.conf
    - icinga.conf
    - webdav.conf

```webdav.conf``` is only needed when an external HTTP share is not
available.

##### dns-admin.conf:

    <VirtualHost [2607:f160:b:10f1::e]:443>
        ServerName dns-admin.faredge.vzwops.com
        ServerAdmin carlos.konstanski@verizonwireless.com
        Protocols http/1.1
        SSLEngine on
        SSLProxyEngine on
        SSLCertificateFile /etc/ssl/certs/vcpfe-lb-vip.faredge.vzwops.com.pem
        SSLCertificateKeyFile /etc/ssl/private/vcpfe-lb-vip.faredge.vzwops.com.key
        SSLProtocol all -SSLv2 -SSLv3
        SSLCipherSuite ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
        SSLHonorCipherOrder on
        SSLCompression off
        SSLOptions +StrictRequire
        ProxyPass "/" "http://[::1]:13000/"
        ProxyPassReverse "/" "http://[::1]:13000/"
        SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown
        ErrorLog "/var/log/apache2/dnsadmin_error_log"
        CustomLog "/var/log/apache2/dnsadmin_access_log" common
    </VirtualHost>

##### far\_edge\_ops\_api.conf:

    <VirtualHost [2607:f160:b:10f1::e]:443>
        ServerName middleware.faredge.vzwops.com
        ServerAdmin carlos.konstanski@verizonwireless.com
        Protocols http/1.1
        SSLEngine on
        SSLProxyEngine on
        SSLCertificateFile /etc/ssl/certs/vcpfe-lb-vip.faredge.vzwops.com.pem
        SSLCertificateKeyFile /etc/ssl/private/vcpfe-lb-vip.faredge.vzwops.com.key
        SSLProtocol all -SSLv2 -SSLv3
        SSLCipherSuite ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
        SSLHonorCipherOrder on
        SSLCompression off
        SSLOptions +StrictRequire
        ProxyPass "/" "http://[::1]:13080/"
        ProxyPassReverse "/" "http://[::1]:13080/"
        ErrorLog "/var/log/apache2/middleware_error_log"
        CustomLog "/var/log/apache2/middleware_access_log" common
    </VirtualHost>

##### icinga.conf:

    <VirtualHost [2607:f160:b:10f1::e]:443>
        ServerName icinga.faredge.vzwops.com
        ServerAdmin carlos.konstanski@verizonwireless.com
        DocumentRoot "/usr/share/icingaweb2/public"
        SSLEngine on
        SSLProxyEngine on
        SSLCertificateFile /etc/ssl/certs/vcpfe-lb-vip.faredge.vzwops.com.pem
        SSLCertificateKeyFile /etc/ssl/private/vcpfe-lb-vip.faredge.vzwops.com.key
        SSLProtocol all -SSLv2 -SSLv3
        SSLCipherSuite ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
        SSLHonorCipherOrder on
        SSLCompression off
        SSLOptions +StrictRequire

        <Directory "/usr/share/icingaweb2/public">
            Options SymLinksIfOwnerMatch
            AllowOverride None

            <IfModule mod_authz_core.c>
                <RequireAll>
                    Require all granted
                </RequireAll>
            </IfModule>

            SetEnv ICINGAWEB_CONFIGDIR "/etc/icingaweb2"
            EnableSendfile Off

            <IfModule mod_rewrite.c>
                RewriteEngine on
                RewriteBase /
                RewriteCond %{REQUEST_FILENAME} -s [OR]
                RewriteCond %{REQUEST_FILENAME} -l [OR]
                RewriteCond %{REQUEST_FILENAME} -d
                RewriteRule ^.*$ - [NC,L]
                RewriteRule ^.*$ index.php [NC,L]
            </IfModule>
        </Directory>
    </VirtualHost>

##### webdav.conf:

    <VirtualHost *:80>
        ServerName webdav.faredge.vzwops.com
        ServerAdmin carlos.konstanski@verizonwireless.com
        DocumentRoot "/var/www/html/webdav"
        SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown
        ErrorLog "/var/log/apache2/webdav_error_log"
        CustomLog "/var/log/apache2/webdav_access_log" common
        <Directory "/var/www/html/webdav">
            AllowOverride limit
            Options Indexes FollowSymLinks
        </Directory>
    </VirtualHost>

##### Webdav directory:

Create this directory only if you don't have access to an external HTTP share and you are using the webdav VirtualHost above:

    - /var/www/html/webdav

### HAProxy

Use the following haproxy config, adjusting IP addresses, hostnames and credentials as needed:

    global
        tune.ssl.default-dh-param 2048
        log /dev/log local0
        log /dev/log local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin
        stats timeout 30s
        maxconn 5000
        user haproxy
        group haproxy
        daemon
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private
        ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
        ssl-default-bind-options no-sslv3

    defaults
        log global
        option dontlognull
        timeout connect 10s
        timeout client 60s
        timeout server 180s
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

    listen stats
        bind 127.0.0.1:1993
        mode http
        option httplog
        option forwardfor
        stats enable
        stats uri /stats
        stats auth someuser:somepass
        stats refresh 30s
        stats show-node
        stats hide-version
        log global

    frontend dns-admin
        bind 0.0.0.0:13000
        bind :::13000
        mode http
        option httplog
        option forwardfor
        default_backend dns-admin-backend

    backend dns-admin-backend
        balance roundrobin
        mode http
        option log-health-checks
        option httpchk GET /dns HTTP/1.1\r\nHost:\ dns-admin.faredge.vzwops.com\r\nUser-Agent:\ curl/7.35.0\r\nAccept:\ */*\r\nAuthorization:\ Basic\ b3BlbmJvb2s6WnlsejNmZE1TZDhTVmlod0w5TnlaTHpU
        default-server inter 10s fall 1 rise 1
        server server-1 [2607:f160:b:10f1::7]:80 check
        #server server-2 [2607:f160:b:10f1::8]:80 check

    frontend far_edge_ops_api
        bind 0.0.0.0:13080
        bind :::13080
        mode http
        option httplog
        option forwardfor
        reqdel ^Host:
        reqadd Host:\ middleware.faredge.vzwops.com
        default_backend far_edge_ops_api-backend

    backend far_edge_ops_api-backend
        balance roundrobin
        mode http
        option log-health-checks
        option httpchk GET /caas/ HTTP/1.1\r\nHost:\ middleware.faredge.vzwops.com\r\nUser-Agent:\ curl/7.35.0\r\nAccept:\ */*\r\nAuthorization:\ Basic\ bWlkZGxld2FyZXVzZXI6bWlkZGxld2FyZXBhc3N3b3Jk
        default-server inter 10s fall 1 rise 1
        server server-1 [2607:f160:b:10f1::6]:3080 check
        #server server-2 [2607:f160:b:10f1::5]:3080 check

We currently only proxy to one backend for both ```dns-admin``` and
```far_edge_ops_api``` because there has been no opportunity to test
the behavior of the apps in a multiple-backend configuration. This is
why ```server-2``` is commented out. TODO: update this document when
this testing is complete.

### Icinga2

The icinga2 configuration is mostly standard. TODO: provide a link to
an internet HOWTO from which the rest of this documentation can
build. There will be only minor additions to the standard base
install.

## Jenkins

The applications are built and deployed from jenkins. The build jobs
already exist. But a new deploy job needs to be added for the new
environment which you are building.

### env.yaml

All of the far-edge builds and deploymenets are run on the jenkins
slave ```openbook-docker-slave-4.meter.vzwops.com```. Log onto this
host and edit three files in the jenkins home directory
```/var/lib/jenkins/```:

    - env.yaml.ansible-queue
    - env.yaml.dns-admin
    - env.yaml.far-edge-ops-api

A new deployment target section needs to be added to each of these
files. Give the environment a new unique name and use the same name in
all three files.

### jenkins-job-builder (jjb)

Clone the following git repo onto your laptop:

    git@gitlab.verizon.com:ONV6661_VCPSPUBLIC/jjb.git

Create an ini file called ```verizoncloudplatform.com.ini``` in the
top-level directory of your jjb git checkout with the following
contents:

#### verizoncloudplatform.com.ini:
    [job_builder]
    ignore_cache=True
    keep_descriptions=False
    include_path=.:scripts:~/git/
    recursive=False
    exclude=.*:manual:./development
    allow_duplicates=False

    [jenkins]
    user=<your_jenkins_username>
    password=<your_jenkins_api_token>
    url=https://openbook-jenkins-master.meter.vzwops.com:8080/
    timeout=120
    query_plugins_info=False

Follow these steps to create the new deploy jobs:

- Open the file ```yaml/verizoncloudplatform.com/deploy-vcpfe.yaml```
  in an editor.

- Add three new jobs to the list, one for each app
  (```ansible-queue```, ```dns-admin``` and
  ```far-edge-ops-api```). Set the target attribute to the new ansible
  target which you created in env.yaml in the previous section.

- ```# ./run.sh verizoncloudplatform.com```

Once this is done, log into the jenkins UI to verify that the new jobs
were created.

Push your jjb changes to git. (The ini file is gitignored; it is your
private file.)

## Application hosts

The services which run on the application hosts are as follows:

    - far_edge_ops_api

### far\_edge\_ops\_api

The middleware (as ```far_edge_ops_api``` is ubiquitously known) is a
docker containerized django web application.

#### Build/deploy

The application can be built and deployed from jenkins:

https://openbook-jenkins-master.meter.vzwops.com:8080

Run the job ```docker-container-far-edge-ops-api``` to build the
docker container, and the appropriate
```far-edge-ops-api-deploy-vcpfe-<env>``` job to deploy it to an
environment.

#### Configure

It might seem backwards to perform the build and deployment before
configuring. It's a chicken-and-egg problem: the deployment needs to
create the required files and directories before configuration can
commence. For first-time deployments the process will look like: build
-> deploy -> configure -> deploy. This is true of all three
applications.

The main configuration directory is ```/etc/far-edge-ops-api/```. It
looks like the following on a working system:

    $ tree /etc/far-edge-ops-api/
    /etc/far-edge-ops-api/
    ├── modules.d
    │   ├── wsgi.conf
    │   └── wsgi.load
    ├── settings.py
    └── vhosts.d
        └── far-edge-ops-api.conf

```settings.py``` is a rather long file, too long to include here. Get
a copy from a known good source (a production or lab server) and edit
to suit the new environment.

```far-edge-ops-api.conf``` is a rather typical-looking apache
VirtualHosts file, but with LDAP auth added. First the file:

##### far-edge-ops-api.conf:

    LDAPTrustedGlobalCert CA_BASE64 "/etc/apache2/auth/uswin_ca.pem"
    LDAPTrustedGlobalCert CA_BASE64 "/etc/ssl/certs/selfsigned_root_cert.crt"
    LDAPTrustedMode SSL
    LDAPLibraryDebug 7

    <VirtualHost *:3080>
        ServerName middleware.<fqdn>
        DocumentRoot "/opt/far-edge-ops-api/src/src"
        WSGIScriptAlias / /opt/far-edge-ops-api/src/src/far_edge_ops_api/wsgi.py
        WSGIDaemonProcess middleware.<fqdn> processes=20 threads=15 display-name=%{GROUP} python-home=/opt/far-edge-ops-api/src/venv
        WSGIProcessGroup middleware.<fqdn>
        Alias /static/ /opt/far-edge-ops-api/src/src/static/
        TimeOut 600
        SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown
        ErrorLog "/var/log/apache2/far-edge-ops-api_error_log"
        CustomLog "/var/log/apache2/far-edge-ops-api_access_log" common
    </VirtualHost>

    <Directory "/opt/far-edge-ops-api/src/src">
        AllowOverride all
        Options FollowSymlinks
        AuthName "USWIN"
        AuthType Basic
        AuthBasicProvider ldap
        AuthLDAPURL "ldaps://vcpfe-ldap-prod.vcpfe.vzwops.com:636/DC=uswin,DC=ad,DC=vzwcorp,DC=com?sAMAccountName?sub?(objectclass=*)"
        AuthLDAPBindDN "CN=SVC-VCP-LDAP,OU=SVC,OU=FNA,DC=uswin,DC=ad,DC=vzwcorp,DC=com"
        AuthLDAPBindPassword "<password_for_SVC-VCP-LDAP_account>"
        Require valid-user
    </Directory>

    <Directory "/opt/far-edge-ops-api/src/src/static">
        AllowOverride limit
        Require all granted
        Options Indexes FollowSymLinks
    </Directory>

There are two SSL certs because the far-edge stack is IPv6-only while
the production LDAP endpoint (USWIN) is IPv4-only. So we must use a
proxy. The first cert is the actual uswin cert, while the second cert
is the one for the proxy. Both of these certs must be placed on the
app hosts.

Replace ```<fqdn>``` with the DNS domain. Replace
```<password_for_SVC-VCP-LDAP_account>``` with the correct password.

OpenLDAP needs to be configured with these same certs. Since they are
self-signed, openldap will reject them unless they are listed as
exceptions.

#### /etc/ldap/ldap.conf:

    TLS_CACERT /etc/apache2/auth/uswin_ca.pem
    LDAPTLS_CACERT /etc/ssl/certs/self_ca_signed_cert_and_key_2.pem

Now that the configuration is complete, rerun the deployment from
jenkins. This time the application should start normally.

## Queue hosts

The services which run on the application hosts are as follows:

    - ansible-queue
    - dns-admin

### ansible-queue

```ansible-queue``` is a docker containerized common lisp
application.

#### Build/deploy

The application can be built and deployed from jenkins:

https://openbook-jenkins-master.meter.vzwops.com:8080

Run the job ```docker-container-ansible-queue``` to build the docker
container, and the appropriate ```ansible-queue-deploy-vcpfe-<env>```
job to deploy it to an environment.

#### Configure

Two files are required in the directory ```/etc/ansible-queue/```:

    - .ansible_pass.far_edge
    - options.lisp

```.ansible_pass.far_edge``` is the ansible vault password
file. Obtain a copy from an existing queue host.

```options.lisp``` is the config file for the ```ansible-queue```
application.

##### options.lisp:

    ((:queue (:num-icinga-process-threads 1
              :num-bmc-process-threads 30
              :num-nic-process-threads 20
              :num-wr-process-threads 25
              :num-dns-process-threads 1
              :num-patch-process-threads 1
              :num-hw-process-threads 20
              :num-fix-process-threads 20
              :icinga-wait-interval 0
              :bmc-wait-interval 1
              :nic-wait-interval 30
              :wr-wait-interval 0
              :dns-wait-interval 0
              :patch-wait-interval 30
              :hw-wait-interval 1
              :fix-wait-interval 1
              :sleep-interval 60
              :ipv6 t
              :vault-password-file "/etc/ansible-queue/.ansible_pass.far_edge"
              :icinga-queue-url "tcp://*:5555"
              :bmc-queue-url "tcp://*:5556"
              :nic-queue-url "tcp://*:5557"
              :wr-queue-url "tcp://*:5558"
              :dns-queue-url "tcp://*:5559"
              :patch-queue-url "tcp://*:5560"
              :hw-queue-url "tcp://*:5561"
              :fix-queue-url "tcp://*:5562"
              :middleware-url "https://middleware.vcpfe.vzwops.com"
              :middleware-username "SVC-Far-Edge"
              :middleware-password "<password_for_SVC-Far-Edge_account>")))

Replace ```<password_for_SVC-Far-Edge_account>``` with the correct
password.

Now that the configuration is complete, rerun the deployment from
jenkins. This time the application should start normally.

### dns-admin

```dns-admin``` is a docker containerized web application written in
common lisp and clojurescript.

#### Build/deploy

The application can be built and deployed from jenkins:

https://openbook-jenkins-master.meter.vzwops.com:8080

Run the job ```docker-container-dns-admin``` to build the docker
container, and the appropriate ```dns-admin-deploy-vcpfe-<env>``` job
to deploy it to an environment.

#### Configure

Only one new config file is required. Use the following content
verbatim:

##### /etc/dns-admin/conf.lisp:

    (:name "dns-admin"
     :document-root "dns-admin"
     :title "DNS Administration"
     :meta-description "A website for administering DNS. Supports nsupdate and infoblox."
     :dns (:label "CDS Infoblox"
           :backend-type "infoblox"
           :url "https://infoblox.alias/wapi/v2.6"))

Notice the odd-looking hostname ```infoblox.alias```. The following
needs to be added to ```/etc/hosts```:

    2607:f160:8a02:8016:a0:103:0:10 infoblox.alias

Now that the configuration is complete, rerun the deployment from
jenkins. This time the application should start normally.