diff options
Diffstat (limited to 'doc/markdown/automation-stack-architecture.md')
| -rw-r--r-- | doc/markdown/automation-stack-architecture.md | 767 |
1 files changed, 767 insertions, 0 deletions
diff --git a/doc/markdown/automation-stack-architecture.md b/doc/markdown/automation-stack-architecture.md new file mode 100644 index 0000000..2ae775c --- /dev/null +++ b/doc/markdown/automation-stack-architecture.md @@ -0,0 +1,767 @@ +# Far-Edge Automation Stack Deployment Guide + +The Far-Edge automation stack consists of nine Ubuntu 18.04 hosts: + +- Two load balancers +- Two middleware application hosts +- Two ansible hosts +- Three database hosts + +The provisioning of these hosts in openstack is outside the scope of +this document. There will be a separate document to address this. + +There is an additional host in the production stack to handle LDAP +dual-stack proxying to reach USWIN. This host does not live in the +automation stack tenant space. It is a standalone VM managed by VCPe. + +## OpenStack tenant space + +In both production and MTCE the far-edge automation stack runs in an +OpenStack tenant space. (Production runs in VCPe.) Creating the +OpenStack objects (neutron networks, cinder volumes, instances) is +beyond the scope of this document and will be documented separately. + +Secgroup rules must be created to allow network traffic. Use a script +like the following to create the secgroup rules: + + #!/bin/bash + + private_network_cidr="2001:4888:a21:3102:245:29::/112" + public_network_cidrs="2001:4888::/32" + + for name in default-grp; do + openstack security group show ${name} || openstack security group create --description "${name}" ${name} + for protocol in tcp udp; do + if [ "openstack security group show ${name} -f shell -c rules | grep -F 'ingress' | grep -F \"port_range_min='1'\" | grep -F \"port_range_max='65535'\" | grep -F \"${protocol}\" | grep -F 'IPv6'" == "" ]; then + openstack security group rule create --egress --protocol ${protocol} --src-ip ::/0 --dst-port 1:65535 --ethertype IPv6 ${name} + fi + done + for cidr in ${private_network_cidr} ${public_network_cidrs}; do + if [ "$(openstack security group show ${name} -f shell -c rules | grep -F \"icmp\" | grep -F \"remote_ip_prefix='${cidr}'\")" == "" ]; then + openstack security group rule create --ingress --protocol icmp --src-ip ${cidr} --ethertype IPv6 ${name} + fi + done + for port in 22; do + for cidr in ${private_network_cidr} ${public_network_cidrs}; do + if [ "$(openstack security group show ${name} -f shell -c rules | grep -F \"port_range_min='${port}'\" | grep -F \"remote_ip_prefix='${cidr}'\")" == "" ]; then + openstack security group rule create --ingress --protocol tcp --dst-port ${port} --src-ip ${cidr} --ethertype IPv6 ${name} + fi + done + done + done + + for name in icinga; do + openstack security group show ${name} || openstack security group create --description "${name}" ${name} + for port in 5665; do + for cidr in ${private_network_cidr}; do + if [ "$(openstack security group show ${name} -f shell -c rules | grep -F \"port_range_min='${port}'\" | grep -F \"remote_ip_prefix='${cidr}'\")" == "" ]; then + openstack security group rule create --ingress --protocol tcp --dst-port ${port} --src-ip ${cidr} --ethertype IPv6 ${name} + fi + done + done + done + + for name in ldap; do + openstack security group show ${name} || openstack security group create --description "${name}" ${name} + for port in 636; do + for cidr in ${private_network_cidr}; do + if [ "$(openstack security group show ${name} -f shell -c rules | grep -F \"port_range_min='${port}'\" | grep -F \"remote_ip_prefix='${cidr}'\")" == "" ]; then + openstack security group rule create --ingress --protocol tcp --dst-port ${port} --src-ip ${cidr} --ethertype IPv6 ${name} + fi + done + done + done + + for name in web; do + openstack security group show ${name} || openstack security group create --description "${name}" ${name} + for port in 443 3000; do + for cidr in ${private_network_cidr} ${public_network_cidrs}; do + if [ "$(openstack security group show ${name} -f shell -c rules | grep -F \"port_range_min='${port}'\" | grep -F \"remote_ip_prefix='${cidr}'\")" == "" ]; then + openstack security group rule create --ingress --protocol tcp --dst-port ${port} --src-ip ${cidr} --ethertype IPv6 ${name} + fi + done + done + for port in 80 3000 3080 3128:3130 8000; do + for cidr in ${private_network_cidr}; do + if [ "$(openstack security group show ${name} -f shell -c rules | grep -F \"port_range_min='${port}'\" | grep -F \"remote_ip_prefix='${cidr}'\")" == "" ]; then + openstack security group rule create --ingress --protocol tcp --dst-port ${port} --src-ip ${cidr} --ethertype IPv6 ${name} + fi + done + done + done + + for name in galera; do + openstack security group show ${name} || openstack security group create --description "${name}" ${name} + for port in 3306 4444 4567 9200 13306; do + for cidr in ${private_network_cidr}; do + if [ "$(openstack security group show ${name} -f shell -c rules | grep -F \"port_range_min='${port}'\" | grep -F \"remote_ip_prefix='${cidr}'\")" == "" ]; then + openstack security group rule create --ingress --protocol tcp --dst-port ${port} --src-ip ${cidr} --ethertype IPv6 ${name} + fi + done + done + done + + for name in zmq; do + openstack security group show ${name} || openstack security group create --description "${name}" ${name} + for port in 5555:5559; do + for cidr in ${private_network_cidr}; do + if [ "$(openstack security group show ${name} -f shell -c rules | grep -F \"port_range_min='${port}'\" | grep -F \"remote_ip_prefix='${cidr}'\")" == "" ]; then + openstack security group rule create --ingress --protocol tcp --dst-port ${port} --src-ip ${cidr} --ethertype IPv6 ${name} + fi + done + done + done + + exit 0 + +## Database + +```far_edge_ops_api``` and icinga2 require a database backend. We +currently use postgresql. The goal is to migrate to +galera/mariadb. For now we are using only one database host, but when +we switch to galera we will use all three. The galera solution is +dockerized. It is a mature and well-tested product of the VCP Metering +project. + +### Installation + +Install postgresql on the first database host per the typical +procedure that can be found on the internet. TODO: provide a link to +an internet HOWTO that best fits our needs. + +#### postgresql.conf: + + data_directory = '/var/lib/postgresql/12/data' + listen_addresses = '*' + port = 5432 + max_connections = 2000 + password_encryption = md5 + ssl = off + shared_buffers = 2000MB + work_mem = 4MB + maintenance_work_mem = 256MB + dynamic_shared_memory_type = posix + wal_buffers = 2MB + checkpoint_timeout = 15min + max_wal_size = 1GB + min_wal_size = 80MB + checkpoint_completion_target = 0.9 + logging_collector = on + log_directory = '/var/lib/postgresql/12_log/' + log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log' + log_rotation_size = 50MB + log_min_duration_statement = 5 + log_checkpoints = on + log_connections = on + log_disconnections = on + log_duration = on + log_error_verbosity = default + log_line_prefix = '%m, %d, %a. %r, %s, %x, %p ' + log_timezone = 'GMT' + autovacuum = on + idle_in_transaction_session_timeout = 43200000 + datestyle = 'iso, dmy' + timezone = 'GMT' + lc_messages = 'en_US.utf8' + lc_monetary = 'en_US.utf8' + lc_numeric = 'en_US.utf8' + lc_time = 'en_US.utf8' + default_text_search_config = 'pg_catalog.simple' + plperl.on_init = 'use utf8; use re; package utf8; require "utf8_heavy.pl";' + +#### pg_hba.conf: + + local icinga icinga md5 + local all all password + local replication postgres trust + host icinga icinga 127.0.0.1/32 md5 + host icinga icinga ::1/128 md5 + host all all 127.0.0.1/32 password + host all all ::1/128 password + host icinga icinga 2001:4888:a21:3102::/32 md5 + host all all 2001:4888:a21:3102::/32 password + +#### sysctl.conf: + +Configure ```kernel.shmmax``` and ```kernel.shmall``` to be 25\% of +available RAM. For example on a 16GB host: + + kernel.shmmax=4199411712 + kernel.shmall=4199411712 + +### User and database creation + +As the postgres user: + + # createuser -l -P -e faredge + # createdb -e -O faredge faredge + +We do not need to populate the database with a schema. Django will do +this automatically. + +## Load Balancers + +The services which run on the loadbalancer hosts are as follows: + +- apache +- haproxy +- icinga2 + +### Apache + +#### SSL Certificates: + +A prerequisite for configuring apache is an SSL x509 certificate that +answers to the following names: + + middleware.<fqdn> + icinga.<fqdn> + dns-admin.<fqdn> + +Place the certificate in ```/etc/ssl/certs/``` and the key in +```/etc/ssl/private```. + +#### Modules: + +Apache needs the following modules enabled: + + - access_compat.load + - alias.conf + - alias.load + - auth_basic.load + - authn_core.load + - authn_file.load + - authz_host.load + - authz_user.load + - authz_core.load + - autoindex.conf + - autoindex.load + - deflate.conf + - deflate.load + - dir.conf + - dir.load + - env.load + - filter.load + - http2.load + - mime.conf + - mime.load + - mpm_prefork.conf + - mpm_prefork.load + - negotiation.conf + - negotiation.load + - php7.2.conf + - php7.2.load + - proxy.conf + - proxy.load + - proxy_http.load + - proxy_http2.load + - proxy_wstunnel.load + - reqtimeout.conf + - reqtimeout.load + - rewrite.load + - setenvif.conf + - setenvif.load + - socache_shmcb.load + - ssl.conf + - ssl.load + - status.conf + - status.load + +#### VirtualHosts: + +Apache needs the following sites enabled: + + - dns-admin.conf + - far_edge_ops_api.conf + - icinga.conf + - webdav.conf + +```webdav.conf``` is only needed when an external HTTP share is not +available. + +##### dns-admin.conf: + + <VirtualHost [2607:f160:b:10f1::e]:443> + ServerName dns-admin.faredge.vzwops.com + ServerAdmin carlos.konstanski@verizonwireless.com + Protocols http/1.1 + SSLEngine on + SSLProxyEngine on + SSLCertificateFile /etc/ssl/certs/vcpfe-lb-vip.faredge.vzwops.com.pem + SSLCertificateKeyFile /etc/ssl/private/vcpfe-lb-vip.faredge.vzwops.com.key + SSLProtocol all -SSLv2 -SSLv3 + SSLCipherSuite ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA + SSLHonorCipherOrder on + SSLCompression off + SSLOptions +StrictRequire + ProxyPass "/" "http://[::1]:13000/" + ProxyPassReverse "/" "http://[::1]:13000/" + SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown + ErrorLog "/var/log/apache2/dnsadmin_error_log" + CustomLog "/var/log/apache2/dnsadmin_access_log" common + </VirtualHost> + +##### far\_edge\_ops\_api.conf: + + <VirtualHost [2607:f160:b:10f1::e]:443> + ServerName middleware.faredge.vzwops.com + ServerAdmin carlos.konstanski@verizonwireless.com + Protocols http/1.1 + SSLEngine on + SSLProxyEngine on + SSLCertificateFile /etc/ssl/certs/vcpfe-lb-vip.faredge.vzwops.com.pem + SSLCertificateKeyFile /etc/ssl/private/vcpfe-lb-vip.faredge.vzwops.com.key + SSLProtocol all -SSLv2 -SSLv3 + SSLCipherSuite ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA + SSLHonorCipherOrder on + SSLCompression off + SSLOptions +StrictRequire + ProxyPass "/" "http://[::1]:13080/" + ProxyPassReverse "/" "http://[::1]:13080/" + ErrorLog "/var/log/apache2/middleware_error_log" + CustomLog "/var/log/apache2/middleware_access_log" common + </VirtualHost> + +##### icinga.conf: + + <VirtualHost [2607:f160:b:10f1::e]:443> + ServerName icinga.faredge.vzwops.com + ServerAdmin carlos.konstanski@verizonwireless.com + DocumentRoot "/usr/share/icingaweb2/public" + SSLEngine on + SSLProxyEngine on + SSLCertificateFile /etc/ssl/certs/vcpfe-lb-vip.faredge.vzwops.com.pem + SSLCertificateKeyFile /etc/ssl/private/vcpfe-lb-vip.faredge.vzwops.com.key + SSLProtocol all -SSLv2 -SSLv3 + SSLCipherSuite ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA + SSLHonorCipherOrder on + SSLCompression off + SSLOptions +StrictRequire + + <Directory "/usr/share/icingaweb2/public"> + Options SymLinksIfOwnerMatch + AllowOverride None + + <IfModule mod_authz_core.c> + <RequireAll> + Require all granted + </RequireAll> + </IfModule> + + SetEnv ICINGAWEB_CONFIGDIR "/etc/icingaweb2" + EnableSendfile Off + + <IfModule mod_rewrite.c> + RewriteEngine on + RewriteBase / + RewriteCond %{REQUEST_FILENAME} -s [OR] + RewriteCond %{REQUEST_FILENAME} -l [OR] + RewriteCond %{REQUEST_FILENAME} -d + RewriteRule ^.*$ - [NC,L] + RewriteRule ^.*$ index.php [NC,L] + </IfModule> + </Directory> + </VirtualHost> + +##### webdav.conf: + + <VirtualHost *:80> + ServerName webdav.faredge.vzwops.com + ServerAdmin carlos.konstanski@verizonwireless.com + DocumentRoot "/var/www/html/webdav" + SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown + ErrorLog "/var/log/apache2/webdav_error_log" + CustomLog "/var/log/apache2/webdav_access_log" common + <Directory "/var/www/html/webdav"> + AllowOverride limit + Options Indexes FollowSymLinks + </Directory> + </VirtualHost> + +##### Webdav directory: + +Create this directory only if you don't have access to an external HTTP share and you are using the webdav VirtualHost above: + + - /var/www/html/webdav + +### HAProxy + +Use the following haproxy config, adjusting IP addresses, hostnames and credentials as needed: + + global + tune.ssl.default-dh-param 2048 + log /dev/log local0 + log /dev/log local1 notice + chroot /var/lib/haproxy + stats socket /run/haproxy/admin.sock mode 660 level admin + stats timeout 30s + maxconn 5000 + user haproxy + group haproxy + daemon + ca-base /etc/ssl/certs + crt-base /etc/ssl/private + ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS + ssl-default-bind-options no-sslv3 + + defaults + log global + option dontlognull + timeout connect 10s + timeout client 60s + timeout server 180s + errorfile 400 /etc/haproxy/errors/400.http + errorfile 403 /etc/haproxy/errors/403.http + errorfile 408 /etc/haproxy/errors/408.http + errorfile 500 /etc/haproxy/errors/500.http + errorfile 502 /etc/haproxy/errors/502.http + errorfile 503 /etc/haproxy/errors/503.http + errorfile 504 /etc/haproxy/errors/504.http + + listen stats + bind 127.0.0.1:1993 + mode http + option httplog + option forwardfor + stats enable + stats uri /stats + stats auth someuser:somepass + stats refresh 30s + stats show-node + stats hide-version + log global + + frontend dns-admin + bind 0.0.0.0:13000 + bind :::13000 + mode http + option httplog + option forwardfor + default_backend dns-admin-backend + + backend dns-admin-backend + balance roundrobin + mode http + option log-health-checks + option httpchk GET /dns HTTP/1.1\r\nHost:\ dns-admin.faredge.vzwops.com\r\nUser-Agent:\ curl/7.35.0\r\nAccept:\ */*\r\nAuthorization:\ Basic\ b3BlbmJvb2s6WnlsejNmZE1TZDhTVmlod0w5TnlaTHpU + default-server inter 10s fall 1 rise 1 + server server-1 [2607:f160:b:10f1::7]:80 check + #server server-2 [2607:f160:b:10f1::8]:80 check + + frontend far_edge_ops_api + bind 0.0.0.0:13080 + bind :::13080 + mode http + option httplog + option forwardfor + reqdel ^Host: + reqadd Host:\ middleware.faredge.vzwops.com + default_backend far_edge_ops_api-backend + + backend far_edge_ops_api-backend + balance roundrobin + mode http + option log-health-checks + option httpchk GET /caas/ HTTP/1.1\r\nHost:\ middleware.faredge.vzwops.com\r\nUser-Agent:\ curl/7.35.0\r\nAccept:\ */*\r\nAuthorization:\ Basic\ bWlkZGxld2FyZXVzZXI6bWlkZGxld2FyZXBhc3N3b3Jk + default-server inter 10s fall 1 rise 1 + server server-1 [2607:f160:b:10f1::6]:3080 check + #server server-2 [2607:f160:b:10f1::5]:3080 check + +We currently only proxy to one backend for both ```dns-admin``` and +```far_edge_ops_api``` because there has been no opportunity to test +the behavior of the apps in a multiple-backend configuration. This is +why ```server-2``` is commented out. TODO: update this document when +this testing is complete. + +### Icinga2 + +The icinga2 configuration is mostly standard. TODO: provide a link to +an internet HOWTO from which the rest of this documentation can +build. There will be only minor additions to the standard base +install. + +## Jenkins + +The applications are built and deployed from jenkins. The build jobs +already exist. But a new deploy job needs to be added for the new +environment which you are building. + +### env.yaml + +All of the far-edge builds and deploymenets are run on the jenkins +slave ```openbook-docker-slave-4.meter.vzwops.com```. Log onto this +host and edit three files in the jenkins home directory +```/var/lib/jenkins/```: + + - env.yaml.ansible-queue + - env.yaml.dns-admin + - env.yaml.far-edge-ops-api + +A new deployment target section needs to be added to each of these +files. Give the environment a new unique name and use the same name in +all three files. + +### jenkins-job-builder (jjb) + +Clone the following git repo onto your laptop: + + git@gitlab.verizon.com:ONV6661_VCPSPUBLIC/jjb.git + +Create an ini file called ```verizoncloudplatform.com.ini``` in the +top-level directory of your jjb git checkout with the following +contents: + +#### verizoncloudplatform.com.ini: + [job_builder] + ignore_cache=True + keep_descriptions=False + include_path=.:scripts:~/git/ + recursive=False + exclude=.*:manual:./development + allow_duplicates=False + + [jenkins] + user=<your_jenkins_username> + password=<your_jenkins_api_token> + url=https://openbook-jenkins-master.meter.vzwops.com:8080/ + timeout=120 + query_plugins_info=False + +Follow these steps to create the new deploy jobs: + +- Open the file ```yaml/verizoncloudplatform.com/deploy-vcpfe.yaml``` + in an editor. + +- Add three new jobs to the list, one for each app + (```ansible-queue```, ```dns-admin``` and + ```far-edge-ops-api```). Set the target attribute to the new ansible + target which you created in env.yaml in the previous section. + +- ```# ./run.sh verizoncloudplatform.com``` + +Once this is done, log into the jenkins UI to verify that the new jobs +were created. + +Push your jjb changes to git. (The ini file is gitignored; it is your +private file.) + +## Application hosts + +The services which run on the application hosts are as follows: + + - far_edge_ops_api + +### far\_edge\_ops\_api + +The middleware (as ```far_edge_ops_api``` is ubiquitously known) is a +docker containerized django web application. + +#### Build/deploy + +The application can be built and deployed from jenkins: + +https://openbook-jenkins-master.meter.vzwops.com:8080 + +Run the job ```docker-container-far-edge-ops-api``` to build the +docker container, and the appropriate +```far-edge-ops-api-deploy-vcpfe-<env>``` job to deploy it to an +environment. + +#### Configure + +It might seem backwards to perform the build and deployment before +configuring. It's a chicken-and-egg problem: the deployment needs to +create the required files and directories before configuration can +commence. For first-time deployments the process will look like: build +-> deploy -> configure -> deploy. This is true of all three +applications. + +The main configuration directory is ```/etc/far-edge-ops-api/```. It +looks like the following on a working system: + + $ tree /etc/far-edge-ops-api/ + /etc/far-edge-ops-api/ + ├── modules.d + │ ├── wsgi.conf + │ └── wsgi.load + ├── settings.py + └── vhosts.d + └── far-edge-ops-api.conf + +```settings.py``` is a rather long file, too long to include here. Get +a copy from a known good source (a production or lab server) and edit +to suit the new environment. + +```far-edge-ops-api.conf``` is a rather typical-looking apache +VirtualHosts file, but with LDAP auth added. First the file: + +##### far-edge-ops-api.conf: + + LDAPTrustedGlobalCert CA_BASE64 "/etc/apache2/auth/uswin_ca.pem" + LDAPTrustedGlobalCert CA_BASE64 "/etc/ssl/certs/selfsigned_root_cert.crt" + LDAPTrustedMode SSL + LDAPLibraryDebug 7 + + <VirtualHost *:3080> + ServerName middleware.<fqdn> + DocumentRoot "/opt/far-edge-ops-api/src/src" + WSGIScriptAlias / /opt/far-edge-ops-api/src/src/far_edge_ops_api/wsgi.py + WSGIDaemonProcess middleware.<fqdn> processes=20 threads=15 display-name=%{GROUP} python-home=/opt/far-edge-ops-api/src/venv + WSGIProcessGroup middleware.<fqdn> + Alias /static/ /opt/far-edge-ops-api/src/src/static/ + TimeOut 600 + SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown + ErrorLog "/var/log/apache2/far-edge-ops-api_error_log" + CustomLog "/var/log/apache2/far-edge-ops-api_access_log" common + </VirtualHost> + + <Directory "/opt/far-edge-ops-api/src/src"> + AllowOverride all + Options FollowSymlinks + AuthName "USWIN" + AuthType Basic + AuthBasicProvider ldap + AuthLDAPURL "ldaps://vcpfe-ldap-prod.vcpfe.vzwops.com:636/DC=uswin,DC=ad,DC=vzwcorp,DC=com?sAMAccountName?sub?(objectclass=*)" + AuthLDAPBindDN "CN=SVC-VCP-LDAP,OU=SVC,OU=FNA,DC=uswin,DC=ad,DC=vzwcorp,DC=com" + AuthLDAPBindPassword "<password_for_SVC-VCP-LDAP_account>" + Require valid-user + </Directory> + + <Directory "/opt/far-edge-ops-api/src/src/static"> + AllowOverride limit + Require all granted + Options Indexes FollowSymLinks + </Directory> + +There are two SSL certs because the far-edge stack is IPv6-only while +the production LDAP endpoint (USWIN) is IPv4-only. So we must use a +proxy. The first cert is the actual uswin cert, while the second cert +is the one for the proxy. Both of these certs must be placed on the +app hosts. + +Replace ```<fqdn>``` with the DNS domain. Replace +```<password_for_SVC-VCP-LDAP_account>``` with the correct password. + +OpenLDAP needs to be configured with these same certs. Since they are +self-signed, openldap will reject them unless they are listed as +exceptions. + +#### /etc/ldap/ldap.conf: + + TLS_CACERT /etc/apache2/auth/uswin_ca.pem + LDAPTLS_CACERT /etc/ssl/certs/self_ca_signed_cert_and_key_2.pem + +Now that the configuration is complete, rerun the deployment from +jenkins. This time the application should start normally. + +## Queue hosts + +The services which run on the application hosts are as follows: + + - ansible-queue + - dns-admin + +### ansible-queue + +```ansible-queue``` is a docker containerized common lisp +application. + +#### Build/deploy + +The application can be built and deployed from jenkins: + +https://openbook-jenkins-master.meter.vzwops.com:8080 + +Run the job ```docker-container-ansible-queue``` to build the docker +container, and the appropriate ```ansible-queue-deploy-vcpfe-<env>``` +job to deploy it to an environment. + +#### Configure + +Two files are required in the directory ```/etc/ansible-queue/```: + + - .ansible_pass.far_edge + - options.lisp + +```.ansible_pass.far_edge``` is the ansible vault password +file. Obtain a copy from an existing queue host. + +```options.lisp``` is the config file for the ```ansible-queue``` +application. + +##### options.lisp: + + ((:queue (:num-icinga-process-threads 1 + :num-bmc-process-threads 30 + :num-nic-process-threads 20 + :num-wr-process-threads 25 + :num-dns-process-threads 1 + :num-patch-process-threads 1 + :num-hw-process-threads 20 + :num-fix-process-threads 20 + :icinga-wait-interval 0 + :bmc-wait-interval 1 + :nic-wait-interval 30 + :wr-wait-interval 0 + :dns-wait-interval 0 + :patch-wait-interval 30 + :hw-wait-interval 1 + :fix-wait-interval 1 + :sleep-interval 60 + :ipv6 t + :vault-password-file "/etc/ansible-queue/.ansible_pass.far_edge" + :icinga-queue-url "tcp://*:5555" + :bmc-queue-url "tcp://*:5556" + :nic-queue-url "tcp://*:5557" + :wr-queue-url "tcp://*:5558" + :dns-queue-url "tcp://*:5559" + :patch-queue-url "tcp://*:5560" + :hw-queue-url "tcp://*:5561" + :fix-queue-url "tcp://*:5562" + :middleware-url "https://middleware.vcpfe.vzwops.com" + :middleware-username "SVC-Far-Edge" + :middleware-password "<password_for_SVC-Far-Edge_account>"))) + +Replace ```<password_for_SVC-Far-Edge_account>``` with the correct +password. + +Now that the configuration is complete, rerun the deployment from +jenkins. This time the application should start normally. + +### dns-admin + +```dns-admin``` is a docker containerized web application written in +common lisp and clojurescript. + +#### Build/deploy + +The application can be built and deployed from jenkins: + +https://openbook-jenkins-master.meter.vzwops.com:8080 + +Run the job ```docker-container-dns-admin``` to build the docker +container, and the appropriate ```dns-admin-deploy-vcpfe-<env>``` job +to deploy it to an environment. + +#### Configure + +Only one new config file is required. Use the following content +verbatim: + +##### /etc/dns-admin/conf.lisp: + + (:name "dns-admin" + :document-root "dns-admin" + :title "DNS Administration" + :meta-description "A website for administering DNS. Supports nsupdate and infoblox." + :dns (:label "CDS Infoblox" + :backend-type "infoblox" + :url "https://infoblox.alias/wapi/v2.6")) + +Notice the odd-looking hostname ```infoblox.alias```. The following +needs to be added to ```/etc/hosts```: + + 2607:f160:8a02:8016:a0:103:0:10 infoblox.alias + +Now that the configuration is complete, rerun the deployment from +jenkins. This time the application should start normally. |