diff options
Diffstat (limited to 'doc/markdown/subcloud-install-remediation.md')
| -rw-r--r-- | doc/markdown/subcloud-install-remediation.md | 253 |
1 files changed, 253 insertions, 0 deletions
diff --git a/doc/markdown/subcloud-install-remediation.md b/doc/markdown/subcloud-install-remediation.md new file mode 100644 index 0000000..753c5fb --- /dev/null +++ b/doc/markdown/subcloud-install-remediation.md @@ -0,0 +1,253 @@ +# Far-Edge Subcloud Installation and Remediation Guide + +## Installation + +A Wind River subcloud installation normally runs with no manual +intervention. In the event that an installation needs to be started +manually: + +- Visit https://middleware.vcpfe.vzwops.com/caas/wrfix/ in a browser. + +- Enter the iLO IP address into the first text field (even though it's + labeled "cluster name"). + +- Type ```wr``` in the playbook key field. + +- Click the submit button. + +If an OK message appears, the job has been successfully sent to +ansible-queue. The remaining steps require an SSH shell on +```root@vcpfe-queue-2-prod-birmingham```: + +- View ```/var/log/lisp/ansible-queue.log``` to ensure that the + playbook started. + +- View the specific ansible log in ```/var/log/ansible/queue/``` to + watch the playbook's progress. + +If you want to watch the OS installation (the first stage of the +overall deployment), attach to the iLO console: + +- Find the subcloud data (see below for the query) and find the + ```ilo_host_address``` and ```bmc_password```. + +- ```# ssh bladecenter-lb-vip.meter.vzwops.com``` + +- ```# do-bmc <ilo_host_address>``` + +- Enter the password when prompted. + +- Type ```vsp``` to start the serial console. + +## Remediation + +All too often it is necessary to remediate failed Wind River +installations. Even successful ones need a visit by an engineer to +ensure that they are in fact complete. It is not possible to discover +this information from the middleware database, the subcloud, nor the +central controller alone. All three of these data sources must be +accessed. TODO: write a dashboard which aggregates the required +information from the central controller, subcloud and database. + +Common causes of failure (this is not an exhaustive list): + +- OAM network unreachable. +- MGMT network unreachable. +- MGMT subnet overlaps with another already-installed subcloud. +- The subcloud does not sync in time after coming online. +- The platform-integ-apps application fails to apply. + +(This does not include servers that have issues which prevent them +from advancing through the automation pipeline far enough to begin a +Wind River installation. These include, but are not limited to, +hardware problems and incorrect firmware versions.) + +### Inspecting a subcloud + +Three data sources were listed above. Let's visit each one in detail. + +#### Middleware database + +The middlware database is our record of what should be installed and +whether or not we have marked it as complete. The ```caas_wrbatch``` +view and ```caas_wrinstallschedule``` table are important in this +context. + +Here is a handy query to pull up all relevant information by cluster +name: + + select b.cluster_name || '.vcpfe.vzwops.com' as cluster_name, + b.*, + s.* + from caas_wrbatch b + left outer join caas_wrinstallschedule s + on b.cluster_id = s.cluster_id + inner join caas_location l + on b.fuze_spm_site_id = l.fuze_spm_site_id + where b.cluster_name in ('<cluster_name>') + order by b.parent_cluster_name asc, + b.cluster_name asc; + +(The expression in the first column is a convenience: it builds a FQDN +for the subcloud that can be directly used in an SSH command from your +laptop.) + +Here is a sample record showing a completed subcloud: + + cluster_name | wsbomagj-d663366-001.vcpfe.vzwops.com + id | 423 + cluster_id | 423 + fuze_spm_site_id | 663366 + parent_oam_vip_hostname | + parent_oam_vip_address | 2001:4888:a12:3221:106:290:0:10 + parent_mgmt_address_range_start | 2001:4888:a12:3220:106:290:0:10 + parent_mgmt_address_range_end | 2001:4888:a12:3220:106:290:0:ffff + parent_mgmt_default_gateway | 2001:4888:a12:3220:106:28:: + parent_mgmt_subnet | 2001:4888:a12:3220:: + pxe_mac_address | 48:df:37:e2:67:34 + ilo_hostname | wsbomagj-663366-rh-pe0e910-001 + ilo_host_address | 2001:4888:2a10:90fd:103:40a:0:e001 + oam_hostname | wsbomagj-663366-rh-pe2e910-001 + oam_host_address | 2001:4888:2a10:90fd:103:40a:0:400 + oam_vip_address | 2001:4888:2a10:90fd:103:40a:0:f400 + oam_default_gateway | 2001:4888:2a10:90fd:103:2a0:: + mgmt_address_range_start | 2001:4888:2a10:982b:103:40a:: + mgmt_address_range_end | 2001:4888:2a10:982b:103:40a:0:f + mgmt_default_gateway | 2001:4888:2a10:982b:103:2a0:: + mgmt_subnet | 2001:4888:2a10:982b:103:40a:: + host_vlan | 2845 + oam_vlan | 410 + mgmt_vlan | 420 + cluster_name | wsbomagj-d663366-001 + parent_cluster_name | wsbomagj-c319918-003 + vendor_name | HPE + intel_nic_firmware_version | 1.2585.0 + maint_window_p | f + namespace_id | 425 + namespace_name | wsbomagj-663366vzwcvdu-y-ss-x-05690012222 + bmc_username | k8sctl + bmc_password | <redacted> + id | 709 + date_scheduled | 2020-11-19 19:15:16.885864+00 + date_completed | 2020-12-04 19:28:17.644994+00 + date_last_failed | 2020-12-03 20:47:18.199344+00 + kirke_ticket_number | 709 + kirke_ticket_status | + kirke_ticket_completed | 2020-11-20 02:28:30.908054+00 + cluster_id | 423 + +The key points: + +- ```pxe_mac_address``` is not null +- ```intel_nic_firmware_version``` == 1.2585.0 +- ```date_completed``` is not null +- ```kirke_ticket_number``` is not null + +The last point is the crucial one. Setting ```kirke_ticket_number``` +to something other than null is the way we manually mark the Wind +River installation as complete. It must be done with a query similar +to the following: + + update caas_wrinstallschedule + set kirke_ticket_number = id + where id = 709; + +#### Central controller + +Use the following command to see the status of every subcloud attached +to a central controller: + + dcmanager subcloud list + +If the subcloud in question is complete and healthy from the central +controller's point of view, it will look like the following. This does +not necessarily mean that all is well. The subcloud itself must be +visited and checked. + + [sysadmin@controller-0 ~(keystone_admin)]$ dcmanager subcloud list | grep wsbomagj-d663366-001 + | 546 | wsbomagj-d663366-001 | managed | online | complete | in-sync | + +#### Subcloud + +If the central controller says that the subcloud is offline, the first +thing to check is the mgmt network. If you cannot ping the central +controller's mgmt IP, there is no hope for remediating the subcloud. + + ping6 <parent_mgmt_address_range_start> + +If the subcloud is managed, online and in-sync, then the remaining +checks can be performed. Check the alarms: + + fm alarm-list + +Check the applied status of the applications: + + [sysadmin@controller-0 ~(keystone_admin)]$ system application-list + +--------------------------+----------+-----------------------------------+------------------------------------+---------+-----------+ + | application | version | manifest name | manifest file | status | progress | + +--------------------------+----------+-----------------------------------+------------------------------------+---------+-----------+ + | cert-manager | 20.06-4 | cert-manager-manifest | certmanager-manifest.yaml | applied | completed | + | nginx-ingress-controller | 20.06-0 | nginx-ingress-controller-manifest | nginx_ingress_controller_manifest. | applied | completed | + | | | | yaml | | | + | | | | | | | + | oidc-auth-apps | 20.06-26 | oidc-auth-manifest | manifest.yaml | applied | completed | + | platform-integ-apps | 20.06-9 | platform-integration-manifest | manifest.yaml | applied | completed | + +--------------------------+----------+-----------------------------------+------------------------------------+---------+-----------+ + +Check the drbd resize. drbd8 should be set to 32GB: + + [sysadmin@controller-0 ~(keystone_admin)]$ lsblk | grep drbd8 + │ └─drbd8 147:8 0 32G 0 disk /var/lib/docker-distribution + +Check the PTP configuration. ```tx_timestamp_timeout``` should be set +to 50: + + [sysadmin@controller-0 ~(keystone_admin)]$ grep tx_timestamp_timeout /etc/ptp4l.conf + tx_timestamp_timeout 50 + +Check the ISOL CPU configuration which should be enabled on cores 3, +4, 27 and 28: + + [sysadmin@controller-0 ~(keystone_admin)]$ system host-cpu-list controller-0 | grep -F 'Application-isolated' + | 72499a12-e2f2-40fc-a069-a2b002908843 | 3 | 0 | 3 | 0 | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | Application-isolated | + | 8498669d-9048-45f6-b9dd-c81262934b70 | 4 | 0 | 4 | 0 | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | Application-isolated | + | 4a70893f-b2cf-4405-9b69-11ed823a81b3 | 27 | 0 | 3 | 1 | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | Application-isolated | + | 1d6e330d-7d2f-416c-b375-9b909d6a1f3c | 28 | 0 | 4 | 1 | Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz | Application-isolated | + +### Marking an installation as complete + +If the subcloud passes all of the checks, it should be marked complete +with the update query: + + update caas_wrinstallschedule + set kirke_ticket_number = id + where id = <id>; + +### Remediating + +If the subcloud fails to pass all of the checks, it must be +remediated. Deciding which steps to take requires operational +experience. Here is a guide to some common problems and solutions: + +- The subcloud is unmanaged: ping the central controller mgmt IP. If + it fails, the network is misconfigured and must be fixed by the + region. + +- ```platform-integ-apps``` is not applied: perform a wipedisk and + reinstall. + +- ```oidc-auth-apps``` is uploaded but not applied: run the + ```wr_remediate``` playbook. + +- The drbd resize hasn't happened: run the ```wr_remediate``` + playbook. + +- PTP isn't configured: run the ```wr_remediate``` playbook. + +- ISOL CPUS are not configured: run the ```wr_remediate``` playbook. + +The very last thing that the ```wr-installer``` playbook does is a +```collect all```. You can tell if the playbook ran to completion by +checking the contents of the ```/scratch``` directory on the +subcloud. If it contains a collect tarball, the playbook completed. + |