Service Protector High Availability Installation and Upgrade
=============================================================
Table of content
0. Scope
1. Installation
2. Troubleshooting
3. Upgrade - Shared Storage Solution
4. Upgrade - DRBD Solution
5. References

0. Scope
=============================================================
This document describes installation and upgrade procedure 
for DDoS Controller (DSC) in High Availability (HA) configuration.

1. Installation
=============================================================
1.0. Installation must be performed by root user. The system includes two nodes
(hosts running Service Protector application). Both nodes are connected to 
management network and shared storage (DataStorage device used in installation 
should be configured in advance, see appropriate manual). Also, there is a 
dedicated network connection between two hosts. Each network connection uses 
two network cards for redundancy. Following scheme shows system connectivity:

       +===========================+     
       |        DataStorage        |
       +===========================+ 
       |     |               |     |   
       | SAS |               | SAS |   
    +-------------+     +--------------+
    | Primary node|     |Secondary node|
    |             |BOND1|              |
    |         NIC2|-----|NIC2          |
    |         NIC3|-----|NIC3          |
    |             |     |              |
    | NIC0 NIC1   |     |   NIC0 NIC1  |
    +-------------+     +--------------+
       |BOND0|               |BOND0|   
       |     |               |     |   
       |     |   DSC-HA-VIP   |     |
       =============================
                   |    |
             Management-Network
             
   * Each network connection is supported by two network cards for redundancy,
     i.e., 4 network cards should be installed on each host prior to system
     installation.     
   * NIC0/1 (BOND0) are connected to Management network, custom IP set on step 
     1.3 below (it may be the same as set on step 1.1). DSC HA VIP should belong
     to the same network as these IPs.
   * NIC2/3 (BOND1) are connected to appropriate ports on pair node, internal 
     IP is set automatically on step 1.3 below.
   
One of the hosts (nodes) used in HA system will be called 'primary' in this
document - please be consistent when choosing primary node during installation
(i.e., use the same host as primary). Primary and Secondary hosts are assigned 
voluntarily during DSC HA configuration (step 3 below). These terms are used for
instruction convenience only and do not mean any difference in functionality.
Pay attention that 

For installation steps 1.1-1.2 details see appropriate Allot installation 
manuals.
Make sure that DataStorage device is disconnected from both hosts when running
steps 1.1-1.2.

1.1. Install ACP17.8.0 or higher version on both hosts. Perform preliminary 
    network configuration (IP address of management connection is configured).

1.2. Install Service Protector v.15.1.10 or higher on both hosts. 

1.3. Configure DSC HA system on both sides.
    1.3.1. Check network connections, verify that storage is connected and 
    switched on. 

    1.3.2. On secondary node run:
        cd /opt/allot/conf/ha
        ./Allot_HA7_Setup.sh
        
        Answer the question:
        Is this a Primary Machine (y/n) [y]? n
        
        Then a following menu appears 
      (hereafter '%' in the beginning of the line represents the screen output):
        
%    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
%           Allot DSC High availability for ACP 17.x Setup menu
%    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
%
% +------------------------------------------------------+
% | Secondary                                            |
% | HOST Name = nms211-116.lab.local                     |
% | GATEWAY =   10.150.0.1                               |
% | VIP =                                                |
% |                                                      |   |BOND0-Management
% | External-Storage = y                                 |-->|
% | Node2 =                                              |   |
% | DNS =       172.18.1.10                              |
% |                                                      |   |BOND1-Cluster
% |                                                      |-->|
% |                                                      |   |
% +------------------------------------------------------+
%
% #     I/F             MAC Address       IP Address         NETMASK     Link
%~~~ ~~~~~~~~~~~~    ~~~~~~~~~~~~~~~~~  ~~~~~~~~~~~~~~~  ~~~~~~~~~~~~~~~  ~~~~
% #) eno1         :  5c:f3:fc:e7:69:2c   10.150.211.116      255.255.0.0   yes
% #) eno2         :  5c:f3:fc:e7:69:2e      Not-defined      Not-defined    no
% #) eno3         :  5c:f3:fc:6b:42:98      Not-defined      Not-defined    no
% #) eno4         :  5c:f3:fc:6b:42:9a      Not-defined      Not-defined    no
% #) ens3f0       :  00:15:17:8a:a8:64      Not-defined      Not-defined    no
% #) ens3f1       :  00:15:17:8a:a8:65      Not-defined      Not-defined    no
%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
%
% HA Configuration :
% ~~~~~~~~~~~~~~~~~~
% 1) Bonding      
%    BOND0/MGT    :  MGT1=              MGT2=             
%    BOND1/Cluster:  CLT1=              CLT2=             
%~~~ ~~~~~~~~~~~~    ~~~~~~~~~~~~~~~~~  ~~~~~~~~~~~~~~~~~
% 2) Continue HA Setup
% 3) Maintanance restore (Currently not implemented)
% 4) Quit
%
%Choose a number of an item you want to configure:     

    The values presented (IP, hostname, DNS etc.) were defined on step I. NIC 
    names and MAC addresses may differ.
    
        Choose an item number to edit/perform, type the value when prompted, 
    press Enter. Start with menu option 1) Bonding (type 1, press Enter). 
    Select the apropriate interfaces for Bond0 and define the node IP & Mask. 
    Select the apropriate interfaces for Bond1.
        When done and return to the main menu, continue with option 2) Continue
    HA Setup. When done, continue on Primary node.

    1.3.3. On primary node run:
        cd /opt/allot/conf/ha
        ./Allot_HA7_Setup.sh
        
        Answer the questions (enter relevant IP values):
% Is this a Primary Machine (y/n) [y]? y    
% Please type the IP address of the VIP [] : 10.150.211.215
% Please type the Management IP address of the Secondary host []: 10.150.211.116

        Then an above menu appears. Start with menu option 1) Bonding, as 
        described in step 2. After bonding configuration, the network service 
        is restarted, it takes some time. When asked, confirm connection to 
        secondary node:
% Are you sure you want to continue connecting (yes/no)? yes
        and enter root password (twice if needed):
% root@192.168.168.2's password:
        
        Continue with option 2) Continue HA Setup. When the configuration 
        process finishes, HA system should run with resources active on one 
        node (dsc service may take aboute a minute more to start after the
        script exits). For solutions of some possible problems see 
        "Troubleshooting" below.
    
2. Troubleshooting
=============================================================
    2.1. If the connection between nodes cannot be established, a message 
    appears: "No connection to 192.168.168.2. Do you want to try again? (y/n)"
    In such a situation, a system may be checked for a problem (all the cables
    are connected etc.) and configuration resumed (type 'y', press Enter).    
        
    2.2. If the script cannot determine shared device, the following menu 
    appears, allowing to select device:
% Model: IBM ServeRAID M5015 (scsi). Disk /dev/sda: 299GB
% Model: IBM 1746 FAStT (scsi). Disk /dev/sdb: 1797GB
% Evailable choice:
% 1) /dev/sda
% 2) /dev/sdb
% 3) Quit
% Select shared device (enter number): 2
    Type needed number (as 2 in above example), press Enter. Installation will
    continue.
    
3. Upgrade - Shared Storage Solution (Physical installations)
=============================================================
    3.1. Copy new Allot_dsc-<version.build>.tgz and Allot_dsc.sh
        files to local directory on both nodes.
    3.2. Stop the service on 'standby' node (use crm_mon command to determine
        which node is in standby mode): 'service pacemaker stop'
    3.3. Stop the service on 'active' node: 'service pacemaker stop'
    3.4. Make sure the relative processes are down on both nodes: run 
        'ps ax |grep -v grep |grep -E "opt|pace" -c'
        The output should be: 0.
    3.5. Mount the storage on the 'primary' server ONLY: 
        'mount /dev/dm-1 /opt/shared'
    3.6. On both nodes: change directory to one where installation files 
         are located; run upgrade  './Allot_dsc.sh -u'
    3.7. Reboot both hosts.

4. Upgrade - Shared Storage Solution (Physical installations)
=============================================================
    4.1. Copy new Allot_dsc-<version.build>.tgz and Allot_dsc.sh
        files to local directory, /opt/admin/insatll/DSC_<VERSION>_B<BUILD>, on both nodes.
    4.2. Disable PostgresSQL and sp/dsc resources, on primary node and secondary node.
        'pcs resource disable dsc/sp PostgresSQL'.
    4.3. Perform upgrade on primary node than do switch-over to the secondary.
    4.4 Perform upgrade on secondary than enable dsc and PostgresSQL resources.
        'pcs resource enable dsc/sp PostgresSQL'.
    4.5 Reboot/Restart pacemaker in both nodes.
    4.6 Make sure that the processes up and running on the desired node.
	
4a. Upgrade resource file('on the fly') after upgrade DSC HA from 16.X to 17.1.10 and higher 
====================================================================
  After DSC upgrade to 17.1.10 and higher from DSC 16.X (change PostgreSQL version from 9.2 to 12.6)
  Run on active node script /opt/allot/conf/ha/update_resources_xml.sh 
  Not required DSC/pacemaker restart (change done "on the fly")
  After check with pcs status - Resource PostgreSQL is started    
    
5. REFERENCES
=============================================================  
1. DSC Installation and Admin Guide
2. README_ACP_17.8.0_CO73
