Sacrificial Funk

I have been working on implementing a new pair of redundant load balancers using the KeepAlived daemon. One of the common problems with failover is that when the passive server takes over the active servers IP addresses, the router can get confused by the MAC address changing. Until the MAC address times out, the router doesn't know how to talk to the new server.

Keepalived uses a method called "gratuitous arp" to send out 5 arp packets onto the ethernet, to let other servers and routers know that the MAC address has changed for a particular IP. However, I have found that the gratuitous ARP method doesn't work well on my Cisco routers. I have also seen many posts on this problem, and last night I found a solution on the Cisco site using SNMP to delete individual entries from Cisco routers. You can download the perl based arp reset script below. It requires the Net-SNMP utilities snmpset and snmpwalk.

MD5SUM
51dbdc2a58fb3d4c0738898b4a166a53 arpreset-0.2.tar.gz

The Cisco router needs to have a read/write community setup for the utility to work. I recommend using the following IOS commands to setup a restricted community that can only work with the MAC table. You will need ENABLE level access to the router to execute these commands.

access-list 50 permit 192.168.1.10
access-list 50 permit 192.168.1.11
access-list 50 deny any
snmp-server view arpchange ipNetToMediaEntry.4 included
snmp-server community badpassword view arpchange RW 50

Set the access-list permit to the IP addresses of the systems you want to be able to make changes to the MAC table. Set the community name (above blahblah) to something random and password-like.

I was setting up a system for 2 failover load balancers with 3 web servers behind them in an LV-NAT setup. Where the load balancers act as NAT routers for the web servers behind them.

              +-+         +-+
              |X|---LB1---|X|----Web1
    Router----|X|         |X|----Web2
              |X|---LB2---|X|----Web3
              +-+         +-+
           switch         switch
   192.168.1.0/24         192.168.2.0/24

In your keepalived.conf file you can cause the arp reset script to be executed each time one of the servers takes over the MASTER role for any of the VIPs (virtual IP addresses) by using the notify_master keyword.

My config looks something like this:

global_defs {
    notification_email {
        admin@nowhere.com
   }
   notification_email_from keepalived@nowhere.com
   smtp_server 192.168.1.100
   lvs_id loadbal001
}

vrrp_sync_group VG1 {
   group {
      VI_OUTSIDE
      VI_INSIDE
   }
}


vrrp_instance VI_OUTSIDE {
    state MASTER
    interface eth0
    lvs_sync_daemon_inteface eth0
    virtual_router_id 101
    priority 150
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass badpassword
    }
    virtual_ipaddress {
        192.168.1.100
    }
    notify_master "/usr/local/bin/arpreset --router=192.168.1.1 \
--community=badpassword --address=192.168.1.100"
}

vrrp_instance VI_INSIDE {
    state MASTER
    interface eth1
    lvs_sync_daemon_inteface eth1
    virtual_router_id 201
    priority 150
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass badpassword
    }
    virtual_ipaddress {
        192.168.2.1
    }
}

virtual_server 192.168.1.100 80 {
    delay_loop 6
    lb_algo wrr
    lb_kind NAT
    nat_mask 255.255.255.0
    persistence_timeout 900
    protocol TCP

    real_server 192.168.2.10 80 {
        weight 1
        HTTP_GET {
            url {
              path /test-file.html
              digest 4465000740005920008800050c000797
            }
            connect_timeout 10
            connect_port 80
            nb_get_retry 3
            delay_before_retry 10
        }
    }

    real_server 192.168.2.11 80 {
        weight 1
        HTTP_GET {
            url {
              path /test-file.html
              digest 4465000740005920008800050c000797
            }
            connect_timeout 10
            connect_port 80
            nb_get_retry 3
            delay_before_retry 10
        }
    }

    real_server 192.168.2.12 80 {
        weight 1
        HTTP_GET {
            url {
              path /test-file.html
              digest 4465000740005920008800050c000797
            }
            connect_timeout 10
            connect_port 80
            nb_get_retry 3
            delay_before_retry 10
        }
    }

}

Note: the notify_master line is broken across two lines for formatting reasons, it needs to be on one line in your keepalived.conf file.