Iptables based uplink load balancing

Synopsis

You two uplinks wan1 and wan2. The goal is to have traffic load balanced over them.

Routing policies

iproute2 supports routing policies implemented as multiple routing table and routing rule governing when to use which table. A routing rule can consider the source ip or a firewall mark set by iptables.

To set up a new routing table, edit the /etc/iproute2/rt_tables:

#
# reserved values
#
255     local
254     main
253     default
0       unspec
#
# local
#
#1      inr.ruhep
11   wan1
12   wan2

I've added two new routing tables, wan1 and wan2. These new tables are not used yet, as evident by ip rule list. First, we need to add a rule routing traffic originating from the wan interfaces IP address to the corresponding gateway - this may not be necessary if both gateways provide a route 0.0.0.0 and the static local subnet routes exist forcing selection of the appropriate gateway. When in doubt add the policy rule anyway, the do no harm.

ip rule add from $WAN1_IP table 11
ip rule add from $WAN2_IP table 12
ip rule add fwmark 1 table 11
ip rule add fwmark 2 table 12

The fwmark policy rules are necessary for routing traffic that can't have it's fate decided by the routing policy based on source ip address, sometime you can do without them E.G. you are routing traffic (not NAT, routing) from a subnet to multiple upstreams - personally i'd just use a multi-path route on such a case but sometimes crazy things must be done ;). It's really out of scope here anyway.

Iptables

iptables/netfilter include a few modules probabiltyuseful for load balancing setups. The first is statistic, the module can operate in two modes: random and nth, matching at random according to a probability factor or matching every nth packet. statistic can be used for statistical load balancing, which is good if you don't know the actual bandwidth of the connections or for a subset of the traffic.

Second useful module is rateest, a module that implements a traffic rate monitor and allows decision based on comparisons with the counter.

Other useful module are limit, hashlimit, state, mark and connmark.

FW marks

Theres really no point in LB packets, only connections. The last thing i'd want are packets from the same connection dispersed over two different uplinks. This might actually work if you are using true routing, but with NATed uplinks not only will your connections fail, you will also probably be blacklisted as attacker. Even with true routing, this is probably a bad idea. Differences in traversal time will cause packets to arrive out of order and wait longer for recombination.

So really, we need iptables connection tracking and mark packets consistently. The connmark module is just the thing for the job, it marks connection with a special mark - ctmark. ctmark isn't supported outside netfilter, so the connection mark will copied to nfmark, the standard packet mark.

Example LB setup

iptables -t mangle -N LB_FATE
iptables -t mangle -A LB_FATE -m rateest --rateest1 wan1 --rateest1-bps $WAN1_RATE --rateest2 wan2 --rateest2-bps $WAN2_RATE --rateest-delta --rateest-gt -j CONNMARK --set-mark 1
iptables -t mange -A LB_FATE -m connmark ! --mark 1 -j CONNMARK --set-mark 2

iptables -t mangle -A PREROUTING -m state --state NEW -i lan0 -s $LAN_NET -d ! $LAN_NET -j LB_FATE
iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark

iptables -t mangle -A POSTROUTING -m rateest -o wan1 -j RATEEST --rateest-name wan1 --rateest-interval 250ms --rateest-ewmalog 0.5s
iptables -t mangle -A POSTROUTING -m rateest -o wan2 -j RATEEST --rateest-name wan2 --rateest-interval 250ms --rateest-ewmalog 0.5s

iptables -t nat -A POSTROUTING -o wan1 -j SNAT --to-source $WAN1_IP
iptables -t nat -A POSTROUTING -o wan2 -j SNAT --to-source $WAN2_IP

The wan1 routing table:

WAN1_NET dev wan1  proto kernel  scope link  src WAN1_IP 
LAN_NET dev lan  proto kernel  scope link  src LAN_IP 
default dev wan1 via WAN1_GW proto static

The wan2 table is roughly the same with connection specific data of course. The magic is the different default route on both tables.

Rate Estimation and Async lines

DSL and many cable provider are async, so using rate estimation based on outgoing traffic will provide poor results. Instead, measure the rate of incoming traffic and use the rate to determine the route of outgoing traffic. The rate estimation rule will thus be:

iptables -t mangle -A POSTROUTING -m rateest -i wan1 -j RATEEST --rateest-name wan1 --rateest-interval 250ms --rateest-ewmalog 0.5s
iptables -t mangle -A POSTROUTING -m rateest -i wan2 -j RATEEST --rateest-name wan2 --rateest-interval 250ms --rateest-ewmalog 0.5s

-- AvishaiIshShalom - 26 Jul 2010