ARP flux

Понадобилось мне на одном сервере CentOS 6 терминировать два ip-адреса из одной подсети, да так, чтобы у каждого ip был свой MAC и трафик ходил симметрично, т.е. на какой интерфейс пришёл с него же и ушёл.

Не такая простая задача, как кажется на первый взгляд - стек tcp/ip в linux следует
weak host model.


C назначением ip-адресов проблем нет:

# ip addr show dev eth1
4: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:14:5e:cd:44:85 brd ff:ff:ff:ff:ff:ff
    inet 192.168.253.85/28 brd 192.168.253.95 scope global eth1
    inet6 fe80::214:5eff:fecd:4485/64 scope link
       valid_lft forever preferred_lft forever
# ip addr show dev eth2
5: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:14:5e:cd:44:86 brd ff:ff:ff:ff:ff:ff
    inet 192.168.253.86/28 brd 192.168.253.95 scope global eth2
    inet6 fe80::214:5eff:fecd:4486/64 scope link
       valid_lft forever preferred_lft forever

ARP победил следующим образом:
# это, на самом деле, значения по-умолчанию 
sysctl net.ipv4.conf.eth1.arp_filter=0
sysctl net.ipv4.conf.eth2.arp_filter=0
 
#Define different modes for sending replies in response to
#    received ARP requests that resolve local target IP addresses:
#    0 - (default): reply for any local target IP address, #configured
#    on any interface
#    1 - reply only if the target IP address is local address
#    configured on the incoming interface
#    2 - reply only if the target IP address is local address
#    configured on the incoming interface and both with the
#    sender's IP address are part from same subnet on this interface sysctl net.ipv4.conf.eth1.arp_ignore=2
sysctl net.ipv4.conf.eth2.arp_ignore=2
 
#Define different restriction levels for announcing the local
#    source IP address from IP packets in ARP requests sent on
#    interface:
#    0 - (default) Use any local address, configured on any #interface
#    1 - Try to avoid local addresses that are not in the target's
#    subnet for this interface. This mode is useful when target
#    hosts reachable via this interface require the source IP
#    address in ARP requests to be part of their logical network
#    configured on the receiving interface. When we generate the
#    request we will check all our subnets that include the
#    target IP and will preserve the source address if it is from
#    such subnet. If there is no such subnet we select source
#    address according to the rules for level 2.
#    2 - Always use the best local address for this target.
#    In this mode we ignore the source address in the IP packet
#    and try to select local address that we prefer for talks with
#    the target host. Such local address is selected by looking
#    for primary IP addresses on all our subnets on the outgoing
#    interface that include the target IP address. If no suitable
#    local address is found we select the first local address
#    we have on the outgoing interface or on all other interfaces,
#    with the hope we will receive reply for our request and
#    even sometimes no matter the source IP address we announce.

sysctl net.ipv4.conf.eth1.arp_announce=2sysctl net.ipv4.conf.eth2.arp_announce=2
 
ARP заработал.
# ip nei sh dev eth1
192.168.253.92 lladdr 00:90:1a:a3:9c:77 REACHABLE
192.168.253.90 lladdr 00:90:1a:a4:f7:69 REACHABLE
192.168.253.91 lladdr 00:90:1a:a4:25:cd REACHABLE
 
# ip nei sh dev eth2
192.168.253.90 lladdr 00:90:1a:a4:f7:69 REACHABLE
192.168.253.91 lladdr 00:90:1a:a4:25:cd REACHABLE
192.168.253.92 lladdr 00:90:1a:a3:9c:77 REACHABLE

Запросы приходят, сервер отвечает, но трафик с сервера возвращается через один интерфейс, несмотря на то, что на сервере прописаны одинаковые маршруты через оба интерфейса. На linux есть очень продвинутый policy routing. В итоге получилось вот что:

Добавляем в /etc/iproute2/rt_tables таблицу для каждого интерфейса:
11      eth1
12      eth2


# common routes
ip route add 172.31.64.0/19 via 192.168.253.90 dev eth1 table eth1
ip route add 172.31.96.0/20 via 192.168.253.91 dev eth1 table eth1
ip route add 172.31.112.0/20 via 192.168.253.92 dev eth1 table eth1

ip route add 192.168.253.80/28 dev eth1 src 192.168.253.85
ip rule add from 192.168.253.85 table eth1
ip rule add fwmark 0x85/0x85 lookup eth1
 

# common routes
ip route add 172.31.64.0/19 via 192.168.253.90 dev eth2 table eth2
ip route add 172.31.96.0/20 via 192.168.253.91 dev eth2 table eth2
ip route add 172.31.112.0/20 via 192.168.253.92 dev eth2 table eth2

ip route add 192.168.253.80/28 dev eth2 src 192.168.253.86
ip rule add from 192.168.253.86 table eth2
ip rule add fwmark 0x86/0x86 lookup eth2
 
Пришлось полностью отключить uRPF для данных интерфейсов:
# no rpf check
sysctl net.ipv4.conf.eth1.rp_filter=0
sysctl net.ipv4.conf.eth2.rp_filter=0