MultihomedLinuxNetworking < Networking

Note: This article is intended to be written for lartc so the numbering is from 4.

4. Rules - routing policy database

4.1. Simple source policy routing

4.2. Networking of a multihomed host

This article tries to show you my "proof of concept" solution for this problem: We have a multihomed host - a host with multiple up-links (I use "link" to mean the connection to the internet to avoid confusion with TCP connections). How can we use these links efficiently? This is a basic article, so some issues could be missed; most of them are intended to be so, in the interest of simplicity. I'll cover them later in another place.

4.2.1. Network topology

Let's assume that we have 2 DSL internet connections, each with its own modem that allows us to connect to the internet by PPPOE. Setting up PPPOE connections is easy, by using either the rp-pppoe or the pppoe kernel module. If you want to know more about PPPOE, go to http://selab.edu.ms/twiki/bin/view/Networking/PPPoEWithLinux to read my experience about PPPOE.

After setting up PPPOE, we have 2 ppp connections up and running: ppp0 with IP address of 11.1.1.1 and ppp1 with IP address of 22.2.2.2, but we have to set up PPPOE to not set the default route because we will do it manually in order to load balance 2 links.

                                              /
+-----------------+   +---------+ DSL link X / +------
| 11.1.1.1  ppp0  +---+ Modem X +--------------+ ISP
|                 |   +---------+           |  +------
| Linux host      |                         |        Internet
|                 |   +---------+           |  +------
| 22.2.2.2  ppp1  +---+ Modem Y +--------------+ ISP
+-----------------+   +---------+ DSL link Y \ +------
                                              \

4.2.2. Separate connections from different links

A friend with an IP address of X.X.X.X wants to connect to 11.1.1.1. Another friend from Y.Y.Y.Y wants to connect to 22.2.2.2. We want to work with both of them simultaneously. It's clear that ISPs have the responsibility to route packets coming from them to our host via the correct link, so our job is just to route outgoing packets from us to them via the corresponding interface.
Note: We have to route to corresponding because most ISPs will DROP ip spoofing packets.

Outgoing packets belonging to the link X connected to our host have a source of 11.1.1.1, and outgoing packets of the connection from Y have the source of 22.2.2.2. Thus we need to create 2 tables - one for each of the sources:

# ip rule add prio 1 from 11.1.1.1 lookup 1
# ip rule add prio 2 from 22.2.2.2 lookup 2

Then set default routes for packets from each source

# ip route add table 1 to default dev ppp0
# ip route add table 2 to default dev ppp1

Now we can handle connections from both interfaces. If we want to run a web server on the host, we can set up a round robin DNS to load balance requests effectively.

4.2.3. Spread outgoing connections over multiple links for outgoing connections

Now we try to ping gnu.org to see what we have:

# ping 199.232.76.164
connect: Network is unreachable

another try:

# ping -I 11.1.1.1 199.232.76.164
PING 199.232.76.164 from 11.1.1.1 : 56(84) bytes of data.
64 bytes from 199.232.76.164: icmp_seq=1 ttl=47 time=151 ms

We realize that we can ping only if we bind the ICMP packet to a proper IP address! It's because we have only created routes for packets having source addresses of 11.1.1.1 and 22.2.2.2, while packets initializing new outgoing connections don't have any source address yet.

We can solve this by creating a generic route for all packets that don't match any of our 2 source addresses. In this case, we will spread these packets equally to our links to share the loads. This technique is called multipath route:

# ip route add to default nexthop dev ppp0 nexthop dev ppp1

Now we have finished load balancing our host to the internet using 2 different links.

Note that this type of load balancing will not be perfect, as it is connection based and route based. Some connections could use more bandwidth than the others, and routes are cached, which means that routes to often-used sites may always be over the same link. But in most cases, in the long run, you will have an acceptable balance between 2 links.

4.2.4. Configuring the multihomed host as an internet gateway

Usually, only 1 broadband line is enough for most single hosts. So when we have a multihomed host, usually it's because that host will serve as the internet gateway for a big LAN. So we will discuss MASQUERADE/SNAT in a multihomed server. Firstly - IP forwarding is required:

# echo 1 > /proc/sys/net/ipv4/ip_forward

Secondly - we must bind each connection to just one link. It's important because most ISPs prohibit IP spoofing, so each connection has to choose just one link to go through, otherwise it is likely that the packet will be dropped silently by the ISP.

The first packet of each connection is free to choose it's link (the decision is made in the multipath route of the routing system), then all following packets must go through it. Using common techniques, we have no clue for netfilter to recognize which link the first packet chose. However, CONNMARK is here to save our life. It can store a connection-persistent value in the state machine so we can use it to mark the connection the routing system choose for the first packet, and following packets just use that.
Note: From kernel 2.6.10, CONNMARK is included in the vanilla kernel; prior to that, you have to patch the kernel with patch-o-matic to get that feature.

Now, mark first packets based on the interface selectected by the multipath route

# iptables -A POSTROUTING -t mangle -j MARK --set-mark 1 \
-m state --state NEW -o ppp0
# iptables -A POSTROUTING -t mangle -j MARK --set-mark 2 \
-m state --state NEW -o ppp1
# iptables -A POSTROUTING -t mangle -j CONNMARK --save-mark \
-m state --state NEW

We restore the saved mark for following packets before it reaches the routing system

# iptables -A PREROUTING -t mangle -j CONNMARK --restore-mark

Route following packets based on the restored mark

# ip rule add fwmark 1 lookup 1
# ip rule add fwmark 1 lookup 2

The last step: Source NAT them to the public IP addresses before sending to the internet so the returning packets know where to come back to us

# iptables -A POSTROUTING -t nat -m mark --mark 1 \
-j SNAT --to-source 11.1.1.1
# iptables -A POSTROUTING -t nat -m mark --mark 2 \
-j SNAT --to-source 22.2.2.2

Note: Some people think SNAT is just another name for MASQUERADE, but actually MASQUERADE is a special case of SNAT. Don't try to MASQUERADE with multipath routes; you may get undesired effects.

Now you will see that the computers in the LAN can access the internet via both links perfectly. Congratulations!

4.2.5. Dealing with DHCP links and routers

Under certain circumstances you may need to use existing routers instead of PPPOE. For example, when you have internet gateways available in your LAN already, and you want to route packets there to make sure that you do everything correctly before changing the topo of your network. Another example is that your ISPs assign the IP of your eth interface by DHCP.

If you have 1 NIC for each router, everything is simple. Just replace ppp interfaces by eth interfaces. For example, if we have 2 routers 10.0.0.1 and 10.0.0.2 connected to eth1, then we replace

# ip route add table 1 to default dev ppp0
# ip route add table 2 to default dev ppp1
# ip route add to default nexthop dev ppp0 nexthop dev ppp1

with

# ip route add table 1 to default via 10.0.0.1
# ip route add table 2 to default via 10.0.0.2
# ip route add to default nexthop via 10.0.0.1 nexthop via 10.0.0.2

Previously, we had rules:

# iptables -A POSTROUTING -t mangle -j MARK --set-mark 1 \
-m state --state NEW -o ppp0
# iptables -A POSTROUTING -t mangle -j MARK --set-mark 2 \
-m state --state NEW -o ppp1

the conditions "-o ppp0" or "-o ppp1" can help MARK-ing different links with different values. When there is more than one router wired to one network interface (all connected to a switch), we will have trouble because although links have different via values - "via 10.0.0.1" and "via 10.0.0.2", the outgoing interface is the same - eth1. Thus just replacing outgoing interface (-o match) will make 2 identical conditions.

In this case, we will need rules that match based on "via" value ("via 10.0.0.1" or "via 10.0.0.2"). Samuel Jean (http://cookinglinux.org/tm.html) wrote a nexthop patch-o-matic that does that, but the netfilter team has refused to add it to the POM repository so you can't find it on netfilter's website. Samuel makes it available at www.cookinglinux.org/pub/netfilter/nexthop/

With the nexthop patch applied, we replace

# iptables -A POSTROUTING -t mangle -j MARK --set-mark 1 \
-m state --state NEW -o ppp0
# iptables -A POSTROUTING -t mangle -j MARK --set-mark 2 \
-m state --state NEW -o ppp1

with

# iptables -A POSTROUTING -t mangle -j MARK --set-mark 1 \
-m state --state NEW -m nexthop --nexthop-ip 10.0.0.1
# iptables -A POSTROUTING -t mangle -j MARK --set-mark 2 \
-m state --state NEW -m nexthop --nexthop-ip 10.0.0.2

4.2.6. A real world application

Now you know the concept, but you will soon realize that you want more - a complete solution that can solve all kinds of multihomed networks. For example, when the PPPOE links have dynamic IP addresses, or you have cable links where they assign your IP addresses by DHCP, then your SNAT rules must be changed frequently. You need a daemon to watch everything and tune the routes for you.

I've been working on it, a GPL licensed daemon that does that is in beta state. Go to http://selab.edu.ms/twiki/bin/view/Networking/RoutesKeeperProject for more info
to top

End of topic
Skip to action links | Back to top