When a packet leaves a Linux machine, the kernel does not simply blast it out the first network interface it finds. It performs a structured decision process against a set of routing tables, consulting rules in a specific order, applying longest-prefix matching against a compressed trie data structure, and finally selecting a next-hop and output interface. Understanding this pipeline -- not just the commands, but the architecture underneath -- is what separates administrators who can fix routing problems from those who restart the interface and hope for the best.
This guide covers the full picture: the kernel's Forwarding Information Base (FIB), the main and local routing tables, the fields in every route entry, the tools for managing routes, and finally policy routing with the Routing Policy Database (RPDB) for those situations where destination-based routing alone is not enough.
How the Kernel Routes a Packet
Before looking at any commands, it helps to understand what the kernel is doing mechanically. When an outbound IPv4 packet is generated, or an inbound packet needs to be forwarded, the kernel calls fib_lookup(). This function queries the Forwarding Information Base -- the kernel's internal routing database -- with a lookup key that includes the destination IP address, source address, incoming interface, Type of Service field, and firewall mark.
The FIB is not a flat table. Since kernel 2.6.39, Linux stores IPv4 routes in an LPC-trie (Level- and Path-Compressed trie), also called a FIB trie. This data structure places all route prefixes into a binary trie and applies path compression and level compression to minimize the number of nodes. The result is that even routing tables with hundreds of thousands of entries can be searched in roughly 50 nanoseconds on modern hardware -- fast enough to keep up with 10 Gbps line rates on a single core.
You can inspect FIB trie statistics directly at /proc/net/fib_triestat. This file shows the number of leaves, prefixes, internal nodes, and average lookup depth for each routing table on the system. It is a useful diagnostic when you suspect routing table size is affecting performance.
The lookup finds the longest matching prefix -- the most specific route that covers the destination address. If the destination is 192.168.0.5 and the table contains entries for 0.0.0.0/0, 192.168.0.0/16, and 192.168.0.0/24, the kernel selects the /24 entry because it is the most specific match. Only if no more-specific prefix exists does the kernel fall back to a shorter one.
Based on the result, the kernel either delivers the packet locally via ip_local_deliver() or forwards it to the next hop via ip_forward(). The routing entry tells the kernel which output interface to use and the address of the next-hop gateway, if one exists.
The Routing Tables: main, local, and default
Linux supports multiple independent routing tables simultaneously. The kernel tracks each table by a u32 integer, and the kernel header defines RT_TABLE_MAX=0xFFFFFFFF, so table IDs from 0 to 4,294,967,295 are valid. There is a two-tier mechanism behind this: the legacy rtm_table field in struct rtmsg is an unsigned char, limiting it to 0--255; for any ID above 255, the kernel uses the RTA_TABLE rtnetlink attribute, which is a full 32-bit field and takes precedence over rtm_table when present. Tools like ip route handle this transparently -- you can specify table 1000 on the command line without any entry in /etc/iproute2/rt_tables, and the kernel will accept it. Three IDs are reserved for built-in tables; the rest are available for administrator-defined policy routing. Each table operates exactly like a traditional routing table -- a collection of route entries keyed on destination prefix -- and the kernel chooses which table to consult based on rules in the RPDB.
The three built-in tables are:
- local (table 255) -- maintained automatically by the kernel. It contains routes for locally hosted IP addresses and broadcast addresses. When you assign an IP to an interface, the kernel adds a corresponding entry here. You should not manually modify this table.
- main (table 254) -- the table that tools like
ip routeread and write by default. This is where static routes, DHCP-assigned routes, and dynamically learned routes from daemons like FRRouting or BIRD end up. - default (table 253) -- empty by default. Reserved for post-processing fallback if no other rule matches. Rarely used in practice.
The table names and their numeric IDs are defined in /etc/iproute2/rt_tables. You can add custom entries to this file to create named tables for policy routing, which is covered later in this article.
To view the local table -- which is hidden from the default ip route output -- run ip route show table local. You will see entries for each IP address assigned to your interfaces, plus broadcast addresses. This is useful when debugging why traffic to a local IP is not reaching its destination.
Reading the Routing Table
The ip route command from the iproute2 suite is the correct tool for reading and modifying routing tables. The older route and netstat -r commands still work on many systems but are deprecated. They lack support for policy routing and ECMP next-hops, their IPv6 support is rudimentary, and their output format is less precise.
$ ip route show default via 192.168.1.1 dev eth0 proto dhcp src 192.168.1.105 metric 100 192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.105 metric 100 10.10.0.0/16 via 10.10.0.1 dev eth1 proto static metric 200
Each field in a route entry carries specific meaning:
- Destination -- the network prefix this route matches.
defaultis shorthand for0.0.0.0/0, matching any address not covered by a more specific route. - via -- the next-hop gateway address. If absent, the destination is directly reachable on the connected network (scope link).
- dev -- the output interface.
- proto -- who created this route.
kernelmeans the kernel added it automatically when an interface came up.dhcpmeans a DHCP client installed it.staticmeans an administrator added it manually. - scope -- the distance to the destination.
linkmeans the destination is on the directly attached network segment.global(the default when not shown) means routing via a gateway is required. - src -- the preferred source address for packets matching this route. The kernel uses this when selecting which local IP to put in the source field of outbound packets.
- metric -- a tie-breaker when multiple routes match the same prefix. Lower metric wins.
Looking Up a Specific Route
The ip route get command shows exactly which route the kernel would select for a given destination -- including the resolved next-hop and output interface. This is the most reliable way to debug routing decisions because it reflects what the kernel actually sees, not just what is in the table.
$ ip route get 8.8.8.8 8.8.8.8 via 192.168.1.1 dev eth0 src 192.168.1.105 uid 1000 cache $ ip route get 192.168.1.200 192.168.1.200 dev eth0 src 192.168.1.105 uid 1000 cache
Notice that the first lookup selects a gateway because 8.8.8.8 is not on a directly connected network. The second lookup has no gateway -- the kernel knows the destination is reachable directly on eth0 because 192.168.1.0/24 is a connected route with scope link. The uid field reflects the user ID of the process that issued the query and only appears when running as a non-root user; the same command run as root will omit it.
Adding and Removing Routes
Static routes are added with ip route add and removed with ip route del. Routes added this way are ephemeral -- they exist only until the network is restarted or the system reboots. Persistence requires writing routes into distribution-specific configuration files.
# Route to a specific network via a gateway $ sudo ip route add 10.50.0.0/16 via 192.168.1.1 dev eth0 # Default route (gateway of last resort) $ sudo ip route add default via 192.168.1.1 dev eth0 # Host route (single IP address) $ sudo ip route add 203.0.113.5/32 via 10.0.0.1 # Remove a route $ sudo ip route del 10.50.0.0/16 # Replace a route atomically (add if missing, update if present) $ sudo ip route replace 10.50.0.0/16 via 192.168.1.254 dev eth0
Routes added with ip route add are lost on reboot or when the network service restarts. On systems using Netplan (Ubuntu 18.04+), persist routes in /etc/netplan/*.yaml. On RHEL/CentOS systems using NetworkManager, use nmcli connection modify or add a route-<ifname> file in /etc/sysconfig/network-scripts/. On Debian with ifupdown, add up ip route add ... lines to /etc/network/interfaces.
Route Types
Linux supports several route types beyond the standard unicast route. Each type produces different kernel behavior when a packet matches it:
- unicast -- the default. Forward the packet toward the specified next-hop.
- blackhole -- silently drop matching packets. No ICMP is generated. Useful for null-routing abusive sources.
- unreachable -- drop the packet and send an ICMP "network unreachable" message back to the sender.
- prohibit -- drop the packet and send an ICMP "communication administratively prohibited" message.
- throw -- fail the lookup in the current table and return to the RPDB to try the next rule. Used in multi-table policy routing setups.
# Null-route an abusive source without generating any ICMP $ sudo ip route add blackhole 203.0.113.0/24 # Generate a proper unreachable error for a specific range $ sudo ip route add unreachable 10.99.0.0/16
Multipath Routing and ECMP
Linux supports Equal-Cost Multipath (ECMP) routing, which allows a single route entry to specify multiple next-hops with individual weights. The kernel distributes traffic across the paths using a flow hash, keeping individual flows on the same path to avoid packet reordering. By default (fib_multipath_hash_policy=0), the hash is computed from the source address, destination address, and IP protocol number -- three Layer 3 fields. This can be changed to a full 5-tuple Layer 4 hash (adding source port and destination port) by setting the sysctl net.ipv4.fib_multipath_hash_policy=1, which is preferable when many flows share the same IP pair and protocol.
# Two equal-weight paths to the same destination $ sudo ip route add 10.0.0.0/8 \ nexthop via 192.168.1.1 dev eth0 weight 1 \ nexthop via 192.168.2.1 dev eth1 weight 1 # Weighted -- eth0 carries twice the traffic of eth1 $ sudo ip route add default \ nexthop via 192.168.1.1 dev eth0 weight 2 \ nexthop via 192.168.2.1 dev eth1 weight 1
Weighted ECMP is useful for multi-homed servers with asymmetric uplinks. If one link has twice the bandwidth of the other, set its weight to 2 so the kernel sends proportionally more traffic across it.
IPv6 Routing
IPv6 routing follows the same conceptual architecture as IPv4. The kernel maintains a separate FIB for IPv6 prefixes -- historically implemented as a hash table (fib6_table) rather than the LPC-trie used for IPv4, though significant improvements to IPv6 multipath and route lookup were added in kernel 4.11 and later. The ip -6 route subcommand operates on it identically to the IPv4 counterpart, with the obvious difference that addresses and prefixes are in IPv6 notation.
# Show the IPv6 routing table $ ip -6 route show ::/0 via fe80::1 dev eth0 proto ra metric 1024 expires 1799sec pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium 2001:db8::/32 dev eth0 proto kernel metric 256 pref medium # Add an IPv6 static route $ sudo ip -6 route add 2001:db8:100::/48 via 2001:db8::1 dev eth0 # Resolve which route the kernel selects for a destination $ ip -6 route get 2001:4860:4860::8888
One IPv6-specific detail worth knowing: link-local addresses (fe80::/10) are always scoped to an interface. When specifying a link-local next-hop, you must include the interface name with the dev flag, or the kernel cannot resolve the ambiguity if multiple interfaces are present.
Policy Routing and the RPDB
Standard routing makes decisions based purely on destination address. Policy routing extends this to allow decisions based on source address, incoming interface, firewall mark, or Type of Service. The mechanism that implements this in Linux is the Routing Policy Database (RPDB).
The RPDB is a linear list of rules, each with a numeric priority (lower number = higher priority) and an action. When the kernel needs to route a packet, it scans the RPDB from priority 0 upward. For each rule, it applies the rule's selector to the packet. If the packet matches, the rule's action executes -- typically a lookup in a specified routing table. If that table contains a matching route, the lookup succeeds. If not, the kernel continues to the next rule.
Classic routing algorithms make routing decisions based only on the destination address of packets. Policy routing replaces the conventional destination-based routing table with the RPDB, which selects routes by executing a set of rules. -- Alexey Kuznetsov, ip-cref (iproute2 reference documentation)
At boot, the kernel configures three default RPDB rules:
$ ip rule list 0: from all lookup local 32766: from all lookup main 32767: from all lookup default
Rule 0 is hardwired and cannot be removed. It always runs first, checking the local table for locally hosted addresses. Rule 32766 is the normal case -- consult the main table. Rule 32767 checks the empty default table. For the vast majority of Linux systems, all routing happens through this three-rule default RPDB.
Source-Based Policy Routing in Practice
A canonical use case for policy routing is a server with two uplinks to different ISPs. You want traffic that originates from the IP assigned to ISP-A's interface to always exit through ISP-A's gateway, and traffic from ISP-B's interface to always exit through ISP-B's gateway. Standard destination-based routing cannot express this -- it has no concept of "route traffic differently based on which source IP it uses."
The solution is to create a custom routing table for each ISP, populate it with a default route via that ISP's gateway, then add an RPDB rule that matches traffic from that source IP and directs it to the appropriate table.
# Add custom table names 200 isp_a 201 isp_b
# Populate the ISP-A table with its default route $ sudo ip route add default via 198.51.100.1 dev eth0 table isp_a $ sudo ip route add 198.51.100.0/30 dev eth0 scope link table isp_a # Populate the ISP-B table with its default route $ sudo ip route add default via 203.0.113.1 dev eth1 table isp_b $ sudo ip route add 203.0.113.0/30 dev eth1 scope link table isp_b # Add RPDB rules -- match on source address, route to correct table $ sudo ip rule add from 198.51.100.2 lookup isp_a priority 100 $ sudo ip rule add from 203.0.113.2 lookup isp_b priority 101 # Verify $ ip rule list 0: from all lookup local 100: from 198.51.100.2 lookup isp_a 101: from 203.0.113.2 lookup isp_b 32766: from all lookup main 32767: from all lookup default
Rules and routes added with ip rule and ip route are ephemeral -- they disappear when the network is restarted or the system reboots. To make policy routing persistent, you must encode the rules and routes in your distribution's network configuration files. On systems using NetworkManager, the nmcli connection modify command accepts ipv4.routing-rules and ipv4.routes parameters that survive reboots.
Firewall Mark Routing
RPDB rules can also match on firewall mark (fwmark). iptables or nftables sets a numeric mark on a packet, and the RPDB can route marked packets to a different table. This technique is commonly used with VPNs and split-routing setups where you want specific applications -- identified by their packets being marked by a cgroup or iptables owner rule -- to take a different network path.
# Mark packets from a specific UID with mark 0x1 $ sudo iptables -t mangle -A OUTPUT -m owner --uid-owner 1001 -j MARK --set-mark 0x1 # Add a routing table for marked traffic $ sudo ip route add default via 10.8.0.1 dev tun0 table 100 # Route marked packets through the VPN table $ sudo ip rule add fwmark 0x1 lookup 100 priority 50
Debugging Routing Problems
Routing issues typically fall into a few categories: wrong gateway, missing route, wrong source address selection, or policy routing interference. A systematic approach resolves most of them quickly.
# 1. Check which route the kernel actually selects $ ip route get 8.8.8.8 # 2. Check which route the kernel selects from a specific source $ ip route get 8.8.8.8 from 203.0.113.2 # 3. View all RPDB rules (policy routing) $ ip rule list # 4. View a specific custom table $ ip route show table isp_b # 5. View all routes across all tables $ ip route show table all # 6. Check the local table for missing local routes $ ip route show table local # 7. Check the ARP table for next-hop resolution issues $ ip neigh show # 8. Trace the path with explicit interface binding (traceroute package; flag differs on other implementations) $ traceroute -i eth1 8.8.8.8
The most powerful debugging command is ip route get with the from flag. Standard ip route get 8.8.8.8 simulates outbound traffic. Adding from 203.0.113.2 simulates a packet originating from that source address, which triggers the full RPDB evaluation including source-based policy rules. If the two commands return different routes, you have a policy routing rule active that is affecting traffic from that source.
When you suspect routing asymmetry -- packets going out one interface but replies arriving on another -- run ip route show table all | grep <gateway-ip> to see if that gateway appears in multiple tables. Asymmetric routing causes stateful firewalls and conntrack to drop return packets, producing connections that hang or time out unpredictably.
Persisting Routes Across Reboots
How you persist routes depends on which network management stack your distribution uses. The three most common approaches are:
Netplan (Ubuntu 18.04+)
The gateway4 key was deprecated in Netplan with Ubuntu 22.04. The current syntax defines the default gateway as an explicit route entry using to: default. Using gateway4 still functions but produces a deprecation warning on any modern Ubuntu system.
network: version: 2 ethernets: eth0: dhcp4: no addresses: [192.168.1.105/24] routes: - to: default via: 192.168.1.1 - to: 10.50.0.0/16 via: 192.168.1.1 metric: 200 routing-policy: - from: 192.168.1.105 table: 200 priority: 100
NetworkManager (RHEL / Fedora)
# Add a static route to the eth0 connection profile $ sudo nmcli connection modify eth0 \ +ipv4.routes "10.50.0.0/16 192.168.1.1 200" # Add a routing rule $ sudo nmcli connection modify eth0 \ +ipv4.routing-rules "priority 100 from 192.168.1.105 table 200" # Apply changes $ sudo nmcli connection up eth0
Debian / ifupdown
iface eth0 inet static address 192.168.1.105/24 gateway 192.168.1.1 up ip route add 10.50.0.0/16 via 192.168.1.1 metric 200 up ip rule add from 192.168.1.105 lookup 200 priority 100 down ip route del 10.50.0.0/16 down ip rule del from 192.168.1.105 lookup 200
Putting It Together
Linux routing is a layered system. At the bottom is the FIB -- a compressed trie structure that performs fast longest-prefix lookups across one or more routing tables. Above that sits the RPDB, a prioritized list of rules that controls which table the kernel consults for any given packet. For simple single-homed servers, the default three-rule RPDB and a single main table handle everything. For multi-homed servers, VPN gateways, and anything requiring source-based routing, policy routing gives you the full flexibility to control exactly where every packet goes and how.
The key commands to internalize are ip route show, ip route get, ip rule list, and ip route show table all. These four commands tell you everything the kernel knows about routing on the system. With them and a clear mental model of the RPDB lookup pipeline, diagnosing routing problems becomes a structured exercise rather than a guessing game.