Connectivity

Run the tracepath command from a host that’s on the same LAN as the NAS, and target the RADIUS Server. Then, do the same thing in reverse.

Example: MTU size

If the trace stops reporting path information at an intermediate hop en route to the destination that doesn’t perform packet filtering, it’s likely that there’s a path MTU restriction. However, a device along the already-probed segment of the path is preventing ICMP "Fragmentation Needed" or ICMP "Too Big" responses generated by the restrictive network device from reaching the source.

In this type of scenario, a broken PMTUD shows a trace like the following:

Execute the trace with a lower MTU e.g. 1400 bytes, allowing the packets to reach the destination. If the packets get closer, but still don’t reach the target, reduce the MTU again and try transmitting.

Example: Missing ICMP Messages

This behaviour is a strong indication that PMTUD is broken between those hosts. The source is not receiving the indications that it needs to reduce the packet size to the destination, therefore it will likely continue to send RADIUS packets that are too big to reach their destination, rather than perform IP fragmentation with a viable fragment size. Broken PMTUD is a fundamental network issue that should be fixed.

Take captures of all network devices along the path to determine where IP packets are being dropped due to an MTU restriction. Determine if ICMP "Fragmentation Needed" or ICMP "Too Big" responses are being generated (as is required), and --- if so --- where these ICMP responses are being dropped prior to reaching the source.

Take simulataneous captures at both the NAS and the RADIUS server and look for an instances of fragments being generated at source for an IP packet.

If the network is functioning correctly, the capture taken at the destination will show the arrival of all fragments. It is okay for these fragments to arrive out of order.

In the rare case that an in-path device is performing IP fragment reassembly (and the local MTU exceeds that which was discovered by the sender) then it is also possible to observe a single, complete reassembled packet.

In even rarer cases, for IPv4 packets you might even observe a different arrangement of fragments representing the original packet, either because an in-path host has performed further fragmentation of the fragments, or because fragment reassembly has occurred and then the IP packet has been subsequently refragmented using a different IP fragment size.

Each of these scenarios is fine provided that the destination host is provided with a complete set of fragments representing the original IP packet containing the RADIUS request.

In the case of RADIUS requests being sent to the RADIUS Server, debugging the RADIUS Server (radiusd -X) will show it processing the RADIUS request from the reassembled IP packet. If tcpdump shows some IP fragments arriving but FreeRADIUS does not receive the RADIUS request, then something has gone wrong in the network resulting in the operating system failing to reassemble the original IP packet --- due to either missing or incorrectly formatted IP fragments.

Missing or broken IP fragments always infers the existance of one or more network devices that exhibit impaired IP behaviour. Impaired IP fragment handling is a fundamental network issue that should be fixed.

Captures should be taken at network devices along the path to determine where IP fragments are being dropped, or incorrectly routed.

Supplicants and authenticators anticipate that the MTU of the network over which EAPoL is performed is a standard size for the link type. Some supplicants will generate EAPoL frames that are the full 1500 bytes of a standard Ethernet MTU and cannot be configured to do otherwise. Even when a supplicant can be configured to use a smaller EAP fragment size, it might not be practical to do so, for example in BYOD environments.

Solution: Increase the access network’s MTU so that it meets the standard for the link type technology. If resizing the MTU isn’t possible, configure all supplicants and authenticators to use a smaller fragment size for EAP messages. Also, configure the NAS to advertise the smaller MTU of the EAPoL network in the Framed-MTU attribute of RADIUS requests sent to the RADIUS Server.

Some wireless lan controllers (WLCs) and switches (that do not support asymmetric fragmentation/reassembly) are unable to encapsulate a large EAP message generated by a supplicant into a RADIUS Access-Request that would need to span multiple IP fragments to satisfy the path MTU to the RADIUS Server.

Solution: Upgrade or replace the NAS with a device that performs proper IP fragmentation.

Some WLCs are unable to de-encapsulate an EAP message from a RADIUS Access-Challenge that is received as a set of IP fragments, even though the EAP message would fit within the link MTU for the EAPoL interface.

Solution: Upgrade or replace the NAS with a device that performs proper IP fragment reassembly.

Some firewalls, routers and network load balancers simply drop all IP fragments on egress or ingress as a matter of policy, for reasons other than a link MTU restriction.

Solution: Reconfigure the malfunctioning network device to permit IP fragments to and from the RADIUS servers.

Some firewalls drop IP fragments for an extended period of time in reaction to some global network condition, such as during a fragment-based network attack. Services that depend on IP fragmentation may therefore work at some times but not others.

Solution: Override such protections for traffic to and from the RADIUS servers, and disable virtual reassembly if necessary to protect the resources of the firewall. Ensure that the RADIUS Server’s operating system is up to date and that the host has sufficient resources to mitigate fragment-based network attacks by itself.

Firewalls and routers may be configured to perform "virtual reassembly" of complete IP packets using all IP fragments for policy inspection purposes. If traffic takes multiple paths such that a single device does not see all IP fragments then reassembly will fail, fragments will be dropped, and excessive resources consumed.

Solution: Disable virtual reassembly for packets involving the RADIUS servers or amend the routing policy to ensure that all fragments to a destination are forwarded via the same path.

Stateless routers and load balancers, as well as load balancers with broken flow cache lookup for IP fragments, may steer subsequent IP fragments to a different backend than the initial fragment.

Solution: Configure the device to steer packets based on the Layer 2 addresses only, and not the Layer 4 information which is only present in the initial fragment. Note: This configuration is normally required for EAP since the source port is not guaranteed to remain the same throughout the authentication exchange.

NAT devices with broken flow cache lookup may either drop or incorrectly rewrite IP fragments and ICMP responses.

Solution: Upgrade or replace the broken device.

Some load balancers route fragments to the correct backend except when a backend is offline, in which case they route fragments incorrectly. A single backend becoming unavailable results in degradation of the entire service.

Solution: Upgrade or replace the broken load balancer.

Some routers and firewalls may filter critical ICMP responses, breaking PMTUD, and resulting in authenticators and/or RADIUS servers continuously sending oversized IP packets. These packets are too large for the path and do not reach their destination.

Solution: Configure devices so as not to filter ICMP messages that are essential for basic network services.

Some devices performing ECMP routing and other forms of network load balancing with broken flow caches will route an ICMP message to a different backend than to where the application data that originated the ICMP response is sent. This breaks PMTUD and results in RADIUS servers continuing to send oversized IP packets instead of performing IP fragmentation.

Solution: Either use a device that performs flow tracking to match ICMP messages with their associated data flows and steer them to the same backend, or broadcast ICMP messages required for PMTUD to all backends.