MTU and MSS

MTU

There are different types of MTU

MTU is the largest frame (ethernet) or packet size (TCP/IP) that can be transmitted on an interface.

In a perfect world we have matching MTU on all devices on the path. In the real world is isn't always the case.

DF bit (don't fragment)

If DF bit is set in the IPv4 header that prevents a packet from being fragmented. If a packet arrives that exceeds the MTU then it is dropped. If DF bit not set then it is fragmented.

Ethernet (L1 / Frame)

Usually we are talking about ethernet MTU

How large the maximum amount of data that will be transmitted on that connection

That is 1500 bytes is most cases. 

9000 for jumbo frame, I usually only see this used on SAN storage/switches.

It it set locally on the device, it is not negotiated.

Two devices can have different MTU's

IP MTU

If you have a GRE tunnel. We need space for the GRE encapsulation.

So we lower the MTU

MTU vs MSS

MTU is set manually on interfaces, on each device on the path

MTU is everything including TCP/IP/other protocols and encapsulated data

MSS is just your encapsulated data

MSS is negotiated between client and server at TCP 3 way handshake

536 is the default if MSS isn't exchanged at 3 way handshake

MSS = maximum segment size

When 2 hosts do the TCP 3-way handshake they will negotiate the MSS 

Usually 1460 

1460 + 20 (TCP) + 20 (IP) = 1500

Some other common protocols

GRE 4 byte

PPPOE 8 byte

1430 is a command MSS to leave space for TCP/IP/other protocols

Where to look in wireshark

If you capture with wireshark on the host using its getting the data before it goes to the NIC. A capture on the wire or with a network tap should remove this issue for you.

We need to capture the TCP 3 way handshake

Look in the SYN first

Expand the options side and look for Maximum segment size

Check the SYN-ACK you may see the server (or other network device in the path) adjusted the MSS

Something common to see hosts TX MSS of 1460, but then some middle-box will change that to 1360.

Compare a wireshark from the other side

Did we see 3 way handshake work but when we move to larger packets the connection stops working (retransmissions). The other side never receives the larger packets.

Resets can often make it through as they are small packets. Once side can reset because the big packets were never responded to

Ping command

In windows you use ping to find the MTU

ping -f -l 1490 x.x.x.x

Reduce by 20 until the ping starts responding. Good idea to leave some space you don't want to on the exact size because ping is ICMP and real packets will be TCP. So when you find it reduce again by 20.

Setting the MTU in windows

Once you have identified the MTU issue you can get the other side to lower the MTU on their client. This can resolve the problem. The real fix would be fixing the MTU on the full network path but its not always possible as you won't control everything on the path.

CMD to check the MTU in windows

netsh int ipv4 show subinterface

CMD to set MTU in windows

netsh interface ipv4 set subinterface “Local Area Connection” mtu=1458 store=persistent

Summary

MSS = TCP data (1360/1460)

MTU = IP (1400/1500)

Frame = Ethernet (1414/1514 IIRC)

LRO (Large Receive Offload / TCP Offload) can re-assemble TCP "packets" on the RX side. Same for LSO on the TX side.

TCP might hand the NIC 16K of data and the NIC will chop it up into MSS sized packets 

If you capture with wireshark on the host using its getting the data before it goes to the NIC. A capture on the wire or with a network tap should remove this issue for you.