General
The AWS Transit Gateway MTU Black Hole: Why 8500-Byte Jumbo Frames Silently Drop Traffic
Oct 26, 2026/10 min read
calricon / field note
AWS EC2 instances natively default to 9001-byte Jumbo Frames, but AWS Transit Gateway enforces an 8500-byte MTU limit. When enterprise security groups block ICMP Type 3 Code 4 packets, Path MTU Discovery fails—causing large API payloads and database transfers to silently hang forever. Here is how to fix it.
Oct 26, 2026/General
Advertisement
<h1>The AWS Transit Gateway MTU Black Hole: Why 8500-Byte Jumbo Frames Silently Drop Traffic</h1>
<p><strong>Few production networking outages in Amazon Web Services are as infuriating and deceptively difficult to diagnose as the "partial connection" anomaly.</strong> An application deployed in a workload VPC establishes a TCP connection to an Aurora PostgreSQL database or an internal microservice residing in a shared-services VPC across an AWS Transit Gateway (TGW). The 3-way TCP handshake completes in less than two milliseconds. Simple HTTP health checks succeed with 200 OK. Standard ICMP pings report zero packet loss. Yet, the moment the application executes a query returning more than ten rows, or attempts to stream a 50KB JSON payload across the transit gateway, the connection freezes completely. No error is thrown; no RST packet is returned. The socket simply hangs in an agonizing limbo until the client hits its 60-second read timeout and aborts.</p>
<p>The culprit is an architectural mismatch baked into the very fabric of AWS cloud networking: <strong>The Transit Gateway MTU Black Hole</strong>. Within an individual Amazon Virtual Private Cloud (VPC), EC2 instances natively operate using <strong>Jumbo Frames with a 9,001-byte Maximum Transmission Unit (MTU)</strong>. However, AWS Transit Gateway enforces an absolute MTU ceiling of <strong>8,500 bytes</strong> for VPC attachments. When an EC2 instance emits a standard 9,000-byte TCP packet with the Don't Fragment (DF) bit set, the Transit Gateway drops it. In theory, Path MTU Discovery (PMTUD) should handle this gracefully. But in enterprise environments where security groups, Network ACLs, and next-generation firewalls aggressively block all ICMP traffic, the required ICMP Type 3 Code 4 error packet is silently incinerated on the wire. The client never learns that its packet was too large, creating an invisible, permanent black hole. Here is an architectural deep dive into why TGW MTU mismatches paralyze cloud networks and the engineering playbook required to eliminate transit black holes permanently.</p>
<h2>The Physics of MTU: The 9001 vs. 8500 Byte Collision</h2>
<p>The Maximum Transmission Unit (MTU) defines the largest size of an unfragmented packet (in bytes) that a network interface can transmit. In AWS, MTU parameters vary wildly across network boundaries:</p>
<ul>
<li><strong>Intra-VPC Traffic:</strong> EC2 instances running in the same VPC communicate over AWS's Nitro network card supporting <strong>9,001-byte Jumbo Frames</strong>. This slashes CPU overhead for high-throughput database transfers and analytical workloads by packing more payload into fewer network interrupts.</li>
<li><strong>AWS Transit Gateway (VPC Attachments):</strong> When traffic crosses an AWS Transit Gateway, the maximum supported MTU is strictly <strong>8,500 bytes</strong>. AWS reserves 501 bytes for internal encapsulation headers (such as Geneve overlay metadata used by the Nitro hypervisor).</li>
<li><strong>Inter-Region TGW Peering & Direct Connect:</strong> Peering attachments between Transit Gateways in different AWS regions, as well as AWS Site-to-Site VPN connections, drop the supported MTU further—down to <strong>1,500 bytes</strong> (or 1,420 bytes across IPSec tunnels).</li>
</ul>
<p>When an application server in <code>vpc-workload</code> sends a large response to an internal client in <code>vpc-shared</code>, the transmitting operating system constructs a TCP segment sized to fit its local interface MTU (9,001 bytes). Subtracting the 20-byte IP header and 20-byte TCP header leaves a Maximum Segment Size (MSS) of <strong>8,961 bytes</strong>.</p>
<blockquote>
<p>"Your server sends an 8,961-byte TCP segment because its local interface says it can. The Transit Gateway drops it because its ceiling is 8,500 bytes. If your firewall blocks ICMP, your network is playing Russian roulette with every packet."</p>
</blockquote>
<h2>How Path MTU Discovery (PMTUD) Collapses in Enterprise Clouds</h2>
<p>RFC 1191 specifies <strong>Path MTU Discovery (PMTUD)</strong> to dynamically discover the lowest MTU along an arbitrary network path without causing packet fragmentation:</p>
<ol>
<li>All modern Linux and Windows kernels set the <strong>DF (Don't Fragment)</strong> bit in the IPv4 header of every outbound TCP packet.</li>
<li>When a 9,001-byte packet hits the Transit Gateway interface, the TGW detects that the packet exceeds its 8,500-byte MTU.</li>
<li>Because the DF bit is set, the TGW cannot fragment the packet. It drops the packet immediately.</li>
<li>Under RFC specifications, the Transit Gateway router generates an <strong>ICMP Type 3, Code 4</strong> packet (<em>"Destination Unreachable: Fragmentation Needed and DF Set"</em>). This packet specifies the next-hop MTU (8,500 bytes) and is sent back to the transmitting EC2 instance.</li>
<li>Upon receiving the ICMP message, the transmitting kernel caches the new path MTU, reduces its TCP MSS accordingly, and retransmits the payload cleanly in smaller chunks.</li>
</ol>
<h2>The Fatal Failure Mode: The Corporate ICMP Massacre</h2>
<p>In theory, PMTUD works seamlessly. In real-world enterprise infrastructure, it is systematically destroyed by well-intentioned security engineers:</p>
<ul>
<li><strong>Security Group Hardening:</strong> Security teams enforce baseline Terraform templates that restrict ingress rules strictly to application ports (e.g., <code>Allow TCP 443</code>, <code>Allow TCP 5432</code>). All ICMP traffic is blocked by default.</li>
<li><strong>NACL Blanket Denies:</strong> Subnet Network Access Control Lists (NACLs) explicitly deny ICMP to prevent internal ping-sweeps or ICMP tunneling attacks.</li>
<li><strong>Inspection VPC Firewalls:</strong> Traffic routed through centralized inspection VPCs (Palo Alto VM-Series, Fortinet FortiGate, or AWS Network Firewall) frequently drops non-TCP/UDP traffic unless specifically whitelisted.</li>
</ul>
<p>When the Transit Gateway emits the ICMP Type 3 Code 4 notification, the packet is instantly annihilated by the transmitting instance's security group or firewall. The transmitting kernel never receives the signal!</p>
<h3>The Anatomy of the Hang:</h3>
<ol>
<li>The TCP handshake succeeds because <code>SYN</code> and <code>SYN-ACK</code> packets are only 60 bytes.</li>
<li>The client issues a short HTTP GET request (300 bytes). It slips through unhindered.</li>
<li>The server prepares a 40KB response. It fills its first TCP segment to the maximum allowable MTU (9,001 bytes) and pushes it into the network socket.</li>
<li>The Transit Gateway drops the packet. The ICMP error is blocked.</li>
<li>The server's TCP stack waits for an ACK. When no ACK arrives, it retransmits the exact same 9,001-byte segment after an exponential backoff timer (RTO: 1s, 2s, 4s, 8s, 16s, 32s).</li>
<li>Every retransmitted packet is dropped at the Transit Gateway.</li>
<li>After 60 to 120 seconds, the client application times out and severs the TCP connection. SREs see healthy metrics, zero CPU spikes, and green load balancer targets—while production transactions fail 100% of the time on large payloads!</li>
</ol>
<h2>The Diagnostic Trap: Why Pings and Handshakes Lie to You</h2>
<p>When on-call engineers receive an alert that a cross-VPC microservice is failing, their first diagnostic step is running <code>ping</code> or <code>curl -I</code>:</p>
<pre><code>$ ping 10.100.4.52
PING 10.100.4.52 (10.100.4.52) 56(84) bytes of data.
64 bytes from https://www.google.com/url?q=http://10.100.4.52&source=gmail&ust=1793082957750000&sa=E: icmp_seq=1 ttl=254 time=1.24 ms
64 bytes from https://www.google.com/url?q=http://10.100.4.52&source=gmail&ust=1793082957750000&sa=E: icmp_seq=2 ttl=254 time=1.18 ms
$ curl -I https://www.google.com/url?q=https://internal-api.corp/healthz&source=gmail&ust=1793082957750000&sa=E
HTTP/1.1 200 OK
Content-Length: 2
</code></pre>
<p>Because synthetic pings use 84-byte packets and health checks return 200-byte responses, they easily fit beneath the 8,500-byte threshold. The engineer closes the incident ticket as a "transient network glitch." Meanwhile, actual production customer traffic carrying large payloads remains 100% blackholed!</p>
<h2>The MTU Hierarchy across AWS Network Primitives</h2>
<table>
<thead>
<tr>
<th>Network Boundary</th>
<th>Maximum Supported MTU</th>
<th>TCP MSS Equivalent</th>
<th>Failure Mode If Unhandled</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>Intra-VPC (EC2 to EC2)</strong></td>
<td>9,001 Bytes (Jumbo)</td>
<td>8,961 Bytes</td>
<td>None (Direct hardware delivery)</td>
</tr>
<tr>
<td><strong>AWS Transit Gateway (VPC Attachment)</strong></td>
<td>8,500 Bytes</td>
<td>8,460 Bytes</td>
<td><strong>Silent Drop if ICMP blocked</strong></td>
</tr>
<tr>
<td><strong>AWS Transit Gateway (Peering Attachment)</strong></td>
<td>8,500 Bytes</td>
<td>8,460 Bytes</td>
<td><strong>Cross-region MTU Black Hole</strong></td>
</tr>
<tr>
<td><strong>AWS Direct Connect (Private VIF)</strong></td>
<td>1,500 or 9,001 Bytes</td>
<td>1,460 or 8,961 Bytes</td>
<td>Depends on on-prem router MTU configuration</td>
</tr>
<tr>
<td><strong>AWS Site-to-Site VPN / IPsec</strong></td>
<td>1,420 – 1,500 Bytes</td>
<td>1,380 – 1,460 Bytes</td>
<td>Severe fragmentation & packet loss</td>
</tr>
</tbody>
</table>
<h2>The Calricon Production Playbook: Eliminating TGW MTU Black Holes</h2>
<p>To permanently resolve MTU black holes across enterprise cloud networks, implement this four-pillar defense-in-depth architecture:</p>
<h3>1. Enforce TCP MSS Clamping on Transit Firewalls and Gateways</h3>
<p>The most robust, fool-proof solution is enforcing <strong>TCP MSS Clamping</strong> at your network transit inspection layer (or on Linux routing instances). MSS clamping intercepts the initial TCP SYN packet during the 3-way handshake and rewrites the advertised Maximum Segment Size to fit the path's bottleneck:</p>
<pre><code># Linux iptables / nftables MSS Clamping Rule for Transit Routers
# Clamps TCP SYN packets traversing the Transit Gateway to 8460 bytes
iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN \
-j TCPMSS --set-mss 8460
# For cross-region or VPN tunnels traversing standard internet hops:
iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN \
-j TCPMSS --clamp-mss-to-pmtu
</code></pre>
<p>When an instance in a 9,001-byte VPC establishes a connection across the Transit Gateway, the router rewrites the SYN packet's MSS to 8,460. Both endpoints immediately agree to never emit packets larger than 8,500 bytes—completely bypassing the need for PMTUD and rendering the connection immune to ICMP blocking!</p>
<h3>2. Explicitly Whitelist ICMP Type 3, Code 4 in Security Groups</h3>
<p>Audit your central Terraform and AWS Security Group baselines. Security teams should never block PMTUD signaling. Add an explicit ingress rule to allow ICMP Fragmentation Needed packets across all VPC subnets:</p>
<pre><code># Terraform Security Group Rule: Permit Path MTU Discovery
resource "aws_security_group_rule" "allow_pmtud" {
type = "ingress"
from_port = 4 # Code 4: Fragmentation Needed
to_port = 3 # Type 3: Destination Unreachable
protocol = "icmp"
cidr_blocks = ["https://www.google.com/url?q=http://10.0.0.0/8&source=gmail&ust=1793082957750000&sa=E"] # Internal corporate RFC 1918 range
security_group_id = https://www.google.com/url?q=http://aws_security_group.app_sg.id&source=gmail&ust=1793082957750000&sa=E
description = "Allow Path MTU Discovery (PMTUD) signaling from Transit Gateway"
}
</code></pre>
<h3>3. Standardize Instance MTU via Cloud-Init and DHCP Options</h3>
<p>If your enterprise topology heavily relies on cross-VPC and hybrid interconnects, eliminate interface asymmetry at the operating system layer. Configure your base AMI (Golden Image) or cloud-init bootstrap scripts to set the default network interface MTU to <strong>8,500 bytes</strong> (or 1,500 bytes if predominantly routing over VPN):</p>
<pre><code># /etc/netplan/01-netcfg.yaml (Ubuntu / Debian)
network:
version: 2
ethernets:
eth0:
dhcp4: true
mtu: 8500 # Matches AWS Transit Gateway native limit
</code></pre>
<h3>4. Deploy Automated MTU Black Hole Probes</h3>
<p>Stop relying on standard ICMP pings in your synthetic monitoring suites. Deploy an automated probe that tests network paths with explicit Don't-Fragment (DF) jumbo packets:</p>
<pre><code># Ping test with Don't Fragment bit set and exact payload size
# 8472 bytes payload + 28 bytes IP/ICMP header = 8500 bytes total!
ping -M do -s 8472 -c 3 10.100.4.52
</code></pre>
<p>If the 8,472-byte probe fails while an 84-byte ping succeeds, trigger an automated P1 alert for an MTU black hole before production traffic is impacted.</p>
<h2>Conclusion</h2>
<p>AWS Transit Gateway simplifies cross-VPC network architecture, but treating it as a transparent, uniform conduit is a critical mistake. The 501-byte difference between intra-VPC 9,001-byte Jumbo Frames and the Transit Gateway's 8,500-byte ceiling—combined with ubiquitous ICMP blocking—creates invisible packet traps that bring down production databases and microservices without warning. By implementing TCP MSS clamping, permitting PMTUD ICMP signaling, standardizing OS interface MTUs, and validating paths with jumbo probes, platform teams can eliminate transit black holes and operate seamless, high-throughput cloud networks.</p>
<p><strong>Few production networking outages in Amazon Web Services are as infuriating and deceptively difficult to diagnose as the "partial connection" anomaly.</strong> An application deployed in a workload VPC establishes a TCP connection to an Aurora PostgreSQL database or an internal microservice residing in a shared-services VPC across an AWS Transit Gateway (TGW). The 3-way TCP handshake completes in less than two milliseconds. Simple HTTP health checks succeed with 200 OK. Standard ICMP pings report zero packet loss. Yet, the moment the application executes a query returning more than ten rows, or attempts to stream a 50KB JSON payload across the transit gateway, the connection freezes completely. No error is thrown; no RST packet is returned. The socket simply hangs in an agonizing limbo until the client hits its 60-second read timeout and aborts.</p>
<p>The culprit is an architectural mismatch baked into the very fabric of AWS cloud networking: <strong>The Transit Gateway MTU Black Hole</strong>. Within an individual Amazon Virtual Private Cloud (VPC), EC2 instances natively operate using <strong>Jumbo Frames with a 9,001-byte Maximum Transmission Unit (MTU)</strong>. However, AWS Transit Gateway enforces an absolute MTU ceiling of <strong>8,500 bytes</strong> for VPC attachments. When an EC2 instance emits a standard 9,000-byte TCP packet with the Don't Fragment (DF) bit set, the Transit Gateway drops it. In theory, Path MTU Discovery (PMTUD) should handle this gracefully. But in enterprise environments where security groups, Network ACLs, and next-generation firewalls aggressively block all ICMP traffic, the required ICMP Type 3 Code 4 error packet is silently incinerated on the wire. The client never learns that its packet was too large, creating an invisible, permanent black hole. Here is an architectural deep dive into why TGW MTU mismatches paralyze cloud networks and the engineering playbook required to eliminate transit black holes permanently.</p>
<h2>The Physics of MTU: The 9001 vs. 8500 Byte Collision</h2>
<p>The Maximum Transmission Unit (MTU) defines the largest size of an unfragmented packet (in bytes) that a network interface can transmit. In AWS, MTU parameters vary wildly across network boundaries:</p>
<ul>
<li><strong>Intra-VPC Traffic:</strong> EC2 instances running in the same VPC communicate over AWS's Nitro network card supporting <strong>9,001-byte Jumbo Frames</strong>. This slashes CPU overhead for high-throughput database transfers and analytical workloads by packing more payload into fewer network interrupts.</li>
<li><strong>AWS Transit Gateway (VPC Attachments):</strong> When traffic crosses an AWS Transit Gateway, the maximum supported MTU is strictly <strong>8,500 bytes</strong>. AWS reserves 501 bytes for internal encapsulation headers (such as Geneve overlay metadata used by the Nitro hypervisor).</li>
<li><strong>Inter-Region TGW Peering & Direct Connect:</strong> Peering attachments between Transit Gateways in different AWS regions, as well as AWS Site-to-Site VPN connections, drop the supported MTU further—down to <strong>1,500 bytes</strong> (or 1,420 bytes across IPSec tunnels).</li>
</ul>
<p>When an application server in <code>vpc-workload</code> sends a large response to an internal client in <code>vpc-shared</code>, the transmitting operating system constructs a TCP segment sized to fit its local interface MTU (9,001 bytes). Subtracting the 20-byte IP header and 20-byte TCP header leaves a Maximum Segment Size (MSS) of <strong>8,961 bytes</strong>.</p>
<blockquote>
<p>"Your server sends an 8,961-byte TCP segment because its local interface says it can. The Transit Gateway drops it because its ceiling is 8,500 bytes. If your firewall blocks ICMP, your network is playing Russian roulette with every packet."</p>
</blockquote>
<h2>How Path MTU Discovery (PMTUD) Collapses in Enterprise Clouds</h2>
<p>RFC 1191 specifies <strong>Path MTU Discovery (PMTUD)</strong> to dynamically discover the lowest MTU along an arbitrary network path without causing packet fragmentation:</p>
<ol>
<li>All modern Linux and Windows kernels set the <strong>DF (Don't Fragment)</strong> bit in the IPv4 header of every outbound TCP packet.</li>
<li>When a 9,001-byte packet hits the Transit Gateway interface, the TGW detects that the packet exceeds its 8,500-byte MTU.</li>
<li>Because the DF bit is set, the TGW cannot fragment the packet. It drops the packet immediately.</li>
<li>Under RFC specifications, the Transit Gateway router generates an <strong>ICMP Type 3, Code 4</strong> packet (<em>"Destination Unreachable: Fragmentation Needed and DF Set"</em>). This packet specifies the next-hop MTU (8,500 bytes) and is sent back to the transmitting EC2 instance.</li>
<li>Upon receiving the ICMP message, the transmitting kernel caches the new path MTU, reduces its TCP MSS accordingly, and retransmits the payload cleanly in smaller chunks.</li>
</ol>
<h2>The Fatal Failure Mode: The Corporate ICMP Massacre</h2>
<p>In theory, PMTUD works seamlessly. In real-world enterprise infrastructure, it is systematically destroyed by well-intentioned security engineers:</p>
<ul>
<li><strong>Security Group Hardening:</strong> Security teams enforce baseline Terraform templates that restrict ingress rules strictly to application ports (e.g., <code>Allow TCP 443</code>, <code>Allow TCP 5432</code>). All ICMP traffic is blocked by default.</li>
<li><strong>NACL Blanket Denies:</strong> Subnet Network Access Control Lists (NACLs) explicitly deny ICMP to prevent internal ping-sweeps or ICMP tunneling attacks.</li>
<li><strong>Inspection VPC Firewalls:</strong> Traffic routed through centralized inspection VPCs (Palo Alto VM-Series, Fortinet FortiGate, or AWS Network Firewall) frequently drops non-TCP/UDP traffic unless specifically whitelisted.</li>
</ul>
<p>When the Transit Gateway emits the ICMP Type 3 Code 4 notification, the packet is instantly annihilated by the transmitting instance's security group or firewall. The transmitting kernel never receives the signal!</p>
<h3>The Anatomy of the Hang:</h3>
<ol>
<li>The TCP handshake succeeds because <code>SYN</code> and <code>SYN-ACK</code> packets are only 60 bytes.</li>
<li>The client issues a short HTTP GET request (300 bytes). It slips through unhindered.</li>
<li>The server prepares a 40KB response. It fills its first TCP segment to the maximum allowable MTU (9,001 bytes) and pushes it into the network socket.</li>
<li>The Transit Gateway drops the packet. The ICMP error is blocked.</li>
<li>The server's TCP stack waits for an ACK. When no ACK arrives, it retransmits the exact same 9,001-byte segment after an exponential backoff timer (RTO: 1s, 2s, 4s, 8s, 16s, 32s).</li>
<li>Every retransmitted packet is dropped at the Transit Gateway.</li>
<li>After 60 to 120 seconds, the client application times out and severs the TCP connection. SREs see healthy metrics, zero CPU spikes, and green load balancer targets—while production transactions fail 100% of the time on large payloads!</li>
</ol>
<h2>The Diagnostic Trap: Why Pings and Handshakes Lie to You</h2>
<p>When on-call engineers receive an alert that a cross-VPC microservice is failing, their first diagnostic step is running <code>ping</code> or <code>curl -I</code>:</p>
<pre><code>$ ping 10.100.4.52
PING 10.100.4.52 (10.100.4.52) 56(84) bytes of data.
64 bytes from https://www.google.com/url?q=http://10.100.4.52&source=gmail&ust=1793082957750000&sa=E: icmp_seq=1 ttl=254 time=1.24 ms
64 bytes from https://www.google.com/url?q=http://10.100.4.52&source=gmail&ust=1793082957750000&sa=E: icmp_seq=2 ttl=254 time=1.18 ms
$ curl -I https://www.google.com/url?q=https://internal-api.corp/healthz&source=gmail&ust=1793082957750000&sa=E
HTTP/1.1 200 OK
Content-Length: 2
</code></pre>
<p>Because synthetic pings use 84-byte packets and health checks return 200-byte responses, they easily fit beneath the 8,500-byte threshold. The engineer closes the incident ticket as a "transient network glitch." Meanwhile, actual production customer traffic carrying large payloads remains 100% blackholed!</p>
<h2>The MTU Hierarchy across AWS Network Primitives</h2>
<table>
<thead>
<tr>
<th>Network Boundary</th>
<th>Maximum Supported MTU</th>
<th>TCP MSS Equivalent</th>
<th>Failure Mode If Unhandled</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>Intra-VPC (EC2 to EC2)</strong></td>
<td>9,001 Bytes (Jumbo)</td>
<td>8,961 Bytes</td>
<td>None (Direct hardware delivery)</td>
</tr>
<tr>
<td><strong>AWS Transit Gateway (VPC Attachment)</strong></td>
<td>8,500 Bytes</td>
<td>8,460 Bytes</td>
<td><strong>Silent Drop if ICMP blocked</strong></td>
</tr>
<tr>
<td><strong>AWS Transit Gateway (Peering Attachment)</strong></td>
<td>8,500 Bytes</td>
<td>8,460 Bytes</td>
<td><strong>Cross-region MTU Black Hole</strong></td>
</tr>
<tr>
<td><strong>AWS Direct Connect (Private VIF)</strong></td>
<td>1,500 or 9,001 Bytes</td>
<td>1,460 or 8,961 Bytes</td>
<td>Depends on on-prem router MTU configuration</td>
</tr>
<tr>
<td><strong>AWS Site-to-Site VPN / IPsec</strong></td>
<td>1,420 – 1,500 Bytes</td>
<td>1,380 – 1,460 Bytes</td>
<td>Severe fragmentation & packet loss</td>
</tr>
</tbody>
</table>
<h2>The Calricon Production Playbook: Eliminating TGW MTU Black Holes</h2>
<p>To permanently resolve MTU black holes across enterprise cloud networks, implement this four-pillar defense-in-depth architecture:</p>
<h3>1. Enforce TCP MSS Clamping on Transit Firewalls and Gateways</h3>
<p>The most robust, fool-proof solution is enforcing <strong>TCP MSS Clamping</strong> at your network transit inspection layer (or on Linux routing instances). MSS clamping intercepts the initial TCP SYN packet during the 3-way handshake and rewrites the advertised Maximum Segment Size to fit the path's bottleneck:</p>
<pre><code># Linux iptables / nftables MSS Clamping Rule for Transit Routers
# Clamps TCP SYN packets traversing the Transit Gateway to 8460 bytes
iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN \
-j TCPMSS --set-mss 8460
# For cross-region or VPN tunnels traversing standard internet hops:
iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN \
-j TCPMSS --clamp-mss-to-pmtu
</code></pre>
<p>When an instance in a 9,001-byte VPC establishes a connection across the Transit Gateway, the router rewrites the SYN packet's MSS to 8,460. Both endpoints immediately agree to never emit packets larger than 8,500 bytes—completely bypassing the need for PMTUD and rendering the connection immune to ICMP blocking!</p>
<h3>2. Explicitly Whitelist ICMP Type 3, Code 4 in Security Groups</h3>
<p>Audit your central Terraform and AWS Security Group baselines. Security teams should never block PMTUD signaling. Add an explicit ingress rule to allow ICMP Fragmentation Needed packets across all VPC subnets:</p>
<pre><code># Terraform Security Group Rule: Permit Path MTU Discovery
resource "aws_security_group_rule" "allow_pmtud" {
type = "ingress"
from_port = 4 # Code 4: Fragmentation Needed
to_port = 3 # Type 3: Destination Unreachable
protocol = "icmp"
cidr_blocks = ["https://www.google.com/url?q=http://10.0.0.0/8&source=gmail&ust=1793082957750000&sa=E"] # Internal corporate RFC 1918 range
security_group_id = https://www.google.com/url?q=http://aws_security_group.app_sg.id&source=gmail&ust=1793082957750000&sa=E
description = "Allow Path MTU Discovery (PMTUD) signaling from Transit Gateway"
}
</code></pre>
<h3>3. Standardize Instance MTU via Cloud-Init and DHCP Options</h3>
<p>If your enterprise topology heavily relies on cross-VPC and hybrid interconnects, eliminate interface asymmetry at the operating system layer. Configure your base AMI (Golden Image) or cloud-init bootstrap scripts to set the default network interface MTU to <strong>8,500 bytes</strong> (or 1,500 bytes if predominantly routing over VPN):</p>
<pre><code># /etc/netplan/01-netcfg.yaml (Ubuntu / Debian)
network:
version: 2
ethernets:
eth0:
dhcp4: true
mtu: 8500 # Matches AWS Transit Gateway native limit
</code></pre>
<h3>4. Deploy Automated MTU Black Hole Probes</h3>
<p>Stop relying on standard ICMP pings in your synthetic monitoring suites. Deploy an automated probe that tests network paths with explicit Don't-Fragment (DF) jumbo packets:</p>
<pre><code># Ping test with Don't Fragment bit set and exact payload size
# 8472 bytes payload + 28 bytes IP/ICMP header = 8500 bytes total!
ping -M do -s 8472 -c 3 10.100.4.52
</code></pre>
<p>If the 8,472-byte probe fails while an 84-byte ping succeeds, trigger an automated P1 alert for an MTU black hole before production traffic is impacted.</p>
<h2>Conclusion</h2>
<p>AWS Transit Gateway simplifies cross-VPC network architecture, but treating it as a transparent, uniform conduit is a critical mistake. The 501-byte difference between intra-VPC 9,001-byte Jumbo Frames and the Transit Gateway's 8,500-byte ceiling—combined with ubiquitous ICMP blocking—creates invisible packet traps that bring down production databases and microservices without warning. By implementing TCP MSS clamping, permitting PMTUD ICMP signaling, standardizing OS interface MTUs, and validating paths with jumbo probes, platform teams can eliminate transit black holes and operate seamless, high-throughput cloud networks.</p>
Advertisement