calricon.
All journal notes

General

The AWS Network Firewall Asymmetric Routing Trap: Why Cross-AZ Transit Gateway Attachments Silently Drop TCP Handshakes

Oct 24, 2026/10 min read

calricon / field note

When enterprises deploy centralized AWS Network Firewalls behind a Transit Gateway, cross-AZ traffic triggers a maddening mystery: same-AZ connections succeed, while two-thirds of cross-AZ TCP handshakes silently time out. Default TGW AZ affinity routes SYN and SYN-ACK packets to different firewall endpoints, causing stateful Suricata engines to drop return traffic. Here is how to architect symmetric inspection hubs.

Oct 24, 2026/General
240
Advertisement

The AWS Network Firewall Asymmetric Routing Trap: Why Cross-AZ Transit Gateway Attachments Silently Drop TCP Handshakes

In enterprise cloud networking, the centralized Hub-and-Spoke inspection model is the gold standard for enforcing zero-trust perimeter and east-west security. Instead of deploying expensive, isolated security appliances inside dozens of workload VPCs, cloud architects place an AWS Transit Gateway (TGW) at the core and route all inter-VPC, egress, and hybrid Direct Connect traffic through a dedicated Inspection VPC housing AWS Network Firewall endpoints across three Availability Zones (AZs). On paper, the architecture is clean, scalable, and compliant. In production, however, a single omitted configuration flag triggers one of the most maddening network outages in cloud engineering: roughly one-third of internal microservice calls succeed instantaneously, while the remaining two-thirds hang indefinitely and fail with TCP connection timeouts. The culprit is the AWS Network Firewall Asymmetric Routing Trap: by default, AWS Transit Gateway enforces Availability Zone affinity, routing forward TCP SYN packets through the firewall endpoint in the source AZ while routing return SYN-ACK packets through a completely different firewall endpoint in the destination AZ. Because stateful firewall engines do not synchronize connection state tables across Availability Zones, the second firewall treats the unsolicited return packet as an invalid mid-stream anomaly and silently drops it into a black hole.

When you deploy a 3-AZ AWS Network Firewall behind a Transit Gateway with default settings, exactly 33% of your traffic works and 66% silently dies. Forward SYN packets pass through AZ-A, return SYN-ACK packets get routed through AZ-B, and the stateful firewall in AZ-B drops the return traffic because it never saw the handshake begin.

The Mechanics of Centralized Inspection: How Transit Gateway AZ Affinity Works

To understand why two-thirds of cross-VPC connections fail while one-third succeeds, cloud architects must trace how AWS Transit Gateway selects network interfaces across Availability Zones:

  • Default AZ Affinity (Locality Preservation): When a Transit Gateway forwards traffic from a source VPC attachment to a destination VPC attachment (such as an Inspection VPC), its default routing behavior prioritizes keeping traffic within the exact same Availability Zone where the packet originated. AWS designs this default to minimize cross-AZ latency and save customers from cross-AZ data transfer charges.
  • Stateful Flow Tracking in AWS Network Firewall: Under the hood, AWS Network Firewall runs a managed, high-performance Suricata intrusion detection and prevention engine paired with a Gateway Load Balancer (GWLB) endpoint architecture. When a stateful rule group is active—especially under Strict TCP Ordering (aws:drop_strict)—the firewall maintains an in-memory flow state table tracking the 5-tuple (Source IP, Destination IP, Source Port, Destination Port, Protocol) and TCP sequence numbers of every active connection.
  • Zero Cross-AZ State Synchronization: Crucially, each AWS Network Firewall endpoint deployed in an Availability Zone is an independent, isolated inspection node. The firewall endpoint in us-east-1a shares zero memory, zero flow state, and zero TCP session tables with the firewall endpoint in us-east-1b or us-east-1c.

The Anatomy of the 66% Packet Drop: Tracing a Cross-AZ TCP Handshake

Consider a standard three-AZ enterprise environment (us-east-1a, us-east-1b, us-east-1c) where an order-processing service in Spoke-VPC-A attempts to open a PostgreSQL database connection or HTTPS call to an inventory service in Spoke-VPC-B.

Scenario 1: Same-AZ Traffic (The 33% That Works)

If the client pod happens to be scheduled on a node in us-east-1a and the target database instance is also located in us-east-1a, both the outbound TCP SYN and the returning TCP SYN-ACK originate in us-east-1a. Both packets traverse the exact same firewall endpoint in us-east-1a. The state table matches, the three-way handshake completes in 2 milliseconds, and engineers assume the network is healthy.

Scenario 2: Cross-AZ Traffic (The 66% That Silently Dies)

Now watch what happens when the client pod is in us-east-1a, but the target service sits in us-east-1b:

  1. The Outbound SYN: The client in Spoke-VPC-A (us-east-1a) sends a TCP SYN packet. It enters the Transit Gateway in us-east-1a. Honoring default AZ affinity, the TGW routes the packet into the Inspection VPC via the TGW attachment ENI in us-east-1a.
  2. Stateful Registration in AZ-A: The AWS Network Firewall endpoint in us-east-1a inspects the SYN packet, records a new half-open connection in its local us-east-1a flow table, and sends the packet back to the TGW, which delivers it across zones to the target server in Spoke-VPC-B (us-east-1b).
  3. The Asymmetric Return (SYN-ACK): The target server in Spoke-VPC-B (us-east-1b) responds with a TCP SYN-ACK. The packet enters the Transit Gateway in us-east-1b. Because default AZ affinity is active on the Inspection VPC attachment, the TGW refuses to cross back to us-east-1a; instead, it forwards the SYN-ACK to the Inspection VPC attachment in us-east-1b.
  4. The Silent Stateful Drop: The AWS Network Firewall endpoint in us-east-1b receives a raw TCP SYN-ACK packet. It checks its local flow table and finds no record of a preceding SYN packet for that 5-tuple. Treating the packet as an out-of-state TCP injection or spoofing attempt, the firewall drops the packet immediately. The client retransmits its SYN until the connection times out.

Default AZ Affinity vs. Stateless Bypass vs. Transit Gateway Appliance Mode

Evaluation Dimension Default TGW Attachment (Appliance Mode Off) Stateless Firewall Rule Workaround Calricon Appliance Mode Architecture
Cross-AZ Stateful TCP Handshakes Fails (Drops 66% of flows in a 3-AZ topology) Succeeds (At the cost of blinding security inspection) 100% Symmetric (Pinned 5-tuple AZ stickiness)
Deep Packet Inspection (TLS SNI / Suricata IPS) Broken on cross-AZ flows (Stream reassembly fails) Impossible (Stateless engine inspects single packets only) Full Bidirectional Stream Reassembly
Availability Zone Failover Resilience Unpredictable state resets High (No state to lose) Automatic Re-Hashing to surviving AZs
Cross-AZ Traffic Traversal Overhead Zero within Inspection VPC (Stays in local AZ) Zero within Inspection VPC Moderate (Flows hashed evenly across inspection AZs)

The Calricon Production Playbook: Architecting Symmetric Inspection Hubs

Eliminating asymmetric drops without sacrificing deep stateful inspection requires reconfiguring Transit Gateway attachment physics, isolating subnet route tables, and instrumenting strict drop telemetry.

Step 1: Enable Appliance Mode on the Inspection VPC Transit Gateway Attachment

To guarantee that both directions of a network flow traverse the exact same AWS Network Firewall endpoint, you must enable Appliance Mode (appliance_mode_support = "enable") on the Transit Gateway VPC attachment belonging to the Inspection VPC.

When Appliance Mode is enabled, the Transit Gateway abandons local AZ affinity for traffic destined to the Inspection VPC. Instead, it calculates a deterministic hash of the IP 5-tuple (forcing forward and reverse packets to yield the exact same hash value) and pins the entire bidirectional conversation to a single Availability Zone inside the Inspection VPC for the lifetime of the flow.

In Terraform, enforce this explicitly on the inspection attachment:

resource "aws_ec2_transit_gateway_vpc_attachment" "inspection_vpc" {
  transit_gateway_id = aws_ec2_transit_gateway.core_hub.id
  vpc_id             = aws_vpc.inspection_hub.id
  subnet_ids         = aws_subnet.tgw_attachment[*].id

  # CRITICAL: Prevents cross-AZ asymmetric drops on stateful firewalls
  appliance_mode_support = "enable"

  transit_gateway_default_route_table_association = false
  transit_gateway_default_route_table_propagation = false

  tags = {
    Name = "tgw-att-inspection-hub-symmetric"
  }
}

Or apply it immediately to a live production attachment via AWS CLI:

aws ec2 modify-transit-gateway-vpc-attachment \
  --transit-gateway-attachment-id tgw-attach-0a1b2c3d4e5f6g7h8 \
  --options ApplianceModeSupport=enable

Step 2: Enforce Strict Two-Tier Subnet Segregation Per Availability Zone

Enabling Appliance Mode on the Transit Gateway is useless if your internal Inspection VPC routing tables cross Availability Zones. Every Availability Zone in your Inspection VPC must have two dedicated, strictly isolated subnets:

  1. Transit Gateway Attachment Subnet (per AZ): Hosts only the TGW ENIs. Its dedicated route table must route 0.0.0.0/0 exclusively to the AWS Network Firewall VPC Endpoint (vpce-...) located in that exact same Availability Zone.
  2. Firewall Endpoint Subnet (per AZ): Hosts only the AWS Network Firewall Gateway Load Balancer endpoint. Its dedicated route table must route 0.0.0.0/0 back to the Transit Gateway (tgw-...).

Never share route tables across Availability Zones in an Inspection VPC; cross-wiring a TGW subnet in us-east-1a to a firewall endpoint in us-east-1b reintroduces asymmetry inside the VPC itself.

Step 3: Eliminate Asymmetric Hairpinning on Ingress/Egress Paths

A subtle secondary failure occurs when architects place Internet Gateways (IGW) or NAT Gateways inside the same Inspection VPC that inspects east-west TGW traffic. Appliance Mode stickiness only applies to traffic entering and leaving the Inspection VPC through the same Transit Gateway. If inbound traffic enters via an Internet Gateway in the Inspection VPC and forwards to a Spoke VPC via TGW, the return traffic may hash to a different AZ. Always separate your East-West Inspection VPC (or dedicated TGW attachment subnets) from North-South Egress VPCs, ensuring every inspected flow enters and exits the inspection attachment symmetrically via the Transit Gateway.

Step 4: Enable Strict TCP State Enforcement with Explicit Drop Alerting

By default, if Suricata encounters an asymmetric stream and drops or resets it without explicit logging rules, network engineers are left staring at silent packet captures. Configure your AWS Network Firewall Stateful Engine to use Strict Order evaluation with explicit drop logging, and deploy a CloudWatch Logs Insights query to catch asymmetric misconfigurations immediately:

{
  "StatefulEngineOptions": {
    "RuleOrder": "STRICT_ORDER",
    "StreamExceptionPolicy": "DROP"
  }
}

Combine this with a CloudWatch Logs Insights query monitoring dropped TCP SYN-ACK packets across Availability Zones:

fields @timestamp, event.src_ip, event.dest_ip, event.tcp.tcphdr, event.alert.action
| filter event.alert.action = "blocked" and event.tcp.syn = 1 and event.tcp.ack = 1
| stats count(*) as asymmetric_syn_ack_drops by availability_zone, bin(1m)
| sort asymmetric_syn_ack_drops desc

Stop Disabling Stateful Inspection to Fix Routing Bugs

When confronted with mysterious cross-AZ connection timeouts after deploying AWS Network Firewall, desperate engineering teams frequently commit a fatal security blunder: they switch their firewall rules from stateful to stateless, or set StreamExceptionPolicy to CONTINUE (allowing mid-stream unestablished TCP packets to bypass inspection entirely). That does not fix your network architecture; it lobotomizes your firewall and turns an enterprise Intrusion Prevention System into a glorified, overpriced Network ACL.

Audit your Transit Gateway inspection attachments today. Enable ApplianceModeSupport on every VPC attachment hosting stateful middleboxes, verify that each Availability Zone's TGW subnet routes strictly to its local AZ firewall endpoint, and monitor dropped SYN-ACK metrics in CloudWatch. True zero-trust cloud networking demands both uncompromising stateful inspection and rock-solid routing symmetry.

Advertisement
Talk to Calricon