72ns Gateway: A Versal ACAP HFT Accelerator

01. The Motivation: Chasing Zero Latency

In the world of high-frequency trading, the speed of light becomes a tangible constraint. At 299,792,458 meters per second, light travels approximately 30 centimeters in one nanosecond. This means that by the time a photon traverses a single meter of fiber optic cable (~5ns), market opportunities may have already evaporated. The question becomes: How fast can we process incoming market data before physics itself becomes the bottleneck?

Traditional software-based network stacks—even those optimized with kernel bypass (DPDK), zero-copy techniques (io_uring), and RDMA—still operate in the realm of microseconds. A well-tuned C++ application with SR-IOV and CPU pinning might achieve 1-2 microseconds of processing latency. But this is three orders of magnitude slower than what's physically possible.

The shift from software to hardware represents the ultimate frontier of optimization. By moving packet processing logic into reconfigurable fabric—specifically, Field-Programmable Gate Arrays (FPGAs)—we can achieve deterministic, sub-100ns latency. This is not about incremental improvement; this is a paradigm shift.

This project is a personal R&D initiative to master the art of heterogeneous computing—bridging the gap between software algorithms and hardware implementation. The goal is not just to build a faster system, but to understand the engineering trade-offs involved in nanosecond-latency design: clock domain crossings, metastability, pipeline hazards, and the cruel reality of propagation delay.

02. The Architecture: From Wire to Application

System Overview

The current prototype implements a receive (RX) path for processing inbound Ethernet frames. The architecture is modular, pipelined, and designed for scalability—while the current implementation uses 1G/2.5G Ethernet, the design can be extended to 40G/100G using the Versal MRMAC (Multirate Ethernet MAC) hard IP.

Component Breakdown

1. AXI 1G/2.5G Ethernet Subsystem

Xilinx IP core that handles the MAC layer (Media Access Control) and presents received frames via AXI4-Stream. The interface runs at 125 MHz with an 8-bit data bus (TDATA[7:0]), providing 1 Gbps of throughput (125 MHz × 8 bits = 1000 Mbps).

2. UDP Filter (Custom RTL)

A byte-by-byte state machine that processes incoming Ethernet frames and performs the following operations:

• Header Validation: Checks EtherType = 0x0800 (IPv4) at byte 12
• Protocol Check: Verifies IP Protocol = 0x11 (UDP) at byte 23
• Port Filtering: Matches UDP Destination Port (default: 1234) at bytes 36-37
• Header Stripping: Removes 42 bytes (14 Ethernet + 20 IP + 8 UDP) and forwards only the payload

3. Async FIFO (Clock Domain Crossing)

A metastability-safe FIFO implementing the Cummings 2-flop synchronizer technique. This module bridges two asynchronous clock domains:

• Write Domain: 250 MHz (Ethernet subsystem)
• Read Domain: 100 MHz (Application logic)

The design uses Gray code for pointer synchronization—a critical technique to avoid race conditions when crossing clock boundaries. Gray code ensures that only one bit changes between consecutive values, minimizing the risk of metastability during clock domain crossing.

4. Packet Parser (FSM Decoder)

A finite state machine that decodes application-layer protocols. The current implementation supports a custom framing protocol (0xAA55 sync pattern + length + payload + checksum), but is designed to be extended for industry-standard protocols like MoldUDP64 and NASDAQ TotalView-ITCH 5.0.

5. Testbench: Laptop as "Exchange Venue"

To validate the design, a Python/Scapy script running on a laptop injects raw Ethernet frames directly into the VCK190's SFP+ port. This simulates a realistic market data feed and allows for controlled testing of edge cases (fragmented packets, out-of-order delivery, checksum errors).

Hardware Specifications

02.5 The Complete Vision: From Market Feed to Strategy Execution

System Architecture Overview

While the current implementation focuses on the packet processing pipeline (Sections 02 and 03), the broader vision is to build a complete end-to-end HFT system that spans from simulated market data generation to strategy execution. This demonstrates not just hardware expertise, but a comprehensive understanding of the full trading stack.

Xilinx Versal VCK190 Evaluation Board: 1M+ logic cells, 400 AI engines, 4x 100GbE MRMAC

Component Details

📡 Stage 1: Market Data Simulation (Laptop)

A Python script running on a standard laptop generates synthetic market data packets conforming to the MoldUDP64 protocol. This simulates the NASDAQ ITCH 5.0 feed by:

• Generating realistic order book events: Add Order, Order Executed, Order Cancel, Trade messages
• Encapsulating messages in MoldUDP64 frames: Session ID, sequence numbers, message count headers
• Transmitting via UDP: Sends packets to the VCK190's 1G Ethernet port (destination port 1234)
• Rate control: Configurable message rate to test throughput and latency under load

⚡ Stage 2-4: Packet Processing Pipeline (FPGA - Implemented)

These modules are currently functional and documented in Sections 02-04:

• UDP Filter: Validates IP/UDP headers, strips 42-byte header, forwards payload (40ns latency)
• Async FIFO: Clock domain crossing with Gray code synchronizers (24ns latency)
• MoldUDP64 Parser: Unpacks session header, extracts individual ITCH messages
• ITCH Message Parser: Decodes message types (Add Order, Execute, Cancel, etc.)

📖 Stage 5: Book Builder (In Development)

The Order Book Reconstruction module maintains a real-time view of market depth by:

• Order tracking: Uses on-chip BRAM to store active orders (Order Reference Number → Price/Quantity)
• Price level aggregation: Maintains sorted bid/ask queues with total quantity at each price
• Update logic: Handles Add Order (insert), Cancel (remove), Execute (modify quantity)
• Latency target: <10ns per order book update (1-2 clock cycles @ 125 MHz)

📋 Stage 6: Order Manager (In Development)

Tracks positions, P&L, and risk metrics in real-time:

• Position tracking: Net shares held per symbol (long/short)
• P&L calculation: Mark-to-market using latest trade price
• Risk limits: Max position size, max notional exposure, loss limits
• Trade attribution: Links executions to strategy signals for performance analysis

🎯 Stage 7: Strategy Creator (In Development)

The alpha generation engine—where market microstructure patterns are exploited:

• Market making: Post quotes on both sides, capture spread (bid-ask arbitrage)
• Order flow imbalance: Detect aggressive buying/selling pressure from order book dynamics
• Momentum signals: Trade direction based on recent price moves and volume
• Arbitrage detection: Cross-exchange price discrepancies (requires multiple feeds)

Note: The strategy logic will be parameterizable via PCIe register writes, allowing dynamic reconfiguration without FPGA reprogramming.

Why This Matters

This end-to-end implementation demonstrates:

• Heterogeneous computing mastery: Bridging Python simulation, network protocols, and FPGA RTL
• Vertical integration: Understanding the full stack from packets to P&L
• Production-grade thinking: Not just a prototype—designed for real-world trading scenarios
• Latency obsession: Every clock cycle counts; every module is timed and optimized

03. The Protocol: NASDAQ TotalView-ITCH 5.0

Why NASDAQ ITCH?

NASDAQ TotalView-ITCH 5.0 is the de facto standard for low-latency market data feeds in U.S. equities. Broadcast from the NASDAQ data center in Carteret, New Jersey, this feed delivers:

• Order Book Updates: Add, delete, replace, and execute messages
• Trade Reports: Matched executions with timestamps
• NOII (Net Order Imbalance Indicator): Auction data
• System Events: Trading halts, circuit breakers

The protocol uses UDP multicast for one-to-many delivery, with MoldUDP64 as the session-layer framing protocol. This design eliminates TCP's handshake overhead and provides deterministic latency—critical for time-sensitive strategies like market making and statistical arbitrage.

MoldUDP64 Frame Structure

MoldUDP64 is NASDAQ's proprietary session-layer protocol. Each UDP datagram contains:

Each message in the block is prefixed with a 2-byte length field (big-endian) followed by the ITCH message itself. The parser must:

1. Verify the session ID matches the expected value
2. Check sequence numbers for gaps (indicating packet loss)
3. Iterate through the message block, parsing each ITCH message type

ITCH 5.0 Message Types (Sample)

Target Deployment: Carteret, NJ

NASDAQ's primary data center is located in Carteret, New Jersey, approximately 10 miles from New York City. Co-location providers (e.g., Equinix NY4, NY5) offer rack space within the same facility, providing sub-millisecond fiber latency to the exchange. For HFT strategies, proximity is critical:

• 500 meters of fiber: ~2.5 microseconds round-trip (light travels at ~2/3 speed of light in glass)
• 50 meters of fiber: ~250 nanoseconds round-trip
• 5 meters of fiber: ~25 nanoseconds round-trip

With 72ns of FPGA processing latency and ~25ns of fiber delay, the total system latency can approach 100ns—faster than a single memory access on a modern CPU.

04. The Engineering: Clock Domain Crossing & Metastability

Cummings 2-Flop Synchronizer: The Gold Standard for CDC

When data crosses from one clock domain to another, there's a risk of metastability—a state where a flip-flop's output oscillates unpredictably because the setup/hold timing was violated. This can propagate through the design, causing functional failures or even system crashes.

The 2-flop synchronizer (also called a "double synchronizer") is the industry-standard solution. By passing the signal through two flip-flops in the destination clock domain, we give it two full clock cycles to resolve to a stable state. The probability of metastability persisting through both stages is astronomically low (MTBF > 10^15 years for typical FPGA process nodes).

Why Gray Code?

Gray code is a binary numeral system where two successive values differ in only one bit. This is critical for CDC because if multiple bits changed simultaneously during a clock edge, the synchronizer might capture an invalid intermediate state. For example:

Binary:  0011 (3) → 0100 (4)  [2 bits change]
Gray:    0010 (3) → 0110 (4)  [1 bit changes]

If the synchronizer captures the binary transition at the wrong moment, it might see 0000, 0001, 0101, or 0111—none of which are valid. With Gray code, the only possible captured values are 0010 (old value) or 0110 (new value)—both correct.

FIFO Full/Empty Logic

The async FIFO uses separate logic for detecting full and empty conditions:

• Full Detection (Write Domain): Compares the write pointer (in Gray code) with the synchronized read pointer. If they match (after accounting for wraparound), the FIFO is full.
• Empty Detection (Read Domain): Compares the read pointer (in Gray code) with the synchronized write pointer. If they match, the FIFO is empty.

System Data Flow (Detailed)

05. The Library: Recommended Reading

This project builds upon decades of research in asynchronous design, network protocols, and quantitative finance. The following texts are essential reading for anyone serious about low-latency systems engineering:

06. The Milestone: First Packet & The 32-bit Trap

Phase 1 Complete: End-to-End Pipeline Architecture

The complete packet processing pipeline spans six modules, transforming raw Ethernet frames at 125 MHz into structured market tick data ready for the order book builder. This architecture demonstrates the full journey from physical layer (PHY) to application layer—every byte accounted for, every clock cycle optimized.

Pipeline Stage Breakdown

Protocol Format: Custom Framing

The current implementation uses a lightweight custom framing protocol optimized for simplicity and determinism:

[Byte 0-1]   Header:   0xAA55 (sync pattern for frame alignment)
[Byte 2]     Length:   N (payload size in bytes, excludes header/checksum)
[Byte 3..N+2] Payload:  N bytes of application data
[Byte N+3]   Checksum: XOR of all payload bytes (simple error detection)

Example: Market Tick (9-byte payload)
AA 55 09 | 00 00 27 10 | 00 00 03 E8 | 42 | A3
^Header  ^Len ^Price=$10000 ^Qty=1000   ^Buy ^XOR
            

This format provides:

• Fast synchronization: 2-byte header is unique and easy to detect with a simple FSM
• Variable-length support: Length field allows payloads from 1-255 bytes
• Basic error detection: XOR checksum catches single-bit errors (not CRC-grade, but sufficient for development)
• Minimal overhead: Only 4 bytes of framing per packet (header + length + checksum)

The Bring-up Challenge: When Reset Meets Reality

Hardware bring-up is where theory meets silicon—and where assumptions are ruthlessly tested. The initial attempt to establish the 1000BASE-X link failed spectacularly: the link stayed stubbornly down despite correct clock frequencies, proper transceiver configuration, and verified cable connections.

The root cause was a classic Reset Polarity mismatch. The Xilinx 1G/2.5G Ethernet Subsystem IP expects pma_reset (Physical Medium Attachment reset) to be driven by peripheral_reset from the Processor System Reset IP. However, the initial design erroneously connected it to interconnect_aresetn, which has inverted polarity. The transceiver was perpetually held in reset, preventing link negotiation.

Adding to the chaos: a wiring conflict where tvalid (active-high data valid signal) and aresetn (active-low reset) were shorted together on the PCB. This created a paradoxical state where asserting reset would inadvertently signal "data valid," corrupting the AXI-Stream handshake. Isolating and rewiring these signals restored proper reset behavior.

MAC Configuration: The Promiscuous Mode Hack

Standard Ethernet communication requires an ARP handshake to map IP addresses to MAC addresses before packets can be exchanged. However, in a pure FPGA-to-PC test environment (no OS network stack on the FPGA side), implementing a full ARP responder would be overkill for initial validation.

The solution: Enable Promiscuous Mode on the Xilinx Ethernet MAC. In this mode, the FPGA accepts all incoming packets regardless of destination MAC address—no ARP required. This allowed the PC to blast UDP packets directly to the FPGA's physical port, bypassing Layer 2 address resolution entirely. Think of it as the hardware equivalent of tcpdump mode: listen to everything, filter in software (or hardware, in this case).

The "32-bit Trap": When Bytes Aren't Bytes

Here's where things got interesting. The initial assumption was that the AXI-Stream interface from the Ethernet MAC would deliver data as a simple byte stream: 0x55, 0xBB, 0xCC, 0xDD, etc. Architecturally clean, easy to parse.

Reality check: The interface runs in 32-bit Little Endian mode at 125 MHz. A single AXI-Stream transaction presents four bytes simultaneously on tdata[31:0], with byte ordering reversed. For example, the byte sequence 0x55 0xBB 0xCC 0xDD appears on the bus as:

tdata[31:0] = 0xDDCCBB55  // Little Endian: LSB (0x55) in tdata[7:0]
            

This is not a bug—it's the standard AXI-Stream convention for maximizing throughput. At 1 Gbps (125 MHz × 8 bits), the MAC naturally emits data in 32-bit chunks to match the fabric clock rate. However, the downstream UDP parser state machine expects sequential 8-bit bytes for header field extraction (IP addresses, port numbers, checksums).

The disconnect is profound: You can't simply wire tdata[7:0] to the parser's byte input. You'd only see the first byte of every 4-byte word, skipping 75% of the packet. The parser would interpret every fourth byte as consecutive, producing gibberish.

The Solution: An "Internal Serializer" in Verilog

The fix required embedding a word-to-byte unpacking mechanism directly into the packet parser module. This "internal serializer" buffers incoming 32-bit words and processes them one byte at a time, effectively converting the parallel 32-bit AXI-Stream interface into a sequential byte stream for the parser state machine.

// Internal Serializer: 32-bit Word → 8-bit Byte Stream
reg [31:0] data_buffer;      // Holds the 4 bytes we just received
reg [2:0]  byte_index;       // Tracks which byte (0-3) we are processing
reg        buffer_valid;     // Do we have data in the buffer?
wire [7:0] current_byte;     // The specific byte we are looking at

// Extract the current byte (LSB of buffer)
assign current_byte = data_buffer[7:0];

always @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
        s_axis_tready <= 0;
        buffer_valid  <= 0;
        byte_index    <= 0;
        data_buffer   <= 0;
    end 
    else begin
        // ---------------------------------------------------------
        // SERIALIZER: Ingest 32-bit Words → Output 8-bit Bytes
        // ---------------------------------------------------------
        
        // If buffer is empty, try to grab new data from UDP Filter
        if (!buffer_valid) begin
            s_axis_tready <= 1; // "Give me data"
            if (s_axis_tvalid && s_axis_tready) begin
                data_buffer   <= s_axis_tdata; // Capture 32 bits
                buffer_valid  <= 1;
                byte_index    <= 0; // Start at Byte 0
                s_axis_tready <= 0; // Hold off on next word
            end
        end 
        
        // If buffer has data, process 1 byte per clock
        else if (buffer_valid && m_axis_tready) begin
            // Process current_byte with parser state machine...
            // (Parser logic operates on current_byte)
            
            // Shift buffer to next byte
            if (byte_index == 3) begin
                buffer_valid <= 0; // Done with this 32-bit word
            end else begin
                data_buffer <= data_buffer >> 8; // Shift right by 8 bits
                byte_index  <= byte_index + 1;
            end
        end
    end
end
            

This implementation uses a right-shift register approach: each clock cycle, the buffer shifts right by 8 bits, exposing the next byte at data_buffer[7:0]. The byte_index counter tracks progress through the 4-byte word, and buffer_valid signals when the serializer needs to fetch the next 32-bit chunk from the MAC.

Critically, the serializer respects AXI-Stream backpressure: it only advances when m_axis_tready is asserted (downstream is ready to accept data). This prevents data loss during clock domain crossings, FIFO congestion, or when the parser state machine is busy processing headers. The handshake mechanism ensures lossless, deterministic packet processing at line rate.

Validation: Three Layers of Proof

Claiming "it works" requires evidence. Three tools provided irrefutable validation:

1. Wireshark: Packet Egress Confirmation

On the PC side, Wireshark captured outgoing UDP packets on the Ethernet interface connected to the VCK190. The capture confirmed:

• Packets were transmitted with correct source/destination IP addresses and UDP ports
• Payload length matched expectations (e.g., 1024 bytes for test frames)
• Frame Check Sequence (FCS) validated—no corruption in transit

Wireshark Capture: UDP packets successfully transmitted from PC to FPGA over 1000BASE-X SFP+ link

2. IBERT: Physical Layer Verification

Xilinx's Integrated Bit Error Rate Tester (IBERT) confirmed the 1.25 Gbps physical link (1000BASE-X uses 8b/10b encoding: 1 Gbps data rate + 25% overhead = 1.25 Gbps line rate). Key metrics:

• RX Eye Diagram: Clean eye opening with >200 mV margin—no inter-symbol interference (ISI)
• Bit Error Rate: Zero errors over 10^12 bits—well below the 10^-15 threshold for reliable operation
• Link Status: PCS/PMA lock achieved, no loss-of-signal events

Vivado IBERT: 1.25 Gbps physical link validated with clean eye diagram and zero bit errors

3. Vivado ILA: The "Money Shot"

The Integrated Logic Analyzer (ILA)—Xilinx's on-chip oscilloscope—provided the smoking gun. By instrumenting the AXI-Stream bus inside the FPGA fabric, the ILA captured live packet data at the parser's input. The critical waveform showed:

tdata[31:0] = 0xDDCCBB55
tvalid      = 1
tready      = 1
            

Translation: The FPGA was actively receiving 32-bit words from the Ethernet MAC with valid data present. This confirmed end-to-end signal integrity from the PC's NIC → SFP+ fiber → VCK190 transceiver → AXI-Stream fabric. The packet wasn't just "arriving"—it was parsable and actionable.

This single ILA screenshot represents hundreds of hours of debugging, schematic review, and constraint tweaking. It's the hardware equivalent of a successful printf("Hello, World!")—except at 125 MHz and with <72ns latency.

What This Milestone Unlocks

With the physical link validated and the parser receiving clean byte streams, the foundation is set for:

• Protocol Decoding: Implement MoldUDP64 deframing to extract individual ITCH 5.0 messages
• Message Parsing: Decode NASDAQ ITCH 5.0 message types (Add Order, Execute, Cancel, Trade)
• Order Book Construction: Build a real-time Limit Order Book (LOB) in BRAM with price-time priority
• Strategy Logic: Implement trading algorithms (e.g., market-making, arbitrage) with sub-microsecond reaction times

The hardest part of any hardware project isn't the algorithm—it's getting that first LED to blink (or in this case, that first packet to parse). With the physical layer proven, the real fun begins. 🚀