Seeing Through Walls: WiFi Human-Activity Recognition on the Versal AI Engine

My ECE Master's at Mississippi State University was in software-defined-radio (SDR) based sensing. One project asked a deceptively simple question: can an ordinary WiFi signal tell what a person is doing in a room?

The answer is yes — and it comes from Channel-State-Information (CSI). Every OFDM WiFi receiver estimates how the channel distorted each subcarrier. When a body moves, the multipath reflections change, and those changes carry a signature of the motion. I built a WiFi-CSI human-activity recognition (HAR) pipeline, benchmarked it against a radar sensing baseline for accuracy, and published the results across several papers (see Publications).

The visual heart of that work is the spectrogram: take the Short-Time Fourier Transform of the CSI time series and each activity paints a distinct time–frequency fingerprint. These are real CSI spectrograms from the dataset — one per activity class:

When I presented that thesis to a former manager, they gently suggested that the whole pipeline could, one day, run in an FPGA — the packet parsing in fabric, the heavy DSP and inference on an accelerator. It was said kindly, almost in passing. But it lodged in my head as a personal challenge I promised myself I'd take on someday.

The napkin sketch of that idea looked like this:

The idea sat on the shelf while life moved on. What finally unblocked it was hardware and time: a VCK190 on the bench and a run of weekends and evenings. Modern tooling — and, honestly, LLMs for grinding through the deep Versal tool-flow issues — made a solo build of this scope tractable.

The pipeline now runs on real hardware. A Raspberry Pi running the nexmon_csi firmware patch turns its WiFi chip into a CSI sensor and packetizes CSI as UDP over Ethernet. That stream enters the VCK190 over an SFP link; the PL parses the UDP/CSI packet and streams the I/Q samples straight into the AI Engine, which computes the motion, breathing and phase-variance features. Only the compact results cross into Linux on the Arm cores, where an on-board web dashboard renders what's happening in the room.

Why this device is the right fit:

• PL streams and parses the Ethernet/UDP packet at line rate.
• The AI Engine is a vector-VLIW array built for FFTs and MAC-heavy DSP — exactly the CSI workload.
• The NoC + DDR and Arm PS handle results and visualization.
• It's a single-chip edge sensor: no PC in the loop.

The full path from WiFi chip to browser. Raw CSI (hundreds of KB/s) never touches the CPU in the target design — only the tiny feature/label vector is DMA'd to Linux. Click any diagram to open it fullscreen and zoom / pan.

No camera, no wearable, often through a wall. The trick is that your body is part of the radio channel. Here is the intuition, in four steps.

Why STFT. Activities are non-stationary, so we take the Short-Time Fourier Transform of the CSI time series — window, DFT, magnitude — producing the time–frequency spectrograms you saw in Act 1. (This directly builds on my paper Effect of the Short-Time Fourier Transform on the Classification of Complex-Valued Mobile Signals.)

In the target architecture the PL owns the packet. RX frames enter as an AXI4-Stream; a custom csi_udp_parser (Vitis HLS) filters the CSI UDP port, strips the Ethernet/IP/UDP headers, extracts the I/Q payload, and streams samples through a csi_mux straight into the AI Engine. This is the real Vivado block design that was built and run on the board:

A simplified view of the same PL datapath, for reference:

Under the hood the parser is a one-byte-per-cycle state machine (Vitis HLS, II=1): it walks each Ethernet frame byte by byte, checks a handful of fixed offsets to accept or drop the frame, and reassembles the little-endian int16 I/Q payload into 32-bit words. The whole wire contract is just these offsets:

The complex pair. From byte 60 the parser takes four wire bytes at a time — two for the real part, two for the imaginary — sign-extends each int16, and packs them into one 32-bit AXI4-Stream beat. That packed word is the complex CSI sample H[k] = I + jQ for a single subcarrier:

// CSI payload from byte 60: int16 LE real, int16 LE imag.
ap_int<16> re; re(7,0) = b0; re(15,8) = b1;
ap_int<16> im; im(7,0) = b2; im(15,8) = b;
axis_csi o;
o.data(15,0)  = (ap_uint<16>)re;   // real -> low half
o.data(31,16) = (ap_uint<16>)im;   // imag -> high half
o.last = beat.last;                 // TLAST marks end-of-CSI for this frame
csi_out.write(o);                   // one 32-bit beat per subcarrier
n_sub++;

So the parser emits two synchronized AXI4-Streams per WiFi frame: csi_out, the run of complex subcarriers, and meta_out, a single 64-bit record carrying everything the downstream needs to normalize and label the window:

The whole parser costs about 1.3% of the PL LUTs, zero DSPs and a few BRAMs — line-rate protocol offload is exactly what the fabric is cheapest at, which is why it lives here and not on the Arm.

The parser produces per-packet complex vectors, but the AI Engine branches want real-valued time windows — a stretch of many packets for one subcarrier. Bridging the two is a short conditioning chain, and deciding where each step runs — PL, AI Engine, or Arm — is the heart of the hardware/software co-design.

Magnitude feeds the motion and breathing branches; phase feeds the phase-var branch. Phase first gets a per-packet linear de-trend across subcarriers, which cancels the carrier-frequency offset (the intercept) and the sampling/packet-delay offset (the slope) that otherwise swamp the real motion:

Where each stage of the pipeline lands is a deliberate partitioning decision:

Rule of thumb across the whole design: streaming & bit-twiddling → PL; FFT / GEMM / filters → AI Engine; control & UX → the Arm.

Those three conditioned windows are exactly what the graph waits on — each lands on its own PLIO stream input, and from here the whole pipeline stays on the AI Engine array. It runs a three-branch feature graph, one branch per window: the two amplitude windows drive motion and breathing, while the sanitized-phase window drives phase-var.

Each branch maps directly to a piece of the physics above, and every kernel is a small streaming function connected window-to-window over PLIO:

Why three? Human activity shows up in the CSI at three different mechanisms and time-scales at once, so each branch is tuned to one of them and they run in parallel on separate AI Engine tiles:

The graph wiring is pure ADF — kernels connected stream-to-stream on the array:

// motion branch: FIR band-pass -> stats (kernel-to-kernel)
connect<>(in.out[0],    kfir.in[0]);
connect<>(kfir.out[0],  kstats.in[0]);
connect<>(kstats.out[0], out.in[0]);
// breathing branch: windowed-DFT magnitude
connect<>(brt_in.out[0], kdft.in[0]);
connect<>(kdft.out[0],   brt_out.in[0]);
// phase-variance branch: stats -> variance is the phase-var feature
connect<>(phase_in.out[0], kphase.in[0]);
connect<>(kphase.out[0],   phase_out.in[0]);

static inline void stats_core(const float *x, float *out) {
    float s = 0.0f;
    for (int i = 0; i < L; i++) s += x[i];
    float mean = s / (float)L;
    float sd = 0.0f, s2 = 0.0f;
    for (int i = 0; i < L; i++) {
        float d = x[i] - mean;
        sd += d * d;
        s2 += x[i] * x[i];
    }
    out[0] = mean;          // DC / bias
    out[1] = sd / (float)L; // variance  == motion-band / phase-var energy
    out[2] = s2 / (float)L; // power
}

Per window the three branches emit 39 floats — mot[3], brt[33], phs[3]. Together with the RSSI from the parser metadata, those collapse into the compact 8-feature vector the classifier actually consumes:

The classifier is deliberately tiny — a pretrained 8→64→128 MLP (~9k parameters, GELU + batch-norm, L2-normalized embedding) plus a linear presence head. All the heavy lifting is the DSP feature stage above: the vector-VLIW AI Engine chews through the FIR taps and DFT butterflies far faster than the scalar A72 could — which is the whole point of the offload.

The last hop is deliberately thin. Two s2mm data movers — one for the AI Engine features, one for the parser metadata — are PL AXI-MM masters that push their results across the Versal NoC into a small reserved region of DDR4. The Arm A72 running PetaLinux reads that region and serves the dashboard. The multi-hundred-KB/s raw CSI stream never crosses into Linux — only a few hundred bytes of features and metadata per window do.

Because s2mm and s2mm_meta are driverless PL masters with no IOMMU, they write to a 1 MB no-map carve-out that Linux does not own: the packed metadata record sits at the base and the feature results 64 KB in, so the two never share a page. Everything is memory-mapped at fixed addresses that the on-target control tool (sw/csi_ctl.c — mac-init, start, mux, meta) and the reader agree on:

The csi_mux is what makes both worlds testable: flip it one way and the AI Engine is fed by the live Ethernet parser; flip it the other and it's fed by mm2s replaying a known CSI window from DDR — the bit-exact golden test that anchors every silicon bring-up below.

Talk is cheap — here is the datapath caught in the act on a real VCK190 (board chanterelle10). Four ILA taps were inserted on the live pipeline via the Versal debug automation: ETH_RX (nexmon UDP bytes), CSI_PARSED (parser output), AIE_IN (mux → AI Engine) and AIE_OUT (AIE features → DMA).

The end-to-end numerical check, on hardware:

The VCK190 serves its own dashboard (pure-Python, standard-library only, no CDN — it runs on a bare PetaLinux rootfs). It consumes the AI Engine feature groups and derives presence, activity, breathing rate and fall alerts — and renders a live CSI spectrogram by stacking the streamed brt[33] windowed-DFT magnitude vectors over time (the same STFT view from Act 1, now updating in real time). Below is that dashboard's logic, replaying a representative feature stream right in your browser — empty room → someone walks in → sits and breathes → a fall → stillness:

The widget above is a representative feature-stream replay running the same metrics and thresholds as the on-board dashboard — it is not a live capture. On the board, these tiles update from live AI-Engine features.

The gap between "works in simulation" and "works on silicon" was where the real engineering happened. A few that cost days: