Per-UE Uplink DMRS Channel Estimate Extraction from Aerial Data Lake

This notebook demonstrates how to extract and visualize pre-computed channel estimates (H-matrices) on a per-UE basis from the Aerial Data Lake.

Unlike the datalake_channel_estimation notebook which re-derives H-estimates from raw IQ using pyAerial/cuPHY, this notebook reads H-estimates that cuPHY already computed during live operation and stored in ClickHouse. No GPU or pyAerial is required.

The hest table stores one row per cell per slot, holding that cell’s UE groups concatenated into a single blob. The fapi table stores one row per UE per slot with the metadata needed to locate each UE inside its cell’s blob: hOffset, hSize, layerOffset, nSubcarriers, nDmrsEstimates, nBsAnts and nrOfLayers. Both tables carry CellId, so joins and per-cell UE counts must include it. Note that nUEs is the slot total across all cells, not a per-cell count.

H-estimate memory layout: [n_dmrs_estimates, n_subcarriers, n_bs_ants, n_layers_group] (row-major, complex64), with hOffset relative to the start of the cell’s blob.

Prerequisites: In cuphycontroller YAML config, enable Data Lake with at least channel estimate and pusch collections:

data_config:
  datalake_db_write_enable: 1
  datalake_data_types: [fh, pusch, hest]

The pusch type populates the fapi table (per-UE metadata), and hest populates the hest table (raw H-estimate blobs). Both are required.

Configuration, Imports, and Setup

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import clickhouse_connect

plt.rcParams['figure.figsize'] = [12, 4]

# --- Configuration ---
CLICKHOUSE_HOST = 'localhost'

client = clickhouse_connect.get_client(host=CLICKHOUSE_HOST)

Per-UE H-estimate Extraction

The get_ue_hest function extracts a single UE’s channel matrix from the concatenated H-estimate blob.

For FDM (each UE in its own group), the UE’s data is a contiguous block at hOffset with hSize elements. For MU-MIMO (multiple UEs sharing a group), layerOffset selects the UE’s layers within the group matrix.

def get_ue_hest(hest_blob, ue_row):
    """Extract per-UE H-estimate from the concatenated per-cell blob.

    Args:
        hest_blob: Flat float32 array from the hest row of this cell and slot
            (interleaved real/imag).
        ue_row: Row from fapi table with per-UE metadata.

    Returns:
        Complex numpy array of shape (n_dmrs, n_subcarriers, n_bs_ants, n_layers_ue).
    """
    h_offset = int(ue_row['hOffset'])
    h_size = int(ue_row['hSize'])
    n_sc = int(ue_row['nSubcarriers'])
    n_dmrs = int(ue_row['nDmrsEstimates'])
    n_bs_ants = int(ue_row['nBsAnts'])
    n_layers_ue = int(ue_row['nrOfLayers'])
    layer_offset = int(ue_row['layerOffset'])

    # h_size is in float2 (complex) elements; total group layers derived from dimensions
    n_layers_group = h_size // (n_dmrs * n_sc * n_bs_ants)

    # h_offset/h_size are in float2 units; multiply by 2 for float32 array indexing
    grp_ri = hest_blob[h_offset * 2 : (h_offset + h_size) * 2]
    grp = grp_ri.reshape(-1, 2).astype(np.float32).view(np.complex64).squeeze()
    grp = grp.reshape(n_dmrs, n_sc, n_bs_ants, n_layers_group)

    # layer_offset selects this UE's layers within the group (0 for FDM, >0 for MU-MIMO)
    return grp[:, :, :, layer_offset : layer_offset + n_layers_ue]

Single-UE Channel Estimates

Query a single-UE slot and plot the channel magnitude across subcarriers for each antenna and DMRS symbol.

# Pick a cell and slot carrying exactly one UE, then join fapi and hest server-side.
# Both the join and the UE count are keyed on CellId: hest holds one row per cell
# per slot, and fapi.nUEs is the slot total across all cells. The count covers
# every scheduled UE, so filtering happens in HAVING rather than WHERE; filtering
# first would let a two-UE slot with one failed transport block look single-UE.
row_1ue = client.query_df("""
    WITH one_ue AS (
        SELECT CellId, SFN, Slot, TsTaiNs FROM fapi
        GROUP BY CellId, SFN, Slot, TsTaiNs
        HAVING count() = 1 AND countIf(tbCrcFail = 0 AND hSize > 0) = 1
        ORDER BY TsTaiNs
        LIMIT 1
    )
    SELECT f.CellId, f.SFN, f.Slot, f.rnti, f.nrOfLayers, f.nBsAnts,
           f.rbStart, f.rbSize, f.nSubcarriers, f.nDmrsEstimates, f.dmrsSymbPos,
           f.layerOffset, f.ueGrpIdx, f.hOffset, f.hSize, f.sinr, f.rsrp,
           h.hestData
    FROM fapi AS f
    INNER JOIN hest AS h USING (CellId, SFN, Slot, TsTaiNs)
    WHERE (f.CellId, f.SFN, f.Slot, f.TsTaiNs) IN (SELECT CellId, SFN, Slot, TsTaiNs FROM one_ue)
""")

if row_1ue.empty:
    raise RuntimeError("No single-UE slot with paired hest blob found. Check that data has been collected.")

ue = row_1ue.iloc[0]
print(f"Cell: {ue['CellId']}  SFN/Slot: {ue['SFN']}/{ue['Slot']}  RNTI: {ue['rnti']}  "
      f"RBs: {ue['rbStart']}-{ue['rbStart']+ue['rbSize']-1}  "
      f"Layers: {ue['nrOfLayers']}  Group: {ue['ueGrpIdx']}")
print(f"hOffset: {ue['hOffset']}  hSize: {ue['hSize']}  "
      f"nSubcarriers: {ue['nSubcarriers']}  nDMRS: {ue['nDmrsEstimates']}  "
      f"SINR: {ue['sinr']:.1f} dB  RSRP: {ue['rsrp']:.1f} dB")

hest_blob = np.array(ue['hestData'], dtype=np.float32)

Cell: 51  SFN/Slot: 504/7  RNTI: 54873  RBs: 0-90  Layers: 1  Group: 0
hOffset: 0  hSize: 13104  nSubcarriers: 1092  nDMRS: 3  SINR: 27.0 dB  RSRP: 5.1 dB

h_ue = get_ue_hest(hest_blob, ue)
n_dmrs, _, n_ants, _ = h_ue.shape

fig, axes = plt.subplots(1, n_dmrs, figsize=(5 * n_dmrs, 4), squeeze=False)
fig.suptitle(f"Channel Magnitude  |  Cell {ue['CellId']}  |  RNTI {ue['rnti']}  |  SFN.Slot {ue['SFN']}.{ue['Slot']}")

for d in range(n_dmrs):
    ax = axes[0, d]
    for ant in range(n_ants):
        ax.plot(np.abs(h_ue[d, :, ant, 0]), label=f'Ant {ant}')
    ax.set_title(f'DMRS {d}')
    ax.set_xlabel('Subcarrier')
    ax.set_ylabel('|H|')
    ax.legend(fontsize=8)
    ax.grid(True, alpha=0.3)

plt.tight_layout()
plt.show()

Multi-UE Channel Estimates

When multiple UEs are scheduled on the same cell in the same slot, cuPHY groups them by frequency allocation (FDM) or spatial multiplexing (MU-MIMO). The per-UE metadata in fapi provides everything needed to extract each UE’s channel independently, regardless of the number of UEs.

• FDM: Each UE has a distinct hOffset/hSize (different frequency allocations), layerOffset = 0.

• MU-MIMO: UEs in the same group share the same hOffset/hSize but differ in layerOffset.

# Find the first cell and slot carrying two or more UEs and pull its fapi rows
# joined with that cell's hest blob. All returned rows share the same hestData.
# Every scheduled UE must have a channel estimate, since all of them get plotted.
fapi_multi = client.query_df("""
    WITH multi_ue AS (
        SELECT CellId, SFN, Slot, TsTaiNs FROM fapi
        GROUP BY CellId, SFN, Slot, TsTaiNs
        HAVING count() >= 2 AND countIf(hSize > 0) = count()
        ORDER BY TsTaiNs
        LIMIT 1
    )
    SELECT f.CellId, f.SFN, f.Slot, f.rnti, f.nrOfLayers, f.nBsAnts,
           f.rbStart, f.rbSize, f.nSubcarriers, f.nDmrsEstimates, f.dmrsSymbPos,
           f.layerOffset, f.ueGrpIdx, f.hOffset, f.hSize, f.sinr, f.rsrp,
           h.hestData
    FROM fapi AS f
    INNER JOIN hest AS h USING (CellId, SFN, Slot, TsTaiNs)
    WHERE (f.CellId, f.SFN, f.Slot, f.TsTaiNs) IN (SELECT CellId, SFN, Slot, TsTaiNs FROM multi_ue)
    ORDER BY f.rnti
""")

if fapi_multi.empty:
    print("No multi-UE slots with paired hest blob found.")
    hest_blob_m = None
else:
    ref = fapi_multi.iloc[0]
    print(f"Cell: {ref['CellId']}  SFN/Slot: {ref['SFN']}/{ref['Slot']}  --  {len(fapi_multi)} UEs")
    for i, ue_r in fapi_multi.iterrows():
        print(f"  UE {i}: RNTI {ue_r['rnti']}  Group {ue_r['ueGrpIdx']}  "
              f"RBs {ue_r['rbStart']}-{ue_r['rbStart']+ue_r['rbSize']-1}  "
              f"hOffset={ue_r['hOffset']}  hSize={ue_r['hSize']}  "
              f"SINR={ue_r['sinr']:.1f} dB")

    hest_blob_m = np.array(fapi_multi.iloc[0]['hestData'], dtype=np.float32)

Cell: 51  SFN/Slot: 598/9  --  2 UEs
  UE 0: RNTI 33163  Group 1  RBs 223-272  hOffset=32112  hSize=14400  SINR=30.4 dB
  UE 1: RNTI 54873  Group 0  RBs 0-222  hOffset=0  hSize=32112  SINR=30.2 dB

if len(fapi_multi) >= 2:
    n_ues = len(fapi_multi)
    fig, axes = plt.subplots(1, n_ues, figsize=(6 * n_ues, 4), squeeze=False)
    sfn_slot = f"{ref['SFN']}.{ref['Slot']}"
    fig.suptitle(f"Multi-UE Channel Magnitude  |  Cell {ref['CellId']}  |  SFN.Slot {sfn_slot}  |  {n_ues} UEs")

    for idx in range(n_ues):
        ue_r = fapi_multi.iloc[idx]
        h = get_ue_hest(hest_blob_m, ue_r)
        ax = axes[0, idx]
        for ant in range(h.shape[2]):
            ax.plot(np.abs(h[0, :, ant, 0]), label=f'Ant {ant}')
        ax.set_title(f'RNTI {ue_r["rnti"]}  |  Group {ue_r["ueGrpIdx"]}  |  '
                     f'RBs {ue_r["rbStart"]}-{ue_r["rbStart"]+ue_r["rbSize"]-1}')
        ax.set_xlabel('Subcarrier')
        ax.set_ylabel('|H|')
        ax.legend(fontsize=8)
        ax.grid(True, alpha=0.3)

    plt.tight_layout()
    plt.show()

Channel Heatmap

Channel magnitude heatmap (antenna vs subcarrier) for each DMRS symbol.

# Use the single-UE data from above
h_ue = get_ue_hest(hest_blob, ue)
n_dmrs, _, n_ants, _ = h_ue.shape

fig, axes = plt.subplots(1, n_dmrs, figsize=(5 * n_dmrs, 3), squeeze=False)
fig.suptitle(f"Channel Heatmap |H|  |  Cell {ue['CellId']}  |  RNTI {ue['rnti']}")

for d in range(n_dmrs):
    ax = axes[0, d]
    # Shape: (n_subcarriers, n_bs_ants) for layer 0
    img = ax.imshow(np.abs(h_ue[d, :, :, 0]).T, aspect='auto',
                    interpolation='nearest', cmap='viridis')
    ax.set_title(f'DMRS {d}')
    ax.set_xlabel('Subcarrier')
    ax.set_ylabel('Antenna')
    ax.set_yticks(range(n_ants))
    plt.colorbar(img, ax=ax, shrink=0.8)

plt.tight_layout()
plt.show()

Channel Phase

Channel phase across subcarriers per antenna.

h_ue = get_ue_hest(hest_blob, ue)

fig, ax = plt.subplots(1, 1, figsize=(12, 4))
fig.suptitle(f"Channel Phase (DMRS 0)  |  Cell {ue['CellId']}  |  RNTI {ue['rnti']}")

for ant in range(h_ue.shape[2]):
    ax.plot(np.angle(h_ue[0, :, ant, 0]), '.', markersize=2, label=f'Ant {ant}')

ax.set_xlabel('Subcarrier')
ax.set_ylabel('Phase (rad)')
ax.legend(fontsize=8)
ax.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()