Module ple 

PLE — hashed n-gram injection into the hyper-connection highway.

Qwen3.8-Flash-Next runs this on ONE layer (ple_layer_ids is 1-indexed, so [2] means model layer 1). From the reference’s own docstring:

PLE projects each token’s concatenated n-gram embedding to a shared value and one key per residual stream. The normalized stream activations gate those values, then a dilated depthwise convolution adds local lexical context.

That is cross-attention into an n-gram table, not an additive embedding — the reading that cost real time on LongCat. The forward, from Qwen4ExpTextPLELayer.forward:

embeddings   = ple_embedding(input_ids)                  # [T, 2560]
key_normed   = norm_key(key_proj(emb)) -> [T, hc, H]
value        = value_proj(emb)                           # [T, 2560]
query_normed = norm_query(hidden)      -> [T, hc, H]     # hidden is [T, 10240]
gate  = (key_normed * query_normed).sum(-1) / sqrt(H)    # [T, hc]
gate  = sign(gate) * sqrt(max(|gate|, 1e-6))             # SIGNED SQRT
gated = sigmoid(gate) * value                            # [T, hc, H]
out   = gated.flatten() + silu(conv1d(norm_conv(gated.flatten())))

and the decoder layer adds it to the highway BEFORE that layer’s attention hyper-connection: hidden_states = hidden_states + ple(...).

Three things here bite, all quietly:

1. The signed square root on the gate. Nobody would guess it; omit it and the gate distribution is wrong but perfectly finite.
2. conv1d is depthwise AND dilated — groups = 10240, kernel_size = 4, dilation = ngram_size = 3, so the state is (4-1)*3 = 9 steps, not 3.
3. All three norms are the offset-from-1 form (normed * (1 + w), Qwen4ExpTextRMSNorm) and grouped with group_size = hidden_size — four independent 2560-wide norms inside the 10240 vector, same as hc_norm. See bench/qwen4_exp/ARCHITECTURE.md §6.

The n-gram table is ~320M rows x 160 dims. It is NOT resident: the row cache, pinned arena and deferred-load path from #746 serve it off NVMe. What does NOT transfer from #746 is the id computation — see ids.rs.

Re-exports

pub use ids::PleIdDims;

pub use ids::ple_ngram_ids;

Modules

dump

Highway taps for the qwen4_exp bisect.

ids

PLE n-gram row ids: EOS-aware right-shift plus the multiply-XOR hash.

Structs

PleLayer

PleSeqState

Per-SEQUENCE carry: the dilated conv’s 9 steps and the token history the id hash needs. Owned by the sequence’s crate::layer::SsmLayerState (Avarok #753 item B: concurrency needs one of these per in-flight sequence, not a layer singleton).

PleWeights

The dense weights of one PLE site.