bounded_gate

Function bounded_gate 

Source
pub fn bounded_gate(
    g_lowrank: &[f32],
    dt_bias: &[f32],
    a_log: &[f32],
    dims: KdaDims,
    lower_bound: f32,
) -> Vec<f32>
Expand description

The bounded GLM forget gate, item 3 of the four NEW structural items.

gate[t, h, d] = lower_bound * sigmoid(exp(a_log[h]) * (g_lowrank[t, h*D + d] + dt_bias[h*D + d]))

a_log is per head; dt_bias is per channel. Getting that asymmetry wrong is the single highest-risk line in the port, and it is why the two are separate arguments here.