pub fn bounded_gate(
g_lowrank: &[f32],
dt_bias: &[f32],
a_log: &[f32],
dims: KdaDims,
lower_bound: f32,
) -> Vec<f32>Expand description
The bounded GLM forget gate, item 3 of the four NEW structural items.
gate[t, h, d] = lower_bound * sigmoid(exp(a_log[h]) * (g_lowrank[t, h*D + d] + dt_bias[h*D + d]))
a_log is per head; dt_bias is per channel. Getting that asymmetry wrong is the
single highest-risk line in the port, and it is why the two are separate arguments here.