Suggestions for getting started with local AI

A technical guide details a 46-layer hybrid model architecture alternating between Linear Attention (LA) and Full Attention (FA) layers, with LA layers using qkv, z, and out projections plus A_log, dt_bias, conv1d, a/b projections, and state norm, while FA layers use q, k, v, o projections with q_norm and k_norm, and all layers include gate, up, and down projections. The guide offers suggestions for getting started with local AI, focusing on implementing this specific architecture.

| 0 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 1 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 2 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 3 | FA | q, k, v, o | q norm, k norm | gate, up, down | | 4 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 5 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 6 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 7 | FA | q, k, v, o | q norm, k norm | gate, up, down | | 8 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 9 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 10 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 11 | FA | q, k, v, o | q norm, k norm | gate, up, down | | 12 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 13 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 14 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 15 | FA | q, k, v, o | q norm, k norm | gate, up, down | | 16 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 17 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 18 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 19 | FA | q, k, v, o | q norm, k norm | gate, up, down | | 20 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 21 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 22 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 23 | FA | q, k, v, o | q norm, k norm | gate, up, down | | 24 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 25 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 26 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 27 | FA | q, k, v, o | q norm, k norm | gate, up, down | | 28 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 29 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 30 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 31 | FA | q, k, v, o | q norm, k norm | gate, up, down | | 32 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 33 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 34 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 35 | FA | q, k, v, o | q norm, k norm | gate, up, down | | 36 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 37 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 38 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 39 | FA | q, k, v, o | q norm, k norm | gate, up, down | | 40 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 41 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 42 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 43 | FA | q, k, v, o | q norm, k norm | gate, up, down | | 44 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 45 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 46 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 47 | FA | q, k, v, o | q norm, k norm | gate, up, down | | 48 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 49 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 50 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 51 | FA | q, k, v, o | q norm, k norm | gate, up, down | | 52 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 53 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 54 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 55 | FA | q, k, v, o | q norm, k norm | gate, up, down | | 56 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 57 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 58 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 59 | FA | q, k, v, o | q norm, k norm | gate, up, down | | 60 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 61 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 62 | LA | qkv, z, out | A log, dt bias, conv1d, a/b projections, state norm | gate, up, down | | 63 | FA | q, k, v, o | q norm, k norm | gate, up, down |