it’s kinda obvious in retrospect. it’s amazing it took people 20 years to figure it out.

There’s the Neural Network inference implementation in 3000 lines of C code (can’t find the link rn), and looking only at the quantity and complexity of the code, a PhD student might have figured that out over the course of 1-2 years. Of course there was a lot of trial and error involved, but still.

What i learn from that: It is an utterly wrong and falsified statement to say that “the human brain is infinitely complicated, and we cannot even begin to understand how it works”. Because ultimately, neural networks are largely modelled after the brain, and they are surprisingly understandable/intellegible.

One of my favorite formats

A brief, but surprisingly accurate meme. I spy feed-forward neural nets, KQV attention, tokenizers… what’s in the top right panel? Backpropagation?

Dude opens his mouth and white noise comes out