I followed a link on Twitter to a fun and informative 2015 blog post by Tim Dettmers:

The Brain vs Deep Learning Part I: Computational Complexity — Or Why the Singularity Is Nowhere Near

The headline conclusion is that it takes at least $10^{21}$ FLOP/s to run the algorithms of a human brain, and therefore "it is unlikely that there will be a technological singularity in this century." I disagree with that, and this post explores why.

(Specifically, I disagree with "at least $10^{21}$ FLOP/s". There's a separate step to go from "at least $10^{21}$ FLOP/s" to "it is unlikely that there will be a technological singularity in this century"—this step is related to Moore's law, bandwidth requirements for parallelization, etc. Tim's blog post has extensive discussion of this second step, and I won't say anything about that here; I'd have to think about it more.)

(I'm writing this in 2021, six years later, but Tim has a comment on this very site that says he still stands by that post; in fact he now goes even further and says "I believe that AGI will be physically impossible with classical computers.")

I highly recommend the original post. Indeed, if I didn't like the post so much, I would not have bothered writing a response. :-)

Are brain algorithms computationally expensive to simulate?

Yes! Definitely! I think it's especially telling that nobody has applied the Dileep George brain-inspired image-processing model to ImageNet, sticking to much smaller images with far fewer categories of objects (MNIST, CAPTCHAs etc.).

Likewise, this Randall O'Reilly paper has a fascinating computational exploration of (in my opinion) different and complementary aspects of the human visual system. That paper tests its theories on a set of ≈1000 256×256-pixel, 8-frame movies from 100 categories—compare to ImageNet's 14 million images from 20,000 categories ... or compare it to the number of visual categories that you can recognize. Training the model still took 512 InfiniBand-connected processor nodes running for ≈24 hours on their campus supercomputer (source: personal communication). The real human vision system is dramatically larger and more complicated than this model, and the whole brain is larger and more complicated still!

But, when I say "computationally expensive to simulate" above, I mean it in, like, normal-person-in-2021 standards of what's computationally e...