Thanks for your thoughts! When writing this up I also felt that the algorithm one is the weakest one, so let me answer from two perspectives:
From the room to invent new algorithms: Convolutional neural networks have been around since the 80s, weâve been using GPUs to run them since about 10 years. If there really would be huge potential left, Iâd be a bit surprised that we didnât find it in the last 40 years alreadyâwe certainly had incentives because hardware was so slow and people had to optimize, but of course you never know. I tried to find a paper reviewing efficiency improvements of non-negative matrix factorization over time, I think that could be a fun guide, but couldnât find one.
From the brain perspective: Yes, itâs puzzling that the brain can do all this on 12 watts power while OpenAI is using server farms that consume much much more than that. So somewhere there must be huge efficiency gains. Note that thatâs mostly on the training sideââevaluatingâ a network is pretty efficient as far as I know. For training, there could be different reasons:
Transfer learning: Maybe the âcomputation of evolutionâ just âpre-programmedâ our brain similar to how we use transfer learning. Itâs already pretty close to where we want it and we just need to fine tune. Transfer learning on neural networks is already pretty cheap today. One argument supporting this would be that many animals are perfectly functional from day 1 of their life without much learning. Of course not same level of intelligence, but still.
Hardware: The brain doesnât run on silicone. We use a very very abstracted version of our brain and there is much more going on biologically. Some people argue that a lot of computation is already happening in the dendrites, maybe the morphology of neurons has effects on computation, maybe the specific nonlinearity applied by the neurons is more relevant than we think, ⌠. One way to try to adress this would be to build chips that are more similar (âneuromorphicâ) but I havenât seen much progress there
Architecture: The brain isnât a CNN. This might be a good approximation for our sensory cortices but even there itâs not the same. The brain is very recurrent, not feed-forward, and it canât send signals back through itâs synapses and therefore canât implement backpropagation. Maybe weâre just using the wrong architecture and if we find the right one itâs going to go much faster. I did my PhD on something related to this and I gave up haha, but of course, Iâm sure there are lots of things to be discovered here.
Thanks for your thoughts! When writing this up I also felt that the algorithm one is the weakest one, so let me answer from two perspectives:
From the room to invent new algorithms: Convolutional neural networks have been around since the 80s, weâve been using GPUs to run them since about 10 years. If there really would be huge potential left, Iâd be a bit surprised that we didnât find it in the last 40 years alreadyâwe certainly had incentives because hardware was so slow and people had to optimize, but of course you never know. I tried to find a paper reviewing efficiency improvements of non-negative matrix factorization over time, I think that could be a fun guide, but couldnât find one.
From the brain perspective: Yes, itâs puzzling that the brain can do all this on 12 watts power while OpenAI is using server farms that consume much much more than that. So somewhere there must be huge efficiency gains. Note that thatâs mostly on the training sideââevaluatingâ a network is pretty efficient as far as I know. For training, there could be different reasons:
Transfer learning: Maybe the âcomputation of evolutionâ just âpre-programmedâ our brain similar to how we use transfer learning. Itâs already pretty close to where we want it and we just need to fine tune. Transfer learning on neural networks is already pretty cheap today. One argument supporting this would be that many animals are perfectly functional from day 1 of their life without much learning. Of course not same level of intelligence, but still.
Hardware: The brain doesnât run on silicone. We use a very very abstracted version of our brain and there is much more going on biologically. Some people argue that a lot of computation is already happening in the dendrites, maybe the morphology of neurons has effects on computation, maybe the specific nonlinearity applied by the neurons is more relevant than we think, ⌠. One way to try to adress this would be to build chips that are more similar (âneuromorphicâ) but I havenât seen much progress there
Architecture: The brain isnât a CNN. This might be a good approximation for our sensory cortices but even there itâs not the same. The brain is very recurrent, not feed-forward, and it canât send signals back through itâs synapses and therefore canât implement backpropagation. Maybe weâre just using the wrong architecture and if we find the right one itâs going to go much faster. I did my PhD on something related to this and I gave up haha, but of course, Iâm sure there are lots of things to be discovered here.