Thank you so much for the detailed comment! I would be very excited to chat offline, but I’ll put a few questions here that are directly related to the comment:
For one, I think while the forecasts in that report are the best publicly available thing we have, there’s significant room to do better
These are all super interesting points! One thing that strikes me as a similarity between many of them is that it’s not straightforward what metric (# transistors per chip, price performance, etc.) is the most useful to forecast AI timelines. Do you think price performance for certain applications could be one of the better ones to use on its own? Or is it perhaps better practice to keep an index of some number of trends?
I think it’s not the case that we have access to enough people with sufficient knowledge and expert opinion. I’ve been really interested in talking to hardware experts, and I think I would selfishly benefit substantially from experts who had thought more about “the big picture” or more speculative hardware possibilities
Are there any specific speculative hardware areas you think may be neglected? I mentioned photonics and quantum computing in the post because these are the only ones I’ve spent more than an hour thinking about. I vaguely plan to read up on the other technologies in the IRDS, but if there are some that might be worth looking more into than others (or some that didn’t make their list at all!) that would help focus this plan significantly.
A recent report estimates that ASICs are poised to take over 50% of the hardware market in the coming years
Thank you for pointing this out! I talked with someone working in hardware that gave me the opposite impression and I haven’t thought to actually look into this myself. (In retrospect this may have been their sales pitch to differentiate themselves from their competitors. FWIW, I think their argument was that AI moves fast enough that an ASIC start-up will be irrelevant before their product hits the market.) I look forward to updating this impression!
I think things like understanding models of hardware costs, the overall hardware market, cloud computing, etc. are not well-encapsulated by the kind of understanding technical experts tend to have.
I would naively think this would be another point in favor of working at start-ups compared to more established companies. My impression is that start-ups have to spend more time thinking carefully about their market is in order to attract funding (and the small size means technical people are more involved with this thinking). Does that seem reasonable?
Do you think price performance for certain applications could be one of the better ones to use on its own? Or is it perhaps better practice to keep an index of some number of trends?
I think price performance, measured in something like “operations / $”, is by far the most important metric, caveating that by itself it doesn’t differentiate between one-time costs of design and purchase and ongoing costs to run hardware, and it doesn’t account for limitations in memory, networking, and software for parallelization that constrain performance as the number of chips are scaled up.
Are there any specific speculative hardware areas you think may be neglected? I mentioned photonics and quantum computing in the post because these are the only ones I’ve spent more than an hour thinking about. I vaguely plan to read up on the other technologies in the IRDS, but if there are some that might be worth looking more into than others (or some that didn’t make their list at all!) that would help focus this plan significantly.
There has been a lot of recent work in optical chips / photonics, so I’ve been following them closely—I have my own notes on publicly available info here. I think quantum computing is likely further from viability but good to pay attention to. I also think it’s worth understanding the likelihood and implications of 3D CMOS chips, just because at least IRDS predictions suggest that might be the way forward in the next decade (I think these are much less speculative than the two above). I haven’t looked into this as much as I’d like, though—I actually also have on my todo list to read through the IRDS list and identify the things that are most likely and have the highest upside. Maybe we can compare notes. :)
I would naively think this would be another point in favor of working at start-ups compared to more established companies. My impression is that start-ups have to spend more time thinking carefully about their market is in order to attract funding (and the small size means technical people are more involved with this thinking). Does that seem reasonable?
I suspect in most roles in either a start-up or a large company you’ll be quite focused on the tech and not very focused on the market or the cost model—I don’t think this strongly favors working for a start-up.
Thank you so much for the detailed comment! I would be very excited to chat offline, but I’ll put a few questions here that are directly related to the comment:
These are all super interesting points! One thing that strikes me as a similarity between many of them is that it’s not straightforward what metric (# transistors per chip, price performance, etc.) is the most useful to forecast AI timelines. Do you think price performance for certain applications could be one of the better ones to use on its own? Or is it perhaps better practice to keep an index of some number of trends?
Are there any specific speculative hardware areas you think may be neglected? I mentioned photonics and quantum computing in the post because these are the only ones I’ve spent more than an hour thinking about. I vaguely plan to read up on the other technologies in the IRDS, but if there are some that might be worth looking more into than others (or some that didn’t make their list at all!) that would help focus this plan significantly.
Thank you for pointing this out! I talked with someone working in hardware that gave me the opposite impression and I haven’t thought to actually look into this myself. (In retrospect this may have been their sales pitch to differentiate themselves from their competitors. FWIW, I think their argument was that AI moves fast enough that an ASIC start-up will be irrelevant before their product hits the market.) I look forward to updating this impression!
I would naively think this would be another point in favor of working at start-ups compared to more established companies. My impression is that start-ups have to spend more time thinking carefully about their market is in order to attract funding (and the small size means technical people are more involved with this thinking). Does that seem reasonable?
I think price performance, measured in something like “operations / $”, is by far the most important metric, caveating that by itself it doesn’t differentiate between one-time costs of design and purchase and ongoing costs to run hardware, and it doesn’t account for limitations in memory, networking, and software for parallelization that constrain performance as the number of chips are scaled up.
There has been a lot of recent work in optical chips / photonics, so I’ve been following them closely—I have my own notes on publicly available info here. I think quantum computing is likely further from viability but good to pay attention to. I also think it’s worth understanding the likelihood and implications of 3D CMOS chips, just because at least IRDS predictions suggest that might be the way forward in the next decade (I think these are much less speculative than the two above). I haven’t looked into this as much as I’d like, though—I actually also have on my todo list to read through the IRDS list and identify the things that are most likely and have the highest upside. Maybe we can compare notes. :)
I suspect in most roles in either a start-up or a large company you’ll be quite focused on the tech and not very focused on the market or the cost model—I don’t think this strongly favors working for a start-up.