What might be interesting in future editions is to note the rate of change (assuming it exists and is >0) over each generation of LLMs, given the truism that their capabilities and thus cognitive complexity, which at least some models favor as relevant, is increasing quite rapidly. I find these questions quite important and I hope that we can continue to interrogate and refine these and other methodologies for answering them.
Hi Andrew, I definitely agree. Tracking how these systems evolve over time — both what the models reveal about their changes and how we can formulate projections based on those insights — has been one of our main motivations for this work. We’re in fact hoping to gather data on newer systems and develop some more informed forecasts based on what we find.
What might be interesting in future editions is to note the rate of change (assuming it exists and is >0) over each generation of LLMs, given the truism that their capabilities and thus cognitive complexity, which at least some models favor as relevant, is increasing quite rapidly. I find these questions quite important and I hope that we can continue to interrogate and refine these and other methodologies for answering them.
Hi Andrew, I definitely agree. Tracking how these systems evolve over time — both what the models reveal about their changes and how we can formulate projections based on those insights — has been one of our main motivations for this work. We’re in fact hoping to gather data on newer systems and develop some more informed forecasts based on what we find.