Hello Agustín, thanks for engaging with our writings and sharing your feedback.
Regarding the ambitiousness, low chances of overall success, and low chances of uptake by human developers and decision-makers (I emphasize “human” because if some tireless near-AGI or AGI comes along it could change the cost of building agents for participation in the Gaia Network dramatically), we are in complete agreement.
But notice Gaia Network could be seen as a much-simplified (from the perspectives of mathematics and Machine Learning) version of Davidad’s OAA, as we framed it in the first post. Also, Gaia Network tries to leverage (at least approximately and to some degree) the existing (political) institutions and economic incentives. In contrast, it’s very unclear to me how the political economy in the “OAA world” could look like, and what is even a remotely plausible plan for switching from the incumbent political economy of the civilisation to OAA, or “plugging” OAA “on top” of the incumbent political economy (and hasn’t been discussed publicly anywhere, to the best of our knowledge). We also discussed this in the first post. Also, notice that due to its extreme ambitiousness, Davidad doesn’t count on humans implementing OAA with their bare hands, it’s a deal-breaker if there isn’t an AI that can automate 99%+ of technical work needed to convert the current science into Infra-Bayesian language.[1] And yes, the same applies to Gaia Network: it’s not feasible without massive assistance from AI tools that can do most of the heavy lifting. But if anything, this reliance on AI is less extreme in the case of Gaia Network than in the case of OAA.
The above makes me think that you should therefore be even more skeptical of OAA’s chances of success than you are about Gaia’s chances. Is this correct? If not, what do you disagree about in the reasoning above, or what elements of OAA make you think it’s more likely to succeed?
Adoption
The “cold start” problem is huge for any system that counts on network effect, and Gaia Network is no exception. But this also means that the cost of convincing most decision-makers (businesses, scientists, etc.) to use the system is far smaller than the cost of convincing the first few, multiplied by the total number of agents. We have also proposed how various early adopters could get value out of the “model-based and free energy-minimising way” of doing decision-making (we don’t need the adoption of Gaia Network right off the bat, more on this below) very soon, in absolutely concrete terms (monetary and real-world risk mitigation) in this thread.
In fact, we think that if there are sufficiently many AI agents and decision intelligence systems that are model-based, i.e., use some kinds of executable state-space (“world”) models to do simulations, hypothesise counterfactually about different courses of actions and external conditions (sometimes in collaboration with other agents, i.e., planning together), and deploy regularisation techniques (from Monte Carlo aggregation of simulation results to amortized adversarial methods suggested by Bengio on slide 47 here) to permit compositional reasoning about risk and uncertaintly that scales beyond the boundary of a single agent, the benefits of collaborative inference of the most accurate and well-regularised models will be so huge that something like Gaia Network will emerge pretty much “by default” because a lot of scientists and industry players will work in parallel to build some versions and local patches of it.
Blockchains, crypto, DeFi, DAOs
I understand why the default prior when hearing anything about crypto, DeFi, and DAOs now is that people who propose something like this are either fantaseurs, or cranks, or, worse, scammers. That’s unfortunate to everyone who just wants to use the technical advances that happen to be loosely associated with this field, which now includes almost anything that has to do with cryptography, identity, digital claims, and zero-knowledge computation.
Generally speaking, zero-knowledge (multi-party) computation is the only solution to make some proofs (of contribution, of impact, of lack of deceit, etc.) without compromising privacy (e.g., proprietary models, know-how, personal data). The ways to deal with this dilemma “in the real world” today inevitably come down to some kind of surveillance which many people become very uneasy about. For example, consider the present discussion of data center audits and compute governance. It’s fine with me and most other people except for e/accs, for now, but what about the time when the cost of training powerful/dangerous models will drop so much that anyone can buy a chip to train the next rogue AI for 1000$? How does compute governance look in this world?
Governance
I’m also skeptical of the theory of change. Even if AI Safety timelines were long, and we managed to pull this Herculean effort off, we would still have to deal with problems around AI Safety governance.
I don’t think AI Safety governance is that special among other kinds of governance. But more generally on this point, of course, governance is important, and Gaia Network doesn’t claim to “solve” it; rather, it plans to rely on some solutions developed by other projects (see numerous examples in CIP ecosystem map, OpenAI’s “Democratic Inputs to AI” grantees, etc.).
We just mention in passing incorporating preferences of system’s stakeholders into Gaia agents’ subjective value calculations (i.e., building reward models for these agents/entities, if you wish), but there is a lot to be done there: how the preferences of the stakeholders are aggregated and weighted, who can claim to be a stakeholders of this or that system in the first place, etc. Likewise, on the general Gaia diagram in the post, there is a small arrow from “Humans and collectives” box to “Decision Engines” box labelled “Review and oversight”, and, as you can imagine, there is a lot to be going on there as well.
Why would AGI companies want to stick to this way of developing systems?
IDK, convinced that this is a safe approach? Being coerced (including economically, not necessary by force) by the broader consensus of using such Gaia Network-like systems? This is a collective action problem. This question could be addressed to any AI Safety agenda and the answer would be the same.
I wouldn’t say that we “claim to be able to slay Moloch”. Rafael is more bold in his claims and phrasing than me, but I think even he wouldn’t say that. I would say that the project looks very likely to help to counteract Molochian pressures. But this seems to me almost a self-evident statement, given the nature of the proposal.
Compare with Collective Intelligence Project. It has started with the mission to “fix governance” (and pretty much “help to counteract Moloch” in the domain of political economy, too, they barely didn’t use this concept, or maybe they even did, I don’t want to check it now), and now they “pivoted” to AI safety and achieved great legibility on this path: e.g., they partner with OpenAI, apparently, on more than one project now. Does this mean that CIP is a “solution looking for a problem”? No, it’s just the kind of project that naturally lends to helps both with Moloch and AI safety. I’d say the same could be said of Gaia Network (if it is realised in some forms) and this lies pretty much in plain sight.
Furthermore, this shouldn’t be surprising in general, because AI transition of the economy is evidently an accelerator and a risk factor in the Moloch model, and therefore these domains (Moloch and AI safety) almost merge in my overall model of risk. Cf. Scott Aaronson’s reasoning that AI will inevitably be in the causal structure of any outcome of this century so “P(doom from AI)” is not well defined; I agree with him and only think about “P(doom)” without specification what this doom “comes from”. Again, note that it seems that most narratives about possible good outcomes (take OpenAI’s superalignment plan, Conjecture’s CoEm agenda, OAA, Gaia Network) all rely on developing very advanced (if not superhuman) AI along the way.
Notice here again: you mention that most scientists don’t know about Bayesian methods, but perhaps at least two orders of magnitude still fewer scientists have even heard of Infra-Bayesianism, let alone being convinced it’s a sound and a necessary methodology for doing science. Whereas for Bayesianism, from my perspective, it seems there is quite a broad consensus of its soundness: there are numerous pieces and even books written about how P-values are a bullshit value of doing science and that scientists should take up (Bayesian) causal inference instead.
There are a few notable voices that dismiss Bayesian inference, for example, David Deutsch, but then no less notable voices, such as Scott Aaronson and Sean Carroll (of the people that I’ve heard, anyway), that dismiss Deutsch’s dismissal in turn.
The above makes me think that you should therefore be even more skeptical of OAA’s chances of success than you are about Gaia’s chances.
I am, but OAA also seems less specific, and it’s harder to evaluate its feasibility compared to something more concrete (like this proposal).
In fact, we think that if there are sufficiently many AI agents and decision intelligence systems that are model-based, i.e., use some kinds of executable state-space (“world”) models to do simulations, hypothesise counterfactually about different courses of actions and external conditions (sometimes in collaboration with other agents, i.e., planning together), and deploy regularisation techniques (from Monte Carlo aggregation of simulation results to amortized adversarial methods suggested by Bengio on slide 47 here) to permit compositional reasoning about risk and uncertaintly that scales beyond the boundary of a single agent, the benefits of collaborative inference of the most accurate and well-regularised models will be so huge that something like Gaia Network will emerge pretty much “by default” because a lot of scientists and industry players will work in parallel to build some versions and local patches of it.
My problem with this is that it sounds good, but this argument relies on many hidden premises, that make me inherently skeptical of any strong claims like “(…) the benefits of collaborative inference of the most accurate and well-regularised models will be so huge that something like Gaia Network will emerge pretty much ‘by default’”.
I think this could be addressed by a convincing MVP, and I think that you’re working on that, so I won’t push further on this point.
It’s fine with me and most other people except for e/accs, for now, but what about the time when the cost of training powerful/dangerous models will drop so much that anyone can buy a chip to train the next rogue AI for 1000$? How does compute governance look in this world?
The current best proposals for compute governance rely on very specific types of math. I don’t think throwing blockchain or DAOs at the problem makes a lot of sense, unless you find an instance of the very specific set of problems they’re good at solving.
My priors against the crypto world comes mostly from noticing a lot of people throwing tools to problems without a clear story of how these tools actually solve the problem. This has happened so many times that I have come to generally distrust crypto/blockchain proposals unless they give me a clear explanation of why using these technologies makes sense.
But I think the point I made here was kinda weak anyway (it was, at best, discrediting by association), so I don’t think it makes sense to litigate this particular point.
Compare with Collective Intelligence Project. It has started with the mission to “fix governance” (and pretty much “help to counteract Moloch” in the domain of political economy, too, they barely didn’t use this concept, or maybe they even did, I don’t want to check it now), and now they “pivoted” to AI safety and achieved great legibility on this path: e.g., they partner with OpenAI, apparently, on more than one project now. Does this mean that CIP is a “solution looking for a problem”? No, it’s just the kind of project that naturally lends to helps both with Moloch and AI safety. I’d say the same could be said of Gaia Network (if it is realised in some forms) and this lies pretty much in plain sight.
I find this decently convincing, actually. Like, maybe, I’m pattern matching too much on other projects which have in the past done something similar (just lightly rebranding themselves while tacking a completely different problem).
Overall, I still don’t feel very good about the overall feasibility of this project, but I think you were right to push back on some of my counterarguments here.
Thanks @Agustín Covarrubias . Glad to hear that you feel this is concrete enough to be critiqued and cross-validated, that was exactly our goal in writing and posting this. From your latest responses, it seems like the main reason why you “still don’t feel very good about the overall feasibility of this project” is the lack of a “convincing MVP”, is that right? We are indeed working on this along a few different lines, so I would be curious to understand what kind of evidence from an MVP it would take to convince you or shift your opinion about feasibility.
Hello Agustín, thanks for engaging with our writings and sharing your feedback.
Regarding the ambitiousness, low chances of overall success, and low chances of uptake by human developers and decision-makers (I emphasize “human” because if some tireless near-AGI or AGI comes along it could change the cost of building agents for participation in the Gaia Network dramatically), we are in complete agreement.
But notice Gaia Network could be seen as a much-simplified (from the perspectives of mathematics and Machine Learning) version of Davidad’s OAA, as we framed it in the first post. Also, Gaia Network tries to leverage (at least approximately and to some degree) the existing (political) institutions and economic incentives. In contrast, it’s very unclear to me how the political economy in the “OAA world” could look like, and what is even a remotely plausible plan for switching from the incumbent political economy of the civilisation to OAA, or “plugging” OAA “on top” of the incumbent political economy (and hasn’t been discussed publicly anywhere, to the best of our knowledge). We also discussed this in the first post. Also, notice that due to its extreme ambitiousness, Davidad doesn’t count on humans implementing OAA with their bare hands, it’s a deal-breaker if there isn’t an AI that can automate 99%+ of technical work needed to convert the current science into Infra-Bayesian language.[1] And yes, the same applies to Gaia Network: it’s not feasible without massive assistance from AI tools that can do most of the heavy lifting. But if anything, this reliance on AI is less extreme in the case of Gaia Network than in the case of OAA.
The above makes me think that you should therefore be even more skeptical of OAA’s chances of success than you are about Gaia’s chances. Is this correct? If not, what do you disagree about in the reasoning above, or what elements of OAA make you think it’s more likely to succeed?
Adoption
The “cold start” problem is huge for any system that counts on network effect, and Gaia Network is no exception. But this also means that the cost of convincing most decision-makers (businesses, scientists, etc.) to use the system is far smaller than the cost of convincing the first few, multiplied by the total number of agents. We have also proposed how various early adopters could get value out of the “model-based and free energy-minimising way” of doing decision-making (we don’t need the adoption of Gaia Network right off the bat, more on this below) very soon, in absolutely concrete terms (monetary and real-world risk mitigation) in this thread.
In fact, we think that if there are sufficiently many AI agents and decision intelligence systems that are model-based, i.e., use some kinds of executable state-space (“world”) models to do simulations, hypothesise counterfactually about different courses of actions and external conditions (sometimes in collaboration with other agents, i.e., planning together), and deploy regularisation techniques (from Monte Carlo aggregation of simulation results to amortized adversarial methods suggested by Bengio on slide 47 here) to permit compositional reasoning about risk and uncertaintly that scales beyond the boundary of a single agent, the benefits of collaborative inference of the most accurate and well-regularised models will be so huge that something like Gaia Network will emerge pretty much “by default” because a lot of scientists and industry players will work in parallel to build some versions and local patches of it.
Blockchains, crypto, DeFi, DAOs
I understand why the default prior when hearing anything about crypto, DeFi, and DAOs now is that people who propose something like this are either fantaseurs, or cranks, or, worse, scammers. That’s unfortunate to everyone who just wants to use the technical advances that happen to be loosely associated with this field, which now includes almost anything that has to do with cryptography, identity, digital claims, and zero-knowledge computation.
Generally speaking, zero-knowledge (multi-party) computation is the only solution to make some proofs (of contribution, of impact, of lack of deceit, etc.) without compromising privacy (e.g., proprietary models, know-how, personal data). The ways to deal with this dilemma “in the real world” today inevitably come down to some kind of surveillance which many people become very uneasy about. For example, consider the present discussion of data center audits and compute governance. It’s fine with me and most other people except for e/accs, for now, but what about the time when the cost of training powerful/dangerous models will drop so much that anyone can buy a chip to train the next rogue AI for 1000$? How does compute governance look in this world?
Governance
I don’t think AI Safety governance is that special among other kinds of governance. But more generally on this point, of course, governance is important, and Gaia Network doesn’t claim to “solve” it; rather, it plans to rely on some solutions developed by other projects (see numerous examples in CIP ecosystem map, OpenAI’s “Democratic Inputs to AI” grantees, etc.).
We just mention in passing incorporating preferences of system’s stakeholders into Gaia agents’ subjective value calculations (i.e., building reward models for these agents/entities, if you wish), but there is a lot to be done there: how the preferences of the stakeholders are aggregated and weighted, who can claim to be a stakeholders of this or that system in the first place, etc. Likewise, on the general Gaia diagram in the post, there is a small arrow from “Humans and collectives” box to “Decision Engines” box labelled “Review and oversight”, and, as you can imagine, there is a lot to be going on there as well.
IDK, convinced that this is a safe approach? Being coerced (including economically, not necessary by force) by the broader consensus of using such Gaia Network-like systems? This is a collective action problem. This question could be addressed to any AI Safety agenda and the answer would be the same.
Moloch
I wouldn’t say that we “claim to be able to slay Moloch”. Rafael is more bold in his claims and phrasing than me, but I think even he wouldn’t say that. I would say that the project looks very likely to help to counteract Molochian pressures. But this seems to me almost a self-evident statement, given the nature of the proposal.
Compare with Collective Intelligence Project. It has started with the mission to “fix governance” (and pretty much “help to counteract Moloch” in the domain of political economy, too, they barely didn’t use this concept, or maybe they even did, I don’t want to check it now), and now they “pivoted” to AI safety and achieved great legibility on this path: e.g., they partner with OpenAI, apparently, on more than one project now. Does this mean that CIP is a “solution looking for a problem”? No, it’s just the kind of project that naturally lends to helps both with Moloch and AI safety. I’d say the same could be said of Gaia Network (if it is realised in some forms) and this lies pretty much in plain sight.
Furthermore, this shouldn’t be surprising in general, because AI transition of the economy is evidently an accelerator and a risk factor in the Moloch model, and therefore these domains (Moloch and AI safety) almost merge in my overall model of risk. Cf. Scott Aaronson’s reasoning that AI will inevitably be in the causal structure of any outcome of this century so “P(doom from AI)” is not well defined; I agree with him and only think about “P(doom)” without specification what this doom “comes from”. Again, note that it seems that most narratives about possible good outcomes (take OpenAI’s superalignment plan, Conjecture’s CoEm agenda, OAA, Gaia Network) all rely on developing very advanced (if not superhuman) AI along the way.
Notice here again: you mention that most scientists don’t know about Bayesian methods, but perhaps at least two orders of magnitude still fewer scientists have even heard of Infra-Bayesianism, let alone being convinced it’s a sound and a necessary methodology for doing science. Whereas for Bayesianism, from my perspective, it seems there is quite a broad consensus of its soundness: there are numerous pieces and even books written about how P-values are a bullshit value of doing science and that scientists should take up (Bayesian) causal inference instead.
There are a few notable voices that dismiss Bayesian inference, for example, David Deutsch, but then no less notable voices, such as Scott Aaronson and Sean Carroll (of the people that I’ve heard, anyway), that dismiss Deutsch’s dismissal in turn.
I am, but OAA also seems less specific, and it’s harder to evaluate its feasibility compared to something more concrete (like this proposal).
My problem with this is that it sounds good, but this argument relies on many hidden premises, that make me inherently skeptical of any strong claims like “(…) the benefits of collaborative inference of the most accurate and well-regularised models will be so huge that something like Gaia Network will emerge pretty much ‘by default’”.
I think this could be addressed by a convincing MVP, and I think that you’re working on that, so I won’t push further on this point.
The current best proposals for compute governance rely on very specific types of math. I don’t think throwing blockchain or DAOs at the problem makes a lot of sense, unless you find an instance of the very specific set of problems they’re good at solving.
My priors against the crypto world comes mostly from noticing a lot of people throwing tools to problems without a clear story of how these tools actually solve the problem. This has happened so many times that I have come to generally distrust crypto/blockchain proposals unless they give me a clear explanation of why using these technologies makes sense.
But I think the point I made here was kinda weak anyway (it was, at best, discrediting by association), so I don’t think it makes sense to litigate this particular point.
I find this decently convincing, actually. Like, maybe, I’m pattern matching too much on other projects which have in the past done something similar (just lightly rebranding themselves while tacking a completely different problem).
Overall, I still don’t feel very good about the overall feasibility of this project, but I think you were right to push back on some of my counterarguments here.
Thanks @Agustín Covarrubias . Glad to hear that you feel this is concrete enough to be critiqued and cross-validated, that was exactly our goal in writing and posting this. From your latest responses, it seems like the main reason why you “still don’t feel very good about the overall feasibility of this project” is the lack of a “convincing MVP”, is that right? We are indeed working on this along a few different lines, so I would be curious to understand what kind of evidence from an MVP it would take to convince you or shift your opinion about feasibility.