hard to assess counterfactuals … [suggest specific topics that the] … person wouldn’t otherwise [pick]
But you select topics, not people? Maybe, people could select which ones work best for them or form teams considering all others’ expertise. This should always counterfactually optimize allocation of resources given a set of topics, using collective intelligence.
depends on lots of open questions from priorities research and on your personal fit for the investigations
Interest in having answers should be considered, though. If this framework just creates interest in questions that relate to impact, then that is a significant contribution. Imagine you never wrote the list and I came up with this post—maybe you would have not even noticed or thought well no we do it better—so, interest increases engagement, for better or worse.
issuers may be biased about their actual vs. counterfactual impact because they don’t want to believe that they’ve made a mistake and they may be afraid that their certificate price will be lower.
They always try to buy for a good value, so should make sure they are not making a mistake before purchasing anything. But yes, for the total impact that an issuer makes even the non-purchased certificates’ counterfactuals should be used. For instance, if someone buys only 1⁄10 certificates in which they expressed interested and thus diverts the attention of 9 additional people from projects of different impact, then the sum of differences should be added to that of the 1⁄10 if the issuer seeks to evaluate the impact of their suggestions list, which they can choose to sell or not.
it’s probably by and large a valid approximation that the same projects get funded, just differently (and hopefully faster and smarter)
Hmm… yes, but maybe actually some organizers would not take the risk, so some projects would get unfunded or delayed and many more projects could be run but eventually not purchased (which could reduce some persons’ ‘risk or learning’ budgets or be counterfactually beneficial).
number of investors … in the retrospective case, you get profit-seeking investors who would otherwise invest into pure for-profit ventures. The degree to which we’re able to attract these is an important part of the success of impact markets.
That makes sense. I would have not initially imagined people motivated purely by profit investing into impact but yes, should be key.
good proxy for (1) additional risk capital that flows into the space, (2) effort that would not otherwise have been expended on (at all) impactful things, and (3) time saved for retro funders.
So, (1)*(2)+(3)*funders’ counterfactual productivity[4], where 1 is e. g. in $, 2 in impact/$, 3 is time, and 4 in impact/time. (2 can also be a difference with respect to the counterfactual.) For 1 and 2, you can ask the investors and organizers and their networks (what they would have done if they did not know about this opportunity), 3 by funders’ time tracking, and 4 by assessing their reasoning about the prioritization of the set of projects that they fund or are interested in (whether they are aware of alternatives and complementarity of their investments).
random ideas
1. surveys make sense but why anonymous—you could get greater sincerity in a (‘safe’ group) conversation, perhaps. 2. yes, that can be convincing. You can maybe observe the EAG/EAGx data on ‘path to impact’ before and after posting and the time a user spent reviewing your post/whether they submitted any certificates. 3. Almost like a lab experiment …
I’m also worried that small methodological mistakes can ruin the analysis for us. And a power analysis will probably show that we’d need to recruit hundreds of participants to be able to learn anything from the results.
Well, stating assumptions and continuing to improve methods can be the way .. Hm, not really, maybe like 126 for an RCT if you observe a greater than 50% difference (alter only E, the minimal difference in the impact metric that the research would detect).
quick and dirty metric that is still informative and not biased to an unknowable extent.
Well, then the funders’ impact perception score that people competitively critique. That uses a network of human brains to assign appropriate weights to different factors.
What do you think? Do any solutions come to mind?
Ask funders: Estimate the impact of our framework compared to its non-existence. Explain your reasoning. Then, summarize quotes and see what people critique.
sum up the invested amounts weighed by the degree to which they are profit-oriented
But why would you weigh by the extent of profit-orientation? Does it matter the rational or apparent justification of the investment? For example, some for-profit fund managers interested in impact need to justify their actions to their clients by profit, so should not be perceived any negatively, or differently from those actually interested in profit, to support positive dynamics.
Maybe, people could select which ones work best for them or form teams considering all others’ expertise.
Oh yes, defs! :-)
But yes, for the total impact that an issuer makes even the non-purchased certificates’ counterfactuals should be used.
Agreed. Preregistration also solves a lot of problems with our version of p-hacking (i-hacking ^.^).
Hmm… yes, but maybe actually some organizers would not take the risk, so some projects would get unfunded or delayed and many more projects could be run but eventually not purchased (which could reduce some persons’ ‘risk or learning’ budgets or be counterfactually beneficial).
I think we’d need more assumptions for this to happen. When it comes to business ventures, some are better funded from investments and some from loans. But if we imagine a world without investments, with only loans, and then offer investments, it’s not straightforward to me that that would harm the businesses that are better suited for loans. Similarly, I’d think (at first approximation at least) that the projects that are not suited for retro funding will still apply for and receive prospective funding at almost the same rate that it is available today. The bottleneck is currently more the investment of funding than the availability of funding.
So, (1)*(2)+(3)*[4]
Right, those are multiplicative too… That makes them a bit not-robust (volatile?). Would it also be valid to conceive of each of those as a multiplier on the current counterfactual impact, e.g., 10x increase in seed funding, 10x improvement in the allocation of seed funding, 10x through economies of scale, etc.? Here’s a sample Guesstimate. But it feel to me like we’re much too likely to err in upward direction this way even if it’s just an estimate of the success (non–black swan) case. Sort of like the multi-stage fallacy give too low estimates if you multiply many probabilities, so I feel like multiplying all these multipliers probably also ignores lots of bottlenecks.
But that said, almost all the factors are multiplicative here except for two components of the allocation – better allocation thanks to knowledge of more languages and cultures and thanks to being embedded in more communities. (I suppose a person is in about as many communities regardless which language/s they speak.) The language component may seem unusually low, which is because so much of the world speaks English and because the US are such a big part of the world economy.
I’m assuming that more entrepreneurial people will join the fray who are not sufficiently altruistic to already be motivated to start EA charities but who will do a good job if they can get at least somewhat rich from it.
Finally, I’m assuming that retro funders reinvest all their savings into the retro funding and priorities research, and that priorities research still has < 10x room for improvement, which seems modest to me (given evidential cooperation in large worlds for example).
I’d be interested what you think of this version – any other factors that need to be considered, or anything that should be additive instead of multiplicative? But we can discuss all that in our call if you like. No need to reply here.
1. surveys make sense but why anonymous—you could get greater sincerity in a (‘safe’ group) conversation, perhaps.
Perhaps, but I’m worried that they may be incentivized to fib about it because it makes their impact seem more valuable and because it makes it more likely that we’ll continue to program (from which they want to benefit again), since we might discontinue it if we don’t think it’s sufficiently impactful.
Hm, not really, maybe like 126 for an RCT if you observe a greater than 50% difference
Thanks! Still a lot. I’ve updated away from this being an important factor to measure… I’d rather find a way to compare the success rates of investors vs. current prospective funders. If none of them are consistently better, they may lose interest in the system.
Ask funders: Estimate the impact of our framework compared to its non-existence. Explain your reasoning. Then, summarize quotes and see what people critique.
Yeah, that sounds good in both cases. It’s probably not easy to talk to them but it would be valuable.
But why would you weigh by the extent of profit-orientation?
If we’re just reallocating EA money, we’re not adding more. A big source of the impact stems from attracting for-profit money.
Looking forward to our call! Feel free to just respond verbally then. No need to type it all out. :-D
Thank you. I probably should schedule a call.
But you select topics, not people? Maybe, people could select which ones work best for them or form teams considering all others’ expertise. This should always counterfactually optimize allocation of resources given a set of topics, using collective intelligence.
Interest in having answers should be considered, though. If this framework just creates interest in questions that relate to impact, then that is a significant contribution. Imagine you never wrote the list and I came up with this post—maybe you would have not even noticed or thought well no we do it better—so, interest increases engagement, for better or worse.
They always try to buy for a good value, so should make sure they are not making a mistake before purchasing anything. But yes, for the total impact that an issuer makes even the non-purchased certificates’ counterfactuals should be used. For instance, if someone buys only 1⁄10 certificates in which they expressed interested and thus diverts the attention of 9 additional people from projects of different impact, then the sum of differences should be added to that of the 1⁄10 if the issuer seeks to evaluate the impact of their suggestions list, which they can choose to sell or not.
Hmm… yes, but maybe actually some organizers would not take the risk, so some projects would get unfunded or delayed and many more projects could be run but eventually not purchased (which could reduce some persons’ ‘risk or learning’ budgets or be counterfactually beneficial).
That makes sense. I would have not initially imagined people motivated purely by profit investing into impact but yes, should be key.
So, (1)*(2)+(3)*funders’ counterfactual productivity[4], where 1 is e. g. in $, 2 in impact/$, 3 is time, and 4 in impact/time. (2 can also be a difference with respect to the counterfactual.) For 1 and 2, you can ask the investors and organizers and their networks (what they would have done if they did not know about this opportunity), 3 by funders’ time tracking, and 4 by assessing their reasoning about the prioritization of the set of projects that they fund or are interested in (whether they are aware of alternatives and complementarity of their investments).
1. surveys make sense but why anonymous—you could get greater sincerity in a (‘safe’ group) conversation, perhaps. 2. yes, that can be convincing. You can maybe observe the EAG/EAGx data on ‘path to impact’ before and after posting and the time a user spent reviewing your post/whether they submitted any certificates. 3. Almost like a lab experiment …
Well, stating assumptions and continuing to improve methods can be the way .. Hm, not really, maybe like 126 for an RCT if you observe a greater than 50% difference (alter only E, the minimal difference in the impact metric that the research would detect).
Well, then the funders’ impact perception score that people competitively critique. That uses a network of human brains to assign appropriate weights to different factors.
Ask funders: Estimate the impact of our framework compared to its non-existence. Explain your reasoning. Then, summarize quotes and see what people critique.
But why would you weigh by the extent of profit-orientation? Does it matter the rational or apparent justification of the investment? For example, some for-profit fund managers interested in impact need to justify their actions to their clients by profit, so should not be perceived any negatively, or differently from those actually interested in profit, to support positive dynamics.
Oh yes, defs! :-)
Agreed. Preregistration also solves a lot of problems with our version of p-hacking (i-hacking ^.^).
I think we’d need more assumptions for this to happen. When it comes to business ventures, some are better funded from investments and some from loans. But if we imagine a world without investments, with only loans, and then offer investments, it’s not straightforward to me that that would harm the businesses that are better suited for loans. Similarly, I’d think (at first approximation at least) that the projects that are not suited for retro funding will still apply for and receive prospective funding at almost the same rate that it is available today. The bottleneck is currently more the investment of funding than the availability of funding.
Right, those are multiplicative too… That makes them a bit not-robust (volatile?). Would it also be valid to conceive of each of those as a multiplier on the current counterfactual impact, e.g., 10x increase in seed funding, 10x improvement in the allocation of seed funding, 10x through economies of scale, etc.? Here’s a sample Guesstimate. But it feel to me like we’re much too likely to err in upward direction this way even if it’s just an estimate of the success (non–black swan) case. Sort of like the multi-stage fallacy give too low estimates if you multiply many probabilities, so I feel like multiplying all these multipliers probably also ignores lots of bottlenecks.
But that said, almost all the factors are multiplicative here except for two components of the allocation – better allocation thanks to knowledge of more languages and cultures and thanks to being embedded in more communities. (I suppose a person is in about as many communities regardless which language/s they speak.) The language component may seem unusually low, which is because so much of the world speaks English and because the US are such a big part of the world economy.
I’m assuming that more entrepreneurial people will join the fray who are not sufficiently altruistic to already be motivated to start EA charities but who will do a good job if they can get at least somewhat rich from it.
Finally, I’m assuming that retro funders reinvest all their savings into the retro funding and priorities research, and that priorities research still has < 10x room for improvement, which seems modest to me (given evidential cooperation in large worlds for example).
I’d be interested what you think of this version – any other factors that need to be considered, or anything that should be additive instead of multiplicative? But we can discuss all that in our call if you like. No need to reply here.
Perhaps, but I’m worried that they may be incentivized to fib about it because it makes their impact seem more valuable and because it makes it more likely that we’ll continue to program (from which they want to benefit again), since we might discontinue it if we don’t think it’s sufficiently impactful.
Thanks! Still a lot. I’ve updated away from this being an important factor to measure… I’d rather find a way to compare the success rates of investors vs. current prospective funders. If none of them are consistently better, they may lose interest in the system.
Yeah, that sounds good in both cases. It’s probably not easy to talk to them but it would be valuable.
If we’re just reallocating EA money, we’re not adding more. A big source of the impact stems from attracting for-profit money.
Looking forward to our call! Feel free to just respond verbally then. No need to type it all out. :-D