I’m currently researching forecasting and epistemics as part of the Quantified Uncertainty Research Institute.
Ozzie Gooen(Ozzie Gooen)
Karma: 4,361
Same!
Happy to see more discussion on these topics.
Much of this is a part of what both some of the EA forecasting community, and what we at [QURI](https://quantifieduncertainty.org/), are working on.
I think the full thing is much more work than you think it is. I suggest trying to take one subpart of this problem and doing it very well, instead of taking the entire thing on at once.
Here’s my super quick take, if I were evaluating this for funding:
Startups are pretty competitive. For me to put money into a business venture, I’d want quite a bit of faith that the team is very strong. This would be pretty high bar.
From looking at this, it’s not clear to me promising the team is at this point.
Generally, the bar for many sorts of projects is fairly high.
A few quick things:
- I agree that many grantmakers don’t have enough time to give much feedback, and that this leads to suboptimal outcomes.
- I think it’s pretty difficult for people outside these organizations to help much with what are basically internal processes. People outside have very little context, so I would expect them to have a tough time suggesting many ideas.
- In this specific proposal, I think it would be tricky for it to help much. A lot of what I’ve seen (which isn’t all too much) around grant applications is about people sharing the negative information they have about applicants. I imagine this would be exceedingly awkward to show publicly.
If people want to help with the larger grantmaking process, some things they could do include:Advise groups requesting money. See if you could provide useful feedback (I think many groups could use a bigger team of advisors)
Help newish people to write more content on the EA Forum and similar. This can be a proving ground for some grant organizations.
This looks really interesting, will take me some time to get through.
Very minor thing: Many of the links (the titles) are broken.
I think this is neat.
Perhaps-minor note: if you’d do it at scale, I imagine you’d want something more sophisticated than coarse base rates. More like, “For a project that has these parameters, our model estimates that you have a 85% chance of failure.”
I of course see this as basically a bunch of estimation functions, but you get the idea.
No worries! I assumed as such.
Minor note about the name: “Never Again” is a slogan often associated with the Holocaust. I think that people using it for COVID might be taken as appropriation or similar. I might suggest a different name.
https://en.wikipedia.org/wiki/Never_again
One quick thought; often, when things are very grim, you’re pretty okay taking chances.
Imagine we need 500 units of AI progress in order to save the world. In expectation, we’d expect 100. Increasing our amount to 200 doesn’t help us, all that matters is if we can get over 500. In this case, we might want a lot of bifurcation. We’d much prefer a 1% chance of 501 units, than a 100% chance of 409 units, for example.
In this case, lots of randomness/bifurcation will increase total expected value (which is correlated with our chances of getting over 500 units, moreso than it is correlated with the expected units of progress).
I imagine this mainly works with discontinuities, like the function described above (Utility = 0 for units 0 to 499, and Utility = 1 for units of 500+)
It seems to me like quantum randomness can be a source of creating legitimate divergence of outcomes. Lets call this “bifurcation”. I could imagine some utility functions for which increasing the bifurcation of outcomes is beneficial. I have a harder time imagining situations where it’s negative.
I’d expect that interventions that cause more quantum bifurcation generally have other costs. Like, if I add some randomness to a decision, the decision quality is likely to decrease a bit on average.
So there’s a question of the trade-offs of a decrease in the mean outcome, vs. an increase in the amount of bifurcation.There’s a separate question on if EAs should actually desire bifurcation or not. I’m really unsure here, and would like to see more thinking on this aspect of it.
Separately, I’d note that I’m not sure how important it is if there’s already a ton of decisive quantum randomness happening. Even if there’s a lot of it, we might want even more.
My impression is that you’re arguing that quantum randomness creates very large differences between branches. However, couldn’t it still be the case that even more differences would be preferable? I’m not sure how much that first argument would impact the expected value of trying to create even more divergences.
So the incentives are not pointing in the right direction. Capable forecasters can earn significantly more by predicting societally-useless sports stuff, or simply by arbitraging between the big European sports-houses and crypto markets. Meanwhile, the people who remain forecasting socially useful stuff on Metaculus, like whether Russia will invade the Ukraine or whether there will be any new nuclear explosions in wartime, do so to a large extent out of the goodness of their heart.
I think that the clear solution to this is to either increase the overall willingness to pay forecasters, or to be willing to subsidize liquidity in prediction markets for questions that are of general value.
I think I don’t quite see it this way. I agree that more money directed towards forecasters is good. However, there are different lessons one could take here.
Altruistic jobs often pay less than very selfish ones. A big reason why is because many people care about altruistic roles more, and are willing to do them for much cheaper. The “selfish” companies essentially have to pay more for people to be willing to work for them.
Telsa and SpaceX both paid less, and had worse working conditions, than other silicon valley companies, for example. Nonprofits often pay even lower. (Some of this is unjustified, but my guess is that some of it just comes from higher supply of workers).
do so to a large extent out of the goodness of their heart
I feel like the phrase “goodness of their heart” implies that it’s unusual for people to do things out of altruism or meaning. But I think it’s very frequent. Basically all of the Wikipedia editors, and almost all of Open Source, is done due a mix of interest/altruism. Much of the funding would be due to altruism, so this really doesn’t seem much different.
This is an interesting idea, thanks for raising it!
I think intuitively, it worries me. As someone around hiring in these sorts of areas, I’m fairly nervous around the liabilities that come from hiring, and this seems like it could increase these. (Legal, and just upsetting people).
I’m imagining:There’s a person who thinks they’re great, but the hiring manager really doesn’t see it. They get rejected.
They decide to work on it anyway, saying they’ll get the money later.
They continue to email the org about their recent results, hoping to get feedback, sort of similar to as an employee.
6-20 months later, they have some work, and are sure that it deserves funding.
The work isn’t that great, and the prize is denied.
They get really upset that their work has been denied.
This system could create “pseudo-employees” who are trying to act as employees, but aren’t really employees. This just seems pretty messy.
In addition, funding seems tricky. Like, a lot of research nonprofits don’t have that much extra funding allocated in their budgets for this. I imagine it would have to be coordinated with funders, on-demand. (“Hey, funder X… person Y, who we rejected, just did good work, and now we need $160k to fund them. Can you donate that money to us, so we can retrospectively pay them?”)
I could also see the tax/legal implications as messy, though that could be resolved with time.
Generally, if someone seems pretty strong and capable of doing independent work, I suggest they apply to the LTFF, and say that I could help discuss their application. The LTFF funds a lot of people at this point. Small funders like the LTFF seem like great escape hatches for these situations. So this technique would really make sense, I assume, if both the LTFF rejects them, and I’m pretty confident they have a solid chance of doing good research. This is pretty unusual.
It’s quite possible the benefits overcome these negatives. I’m not sure, I just wanted to share my quick feelings on this.
That makes more sense, thanks!
Quick thoughts of possible improvements to the format:
1) Make both the start time and the end time clear
2) Include a link to the person’s website/linkedin. Right now I just search each person and choose whatever is on top of Google, anyway. Much of the grants really depend on the specific person, so linking to more information about the person would be valuable. (I realize this might be a bit annoying in terms of getting their buy-in/information)
Just to clarify:
That last report was for May 2021.
So does this report mainly cover grants for June and July, only?
The health interventions seem very different to me than the productivity interventions.
The health interventions have issues with long time-scales, which productivity interventions don’t have as much.
However, productivity interventions have major challenges with generality. When I’ve looked into studies around productivity interventions, often they’re done in highly constrained environments, or environments very different from mine, and I have very little clue what to really make of them. If the results are highly promising, I’m particularly skeptical, so it would take multiple strong studies to make the case.
I think it’s really telling that Google and Amazon don’t have internal testing teams to study productivity/management techniques in isolation. In practice, I just don’t think you learn that much, for the cost of it.
What these companies do do, is to allow different managers to try things out, survey them, and promote the seemingly best practices throughout. This happens very quickly. I’m sure we could make tools to make this process go much faster. (Better elicitation, better data collection of what already happens, lots of small estimates of impact to see what to focus more on, etc).
In general, I think traditional scientific experimentation on humans is very inefficient, and we should be aiming for much more efficient setups. (But we should be working on these!)
I think the obvious answer is that doing controlled trials in these areas is a whole lot of work/expense for the benefit.
Some things like health effects can take a long time to play out; maybe 10-50 years. And I wouldn’t expect the difference to be particularly amazing. (I’d be surprised if the average person could increase their productivity by more than ~20% with any of those)
On “challenge trials”; I imagine the big question is how difficult it would be to convince people to accept a very different lifestyle for a long time. I’m not sure if it’s called “challenge trial” in this case.
Interesting take, quick notes:
1) I worked on a similar model with Justin Shovelain a few years back. See: https://www.lesswrong.com/posts/BfKQGYJBwdHfik4Kd/fai-research-constraints-and-agi-side-effects
2) Rather, one’s impact is positive if the ratio of safety and capabilities contributions si/ci is greater than the average of the rest of the world.
I haven’t quite followed your model, but this doesn’t see exactly correct to me. For example, if the mean player is essentially “causing a lot of net-harm”, then “just causing a bit of net-harm”, clearly isn’t a net-good.