If people think this is generally good idea I would guess CZEA can make it running in few weeks. Most of the work likely comes from curating the content, not from setting up the service
Jan_Kulveit
Karma: 1,638
To clarify—agree with the benefits of splitting the discussion threads for readability, but I was unenthusiastic about the motivation be voting.
I don’t think karma/voting system should be given that much attention or should be used as a highly visible feedback on project funding.
I don’t think anyone should be trying to persuade IMO participants to join the EA community, and I also don’t think giving them “much more directly EA content” is a good idea.
I would prefer Math Olympiad winners to think about long-term, think better, and think independently, than to “join the EA community”. HPMoR seems ok because it is not a book trying to convince you to join a community, but mostly a book about ways how to think, and a good read.
(If they readers eventually become EAs after reasoning independently, it’s likely good; if they for example come to the conclusion there are mayor flaws in EA and it’s better to engage with the movement critically, it’s also good.)
I don’t think risk of this type is given too much weight now. In my model, considerations like this got at some point in the past rounded of to some over-simplified meme like “do not start projects, they fail and it is dangerous”. This is wrong and led to some counterfactual value getting lost.
This was to some extent reaction to the previous mood, which was more like “bring in new people; seed groups; start projects; grow everything”. Which was also problematic.
In my view we are looking at something like pendulum swings, where we were somewhere at the extreme position of not many projects started recently, but the momentum is in direction of more projects, and the second derivative is high. So I expect many projects will actually get started. In such situation the important thing is to start good projects, and avoid anti-unicorns.
IMO the risk was maybe given too much weight before, but is given too little weight now, by many people. Just look at many of the recent discussions, where security mindset seem rare, and many want to move fast forward.
Discussing specific examples seems very tricky—I can probably come up with a list of maybe 10 projects or actions which come with large downside/risks, but I would expect listing them would not be that useful and can cause controversy.
Few hypothetical examples
influencing mayor international regulatory organisation in a way leading to creating some sort of “AI safety certification” in a situation where we don’t have the basic research yet, creating false sense of security/fake sense of understanding
creating a highly distorted version of effective altruism in a mayor country e.g. by bad public outreach
coordinating effective altruism community in a way which leads to increased tension and possibly splits in the community
producing and releasing some infohazard research
influencing important players in AI or AI safety in a harmful leveraged way, e.g. by bad strategic advice
How x-risk projects are different from startups
My impression is you have in mind something different than what was intended in the proposal.
What I imagined was ‘priming’ the argument-mappers with prompts like
Imagine this projects fails. How?
Imagine this project works, but has some unintended bad consequences. What they are?
What would be a strong reason not to associate this project with the EA movement?
(and the opposites). When writing their texts the two people would be communicating and looking at the arguments from both sides.
The hope is this would produce more complete argument map. One way to think about it, is each person is ‘responsible’ for the pro/con section, trying to make sure it captures as much important considerations as possible.
It seems quite natural for people to think about arguments in this way, with “sides” (sometimes even single authors expose complex arguments in the “dialogue” way).
There are possible benefits—related to why ‘debate’ style is used in justice
It levels the playing field in interesting ways (when compared to public debate on the forum). In the public debate, what “counts” is not just arguments, but also discussion and social skills, status of participants, moods and emotions of the audience, and similar factor. The proposed format would mean both the positives and negatives have “advocates” ideally of “similar debate strength” (anonymous volunteer). This is very different from a public forum discussion, where all kinds of “elephant in the brain” biases may influence participants and bias judgements.
It removes some of the social costs and pains associated with project discussions. Idea authors may get discouraged by negative feedback, downvotes/karma, or similar.
Also, just looking at how discussions on the forum look now, it seems in practice it is easy for people to look at things from positive or negative perspectives: certainly I have seen arguments structured like (several different ways how something fails + why is it too costly if it succeeded + speculation what harm it may cause anyway).
Overall: in my words, I’m not sure whether your view is ‘in the space of argument-mapping, noting in the vicinity of debate, will work—at least when done by humans and applied to real problems’. Or ‘there are options in this space which are bad’ - where I agree something like bullet-pointed lists of positives and negatives where the people writing them would not communicate seems bad.
My impression was based mostly on our conversations several months ago—quoting the notes from that time
lot of the discussion and debate derives from differing assumptions held by the participants regarding the potential for bad/risky projects: Benjamin/Brendon generally point out the lack of data/signal in this area and believe launching an open project platform could provide data to reduce uncertainty, whereas Jan is more conservative and prioritizes creating a rigorous curation and evaluation system for new projects.
I think it is fair to say you expected very low risk from creating an open platform where people would just post projects and seek volunteers and funding, while I expected with minimum curation this creates significant risk (even if the risk is coming from small fraction of projects). Sorry if I rounded off suggestions like “let’s make an open platform without careful evaluation and see” and “based on the project ideas lists which existed several years ago the amount of harmful projects seems low” to “worrying about them is premature”.
Reading your recent comment, it seems more careful, and pointing out large negative outcomes are more of a problem with x-risk/long-term oriented projects.
In our old discussions I also expressed some doubt about your or altruism.vc ability to evaluate x-risk and similar projects, where your recent post states that projects that impact x-risks by doing something like AI safety research has not yet applied to the EA Angel Group.
I guess part of the disagreement comes from the fact that I have focus on x-risk and the long-term future, and I’m more interested both in improving the project landscape in these areas, and more worried about negative outcomes.
If open platforms or similar evaluation process also accept mitigating x-risk and similar proposals, in my opinion, unfortunately the bar how good/expert driven evaluations you need is higher, and unfortunately signals like “this is a competent team” which VCs would mainly look at are not enough.
Because I would expect the long-term impact will come mainly from long-term, meta-, exploratory or very ambitious projects, I think you can be basically right about low obvious risk of all the projects historically posted on hackpad or proposed to altruism.vc, and still miss the largest term in the EV.
Milan asked this question and I answered it.
Thanks—both of that happened after I posted my comment, and also I still do not see the numbers which would help me estimate the ratio of projects which applied and which got funded. I take as mildly negative signal that someone had to ask, and this info was not included in the post, which solicits project proposals and volunteer work.
In my model it seems possible you have something like chicken-and-egg problem, not getting many great proposals, and the group of unnamed angels not funding many proposals coming via that pipeline.
If this is the case and the actual number of successfully funded projects is low, I think it is necessary to state this clearly before inviting people to work on proposals. My vague impression was we may disagree on this, which seems to indicate some quite deep disagreement about how funders should treat projects.
I’m not entirely sure what your reasons are for having this opinion, or what you even mean
The whole context was, Ryan suggested I should have sought some feedback from you. I actually did that, and your co-founder noted that he will try to write the feedback on this today or tomorrow, on 11th of Mar—which did not happen. I don’t think this is large problem, as we had already discussed the topic extensively.
When writing it I was somewhat upset about the mode of conversation where critics do ask whether I tried to coordinate with someone, but just assume I did not. I apologize for the bad way it was written.
Overall my summary is we probably still disagree in many assumptions, we did invest some effort trying to overcome them, it seems difficult for us to reach some consensus, but this should not stop us trying to move forward.
Summary impressions so far: object-level
It seems many would much prefer expediency in median project cases to robustness and safety in rare low frequency possibly large negative impact cases. I do not think this is the right approach, when the intention is also to evaluate long-term oriented, x-risk, meta-, cause-X, or highly ambitious projects.
I’m afraid there is some confusion about project failure modes. I’m more worried about projects which would be successful in having a team, working successfully in some sense, changing the world, but achieving large negative impact in the end.
I feel sad about the repeated claims the proposal is rigid, costly or large-scale. If something would not work in practice it could be easily changed. Spending something like 5h of time on a project idea which likely was result of much longer deliberation and which may lead to thousands hours of work seems reasonable. Paradoxically, just the discussion about whether the project is costly or not likely already had higher cost than what setting the whole proposed infrastructure for the project + phases 1a,1d,1c would cost.
Meta:
I will .not have time to participate in the discussion in next few days. Thanks for the comments so far.
Thanks Sundanshu! Sorry for not replying sooner, I was a bit overwhelmed by some of the negative feedback in the comments.
I don’t think step 1b. has the same bottleneck as current grant evaluator face, because it is less dependent on good judgement.
With your proposal, I think part of it may work, I would be worried about other parts. With step 2b I would fear nobody would feel responsible for producing the content.
With 3a or any automatic steps like that, what does that lack is some sort of (reasonably) trusted expert judgement. In my view this is actually the most critical step in case of x-risk, long-term, meta-, and similarly difficult to evaluate proposals.
Overall
I’m sceptical the karma or similar automated system is good for tracking what is actually important here
I see some beauty in automation, but I don’t see it applied here in the right places
FWIW, part of my motivation for the design, was
1. there may be projects, mostly in long-term, x-risk, meta- and outreach spaces, which are very negative, but not in an obvious way
2. there may be ideas, mostly in long-term and x-risk, which are infohazard
The problem with 1. is most of the EV can be caused by just one project, with large negative impact, where the downside is not obvious to notice.
It seems to me standard startup thinking does not apply here, because startups generally can not go way bellow zero.
I also do not trust arbitrary set of forum users to handle this well.
Overall I believe the very lightweight unstructured processes are trading some gain in speed and convenience in most cases for some decreased robustness in worst cases.
In general I would feel much better if the simple system you want to try would avoid projects in long-term, x-risk, meta-, outreach, localization, and “searching for cause X” areas.
It is possible my reading of your post somewhat blended with some other parts of the discussion, which are in my opinion quite uncharitable reading of the proposal. Sorry for that.
Actually from the list, I talked about it and shared the draft with people working on EA grants, EA funds, and Brendon, and historically I had some interactions with BERI. What I learned is people have different priors over existence of bad projects, ratio of good projects, number of projects which should or should not get funded. Also opinions of some of the funders are at odds with opinions of some people I trust more than the funders.
I don’t know, but it seems to me you are either a bit underestimating the amount of consultation which went into this, or overestimating how much agreement is there between the stakeholders. Also I’m trying to factor in the interests of the project founders, and overall I’m more concerned whether the impact in the world would be good, and what’s good for the whole system.
Despite repeated claims the proposal is very heavy, complex, rigid, etc. I think the proposed project would be in fact quite cheap, lean, and flexible (and would work). I’m also quite flexible in modifying it in any direction which seems consensual.
You are missing one major category here: projects which are simply bad because they do have approximately zero impact, but aren’t particularly risky. I think this category is the largest of the the four.
I agree that’s likely. Please take the first paragraphs more as motivation than precise description of the categories.
Which projects have a chance of working and which don’t is often pretty clear to people who have experience evaluating projects quite quickly (which is why Oli suggested 15min for the initial investigation above).
I think we are comparing apples and oranges. As far as the output should be some publicly understandable reasoning behind the judgement, I don’t think this is doable in 15m.
It sounds to me a bit like your model of ideas which get proposed is that most of them are pretty valuable. I don’t think this is the case.
I don’t have strong prior on that.
To do this well they need to have a good mental map of what kind of projects have worked or not worked in the past,...
From a project-management perspective, yes, but with slow and bad feedback loops in long-term, x-risk and meta oriented projects, I don’t think it is easy to tell what works and what does not. (Even with projects working in the sense they run smoothly and are producing some visible output.)
I’m not sure if we agree or disagree, possibly we partially agree, partially disagree. In case of negative feedback, I think as a funder, you are in greater risk of people over-updating in the direction “I should stop trying”.
I agree friends and social neighbourhood may be too positive (that’s why the proposed initial reviews are anonymous, and one of the reviewers is supposed to be negative).
When funders give general opinions on what should or should not get started or how you value or not value things, again, I think you are at greater risk of having too much of an influence on the community. I do not believe the knowledge of the funders is strictly better than the knowledge of grant applicants.
On a meta-level
I’m happy to update the proposal to reflect some of the sentiments. Openly, I find some of them quite strange—e.g. it seems, coalescing the steps into one paragraph and assuming all the results (reviews, discussion, “authoritative” summary of the discussion) will just happen may make it look more flexible. Ok, why not.
Also it seems you and Oli seem to be worried that I want to recruit people who are currently not doing some high-impact direct work … instead of just asking a couple of people around me, which would often mean people already doing impactful volunteer work.
Meta-point is, I’m not sure if you or Oli realize how big part of solving
new EA projects evaluation pipeline
is in consensus-building. Actually I think the landscape of possible ways how to do evaluations looks like in such a way that it is very hard to get consensus on what the “strongest form” is. I’m quite happy to create a bunch of proposals, e.g.
with removing final expert evaluation
removing initial reviews
removing public forum discussions
writing an unrealistic assumption that the initial reviews will take 15m instead of hours,
suggesting that the volunteers will be my busy friends (whose voluntary work does not count?)
emphasising public feedback more, or less
giving stronger or weaker voice to existing funders.
I have stronger preference for the platform to happen than for one option in any single of these choices. But what is the next step? After thinking about the landscape for a some time I’m quite skeptical any particular combination of options would not have some large drawback.
On the object level:
Re: funder involvement
Cross-posting from another thread
Another possible point of discussion is whether the evaluation system would work better if it was tied to some source of funding. My general intuition is this would create more complex incentives, but generally I don’t know and I’m looking for comments.
I think it much harder to give open feedback if it is closely tied with funding. Feedback from funders can easily have too much influence on people, and should be very careful and nuanced, as it comes from some position of power. I would expect adding financial incentives can easily be detrimental for the process. (For self-referential example, just look on this discussion: do you think the fact that Oli dislikes my proposal and suggest LTF can back something different with $20k will not create at least some unconscious incentives?)
We had some discussion with Brendon, and I think his opinion can be rounded to “there are almost no bad projects, so to worry about them is premature”. I disagree with that. Also, given the Brendon’s angel group is working, evaluating and funding projects since October, I would be curious what projects were funded, what was the total amount of funding allocated, how many applications they got.
Based on what I know I’m unconvinced that Brendon or BERI should have some outsized influence how evaluations should be done; part of the point of the platform would be to serve broader community.
It is very easy to replace this stage with e.g. just two reviews.
Some of the arguments for the contradictory version
the point of this stage is not to produce EV estimate, but to map the space of costs, benefits, and considerations
it is easier to be biased in a defined way than unbiased
it removes part of the problem with social incentives
Some arguments against it are
such adversarial setups for truth-seeking are uncommon outside of judicial process
it may contribute to unnecessary polarization
the splitting may feel unnatural
I don’t see why continuous coordination of a team of about 6 people on slack would be very rigid, or why people would have very narrow responsibilities.
For the panel, having some defined meeting and evaluating several projects at once seems time and energy conserving, especially when compared to the same set of people watching the forum often, being manipulated by karma, being in a way forced to reply to many bad comments, etc.
On the contrary: on slack, it is relatively easy to see the upper bound of attention spent. On the forum, you should look not on just the time spent to write comments, but also on the time and attention of people not posting. I would be quite interested how much time for example CEA+FHI+GPI employees spend reading the forum, in aggregate (I guess you can technically count this.)
I would expect if organizations working in the area have reviews of expected technologies and how they enable individuals to manufacture pathogens, which is likely the background necessary for constructing timelines, they would not publish too specific documents.