Founder of CEEALAR (née the EA Hotel; ceealar.org)
Greg_Colbourn(Greg Colbourn)
Karma: 1,987
CEEALAR is hiring for a full-time Community Manager, please share with anyone you think may be interested: https://ceealar.org/job-community-manager/
To start mid-late September. £31,286 – £35,457 per year (full time, 40 hours a week).
Just going to say that your donation to CEEALAR (EA Hotel) in late 2019 when we were facing running out of money was very much appreciated, and, I think, probably pretty high EV! (although obviously I’m biased :))
Something I wrote a little while back regarding whether EA should be a “narrow movement of people making significant impact, or a much broader one of shallower impact”:
I’ve sometimes wondered whether it would be good for there to be a distinct brand and movement for less hardcore EA, that is less concerned with prestige, less elitist, more relaxed, and with more mainstream appeal. Perhaps it could be thought of as the Championship to EA’s Premier League. I think there are already examples, e.g. Probably Good (alternative to 80,000 Hours), TLYCS and OFTW (alternatives to GWWC), and the different tiers of EA investing groups (rough and ready vs careful and considered). Places where you feel comfortable only spending 5 minutes editing a post, rather than agonising about it for hours; where you feel less pressure to compete with the best in the world; where you are less prone to analysis paralysis or perfect being the enemy of the good; where there is less stress, burnout and alienation; where ultimately the area under the impact curve could be comparable, or even bigger..? Perhaps one of the names mentioned here could be used.
CEEALAR is hiring for a full-time Operations Manager, please share with anyone you think may be interested: https://ceealar.org/job-operations-manager/ Blackpool, UK.
To start mid-late July. £31,286 – £35,457 per year (40 hours a week).
Thanks! Here is Table S7 (I’ve highlighted the relevant years):
He’s just one person, so I wouldn’t say that’s significant empirical evidence. Unless a bunch of other people they approached turned them down (and if they did, it would be interesting to know why).
Also, we can generalise “”Geniuses” with nice legible accomplishments” (e.g. Fields Medalists) to “people who a panel of top people in AGI Safety would have on their dream team”. Is there such a list? Who would you nominate to be on the dream team? A great first step would be a survey of top Alignment researchers on this question.
The way I see it is that we need to throw all we have at both AGI Alignment research and AGI governance. On the research front, getting the most capable people in the world working on the problem—a Manhattan Project for alignment as it were—seems like our best bet. (On the governance front, it would be something along the lines of global regulation of, or a moratorium on, AGI capabilities research. Or in the extreme, a full-on Butlerian Jihad. That seems harder.)
40. “Geniuses” with nice legible accomplishments in fields with tight feedback loops where it’s easy to determine which results are good or bad right away, and so validate that this person is a genius, are (a) people who might not be able to do equally great work away from tight feedback loops, (b) people who chose a field where their genius would be nicely legible even if that maybe wasn’t the place where humanity most needed a genius, and (c) probably don’t have the mysterious gears simply because they’re rare. You cannot just pay $5 million apiece to a bunch of legible geniuses from other fields and expect to get great alignment work out of them. They probably do not know where the real difficulties are, they probably do not understand what needs to be done, they cannot tell the difference between good and bad work, and the funders also can’t tell without me standing over their shoulders evaluating everything, which I do not have the physical stamina to do.
This may well be true, but I think we are well past the stage where this should at least be established by significant empirical evidence. We should at least try, whilst we have the opportunity. I would feel a lot better to be in a world where a serious attempt was made at this project (to the point where I’m willing to personally contribute a significant fraction of my own net worth toward it). More.
We found a solution to the bank card issue—using Wise (they offer Visa debit cards).
One bit of feedback about the grants database spreadsheet (.csv download) - I think it’s always been like it currently is, but it would be good if the amounts were formatted as numbers to make sorting and totalling up possible. Also, linking to a Google Sheet would be a nice bonus.
AlphaZero isn’t smart enough (algorithmically speaking). From Human Compatible (p.207):
Life for AlphaGo during the training period must be quite frustrating: the better it gets, the better its opponent gets—because its opponent is a near-exact copy of itself. Its win percentage hovers around 50 percent, no matter how good it becomes. If it were more intelligent—if it had a design closer to what one might expect of a human-level AI system—it would be able to fix this problem. This AlphaGo++ would not assume that the world is just the Go board, because that hypothesis leaves a lot of things unexplained. For example, it doesn’t explain what “physics” is supporting the operation of AlphaGo++’s own decisions or where the mysterious “opponent moves” are coming from. Just as we curious humans have gradually come to understand the workings of our cosmos, in a way that (to some extent) also explains the workings of our own minds, and just like the Oracle AI discussed in Chapter 6, AlphaGo++ will, by a process of experimentation, learn that there is more to the universe than the Go board. It will work out the laws of operation of the computer it runs on and of its own code, and it will realize that such a system cannot easily be explained without the existence of other entities in the universe. It will experiment with different patterns of stones on the board, wondering if those entities can interpret them. It will eventually communicate with those entities through a language of patterns and persuade them to reprogram its reward signal so that it always gets +1. The inevitable conclusion is that a sufficiently capable AlphaGo++ that is designed as a rewardsignal maximizer will wirehead.
From wireheading, it might then go on to resource grab to maximise the probability that it gets a +1 or maximise the number of +1s it’s getting (e.g. filling planet sized memory banks with 1s); although already it would have to have a lot of power over humans to be able to convince them to reprogram it by sending messages via the go board!
I don’t think the examples of humans (Bezos/Witten) are that relevant, in as much as we are products of evolution, and are “adaption executors” rather than “fitness maximisers”, are imperfectly rational, and tend to be (broadly speaking) aligned/human-compatible, by default.
Would it be fair to say that Triplebyte is a similar thing for the software engineering industry?
So by ‘by default’ I mean without any concerted effort to address existential risk from AI, or just following “business as usual” with AI development. Yes, Drexler’s CAIS would be an example of this. But I’d argue that “just don’t code AI as an unbounded optimiser” is very likely to fail due to mesa-optimisers and convergent instrumental goals emerging in sufficiently powerful systems.
Interesting you mention climate change, as I actually went from focusing on that pre-EA to now thinking that AGI is a much more severe, and more immediate, threat! (Although I also remain interested in other more “mundane” GCRs.)
So the last Caplan says there is:
“1′. AIs have a non-trivial chance of being dangerously un-nice.
I do find this plausible, though only because many governments will create un-nice AIs on purpose.”
Which to me sounds like he doesn’t really get it. Like he’s ignoring “by default does things we regard as harmful” (which he kind of agrees to above; he agrees with “2. Instrumental convergence”). You’re right in that the Orthogonality Thesis doesn’t carry the argument on it’s own, but in conjunction with Instrumental Convergence (and to be more complete, mesa-optimisation), I think it does.
It’s a shame that Caplan doesn’t reply to Yudkowsky’s follow up:Bryan, would you say that you’re not worried about 1′ because:
1’a: You don’t think a paperclip maximizer is un-nice enough to be dangerous, even if it’s smarter than us.
1’b: You don’t think a paperclip maximizer of around human intelligence is un-nice enough to be dangerous, and you don’t foresee paperclip maximizers becoming much smarter than humans.
1’c: You don’t think that AGIs as un-nice as a paperclip maximizer are probable, unless those durned governments create AGIs that un-nice on purpose.
Right, but I think “by default” is important here. Many more people seem to think alignment will happen by default (or at least something along the lines of us being able to muddle through, reasoning with the AI and convincing it to be good, or easily shutting it down if it’s not, or something), rather than the opposite.
The Orthogonality Thesis is useful to counter the common naive intuition that sufficiently intelligent AI will be benevolent by default (which a lot of smart people tend to hold prior to examining the arguments in any detail). But as Steven refers to above, it’s only one component of the argument for taking AGI x-risk seriously (and Yudkowsky lists several others in that example. He leads with orthogonality to prime the pump; to emphasise that common human intuitions aren’t useful here.).
Riffing on possible reasons to be hopeful, I recently compiled a list of potential “miracles” (including empirical “crucial considerations” [/wishful thinking]) that could mean the problem of AGI x-risk is bypassed:
Possibility of a failed (unaligned) takeoff scenario where the AI fails to model humans accurately enough (i.e. realise smart humans could detect its “hidden” activity in a certain way). [This may only set things back a few months to years; or could lead to some kind of Butlerian Jihad if there is a sufficiently bad (but ultimately recoverable) global catastrophe (and then much more time for Alignment the second time around?)].
Valence realism being true. Binding problem vs AGI Alignment.
Omega experiencing every possible consciousness and picking the best? [Could still lead to x-risk in terms of a Hedonium Shockwave].
Moral Realism being true (and the AI discovering it and the true morality being human-compatible).
Natural abstractions leading to Alignment by Default?
Rohin’s links here.
AGI discovers new physics and exits to another dimension (like the creatures in Greg Egan’s Crystal Nights).
Simulation/anthropics stuff.
Alien Information Theory being true!? (And the aliens having solved alignment).
I don’t think I put more than 10% probability on them collectively though, and my P(doom) is high enough to consider it “crunch time”.
CEEALAR (aka the EA Hotel) is hiring for a full-time Community Manager.
Location: Blackpool, UK. To start mid-late September. Deadline to apply is Aug 21st (end of the day in your own timezone).
We are looking for a personable, empathetic, sociable, highly organised and conscientious person with high agency, or someone with a strong commitment to develop and embody those characteristics, to take on the challenge of managing our community.
Responsibilities:
Answering email enquiries;
Coordinating with Trustees and Advisors to vet applicants;
Interviewing applicants via video call;
Grantee onboarding;
Maintain community morale at a high level, and resolve conflict if it arises;
Organise and host regular group meetings (e.g. a weekly Friday night meeting to discuss logistics, talks, events and any other business), in coordination with the Operations Manager;
Admin the CEEALAR Discord server;
Public communication (e.g. keeping the website up to date, posting updates to social media and the EA Forum);
Working with the other managers to ensure that all tasks have someone taking responsibility for them, and none fall between the cracks;
Covering holidays for other staff (food/inventory management, cooking, admin), and ensuring your own holidays are covered;
Ad hoc tasks as required (can be outsourced), including back end administration and organisation. We are a “start-up” charity, with few staff and many varied tasks that need doing. Part of the job will be prioritising among, and figuring out how to get done, many tasks that aren’t on this list. We want you to be enthusiastic about the organisation thriving, and do what it takes for this to happen, in collaboration with our Operations Manager and Trustees.
More info here.