Donald Hobson

Karma: 44

Donald Hobson Feb 17, 2023, 7:53 PM
1 point
0 ∶ 0
on: “Develop Anthropomorphic AGI to Save Humanity from Itself” (Future Fund AI Worldview Prize submission)
The problem with “anthropomorphic AI” approaches is
1. The human mind is complicated and poorly understood.
2. Safety degrades fast with respect to errors.
Lets say you are fairly successful. You produce an AI that is really really close to the human mind in the space of all possible minds. A mind that wouldn’t be particularly out of place at a mental institution. They can produce paranoid ravings about the shapeshifting lizard conspiracy millions of times faster than any biological human.
Ok, so you make them a bit smarter. The paranoid conspiricies get more complicated and somewhat more plausible. But at some points, they are sane enough to attempt AI research and produce useful results. Their alignment plan is totally insane.
In order to be useful, anthropomorphic AI needs to not only make AI that thinks similarly to humans. They need to be able to target the more rational, smart and ethical portion of mind space.
Humans can chuck the odd insane person out of the AI labs. Sane people are more common and tend to think faster. A team of humans can stop any one of their number crowning themselves as world king.
In reality, I think your anthropomorphic AI approach gets an arguably kind of humanlike in some ways AI that takes over the world. Because it didn’t resemble the right parts of the right humans in the right ways closely enough in the places where it matters.

Donald Hobson Aug 29, 2022, 12:29 AM
1 point
0 ∶ 0
on: Nuclear Fine-Tuning: How Many Worlds Have Been Destroyed?
If that B52 had exploded, the death toll would probably have been smaller than that of the Hiroshima bomb. (It landed in a random bit of countryside, not a city). Unless you think that accident would have triggered all out nuclear war? Sure, it would have a large impact on American politics. Quite possibly lead to total nuclear disarmament, at least of America. But any anthropic or fine tuning argument doesn’t apply.
Suppose we were seeing lots of “near misses”. These were near misses of events that, had they happened, would have destroyed a random american town. Clearly this isn’t anthropic effects or similar. I would guess something about nuclear safety engineers being more/less competent in various ways. Or nuclear disarmament supporters in high places that want lots of near miss scares. Or the bombs are mostly duds, but the government doesn’t want to admit this.

Donald Hobson Aug 15, 2022, 4:06 PM
1 point
0 ∶ 0
on: Prioritizing x-risks may require caring about future people
Lets imagine a toy model, where we only care about current people. It’s still possible for most of the potential qualies to be far into the future. (If either lifetimes are very long, or quality is very high, maybe both.) So in this model, almost all the qualies come from currently existing people living for trillion years in an FAI utopia.
So lets suppose there are 2 causes, AI safety, and preventing nuclear war. Nuclear war will happen unless prevented, and will lead to half as many current people reaching ASI. (Either it kills them directly, or it delays the development of ASI) Let the QALY’s of (No nukes, FAI) be X.
Case 1) Currently P(FAI)=0.99, And AI safety research will increase that to P(FAI)=1. If we work on AI safety, a nuclear war happens, half the people survive to ASI, and we get U=0.5X. If we work on preventing nuclear war, the AI is probably friendly anyway, so U=0.99X
Case 2) Currently P(FAI)=0, but AI safety research can increase that to P(FAI)=0.01. Then if we prevent nuclear war, we get practically 0 utility, and if we work on AI safety, we get 0.005 utility, a 1% chance of the 50% of survivors living in a post FAI utopia.
Of course, this is assuming all utility ultimately resides in post ASI utopia, as well as not caring about future people. If you put a substantial fraction of utility on pre ASI world states, then the calculation is different. (Either by being really pesimistic about the chance of alignment, or by applying some form of time discounting to not care too much about the far future of existing people either. )

Donald Hobson Jul 29, 2022, 5:53 PM
1 point
0 ∶ 0
in reply to: brb243’s comment on: Does the idea of AGI that benevolently control us appeal to EA folks?
I think sentience is purely computational, it doesn’t matter what the substrate is. Suppose you are asleep. I toss a coin, heads I upload your mind into a highly realistic virtual copy of your room. Tails I leave you alone. Now I offer you some buttons that switch the paths of various trolleys in various real world trolley problems. (With a dependency on the coin flip) So if you are real, pressing the red button gains 2 util, if you are virtual, pressing costs 3 util. As you must (by the assumption the simulation is accurate) make the same decisions in reality and virtuality, then to get max util, you must act as if you are uncertain.
“I have no idea if I’m currently sentient or not” is a very odd thing to say.
Maybe it is chemical structure. Maybe a test tube full of just dopamine and nothing else is everso happy as it sits forgotten in the back shelves of a chemistry lab. Isn’t it convenient the sentient chemicals are full of carbon and get on well with human biochemistry. Like what if all the sentient chemical structures contained americium. No one would be sentient until the nuclear age, and people could make themselves a tiny bit sentient at the cost of radiation poisoning.
“But, humans could suffer digital viruses, which could be perhaps worse than the biological ones.” Its possible for the hardware to get a virus, like some modern piece of malware, that just happens to be operating on a computer running a digital mind. Its possible for nasty memes to spread. But in this context we are positing a superintelligent AI doing the security, so neither of those will happen.
Fixing digital minds is easier than fixing chemical minds, for roughly the reason fixing digital photos is easier than fixing chemical ones. With chemical photos, often you have a clear idea what you want to do, just make this area lighter, yet doing it is difficult. With chemical minds, sometimes you have a clear idea what you want to do, just reduce the level of this neurotransmitter, yet doing it is hard.
“But then, how would you differentiate a digital virus from an interaction, if both would change some aspects of the code or parameters?” If those words describe a meaningful difference, then there must be some way to tell. We are positing a superintelligence with total access to every bit flipped, so yes it can tell. “how can you tell between pictures of cats and dogs when they are both just grids of variously coloured pixels?”
“Ah, I think there is nothing beyond ‘healthy.’ Once one is unaffected by external and internal biological matters, they are healthy.”
Sure. But did you define healthy in a way that agrees with this. And wouldn’t mind uploading reduce the chance of getting cancer in the future. The AI has no reason not to apply whatever extreme tech it can to reduce the chance of you ever getting ill by another 0.0001%

Donald Hobson Jul 20, 2022, 12:23 AM
1 point
0 ∶ 0
in reply to: brb243’s comment on: Does the idea of AGI that benevolently control us appeal to EA folks?
I don’t think the human difficulty imagining what super-healthy is is the reason the AI needs nanobots. A person who is say bulletproof is easy to imagine, and probably not achievable with just good nutrition, but is achievable with nanobots. The same goes for biology that is virus proof, cancer proof etc.
I can imagine mind uploading quite easily.
There may be some “super-healthy” so weird and extreme that I can’t imagine it. But there is already a bunch of weird extreme stuff I can imagine.

Donald Hobson Jul 17, 2022, 5:13 PM
1 point
0 ∶ 0
in reply to: Austin’s comment on: Retroactive funding with subsidy markets
In a competitive market, the investors make a tiny amount of profit. Suppose widgets cost $100 to make. All widget buyers always choose the cheapest widget. If you start selling them for $200, someone else can undercut you by selling them for $180. The only equilibrium is selling widgets for $102 or something. Just enough to make a slim profit, but not enough for anyone to try undercutting you. Of course, if you and you alone have a way to magic widgets out of nothing, then you can make fat profits. This is roughly how a lot of markets work. Its why there are commodity prices, and those making commodities have slim profit margins.
Note that it is easy for investors to loose money by being stupid. And they can potentially turn a large profit if they are a unique source of exceptionally good info. They just can’t turn a large profit in a large pool of similarly competent investors.
The investors have skin in the game, and as long as the investors maximize money, the system should work. The problem is that if people are prepared to burn their own money, they can burn other money with it. (Ultimately meaning some charities get less funding if the silly money gets added in the wrong place than if the silly money had just directly burned their cash at home)

Donald Hobson Jul 17, 2022, 12:58 PM
1 point
0 ∶ 0
in reply to: brb243’s comment on: Does the idea of AGI that benevolently control us appeal to EA folks?
I’d not like to ingest nanobots which would be something like a worm infection but worse!
For a huge range of goals, the optimum answer involves some kind of nanobot. (Unless even deeper magic tech exists). If you want a person to be healthy, the nanobots can make them healthier than any good nutrition.
The idea I was getting at is that asking an AI for better nutrition, meant the way you mean it, is greatly limiting the options for what you actually want. Suppose you walk a lot to get to places, and your shoes are falling apart. You ask the AI for new shoes, when it could have given you a fancy car. By limiting the AI to “choice of food” rather than “choice of every arrangement of atoms allowed by physics” you are greatly reducing the amount the AI can optimize your health.

Donald Hobson Jul 17, 2022, 11:41 AM
3 points
0 ∶ 0
in reply to: Noah Scales’s comment on: Does the idea of AGI that benevolently control us appeal to EA folks?
You know, as far as seeing ourselves on a path to doom, I don’t see why development of a superintelligent rogue AI isn’t treated like development of a superweapon.
Because distinguishing it from benign, not superintelligent AI is really hard.
So you are the FBI, you have a big computer running some code. You can’t tell if its a rouge superintelligence or the next DALL-E by looking at the outputs. A rouge superintelligence will trick you until its too late. Once its run at all on a computer that isn’t in a sandboxed bunker its probably too late. So you have to notice people writing code, and read that code before its run. There are many smart people writing code all the time. That code is often illegible spaghetti. Maybe the person writing the code will know, or at least suspect, that it might be a rouge superintelligence. Maybe not.
Lots of computer scientists are in practice rushing to develop self driving cars, the next GPT. All sorts of AI services. The economic incentive is strong.

Donald Hobson Jul 17, 2022, 11:08 AM
1 point
0 ∶ 0
in reply to: Austin’s comment on: Retroactive funding with subsidy markets
1)The subsidy market is a buffer of money, not a long term source. R should be adjusted so the amount of money flowing in and flowing out are the same.
3) I was trying to generalize a Vickrey auction, but I think I messed that up. Just use whatever kind of auction you feel like.
5) If the amount raised is less than the cost bar, the project can try to muddle through with the money they have, or can refund the investors/ cancel the thing.
7) Yes, and scaled down to reverse the scaling up.
None of the examples illustrate the investors making positive returns. The scheme is delibirately set up, so that in the limit of ideal markets, the investors make nothing. In practice the investors would probably make something, but hopefully not much. Well the investors sometimes win, but its counterbalanced by a loss.
This scheme should pay any project that produces at least $\frac{1}{R}$ util/$, and not pay any other projects, always selecting the highest util/$ variant if there are multiple variants. If we assume project managers are paid a fixed fair salary, there is nothing to threaten them with, unless you want to threaten to cut off funding to an effective charity, because you think it could be even more effective.
This lack of incentive (so long as your util/$ is above some threshold) is what would happen in a regular retroactive system, where all shares are sold, or where you didn’t in practice care how much “credit” you ended up with. (Because you couldn’t turn that credit into money)
Giving the project manager 1% of the shares would produce some incentive in the right direction. Or you could just hope they are all altruists.
I was kind of thinking like the subsidy market more as a currency exchange.
You could replace the scaled down and scaled up money with 2 different kinds of crypto token.
The whole point of the subsidy market is to move money from the easy wins with loads of money, to the marginal cases that are just about worth doing, while leaving the no hopers behind.

Donald Hobson Jul 16, 2022, 8:26 PM
1 point
0 ∶ 0
on: Does the idea of AGI that benevolently control us appeal to EA folks?
Well if you have it, I’ll take it. In the general scenario, a very powerful benevolent AI is left to do whatever it thinks is best. If the AI decides that freedom is one of humans top values, it will try to make the world better while optimizing human freedom. Giving humans more freedom in practice than the typical government is not a particularly high bar. Of course, plenty of people might want the AI micromanaging every detail of their life, the AI will do a really good job of it. But I would think ideally freedom should be there for those who want it.
Its also worth noting there is a fairly common belief that we are on a path to probable doom, and any AI that offered anything better than paperclips is work taking. So, even if your AI was much too controlling and humans would prefer a less controlling one, many EA would say ” best AI we are going to get”.

Donald Hobson Jul 16, 2022, 8:18 PM
5 points
0 ∶ 0
in reply to: brb243’s comment on: Does the idea of AGI that benevolently control us appeal to EA folks?
AGI optimizing nutrition: Ok, first on the menu is mind uploading nanobots.
Human: Err.
AI: You want me to optimize nutrition right? In other words, you want whichever configuration of atoms will give you the longest healthiest life.
Human: How many calories are in that?
AI: Billions, its powered by antimatter.

Donald Hobson Jul 16, 2022, 8:11 PM
5 points
0 ∶ 0
in reply to: david_reinstein’s comment on: Confused about “making people happy” vs. “making happy people”
The limited advocacy for pronatalist policies may also be evidence against it.
In a context where we can have a cosmically vast future, if we avoid X-risk today, advocating for a few more people on earth tomorrow is totally missing the point. Pronatalism only makes sense when thinking total utilitarian but completely forgetting anything longtermist. If you were a total utilitarian, but had “here be dragons” on your map of the long term future, then pronatalism makes sense.

Donald Hobson Jul 16, 2022, 8:07 PM
2 points
0 ∶ 0
on: Confused about “making people happy” vs. “making happy people”
When a range of different happiness levels of future people are available, it is hard to have a time consistent preferences that doesn’t consist of “making happy people”, or “stopping miserable people being made”.
Suppose you are presented with these 3 options.
1. Alice is born and has happiness level A
2. Alice is born and has happiness level B
3. Alice is not born.
In order to be indifferent between all 3 options and have time consistent preferences, you must be indifferent towards
1. Alice has happiness level A
2. Alice has happiness level B
Its possible to have some level of happiness L, and be indifferent to people with levels >L existing, but against people having levels lower than L. This does mean a form of negative utilitarianism that would have killed the first primitive life given the chance.
Its also allowed to say, my preferences aren’t time consistent. I can be money pumped. Every time a baby is born, my preferences shift from not caring about that baby to caring about that baby.
This means you should want to sign a binding oath saying you won’t go out of your way to help anyone who hadn’t been born at the time of signing the oath. (At the time of signing, you by hypothesis don’t care about them. You want to stop your future self wasting resources on someone you currently don’t care about.)
Or maybe you decide you are a human, not an AI. Any clear utility function you write down will be goodheartable. You will just wing it based on intuition and hope.

Retroactive funding with subsidy markets

Donald HobsonJul 16, 2022, 7:41 PM

16 points

4 comments7 min readEA link

Donald Hobson Apr 6, 2022, 10:16 AM
3 points
0 ∶ 0
on: Ultra-Near-Termism: Literally An Idea Whose Time Has Come
Ultra near termism doesn’t mean stopping AI. The ultra near termist would really really like a FAI FOOM that simulates a billion subjective years of utopia in the next few days.
Of course, they would choose a 0.1% chance of this FAI FOOM now, as opposed to a 100% chance of FAI FOOM in 2 months.
If a FAI with nanotech could make simulations that are X times faster, and Y times nicer than the status quo, you should encourage AI so long as you think it has at least 1 in XY chance of being friendly.
One thing that potentially breaks ultra-near-termism is the possibility of timetravel. If you have a consistent discount rate over all time, this implies a goal of taking a block of edumonium back in time to the big bang. With pretty much any other goal having negligible utility in comparison. If you consider the past to have no value, then the goal would be to send a block of edumonium back in time to now.

Donald Hobson Dec 30, 2021, 9:18 PM
10 points
0 ∶ 0
on: The Unweaving of a Beautiful Thing
The time of a single witch or wizard trying to snare death in their dying spell was over a decade ago. Such techniques could only be used by a skilled witch or wizard upon their deaths, and such events were hard to plan for ethically.
Now a team of thaumotological engineers laboured over a contrivance of electronics and crystals. This device would be slid under a bed of a terminally ill patient. Slotted into the centre were a handful of cells in a dish, taken from the dying patient themselves.Theoretical research had been done, software had been written hundreds of different strategies and counter strategies had been tested in simulations. Death was far faster than any human, but the right engineering could be faster than death.
If you had walked through the corridor of the Royal Infirmary, you would have seen very little out of place compared to a normal hospital. One dying patient had slightly more sensors on their body than usual, the wires running under the bed. Small puffs of cool cloud flowed out one side of the bed as nitrogen gas was vented from the cooling system.
Death approached the bed. They felt a brief disorienting pulse of magic, just enough to throw them off balance for a few seconds. Shaking his head at the crudeness of the supposed trap, death claimed another soul.
Half an hour later, a man arrived at the hospital. He slid the box out from under the bed. Opening the top, he observed the contents. Most of the box was full of slag, solidified but still hot. In one insulated corner is an SSD chip full of encrypted data. The password, 100KB of quantum randomness unique to this particular box and stored offsite. He plugs the SSD chip into his laptop and uploads the data back to the main control room. They have significantly improved their sensors from last time and have several gigs of new scans and telemetry.

Donald Hobson Dec 20, 2021, 1:37 AM
3 points
0 ∶ 0
on: Measuring the “apocalyptic residual”
Another key factor is levels of competence to achieve their objective. The people trying to wake chuthulu by chanting magic words aren’t a problem. The doomsday cult doing cutting edge biotech or AI research is a problem. How many rationality points do these doomsday cultists have?

Donald Hobson Dec 20, 2021, 1:30 AM
1 point
0 ∶ 0
on: Nines of safety: Terence Tao’s proposed unit of measurement of risk
Nines of unsafety, for the pessimists. So 2 9′s of unsafety is a 99% chance of doom.

Donald Hobson Dec 11, 2021, 12:59 AM
1 point
0 ∶ 0
on: What “defense layers” should governments, AI labs, and businesses use to prevent catastrophic AI failures?
The boring answers
Don’t give your AI system excess compute. Like ideally on a hardware level. Run it on a small isolated machine not a 0.1% timeshare on a supercomputer.
Use the coding practices developed by Nasa to minimize standard bugs.
Record all random seeds and input data to make everything reproducible.
Put in hard coded sanity checks between AI and output. A robot arm isn’t allowed to move beyond safe limits by a simple max(AI_OUTPUT, MAXIMUM_ARM_ANGLE) type code.
Humans checking in the loop.
Hardware minimization of unneeded action space. Suppose you are programming a social media bot. The AI is quite smart. If you have 1 computer running the AI that has a network connection, and it rowhammers, it can send arbitrary internet packages. Computer A runs the AI and sends raw ascii down a wire to computer B. Computer B checks every simple check that you would expect to be true in normal operation, and performs a hard shutdown if it spots anything odd. (Ie funny characters, post too long or short. ) Computer A of course has no other connections, has a power supply resistant to modulated energy demand attacks, has a radio and audio noise generator next to it and is in a Farady cage.
2 or 3 redundant computers (cosmic ray errors)
Thermite on a timer. Slag the whole computer system. Stops the AI thinking too long. Makes it harder to hide info for future versions of itself.
The speculative answers.
Holomorphic encryption.
Run it on a quantum computer using amplitude amplification techniques. Any breaking out of the box technique must be reversible to work, or use new physics.
Honeypots that look like good hacking targets.
A fairly easy way for any moderately smart AI to wirehead, setting its own utility to maximum. (and triggering some alarm/ delayed shutdown.)
The silly answers
Chuck the AI system into a black hole, set to only start computing after they cross the event horizon.

Donald Hobson Jul 25, 2020, 10:21 PM
2 points
0 ∶ 0
in reply to: EdoArad’s comment on: Quotes about the long reflection
The rot13 is to make it harder to search for. I think that this is a discussion that would be easy to misinterpret as saying something offensive.

Donald Hobson

Retroac­tive fund­ing with sub­sidy markets

Retroactive funding with subsidy markets