I might do AI safety or ML after undergrad
sammyboiz🔸
Karma: 432
You say that an LLM would optimize towards a reward function that not reasonable or balanced by humans.
But human preference IS what a model optimizes for, how do you reconcile this?
For example, Construct a scenario or give a moral thought experiment in which you believe GPT acts in an unbalanced way. Could you find one? If so, can this not be solved with more and better RLHF?
You say that there is a gap between how the model professes it will act and how it will actually act. However, a model trained to obey the RLHF objective will expect negative reward if decided taking over the world, so why would it? Saying that a model will make harmful, unhelpful choices is akin to saying the base model will output typos. Both of these things are trained against. If you refer to deceptive alignment, this is a engineering problem as I stated.
I agree with the “fresh sheet of paper.” Reading the alignment faking paper and the current alignment challenges has been way more informative than reading Yudkowsky.
I think theese circles have granted him too many bayes points for predicting alignment when the technical details of his alignment problems basically don’t apply to deep learning as you said.
Oh I see. I was quick to bifurcate between deontology and utilitarianism. I guess I’m less familiar with other branches of consequentialism. Sorry for being unclear in the critique. My whole reply was just centered around being bad deontologically.
I see your point.
For the interest of the people today, there is an argument to be made for taking on risk of extinction. However, if this is not a purely utilitarian argument, I think it’s extremely careless and condemnable to impose this risk on humanity just because you have personally deemed it acceptable. This would be a deontological nightmare. Who gave AI labs the right to risk the lives of 8 billion people?
I was reluctant to get into the weeds here but how can anything near this model be possible if 2^300 is around how many atoms there are in the universe and we already have conquered 2^150 of them. At some point, there will likely be no more growing and then there will be millions of stable utopia years.
Reaping the benefits of AGI later is pretty insignificant in my opinion. If we get aligned AGI utopia, we will have utopia for millions of years. Acceleration by a few years if negligible if it increase p(doom) by >1%.
1% X 1 million utopia years = 10 thousand utopia years (better than 2 utopia years)
Dario gives a 25% p(doom) if I’m not mistaken. He still continues the build the tech that could knowingly bring doom. Dario and Anthropic are pro-acceleration via their messaging and actions according to a LW’er. How is this position coherent?
I don’t think you can name another company that admits to building technology with a >1% chance of killing everyone… besides maybe OpenAI.
Im 0⁄2
I’m a Canadian. I don’t understand the mindset of a republican. How is anything Trump is doing justifiable in any way!!!!
Are you implying that vegans will not eat lab meat because it is still imitation of flesh which is symbolically bad (or something similar)?
There are probably many vegans who aren’t like this.
I’ve grown accustomed to not bringing up EA and AI safety in my regular non-EA circles. It is too much of a headache trying to explain to people and although some are curious, I don’t enjoy being pushy and opinionated.
In EA circles, I get a lot of validation and a sense of connection. My novel ideas and culture are shared so tightly with other EAs. I almost feel as if there is a recoil effect where I am now too lazy to explore EA tenets with non-EAs. I don’t care for trying to build EA in others to give myself someone to talk to. (I am incredibly grateful for the EA community and I think that the recoil is minor compared to the benefit)
I generally find it extremely easy to be a normal dude day to day but mixing my EA world with my normal world is difficult. I don’t talk about EA with my family or my buddies. I don’t try to convince anyone outside of EA of anything and I probably should! I know AGI lab people who I could probably sway towards quitting their jobs or something haha.
And some people have even suggested saving more than normal if you have relatively short TAI timelines
Short timeline AGI is not priced into the stock market. AI labs, big tech, GPU and data companies are good bets to map to short-term AGI. If this is your belief, you can expect thousandfold returns on investment during the singularity which is far superior to holding cash. While mitigating X-risk via donations might be the maximally altruistic thing to do, it wouldn’t hurt to leave yourself in a good position in the case of a post-AGI world where money still matters.I would say the major exception is if you think your job is going to be automated soon. Then savings would be more valuable, though I suspect that even more valuable than savings would be working on being flexible and being able to pivot quickly.
I am a CS undergrad. As soon as I have money to save, I am going to hedge against job automation by investing in AGI stock. This will offer better financial protection compared to holding cash.
What should you do with the money that you are not putting into retirement? Some have suggested doing things on your bucket list, because you might not have an opportunity to later.
I personally only care about being happy day to day. For me, I can’t be sad that I didn’t check off my bucket list because I would be dead. I’m different to most I assume however.
I would recommend donating much of that extra money to the causes you care about while you can still make a difference, whether that is reducing existential risk, increasing animal welfare, reducing global poverty
Given short timelines, certain charities become less appealing. For example, animal welfare campaigns which convince companies to make commitments by a certain date many years from now may not have any impact. Existential risk probably has the highest expected utility with the correct animal welfare charities in second place.
Given the vibes of this post, I would highly recommend investing in AGI stock for mostly non-EA reasons. It seems like it would give you the safety net to then donate to EA causes.
Why security?
Why is SWE/ML low absorbency?
interesting, you offset each species independently using the appropriate charity. e.g. an omnivore causes 8.3 fish to be farmed per year and it costs $117 to offset those fish via fish charity. if you offset all animals via SWP or some chicken welfare project, the number might be around $10 to offset a year of meat eating which is similar to my prior of 5 cents daily. (since you dont use moral weights, i am only estimating the $10 figure)
This makes sense if you are a public figure or an advocate for animal rights. You talk about having suspicions of people who do moral trade and how they lose credibility. But what if someone is just earning to give and whose advocacy extends only to talking about EA with family and friends? Frankly, I don’t value expending my effort to keep appearances and signal my deontological constraints to others. That isn’t what a true utilitarian would do in this scenario.
Your reasoning is solid to me. great response.
love it
You refer to alignment faking/deceptive alignment, where a model in training expects negative reward and gives responses accordingly but outputs it’s true desires outside of training. This a solvable problem which is why I say alignment is not that hard.
Some other counterarguements:
LLMs will have no reason to take over the world before or after RLHF. They do not value it as a terminal goal. It is possible that they gain a cohorent, consistent, and misaligned goal purely by accident midway through RLHF and then fake it’s way through the rest of the fine-tuning. But this is unlikely and again solvable.
Making LLMs unaware they are in training is possible.