From an outsider perspective, this looks like the sort of thing that almost anyone could get started on and I like the phrasing you used to signal that. AI progress moves so fast that you are most likely going to the only one looking at something and so you can do very basic things like
“How deterministic are these models? If you take the first K lines of the CoT and regenerate it, do you get the same output?”
It’s pretty easy to imagine taking 1 line of CoT and regenerating and then 2 lines...
I think a lot of people can just do this and getting to do it under Neel Nanda is likely to lead to a high quality paper.
I think this effect is completely overshadowed by the fact if what you are saying is true, we have 5-10 years on the technical alignment/governance of AI to get things to go well.
Now is the time to donate and work on AI safety stuff. Not to get rich and donate to it later in the hopes that things worked out.