Tamay

• Tamay 3 Sep 2024 18:31 UTC

  4 points

  1 ∶ 0

  in reply to: Luise’s comment on: Will the US Govern­ment Con­trol the First AGI?—Find­ing Base Rates

  He gives explanations for each of his labels, e.g. for the airplane he considers the relevant inventors the Wright brothers who weren’t government-affiliated at the time. It’s probably best to refer to his research if you want to verify how much to trust the labels.

  By that token, AI won’t be government controlled either because neural networks were invented by McCulloch/​Pitts/​Rosenblatt with minimal government involvement. Clearly this is not the right way to think about government control of technologies.

• Tamay 3 Sep 2024 7:33 UTC

  6 points

  1 ∶ 0

  on: Will the US Govern­ment Con­trol the First AGI?—Find­ing Base Rates

  I like the idea, but the data seems sketchy. For example, the notion of “government control” seems poorly applied:

  • you assign a 0 USG control for “airplane” but historicaly government “control” has been very high (Boeing, Lockheed, Grumman were all operating for the USG during WW2).

  • you assign 0 to the “decoding of the human genome” but the human genome project was initiated, largely funded, and directed by the U.S. government

  Some entries are broad categories (e.g., “Nanotechnology”), while others are highly specific (e.g., “Extending the host range of a virus via rational protein design”) which makes the list feel arbitrary. Why are “Standard model of physics” on the list but not other major theories of physics (e.g. QM or relativity)? Why aren’t Neural nets on here?

• Tamay 22 Jan 2024 22:37 UTC

  1 point

  0 ∶ 0

  in reply to: Ben_West🔸’s comment on: Rates of Crim­i­nal­ity Amongst Giv­ing Pledge Signatories

  >I’m mostly just noting that “altruistic people don’t commit crimes” doesn’t seem like a likely hypothesis.

  I think your data is evidence in favor of a far more interesting conclusion.

• Tamay 22 Jan 2024 18:33 UTC

  26 points

  8 ∶ 0

  on: Rates of Crim­i­nal­ity Amongst Giv­ing Pledge Signatories

  I previously estimated that 1-2% of YCombinator-backed companies with valuations over $100M had serious allegations of fraud.

  While not all Giving Pledge signatories are entrepreneurs, a large fraction are, which makes this a reasonable reference class. (An even better reference class would be “non-signatory billionaires”, of course.)

  My guess is that YCombinator-backed founders tend to be young with shorter careers than pledgees, and in part because of this will likely have had fewer run-ins with the law. I think better reference class would be something like founders of Fortune 500/​Fortune 100 companies.

• Tamay 29 Dec 2023 3:36 UTC

  1 point

  0 ∶ 0

  in reply to: yanni’s comment on: Ta­may’s Quick takes

  You asked for examples of advocacy done well with respect to truth-seekingness/​providing well-considered takes, and I provided examples.

• Tamay 28 Dec 2023 22:39 UTC

  7 points

  2 ∶ 0

  in reply to: yanni’s comment on: Ta­may’s Quick takes

  I think much of the advocacy within EA is reasonably thoughtful and truth-seeking. Reasoning and uncertainties are often transparently communicated. Here are two examples based on my personal impressions:

  • advocacy around donating a fraction of one’s income to effective charities is generally focused on providing accounts of key facts and statistics, and often acknowledges its demandingness and its potential for personal and social downsides

  • wild animal suffering advocacy usually does things like acknowledging the second-order effects of interventions on ecosystems, highlights the amount of uncertainty around the extent of suffering and often calls for more research rather than immediate intervention

  
  By contrast, EA veganism advocacy has done a much poorer job in remaining truth-seeking as Elizabeth has pointed out.

• Tamay 24 Dec 2023 18:16 UTC

  30 points

  15 ∶ 8

  on: Ta­may’s Quick takes

  I think EAs should be more critical of its advocacy contingents, and that those involved in such efforts should set a higher bar for offering more thoughtful and considered takes.

  Short slogans and emojis-in-profiles, such as those often used in ‘Pause AI’ advocacy, are IMO inadequate for the level of nuance required for complex topics like these. Falling short can burn credibility and status of those involved in EA in the eyes of onlookers.

• Tamay 5 Oct 2023 10:43 UTC

  1 point

  0 ∶ 0

  in reply to: Brad West🔸’s comment on: Ta­may’s Quick takes

  One issue I have with this is that when someone calls this the ‘default’, I interpret them as implicitly making some prediction about the likelihood of such countermeasures not being taken. The issue then that this is a very vague way to communicate one’s beliefs. How likely does some outcome need to be for it to become the default? 90%? 70%? 50%? Something else?
  

  The second concern is that it’s improbable for minimal or no safety measures to be implemented, making it odd to set this as a key baseline scenario. This belief is supported by substantial evidence indicating that safety precautions are likely to be taken. For instance:

  • Most of the major AGI labs are investing quite substantially in safety (e.g. OpenAI committing some substantial fraction of its compute budget, a large fraction of Anthropic’s research staff seems dedicated to safety, etc.)

  • We have received quite a substantial amount of concrete empirical evidence that safety-enhancing innovations are important for unlocking the economic value from AI systems (e.g. RLHF, constitutional AI, etc.)

  • It seems a priori very likely that alignment is important for unlocking the economic value from AI, because this effectively increases the range of tasks that AI systems can do without substantial human oversight, which is necessary for deriving value from automation

  • Major governments are interested in AI safety (e.g. the UK’s AI Safety Summit, the White House’s securing commitments around AI safety from AGI labs)

  Maybe they think that safety measures taken in a world in which we observe this type of evidence will fall far short from what is neeeded. However, it’s somewhat puzzling be confident enough in this to label it as the ‘default’ scenario at this point.

• Tamay 4 Oct 2023 0:11 UTC

  6 points

  1 ∶ 0

  on: Ta­may’s Quick takes

  The term ‘default’ in discussions about AI risk (like ‘doom is the default’) strikes me as an unhelpful rhetorical move. It suggests an unlikely scenario where little-to-no measures are taken to address AI safety. Given the active research and the fact that alignment is likely to be crucial to unlocking the economic value from AI, this seems like a very unnatural baseline to frame discussions around.

Ta­may’s Quick takes

• Tamay 24 Sep 2023 4:16 UTC

  1 point

  0 ∶ 0

  on: Some global catas­trophic risk estimates

  I’ve updated the numbers based on today’s predictions. Key updates:

  • AI-related risks have seen a significant increase, almost doubling both in terms of catastrophic (from 3.06% in Jun 2022 to 6.16% in September 2023) and extinction risk (from 1.56% to 3.39%).

  • Biotechnology risks have actually decreased in terms of catastrophe likelihood (from 2.21% to 1.52%), while staying constant for extinction risk (0.07% in both periods).

  • Nuclear War has shown an uptick in catastrophic risk (from 1.87% to 2.86%) but remains consistent in extinction risk (0.06% in both periods).

• Tamay 14 Sep 2023 1:54 UTC

  31 points

  7 ∶ 0

  on: Who should we in­ter­view for The 80,000 Hours Pod­cast?

  • Owain Evans on AI alignment (situational awareness in LLM, benchmarking truthfulness)

  • Ben Garfinkel on AI policy (best practices in AI governance, open source, the UK’s AI efforts)

  • Anthony Aguirre on AI governance, forecasting, cosmology

  • Beth Barnes on dangerous capability evals (GPT-4′s and Claude’s eval)

• Tamay 14 Nov 2022 18:43 UTC

  66 points

  27 ∶ 1

  in reply to: EliezerYudkowsky’s comment on: NY Times on the FTX im­plo­sion’s im­pact on EA

  I agree the victim-perpetrator is an important lens through which to view this saga. But, I also think that an investor-investee framing is another important one; a framing that has different prescriptions for what lessons to take away, and what to do next. The EA community staked easily a billion dollars worth of its assets (in focus, time, reputation, etc.), and ended up losing it all. I think it’s crucial to reflect on whether the extent of our due diligence and risk management was commensurate with the size of EA’s bet.

• Tamay 12 Nov 2022 20:30 UTC

  79 points

  44 ∶ 6

  in reply to: Milan_Griffes’s comment on: A per­sonal state­ment on FTX

  One specific question I would want to raise is whether EA leaders involved with FTX were aware of or raised concerns about non-disclosed conflicts of interest between Alameda Research and FTX.

  For example, I strongly suspect that EAs tied to FTX knew that SBF and Caroline (CEO of Alameda Research) were romantically involved (I strongly suspect this because I have personally heard Caroline talk about her romantic involvement with SBF in private conversations with several FTX fellows). Given the pre-existing concerns about the conflicts of interest between Alameda Research and FTX (see examples such as these), if this relationship were known to be hidden from investors and other stakeholders, should this not have raised red flags?

An­nounc­ing Epoch: A re­search or­ga­ni­za­tion in­ves­ti­gat­ing the road to Trans­for­ma­tive AI

• Tamay 23 Jun 2022 15:35 UTC

  6 points

  0 ∶ 0

  in reply to: bgarfinkel’s comment on: Ben Garfinkel’s Shortform

  This is insightful. Some quick responses:

  • My guess would be that the ability to commercialize these models would strongly hinge on the ability for firms to wrap these up with complementary products, that would contribute to an ecosystem with network effects, dependencies, evangelism, etc.

  • I wouldn’t draw too strong conclusions from the fact that the few early attempts to commercialize models like these, notably by OpenAI, haven’t succeeded in creating the preconditions for generating a permenant stream of profits. I’d guess that their business models look less-than-promising on this dimension because (and this is just my impression) they’ve been trying to find product-market-fit, and have gone lightly on exploiting particular fits they found by building platforms to service these

  • Instead, better examples of what commercialization looks like are GPT-3-powered companies, like copysmith, which seem a lot more like traditional software businesses with the usual tactics for locking users in, and creating network effects and single-homing behaviour

  • I expect that companies will have ways to create switching costs for these models that traditional software product don’t have. I’m particularly interested in fine-tuning as a way to lock-in users by enabling models to strongly adapt to context about the users’ workloads. More intense versions of this might also exist, such as learning directly from individual customer’s feedback through something like RL. Note that this is actually quite similar to how non-software services create loyalty

  I agree that it seems hard to commercialize these models out-of-the-box with something like paid API access, but I expect, given the points above, to be superseded by better strategies.

• Tamay 19 Jun 2022 21:28 UTC

  2 points

  0 ∶ 0

  on: Some global catas­trophic risk estimates

  By request, I have updated the predictions based on the latest predictions. Previous numbers can be found here.

Some global catas­trophic risk estimates

• Tamay 6 Nov 2020 12:43 UTC

  45 points

  0 ∶ 0

  on: Progress Open Thread: Novem­ber 2020

  I won the Stevenson prize (a prize given out at my faculty) for my performance in the MPhil in Economics. I gather Amartya Sen won the same prize some 64 years ago, which I think is pretty cool.

• Tamay 22 Feb 2020 19:14 UTC

  0 points

  0 ∶ 0

  on: An­nounc­ing Li Wen­liang Prize for fore­cast­ing the COVID-19 outbreak

  The list of questions can be accessed here.