aviv

• avivNov 9, 2022, 9:55 PM

  2 points

  0 ∶ 0

  on: aviv’s Shortform

  Assuming misaligned AI is a risk, is technical AI alignment enough, or do you need joint AI/​Societal alignment?

  My work has involved trying to support risk awareness and coordination similar to what has been suggested for AI alignment. For example, for mitigating harms around synthetic media /​ “deepfakes” (now rebranded to generative AI) and it worked for a few years with all the major orgs and most relevant research groups.

  But then new orgs jumped in to fill the capability gap! (e.g. eleuther, stability, etc.)
  Due to demand and for potentially good reasons: those capabilities which can harm people can also help people. The ultimate result is the proliferation/​access/​democratization of AI capabilities in the face of risks.

  Question 1) What would stop the same thing from happening for technical AI safety alignment?^[1]

  I’m currently skeptical that this sort of coordination is possible without some addressing deeper societal incentives (AKA reward functions; e.g. around profit/​power/​attention maximization, self-dealing, etc.) and related multi-principal-agent challenges. This joint/​ai societal alignment or holistic alignment would seem to be a prerequisite to the actual implementation of technical alignment.^[2]

  Question 2) Am I missing something here? If one assumes that misaligned AI is a threat worth resourcing, what is the likelihood of succeeding at AI alignment longterm without also succeeding at ‘societal alignment’?

  1. ^

     This is assuming you can even get the major players on board, which isn’t true for e.g. misaligned recommender systems that I’ve also worked on (on the societal side).

  2. ^

     This would also be generally good for the world! E.g. to address externalities, political dysfunction, corruption, etc.

• avivOct 5, 2022, 2:19 PM

  2 points

  0 ∶ 0

  on: Global Gover­nance Read­ing List

  Thanks for sharing! I’m curious if any of these readings were most helpful around forming “theories of change toward achieving a limited form of global governance in specific key domains where it might be most important” or “viable mechanisms for partial global governance in those domains.”

  As someone exploring alternative ways to govern near-global powerful technology organizations that interact closely with nation-states and fund a significant proportion of AI research, this is what I would be most curious about (and which also seems e.g. particularly relevant re. x-risks). In the linked doc, I focus on sortition-based systems as one potential approach, but there are additional routes (e.g. ML-augmented) that I am also exploring using ~this framework, and I’d be interested any I have not considered.

• avivAug 12, 2022, 6:25 PM

  2 points

  0 ∶ 0

  on: Cause Ex­plo­ra­tion: Gover­nance De­sign & Formation

  Would Governance Experiments and Scaling be a better name?

  I’m curious if this is essentially taking on what I allude to here: https://​​aviv.medium.com/​​building-wise-systems-combining-competence-alignment-and-robustness-a9ed872468d3

• avivMar 26, 2022, 4:12 PM

  5 points

  0 ∶ 0

  in reply to: Sean_o_h’s comment on: Donat­ing against Short Term AI risks

  @Sean_o_h , Just seeing this now when searching for my name on the forum, actually to find a talk I did for an EA community! Thanks for the shoutout.

  For context, while I’ve not been super active community-wise, and I don’t to find identities, EA or otherwise, particularly useful to my work, I definitely e.g fit all the EA definitions as outlined by CEA, use ITN, etc.

• avivMar 13, 2022, 8:47 PM

  2 points

  0 ∶ 0

  in reply to: TedSanders’s comment on: Align­ing Recom­mender Sys­tems as Cause Area

  I just added a comment above which aims to provide a potential answer to this question—that you can use “approaches like those I describe here (end of the article; building on this which uses mini-publics)”. This may not directly get you something to measure, but it may be able to elicit the values needed for defining an objective function.

  You provide the example of this very low bar:

  I guarantee 100% that people would rather have the recommendations given by Netflix and YouTube than a uniform random distribution. So in that basic sense, I think we are already aligned.

  The goal here would be to scope out what a much higher bar might look like.

• avivMar 13, 2022, 7:46 PM

  1 point

  0 ∶ 0

  in reply to: Rohin Shah’s comment on: Align­ing Recom­mender Sys­tems as Cause Area

  I’m curious if approaches like those I describe here (end of the article; building on this which uses mini-publics) for determining rec system policy help address the concerns of your first 3 bullets. I should probably do a write-up or modification specifically for the EA audience (this is for a policy audience), but it ideally gets some of the point across re. how to do “deliberative retrospective judgment” in a way that is more likely to avoid problematic outcomes (I will also be publishing an expanded version that has much more sourcing).

• avivMar 13, 2022, 7:28 PM

  2 points

  0 ∶ 0

  in reply to: Zach Roush’s comment on: Zach Roush’s Shortform

  Art, writing, and other creative outputs are incredibly valuable for societal change and movement building. I know for my work, I’m currently looking for graphic designers who can convey complex ideas in glanceable simple images to skeptical stakeholders. Well produced animated videos would be even better.

  I think there is a huge amount of opportunity, and part of the challenge is that EA in my experience has some potential cultural gaps given its seemingly technocratic focus.

• avivMar 13, 2022, 7:26 PM

  5 points

  0 ∶ 0

  on: aviv’s Shortform

  It would may be helpful to grow a cross-national/​ethnic overarching identity around “wisdom and doing good”. EA does this is a bit, but is heavily constrained to the technocratic. While that is it useful subcomponent of that broader identity, it can push away people who share or aspire the underlying ideals of (1) “Doing good as a core goal of existence” and (2) “Being wise about how one chooses to do good”—but who don’t share the disposition or culture of most EA’s. Even the name itself can be a turnoff—it sound intellectual and elitist.

  Having a named identity which is broader than EA, but which contains it, could be incredibly helpful for connecting across neurodiverse divides in daily work, and could be incredibly valuable as a cross-cutting cleavage in national/​ethnic/​ etc. divides in conflict environments, if this can encompass a broad enough population over time.

  I’m not sure what that name might be in English, or if it makes more sense to just expand meaning of EA, but it may be worth thinking about this, and consciously growing a movement around that with aligned movements that perhaps get at other “lenses of wisdom” that focus on best utilizing/​growing resources for broad positive impact.

aviv’s Quick takes

• avivMar 3, 2022, 5:41 PM

  5 points

  0 ∶ 0

  on: The Fu­ture Fund’s Pro­ject Ideas Competition

  Global Mini-public on AI Policy and Cooperation

  Artificial Intelligence (Governance), Epistemic Institutions, Values and Reflective Processes, Great Power Relations

  We’d like to fund an organization to institutionalize regular (e.g. yearly) global mini-publics to create recommendations on AI policy and cooperation; ideally in partnership with the key academic journals (and potentially the UN, major corporations, research instituions, etc.) . Somewhat analogous to globalca.org which focuses on gene editing (https://​​www.science.org/​​doi/​​10.1126/​​science.abb5931) and globalassembly.org which focuses on climate (those are essentially pilots, heavily limited by funding).

• avivMar 3, 2022, 4:26 AM

  13 points

  0 ∶ 0

  on: The Fu­ture Fund’s Pro­ject Ideas Competition

  Platform Democracy Institutions

  Artificial Intelligence (Governance), Epistemic Institutions, Values and Reflective Processes, Great Power Relations

  Facebook/​Meta, YouTube/​Google, and other platforms make incredibly impactful decisions about the communications of billions. Better choices can significantly impact geopolitics, pandemic response, the incentives on politicians and journalists, etc. Right now, those decisions are primarily in the hands of corporate CEO’s—and heavily influenced by pressure from partisan and authoritarian governments aiming to entrench their own power. There is an alternative: platform democracy. In the past decade, a new suite of democratic processes have been shown to be surprisingly effective at navigating challenging and controversial issues, from nuclear power policy in South Korea to abortion in Ireland.

  Such processes have been tested around the world, overcome the pitfalls of elections and referendums, and can work at platform scale. They enable the creation of independent ‘people’s mandates’ for platform policies—something invaluable for the impacted populations, well-meaning governments which are unable to act on speech, and even the platforms themselves (in many cases at least, they don’t want to decide things since it opens them up to more government retaliation). We have a rapidly closing policy window to test and deploy platform democracy and give it real power and teeth. We’d like to see new organizations to advocate for, test, measure, certify, and scale platform democracy processes. We are especially excited about exploring the ways that these approaches can be used beyond just platform policies, but also for governance of the AI systems created and deployed by powerful corporations.

  (Note: This is not as crazy as it sounds; several platforms you have heard are dedicating significant resources to actively explore this, but they need neutral 3rd party orgs to work with; relevant non-profits are very interested but are stretched too thin to do much. The primary approaches I am referring to here are mini-publics and systems like Polis.)

  More detail at platformdemocracy.com (not an org; just a working paper right now)

• avivMar 3, 2022, 3:49 AM

  8 points

  0 ∶ 0

  on: The Fu­ture Fund’s Pro­ject Ideas Competition

  Operations and Execution Support for Impact

  Empowering Exceptional People, Effective Altruism

  The skill of running operations for building and growing a non-profit organization is often very different from doing the “core work” of that org. Figuring out operational details can suck energy away from the core work, leaving many promising people deciding not to start new orgs even when it is appropriate and necessary for scaling impact. We’d like to see an organization that could provide a sort of recruiting and matchmaking service which identifies promising operations people in more traditional domains, vets them, and matches them with potential founder/​grantees. In addition, such an organization could provide executive coaching to the founders. This would unlock people who would only start an org if they could jump into the core work and building out a team with operational support and coaching (and the health etc. benefits that operational support could do the setup for). Over time, such an org could build out a network and knowledge about how best to identify relevant operational talent and support success in other ways.

  What links here?

  • Cristina Schmidt Ibáñez's comment on The Fu­ture Fund’s Pro­ject Ideas Competition by Nick_Beckstead (Mar 6, 2022, 5:30 PM; 13 points)
  • WilliamKiely's comment on The Fu­ture Fund’s Pro­ject Ideas Competition by Nick_Beckstead (Mar 8, 2022, 12:12 AM; 6 points)

• avivMar 3, 2022, 1:42 AM

  19 points

  0 ∶ 0

  on: The Fu­ture Fund’s Pro­ject Ideas Competition

  Bridging-based Ranking for Recommender Systems

  Artificial Intelligence, Epistemic Institutions, Values and Reflective Processes, Great Power Relations

  Recommender systems are used by platforms like FB/​Meta, YouTube/​Google, Twitter, TikTok, etc. to direct the attention of billions of people every day. These systems, due to a combination of psychological, sociological, organizational, etc. factors are currently most likely to reward content producers with attention if they stoke division (e.g. outgroup animosity). Because attention is a currency that can be converted into money, power, and status, this “bias toward division” has impacts groups at every scale; from local school boards to Congress to geopolitics.

  Ensuring that recommender systems can mitigate this bias is crucial to functional democracy, to cooperation on catastrophic risks (e.g. AGI, pandemics, climate change), and simply to reducing the likelihood of escalating wars. We urgently need more research on how to better design recommender systems; we need to create open source implementations that do the right thing from the start which can be adopted by cash-strapped startups; and we need a mix of pressure and support to ensure these improvements will be rapidly deployed at platform scale.

• avivDec 10, 2020, 3:59 AM

  5 points

  0 ∶ 0

  on: Look­ing for col­lab­o­ra­tors af­ter last 80k pod­cast with Tris­tan Harris

  I was informed of this thread by someone in the EA community who suggested I help. I have deep subject matter expertise in this domain (depending on how you count, I’ve been working in it full-time for 5 years, and toward it for 10+ years).

  The reason I started working on this could be characterized as resulting from my beliefs on the “threat multiplier” impact that broken information ecosystems have on catastrophic risk.

  A few caveats about all this though:

  1. Most of the public dialogue around these issues is very simplistic and reductionist (which leads to the following two issues...).

  2. The framing of the questions you provide may not be ideal for getting at your underlying goals/​questions. I would think more about that.

  3. Much of the academic research is terrible, simply due to the lack of quality data and the newness and interdisciplinary nature of the fields; even “top researchers” sometimes draw the unsubstantiated conclusions from their studies.

  All that said, I continue to believe that the set of problems around information systems (and relatedly governance) are a prerequisite for addressing catastrophic global risks—that they are among the most urgent and important issues that we could be addressing—and that we are still heading at a faster and faster rate in the wrong direction.

  I have very limited bandwidth, with a number of other projects in the space, but if people are putting put significant money and time toward this, I may be able to put in some time in an advisory role at least to help direct that energy effectively. My contact info and more context about me is at aviv.me .

• avivJun 10, 2020, 1:53 AM

  21 points

  0 ∶ 0

  on: EAGxVir­tual Un­con­fer­ence (Satur­day, June 20th 2020)

  Avoiding Infocalypse: How a decline in epistemic competence makes catastrophic risks inevitable — and what EA’s can do about it

  This would be shortened and modified version of a talk I gave at Cambridge University, at the Leverhulme Centre for the Future of Intelligence and Centre for the Study of Existential Risk. The general public version of many of the ideas can be found in this TEDx talk that I gave in 2018 (ignore the title, not my choice).

  Part 1: Framing the underlying problem

  Describe what is meant by epistemic competence (the ability or desire for individuals, organizations, governments, etc. to effectively make sense of the world). Illustrate how it is declining, and how that is likely to get worse.

  Part 2: Connect to catastrophic risks
  Describe how lower epistemic competence makes it extremely difficult to do any sort of crucial coordination, making global coordination on catastrophic risks increasingly unlikely. In addition, lower epistemic competence makes catastrophic forcing functions more likely and individual mitigation steps less likely.

  Part 3: Exploring mitigations

  Discuss what can be done, and show that many of these problems are related to other better understood EA cause areas (including e.g. the connection between synthetic media and AGI).

  ------

  More about my work here for context: aviv.me, twitter.com/​metaviv.

  I would be interested in a late session. My goal is to more broadly circulate these concerns in the EA community, which I have been adjacent to for many years (e.g. this podcast episode I did with Julia Galef) but never deeply engaged.