johnjnay

Karma: 104

Stanford A.I. researcher.

Co-founder and CEO of an A.I. technology company, Brooklyn Artificial Intelligence Research (Skopos Labs, Inc.), which owns the investment management firm, Brooklyn Investment Group (https://bkln.com).

Conducted research funded by the U.S. National Science Foundation and the U.S. Office of Naval Research. Created first A.I. course at the NYU School of Law. Published research on A.I., finance, law, policy, economics, and climate change. Publications at http://johnjnay.com, and Twitter at https://twitter.com/johnjnay.

Large Language Models as Fiduciaries to Humans

johnjnay24 Jan 2023 19:53 UTC

25 points

0 comments34 min readEA link

(papers.ssrn.com)

johnjnay 8 Jan 2023 20:22 UTC
4 points
0 ∶ 0
in reply to: aogara’s comment on: Large Language Models as Corporate Lobbyists, and Implications for Societal-AI Alignment
I agree this would require new legislation to fully address (rather than merely a change to a rule under existing statute).
As far as I’m aware there has not been any consideration of relevant legislation, but I would love to learn about anything that others have seen that may be relevant.

johnjnay 7 Jan 2023 19:40 UTC
8 points
0 ∶ 0
in reply to: aogara’s comment on: Large Language Models as Corporate Lobbyists, and Implications for Societal-AI Alignment
Unfortunately, I think the upside of considering amendments to lobbying disclosure laws to attempt to address implications of this outweigh downsides of people learning more about this.
Also, the more well-funded special interest groups are more likely to independently discover and advance AI-driven lobbying than the less well-funded / more diffuse interests of average citizens.

johnjnay 7 Jan 2023 19:37 UTC
2 points
0 ∶ 0
on: Large Language Models as Corporate Lobbyists, and Implications for Societal-AI Alignment
Cross-posted here: https://law.stanford.edu/2023/01/06/large-language-models-as-lobbyists/

Large Language Models as Corporate Lobbyists, and Implications for Societal-AI Alignment

johnjnay4 Jan 2023 22:22 UTC

10 points

6 comments8 min readEA link

johnjnay 19 Nov 2022 19:22 UTC
6 points
0 ∶ 0
in reply to: Geoffrey Miller’s comment on: AI alignment with humans… but with which humans?
I think AI alignment can draw from existing law to a large degree. New legal concepts may be needed but I think there is a lot legal reasoning, legal concepts, legal methods, etc. that are directly applicable now (discussed in more detail here https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4218031).
Also, I think we should keep the involvement of AI in law-making (broadly defined) as limited as we can. And we should train AI to understand when there is sufficient legal uncertainty that a human is needed to resolve the correct action to be taken.

johnjnay 19 Nov 2022 1:25 UTC
6 points
0 ∶ 0
on: AI alignment with humans… but with which humans?
This is a great post.
Law is the best solution I can think to address the issues you raise.
- Here https://forum.effectivealtruism.org/posts/9YLbtehKLT4ByLvos/agi-misalignment-x-risk-may-be-lower-due-to-an-overlooked I argue that law-informed AI is likely the best path forward for societal alignment
- Here https://forum.effectivealtruism.org/posts/4ykDJA57wstYWq9HK/intent-alignment-should-not-be-the-goal-for-agi-x-risk I explore the difference between intent alignment and societal alignment.

johnjnay 26 Oct 2022 19:04 UTC
3 points
0 ∶ 0
on: Intent alignment should not be the goal for AGI x-risk reduction
Related post: AGI misalignment x-risk may be lower due to an overlooked goal specification technology.

johnjnay 26 Oct 2022 19:00 UTC
3 points
0 ∶ 0
on: AGI misalignment x-risk may be lower due to an overlooked goal specification technology
Related post: Intent alignment should not be the goal for AGI x-risk reduction

Intent alignment should not be the goal for AGI x-risk reduction

johnjnay26 Oct 2022 1:24 UTC

5 points

1 comment1 min readEA link

AGI misalignment x-risk may be lower due to an overlooked goal specification technology

johnjnay21 Oct 2022 2:03 UTC

22 points

1 comment1 min readEA link

johnjnay 26 Sep 2022 17:55 UTC
4 points
0 ∶ 0
on: Aligning AI with Humans by Leveraging Legal Informatics
This was cross-posted here as well:
- https://law.stanford.edu/2022/09/25/aligning-ai-with-humans-by-leveraging-law-as-data/
- https://www.lesswrong.com/posts/9xR4KExLQKNK4iggc/leveraging-legal-informatics-to-align-ai
A follow-up thought based on conversations catalyzed by this post:
Much of the research on governing AI and managing its potential unintended consequences currently falls into two ends of a spectrum related to assumptions of the imminence of transformative AGI. Research operating under the assumption of a high probability of near-term transformative AI (e.g., within 10-15 years) is typically focused more on how to align AGI with ideal aggregations of human preferences (through yet to be tested aggregation processes). Research operating under the assumption of a low probability of near-term transformative AI is typically focused on how to reduce discriminatory, safety, and privacy harms posed by present-day (relatively “dumb”) AI systems. The proposal in this post seeks a framework that, over time, bridges these two important ends of the AI safety spectrum.

Climate-contingent Finance, and A Generalized Mechanism for X-Risk Reduction Financing

johnjnay26 Sep 2022 13:23 UTC

6 points

1 comment26 min readEA link

johnjnay 21 Sep 2022 20:04 UTC
2 points
0 ∶ 0
in reply to: Geoffrey Miller’s comment on: Aligning AI with Humans by Leveraging Legal Informatics
Thanks!

johnjnay 20 Sep 2022 14:38 UTC
1 point
0 ∶ 0
in reply to: Charlie Steiner’s comment on: Aligning AI with Humans by Leveraging Legal Informatics
Interesting. I will think more about the sandwiching approach between non-legal experts and legal experts.

johnjnay 19 Sep 2022 11:10 UTC
2 points
0 ∶ 0
in reply to: Geoffrey Miller’s comment on: Aligning AI with Humans by Leveraging Legal Informatics
Hi Geoffrey, thank you for this feedback.
On your background knowledge comment, I agree that is an important open question (for this proposal, and other alignment techniques).
Related to that, I have been thinking through the systematic selection of which data sets are best suited for self-supervised pre-training of large language models—an active area of research in AI capabilities and Foundation Models more generally, which may be even more important for this application to legal data. For self-supervision on legal data, we could use (at least) two filters to guide data selection and data structuring processes.
First, is the goal of training on a data point to embed world knowledge into AI, or legal task knowledge? Learning that humans in the U.S. drive on the right side of the road is learning world knowledge; whereas, learning how to map a statute about driving rules to a new fact pattern in the real world is learning how to conduct a legal reasoning task. World knowledge can be learned from legal and non-legal corpora. Legal task knowledge can primarily be learned from legal data.
Second, is the approximate nature of the uncertainty that an AI could theoretically resolve by training on a data point epistemic or aleatory ? If the nature of the uncertainty is epistemic – e.g., whether citizens prefer climate change risk reduction over endangered species protection – then it is fruitful to apply as much data as we can to learning functions to closer approximate the underlying fact about the world or about law. If the nature of the uncertainty is more of an aleatory flavor – e.g., the middle name of the defendant in a case – then there is enough inherent randomness that we would seek to avoid attempting to learn anything about that fact or data point.
There are many other aspects of self-supervised pre-training data curation that we will need to explore, but figured I’d share a couple that are top of mind in the context of your world knowledge comment.
Public law informs AI more through negative than positive directives; and therefore it’s unclear the extent to which policy – outside of the human-AI “contract and standards” type of alignment we are working on – can inform which goals AI should proactively pursue to improve the world on society’s behalf. I agree with your comment that, “law tends to track situations where humans have conflicts of interest with each other, and it might not track universal values that are so obvious to everyone that conflicts of interest hardly ever arise.” This is a great illustration of the need to complement the Law Informs Code approach with other approaches to specifying human values. But I believe there are challenges with using the “AI Ethics” approach as the core framework, see section IV. PUBLIC LAW: SOCIETY-AI ALIGNMENT of the longer form version of this post, https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4218031 . I think a blend of the frameworks could most fruitful.
Finally, it would be very interesting to conduct research on the possibility of “cross-cultural universals in legal systems that exemplify some common ground for human values,” and which domains of law have the most cultural overlap. There are many exciting threads to pursue here!

johnjnay 19 Sep 2022 10:38 UTC
4 points
0 ∶ 0
in reply to: Charlie Steiner’s comment on: Aligning AI with Humans by Leveraging Legal Informatics
Hi Charlie, thank you for your comment.
I cite many of Dylan’s papers in the longer form version of this post: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4218031
I will check out Xuan’s talk. Thanks for sharing that.
Instead of:
Law is a computational engine that converts human values into legible directives.
I could expand the statement to cover the larger project of what we are working on:
Law and legal interpretation form a computational engine that converts human values into legible directives.
One of the primary goals of this research agenda is to teach AI to follow the spirit of the law in a human-recognizable way. This entails leveraging existing human capabilities for the “law-making” / “contract-drafting” part (how do we use the theory and practice of law about how to tell agents what to do?), and conducting research on building AI capabilities for the interpretation part (how do our machine learning processes use data and processes from the theory and practice of law about how agents interpret those directives / contracts?).
Reinforcement learning with human attorney feedback (there are more than 1.3 million lawyers in the US) via natural language interactions with AI models is potentially a powerful process to teach (through training, or fine-tuning, or extraction of templates for in-context prompting of large language models) statutory interpretation, argumentation, and case-based reasoning, which can then be applied more broadly for AI alignment. Models could be trained to assist human attorney evaluators, which theoretically, in partnership with the humans, could allow the combined human-AI evaluation team to have capabilities that surpass the legal understanding of the expert humans alone.
The Foundation Models in use today, e.g., GPT-3, have, effectively, conducted a form of behavioral cloning on a large portion of the Internet to leverage billions of human actions (through natural language expressions). It may be possible to, similarly, leverage billions of human legal data points to build Law Foundation Models through large-scale language model self-supervision on pre-processed legal text data.
Aspects of legal standards, and the “spirit” of the law, can be learned directly from legal data. We could also codify examples of human and corporate behavior exhibiting standards such as fiduciary duty into a structured format to evaluate the standards-understanding capabilities of AI models. The legal data available for AI systems to learn from, or be evaluated on, includes textual data from all types of law (constitutional, statutory, administrative, case, and contractual), legal training tools (e.g., bar exam outlines, casebooks, and software for teaching the casuistic approach), rule-based legal reasoning programs, and human-in-the-loop live feedback from law and policy human experts. The latter two could simulate state-action-reward spaces for AI fine-tuning or validation, and the former could be processed to do so.
Automated data curation processes to convert textual legal data into either state-action-reward tuples, or contextual constraints for shaping candidate action choices conditional on the state, is an important frontier in this research agenda (and promising for application to case law text data, contracts, and legal training materials). General AI capabilities research has recently found that learning from textual descriptions, rather than direct instruction, may allow models to learn reward functions that better generalize. Fortunately, much of law is embedded more in the form of descriptions and standards than it is in the form of direct instructions and specific rules. Descriptions of the application of standards provides a rich and large surface area to learn from.
Textual data can be curated and labeled for these purposes. We will aim for two outcomes with this labeling. First, data that can be used to evaluate how well AI models understand legal standards. Second, the possibility that the initial “gold-standard” human expert labeled data can be used to generate additional much larger sets of data through automated curation and processing of full corpora of legal text, and through model interaction with human feedback.
I think your statement:
“This research direction does not look like a Bold New Way to do AI alignment, instead it looks like a Somewhat Bold New Way to apply AI alignment work that is fully contiguous with other alignment research”
is spot on. That is how I was thinking about it, but I should have made that more clear; perhaps I should work on a follow-up post at some point that explicitly explores the intersections of Law Informs Code with other strands of alignment research. Some of this is in the longer form version of this post, but with this inspiration from you, I may try to go further in that direction (although I am already beyond the length the Journal editors want!).

johnjnay 18 Sep 2022 15:16 UTC
1 point
0 ∶ 0
in reply to: Cullen_OKeefe’s comment on: Aligning AI with Humans by Leveraging Legal Informatics
That’s awesome—thank you for sharing!
Would love to chat as well.

Aligning AI with Humans by Leveraging Legal Informatics

johnjnay18 Sep 2022 7:43 UTC

20 points

11 comments3 min readEA link

johnjnay

Large Lan­guage Models as Fi­du­cia­ries to Humans

Large Lan­guage Models as Cor­po­rate Lob­by­ists, and Im­pli­ca­tions for So­cietal-AI Alignment

In­tent al­ign­ment should not be the goal for AGI x-risk reduction

AGI mis­al­ign­ment x-risk may be lower due to an over­looked goal speci­fi­ca­tion technology

Cli­mate-con­tin­gent Fi­nance, and A Gen­er­al­ized Mechanism for X-Risk Re­duc­tion Financing

Align­ing AI with Hu­mans by Lev­er­ag­ing Le­gal Informatics