This seems like a great project, so thanks a lot for doing this! <3
Suggestions & feedback
Begin future summary posts with a short but huge standard disclaimer of the main shortcomings of (these / all) AI summaries: can’t guarantee accuracy; the emphasis might be wrong; middle sections of long posts are skipped for now; etc.
Besides the author byline, add additional info pulled from the original posts: like length or reading time; or amount of discussion (none / little / lots)
(Since you already have the author bylines, you could link the author names to their user profiles. Though that might just result in pointless clutter.)
(You could even flag posts which feature disproportionate amounts of scholarship, by e.g. identifying posts which link to a lot of academic papers, or which have lots of footnotes. But that has unclear benefits and feasibility, and is likely very out of scope.)
Currently summaries are grouped by EA Forum vs. LessWrong, but that seems like insufficient structure given the ~16 posts in each section. And I don’t know how posts are sorted right now. So a third level of hierarchy seems warranted. Maybe these posts could be automatically grouped or sorted into topics like “AI” or “Longtermism” or something?
I don’t know where the topics and the categorization would come from, though. Maybe from looking at the post tags? Unfortunately posts are tagged by hand and thus tags are inconsistently applied.
There were some issues with the bylines, which I reported in this comment thread.
Summary limitations: Detect when a post contains mostly content which can’t be properly summarized: probably images; probably some types of embeds (Manifold? Twitter?); maybe also other stuff like tables or LaTeX math formulas.
Regarding summary tone & format
In addition to the current output as a post on the EA Forum / LW, you could also make a spreadsheet of summaries (with columns for e.g. week, title, link, authors, summary paragraph), add new summaries to this spreadsheet, and then link to the spreadsheet at the beginning of each post. I’m not sure how useful that would be, but it would allow for e.g. follow-up automations; quarterly / yearly summaries of these weekly summary posts; etc.
Regarding the formatting of the summaries, one question to ask is “Who are these summaries for?” or “What value do summaries add?” or “What questions do summaries answer?”.
E.g. a crucial one is “Should I read this?” / “Who would benefit from reading this?”. Currently these summaries answer that question implicitly, but you could instead / in addition ask ChatGPT to answer this question explicitly.
Regarding the summary tone, I currently find the summaries a bit dry, so I don’t find them particularly enjoyable to read. But I’m not sure if there’s a tone which is less formal without being less accurate.
Opportunities for collaboration
Maybe you could collaborate with the people from Type III Audio in some way? They generate automatic audio transcripts for new posts while you generate automatic text summaries; so there might be some potential for cross-pollination.
I vaguely recall there also being some efforts to auto-tag posts via LLMs; that’s another avenue for collaboration.
This seems like a great project, so thanks a lot for doing this! <3
Suggestions & feedback
Begin future summary posts with a short but huge standard disclaimer of the main shortcomings of (these / all) AI summaries: can’t guarantee accuracy; the emphasis might be wrong; middle sections of long posts are skipped for now; etc.
Besides the author byline, add additional info pulled from the original posts: like length or reading time; or amount of discussion (none / little / lots)
(Since you already have the author bylines, you could link the author names to their user profiles. Though that might just result in pointless clutter.)
(You could even flag posts which feature disproportionate amounts of scholarship, by e.g. identifying posts which link to a lot of academic papers, or which have lots of footnotes. But that has unclear benefits and feasibility, and is likely very out of scope.)
Currently summaries are grouped by EA Forum vs. LessWrong, but that seems like insufficient structure given the ~16 posts in each section. And I don’t know how posts are sorted right now. So a third level of hierarchy seems warranted. Maybe these posts could be automatically grouped or sorted into topics like “AI” or “Longtermism” or something?
I don’t know where the topics and the categorization would come from, though. Maybe from looking at the post tags? Unfortunately posts are tagged by hand and thus tags are inconsistently applied.
Spelling: “Lesswrong” section header → “LessWrong”
There were some issues with the bylines, which I reported in this comment thread.
Summary limitations: Detect when a post contains mostly content which can’t be properly summarized: probably images; probably some types of embeds (Manifold? Twitter?); maybe also other stuff like tables or LaTeX math formulas.
Regarding summary tone & format
In addition to the current output as a post on the EA Forum / LW, you could also make a spreadsheet of summaries (with columns for e.g. week, title, link, authors, summary paragraph), add new summaries to this spreadsheet, and then link to the spreadsheet at the beginning of each post. I’m not sure how useful that would be, but it would allow for e.g. follow-up automations; quarterly / yearly summaries of these weekly summary posts; etc.
Regarding the formatting of the summaries, one question to ask is “Who are these summaries for?” or “What value do summaries add?” or “What questions do summaries answer?”.
E.g. a crucial one is “Should I read this?” / “Who would benefit from reading this?”. Currently these summaries answer that question implicitly, but you could instead / in addition ask ChatGPT to answer this question explicitly.
Regarding the summary tone, I currently find the summaries a bit dry, so I don’t find them particularly enjoyable to read. But I’m not sure if there’s a tone which is less formal without being less accurate.
Opportunities for collaboration
Maybe you could collaborate with the people from Type III Audio in some way? They generate automatic audio transcripts for new posts while you generate automatic text summaries; so there might be some potential for cross-pollination.
I vaguely recall there also being some efforts to auto-tag posts via LLMs; that’s another avenue for collaboration.
Beaut. Thanks for the detailed feedback!
I think these suggestions make sense to implement immediately:
add boilerplate disclaimer about accuracy / fabrication
links to author pages
note on reading time
group by tags
“Lesswrong” → “LessWrong”
The summaries are in fact generated within a Google sheet, so it does make sense to add a link to that
These things will require a bit of experimentation but are good suggestions:
Agree on the tone being boring. I can think of a couple of fixes:
Prompt GPT to be more succinct to get rid of low information nonsense
Prompt GPT to do bulletpoints rather than paragraphs
Generate little poems to introduce sections
Think about cross pollinating with Type III Audio