Error
Unrecognized LW server error:
Field "fmCrosspost" of type "CrosspostOutput" must have a selection of subfields. Did you mean "fmCrosspost { ... }"?Unrecognized LW server error:
Field "fmCrosspost" of type "CrosspostOutput" must have a selection of subfields. Did you mean "fmCrosspost { ... }"?
This is great! I very much miss the weekly summaries. Maybe it would be nice if these summaries were published as a weekly newsletter, just like Zoe’s?
Thanks for taking on this public service task!
Small note—it seems to occasionally mess up the usernames for posts like ‘EU farmed fish policy reform roadmap’ and ‘New probabilistic simulation tool’.
Thanks for pointing this out! I’ve fixed it in this post and we’ll look into checking it automatically
I’ve identified source of problem and fixed, thanks!
Another username issue: If there are multiple authors, they’re ignored here, e.g. for “Impact obsession”, the attribution for Ewelina_Tur is missing. Same with “CE alert”, “Dimensions of pain”, etc.
I’ve noticed one more username issue: The listed name is the one in the URL (e.g. jessica-liu-taylor) instead of the username (e.g. jessicata). Or e.g. Bill Benzon becomes bill-benzon. It’s not clear to me why e.g. Raemon doesn’t become raemon, though.
should be fixed for the next issue
Thanks for doing this!
Yes, everything gets cross-posted!
cool ty
This seems like a great project, so thanks a lot for doing this! <3
Suggestions & feedback
Begin future summary posts with a short but huge standard disclaimer of the main shortcomings of (these / all) AI summaries: can’t guarantee accuracy; the emphasis might be wrong; middle sections of long posts are skipped for now; etc.
Besides the author byline, add additional info pulled from the original posts: like length or reading time; or amount of discussion (none / little / lots)
(Since you already have the author bylines, you could link the author names to their user profiles. Though that might just result in pointless clutter.)
(You could even flag posts which feature disproportionate amounts of scholarship, by e.g. identifying posts which link to a lot of academic papers, or which have lots of footnotes. But that has unclear benefits and feasibility, and is likely very out of scope.)
Currently summaries are grouped by EA Forum vs. LessWrong, but that seems like insufficient structure given the ~16 posts in each section. And I don’t know how posts are sorted right now. So a third level of hierarchy seems warranted. Maybe these posts could be automatically grouped or sorted into topics like “AI” or “Longtermism” or something?
I don’t know where the topics and the categorization would come from, though. Maybe from looking at the post tags? Unfortunately posts are tagged by hand and thus tags are inconsistently applied.
Spelling: “Lesswrong” section header → “LessWrong”
There were some issues with the bylines, which I reported in this comment thread.
Summary limitations: Detect when a post contains mostly content which can’t be properly summarized: probably images; probably some types of embeds (Manifold? Twitter?); maybe also other stuff like tables or LaTeX math formulas.
Regarding summary tone & format
In addition to the current output as a post on the EA Forum / LW, you could also make a spreadsheet of summaries (with columns for e.g. week, title, link, authors, summary paragraph), add new summaries to this spreadsheet, and then link to the spreadsheet at the beginning of each post. I’m not sure how useful that would be, but it would allow for e.g. follow-up automations; quarterly / yearly summaries of these weekly summary posts; etc.
Regarding the formatting of the summaries, one question to ask is “Who are these summaries for?” or “What value do summaries add?” or “What questions do summaries answer?”.
E.g. a crucial one is “Should I read this?” / “Who would benefit from reading this?”. Currently these summaries answer that question implicitly, but you could instead / in addition ask ChatGPT to answer this question explicitly.
Regarding the summary tone, I currently find the summaries a bit dry, so I don’t find them particularly enjoyable to read. But I’m not sure if there’s a tone which is less formal without being less accurate.
Opportunities for collaboration
Maybe you could collaborate with the people from Type III Audio in some way? They generate automatic audio transcripts for new posts while you generate automatic text summaries; so there might be some potential for cross-pollination.
I vaguely recall there also being some efforts to auto-tag posts via LLMs; that’s another avenue for collaboration.
Beaut. Thanks for the detailed feedback!
I think these suggestions make sense to implement immediately:
add boilerplate disclaimer about accuracy / fabrication
links to author pages
note on reading time
group by tags
“Lesswrong” → “LessWrong”
The summaries are in fact generated within a Google sheet, so it does make sense to add a link to that
These things will require a bit of experimentation but are good suggestions:
Agree on the tone being boring. I can think of a couple of fixes:
Prompt GPT to be more succinct to get rid of low information nonsense
Prompt GPT to do bulletpoints rather than paragraphs
Generate little poems to introduce sections
Think about cross pollinating with Type III Audio
Hey cool, I tried doing something similar throwing in links to NotionAI, telling it to turn the text into a table, and then prompting a column for a summary. NotionAI is about $10/month and your API calls were around $5. Do you think at some point in scale it might be cheaper to use notion? Then maybe you can get adhoc table filtering for free?
Hm, I think Hamish’s estimate of the cost included a bunch of tinkering with the settings; I can see it going either way. Another thing I think is more important is flexibility to make code changes and iteratively improve—how do you feel notion would do with that? I’m curious to see what you managed to get on Notion if you’re willing to talk through it with us.
(BTW, are you the Patrick Liu who participated in the Stampy hackathon this past weekend?)