This is an awesome response, thank you Johannes—especially while you’re on leave! I hope it didn’t take eat too much into your time off.
Obviously, there are inevitably follow up questions/other queries that come to mind, but your answer is an exceptionally thorough and concise account of your reasoning, that massively increases my confidence (coming from a place of naivety) in this grant and your decision process in general.
It makes me wonder if there might be some middle ground where such an account could be provided, in the interim before publishing a more extensive analysis, when grants are made—especially since you seem to have been able to put this together (I hope) relatively quickly. Take this as a humble passing thought, however, you, of course have a much better understanding of the relevant constraints/considerations.
Thanks again for giving such a great answer so quickly, and enjoy the rest of your time off!
you seem to have been able to put this together (I hope) relatively quickly.
Johannes is being polite, which is a good response to public criticism, but as someone unrelated I’m going to be more blunt. You looked into a decision an org made and ended up with a lot of questions. Instead of asking the org or running a draft by them so they could prepare a response you posted your questions publicly in a way that looks a lot like an accusation of corruption. Practically, the organization needs to respond as soon as possible or many people will see your post, some will downgrade their view of the org, and most will never see the follow-up. That he was able to assemble something so quickly while on leave mostly speaks to the (unnecessary) urgency you gave to this situation, and not to how easy the task was. His response probably also skipped some steps grantmakers commonly have in writing publicly about their decisions, like running it by the grantee for accuracy.
As a positive example, I think the recent critical post
Why I don’t agree with HLI’s estimate of household spillovers from therapy handled this well: if James had published that publicly on a Sunday night with no warning then HLI would have been scrambling to put together a response. Instead, James shared it in advance and we got a much more detailed response form HLI, published at the same time as the rest of the piece, which is really helpful for outsiders trying to make sense of the situation.
While I agree it would have been significantly better to send this to the org ahead of time, I think on the margin I really wish we had more random spot-checks and discussions of org decisions, and still prefer seeing a post that puts an accidentally heavy burden on the org than not seeing one at all.
I do think this is a hard balance, and as someone whose writing motivation is far stronger with the reward of immediate publication, it’s one I’ve struggled with and probably one I’ve gotten this wrong in both directions. A norm of sharing a draft ahead of time and giving them, say, a week to prepare a response if they want, though, seems pretty good?
I’m starting to think posts should get a pinned mod comment if the poster doesn’t assert that the person/organization had a specified amount of advance notice. That could be a tricky norm to define, as there can be valid reasons not to provide advance notice (e.g., breaking news or a situation where delay could risk clearly identifiable harm), and it’s not trivial to define with precision what type of posts warrant an advance-notice norm. I’m not envisioning a hostile pinned comment, but I am wondering if there should be an “official” statement that says something along the lines of: “we don’t delete criticisms that were not shared with the person/organization in advance, but—at least absent special circumstances—no one should expect a prompt response where the poster chose not to share the post in advance.”
It was always my intention to avoid criticism and merely put questions to FP. It is difficult to defend myself here without now straying into criticism, which I am also disinclined towards (particularly publicly for the reasons you state), and because it would seem particularly unfair given Johannes’ sincere engagement with those questions.
I therefore make note of a few things that were relevant to my decision: - It has been ~nine months since the grant was made. - The grant was for a large amount of money. - Near zero information is publicly available regarding the decision making for the grant, or for the grantee. - In comparison to the HLI case, the post was made on the basis of a lack of information, not on the basis of detailed information made publicly available. - In comparison to the HLI case, the post took the form of a request for further information, rather than a criticism of publicly available information.
Obviously, the timing of the researcher responsible being on leave was unanticipated and unfortunate. Happy to take on board criticism/feedback regarding whether questions should be put privately first/notice should be given (particularly from Johannes), but I’ll also just note that, obviously, I’m just some guy. The administrative burden of having to follow up with an org seems greater than should be required (though perhaps it should be preferred) for mere questions about funding decisions on a public forum, in my view.
Along the lines of Ben’s comment, this seems like it would be a disincentive for questions to be asked regarding these decisions, and my personal view is that many more questions should be asked. Not because there are necessarily any issues, but because facing questions and giving transparent answers increases credibility, and makes the org stronger and more effective.
I don’t think that any of those justify not sending either your questions or a writeup of the post to the org in advance. They have a public email address. It’s at the bottom of their home page. I don’t think it’s a particularly excessive burden to send a copy once you’re done and give them a week. Perhaps two if they apologize and ask for a bit more time. I understand why people might be suspicious at the moment, but forcing people to scramble while on vacation is not a good norm. As you say, this post clearly wasn’t that time-sensitive. I don’t think that the Forum should have taken your post down, but that’s a much higher bar.
For comparison, when I posted a piece that was somewhat critical of CEA’s admissions and transparency policies, it was after I had asked in a more private Slack channel and gotten an answer I was not satisfied with. You can see that they clarified that they did inform people, and that others chimed in to thank me for informing them with the post.
Looking at this comment after Nonlinear, I think it holds up. There exists a point at which an org loses the (moral, not legal) right to see questions / a writeup in advance, and Nonlinear was past it. Legal threats, contacting the people you spoke with, and contacting your employer are classic examples of this. I am also sympathetic to journalists covering industries that are known to react strongly, such as oil and tobacco. But the items in the list you provide do not come close to the bar of the org being untrustworthy, and that is the bar I think must be cleared.
This is an awesome response, thank you Johannes—especially while you’re on leave! I hope it didn’t take eat too much into your time off.
Obviously, there are inevitably follow up questions/other queries that come to mind, but your answer is an exceptionally thorough and concise account of your reasoning, that massively increases my confidence (coming from a place of naivety) in this grant and your decision process in general.
It makes me wonder if there might be some middle ground where such an account could be provided, in the interim before publishing a more extensive analysis, when grants are made—especially since you seem to have been able to put this together (I hope) relatively quickly. Take this as a humble passing thought, however, you, of course have a much better understanding of the relevant constraints/considerations.
Thanks again for giving such a great answer so quickly, and enjoy the rest of your time off!
Johannes is being polite, which is a good response to public criticism, but as someone unrelated I’m going to be more blunt. You looked into a decision an org made and ended up with a lot of questions. Instead of asking the org or running a draft by them so they could prepare a response you posted your questions publicly in a way that looks a lot like an accusation of corruption. Practically, the organization needs to respond as soon as possible or many people will see your post, some will downgrade their view of the org, and most will never see the follow-up. That he was able to assemble something so quickly while on leave mostly speaks to the (unnecessary) urgency you gave to this situation, and not to how easy the task was. His response probably also skipped some steps grantmakers commonly have in writing publicly about their decisions, like running it by the grantee for accuracy.
As a positive example, I think the recent critical post Why I don’t agree with HLI’s estimate of household spillovers from therapy handled this well: if James had published that publicly on a Sunday night with no warning then HLI would have been scrambling to put together a response. Instead, James shared it in advance and we got a much more detailed response form HLI, published at the same time as the rest of the piece, which is really helpful for outsiders trying to make sense of the situation.
While I agree it would have been significantly better to send this to the org ahead of time, I think on the margin I really wish we had more random spot-checks and discussions of org decisions, and still prefer seeing a post that puts an accidentally heavy burden on the org than not seeing one at all.
I do think this is a hard balance, and as someone whose writing motivation is far stronger with the reward of immediate publication, it’s one I’ve struggled with and probably one I’ve gotten this wrong in both directions. A norm of sharing a draft ahead of time and giving them, say, a week to prepare a response if they want, though, seems pretty good?
[EDIT: expanded this into a post]
I’m starting to think posts should get a pinned mod comment if the poster doesn’t assert that the person/organization had a specified amount of advance notice. That could be a tricky norm to define, as there can be valid reasons not to provide advance notice (e.g., breaking news or a situation where delay could risk clearly identifiable harm), and it’s not trivial to define with precision what type of posts warrant an advance-notice norm. I’m not envisioning a hostile pinned comment, but I am wondering if there should be an “official” statement that says something along the lines of: “we don’t delete criticisms that were not shared with the person/organization in advance, but—at least absent special circumstances—no one should expect a prompt response where the poster chose not to share the post in advance.”
Edit: typo
It was always my intention to avoid criticism and merely put questions to FP. It is difficult to defend myself here without now straying into criticism, which I am also disinclined towards (particularly publicly for the reasons you state), and because it would seem particularly unfair given Johannes’ sincere engagement with those questions.
I therefore make note of a few things that were relevant to my decision:
- It has been ~nine months since the grant was made.
- The grant was for a large amount of money.
- Near zero information is publicly available regarding the decision making for the grant, or for the grantee.
- In comparison to the HLI case, the post was made on the basis of a lack of information, not on the basis of detailed information made publicly available.
- In comparison to the HLI case, the post took the form of a request for further information, rather than a criticism of publicly available information.
Obviously, the timing of the researcher responsible being on leave was unanticipated and unfortunate. Happy to take on board criticism/feedback regarding whether questions should be put privately first/notice should be given (particularly from Johannes), but I’ll also just note that, obviously, I’m just some guy. The administrative burden of having to follow up with an org seems greater than should be required (though perhaps it should be preferred) for mere questions about funding decisions on a public forum, in my view.
Along the lines of Ben’s comment, this seems like it would be a disincentive for questions to be asked regarding these decisions, and my personal view is that many more questions should be asked. Not because there are necessarily any issues, but because facing questions and giving transparent answers increases credibility, and makes the org stronger and more effective.
I don’t think that any of those justify not sending either your questions or a writeup of the post to the org in advance. They have a public email address. It’s at the bottom of their home page. I don’t think it’s a particularly excessive burden to send a copy once you’re done and give them a week. Perhaps two if they apologize and ask for a bit more time. I understand why people might be suspicious at the moment, but forcing people to scramble while on vacation is not a good norm. As you say, this post clearly wasn’t that time-sensitive. I don’t think that the Forum should have taken your post down, but that’s a much higher bar.
For comparison, when I posted a piece that was somewhat critical of CEA’s admissions and transparency policies, it was after I had asked in a more private Slack channel and gotten an answer I was not satisfied with. You can see that they clarified that they did inform people, and that others chimed in to thank me for informing them with the post.
Looking at this comment after Nonlinear, I think it holds up. There exists a point at which an org loses the (moral, not legal) right to see questions / a writeup in advance, and Nonlinear was past it. Legal threats, contacting the people you spoke with, and contacting your employer are classic examples of this. I am also sympathetic to journalists covering industries that are known to react strongly, such as oil and tobacco. But the items in the list you provide do not come close to the bar of the org being untrustworthy, and that is the bar I think must be cleared.
Do you mean Leverage or Nonlinear?
Nonlinear, thank you. Edited.