This seems like an excellent program. If the program is selective at all or you have outreach that is limited and could be varied randomly, have you thought of doing a randomized evaluation of some sort? I’ve realized that even when we’re working on the far future, there are often intermediate outcomes (e.g. changed attitudes) to focus on, and we should test our impacts on those as rigorously as possible.
Our conference does have an application process. To provide accommodation for participants, travel reimbursement, and events that allow participants to engage with speakers we can only have a limited number of participants. Right now, we seek to maximize the impact that the conference has by selecting participants who are likely to be leaders in the future. We’ve narrowed down criteria we believe are indicative of becoming future leaders.
That said, I do agree with your point and I worry about our ability to accurately predict who will become a leader in the future. There are many possible paths to becoming a leader in the future, and what you do in college may not reflect well what you’ll do in the future. Beyond that, applications are subjective and we likely have a high false negative rate. Some people are better at writing applications, and skill in writing applications doesn’t imply (I think) the applicant will be successful in the future.
We’ve considered doing randomized evaluation, but haven’t yet because we don’t know of a rigorous way to measure how we did selecting participants that will have an impact in the future. An impact in the future would, by construction, have to be measured years after the conference. We’d have to track randomized applicants and applicants we selected and have a way to measure their impact, which would be challenging if not infeasible. Right now, the best heuristic we have for identifying applicants that will be leaders in the future is by looking at what they’re doing now.
The act of having an application also provides self selection for applicants; many people have communicated to us that completing the application is a barrier to entry. The application itself selects for a certain level of interest in the conference (the interest required to complete the application).
We like to believe that this approach is better than random, but we do admit that it may not be.
This seems like an excellent program. If the program is selective at all or you have outreach that is limited and could be varied randomly, have you thought of doing a randomized evaluation of some sort? I’ve realized that even when we’re working on the far future, there are often intermediate outcomes (e.g. changed attitudes) to focus on, and we should test our impacts on those as rigorously as possible.
Our conference does have an application process. To provide accommodation for participants, travel reimbursement, and events that allow participants to engage with speakers we can only have a limited number of participants. Right now, we seek to maximize the impact that the conference has by selecting participants who are likely to be leaders in the future. We’ve narrowed down criteria we believe are indicative of becoming future leaders.
That said, I do agree with your point and I worry about our ability to accurately predict who will become a leader in the future. There are many possible paths to becoming a leader in the future, and what you do in college may not reflect well what you’ll do in the future. Beyond that, applications are subjective and we likely have a high false negative rate. Some people are better at writing applications, and skill in writing applications doesn’t imply (I think) the applicant will be successful in the future.
We’ve considered doing randomized evaluation, but haven’t yet because we don’t know of a rigorous way to measure how we did selecting participants that will have an impact in the future. An impact in the future would, by construction, have to be measured years after the conference. We’d have to track randomized applicants and applicants we selected and have a way to measure their impact, which would be challenging if not infeasible. Right now, the best heuristic we have for identifying applicants that will be leaders in the future is by looking at what they’re doing now.
The act of having an application also provides self selection for applicants; many people have communicated to us that completing the application is a barrier to entry. The application itself selects for a certain level of interest in the conference (the interest required to complete the application).
We like to believe that this approach is better than random, but we do admit that it may not be.