1) >”The numbers on how useful things are seem quite low to me...”
On the scale 1 was “Useless” and 10 was “Life-transforming”. But just before asking for feedback, I made a change in the slides and added this meaning to the ratings of the events:
“3 - £100, 5 - £1,000, 8 - £10,000, 10 - £100,000 (e. g. career change)”
I explained it to people as well. This was… not smart. Because of this, some respondents gave low scores to all the events. E.g. someone said that the weekend was “Far more valuable (10-30x the counterfactual)” but did not gave any event a rating that is higher than 4. Others ignored the point and gave high ratings for all events.
That’s why I weighted and normalised the ratings. If someone said that the weekend was “Vastly more valuable (>30x counterfactual)”, I multiplied all their ratings by a constant so that their highest rating would be 10. If they rated the weekend as “Far more valuable (10-30x the counterfactual)”, I multiplied all their ratings so that the highest rating would be 9. 8 for “Much more valuable”, 7 for “Somewhat more valuable”, and 6 for “About as valuable”.
Yeah, I thought the ends of the scales might have been more extreme than we’d normally use. It’s probably quite hard to get people to sensibly answer unfamiliar, tricky questions.
1) >”The numbers on how useful things are seem quite low to me...”
On the scale 1 was “Useless” and 10 was “Life-transforming”. But just before asking for feedback, I made a change in the slides and added this meaning to the ratings of the events:
I explained it to people as well. This was… not smart. Because of this, some respondents gave low scores to all the events. E.g. someone said that the weekend was “Far more valuable (10-30x the counterfactual)” but did not gave any event a rating that is higher than 4. Others ignored the point and gave high ratings for all events.
That’s why I weighted and normalised the ratings. If someone said that the weekend was “Vastly more valuable (>30x counterfactual)”, I multiplied all their ratings by a constant so that their highest rating would be 10. If they rated the weekend as “Far more valuable (10-30x the counterfactual)”, I multiplied all their ratings so that the highest rating would be 9. 8 for “Much more valuable”, 7 for “Somewhat more valuable”, and 6 for “About as valuable”.
Yeah, I thought the ends of the scales might have been more extreme than we’d normally use. It’s probably quite hard to get people to sensibly answer unfamiliar, tricky questions.