As long as we keep asking Alice and Bob questions via the same platform, and their incentives don’t change, I agree. But if we now need to decide whether to hire Alice and/or Bob to do some forecasting for us, comparing their average daily Brier score is problematic. If Bob just wasn’t motivated enough to update his forecast every day like Alice did, his lack of motivation can be fixed by paying him.
As long as we keep asking Alice and Bob questions via the same platform, and their incentives don’t change, I agree. But if we now need to decide whether to hire Alice and/or Bob to do some forecasting for us, comparing their average daily Brier score is problematic. If Bob just wasn’t motivated enough to update his forecast every day like Alice did, his lack of motivation can be fixed by paying him.