[Stats4EA] Expectations are not Outcomes

This is the first in what might be­come a bunch of posts pick­ing out is­sues from statis­tics and prob­a­bil­ity of rele­vance to EA. The for­mat will be in­for­mal and fairly bite-size. None of this will be origi­nal, hope­fully.


Ex­pec­ta­tions are not outcomes


Here we at­tempt to trim back the in­tu­ition that an ex­pected value can be safely thought of as a rep­re­sen­ta­tive value of the ran­dom vari­able.


Si­tu­a­tion 1


A Rademacher ran­dom vari­able X takes the value 1 with prob­a­bil­ity 12 and oth­er­wise −1. Its ex­pec­ta­tion is zero. We will al­most surely never see any value other than −1 or 1.


This means that the ex­pected value might not even be a num­ber the dis­tri­bu­tion could pro­duce. We might not even be able to get ar­bi­trar­ily close to it.

Imag­ine walk­ing up to a table in a cas­ino and bet­ting that the next roll of a die will be 72.


Si­tu­a­tion 2


Re­searchers cre­ate a nat­u­ral lan­guage simu­la­tion model. Upon re­ceiv­ing a piece of text as stim­u­lus it out­puts a ran­dom short story. What is the ex­pec­ta­tion of the story?


Let’s think about the first word. There will be some im­plied prob­a­bil­ity dis­tri­bu­tion over a dic­tio­nary. Its ex­pec­ta­tion is some frac­tional com­bi­na­tion of ev­ery word in the dic­tio­nary. What­ever that means, and what­ever it is use­ful for, it is not the start of a leg­ible story—and should not be used as such.


What is the ex­pected length of the story? What would a solu­tion to that prob­lem mean? Could one, for ex­am­ple, print the ex­pected story?


Si­tu­a­tion 3


Distri­bu­tions with very fat tails. For in­stance, the Cauchy dis­tri­bu­tion has an un­defined ex­pec­ta­tion.


Implication


It is tempt­ing to freely sub­sti­tute an ex­pec­ta­tion in as a rep­re­sen­ta­tive of a ran­dom vari­able. Sup­pose we used the fol­low­ing pro­ce­dure in a blan­ket fash­ion:

  1. We are faced with a de­ci­sion de­pend­ing on an un­cer­tain out­come.

  2. We take the ex­pected value of the out­come.

  3. We use the ex­pec­ta­tion as a sce­nario to plan around.

Step three is un­safe in prin­ci­ple—even if some­times not in prac­tice.

If there is a next time (the length of this se­ries is cur­rently frac­tional) I hope to touch on some sce­nar­ios less eas­ily dis­missed as the con­cerns of a pedant.