A/B testing in general is great. For UI-related changes you generally want to run the experiment sticky per user, to reduce confusion and allow the time for users to adapt to changes. This does add statistical complexity, though, because one heavy user in an experimental treatment can have a large impact on aggregate statistics like “total number of comments per category”.
Happy to talk more about this if you’d find it helpful; this is an area I used to work in.
A/B testing in general is great. For UI-related changes you generally want to run the experiment sticky per user, to reduce confusion and allow the time for users to adapt to changes. This does add statistical complexity, though, because one heavy user in an experimental treatment can have a large impact on aggregate statistics like “total number of comments per category”.
Happy to talk more about this if you’d find it helpful; this is an area I used to work in.