Posted on Sat 19 June 2021

Raises interesting points about variance in decision making, but is unnecessarily verbose. The authors also seem to consider algorithmic decisions noise free for some reason - this is only true in a very literal sense (same output for exact same input image), and even then only for the simplest systems (as machine learning practitioners will be happy to tell you).

The variance reduction strategies recommended by the authors are useful however:

• focus on accuracy, not individual expression (what would another judge think, checklists, etc)
• start from the outside view, think statistically
• structure judgements into independent tasks, do not mix them until evaluating all parts
• sequence exposure to information to avoid premature intuitions (eg blind studies)
• ensemble multiple judges
• prefer relative scales and comparisons over absolute judgements

Tags: book, review, little-brown-spark