A Pen Knife and an Educated Thumb

A couple of weeks ago, I had the privilege of making a presentation on collection assessment at Rice University. During the presentation, I presented the experience of a paint expert from PPG - the best measurement of adhesion was "a pen knife and an educated thumb". My point being there are important things that can not be directly measured with numbers. Since then I have started reading Daniel Kahneman's "Thinking, Fast and Slow". I need to modify my thinking slightly.

Kahneman has a interesting view of expertise. Expert intuition is very good but has two major limitations:

• Experts don't know the limitations of their expertise. Either people don't know what they know until they know until someone else asks. Or like Raymond Babbit from the movie Rain Man, an expert is good one thing - counting cards, but doesn't understand the expertise only applies to a limited number of games of chance.
• Expertise needs to be built in a reasonably consistent environment. If the environment is extremely unreliable, all the expert will do is create a superstition. For example, I had a coworker several years ago, who had worked for a trading firm in New York. One morning he got up. He went to work. He read all the appropriate news sources. He made the same decisions based on the same types of information, and expected to make a nice profit for the firm. He lost $300 million. 

So, my example expert developed his skills over a long period of time, used consistent methods to observe on consistent environment and was able to develop intuition to process a complex situation accurately. I believe library services can be asses in the same way, though active observation. And observations a good supplement to the numbers.