One of the benefits of working during the August lull is that one gets to stretch the brain with ideas that are typically outside the realm of the day-to-day. One example I came across this week is a fascinating article on the psychology of security written by Bruce Schneier. The article was discussed on an IT security mailing list, however Schneier writes essentially about the psychology of risk. A large part of the job of a Product Manager deals with evaluating risk, from analyzing competitors’ behavior, to SWOT analysis of market trends, estimating the impact of product defects as well as the potential for project slippage.

While we all think we are so reasonable, all the experimental evidence points to the contrary. Human decision making is extremely complex and may be influenced by many irrational factors. Ed: Not mine, however, obviously!

Dealing with risks is always about making the right trade-off. Schneier states that the following five aspects are vulnerable to poor decision making:

1. The severity of the risk.
2. The probability of the risk.
3. The magnitude of the costs.
4. How effective the countermeasure is at mitigating the risk.
5. The trade-off itself.

The essay is full of fascinating information, but I pulled out a couple of nuggets for reference.

Availability Heuristic

Considering a particular outcome in one’s imagination makes it appear more likely later.

The moral here is that people will be persuaded more by a vivid, personal story than they will by bland statistics and facts, possibly solely due to the fact that they remember vivid arguments better.

An outcome that is difficult to imagine may actually appear to be less likely.

The most extreme example of a class of things tends to come to mind when thinking about the class.

I have seen the availability heuristic at work in myself, and dare I say, my colleagues? After a customer visit, or a discussion with a pre-sales engineer, I tend to prioritize their wishes/bug reports higher than I might otherwise, even if this is the only customer to register this particular product wish. This behavior is also typical when dealing with internal and external users of the product - they tend to prioritize their current issues much higher, even if they saw more severe issues in a project 3 months ago. To counter this tendency Product Managers collect “evidence” for wishes (from customer visits and the votes within the wish database) and try to use this information for rational decision making.

Long-range planning is also particularly sensitive to the availability heuristic. When making a three year strategic plan it is very easy to be unduly biased by the competitive event that happened last week.

Anchoring Effect

1. Question 1: Should divorce in this country be easier to obtain, more difficult to obtain, or stay as it is now?
2. Question 2: Should divorce in this country be easier to obtain, stay as it is now, or be more difficult to obtain?

In response to the first question, 23% of the subjects chose easier divorce laws, 36% chose more difficult divorce laws, and 41% said that the status quo was fine. In response to the second question, 26% chose easier divorce laws, 46% chose more difficult divorce laws, and 29% chose the status quo. Yes, the order in which the alternatives are listed affects the results.

Confirmation Bias

People are more likely to notice evidence that supports a previously held position than evidence that discredits it. Even worse, people who support position A sometimes mistakenly believe that anti-A evidence actually supports that position.

Summary

So perhaps by coming to a deeper understanding of how our brain reacts to risks we can make better decisions? We may also be better equipped to detect when unscrupulous people are attempting to manipulate our perception of risk to influence our decision making. Buddhists call this approach to life Mindfulness.

Daniel Gilbert is also eminent in this field. He wrote the article “If Only Gay Sex Caused Global Warming” which must be one of the best headlines ever!

Caveats

I feel compelled to remind you that in this post I am presenting results from benchmarks. Benchmarks are not applications and will not reflect the performance profile of the application you are designing. Below I am also only reporting average rule engine execution time. There are many other factors that may be important to you, such as ruleset parsing time or JVM memory usage.

Benchmarks

I spent today re-running some old performance tests against JRules 5.1.0 (yes, that old thing!). There have been various reports of performance improvements in JDK 6, so I wanted to try to quantify them for JRules. I ran four benchmarks I have used in the past:

• ManyObjects: a very working memory intensive benchmark that inserts 1M objects into working memory and processes them using 7 rules
• PatternMatching: uses advanced Rete constructs to correlate 100 accounts and 400 account events using 15 rules
• Sequential: executes 5959 sequential rules against a single input parameter, a single rule fires per execution
• XmlBinding: processes two XML documents using ILOG XML binding and 401 rules

I tested using the JDKs I had at hand (Sun, IBM and BEA) using a single machine configuration (my Dell D620 core-duo laptop running Windows XP). All benchmarks were run on untuned JVMs, just setting 512 MB as the maximum heap size.

Summary

The chart above shows the average execution time over 10 runs, with the first run discarded. The vertical axis is in milliseconds (smaller is better). You can clearly see that IBM JDK 1.4.2 and Sun JDK 1.5.0_08 both have significant performance problems with these benchmarks. By just upgrading your Sun JDK from patch level 8 to patch level 11 you will see a huge performance improvement (with these benchmarks). In general terms however you can see the JVM vendors tuning and optimizing their implementations and IBM, BEA and Sun are all playing in the same ballpark once we get to JDK 5/6.

So, impressive work by Sun on JDK 6:

• +49% for Sequential
• +20% for PatternMatching
• +17% for ManyObjects
• +24% for XmlBinding

JRules 6.7 will officially support JDK 6.

Details

If you look at the individual benchmark results you can see how the different JVM vendors have optimized for different use cases.

The ManyObjects benchmark performs very well on JRockit. JRockit must be optimized for memory allocation and have an efficient default garbage collector for this benchmark.

The PatternMatching benchmark also performs well on BEA JRockit.

The Sequential benchmark is very interesting because it shows that JRockit is (still) poor at handling dynamically generated classes. In comparison the Sun JDK performs very well with this sequential mode benchmark.

IBM JDK 5 performed very well with ILOG XML binding.