There are some questions that stubbornly refuse to succumb to popular analysis, “How long is a piece of string”, “Is that data or meta-data?” and today I’d like to propose another, specific to business rules, “Is that policy or meta-policy?”

I informally define meta-policy as, “Business rules (captured explicitly, or implicit in an application implementation) that are used to determine the set of business rules to apply to an incoming transaction.”

Imagine an insurance company that offers products to customers across several countries or states. When a request for an insurance quote is made by a prospective customer through the insurance company’s web portal the automated quoting system has to determine the correct pricing rules to fire (activate) based on the characteristics of the customer. Perhaps some surcharge pricing rules are specific to high crime states while other rules provide a discount to female customers over the age of 45. The goal of the BRMS is to allow the Policy Manager to easily express and manage the rules for the insurance products, as well as execute the correct rules at runtime.

Clearly there is one logical set of real business rules however the rules may be implemented using several possible physical strategies. Typically these strategies are determined by a pragmatic trade-off between manageability, extensibility, performance as well as how the insurance company has traditionally talked about and managed their business. I hope some examples will clarify!

Consider the following very simple Action Rule created using ILOG JRules:

The rule has been placed within a package (folder) called surcharge:

Nice and simple and atomic isn’t it? However as the number of states increases it is quite likely that the Policy Manager will want to organize the business rules by geography, allowing them to quickly view the rules that are specific to New York. However the information about “New York” is then duplicated: once in the package hierarchy and once within the condition of the rule. E.g.

The Policy Manager does not like the fact that they have to explicitly add the “the state of customer is "New York” for rules in the New York package as they rightly believe that it is redundant with the package organization, as well as error prone if they should copy/paste or move a business rule. We’ve fallen back into the “data vs. meta-data” debate! The package organization of a rule is meta-data whereas the rule condition is data. It is extremely useful to be able to use the package meta-data as an implicit condition on the rules. Of course one of the drawbacks is that it may be too implicit, detracting from the atomicity and reusability of the rule outside its owning package.

I will now discuss some ILOG JRules mechanisms and patterns that can be useful when dealing with meta-policy.

Determining the rules within a ruleset

First however I must digress a little and explain how you can specify the contents of an executable ruleset archive. By default all the rules within a Rule Project are placed within the ruleset archive. However in some situations it can be useful to filter or select the rules based on their properties (meta-data) using a query:

Here “effective date” is an out-of-the-box rule property; however any custom property can be used in rule queries. Using the query above only the rules within the “New York” package would be extracted to the generated ruleset archive.

Explicit rule conditions

As I stated earlier, the natural thing to do is to capture all rule conditions explicitly on the left-hand-side of the rule (in the “if” part). This has the very nice property that it likely makes the rules atomic so they can be safely moved around the Rule Project and a single rule can be read and analyzed independent of its context. The drawback is that if there are many repeated tests (but the rules are not amenable to a decision table or tree) the rules can appear verbose and somewhat clumsy.

Rule overriding

It is often the case that the meta-policy states that under certain conditions a group of rules should override or take precedence over another set of rules. This is commonly the case when dealing with a hierarchical property such as geographical regions for example. For this case JRules provides a declarative mechanism to specify that a rule or decision table/tree should override a set of rules or decision tables/trees.

Ruleflow

For relatively simple routing of a transaction to rules it is possible to use a Ruleflow with a Choice Node and a guard condition. The ruleflow implements the meta-policy, while the rules in the “New York” and “Maine” packages no longer have to test the state of the customer as it has already been performed by the ruleflow.

Taken too far however this approach can lead to maintenance problems as the procedural Ruleflows can get very complex and hard to maintain, even if broken into sub-flows. Large ruleflows can also become relatively expensive to execute as they must typically be evaluated sequentially from start-to-finish as compared to a declarative pattern-matching based approach using rules.

Multiple rulesets

One can be even more drastic and decide to create separate Rule Projects for each state of interest, perhaps using Rule Project dependencies to a sub-project to import rules that are independent of state. Alternatively one could use a set of queries to build different rulesets from a single Rule Project, filtering the rules based on their meta-data.

At runtime the application first determines the correct ruleset to execute, based on the customer’s state, and then invokes the Rule Execution Server or rule engine. Execution may be moderately faster because the state of the customer is no longer tested at all within the ruleset.

Deploying multiple rulesets can also be useful is there are a very large number of rules that are extremely disjoint or that have different life-cycles or Quality of Service objectives. A frequently updated ruleset might be split out from a very large ruleset to improve deployment and parsing time for example.

A Rule Execution Server Ruleset Interceptor can be used to register a Java class that is automatically called prior to invoking the ruleset. The interceptor can determine the correct ruleset to call (routing) based on data within the incoming transaction as well as meta-data on the deployed rulesets.

Rule property

One may also decide to use a custom property associated with business rules. Adding a custom property “state” to every rule, avoiding the need to test the state condition on every state specific rule. The “state” property will safely follow the rule as it is moved or copied between packages. The state property can be used by a query that builds the ruleset, for runtime rule selection or rule rewriting (see below).

Package name

The package name is often used for meta-policy (either using a query to build the ruleset, a ruleflow to route between packages or using runtime rule selection). Although very convenient, implementing meta-policy using rule packages requires some care, as the rules within a package may now rely on an implicit condition.

Runtime rule selection

Runtime rule selection is a very powerful mechanism that allows the rule engine to determine the rules within a rule task at runtime. The ruleflow definition becomes trivial:

While the definition of the “State Rules” task contains the Runtime Selection logic:

The “dynamic” checkbox can be used to ensure the rules eligible for a task are recomputed each time the task is entered. For rule tasks that do not use dynamic runtime rule selection the computation is performed once, when the rule flow is instantiated by the rule engine.

Rule rewriting

Rule rewriting is an advanced mechanism to implement meta-policy. By customizing the translation of Action Rules to executable IRL rule engine code you can automatically insert conditions or actions into the rules based on their meta-data. I would not typically recommend this option as it is relatively difficult to implement and is very opaque to the business users and developers that have to maintain the business rules application.

Agenda filter

The agenda is another advanced mechanism that has historically been used in some cases to implement meta-policy. Agenda filters are a procedural “hook” of last resort that allows custom code to prevent a rule from firing that has been selected by the rule engine (i.e. all the rule’s conditions are true). The drawback with agenda filters is that they are again very opaque to the business users of the rules, require either IRL or Java code to implement, and are not optimal from a performance perspective, because the rule engine has already done all the work to determine the rule should be fired, only to have that decision overruled by the agenda filter. One of their advantages is that they operate upon rule instances allowing a single rule to be potentially prevented from firing based on the state of the objects that caused the rule to become matched, or even the other rules that have been selected for execution and are present in the agenda.

Summary

I’ve tried to summarize the JRules meta-policy options in the table below. Like most generalizations it doesn’t stand up to too close scrutiny but I hope it will kick start some discussion on how to best determine the rules to apply to incoming transactions, while making smart trade-offs between manageability, performance and dynamicity. Note that these patterns are not mutually exclusive — you can combine the extraction of multiple rulesets with runtime rule selection within each ruleset for example.

We are lucky enough to employ some of the smartest rule engine minds in the industry, so last week I was not too surprised to hear that in JRules 6.x the rule engine team have raised the performance bar yet again. We run many performance tests, but the ones people seem to get most excited/obsessed about are the so-called “academic benchmarks”. Frankly I don’t like these benchmarks very much because they are not representative of the types of problems customers are trying to solve using business rules. Anyone doing N-dimensional graph labeling using rules? Assigning guest seating for a dinner party? Calculating the Fibonacci series? I didn’t think so…! To compound matters the results of some of the academic benchmarks are difficult to verify, leading to performance comparisons between buggy implementations or inadvertent over-optimization.

As an aside, I think it would be much more useful to take a simplified customer problem as the basis for a benchmark. The functional characteristics of the problem would be described, and then all the vendors could provide a solution. Solutions would be graded not purely on performance (my assembler is faster than your Java!), but also on subjective criteria such as the business expressiveness of the rules, the flexibility of the solution to evolution and change etc. In some cases this would also allow vendors to showcase their specialized execution algorithms, such as Sequential or Fastpath from ILOG. I will return to this subject in a future blog post.

To make the results more interesting (and to provide some basis for comparison) today I ran the Waltz and Manners academic benchmarks against Drools 4.0.1 as well as JRules 6.6.1 Trial. As you can see in the table and charts below the results are very encouraging for JRules users, however I’d also like to tip my hat to Mark Proctor and Co. as Drools 4 is certainly moving in the right direction performance-wise. The Drools team is doing a good job at evangelizing business rules to Java developers and we hope to also do our part with the newly launched BRMS Resource Center. We welcome you to recreate these results in your own environment by downloading a trial version of JRules from the BRMS Resource Center. Of course, if you have optimization tips for improving either the Drools or JRules results I’d love to hear from you.

The Manners results show broadly the same performance profile for both products, with JRules between 50% and 150% faster than Drools. Note that for reasons I don’t yet understand the Drools engine fires slightly more rules than JRules for Manners 512, while the two engines fire the same number of rules for the other Manners benchmarks.

The results for Waltz are very different, with JRules showing a clear algorithmic advantage for this benchmark. Again, the two engines do not agree on the number of rules that should be fired.

Naturally our Rules for .NET customers will also experience improved rule engine performance as the vast majority of the rule engine code is shared between JRules and Rules for .NET.

An Excel spreadsheet with the results I recorded is available here. After you’ve downloaded the JRules Trial you can run these benchmarks by downloading the ZIP file here.

Today ILOG launched the BRMS Resource Center, along with this blog. I hope that the BRMS Resource Center will become your one-stop-shop as you evaluate business rules technology in general, and ILOG JRules or Rules for .NET in particular. We’ve pulled together a great initial collection of material as well as making it much easier for you to download and evaluate our BRMS products at your own pace.

I will use this blog to give you a “sneak peek” at some of the amazing innovations being worked on within ILOG Rules R&D, as well as shining a light on technology, tools and best/worst practices that you can use with ILOG’s BRMS products. It should be a fun ride!

I just got back from the SOA Governance Summit in beautiful Copenhagen. The event was organized flawlessly by our partner Software AG. Software AG recently acquired webMethods, so this was an opportunity for them to explain the product integration strategy to their partners and customers, as well as raise the visibility of the CentraSite Community and the governance challenges inherent in an SOA approach to IT architecture. L. Frank Kenney from Gartner gave a great presentation (he took no hostages!) that underscored the importance of incorporating governance within an overall SOA strategy.

I gave a short presentation on the value of Transparent Decision Services within an SOA, and illustrated how CentraSite can be used as a service registry to store the WSDL for Transparent Decision Services, as well as a repository to store the rich rule-specific metadata required to encourage service reuse.

You can download my presentation here.

I’ve always been very interested in the interface between people and technology, and I’ve worked on a couple of large products that simultaneously targeted several different user roles. A BRMS is probably one of the most challenging of such products because the list of user roles spans developers, policy managers (aka line of business managers), system administrators, business analysts as well as others. Each one of these roles brings a unique culture, set of expectations, education and toolset bias to the BRMS. Some vendors attempt to cram all users into a “one-size-fits-all” model, essentially ignoring the needs of one or more user roles. The poster-child for this approach is the plethora of tools for developers, of course built by developers, that attempt (and usually fail) to make inroads amongst members of the “business” roles. There are many examples within the BPM and BRMS communities. The converse is also true - tools that although very popular with business analysts fail to gain acceptance from IT more generally.

You can usually see the heritage of a successful product by looking at how they manage and store their artifacts within their repository. There are essentially two models for this: text-file based or database-based. Developer-centric products tend to favour text-files while business-centric products tends to favour database storage. There are pros to both approaches:

Text Files

• Everything is “code” or “configuration”
• Integrates well with configuration management systems
• Integrates well with text-based tools (search, print, find/replace, difference)
• Changes are isolated
• Easy to work offline (disconnected)
• Possible to automate changes
• Easy to perform ad-hoc searches

Database

• Everything is “data”
• Relational integrity between related artifacts
• Easy deployment (always deployed!)
• Easily shared with team members
• Easy to define security policy and permissions
• Powerful/performant query model
• Integrates well with reporting and business intelligence tools

Think Windows registry vs. Linux configuration files, or mbox email files vs. Microsoft Outlook PST file.

Unfortunately, as you may have seen, it is very hard (perhaps impossible?) to build a single repository that combines the best of both the file-based and database development models. In addition, one is also faced with a workflow challenge: do I want to work alone on repository artifacts or do I want to collaborate with my peers? If I want to collaborate with my peers how frequently do I want to exchange information with them, and how do I want my work to impact theirs? If you and I are collaborating on a set of resources then we can either take an “optimisitc” view of the world or a “pessimistic” view. I.e. I can assume you will not modify my resources or I can assume you will. In technical terms I may wish to take the pessimistic view and “lock” a resource I am editing to prevent you making changes, or I may wish to let you go ahead and make your changes concurrent with mine, postponing the decision on how to reconcile the two sets of changes until the last person has completed their change and the impact of the changes is better understood.

I remain convinced that the solution to this problem is to provide two repository implementations, one tuned for developer-style use cases using an optimisitc locking model and workspace isolation, the other optimized for business-style use cases, using a pessimistic locking model and a shared workspace. The challenge then literally becomes how do you synchronize these IT and business workspaces to ensure cross-role collaboration? The solution requires a robust synchronization framework that can detect modified artifacts in each workspace, prompt the user to reconcile potential differences, and move artifacts from one workspace to the other.

If you are familiar with the JRules 6.x architecture this should sound familiar: Rule Studio uses a file-based persistence model and workspace isolation, while Rule Team Server uses a database persistence model and a shared workspace for Policy Managers. In between we use a synchronization module to move artifacts to and from the two environments. No compromise BRMS indeed!

This week I am working with the Rule Studio team and the Product Managers on fleshing out and prioritizing the User Stories we will target for our JRules 7 release. I recently moved back into R&D, after over four years as a JRules Product Manager, so it is going to be interesting to see the release planning process from a developer’s perspective this time around.

If you are not familiar with Agile development methodology, User Stories are the central communication tool between product managers and the technical team working on a release. A User Story is a short (a paragraph or two) description of a usage scenario from a specific user’s perspective. I’ve found User Stories to be more useful than prescriptive requirements as they conveys the “What” and “Why” of user interaction with the product, whereas requirements inevitably get bogged down in the details of “How”. Sometimes you need a mixture of both, but even then a good user story can help frame a set of requirements within an overall context. User stories also reinforce to developers that they are building software for someone other than themselves!

For example, below is one of the simple stories that motivated changes to how JRules Rule Studio supports Rule Execution Server in JRules 6.6.

Quick Publish of RuleApps

Cynthia uses Rule Studio to develop rules for a RES-based application. She tests her application frequently; running a client deployed to Tomcat as well as using JUnit tests. Every time she builds within Rule Studio her RuleApp is automatically published to her RES server instance over HTTP. In addition she may choose to automatically publish the RuleApp to the file system.

User Stories are not the result of a product manager’s fertile imagination - most are the result of interviews with real customers or internal users such as pre-sales engineers, R&D, support or professional services. The User Stories are prioritized using the weight of evidence backing each story. Frequently requested or strategic stories are prioritized higher than more rarely encountered stories. The art of a good product manager is being able to come up with a set of coherent User Stories from the diverse comments collected during interviews. For a given release it is also common to have an overall theme that groups the User Stories (e.g. Business User Empowerment) or conversely to ensure that the selected User Stories provide something for a range of product stakeholders.

Next week we will perform the “estimation games” with the product managers. During an estimation game each development team votes to determine an initial rough estimate for the development effort for each user story. These rough estimates are then used to select the user stories in scope for a release, and ultimately to draw up a tentative iteration schedule. One of the challenges with this approach is to estimate the user stories that cross team boundaries and to determine the technical dependencies between user stories. I’m not sure we’ve found the optimal approach yet for cross-team stories, so I’d welcome your tips.