Business rules and rules engines, oh my

This is a bit of a meandering post, thoughts and ideas that I put down with regards to our use of a rules engine within the environment. For the past five years I was involved in initially using and later promoting the use of a third-party rules engine within the company that I am at (I became the rules ‘evangelist’ within the company due to the work that was done within the rules environment). The rules engine (no, not one of the open source ones) was not a big name player, however it was flagged as an ‘up and coming’ challenger in the business rules engine space. Unfortunately I am not sure whether one can use the name of the engine, The rules engine provided not only a RETE engine, but also some BRMS capability.

Unfortunately a couple of years ago the company that built the rules engine was bought out by a very large corporate (no, not the one that starts with ‘O’ and ends in ‘racle’. For a change it was another one – think ‘Tree fluid’). As it was a no-no to do business with said big conglomerate, the version of the rules engine has never been updated. However system development and investment in rules technology continued, with – at some stage – a conversion to another rules engine in the making.

So, over the past five years there has been a big learning curve when it comes to the business rules engine space. Before I started at my employer a business rules engine was something one would occasionally read about, where some seemingly lofty promises were made. I’ve played with Drools and Jess and dabbled with expert systems, thought to myself that it is sweet, and there would be some place to use it. But never really saw the need for one. And then I joined my current employer that were ‘using’ a rules engine in a very interesting(*) manner.

Rules were developed in Java code – originally in neatly segmented rules classes but with copy / paste sickness came a spread of the rules all through the code base. Some (yes, only some) of the rules were called from the rules engine as it could invoke methods on Java classes supplied to it. We thus had a system where a third party rules engine was called, and that immediately called out back to the very application that called it to execute logic and return the outcome back to the rules engine. The outcome would then be returned to the calling application that calculated it in the very first place… Yes, this did not make sense at all. Just because the rules engine that was being used could integrate nicely into a Java environment and execute Java code, does not mean that one should abuse it like this – there was no clear separation of concern, no use of the power of RETE.

As architect I faced a choice – either get rid of the rules engine, the superfluous calls to it, or use it properly. A project was identified to try out the rules technology and see whether it can assist us. The outcome of this was an embracing of the rules engine as a concept and after five years the rules engine has become a cornerstone of the architecture of the system. And no, rules are not simply ‘shells’ over Java code, a couple of simple standards have been instituted to combat this and to help us with our use of the rules engine.

What standards were introduced?

• Minimise the calling of pure Java code within the rules engine. Unfortunately we couldn’t get away completely from the ‘do not call Java code in the rules engine’ approach. However we try to keep it to a minimum, limiting it to convenience methods in order to make it easier to obtain data. What is not allowed though is – for example – manipulation of flows, saving (or retrieving) of data within a rule. The question that should arise out of this is how were the rules used then, if business requires a rule to direct information to a specific group? This leads to the next point…
  
• Rules determine outcomes only, it does not do more. The rules engine is one of the components of the system, and how could we facilitate communication between components, yet ensure that components do not necessarily know of each other, but can communicate with each other. What we did on the rules side is that the rules will define outcomes where an outcome has a unique identifier. The identifier is a grouping code that can then be used by other components as an input that they will react to. This allowed us to separate the business rules and the effect (outcome) of the business rules from the actual components that will react to and do the hard work. It also means that if there is a requirement change (send out information to a different target for example) that an outcome can be either added, removed or the unique identifier changed.
  
• Use of decision tables. Way back in the past rules engines used only the RETE algorithm, and if…then… rules were the mainstay. Most of the rules engines these days implement decision tables in one form or another. What makes them particularly attractive in our environment was the integration with Excel – a tool that a business user understands. We ‘trained’ some of our business users to define rules in an Excel spreadsheet that would allow us to import into and export from the business rules engine.
  
• Use of Flow rules. Another new addition, and something which appealed to the modeler in me. Using flow rules allowed us to model graphically flows and the order in which rulesets are to be called. After a meeting with the architects of the rules engine company, we also started using flow rulesets as the entry point to all our rules projects – thus allowing for internal changes to rulesets to be shielded from the outside world (in this case other components in the system). Thus we could change ruleset names without affecting our calling code.
  

We learned some lessons over the five years of active usage. What were the top lessons learnt during five years of rules engine usage?

• Business ownership. This is an aspect that we are – in all honesty – still battling with. Many Business Rules Engines sell themselves on the fact that business can define rules and can thus take ownership of 'their work'. We did not experience this, and are still trying to get business to take ownership of and even understand their rules. This may sound strange, but I have been in meetings where you hear ‘What are the rules in the system’ or ‘If we define it what would you guys do’. It has certainly improved, rules definitions are being shifted off to more business focused people (the business analysts for example). It will be a long road still. Other aspects that have helped us on this road is the format of how a rule is defined (we have a formal definition format), the rules bibles mentioned earlier and the ability to publish rules more easily.
  
• Rules visibility. Another aspect that I believe we can improve, but one that has paid dividends in the past. One can report easier on the rules in terms of generating rules overviews as well as tracking what rules have fired. We developed a simple interceptor that saves the appropriate information for our analysts to drill into (yes, the rules engine has a stunning little API exposed that allows all sorts of information to be retrieved). Although I personally believe that we should actually expose the rules on our internal website for example for members of business to view.
  
• Separation of concerns. The separation of Java code and what will be executed by the rules engine. I personally believe that due to our use of outcomes, and minimizing the execution of any Java code from within the rules engine is making our maintenance and continuous support a lot simpler.
  
• Treat rules seriously. Once we started to understand the worth of the rules engine and building up on it, we changed from rules as an afterthought to it becoming a major part of the system. At one stage a mantra in the team was that ‘prove it cannot be done in the rules engine’ as a way to almost force the development of rules in the engine. That said it was also refreshing to see how the developers and later the business analysts took to the use and understanding of the rules engine.
  

I am (still) passionate about the use of a rules engine, or rather what a rules engine provides. I am after all in the business of building business applications in an environment that is rather heavy on the rules side. Not only from a company and area but also from various national and international standards as well as legislation. Our current rules documents are over 350 pages each and there are two of them. Yes, there is a lot of white space to improve readability, but it is still rather hefty. Has the investment in rules technology paid off? In my mind it has, especially when I compare some of the maintenance tasks that occurs within the legacy system as compared to the system using the rules engine.

The current rules engine will be phased out and replaced due to its age and the company’s lack of interaction with the provider. Replacing it with another rules engine (likely by one of the open source equivalents, ooo I wonder which one). It certainly has been an interesting road…



(*) In this case interesting is used as a euphemism for poor, bad, not interesting at all but rather scary.