Mining Business Rules

 

Michael Oara

CEO, System Renewal, Inc.

 

 

 

What are we after?

 

 

One of the major ways in which an application could be modernized is through the externalization of its business rules, Just like SOA enablement, it does neither put some stitches on the application, nor totally rewriting it. In stead, it allows some of the legacy code to continue in the legacy environment, while moving some parts outside of that environment such that they could be modernized. In fact, once the business rules are identified and externalized, the development team has a spectrum of possibilities for the old code, starting with minimal changes end ending with a complete rewrite.

 

If we look at the actual code of an application, we may distinguish three distinct categories: process management, business rules and mechanisms.

 

The process management part is related to the order in which various tasks are performed. For instance, when processing a customer order, the application will perform certain activities, in a precise order: validating the request, checking the inventory, computing the prices, computing the taxes, computing the shipping costs, sending a confirmation. There are portions of the program code that are involved with the control of these steps, and they are part of the process management. This may involve checking the success or failure of each step, screen navigation and event processing.

 

The “business rules” code is that code which – as the name well describes it – controls and implements various restrictions or way in which the enterprise business operates. Such rule may, for instance, state that a discount is offered for orders over $100 in value or that the sales tax is computed in a particular way in a particular state.

 

Mechanisms refer to purely technical aspects which are very much incidental in a particular implementation. This may be the case when an area in memory is initialized or when the length of a message is computed.

 

The three types of code could be easily identified in the case of a particular program. Let’s consider, for example, three code segments:

 

Segment 1

IF OPERATOR-REQUEST = “R” THEN

    PERFORM REMOVE-ITEM

ELSE

    IF OPERATOR –REQUEST = “U” THEN

         PERFORM UPDATE-ITEM

         END-IF

    END-IF

Segment 2

IF TOTAL-PRICE > 100 THEN

    MULTIPLY TOTAL-PRICE BY DISCOUNT

    END IF

Segment 3

PERFORM VARYING IDX FROM 1 TO NUMBER-OF-ITEMS

    IF ITEM-NAME(IDX) = REQUESTED-NAME THEN

         REQUESTED-INDEX = IDX

 

A simple look at these samples is enough to establish to which category they belong. Segment 1 reacts to an operator request, therefore helps to control the process. Segment 2 expresses the rule that a discount is offered for orders over $100, while Segment 3 simply performs a look-up to find an element in an array.

 

The three categories play separate roles in a modernization effort. Mechanisms are very much irrelevant and may be changed only in order to improve the performance of the application. For instance, a look-up of an array could be improved by changing from a full array scan (as in the example above) to a binary search. However, if the application is re-implemented in a new environment or using another technology, this is irrelevant, as new and modern development facilities already offer a large library of functions to take care of “mechanisms”.

 

On the other hand, the process management and business rules aspects remain relevant. The enterprise will prefer in most cases to preserve the way in which it does business, even if the implementation changes. For both cases, one would wish to first identify and then re-implement. Once identified, business processes may be implemented through SOA and BPM, while business rules may be implemented using business engine technologies. Users of Relativity Technologies tools may employ the SOA Analyzer to identify the processes and the services they use, and may employ the Business Rules Manager to identify the business rules.

 

We will take you now through a business rules mining scenario, to illustrate how the business rules identification may be accomplished in a practical way.

 

Some prerequisites

 

Tools needed:

 

 

Besides tools and technologies, there are some other prerequisites for a successful business rule mining project.

 

Perhaps the most important prerequisite is familiarity with the application. “Familiarity” may be a vague concept, but it is certainly enhanced by employing some analysis tools, like for instance Relativity’s RMW Analyzer. Before the actual mining starts, we should take first a high level view, so we can perceive the whole forest, not just individual trees (or even leaves). The static analysis of the application may help answer some important questions, as for instance:

 

 

As the concept of business rules may be vague to a lot of people, we recommend that at the beginning of any mining project the team involved set a clear definition and objective. The problem with the “let’s collect the business rules” objective is that one is never sure when the project is completed. For this reason it is more useful to have an objective specified in a more concrete form. As an example, any combination of the following objectives will set a clear and achievable target for the mining project:

 

 

 

Finally, it is only natural to assume that familiarity with the analysis and mining tools is a great plus.

 

Identifying the business rules

 

Relativity’s Business Rule Manager allows for three basic techniques for business rule identification: pattern search, validation rules search and backward autodetection. A combination of these could be employed from case to case. The immediate result of such searches will be the discovery of a number of “candidate rules,” some of them of obvious interest, some nothing else then false positives. The mining process will consist of a number of iterations in which the search criteria are constantly improved.

 

As candidate rules are discovered, there are always two questions to be answered:

 

  1. Can I improve the search criteria in order to eliminate false positives?
  2. Can I improve the search criteria such that I catch some candidate rules that were not discovered by the search?

 

Here are two concrete examples.

 

Example 1- Eliminating false positives: Suppose that I established a search criterion for finding the computation of the output fields displayed on a screen. After looking at the results, I discover that I found more candidate rules then expected, due to the fact that I captured the computation of all outputs, including the attributes of the output fields, which determine the color or display style of some data. As these attributes are not important, I can introduce more constraints so that I will find only the computation of data from the screen, not its displayed attributes.

 

Example 2- Discovering more precise rules: Suppose that I look for the error messages produced by an online program during input data validation. The result will be a number of code segments that look like

 

IF ERROR-CODE = 15 THEN

    MOVE “SHIPPING DATE IS TOO CLOSE” TO ERROR-MESSAGE.

 

This is not sufficient, as ERROR-CODE value 15 does not tell me anything about the real expectation about the shipping date. However, a new criterion may discover the code which produce ERROR-CODE 15. It may be:

 

MOVE INPUT-DATE TO DATE1

MOVE CURRENT-DATE TO DATE2

PERFORM DATE-INTERVAL-COMPUTATION

IF DATE-INTERVAL < 3 THEN

    MOVE 15 TO ERROR-CODE

 

This is more relevant, as it captures the fact that the shipping date cannot be less then three days from the current date.

 

Let’s see now how the search criteria are built and used to discover rules.

 

Using pattern search

 

Relativity’s Business Rule Manager allows you to search for patterns in the code, by using a facility called Clipper. With Clipper you can create various queries based on the syntax of the language used by the application. Clipper will assist you in building correct queries, but you must know what you are after. As explained above, queries may be constantly refined in order to obtain more relevant results.

Let’s suppose that with some knowledge of the application I know that somewhere in the code some computations are performed that depend on the state where the supplier of a product resides. That is enough to get started. The Clipper query will look like bellow:

 

 

As you may notice on the left pane, Clipper comes with a number of predefined queries organized in various categories. However, in this case I have created a new query, which I named “States.” Clipper assisted me by indicating at each point the types of objects or relationships that I may look for. I can click on any of the blue underlined words to make changes. I have also added a description on the bottom pane, which will help me remember what the query is doing in case I need it later.

 

As you can easily see, the query is for all paragraphs in the application that are performed on a branch of an IF in which I check for a variable with the name like “*SUPP* (for supplier).  If I run the query by pressing the Find All Constructs button, I get the following result:

 

 

which indicates that the pattern was found in just one of the programs (PRODUPD.CCP), in three instances. As I click on this line, I can see the individual instances and the associated source

 

 

This particular instance refers to a supplier from the state of Iowa (IA). At this point, if we are happy with the current finding, we can take all three instances of the code pattern and make them into “business rules.” I used quotes because we do not have yet rules by the strict definition of the object. We have in fact found implementations of rules, from which we will derive the rules.

 

As I request to create rules from these instances, I am prompted to enter the name of a Business Function and of a Rules Set to which they belong. These are just containers, that help me to keep well organized. Now I can see the code instances captured as “business rules”

 

As you may see, I have given them names like “Computation of inventory value for the state of …”. You may also see that the Business Rule Manager has collected some additional information, like for example information about the use of some data elements in the rule. In the bottom right pane you can see that two of the variables are found to be both inputs and outputs, while the other three are just inputs. I have given them business names and I have also given a plain language description to each of the three rules.

 

Searching validation rules

 

Validation rules are of the highest importance in an application, as they describe restrictions on the data that comes from various input interfaces. Some validation rules are very simple (“Amount must be entered as a numeric value”), some may be more complex (“if the customer is in the state of Ohio, a valid address must be entered”). The Business Rule Manager can collect these rules automatically. The logic for finding such rules is simple: the tool finds all tests applied to data elements that get their values from an input on a screen.

 

To collect validation rules, in Business Rules Manager you point to an input port in the source of a program (as for instance a CICS RECEIVE statement) and request the tool to produce the list of candidates. As a result you will see the validation rules for the program and you may inspect them one by one, to eliminate false positives or rules that are of no interest. The tool may show, for example, rules that simply indicate that a particular field, such as customer name, is mandatory. It is up to you to decide if this is a rule you want to keep, of if it is a “trivial” rule, which could be removed.

 

Here is a validation rule discovered by the Business Rule Manager, which seems more significant:

 

 

Inspecting the code, we can understand the meaning of this validation rule. It declares that when the operator enters a customer order on the screen, a “check-digit” validation is perform, to make sure that all product numbers are correct. Understanding the meaning of the rule, we can now give it a more significant name and a description.

 

Using backward auto-detection

 

We can employ this method when we try to answer the question “how is the value of a particular data element derived at a certain point in a particular program?” To find rules of interest, we should concentrate on the data elements that we know are calculated or derived, such as discount, total price or insurance premium. Other data, such as customer name or address are of less interest, as they come from outside, instead of being determined by a program. The point of interest is usually around the place where the program either displays the results of the calculation (such as a screen or report) or records it in a permanent storage (table or file). It could be the output port itself or a MOVE statement where the output data is constructed.

 

Example: I know that a particular program computes the discount offered to a customer when the order has certain characteristics. The computation of the discount is complicated, as it takes into account parameters such as the state where the customer resides, the number of items ordered and their value. To start the backward auto-detection, I point in the source to the statement where the discount is moved to the screen area and request the tool to find the rule.

 

Business Rule Manager will discover all the statements involved in the computation while ignoring everything else. Furthermore, the “backward” tracing of the discount calculation is not limited to one program, but can cross into other programs called to aid in the calculation. The result is a logic structure extracted from the program, which the Business Rule Manager displays in one of its panes as in the picture bellow:

 

 

As with the other rule mining methods above, at this point I can assign significant names to the rule and give it a plain language description. The tool has calculated the input and the output fields of the rule, to which I can also give significant business names.

 

Putting it together

 

Reviewing all the activities described above, we can see the steps needed to perform a successful business rule mining project:

 

  1. Register all sources into the RMW repository.
  2. Verify (or parse) the sources.
  3. Get a high level understanding of the application with the aid of diagrams and reports
  4. Set clear objectives for the type of rules to be mined.
  5. Use one of the three methods for rule detection: search patterns, search validations and backward auto-detection.
  6. Remove false positives
  7. Document the remaining significant rules.

 

These steps will result in a list of rules stored in the RMW repository. They can be reviewed at any time, and they can be exported in XML format.

 

Having discovered the business rules, what next?

 

The discovered rules may be used in a variety of ways, starting with the simple documentation of the application and ending with registration in a business rule engine, such as the one supplied by Oracle.