Why Processes?

This means to be a sequel of my previous post, where we started discussing the benefits of writing rules. I intend to do a similar thing, but now centred on the concept of Business Processes. As with last time, this post is oriented to people just getting started with the concept of processes, who are still getting into it. Processes are a simpler concept to grasp if you compare it to rules, but I still think we should start with a basic description.

What is a process?

The simplest explanation of a process I can come up with is “a sequence of steps to achieve something”. Actually, I wasn’t able to come up with it, it is a dictionary definition. But it is as simple as that, just a sequence of actions that need to be taken, by either systems or people, to achieve a goal. If we talk about a business process, we just need to add a little more definition to it and say it is a sequence of steps on a specific domain, performed by systems or people, to achieve a specific business objective.

If you come from the developer end you might be thinking “so? any code I ever wrote is a sequence of steps towards something!”. And you’re right! Code itself follows a sequence. It also has a purpose. If it reads inputs from a specific source, it can interact with systems or people, depending if the source is a specific integration component or a user interface. There are, however, several advantages that process definitions provide over plain code. We’ll cover some of them in this post

Diagramming structure

The first and most user friendly feature is the possibility of seeing the process definition as a diagram. We won’t need to see the process as an obscure list of method invocations, but as an image like this:

If defined in the BPMN2 standard (the current most accepted standard for defining executable processes), all the diagramming information is available in the process definition itself, so the same information used to determine which is the next step to follow will also define the structure for visualizing the process. This is very helpful for business users who probably don’t know about a specific technology, but they have a very clear understanding of the what needs to be done at what time. Which leads to the next advantage.

Process definition as a collaborative tool

Once you have a diagram, non-technical people can understand each step of what is happening. Thanks to process editors provided by each implementation of BPM, people with no specific development background can also edit the content to match the real world scenario. This makes the process definition the centre of our development effort. It not only provides itself as a documentation thanks to the diagram, but it also works as a connection point between business analysts and technical people. Business Analysts need to worry only about the specifics of the process completion (that it should match the specific objectives, what order should the steps in the process be following, who should be in charge of what human task, etc), and technical people need to worry about integrating the process with any other existing service. Which leads to another advantage

Business users take care of business. Development takes care of integration

This leads to an eventual freedom from the development side, from having to worry about making each component to follow ever changing processes. Developers, in this paradigm, only need to worry about providing the necessary connectors to different systems and people. By connectors, we mean system invokers that can be called from the process, components that will be able to invoke or signal a process, and forms for people to be able to do human tasks. From a developer’s perspective, the process will be something like this:

Where you will only need to worry about connecting processes with other systems and providing a UI for new forms. In a way, it releases developers from having the complete responsibility of defining the structure of the system, allowing more people to collaborate in the definition of how things are done, and letting developers concentrate on connecting those definitions with other systems.


Since development efforts on the process definition side will decrease over time, people in the development area can start working on what they really know to do best. Better connectors, and better environment configurations. Eventually, as process execution demand starts growing, you can start working on the scalability of your BPM systems. And since all activities move around the same structures (the process executions), you can concentrate on making ways for processes to take as little time executing steps as possible, managing signals in distributed ways, and managing responses from the connectors in a way that more than one node could be able to receive them.

This leads to not only better running systems, but also a more manageable scalability of them. As all core processes of the company start moving to BPM systems instead of legacy applications, high availability and scalability of systems starts becoming an ubiquitous feature.

A company as a process (composed of other processes)

The final stage of this endeavour is getting more and more processes of the company working inside a BPM system. Since processes can invoke other processes inside them (called sub-processes), you can make a hierarchy of components which can be as broad as desired, to cover as many operations of your company. An ideal point in this stage will be reaching the main process of the company, where usually client or service request are the input, and the objectives of the company are the output. Inside that process (which, beleive it or not, could be quite a simple diagram), you will invoke every other process you define for each specific case in a hierarchy of processes invocations.

Summing up

Processes become a great way to involve as many parts of the company in the definition of the steps needed to perform a task. Nobody knows better how to do a specific task than the operation teams, and nobody understands better the objectives of a company than the managers. Also, nobody understand better how to accomplish those tasks in an automated way than the developer teams. With process definitions, each one of them can collaborate on the same component, all working together the same goal: improving the objectives of a company.

Also, process engines should take care of two things: Coordinating tasks between other systems and people, and taking care of storing the state of current active processes. It is like this because it should let other systems with other objectives do what they do best. Web Services should be in charge of services. Rules should be in charge of complex decisions. In general, a good process engine should try to be something like the following description (which I like to read with De Niro’s voice in this scene of “Men of Honor” 🙂 ):

A process engine isn’t a monolithic system, it’s an integration platform. If it should be done by someone else, it invokes it. If it needs a person, it notifies him. If many tasks need to be done at the same time, it synchronizes them. If it is lucky, it will execute for a few milliseconds for a process that should run over a full year, for that is the closest it will ever get to being considered efficient.

And please stay tuned for the next post, where we will be discussing the power of putting rules and processes together!

Why Rules?

I’ve always gone into a lot of detail regarding specific points of the rule engine, but I feel this is a topic which needs to be covered more. So this post is oriented to people who might just be getting started with the concept of rules and are still in doubt. I will try to explain some of the advantages of using business rules and rule engines in a way that, oddly enough, I haven’t found online yet.

What is a rule?

If you got as far as reaching this blog, you probably already have an idea of what a rule is. The most primal explanation we can give is “something that looks like this”:

when a condition is found to be true
then a consequence is executed

And that’s about it. Add as much syntax sugar as you want on top of it, but a rule will still have that basic structure underneath it all. I still remember the first question that rose to the top of my head when they explained what a rule was: How is this any better than writing code? It took me a while to figure it out, but I’ll try to make it easier for you

Let’s begin by explaining the reason of such a simple structure. It’s based on an Artificial Intelligence principle, where a specific structure of code can be represented as data, in a structure similar to the one shown in the following diagram:

Why would we want to represent our code as data? Because manipulating that data makes a system able to change what it is doing, and it is one of the pillars of machine learning. Having the execution code as data means that we can change the data and make the system behave in a different way without having to restart any component. But that is not even the beginning of why rules are something useful:

It is not what one rule can do, but what many together can do

If we think of an extremely simple case, where maybe we have one or two rules to define a scenario, we can definitely think of a simpler way of doing the same scenario using nothing but Java. The power of rules doesn’t reside in the possibility of explaining simple scenarios, but quite the contrary. It lies on explaining complex scenarios in a very simple and performing way. It divides a problem in the most atomic components: each individual decision we could take, together with the minimal information we need to take that decision. No rule should extend beyond that complexity.

Rules don’t know each other

This is a principle most rule engines abide: Rules are independent. Each rule doesn’t know if another rule even exists. All they share is the information about the state of the world. This information is shared, and some rule engines (drools included) allows for rules to modify this information, in order for other rules to be activated. This process of activating other rules by changes in the information of the state of the world (called working memory) is called inference.

Because of this mechanism, rules don’t need to be too complex to describe a complex scenario: a complex rule can be divided into simpler rules that infer parts of information of a complex scenario and feed it to the working memory, and other rules might depend on that data to get activated. Which leads to the next item regarding how to manage complex, ever-growing situations

Scenario becomes more complex? Just add more rules

This is how you should manage complex scenarios. You don’t have to worry about having a very complex list of commands to evaluate each condition on a complex decision. All you need is to keep creating atomic rules, until you have covered all the different cases. Rules become the epitome of decoupling software: Each rule should just worry about understanding a very small and basic concept from the information it can gather from the state of the world

Letting the rule engine do its job

Rules should remain that simple because it is the rule engine that will manage the complexity for us. Whenever we need to detect many similar conditions, the rule engine will create a performing way of evaluating those conditions as fast as possible, through the RETE (in drools 5) and PRHEAK (in drools 6) algorithms. They will do the heavy lifting. It’s something that we have seen before:

If you’re as old as I am, you might have been there when development teams started going from compiled code (like C or C++) to interpreted code (like Java). I didn’t like it that much at first: I needed more resources to run the programs I made, I had to learn new APIs, everything was a class now… it just wasn’t my cup of tea, at least until I got the hang of it. The thing was that, in order to write good Java code, I needed to follow very different standards from those to write good C code. I didn’t need to create fancy optimizations in our code because those optimizations were going to be done by the VM. And eventually, simpler code became more efficient than complex code hacks we brought with us from C, mainly because the VM optimizations were doing it so much better than us.

And in a sense, a rule engine does the same thing. It will take all our rules and transform them into something that can run in the most efficient way to find all the matches to our rules. In a system with many rules, the code to evaluate the conditions will be far quicker than the optimizations we might add to a batch processing on a code block.

And that is pretty similar when talking about rule engines and rules. Its like the VM and Java code all over again. Someone will be able to do a better job than you at optimizing simple scenarios, so we shouldn’t have to worry about optimizing the code so much as making it clearer.

Fitting more hands in a plate

There is a common saying in engineering: You can’t make 9 mother make one baby in a month. We use it as a clear example of how some tasks cannot be divided any more, and have specific dependencies between each other. The phrase is used to describe situations where you have coupled code, and even if we divide a problem into many different tasks, people will end up touching the same components (and experiencing the expected conflict) when trying to solve two different situations.

One of the greatest advantages of rules is how highly detached they are. This means more people can be editing the rules without stepping in each other toes. The simpler the rules, the less experience needed to edit them. And thanks to human language to rules mappings (like DSLs and Decision Tables), you can have non-technical people defining rules. This means more people can get involved in rule definition, and help extend a system that could be as complex as necessary, and do it in the simplest of ways.


Dynamic Control of Processes and Services

This is where the events from processes and services, and external services, are combined to select which processes and services are relevant to use for the current context.
In effect, the CEP engine becomes the controlling agent for the business processes and service engine, handling for example dynamic process selection.

Note that this pattern is a further evolution of patterns 2a and 2b.

Predictive Analytics and Drools

I’ve been looking into Decision Management over the last few days. It is an excellent topic I’ve been missing out for quite some time. A brief description of what I understood so far from it, is a set of methodologies to provide a closed cycle for knowledge management systems, from the point of knowledge discovery and formalization, to exposing it through a knowledge runtime and providing feedback to the knowledge discovery.

This last part, the feedback, is a component I’ve always wondered if was possible to build as part of the tooling that Drools provides. The idea behind it is having an environment where you can play with data from your production systems, finding all sorts of combinations that you weren’t paying attention to before, either because you planned them for later, or because you didn’t thought of that condition was appearing in your environment. This is not always something you want to do so lightly, specially when you’re handling private or sensitive information in your working memory, but after a certain amount of filtering, it is something that you would like to consider a good place to start searching how your environment is working in production.

You might want to do this in a separate environment, however, because of two things:

  • Performance issues: If you’re going to perform queries, groupings, or any heavy search engine related tasks, you don’t want them to slow down your environment. For Drools, this is a very important topic, because so many people use the rule engine because of its high performance, and you don’t want it to decrease by any reasons
  • Simulation generation: Another heavy task. Once you find the patterns that would identify your new cases, or improve your existing cases, you might want to do changes to your knowledge definitions to see how they behave with the existing data. If they work well, you will want to apply the new knowledge definitions to the real production system, but not before.

The branch that targets this analysis is called predictive analytics, because it goes further from just making realtime analysis of data, but also focuses on discovering trends of change in your production environments to be suggests new ways in which production data might be getting inside your environment in a few minutes, hours, months, and so on depending on how far back the production data involved in the discovery process goes.

All these analysis are usually possible because they are thought to be conducted from Big Data. This is the part where Drools puts a distance with Decision Management Software. Drools production data (the working memory) lives in memory. It can be persisted, but even if it is, it is just a serialized blob of information to restore the session elsewhere. So, even if we did these analytical tools for Drools, they would have to work on memory.

This got me thinking if these sort of analysis could be done on top of a Drools rule engine. We certainly do have some tools in DRL to provide us with analysis capabilities.

Analytics with rules and Queries

DRL Queries can give us insight on any specific working memory using the same search patterns that rules use. The one thing it would be able to do is provide us with easy grouping, but rules and specific facts for carrying grouping information could do this quite easily.

rule "Group by init"
insert(new GroupBy());

rule "Group by example"
p: Person(age > 16)
gb: GroupBy()
gb.sum("personAge", p.getAge());

Also, once a query is constructed, group by functionality could be introduced on the Java side (after all, everything is running in memory). But let’s delay the detailed analysis of that for a moment by stating that, for the moment, if we wanted to analyze a working memory to find uncovered cases, we would most likely be able to. Having that as a “for the moment” assumption, lets try to see how we could implement an in-memory analytical tool for Drools

Given that, lets see 3 different scenarios to start building some tooling for this. We will use a few diagrams with the same color coding: white is for already existing systems, yellow for easily built components, and red for components that would be hard to implement:

Case 1: Your own 100% developed environment

Let’s say that, for the simulation environment where we will run our analytics, we were going to build an entirely new system from scratch. Some things might be easy to create, some not so much. Usually when we do this, it is because we have an already existing application dedicated to this, and want to use it for analyzing our Drools environment, so we will assume the environment already exists. But what do we have to build for it?

Without a KMS

As you see, the first thing we will need is a way to send new information to Environment B. Session persistence could take care of this, but usually when we run complex working memories we don’t usually persist in order to gain maximum performance. In this scenario, we would use a single component of persistence: session serialization. Using any pluggable communication methods, we could create a communication between two environments to share the same session , even if that session is not persistent.

The one component that would be hard to implement, however, would be a runtime and UI where we can construct queries or rules to run simulations with the production data. This would involve creating query editors, rule editors, runtime components to perform those searchs in the copied working memory, and UIs to show them to the user. These components would be quite hard to build, not because of any intrinsic complexity, but mostly because they would have to be maintained by 3rd parties (A.K.A. YOU)

Case 2: Using the KIE Workbench functionality embedded in our application

Fortunately, there is an alternative to writing your own editors that would facilitate development a lot. That is, using the guvnor editors for rules to construct a query editor you could use from your application. That leaves environment B to worry only about having an execution environment for execution of simulation scenarios.

With a KMS

It also facilitates deploying, because if you use guvnors internal build and deploy functionality, all new knowledge definitions could be implemented in your production environment by using nothing else than the KieScanner, provided by the kie-ci dependency in Drools 6.

The one problem it would have is that it would have to embed existing editor inside your own applications. It is a very reasonable consideration most Drools tooling users want to be able to do, but for the moment, it is rather complex to integrate. Perhaps there is a way to have everything environment B has in a single place?

Case 3: Extending the KIE Workbench functionality for analytics

In this scenario, we would just extend existing functionality of existing workbenchs. A query editor would be easily built using the guided rule editor. A query executor and analyzer could be created from injectable components and UIs. Event sending could be used to trigger a simple yet powerful session replicator from a different environment. These, I think, would be the best way to go to start building analytical tools on top of Drools right now:
Everything inside Kie WB


Of course, these are just theoretical components for the moment, but they would be very possible to implement. They would provide a huge added value to the discovery stages of knowledge development for Drools tooling. Development over the next few months could prove me wrong, but I hope it will not, as I also hope you found this analysis of drools analytic possibilities informative.