How to design? - the message flow

Message flow

How to model the message flow? The best answer to this question is it must be as simple as possible. Why? Consider that the OSB is the center of the communication. If we just consider the performance of the network then the OSB seems to be unnecessary as this is just a new station between two systems. It may slow down the communication (Ok we know the OSB is a very important IT platform of the company).

The message flow must be quick and contains the least steps. Of course we have to be concerned with some basic things as error handling and monitoring. But we have to put these logics into the message flow not to disturb the main workflow (e.g. At the beginning of each proxy services we have to call a monitoring system. To consider the conclusion above we have to make an asynchronous call to monitor system not a synchronous one).

It is very important to understand that the OSB is not a business workflow engine. You mustn't implement this kind of logic in the message flow. And the next question is evident: What should we consider the business logic? The answer is difficult. You have to think of the OSB as a funnel. You (the client) fill something (e.g. milk) on the one side and You (the server system) get the same (milk) or other (chocolate milk) thing. But You don't do anything just fill! Ok, I think it is enough of the metaphors... :)
Usually the proxy services use just one source system (business service). The OSB service can modify the requests / responses, change the protocol, add some basic functionalities and so on. If You should use more source systems in the proxy service You have to consider to implement it in a workflow engine. If You haven’t got such an engine You can develop enterprise services in the OSB.

To decide what You may put into the OSB depends on your actual IT possibilities as well. If You have got a BPEL engine then You can take out easily the logic which is complicated to implement in the OSB. But if your the company has got just the OSB (and not BPEL engine)... well in this case we can't put this workflow into a non-existent workflow engine. It is sad to say but You don't have any other chance just to implement these logics into the OSB (if the tools of the OSB are sufficient for this...).

The OSB enterprise service use more systems not just one. They are built on the existing services of the company systems but they seem to be new ones. You have to find out the requests/responses, the message flows and how to transform or summarize the results of the called systems. So these ones are the so called enterprise services which are brand new services in the IT platform of the company.

Every message flow is different but there are similar / general parts. We have to find these high level parts to develop the message flow easier. Check the next figure:

I would like to emphasize this is my vision about the basic parts of the message flow. There could be a lot of other area which I haven't met yet and could change the figure above so forgive me if this is not the complete picture. :) If you have got any idea about what to change please write me!

Monitoring
So the first step is the monitoring. I illustrated this with dotted line which means that I don't think this step is compulsory in each cases. This step is necessary in those cases if the proxy service is an asynchronous service. For example if You receives an email and an error occurs then usually it is not reasonable or there is not any chance to send back an email about the error to the sender. Or in case of FTP we should send back a file...?
But it is rarely informative to monitor if a synchronous service is called. The monitoring funcionality is belong to the client in this case not to the OSB. But if we consider an asynchronous service this is the task of the OSB. You should monitor as much information as possible (of course it depends on the possibilities of the monitoring system as well).

Check the incoming data
It is very very essential to check the incoming data before using them (e.g. to send to source system). Here there is a good chance to use the OSB: we can disencumber the source system. It is absolutely unnecessary to call the source system if there is no chance for a successful execution. Usually we have got two tasks to do in this step:

1. to validate the data
   We know the expected structure of the data so the validation of the incoming data can be performed. We have an easy task in case of webservice (the structure is determined in the WSDL) but in the case of other protocols the structure is discovered in the phase of business design already. Usually we use scheme to validate but other validation logic can work as well (e.g. in case of text files).
2. to check if any necessary information is available
   We have to examine if we have got all the data to call the source system. For example: if we develop a service to send an email and we doesn't receive the email address then we have to raise an error to the client. Of course we can handle these cases with the XSD as well but there might be situations when we have to check this without the help of the XSD (e.g. the XSD mustn't be modified).

Transform the incoming data
After we checked the incoming data sometimes we have to transform it. What I mean to transform? Just remember, an important usage of the OSB is to play a gateway role between systems. For example if the CRM system wants to communicate to the billing system and they have got different data structure then it should mean some development works in one of the systems. But it doesn't because You can make the conversion in the OSB! The costs might be lower as if You should start a development project in a legacy system...
When I am talking about conversion I mean XQuery transformation which is a powerful tool to transform an XML to an other one. If the source is not an XML then You have to develop individual solutions. If You should change the namespace of tha XML only then You could use the action 'Rename'.

Create the request for calling the source system
To call a system we need a request message. How to create the request depends on the protocol (I won't specify all of them but the more important ones for me):
- Webservice
We have to create a SOAP message. We have to set the body part of the message and occasionally the SOAP header. Usually we use the header to set the security (e.g. OASIS security or custom solutions).
We have already produced the data to send before this step so we just have to put it into the SOAP body. Sometimes we have to add some tags to the XML. To be clear have a look at the next sample request  for the header part:

<soap-env:Header xmlns:ws="http://ws.billing.acme.com/">
    <ws:username>...</ws:username>
    <ws:password>...</ws:password>
</soap-env:Header>

and for the body part:

<ws:receiveOrder xmlns:ws="http://ws.billing.acme.com/">
    <ws:Container>
        {$incomingTransformedData}
    </ws:Container>
</ws:receiveOrder>

At the first sample we use custom security solution, we put the user and password into the SOAP header. At the second sample we have produced the variable 'incomingTransformedData' already in the previous step so we create the SOAP body and put this data into it.
- Email
To send an email you have to use the action 'Transport Header'. You can set the email address of the sender and receiver and the subject (and a lot of other things as well...). The content of the email has to be put into the internal variable 'body' when you use the action 'Routing' or 'Service Callout'. It can be any text or XML but it depends on the mail system.
If you want to send attachment as well then you have to insert a similar XML to the internal variable 'attachments':

<con:attachment xmlns:con="http://www.bea.com/wli/sb/context">
    <con:Content-Type>application/octet-stream</con:Content-Type>
    <con:Content-Disposition>attachment; filename="{$fileName}"</con:Content-Disposition>
    <con:Content-Transfer-Encoding>base64</con:Content-Transfer-Encoding>    
    <con:body>{$fileContent}</con:body> 
</con:attachment>

- FTP
You have to use the action 'Transport Header' in this case as well. The name of the file which will be sent through FTP can be set in the property 'fileName'. You can set the FTP endpoint dynamically with the action 'Routing Options'. And again you have to put the content of file into the internal variable 'body'.
- any Java serviceYou can call a service of a Java service (e.g. session bean) directly from the OSB. The key is the action 'Java Callout'. If you have got the client (proxy) java classes of the session bean in a JAR file and put to the classpath of the OSB then You can use them in the action 'Java Callout'. This is a simple but not too elegant possibility to use a Java service in the OSB.
You can reach an EJB3.0 service directly from the OSB but this is true only the EJB3.0 services.

Call the source system
To call a system you have got two possibilities: routing or service callout / publish. If you need to execute some steps after calling the system then You have to use service callout / publish (first one is for sync. services, second one for async. ones).
You have to pay attention for the error handling. To thow the exception to the client or catch them? I will deal these questions later in another blog.
You have to be concerned with the parameters (e.g. retry interval, retry count) of the calling. If you don't set the values of these parameters prudently then it can cause an unnecessary bottleneck. The attibute 'Retry Application Errors' of the business services is such a parameter as well (why try again if the source system sent back a response?).


Transform the response of the source system
This step is similar to step 'Transform the incoming data'. This step is unnecessary if the proxy service is an asynchronous service or there is no need to transform the data.


Create and send back the reponse
If our proxy service is synchronous then we have to send back a resposne to the client. If the source system synchronous as well then we send back its response to the client. If it is asynchronous then we have to create a reponse to send back. Usually these reponse are not complicated just a sign that the execution of the operation was correct.

How to design? - the data

How to design? This is the basic question for creating reliable and well-working OSB components. What are those repetitive steps which the designers or the developers must do?

OSB is a communication platform so the data (requests / responses) are very important. The main task of  the OSB is to deal with the data and not with the business logic.
Usually we have to process the incoming data before sending to the other systems or third parties. These process steps are called the message flow. I emphasize the purpose of these processes are to handle the data and not to implement business workflows.
We have to know the answer of the next question as well: what will we do if an error occurs? We have got lots of choices what to do in these cases but the circumstances (business requirements, used protocols, ...) are different and limit our possibilities.
OSB is a gateway among systems it might slow down the communication and cause bottlenecks. You should be concerned with the performance issues as well.
It is very important to check what is happening inside the OSB. The technical and business users want to monitor the events or just become aware of the errors or the more important happenings.

These are the key factors what You should be concerned with:

  - input / output data
  - message flows
  - error handling
  - performance
  - monitoring

Input / output data (requests / responses)

The handling of the requests or responses is different in case of
  - business services
  - proxy services.

Usually the data structure of requests of the business services is determined as the services which must be connected to OSB have been working already. So You might have less tasks to do. You have to create the business service, set the protocol and the structure of input / output data (in case of webservice just use the XSD in the WSDL).
In some cases (e.g. attachment of email) You can't set exactly the data structure for the business service (anything can be sent in attachment), You have to find out and validate it in the message flow (e.g. XSD validation). In these cases You must check the data because You mustn't encumber the source system unnecessarily.
Sometimes You need to create a request or a response message. If You want to convert an asynchronous service to a synchronous one inside the OSB then You need to create a virtual response because the source system won't send a response back. (e.g. webservice → email)

If You might receive unstructured data then You can't validate it simply. Why? The best example is the attachment of emails for this. You can receive ANYTHING in an email. One part of the incoming email is incorrect (for example a JPG file was sent instead of XML) and the other part seems to be good.
If the same data is being sent to an email address then there is no problem. But what if the partners send orders and complaints to the same email address? And what's more there are more versions of the orders and one of the partners send version 1.0, the other partner sends version 2.0 and so on. In these cases You can't validate so simply.
To consider the different business data (orders, complaints) You have to validate the attachment using one of the XSD's and if an error happens You have to use an other schema and so on. If there are more versions of the incoming data and there is version information in the XML (e.g. <order version="1.0">) then You are able to choose the correct schema to validate the XML. If You haven't got version information then You have to use the strictest schema to validate and the second strictest one and so on.

Proxy service is a new service created by You. In this sense You have got freedom to define the input / output data. But this is not true entirely. If You implement an atomic service usually You have to use the requests / responses format of the business service. In the case of the domain proxy services the input / output data are similar to the data of business service but this is not certain. You have to process the input data and to transform them so the difference between the requests / responses  of the business and proxy service may be huge. This is determined by the business requirements. And if we are talking about enterprise services we have to work out the structure completely ourselves.

To allow more request format for an operation is a bad design pattern. I mean You can use the 'choice' XSD element which let You use different request formats. For example:

<xss:choice>
   <xs:element name="mouse" type="mouse"/>
   <xs:element name="cat" type="cat"/>
</xs:choice>

In this case the request may be this one:

<soap:Envelope ...>
   <soap:Body>
       <mouse>
           ...
       </mouse>
   </soap:Body>
</soap:Envelope>

and this one as well:

<soap:Envelope ...>
   <soap:Body>
       <cat>
           ...
       </cat>
   </soap:Body>
</soap:Envelope>

It seems to be simpler to use the 'choice' XSD element but it is a better and cleaner solution to create an operation for the cats and an other one for mice.

I suggest to put the generally used schemas to a separate project which is for common components. You mustn't modify these schemas even if You should do it because of a proxy service. If You need to modify one of them then copy it to your project and modify that one not the original.

What kind of tools can we use to design?

The fact that we have to design the OSB components seems to be pretty clear. What we have to know is the input/output data structure, how to process the input data (message flow) and what protocol we need to use (Ok, there are a lot of other factors but these are the most important ones).

But what I am thinking of what tools we can use to design OSB components. I bet the quickest answer is the OSB IDE. But this is a developer tool not a designer one. So the real question is do we really need a desinger software to model the components? The answer is not easy. I have to admit I did not have to use always this kind of software when developing OSB. And I can't state whether You really need it or You don't.

It is always good to use a designer tool but I think the main reason to use it is to generate code from model.

The proxy services don't contain deep business logic (except maybe the newly created enterprise services) so it is unnecessary to model it.
It is hard to model the input / output data without so much as not to implement them. I think to model the detailed input / output data and message flow is not reasonable. So we should model the services (proxy, business) and their structures (see the previous blog). In some cases there might be justified to design the data structures but You need to make up Your mind the resolution level.

I have already told the magic word: code generation. It means code can be generated from model. You are creating the model adding stereotypes and properties to the components and the code generator will create folders, XML files (for proxy / business services), project descriptors files, etc.
This is a sample model which I created in the designer tool (this is the model of the topic of previous blog ):

You can assign properties to the stereotype and set those ones when creating the model. In this case I have defined if the proxy service is atomic or it isn't.

After the generator created the code You have to implement the message flows, to detail the requests / responses and so on. The generator can create just the frame You must fill it with content.

What are the advantages? You have got the model of the OSB components. You can review the model easier than the real services in the OSB IDE or console. But You can say now 'a model is never static it is changing often!' Yes, You  are right it is not. If your model is being changed and You have to regenerate the source code then the code which You implemented after the previous generation will be lost (because we have got a source code on the computer which we maintain).
However the good news is this is not an obstacle for our generator. It can merge the old source code (the result of the earlier generation) and the new one! So our work can't disappear. The trick is You have to define read only and modifiable parts in your template and the developers can change the modifiable ones (check the //<%startUserCode%> code in the sample below). If they will write into the read only parts of generated code then this new code will really be lost at next code generation event.

The above ones were the advantages but there are some drawbacks as well. You have to write the code generation algorythms. It might be huge work and depends what exactly You want to generate from model. If You want to create more code from model (e.g. not just services but input / output data as well) then You have to deal more time with the code generator. This effort is worth if You have got a lot of services / components on the OSB.
This is a sample for OSB proxy service template:

<%
metamodel http://www.eclipse.org/uml2/2.1.0/UML

import com.astron.uml2.java.services.Uml2Services
import com.astron.uml2.java.services.StringServices
%>

<%script type="Class" name="fullFilePath"%>
<%if (hasStereotype("ProxyService")) {%>
/<%name.toU1Case()%>PS.xml
<%}%>

<%script type="Class" name="generate" file="<%fullFilePath%>"%>
<?xml version="1.0" encoding="UTF-8"?>
<xml-fragment xmlns:ser="http://www.bea.com/wli/sb/services" xmlns:tran="http://www.bea.com/wli/sb/transports" xmlns:env="http://www.bea.com/wli/config/env" xmlns:http="http://www.bea.com/wli/sb/transports/http">
  <ser:coreEntry isProxy="true" isEnabled="true">
    <ser:binding type="abstract XML"/>
    <ser:monitoring isEnabled="false">
      <ser:aggregationInterval>10</ser:aggregationInterval>
      <ser:pipelineMonitoringLevel>Pipeline</ser:pipelineMonitoringLevel>
    </ser:monitoring>
    <ser:reporting>true</ser:reporting>
    <ser:logging isEnabled="true">
      <ser:logLevel>debug</ser:logLevel>
    </ser:logging>
    <ser:sla-alerting isEnabled="true">
      <ser:alertLevel>normal</ser:alertLevel>
    </ser:sla-alerting>
    <ser:pipeline-alerting isEnabled="true">
      <ser:alertLevel>normal</ser:alertLevel>
    </ser:pipeline-alerting>
  </ser:coreEntry>
  <ser:endpointConfig>
    <tran:provider-id>http</tran:provider-id>
    <tran:inbound>true</tran:inbound>
    <tran:URI>
      <env:value>/OSB/<%name.toU1Case()%>PS</env:value>
    </tran:URI>
    <tran:inbound-properties/>
    <tran:provider-specific>
      <http:inbound-properties/>
    </tran:provider-specific>
  </ser:endpointConfig>
  <ser:router>
  //<%startUserCode%> for messageFlow
  //<%endUserCode%> for messageFlow
  </ser:router>
</xml-fragment>

And here it is a sample for Java template as well:

<%
metamodel http://www.eclipse.org/uml2/2.1.0/UML

import com.astron.uml2.java.services.Uml2Services
import com.astron.uml2.java.services.StringServices
%>

<%script type="Class" name="fullFilePath"%>
<%if (hasStereotype("entity")) {%>
/<%getProperty("package.server.hibernate").toPath()%>/<%name.toU1Case()%>Dao.java
<%}%>

<%script type="Class" name="generate" file="<%fullFilePath%>"%>

package <%getProperty("package.server.hibernate")%>;


import java.util.List;

import org.hibernate.HibernateException;
import org.hibernate.Session;
import org.hibernate.Transaction;

import com.astron.common.exception.DaoException;
import com.astron.common.hibernate.SessionFactoryUtil;
//<%startUserCode%> for imports
//<%endUserCode%> for imports


/**
 * Data Access Object
 * Generated by Astron MDA tools
 */
public class <%name.toU1Case()%>Dao {


    /*
     * Get all <%name%> entities
     * 
     * @return a list which contains <%name%> objects
     */
    public static List<<%name%>> getAll<%name.toU1Case()%>s() {
        Transaction tx = null;
        Session session = SessionFactoryUtil.getInstance().getCurrentSession();
        try {
              tx = session.beginTransaction();
              List<<%name%>> <%name.toL1Case()%>s = 
                  session.createQuery("select h from <%name.toU1Case()%> as h").list();
            //<%startUserCode%> for getAll
            //<%endUserCode%> for getAll
              tx.commit();
              return <%name.toL1Case()%>s;
        }
        catch (RuntimeException re) {
              if (tx != null && tx.isActive()) {
                try {
                      tx.rollback();
                }
                catch (HibernateException he) {
                    throw new DaoException("Hibernate: error at rollback!", he);
                }
                throw new DaoException("Hibernate: error at list<%name.toU1Case()%>()!", re);
              }
        }
        return null;
      }

      ...

I recommend two open source products to generate code:
  - Topcased (http://www.topcased.org/) - for designing
  - Obeo (http://obeo.fr/pages/acceleo/en) - for code generating

To use this technique You have to create some roles in Your IT department. Have a look at the next figure:

As You can see on the figure above at least three different roles must exist:

1. MDA architect
   The responsibility of this role is to maintain the Source Code Generator Framework (SCGF). And what does the SCGF mean? It contains:
   - templates: Each artifacts which You want to create from the model need a template (these artifacts for eample: proxy / business service XML files, XSD's). The generated code will be created from these templates.
   - SCGF package for developers: Usually the developers don't need to know anything about the code generation. Their task is to implement the business logic so they have to put the least effort to generate the code. Due to this fact the MDA architect must create a generator package for developers which they just download from the version control system configure at the beginning of the project (e.g. what is the path of thier sour code or model) and use it simply (e.g. they have to execute a batch file on windows systems)
   
2. designer
   The designer will create the initial model and modify it according to the requirements. They have to upload the modified model to the version control system from where the developers can download it to (re)generate their code.
   
3. developers
   The developers maintain the source code. They download the model and the SCGF package from version control system and generate the first version of the code fill the empty places of code and regenerate it if necessary.

The MDA architect role is project independent but the designer and developers roles are not.

The MDA based code generation is not a magic wand to build something from nothing. You have to create the SCFG Yourself and this might be a huge work. But it could be an enormous help to create the frame of the code easily and quickly.