Deploying OSB components onto different environments

Each OSB expert has already faced the problem of deployment. This task seems to be pretty obvious at first glance but it is quite sophisticated if we dig into the details.
There are many values in the OSB components which are different in each environment (local / development / test / ... / production server). We would like to use the same sourcecode to deploy it into the test server or production server, we want to use a CI tool so it is not possible to use constant values in the source.

The customization file could be a perfect solution for this job ... but it is not. At least it was not the best solution to me.

Let me explain it: I had to send an email in a proxy service. The receiver of the email must be different in the production, the test and the development environments (it is obvious as we wouldn't like to send emails to the business users during tests...). I set the receiver of the email in the message flow by modifying the the Transport Header. We cannot modify this value by the help of the Customization File.
Other example is to change the port number of the SMTP server (you can change the endpoint URL of the SMTP server but not the port number which is a separate property of SMTP server).
There are a lot of other examples too where the Customization File fail so I had to find an other solution to change the values in the proxy services.

Unfortunately I could find any other solution (OSB 11.1.1.3.0) how to change these values. I could work out just one solution: change the XML file of proxy service. It seems to be a sophisticated (or tough ...) solution but you can get used to it. :)
An XML file is created when you produce a proxy service. All you need to do is to find the value in the XML that you want to change and create an XPath to find it in the XML. Afterwards you can change the value with an ant task for example. Unfortunately this is not so easy either...

The first problem is that we don't want to modify the files in the source folder of our local machine. Let's imagine your workspace is in C:\work\OSB\workspace (apologize to the Linux users :) ). You are using your workspace for deploying OSB components into your local OSB domain so I don't think that you'd want to mess that folder.
Besides we need a JAR file as a result with all the environment dependent modifications and we just want to deploy it onto the given domain (see Continuous Integration).
So this is my scenario:

1. create a deployable JAR file
2. unzip the JAR file into a temporary folder
3. change the values according to the given environment
4. create the JAR files from temporary folder

I used this ANT task at the first step:

<target name="build">
  <delete failonerror="false" includeemptydirs="true" dir="${metadata.dir}"/>
  <java dir="${eclipse.home}"
        jar="${eclipse.home}/plugins/org.eclipse.equinox.launcher_1.0.201.R35x_v20090715.jar"
        fork="true" failonerror="true" maxmemory="768m">
    <jvmarg line="-XX:MaxPermSize=256m"/>   
    <arg line="-data ${workspace.dir}"/>
    <arg line="-application com.bea.alsb.core.ConfigExport"/>
    <arg line="-configProject ${config.project}"/>
    <arg line="-configJar ${config.jar}"/>
    <arg line="-configSubProjects ${config.subprojects}"/>
    <arg line="-includeDependencies ${config.includeDependencies}"/>
    <sysproperty key="weblogic.home" value="${weblogic.home}"/>
    <sysproperty key="osb.home" value="${osb.home}"/>
    <sysproperty key="osgi.bundlefile.limit" value="500"/>
    <sysproperty key="harvester.home" value="${osb.home}/harvester"/>
    <sysproperty key="osgi.nl" value="en_US"/>
    <sysproperty key="sun.lang.ClassLoader.allowArraySyntax" value="true"/>
  </java>
</target>

Performing the second step wasn't very difficult:

<unzip src="${config.jar}" dest="../${env.BUILD_ENVIRONMENT}/tmp"/>

There are more ways for modifying the values unfortunately I chose an inconvenient and sophisticated one. :)

<target name="_change.values">
  <xmltask source="../${env.BUILD_ENVIRONMENT}/tmp/System/SMTP_Servers/EmailServer.SMTPServer" dest="../${env.BUILD_ENVIRONMENT}/tmp/System/SMTP_Servers/EmailServer.SMTPServer">
    <!-- change email server URL -->
    <replace path="/xml-fragment/*[local-name(.)='serverURL']/text()" withText="${EmailServer.smtp.Server_URL}"/>
    <!-- change email server port number -->
    <replace path="/xml-fragment/*[local-name(.)='portNumber']/text()" withText="${EmailServer.smtp.Port_Number}"/>
  </xmltask>
  ...

And I created a properties file and I defined the values.

EmailServer.smtp.Server_URL=XXXXXX
EmailServer.smtp.Port_Number=25

Creating a JAR file is easy too:

<jar destfile="${config.jar}" basedir="../${env.BUILD_ENVIRONMENT}/tmp"/>

Unfortunately I had to implement a 2.1 and 2.2 step as well... When the JAR file was created by the Weblogic ANT task the Configuration Project level resources (e.g. SMTP server) weren't put into the JAR. I don't know it was a problem of the ANT task or I made something badly (but of course it is less likely...:) ).
So I had to copy some folders (see below) into the JAR by the help of ANT and modify the ExportInfo file into the JAR too.

UPDATE

I uploaded my deploy framework that I used to deploy the components, please let me know if you have any issue.

deploy framework

The beginnings... - how to organize OSB components

To create and support an OSB platform is not an easy task. Ok it might be more precise to say "it is hard and very difficult". I think I have told nobody a great truth...

You have to address and find good solutions for the next problem domains:

•  design
•  develop
•  deploy
•  monitoring
•  management

And we have arrived already ... this is SOA Governance for OSB. But I don't want to begin with this topic now so let's start with the basics: how to desing.

I would like to refine the topic above not how to design but how to make a structure  for OSB components. It doesn't seem to be important but believe me it is! If You've got a lot of services on OSB and You haven't got a well-defined vision about how to arrange your services then ... well You will go crazy after a while.

Let me tell You an example. Suppose that You've got a virgin OSB and You have to develop and deploy a proxy service which will poll an email server. It will send the content of attachment of incoming email to a system via webservice. This email is sent by your customers and contains their orders.
We would think 'it is no problem let's create a folder name it as EmailService and put the OSB proxy service into it'. It is a "good" solution it will work.
After some days our boss will find out to process emails coming from our suppliers using OSB. Perfect, we've got a folder for services polling emails let's put this new proxy services into it as well. Then we have to receive orders from our customers via webservice. And here is the real dilemma: we need to create an other folder for this proxy services? But these two services (receiving orders from customers via email and webservice) belong to the same client... Should we create a new folder and name it as CustomerServices? And there is an other trick here. These services mustn't belong to the customer because the customers here are just the clients of services, the Order Management System must own these services.
I would like to clarify this problem, check the next figure:

As You can see multiple clients try to use the same OSB service so the the service must belong to the Order Management System.
If we consider this idea we will realize that in this aspect the OSB is a part of the source system. Ok it sounds strange but have a look at the next figure:

The OSB is transparent for the clients they just want to call the services of the Order Management System. The OSB can extend the possibilities of the source system so the clients can access it via new protocols, other data structure and so on.

This is my vision about the structures of OSB components:
 

* services_OrderManagementSystem (folder)

       * atomic (folder)

              * order (folder)

                     * webservice (folder)

                            * business (folder)

                                   - BS (business service)

                             * proxy (folder)

                                   - PS (proxy service)

       * domain (folder)

              * order (folder)

                     * email (folder)

                             * proxy (folder)

                                   - PS (proxy service)

* services_UnknownSystem (folder)

       * atomic (folder)

              * unknown_business_logic (folder)

                     * webservice (folder)

                            * business (folder)

                                   - BS (business service)

                             * proxy (folder)

                                   - PS (proxy service)

The systems of the company are on the top level of directory structure. Other name conventions can be used on top level as well e.g. the name of business areas (CRM, billing and so on) but for me it was the best one. As You can see on the figure below these are Oracle Service Bus projects and not folders.
On the second level: Is the service an atomic or domain service?  And what is the difference between atomic and domain services? The simple answer is the complexity. I mean if You want to pass the incoming request to the source system and don't want process it this is an atomic service. But if You need to do something with the input data this is a domain service (e.g. if You have to do protocol transformation then the service mustn't be atomic). It depends on You which one is an atomic or a domain service and actually whether You need this level or You don't.
Third level: I think the services must be distinguished by business logic.  In this sample the client can send orders to source system with the OSB components and it means we use this service managing orders.
The fourth level is the protocol. It can ease the navigation in OSB You can easily find the service which You need.
Last level: the proxy and business services must be put into different folders.

It is up to You to use the levels otherwise. For example change the second and third level because it better for your requirements.

So this is my pattern how to organize the OSB components. It helped me a lot not to get confused.

UPDATE1
********
Nobody can be  experienced enough... :)

So if your services are heavily used (100.000 hits / day) then usually you have to deploy these services onto different services (I am not talking about cluster. In case of clusters all the services can be found on each managed servers.
For example you have got 5 services and 2 of 5 services are on the cluster 1 (which consists of more managed servers) and the remaining services are on cluster 2).

In this case you have to group the services and deploy them onto different servers (if you are using the grouping above you can deploy all the services in a OSB project onto only one server).
It is more reasonable to group services according to your cluster structure. What I mean is:

You want to deploy serviceA and serviceB onto cluster1 and serviceC, serviceD and serviceE onto cluster 2 then:

* cluster_1

       * service_A

       * service_B

* cluster_2
       * service_C

       * service_D

       * service_E

My original idea about grouping is correct if your services are not heavily use and you want to keep together your services in a cluster. However if some of your services are overused then you should think about deploying your services individually.