Monday, April 6, 2015

Fetching Data From Multiple Tables Through Dynamically Invoking Stored Procedure

There is a comment/question posted on following post:Polling data from SQL using WCF-SQL Adapter in BizTalk 2010 - TypedPolling(From Multiple table)  where Vinoth posted the scenario:

"Please help me, I have to poll the data from 15 Tables, Table by Table, should not join in to single select statement. For example, if I have 15 Tables to poll, I have to poll Table1 and store as XML and i have to poll Table 2, then 3, 4, 5...
Multiple Table polling by looping, how to do it dynamically poll the data Table by Table...
Please provide me the solution? its very urgent... "

There are situations where we have to configure WCF port dynamically,

1. The information for the configuration is available at runtime through some content in message or through some other way.

2. The configuration information is already known but it can change or has to be changed depending on the business needs.

To demonstrate how to configure WCF Port dynamically and how  to Invoke stored procedures dynamically, have used following scenario:


 I have to poll the data from Tables, one by one and save it on file system. The challenge is the number of tables can increase or can be reduced .For this demonstration , I have used 3 tables (Logic remains same for 15 tables as well).

Ok, as per the requirement posted the configuration of the port which fetches data has to be done dynamically and in a loop, so nothing has to be static inside orchestration related to the storedprocedure thus used BRE as it provides a way to do changes without having to change anything in orchestration after initial CallRules configuration. 

I have made some assumptions:

1. There is chances of changes in number of tables to be fetched - thus have used BRE which has provision of changing number of table without touching the Orchestration.
2. The response received is not to be mapped, instead it is to be saved as it is.
3. Invoking stored procedure can be anywhere in the designed process, thus have used dummy Order to instantiate the orchestration (which has nothing to do with logic)

Building the solution:

1. Database

a. Creating three tables

Figure1. Demo Database with three tables

Figure2. Order table design

Figure3. Order1 table design

Figure4. Order2 table design

b.Creating three stored procedures with simple query

Figure5. uspGetOrder1 stored procedure
Figure6. uspGetOrder2 stored procedure
Figure7. uspGetOrder stored procedure
Figure8. Three tables and respective stored procedures

2. BRE

a. Creating Policy GetConfigDetails

Figure9. GetConfigDetails Policy

b.Creating four rules under GetConfig

The first rule is used to set the number of tables to be fetched.

Figure10. GetConfigDetails Policy - Rule1

The rest three rules are used to set the properties i.e. Action,Address,Bindingtype and TransportType which are required to configure the port dynamically.

Figure11. GetConfigDetails Policy - Rule2

Figure12. GetConfigDetails Policy - Rule3

Figure13. GetConfigDetails Policy - Rule4


3. BizTalk Solution

For solution the name given is DynamicInvocationOfStoredProcedures which has schemas(intiating message, Configuration message , storedprocedure message), Orchestration, strong name key(used to sign the project) and binding files which are generated by wizard while creating schemas from stored procedures. Steps are as below.

Figure14. Solution structure

a. Create schema for message to instantiate the orchestration

b. Create schema for message to hold configuration details - which is updated using BRE

All the elements are promoted as distinguished field so as to get/set values from those in orchestration

c. Generate schemas from the stored procedures which are to be executed dynamically

We need to do this so that we can refer to the binding files generated to get configuration details.

1. Right click the project and  select Add-->Add Generated Items
2. Then select "Consume Adapter Service"
3. Now the Consume Adapter Service wizard wants us to provide it the required information, so firstly select "sqlbinding" against the label Select a binding.
4. Click the Configure button, on security tab select the Client credential type
5. On URI Properties tab:
      i.Type the name of the server where your database is located ( I have used "." as the database resides on same machine)
     ii. Provide the name of the database you want to use.
     iii. I have left InstanceName as blank because am using the default instance of SQL (If need to use some other instance then provide the name of that instance)

6. After done with above configuration,click on Connect then select Contract type as Client(Outbound operations)
7. Select the category as Procedures then select procedure based on which schema is to be generated, click ADD
8. Give Filename Prefix (it's optional - but it helps) and click ok
Figure. Generating schema from uspGetOrder storedprocedure

9. Repeat above steps for other stored procedures

d. Create Orchestration

Figure.PollMultipleOneByOneDynamically Orchestration

The orchestration as shown above has following variables and types used in it

Ports and Port Type
1. prtRcvStart of type prttypeRcvStart - to receive intial message thus binded to Order schema as intial message is based on it

2. prtExecuteSP of type prttypeExecuteSP - it is dynamic port to send message to invoke storedprocedure and to receive response from storedprocedure,binded to System.XMLDocument.
Why System.XmlDocument? - As per the requirement multiple storedprocedures are to be invoked but those are not static and are to be set dynamically so this port can't be binded to single schema. And as System.XmlDocument is the base schema for all schema it can be used in the scenario where generic approach is required.

3.prtSndInvokeSPResp of type prttypeSndInvokeSPResp - to send the response received from stored procedure to a folder

Figure.PollMultipleOneByOneDynamically Orchestration variables and types

Message and  Variables
1. msgstart - based on Order schema, used to instantiate the orchestration
2. msgConfig - based on Config schema, used to get config details from BRE
3. msgInvokeSP - based on System.XmlDocument, used to send message expected by StoredProcedure
4. msgInvokeSPResp - based on System.XmlDocument, used to get response from stored procedure
5. varLoop - Int variable to hold the value (number of times loop is to be run)
6. varAction - String variable to hold Action value set on msgConfig by GetConfigdetails policy
7. varAddress - String variable to hold Address value set on msgConfig by GetConfigdetails policy
8. varBinding - String variable to hold Binding value set on msgConfig by GetConfigdetails policy
9. varTransportType- String variable to hold TransportType value set on msgConfig by GetConfigdetails policy
7. xmlDoc - Variable of type System.XmlDocument

Steps in Orchestration

1. Receive start message
2. Construct the config message in message assignment shape, this message will be passed, here to note is that GetTableNumber element has values as Yes  

 3. Configure  the CallRules shape, select the GetConfigdetails policy and message expected by it  - msgConfig

4. Assign  the Loop variable to the value received in NumberOfTables field of msgConfig
5. Set the Loop condition

6. Inside loop , call the policy
7. Assign  the varAction,varAddress, varBinding,VarTransportType to the value received in msgConfig

8. Construct the message to invoke the stored procedure and set it's WCF.Action property - this is where we are dynamically setting the action - which we get from Policy we execute in step 6

9. Send the message to prtExecuteSP and collect the response
10. Send the collected response to the folder
11. So  now reset the loop variable by reducing it by one and also now next table is to be fetched so set TableNumber field of msgConfig respectively.

Allright, so why CallRules is used twice in above Orchestration?

Considering that number of tables to be fetched can be changed, the first callrule is used to tackle this scenario. So when Orchestration passes config message to rule engine having GetTableNumber as Yes, then  the NumberOfTables is set  and returned. And in loop, Orchestration is interested in knowing which StoredProcedure is to be executed(the WCF.Action property) and here second callrule is used to get the Action against the TableNumber.

If you see at Rules above, it will be more clear :)

e. Build, Deploy and Configure

After deployment the dynamic port is automatically added to the Application in the Admin console, whose configuration we did in orchestration dynamically.

So we just need to configure one Recieve Port, one static send port and bind it to Logical ports of Orchestration

Figure. After  Binding Orchestration

To test I dropped the instance of Order message (message which instantiates the orchestration) and three xml files were generated in the folder which is binded to the port which sends the response received from stored procedures.

Note : System.XmlDocument is used as base for the message  to communicate with stored procedures so it's untyped . There are two limitation that the response received can't be used in any map and if there is subscriber in your group for untyped message then copy of it will be passed to it as well.

Download Sample:

Wednesday, April 1, 2015

Begineer to BizTalk Expert : Interview with Michael Stephenson

Welcome to tenth interview of the series, today's expert is Michael Stephenson.

Michael is an independent integration & cloud specialist based in the UK. He is primarily focused around integration technologies in the Microsoft integration platform such as BizTalk, Windows Azure, and a number of other open source offerings. 

Michael has many years of technical leadership and coaching experience and has worked with customers to deliver a number of complex real-world hybrid integration solutions connecting business partners and SAAS applications to an organisations on premise line of business systems.  Prior to freelancing Michael also held a technical evangelist role with Edenbrook who were a large UK consultancy.

Michael is also heavily involved in community activities such as the UK Connected Systems User Group and blogging.  Michael also recently pioneered the BizTalk Maturity Assessment ( which is designed to help customers ensure they are doing the right things to deliver successful projects with BizTalk. Recognized by Microsoft as an Integration MVP - 7 Years.

Lets begin the Interview....

Mahesh: Who are you and what you do?
I am Mike Stephenson, i am 38 years old and live in Newcastle in the North East of England.  I work as a freelance Consultant specializing in Integration and Cloud technologies on the Microsoft technology platform. I work with customers in the UK and abroad to help them create traditional integration solutions and also to understand how to leverage the cloud to create hybrid integration solutions.

Mahesh: When did you start working on BizTalk?
I started working on BizTalk in 2002 with a company who specialized in supply chain outsourcing and have worked on every version of BizTalk since then.

Mahesh: How did you mastered BizTalk (Learning path, amount of time)?
Personally I think BizTalk is a product where you always have something to learn.  There are so many different types of solution you can create whether it is integration patterns you have not used before or applications you have not integrated with.  Today technology is also changing very quickly so there are always new things to learn.
I also like to consider myself to be an Integration Specialist rather than just a BizTalk person.  I think its also very important to learn about the other integration technologies too and to understand where to use what.
For training I usually use Pluralsight and Technet wiki and peoples blogs to keep up to speed and also attend #IntegrationMonday web casts

Mahesh: Which are the major projects you handled so far?
I have handled a number of projects over the years, back when I used to work for consultancies I spent time spent time supporting projects at many different customers and I have also worked on a number of projects with healthcare companies but more recently i am working in the higher education sector

Mahesh: How do you see BizTalk compare to other integration platform?
If you compare BizTalk to other products within the microsoft integration technology suite then I consider BizTalk to be like the swiss army knife which has something for almost any integration problem.  This is why many customers use BizTalk at the heart of their integration solutions.
If you compare BizTalk (or microsoft integration) to other vendors then I think you will find it really depends upon what your use cases are.  I guess as a generalization there are two types of vendors those who provide the heavy weight tools which have been around for years and the newer breed of vendors who offer the lighter weight integration platforms where you can provide simpler ways to solve some of the newer world integration problems but often struggle to deal with the more complex integration solutions.
I think what is interesting about microsoft is it is moving quickly towards a good offering in both spaces. 

When it comes to comparing I think it is important to get past the marketing car sales man view of the products you will come across and to look into the real capabilities and how they are implemented.  This is why i contributed to the whitepaper by Kent Weare

Mahesh: What as per you is must to know to become an Integration(BizTalk) Expert?
I am not a great fan of the term integration expert because expert implies you know everything and in integration there is such a wide range of things to consider that you can never know everything about everything and the best skill is to try and be humble and meet new people with different ideas and be learning all of the time.

To excel at integration i think you need to combine knowledge about the integration products you use with knowledge about best practices for application life cycle management and then the non technical skills such as how to work well with other people and how to understand managing dependencies.

I think you need to have a well rounded skill set of the above things to do well in the integration space

Mahesh: What are your thoughts on forums,blogs and articles etc.?
I have always thought the biztalk community have an excellent set of blogs, forums etc.  I have my own preferences about how i like to interact but there is always opportunities to get involved and the community is very supportive of new contributors

Mahesh: Your suggestion to a newcomers? What should be approach to get sound knowledge in BizTalk?
I think the most important thing is hands on experience and the best way to get that is working with lots of different people to discover the practices they use to decide which you like best.  Books, videos etc are all great but to get to the next level its really about hands on experience and appreciating that no matter how much experience you have you should be learning every day

Mahesh: What are your thoughts around BizTalk certification?
I think BizTalk certifications have a value up to a point but over the years i have met people who had certifications who werent that great and i have met many people who have not considered certifications who were truly excellent.

If I was interviewing people a BizTalk certification would not be a factor which would make me choose on person over another for a role but is certainly a nice badge to have on the CV and I guess the best thing about certifications is that it can make you look at parts of BizTalk you may have never used before.

Mahesh: What is the future of BizTalk?
Right now I am more excited about the future of BizTalk and Microsoft Integration than I have been at anypoint for a long time.  I think combining BizTalk with the new azure technologies will let us build some great solutions.


Mahesh: Any thoughts on cloud?
I am a huge fan of cloud and like to work with customers who have a desire to consider where the cloud could benefit their organisation.

I would say to anyone now working commercially with cloud you absolutely must be learning about it in your own time or you will quickly fall behind which could affect your opportunities in the future.

Mahesh: What motivates you to do the community work?
: I think I like the challenge of doing interesting stuff and learning but also over the years i have developed some excellent friendships with the people in the integration community and its cool to work together on little projects and also to help and encourage each other

Mahesh: Being MVP, do you feel that responsibilities get added? What is your thought on MVP?
I dont think the mvp programme adds extra responsibilities.  It is an excellent way to bring together community influencers and to help encourage people to participate in the community.  I think ive been in the programme for 7 or 8 years now and as result ive developed friendships with people in many countries around the world.  It rocks!

Thanks a lot Mike for great insights, this will surely benefit many !!!

Feel Free to ask questions to Mike in the comments!!!!!!!!

Related Post: