Monday, November 10, 2014

BizTalk Server: Multiple XML files to Single FlatFile Using File Adapter


It’s very common in integration project where we have to deal with debatching the batch of files and batching multiple files into single file. Depending upon requirements we can do xml debatching, EDI debatching, Flat file debatching and xml batching, flat file batching, EDI batching with the help of respective assembler and disassembler.

This post demonstrates how to batch multiple xml files into single flat file using flat file assembler and file adapter.


The best example for this would be “Collection of errors for a particular day in a single file”. But to keep this simple and easy to understand following scenario is used for this post – Solution is to be designed to collect all the Employee messages(xml) coming in on that particular day and to be batched into single flat file and saved as “todaysdate.txt”

Say, 10 employee messages came in between 12:00 AM – 12:00 PM on 10th November 2014 then all the employee message should be batched into single flat file and saved as 10-11-2014.txt

How To Do 

Input Schema

Output Schema

Custom Flat File send pipeline

There is no default flat file pipeline provided out of the box, so we need to create custom flat file pipeline. As we need to create flat file at send side, we create flat file send pipeline with the help of Flat file Assembler

Creating Orchestration

Have used Orchestration for the sake of assigning Current date as filename to the outgoing message, but yes this solution can be implemented as pure messaging solution by creating a custom pipeline component for assigning the filename.

Construct shape

1. Transform Shape

Simple one to one mapping from xml to flat file.

2. Message assignment

In message assignment shape, System’s current date is assigned to a string variable SystemDate and later it is assigned to context property FILE.ReceivedFileName, so we are actually overwriting the values of FILE.ReceivedFileName. We did this so that we can use %SourceFileName% macro at send port, which reads this context property and uses it to name the message saved at destination location.

Possible error:  The Filename, directory name, or volume label syntax is incorrect

Build and deploy

After done with the development, sign the project and assign it a Name and deploy.

Configuring and testing the application 

Download Sample

Related Post 

BizTalk Server: Multiple XML files to Single FlatFile Using File Adapter - Configuring and testing

Configuring the application 

Receive Port

Receive Location

Send Port

Send pipeline: MXSFF (Custom flat file)

Destination URI

  • File name: %SourceFileName%.txt (puts the value of the FILE.ReceivedFileName)
  • Copy mode: Append (The file send handler opens a file if it exists and appends a message to the end of the file. If the file does not exist, the file send handler creates a new file)

So for example, when first message or batch of messages are received on date 10-11-2014 then new file will be created at destination location having 10-11-2014.txt and then after whatever messages are received until the end of date are appended to the same file. But a new file will be created for messages received on next day.

Configuring send pipeline (Flat file assembler)

Configuring Orchestration

Testing the application

To test the functionality of application, considered two test cases

Testcase 1: First files of the day

Input: Placed three sample files in XMLIN folder (input location)

Output: A file with today’s date was generated at FFOut (destination location)

Testcase 2: Rest files of the day

Input: Placed two sample files in XMLIN folder (input location)
Output: Records were appended to the existing file

Download Sample

Saturday, November 1, 2014

Begineer to BizTalk Expert : Interview with Tord Glad Nordahl

Welcome to fifth interview of the series, today's expert is Tord Glad Nordahl.

Tord Glad Nordahl is working as a BizTalk Administrator Expert for Bouvet ASA in Norway. He worked with BizTalk for the last 6 years however he’s been working with all versions of BizTalk. His main jobs is Tech Lead of Bouvets Integration Competence Center, health checking, traning, and as ad-hoc support during critical problems for clients. He’s set up BizTalk over a 100 times for different clients all over the world. Tord Glad Nordahl is also an active contributor to the community with his work on his blog, on the MSDN TechNet forums and Wiki.

He was awarded Microsoft Integration MVP for his commitment for the BizTalk community for two consecutive years (2013 and 2014). He’s held presentation all over the world and is always up for a challenge when it comes to BizTalk.

Tord is also a member and co-founder of BizTalkCrew along with Nino Crudele, Saravana Kumar, Steef-Jan Wiggers and Sandro Pereira that are responsible for organizing BizTalk Innovation Day event all across Europe. Before starting the interview, I would congratulate Tord for the new journey which he is going to start today with Microsoft, by becoming part of it.

Mahesh: Who are you and what you do? 
Tord : My name is Tord Glad Nordahl, I'm 27 years and got three kids, Natala, Maxim and Maribell. I've worked as a BizTalk consultant for the lest three years, my nation focus has been operation, but I've done development too.

Mahesh: When did you start working on BizTalk? 
Tord: It all started 6 years ago, I was working for a rather big company in Norway collecting all web services and putting them on a single webfarm. I was in the same room as two BizTalk administrators. One day when both of them were out of office BizTalk stopped working and I was put on the task, i fixed it but at the same time my passion towards BizTalk grew drastically that day.

Mahesh: How did you mastered BizTalk (Learning path, amount of time)? 
Tord : I would say it was learning by doing and playing around with it. When i first started working with BizTalk the stuff for operation like blogs, wiki articles etc didn't exist. I remember learning alot but starting up my blog.

Mahesh: Which are the major projects you handled so far? 
Tord: Although I wish I could talk about all protects I've done, most of them are under NDA. However I can say this, the biggest one was an upgrade from BizTalk 2010 to BizTalk2013 for 16 environments.

Mahesh: How do you see BizTalk compare to other integration platform? 
Tord: BizTalk has its share of "competitor" products although none of these products it's similar to BizTalk, thanks to our message box BizTalk stands out. People tend to focus on the negative side of BizTalk, like there is little innovation, but remember that Microsoft BizTalk is the only product from Microsoft that has never had a security update.

Mahesh: What as per you is must to know to become an Integration(BizTalk) Expert? 
Tord: Becoming an integration expert is I impossible, there is simply too much to grasp it all, you could probably on the other hand be expert on certain areas, like HL7, Maps etc. Either what you choose you need to read, try, fail and do it all over again. 

Mahesh: What are your thoughts on forums,blogs and articles etc.? 
Tord: The BizTalk community has grown a lot the last years, the amount of articles, blogs and forum responses has grown. I love song the community blossom like this. Keep up the good work.

Mahesh: Your suggestion to a newcomers? What should be approach to get sound knowledge in Biztalk? 
Tord: Get yourself a sandbox and play around on it. Practice is the key and you'll get better and better for every day, Simone once said that a good developer it's a developer that knows how to use Google (or Bing). Ask anyone and don't be afraid to do so. You wont be looked at in a bad way.

Mahesh: What are your thoughts around BizTalk certification? 
Tord: Well, it's good to acknowledge your experience, but during recruitments we don't focus too much on the certification, the "solution" to all question is out there and there for focusing on good questions during the interview that is not found online that easy. :-)

Mahesh: What is the future of BizTalk? 
Tord: The future of BizTalk is bright. The marked is focusing on creating cost effective ways to run their business,integrating their systems is a good way to achieve these goals, BizTalk is still marked leader, with new releases every year even Microsoft shows off the fact that they are investing in the product.

Mahesh: Any thoughts on cloud? 
Tord: So the cloud is the way to go, and were all know that moving everything to the cloud in one step is not smart, however for many technologies, including integration, utilizing the cloud for processing can remove some of the dynamic lead you have on the on-prem solutions.i think the main focus for companies is to look at hybrid solutions.

Mahesh: What motivates you to do the community work?
Tord: Remember sharing is caring!

Mahesh: Would you like to share about your journey of becoming MVP and now joining Microsoft?
Tord: So I don't have any travel plans when it comes to my career, however I do have a personality that our me in positions to get awards like the MVP Award. But it was never a goal. I love to skate and the rewards you heartbeat is priceless, getting a thank you from someone you've never met it's worth more than any award.. Do community work because you want to, not because you want an award. It's not a race.

So loosing my MVP award this year is a little sad, although it's a part of my new journey when I join Microsoft (you cannot be an MVP and MSFT). I will however still work with BizTalk in MS as a technology solution professional.

Thanks a lot Tord, great insights, this will surely benefit many.

Feel Free to ask questions to Tord in the comments!!!!!!!!

Related Post: