Thursday, October 12, 2017

Begineer to BizTalk Expert: Interview with Mikael Sand

Welcome to the 30th interview of the series, today's expert is Mikael Sand .

Mikael is Senior integration archiect, BizTalk and Logic Apps expert, as well as general integration greyback at Enfo Integration Stockholm.

Also, avid twitter-user, public speaker, singer and master of ceremonies.

Let’s begin the interview…

Mahesh: Who are you and what you do? When did you start working on BizTalk?
Mikael: I am Mikael Sand, 43 years old, living just outside Stockholm (Sweden). I am an Solution Architect and Business developer at Enfo. I started doing BizTalk by attending a course in 2006. By 2007 I got my first project.

Mahesh: How did you mastered BizTalk (Learning path, amount of time)?
Mikael :Initially by attending the official BizTalk course, then by working with talented people like and reading blogposts, but most importantly: thru a constant strive for knowledge. My main drive is always that I want to know more. To become good at it I think I spent about a year to 18 months.

Mahesh:Which are the major projects you handled so far?
Mikael: I was the tech lead for the integration team at CGI for about 4 years. I was an architect for making a large Swedish hospital integration focused. I made sure everyone got their pensions for a year, and beefed up the integration platform at an airport operating company in Sweden.

Mahesh: What do you think is the most challenging part while working on Integration project?
Mikael: Often, to make the integration parties see why they need integration. It is much easier for them to build a point to point, and the integration team might seem like we make that much more complex. Also, let’s not forget that integration might land right in the middle of a company’s internal conflicts and even making the parties speak to you might be a challenge. And then there is SAP…

Mahesh: How do you see BizTalk compare to other integration platform?
Mikael: Like a warm fuzzy blanket, full of things I know by heart and inside out. Often an easy day at the office, but also as a very troubled child with a lot of special needs and cries for attention. In some ways it feels old and unwilling.

Mahesh: What as per you is must to know to become an Integration(BizTalk) Expert?
Mikael: To work independently. Sure, you can use Google to search for answers, but you really need to know how to get from specification to a finished integration, including design and documentation, without outside help. Hopefully you do not need to work by yourself but if you can’t then you are not an expert.

Mahesh: What are your thoughts on forums, blogs and articles etc.?
Mikael: I would not, and BizTalk would certainly would not, be a thing without them. Blog posts on BizTalk might not be as important anymore as the number of things that are undocumented become less and less, but they are still out there. Most of the best articles was written back in 2006-2010.

Mahesh: Your suggestion to a newcomer? What should be approached to get sound knowledge in BizTalk?
Mikael: Not much has changed since the core was written in 2004 so find someone older that knows BizTalk. That way you don’t make the same mistakes as he/she did. Know what blogs are good and how to search for solutions.

Mahesh: There are many tools from community which support BizTalk in some or the other way(like BTDF, Bizunit etc), what do you say about it? Which ones you would recommend? Why? No Microsoft support available?
Mikael: I have not used BTDF nor BizUnit, but I have used the old blogical SFTP adapter and the schedule adapter. Great stuff that really filled a void in the product. The downside of community supplied components is that they are very dependent on the person that developed them in the first place. If that person decides to do something else, the product dies after the next platform upgrade. This is very unusual in the BizTalk community though.

Mahesh: What are your thoughts around BizTalk certification?
Mikael: Heh, that old thing. Of course, there should be a certification on BizTalk, at least if Microsoft wants to support BizTalk as a product and a profession.

Mahesh: What is the future of BizTalk?
Mikael: Uncertain I think. Things was going well but then Tord quit and now I do not know. There is a joke in the community that BizTalk will become a proxy server for integration with Azure. There are too many installations of BizTalk out there for Microsoft to kill BizTalk but then again, they need to modernize it more or it will die due to lack of BizTalk developers.

Mahesh: Any thoughts on cloud?
Mikael: A very wide question. I have been using and proclaiming the upsides of the public cloud since about 2009. You have to use it the right way and for the right things, but I really see no reason to use on prem or in some cases even infrastructure, when designing a company’s IT landscape.

Mahesh: What motivates you to do the community work?
Mikael: Fame and all that sweet money. No really, I like sharing information. I like being the first on the scene and then teaching everyone else not to make the same mistakes I did.

Mahesh: As per the Roadmap provided by Microsoft, Logic Apps will be able to run on-premise in addition to AZURE. Do you think Azure Stack Logic Apps on prem will supersede BizTalk Server?
Mikael: I think they might complete one another. Clients/companies does not choose a technical platform, like Azure Stack, because of the integration engine. If the client continues to run its VM-ware server cave, you will use BizTalk. But running Logic Apps on prem might be a very good solution to some scenarios.

Mahesh: From my perspective, Microsoft keeps coming up with overlapping technologies like recent ones MABS, Microsoft Flow and Logic Apps, in some situation it gets puzzling. What you say?
Mikael: “The right tool for the job”, etc. Yes, in some ways it might be confusing but not all companies are the same or have the same need or staff. Flow might be enough for a small client without any IT-department.

Mahesh: Do you think BizTalk in cloud (IAAS) is accepted over BizTalk on Prem? Which one you prefer - what pros and cons you see?
Mikael:  I have not yet seen any good documentation on installing BizTalk on Azure. Then again, I have not really been looking. I have said this before: Installing BizTalk on Azure is like upgrading your black and white TV. The image might be sharper and better than ever, but over here we have color TV.

Mahesh: How do you see the step of Microsoft releasing Feature Pack for BizTalk?
Mikael: One of the best things that has happened to the product in years. I have extra features with a shorter release cadence is awesome.

Mahesh: Microsoft is keeping on adding many things under Hybrid Integration, what should be Integration developers be focusing on?
Mikael: Tough question. I really do not know. Based on recent experience I would say to focus on whatever the current project demands. Even after you have chosen a technology, that tech is moving around, and you learn new ways to use it. Just do not be afraid to change, I would say.

Mahesh: Do you think the cost of using cloud services (Logic App, Service Bus etc) will be more than having infrastructure on prem. (Where transactions are in millions or more)?
Mikael: No idea. I usually make sure to have cost in mind when designing integrations right from the start. Then again, looking at the competition (IBM, Mule) their reaction is usually that Microsoft are too cheap.

Mahesh: Recently BizTalk components are made open source, how do you see this Microsoft Move?
Mikael: Moving to open source might be a way to make the community doing development for you, which we already do, and they have not released the good stuff, yet but it is a step in a new direction.I wonder where they are heading. 

Thanks a lot Mikael for taking out time and sharing your insights, experiences, this will surely benefit many !!!

Feel Free to ask questions to Mikael in the comments!!!!!!!!

Related Post:

Monday, October 9, 2017

Getting Started with Logic Apps - Enterprise Application Integration


In last post we saw, how to do file routing using LogicApps. But in real world, Enterprise Application Integration have more than just routing – Messaging Solution or some kind of Business Process Automation(It can be simple or complex in nature).

Messaging solution basically revolves around VETER, it can be either combination of all or few of them. In Logic Apps we have XML Validation to validate, Transform XML to do transformation, xpath can be used to extract xml and to Enrich(compose new message) and route  we have connectors available.

In this post we will see with an example, how to validate, transform and route using Logic App.


Requirement is to route the Trip related details which is received in csv format from sales department in a folder to destination folder but in XML format which is expected by downstream system.

In order to implement this solution In logic app, we need to have schemas related to csv and xml , and a map doing a transformation from csv to xml.

Unlike BizTalk which provides us platform to create the artifacts needed (schemas, maps, pipelines etc) which can be orchestrated or used in work flow, logic app is platform for only designing/orchestrating  workflow around the services available (using Trigger, Connectors, Azure functions etc.).

Although we can design Logic app using browser, we don’t have provision to create schema or map using browser but they can be used in workflow if available to it. 

To make this availability possible– Integration Account is introduced which is part of Enterprise Integration Pack, which acts as a container having capability to store Schemas, Map, Certificates, Trading Partners, and Agreements. And to deal with the creation of Schemas and Maps there is SDK provided, It can be downloaded from below link

Microsoft Azure Logic Apps Enterprise Integration Tools for Visual Studio 2015 2.0

Note: If you already have BizTalk env, then you don’t need to download the SDK, the schemas and maps created in BizTalk application are absolutely compatible. Also note that extension for visual studio version above 2015 is not yet available as extension are part of BizTalk and BizTalk 2016 is the last released version. Once latest version of BizTalk is out, respective extension for visual studio will also be made available.

Once you have schemas and maps created, using either of the above ways – those can be uploaded to the Integration account. Later this Integration account has to be linked to Logic App, in order to use the artifacts.

Steps in creating solution

For demonstration, I have used folder as source and destination which is on-premises. In real world scenario it can be anything - source/destination can be on prem or in cloud. 

Creating schemas and map

As am using BizTalk for creating schemas and map, I create BizTalk project whereas if you are using SDK, you would be creating Integration project(this template gets added to visual studio when SDk is installed.

BizTalk New Project

Create one flat file schema-Trip.xsd .Its extension is xsd as the representation of flat file is in xml but with the help of FlatFileExtension class all properties related to flat file are made available.
flat file schema

and one xml based schema - Trip_Xml.xsd

xml schema

Now create a Map with Flat file schema as source and Xml schema as destination, mapping is one to one, expect have concatenated the address.

flat file to xml schema

One more step, before we move ahead is to save the xslt generated from above map. Right click Map-->Validate Map. In output pane, you will see the path of stored XSLT, copy the file and save it in your current project folder.

Creating Integration Account, Uploading schemas and maps

Login to Azure portal and click on new button, search for Integration Account. Provide the name, resource group, Pricing , Location and click on create.

Create Integration Account

There are two options in Pricing Tier 1. Free and 2. Standard, would recommend to use Free as you are able to create a maximum of one account per region and store up to 10 Agreements and 25 maps. Schemas, certificates, and partners have no limits and you can upload as many as you need.

Now to upload the schemas, select the above created Integration Account-->click on schemas on left pane--> click on Add button --> provide name --> browse to folder having the schema of interest--> click ok
upload schemas in Integration Account

and to upload the map,select the above created Integration Account-->click on Maps on left pane--> click on Add button --> provide name --> browse to folder having the map of interest(choose the xslt file we saved earlier)--> click ok
upload map in Integration Account

Note: Only xslt file is supported as of now, .btm is not supported. Read The provided map definition is not valid

Creating Logic App

Note: Make sure your integration account and logic app are in the same Azure location before you begin.

Now on Azure portal click New and search for Logic App, select the blank Template.Provide the name, region, resource group etc.

First thing to do is after creating and before designing, we need to link the integration Account in order to use schemas and map (you can link it later too but you will get error while you look for schemas or map)

link integration account with Logic apps

The workflow is – Pick flat file from on prem folder, decode it using Flat file decoder, validate it against the flat file schema,  transform the file and save it to another on-prem folder .

Configure the File System connector to trigger the logic app when new file is created in folder after every one minute, followed by action to fetch the content of file. Read Getting Started with Logic Apps - File Routing
file system trigger and action

Add new Action and search for Flat file decoding, use the File content from previous step and select the flat file schema uploaded earlier in Integration Account Trip_FF

Flat file decoding action

Next Action is to add XML Validation, use Body of previous step and select the schema to validate against - Trip_FF
xml validation action
Now we have valid decoded Flat file which needs to be transformed to expected XML, add Transform XML action, select the Body from previous step (here you will see two Body select the one related to Flat File Decoding. Read Required property 'content' expects a value but got null. Path ''.'

Transform xml action

The final step is to save the above transformed file to destination location, add Create File action, provide the out folder, name and use the content from Transform XML
create file action

That's it, final workflow is as below, save it and test.
final eai workflow


For testing I generated an instance from flat file and update the values in it, and saved it in the IN folder

Input to workflow
 And in Out folder I can see the output
Output of workflow

Saturday, October 7, 2017

The provided map definition is not valid

Schemas and Maps created in BizTalk can be used in Logic apps with the help of Integration account, but got below error while uploading map from my BizTalk VM (schemas were uploaded successfully)

The provided map definition is not valid

Why it happened

As error says Provided map definition is not valid , so is something wrong in map? - No. Map is perfectly valid and tested too. Although BizTalk Maps are compatible, the extension is not i.e., .btm is not understood instead it accepts only xslt.

look for xslt file

Once you click to select map file, it explicitly looks for XSLT File.

What to do

BizTalk Map

BizTalk map gets compiled into xslt, and we can get that xslt by simply right clicking the map --> Validate Map. 

validate map

Friday, October 6, 2017

Required property 'content' expects a value but got null. Path ''.'

Upon triggering a logic app which is intended to transform a flat file into xml, saw an error in Logic App run on Transform XML action

Required property content expects a value error

Why it happened

I had to pass input to Transform XML action i.e. the Body which is received after Flat File Decoding, what I was trying to do is - use Body from previous step in Transform XML and I happened to choose the Body parameter from XML Validation action.
Incorrect body to transform XML action

Off-course the Body (Output of XML Validation) will not have the Flat File content, in-fact there is no output. If file is valid, step gets passed else fails.

What to do

Correct Body has to be provided, which conforms to the source schema used in the Map which is executed in the Transform_XML action i.e. the body from the Flat File Decoding stage.

Correct Body to Transform XMl

Learn More about some more Logic App errors