Monday, May 29, 2017

Begineer to BizTalk Expert : Interview with Eldert Grootenboer

Welcome to the 27th interview of the series, today's expert is Eldert Grootenboer

Eldert is Integration architect working on the Microsoft platform. Designing and building integration solutions with Microsoft products, specialized in BizTalk, IoT and Azure.

Very active in the integration community by blogging, writing, publishing on Technet Wiki, creating code samples, and speaking at events internationally. Recognized by Microsoft as a Microsoft Azure MVP for these efforts

Let's Begin Interview....

Mahesh:Who are you and what you do? When did you start working on BizTalk? 
Eldert:I am Eldert Grootenboer, an integration architect from the Netherlands. I work at Motion10, a company focused on integration, business productivity and BI. I’m mainly focused on integration using Azure and BizTalk, as well as on IoT, for which I am a Subject Matter Expert within Motion10 as well. I have been in IT since 2006, went into the integration space in 2010 with BizTalk, and have been working with Azure almost from the beginning as well, with technologies like Service Bus, Logic Apps, API management etc. In my spare time I also like to be active in the community, by attending and speaking at conferences and user groups, blogging (both at my own blog and as a guest blogger for BizTalk360), providing code samples and more.

Mahesh:How did you mastered BizTalk (Learning path, amount of time)? 
Eldert: I started working with BizTalk in 2010 on a project with one of my colleagues, who showed me the ropes when it came to integration in general and BizTalk in particular. I immediately started getting deeper into the technologies myself by lots of self-study, attending community events and getting to know specialists from the field. I did my first project alone one year later, and have been doing several projects, both big and small, ever since.

Mahesh:Which are the major projects you handled so far? 
Eldert:I have been working on several projects, ranging from general integration implementations to projects where I’m more in an architect or consulting role, helping the customers figure out what would be the best solution for them. Currently I am working on a project where the customer wants to go completely to Azure with all their integrations, having several subsidiaries all with their own processes and applications. Together with the customer we are now setting up a complete architecture which will be future proof and can easily incorporate new processes.

Mahesh:What do you think is the most challenging part while working on Integration project?
Eldert: The most challenging part of integration, which is also the most interesting to me, is working with different technologies all the time, whether it’s SQL, SAP, CRM, legacy applications, etc.. When it comes to integration you have to understand all these systems, and be able to determine how to incorporate them in a complete solution.

Mahesh:How do you see BizTalk compare to other integration platform? 
Eldert:In my opinion BizTalk is an amazing product, especially when it comes to on-premises integration. The product is very robust, gives us many options to easily communicate with systems, and gives good insights into what is happening in your integrations, which is key to having a stable environment. The possibilities we now have to also easily connect to the cloud, also makes it a very good choice in hybrid scenarios, where you have part of your application landscape living in the cloud, and another part on-premises.

Mahesh: What as per you is must to know to become an Integration(BizTalk) Expert? 
Eldert: You really have to love working in integration to become an expert. You will be dealing with all technologies, ranging from old legacy software to cutting edge. Also the integration space is evolving rapidly, with Azure technologies pushing out updates all the time. This means you have are never done with learning. And as you keep learning, don’t forget to share your knowledge with others as well.

Mahesh:What are your thoughts on forums, blogs and articles etc.? 
Eldert: I think these are great resources, and I love how generous the integration community is with sharing their knowledge. Whether you are new to the integration space, or have been here for a long time, anyone is welcome to share their ideas and knowledge.

Mahesh:Your suggestion to a newcomers? What should be approach to get sound knowledge in BizTalk? 
Eldert: Start by learning the basics. There is a wealth of information about BizTalk, Azure, integration patterns and more out there. There is Pluralsight where multiple great courses can be found, books going from high level introduction to deep-dives into specific parts of technologies, blogs where the community are sharing their ideas, videos on Integration Monday and Middleware Friday, and instructor led courses. And also make sure to attend your local user groups, people are always willing to welcome new people, so go and meet them.

Mahesh:There are many tools from community which support BizTalk in some or the other way(like BTDF, Bizunit etc), what do you say about it? Which ones you would recommend?Why? No Microsoft support available?
Eldert: The community does a great job filling in the gaps, which will always be there when using commercial tools like BizTalk. Some of these tools target a very specific scenario, where others will recommended be for a larger audience, but it always depends on your own specific scenario. And don’t forget, when you run into one of these gaps, Microsoft has UserVoice pages for their products as well, allowing you to make them aware. And as we have seen with Feature Pack 1 for BizTalk, as well as various updates for Logic Apps, they really do listen.

Mahesh:What are your thoughts around BizTalk certification? 
Eldert: My thoughts around certification in general, is that it’s better to showcase your skills in another way. Speak at your local user group, blog about solutions you have created, things like that. I have found most employers would much rather see you are involved in the community, sharing your knowledge, then having an arbitrary certificate.

Mahesh:What is the future of BizTalk? 
Eldert: BizTalk will remain an important tool in our toolset for a long time to come. I do think we will see more and more hybrid in the future, where BizTalk will play an important part in the on-premises integration space, combined with Azure technologies like Logic Apps, Service Bus, Azure Functions and API Management.

Mahesh:Any thoughts on cloud?
Eldert: As already said in the previous question, hybrid is coming strong, as we see more and more companies moving partly or completely to the cloud. Thanks to Microsoft’s push in the last years we now also have enterprise grade technologies in the cloud, allowing us to provide our customers with optimal solutions for each scenario.

Mahesh:What motivates you to do the community work? 
Eldert: I have had a lot of help from the community over the years, and try to do the same as well. I love sharing my knowledge, and interact with experts from around the world, many of which I can now count among my friends as well.

Mahesh:As per the Roadmap provided by Microsoft,LogicApps can be run on-premise in addition to Azure.Do you think Azure Stack Logic Apps on prem will supersede BizTalk Server? 
Eldert: I don’t think it will, BizTalk has its own strengths and weaknesses, as does Logic Apps, and for now they complement each other. Of course they have some overlap, so for some scenarios Logic Apps may be able to replace BizTalk, but I think BizTalk will have its own place in our toolset for quite some time.

Mahesh:From my perspective, Microsoft keeps coming up with Overlapping technologies like recent ones MABS and Logic Apps, in some situation it gets puzzling. What you say?
Eldert: I think most products have their own specific scenarios which they target, and depending on your scenario, you should find the product which fits best. 

Mahesh:Do you think BizTalk in cloud (IAAS) is accepted over BizTalk on Prem? Which one you prefer - what pros and cons you see?
Eldert:I think IaaS is good for BizTalk DEV and TEST, but personally I don’t see the added value in running your BizTalk PROD in IaaS. Of course there will be scenarios where it would be a good fit, but if you are looking into this, I would recommend looking into the possibilities for shifting your integrations into PaaS / SaaS technologies like Logic Apps.

Mahesh: I see you involved with books, which books would you recommend?
Eldert: A must read for anyone in the integration space is Enterprise Integration Patterns by Hohpe and Woolf, which gives a great explanation of the various patterns which we can use, and which still apply in the current cloud driven world. 

Mahesh: Being an MVP, do you think responsibilities get added?
Eldert:As an MVP, I get approached a lot by people from the community, and I do think it’s one of our responsibilities to make sure we help people out whenever we can. Luckily I love helping people, and I know my fellow MVP’s feel the same as well.

Mahesh: How do you see the step of Microsoft releasing Feature Pack for BizTalk?
Eldert:As I wrote in a blog post when Feature Pack 1 was released, Microsoft is showing us with this feature pack that they still believe in BizTalk, and are finally showing BizTalk the love it deserves. For too long we have seen only minor improvements, but with the new pro-integration team, who really are making a great impact, we now see real improvements again, on both BizTalk and Azure side.

Thanks a lot Eldert for taking out time and sharing your insights,experiences, this will surely benefit many !!!

Feel Free to ask questions to Eldert in the comments!!!!!!!! 

Related Post:

Saturday, May 27, 2017

BizTalk Feature Pack1 - Exploring Azure Application Insights

Continuing from where I left - Installing BizTalk 2016 feature pack 1 , let's walk-through the process of utilizing the feature - Azure Application Insights.

Steps are simple and no coding is required.

1. Create an instance of Application Insight

  • Login to Azure portal and click on New and search for Application Insights
      search Application Insights

  • Select Application Insights and click on Create
select Application Insights and click Create
  • Give name , choose Application Type as General (As of now BizTalk is not added in list),Select subscription,Resource group, Location as per your convenience and click on create. 
set Application Insights properties and click Create

  • That's it, your Application Insight instance is ready to use

Application Insights instance Created

  •  In order to sync data to this instance we need its Instrumentation Key, copy it and use when enabling Analytics on BizTalk Group

Application Insights essentials

2.Install BizTalk Feature Pack1 (If not done already)

After you install Feature Pack, you have to Enable group-level analytics and select Target Type as Application Insight (as of now only this option is available) and against the Connection parameters provide the Instrumentation key of the Application Insight instance to which you want to send the tracked data.

enable Application Insights on group level

Note: By Default group-level tracking is enabled.

3. Create BizTalk application or use existing application whose telemetry data is to be pushed to Azure Application insights

I am using already existing application, and enabled the analytics on the artifacts(RecceivePort,SendPort and Orchestration)

enable Application Insights on receive portenable Application Insights on send port

enable Application Insights on orchestration

4. Check the data on Azure Application insight

Run few test transactions,Login to Azure portal and click on the Instance of Application Insight. On left pane look for Usage and in middle pane for Custom event (Time range I have selected 7 days)

Application insights usage

Note: Events pushed from BizTalk are categorized as custom event in AI.

If interested in getting more details of any event, then just double click and new blade opens with more details (below I selected event marked in yellow and in next blade bar representation of Real User Traffic - it is one of the filter)

Application insights events

You can filter the events based on other properties too, just click on filters and you will see list 
Application insights events filters

You would see that this matches to the context properties when tracked message are looked upon in Admin console.

Does tracking has any effect on data ?

So the question might arise, is Tracking on the artifacts have to be turned on? To verify this, I unchecked all the tracking on orchestration - only analytics was enabled and ran few test followed by verifying the data at AI. All the events (receive/send)happening are reflected in AI.

Application insights usage deprecated

Another way to check the events is to Click on Search and look for custom events (Usage is deprecated)
Application insights search events

So what is the data sent

It is the same data which is visible in Admin console when we enable Track Events. Below are the tracked events of  an orchestration(with Track Events enabled) viewed in group hub.

Biztalk tracked events

With data going to AI, there are lots of options to query the data and also various option around viewing the data.

 How will following matter:  

1. BizTalk group can be synced to only one Application Insight instance as  of now
2.  Live stream can't be used as of now

Monday, May 22, 2017

JSON Decoder : Unexpected character encountered while parsing value

After fixing an error with JSON decoder -Property not specified in JSON decoder, got one more error from JSON decoder-

 Unexpected character encountered while parsing value

Unexpected character encountered while parsing value <

From the error it is clear that the data in response from the api is not valid JSON, thus the decoder throws error.

Why it happened

I went through the tracked service instance and checked what exactly is received, and as can be seen in below snap - it was HTML and no JSON message. 

response html message

And as the returned message starts with <  , the decoder was not able to parse it as valid JSON message is like:

   "company": "Maruti",
   "name": "Swift",
   "price": 800000

Note: The RestApi has not implemented the required functionality, only instance of webapi is provisioned in Azure - thus the HTML is returned.

What to do 

Here, we can't do anything except get the functionality implemented in Rest Api and let it return valid message. This error can also be encountered in case you are using JSON decoder and you receive XML.

Sunday, May 21, 2017

Error xxxx not specified in JSON decoder properties

Am working on a POC where I have to invoke a Rest Api, collect the response and do futher processing on it. So to accept JSON data coming in response, I created a custom pipeline having JSON decoder at decode stage. After deploying the project and configuring it, dropped the test instance at receive location and watching for output in OUT folder(configured to archive API response).

As no output was generated - means some error was encountered, visited event log and found below error:

Root Node is not specified in property

Why this happened

It was my bad, missed to set couple of properties which are required by JSON decoder to convert JSON message in corresponding XML message i.e. Root Node Name and Namespace.

Root Node  and Namespace propertyin JSON decoder

Above error is in situation where you do not specify Root Node Name and Namespace both or if only Namespace is set.
Out of curiosity wanted to check what JSON decoder says if I set Root Node Name only and leave Namespace as Blank - It says, I want it to build XML message for you ;)
Namespace is not specified in property

What to do

When using JSON decoder it is mandatory to provide this two properties. It is because the Key is converted into element and Value as element value.


JSON Message:

   "company": "Maruti",
   "name": "Swift",
   "price": 800000

Corresponding XML:


But a valid XML has a Root Node, thus that needs to be provided explicitly to JSON decoder to attach the converted XML structure under Root Node 

Friday, May 19, 2017

Installing BizTalk 2016 feature pack 1

Microsoft recently released feature pack 1 for BizTalk server 2016, this is first time such feature pack is presented and off course it is for enriching customers experience that too without breaking existing functionality.

Organizations can only use Feature Pack 1 with BizTalk Server 2016 if they have the Enterprise or Developer editions of the product, plus Software Assurance. No other editions are eligible. It's also possible to use Feature Pack 1 if BizTalk Server is run in Azure with an Enterprise Agreement, the MSDN article explained

So thought of installing it and exploring it, installation is quite straight forward - steps below

2. Run the installer 
BizTalk feature Pack1 installer

BizTalk feature Pack1 installer wizard page 1

BizTalk feature Pack1 installer wizard page 2 Accept agreement

BizTalk feature Pack1 installer wizard page 3

BizTalk feature Pack1 installer wizard page 4

BizTalk feature Pack1 instalation complete

BizTalk feature Pack1 instalation complete restart

This installation requires to restart the server to reflect the installed features.

Microsoft claims Feature Pack 1 is production ready and provides only non-breaking features (Off course we need to verify this before thinking of Production). Is there anything noticeable thing(after installation) supporting below features - there saw few things


1. Deploy with VSTS – Enable Continuous Integration to automatically deploy and update applications using Visual Studio Team Services (VSTS)
2.New management APIs – Manage your environment remotely using the new REST APIs with full Swagger support


1.Application Insights – Tap into the power of Azure for tracking valuable application performance and to better understand your workflow 

To support this there is an option provided in tracking section wherein we can Enable Analytics. What is Application insights - After creating Application Insight, capture the instrumentation Key, as this is needed when enabling Analytics on BizTalk group and it is passed as connection parameters

BizTalk feature Pack1 enable Application Insight on Group

BizTalk feature Pack1 enable Application Insight send port

BizTalk feature Pack1 enable Application Insight receive port

BizTalk feature Pack1 enable Application Insight Orchestration

2.Leverage operational data – View operational data from anywhere and with any device using Power BI


1.Support for Always Encrypted – Use the WCF-SQL adapter to connect to SQL Server secure Always Encrypted columns

SQL Server 2016 introduced a feature that enables client applications to read/write encrypted data within a SQL table without actually providing the encryption keys to SQL Server and that's been included in WCF SQL adapter as a property, by default it is Disabled.

BizTalk feature Pack1 enable Sql ColumnEncryptionSetting

2.Advanced Scheduling – Set up advanced schedules using the new capabilities with Feature Pack 1 

If you would have used Scheduled task adapter, this was on developers wanted list for long time to have scheduling capability on granular level. Including Time zone is really thoughtful

Advanced Scheduling

Tuesday, May 16, 2017

The Login is from an untrusted domain and cannot be used with Windows authentication

During daily routine checkup, we found following warning and error in event viewer.

At first it might appear from the error that BizTalk Server machine and SQL server machine are in different domain or the user account used for configuring the BizTalk service is from a different domain than from SQl server's domain  -- None is the case 

The Login is from an untrusted domain and cannot be used with Windows authentication  warning

The Login is from an untrusted domain and cannot be used with Windows authentication error

Why it happened

It is known issue and this errors are result of restart of Domain controller(A domain controller is the centerpiece of the Windows Active Directory service. It authenticates users, stores user account information and enforces security policy for a Windows domain) or unavailability of same over the network,thus the BizTalk applications cannot authenticate users until the Domain Controller(s) is not up.And it is because SQL is not able to find the authenticated domain user (as domain controller is not available at moment) and thus throws error. 

What to do

There were no more errors as soon as Domain controller was up but there can be other reasons too, refer following link:

Saturday, May 13, 2017

BizTalk host BizTalkServerApplication throttled because DatabaseSize exceeded the configured throttling limit

On one of the BizTalk servers were many of  BizTalk WCF service are published started giving  Timeout issue (it was working absolutely fine before that).

The first thought after seeing timeout error is to check the WCF timeout settings, but it couldn't be the reason as those were already configured while deployment, also it worked fine in load scenario too. So it was something else which was to look upon.

Upon visiting the event viewer, found a warning - BizTalkIsolatedHost throttled because DatabaseSize exceeded the configured throttling limit

BizTalk host BizTalkServerApplication throttled because DatabaseSize exceeded the configured throttling limit

Learning - Never ignore Warnings ;)

Why it happened

The most probable reason for increase in database size is due to misbehavior of the jobs which are intended to keep everything flowing smoothly in SQL(BizTalk). Thus it needs to be validated  that the jobs are configured properly and also if required clean/shrink the database. 

So it was MessageBox_Message_ManageRefCountLog_BizTalkMsgBoxDb which was failing continuously with below error.
Executed as user: xxx\xxxx. The SELECT permission was denied on the object 'sysjobs', database 'msdb', schema 'dbo'. [SQLSTATE 42000] (Error 229).  The step failed

MessageBox_Message_ManageRefCountLog_BizTalkMsgBoxDb calls the MessageBox_Message_Cleanup_BizTalkMsgBoxDb which deletes the message which have fulfilled the subscription . Whenever message is published in Message box, subscribers get copy of the message and not the actual message as messages in BizTalk are immutable. Upon fulfilling the subscriptions messages are marked for deletion.

Thus messages were getting piled up.

What to do

Somehow there was access issue for the account, which was granted and the job MessageBox_Message_ManageRefCountLog_BizTalkMsgBoxDb was enabled. By default MessageBox_Message_Cleanup_BizTalkMsgBoxDb  is disabled and should not be run manually.