Saturday, February 29, 2020

Debatching(Splitting) JSON Message in Logic Apps - ForEach and SplitOn

Introduction

In last post Debatching(Splitting) XML Message in Logic Apps - ForEach and SplitOn we understood about Splitter pattern and how to use it to split/debatch messages.

In this post we will see how to debatch/split JSON message

Basic difference between XML (Extensible Markup Language) and JSON (JavaScript Object Notation) is - xml is a markup language and uses tag (that's why its heavier) structure to represent data items whereas JSON is a way of representing objects. 

JSON supports only UTF-8 encoding whereas XML supports many type of encoding


Steps to implement Splitter/Debatching Pattern in Logic App


Scenario : 


Say we want to debatch/split PurchaseOrders json message having multiple Purchase Order in it

{
"PurchaseOrders": {
"PurchaseOrder": [
{
"Address": [
{
"Name": "Ellen Adams",
"Street": "123 Maple Street",
"City": "Mill Valley",
"State": "CA",
"Zip": "10999",
"Country": "USA",
"_Type": "Shipping"
}
],
"DeliveryNotes": "Please leave packages in shed by driveway.",
"Items": {
"Item": [
{
"ProductName": "Lawnmower",
"Quantity": "1",
"USPrice": "148.95",
"Comment": "Confirm this is electric",
"_PartNumber": "872-AA"
},
{
"ProductName": "Baby Monitor",
"Quantity": "2",
"USPrice": "39.98",
"ShipDate": "1999-05-21",
"_PartNumber": "926-AA"
}
]
},
"_OrderDate": "1999-10-20",
"_PurchaseOrderNumber": "99503"
},
{
"Address": [
{
"Name": "Cristian Osorio",
"Street": "456 Main Street",
"City": "Buffalo",
"State": "NY",
"Zip": "98112",
"Country": "USA",
"_Type": "Shipping"
}
],
"DeliveryNotes": "Please notify me before shipping.",
"Items": {
"Item": {
"ProductName": "Power Supply",
"Quantity": "1",
"USPrice": "45.99",
"_PartNumber": "456-NM"
}
},
"_OrderDate": "1999-10-22",
"_PurchaseOrderNumber": "99505"
},
{
"Address": [
{
"Name": "Jessica Arnold",
"Street": "4055 Madison Ave",
"City": "Seattle",
"State": "WA",
"Zip": "98112",
"Country": "USA",
"_Type": "Shipping"
}
],
"Items": {
"Item": [
{
"ProductName": "Computer Keyboard",
"Quantity": "1",
"USPrice": "29.99",
"_PartNumber": "898-AZ"
},
{
"ProductName": "Wireless Mouse",
"Quantity": "1",
"USPrice": "14.99",
"_PartNumber": "898-AM"
}
]
},
"_OrderDate": "1999-10-22",
"_PurchaseOrderNumber": "99504"
}
]
}
}

Splitting using For each control


Create Http based Logic app 

In Azure portal create new instance of Logic App, once ready - in designer 

Add http trigger, followed by ForEach and Response Action
for each logic app

It is in For each control where we specify the path of the repeating node (which is to be splitted) 


adding expression in for each

triggerBody()['PurchaseOrders']['PurchaseOrder']

Using path above - we are asking to traverse through the triggerBody and look out of PurchaseOrder property and if found then add entry in  For each array.(All occurrence of PurchaseOrder property are added).

Now we add next steps within For each by using Add an action, here I have added Send an email step. So Send an email will be executed based on items in For each array, if there are 3 items then 3 times email will be sent out.

email action in for each

Note : The default behavior of For each loop is to run in parallel and each iteration in it runs in different
thread, however it can be made to run in sequential form by setting its Degree of Parallelism to 1.

Another example using For each debatching -- Inserting Multiple Records In On Prem SQL Using Logic App

Splitting using SplitOn property on trigger


Create Http based Logic app

In Logic app Designer add HTTP trigger followed by Create blob action (just keeping it simple :) )

logic app with split on


In Create blob action, I have specified the location where the blob is to be create with what name(PurchaseOrderNumber property in coming JSON) and the content of it(Whatever is received in trigger)

create blob  action

We need to set splitOn property on trigger, but no where it is visible -- Not able to understand why Microsoft has not provided.

No problem, we can still do it by going to Logic app code view, and under triggers, add splitOn property with value as path expression (same path is used as in For each above)


split on in logic app code view

Save it, and now you should be able to see in Designer view as well in trigger Settings


Split On in Trigger settings


Setting this property forces Logic App engine to create equivalent number of instances as that of number of repeating nodes, so if there are three repeating nodes in message posted to this logic app, then 3 different instance of this logic app will be instantiated.

Note: You can't use SplitOn with a synchronous response pattern , read Error related to this
-- The workflow with 'Response' action type should not have triggers with 'splitOn' property




Testing


Using Postman, sent a request(JSON PurchaseOrders) having three PurchaseOrder property in it to both the logic app



Logic App run history with Foreach control

Three emails were sent by this logic app

for each logic app history


Logic App run history with SplitOn property on trigger

Three blob were created in debatchedOrders container by three logic app instance

split on logic app history



Note: In both the logic app above message debatching happens, but in second approach the process/workflow also gets splitted (it scales out based on splitted messages by spinning new instances of itself). However in first logic app single instance does all the work.



Related Post 

Monday, February 24, 2020

SplitOn property doesn't validate expression at design time

While working on a POC about Debatching in Logic Apps using SplitOn,  I was encountered with an below error when tried to test the workflow
expression evaluation failed on splitOn property

                           



 { "error": { "code": "InvalidTemplate", "message": "The template language expression evaluation failed:
'The template language expression 'triggerBody()[‘PurchaseOrders’][‘PurchaseOrder’]' is not valid:
the string character '‘' at position '14' is not expected.'." } }

Why it happened


The error points out to the expression having invalid character in it at position 14 i.e. the single quote. 

First thought was -  if that is the case then Logic app designer should have stopped me from saving the workflow (Design time validation), but it did not.

And when request was posted to it, logic app runtime gave the error.  Struggled a bit with resolving this and it happened to be a character encoding issue… Thanks to Manojkumar Sachdev for pointing it out.

The reason was I typed in the expression in Microsoft Word, copied it from there and pasted in code view. Saved the logic app and when started testing got the error.  


What to do



To resolve this issue, deleted the single quote which was copied from Microsoft word and typed in single quote directly in code view.

replacing incorrect single quote with correct single quote

That’s it.. all working fine.

Still, wanted to know why Design time validation didn’t stop me from saving the workflow?


Tried to emulate it with Foreach loop, copied single quote from Microsoft word and added in expression and tried to save it - And I was not allowed, design time validation stopped me from doing it.

reproducing error with foreach expression


So it was clear that issue is only with SplitON property, so tried couple of things with it

1. Removed @ and added incorrect expression -- design time didn't allow me to save
without @



2. Kept @ and added incorrect expression -- design time didn't allow me to save

with @

Not sure if this is bug, or is this how splitOn is designed -- will try to reach out to Microsoft Product team and update if any answer received.


Let me know if you have some inputs on this in comments !!!

Sunday, February 23, 2020

Debatching(Splitting) XML Message in Logic Apps - ForEach and SplitOn

Introduction

When working on an Enterprise integration Project, it is very likely to have a scenario where the message would be received in the form of Batches containing multiple messages within a single message. So if processing has to be done on individual messages, how it is done?


The design pattern used here is Splitter, where a Splitter/Debatcher splits the composite/Batched message into individual messages and further processes each message separately.


Splitter Pattern

Now depending on the message size, whether aggregation of the Processing results of each individual message is required, whether fast processing is to be done etc ,above pattern can be adjusted with little variation in approach.


Splitter Pattern with self processing
For example , say you want to stop processing of batch messages if any of message is faulty or you want to do some actions only after all messages are processed, then above approach can be used. Here single instance of process/workflow/splitter will split the batch message into individual messages and process them one by one. 

So it has control as a whole, as it has state of individual messages with it. Thus if any message processing fails you can stop further message processing and take corrective measures.

Above pattern in BizTalk it is done by using Orchestration - Debatching XML Batch Message in Orchestration

And in Logic app it is implemented by using For each control
Splitter Patern with multiple instance
But if that is not the case, i.e. if it is Fire and Forget scenario (no need to maintain state) then each splitted messages can be processed by a dedicated instance of process, it surely makes processing faster.

Above pattern in BizTalk it is implemented at Receive Port using Pipeline Debatching XML Batch Message at Receive Port


And in Logic app it is implemented by using SplitOn property on trigger.

In both the above approaches Xpath plays an important role, using which we tell which repeating node is to be looked for and then splitting/debatching is to be done.

XPath is a query language for selecting nodes from an XML document, basically it describes a way to locate and process items in XML documents by using an addressing syntax based on a path through the document's logical structure or hierarchy.

Note: In both the approaches message debatching happens, but in second approach the process/workflow also gets splitted (it scales out based on splitted messages by spinning new instances of itself). However in first approach single instance does all the work.



Steps to implement Splitter/Debatching Pattern in Logic App


Scenario : 


Say we want to debatch/split PurchaseOrders message having multiple Purchase Order in it

xml Purchase Order


Splitting using For each control


Create Logic app Instance


Create DebatchXMLMessageLogic App using for each

Add http trigger, followed by ForEach and Response Action
foreach la

It is in For each control where we specify the xpath of the repeating node (which is to be splitted)
foreach with xpath


xpath(xml(triggerBody()),'//*[local-name()="PurchaseOrder" and namespace-uri()
="http://www.adventure-works.com"]')

Using xpath above - we are asking to traverse through the triggerBody and look out of PurchaseOrder node and if found add entry in  For each array.(All occurence of PurchaseOrder nodes are added).

Now we add next steps within For each by using Add an action, here I have added Send an email step. So Send an email will be executed based on items in For each array, if there are 3 items then 3 times email will be sent out.

Read Error related to this -- The template language function 'xpath' expects its first parameter to be an XML object

Note : The default behavior of For each loop is to run in parallel and each iteration in it runs in different
thread, however it can be made to run in sequential form by setting its Degree of Parallelism to 1.

Another example using For each debatching -- Inserting Multiple Records In On Prem SQL Using Logic App

Splitting using SplitOn property on trigger


Create Logic app Instance

In Logic app Designer add HTTP trigger followed by send email action (just keeping it simple :) )


split on la

We need to set splitOn property on trigger, but no where it is visible -- Not able to understand why Microsoft has not provided.

No problem, we can still do it by going to Logic app code view, and under triggers, add splitOn property with value as xpath expression (same xpath is used as in For each above)


Spliton property in logic app code view

Save it, and now you should be able to see in Designer view as well in trigger Settings


split on property in trigger settings


Setting this property forces Logic App engine to create equivalent number of instances as that of number of repeating nodes, so if there are three repeating nodes in message posted to this logic app, then 3 different instance of this logic app will be instantiated.

Note: You can't use SplitOn with a synchronous response pattern , read Error related to this
-- The workflow with 'Response' action type should not have triggers with 'splitOn' property




Testing


Using Postman, sent a request having three Purchase node in it to both the logic app

And three emails were sent by each logic app, however in first logic app single instance
did it whereas in second logic app 3 separate instances did it. It can be checked in Logic
App run history

Logic App run history with Foreach control

for each logic app history


Logic App run history with SplitOn property on trigger

split on la history





Related Post 

Saturday, February 22, 2020

The workflow with 'Response' action type should not have triggers with 'splitOn' property

While working on a POC about Debatching in Logic Apps using SplitOn,  I was encountered with an below error when tried to add Response action in the workflow

                           SplitOn property error



"Failed to save logic app DebatchXMLUsingSplitON. The workflow with
'Response' action type should not have triggers with 'splitOn' property
defined: 'manual'."


Why it happened


As I had to debatch an xml message, following xpath expression was provided to splitOn property


xpath(xml(triggerBody()),'//*[local-name()="PurchaseOrder" and namespace-uri()
="http://www.adventure-works.com"]')

So this property will make logic app to instantiate equivalent number of the instances
as that of number of splitted message.

Say, a batched message with 3 messages in it is posted to this Logic App, then 3 instances
of logic app gets created.

And this is the reason, why designer stops us when we try to add Response action -
because incoming HTTPS requests state is no more preserved(connection gets closed), as new instances gets
created thus ending the session.

What to do


Remove the Response action and save it :)







The template language function 'xpath' expects its first parameter to be an XML object

While working on a POC about Debatching in Logic Apps using For Each,  I was encountered with an below error when testing it

The template language function 'xpath' expects its first parameter to be an XML object



"InvalidTemplate. Unable to process template language expressions for action
'For_each' at line '1' and column '1658': 'The template language function
'xpath' expects its first parameter to be an XML object.
The provided value is of type 'String'.
Please see https://aka.ms/logicexpressions#xpath for usage details.'."


Why it happened


As I had to debatch an xml message, following xpath expression was provided to ForEach action

xpath(triggerBody(),'//*[local-name()="PurchaseOrder" and namespace-uri()="http://www.adventure-works.com"]')

And when xml message was posted, the logic app was not able to apply xpath on the
triggerbody

It happened because triggerBody() is an Azure Workflow built-in FUNCTION, which is used to access
the output of trigger(shorthand for trigger().outputs.body) and it’s return type
is String.

What to do


Logic app is very well equipped to process XML message apart from JSON, however if any expressions(functions) are to be applied on it then it has to be explicitly handled by casting it to xml and then use xml based functions.

So the correct expression is


xpath(xml(triggerBody()),'//*[local-name()="PurchaseOrder" and namespace-uri()
="http://www.adventure-works.com"]')

In the above expression first triggerBody() (which is string) is casted in XML using xml() function and then xpath function is applied.


That's it, all worked fine.