Wednesday, March 18, 2020

XML Batching(Aggregation) in Logic App

Introduction

One of my connection on Linkedin asked for help 



So basically his requirement is to send all the individual records in a group to CRM endpoint i.e. a Batch of individual records. This is called Aggregator or Batching pattern in Integration.

aggregator pattern
When we say we want to group messages then comes a question as to how to  wrap them.In BizTalk we handle such scenario with help of Envelope schema (for Debatching and Batching). 

In logic app there is a special trigger for grouping of the messages - Batch trigger.


In last post Batching(Aggregating) messages in Logic App we saw how to group JSON messages, in this post let's see how to batch XML message.

One of the primary thing to consider is valid that XML document has single Root Node.

We would create two logic app 1. XMLBatchProcessor 2. BatchProcessCaller


Steps to implement Aggregator/Batching Pattern in Logic App


1. Create a XMLBatchProcessor Logic App

From the Azure Portal select Create a resource -> search of “Logic App” -> click Create button

After instance is created, go to logic app designer and to add trigger - search for Batch. Select the Batch message trigger and configure batch process trigger .

  • Batch Mode 
Here we have two option

- Inline: For defining release criteria inside the batch trigger itself (for this post am using this - it's kind of hard coding). 

IMO This should be opted if you don't have integration account or you have limited batch processing requirement.

- Integration Account: Ideally, this is should be preferred, as it helps for defining multiple release criteria configurations and also you don't have to make updates in Logic app every time release criteria is changed . It is like a configuration file sitting in an integration account,so you can maintain these configurations all in one place rather than in separate logic apps
  • Batch Name
The name for your batch, which is "CommonBatch" in this example  and applies only to Inline batch mode
  • Release Criteria

As we have used Inline batch mode, we need to select the criteria to meet before processing each batch (we can select multiple criteria also):
- Message count based: Release the batch based on the number of messages collected by the batch.
- Size based: Release the batch based on the total size in bytes for all messages collected by that batch.
- Schedule: Release the batch based on a recurrence schedule, which specifies an interval and frequency. In the advanced options, you can also select a time zone and provide a start date and time.
- Select all: Use all the specified criteria.

Here Message count is selected with count as 3

configuring batch process trigger in logic app

Now, what you want to do with Batched Message? 

We want to wrap this group of xml messages with a Root element tag, so that we get a valid XML Batched message and then finally can send it to the required destination for further processing.

So how we do it -- one option would be to create Azure Function - you will have to write code  which will accept the Batched items and return new batched message. And call this function from your logic app.

Another would be to use Inline code - but it is still in preview and currently only supports Java Script.

Here we will make use of string Variable, using which we do following

Initialize variable with Start tag <RootNode> 

So after the Batch Trigger step , add new action and select variable
Add variable

Select Initialize variable


Give the Variable a Name , String as Datatype and against Value provide the start tag you want , here <BatchedMessages> is used

Initialize variable

Keep on appending each Batched Item

Now add new action -- Append to string variable
Append to string variable

Select Message Content of BatchedItems which is output of BatchTrigger step as a value to append. Note that For each gets automatically added.

values to append



Close the Rootnode tag </RootNode>

Next again we add Append Variable action after Foreach and add closing tag to it.

And now the final step is to send to destination -- for this post have used Send email action with attachment

send email action


Note : Batch trigger is a special trigger meant for collecting messages and it only starts the instance once it's release condition is met - so you won't see it's instance in run history till it reaches it's release point. 

Also you would not be able to call it outside of your subscription or from different region.


2. Create a caller Logic App

From the Azure Portal select Create a resource -> search of “Logic App” -> click Create button.

In designer select http tigger and in next action search for Batch, you should see all the logic apps which have Batchtrigger, select the one created in above step.
call xml processor



As soon as you select it, Trigger Name and Workflow will be auto populated, against Batch Name give the same name used in step 1 and for Message content provide triggerbody.

configure call xml processor

That's it, ready to use.


Testing


Using Postman, sent 3 request(xml) having order details in it in it to the BatchProcessCaller logic app
xml input


And below is the email received - which is sent by XMLBatchProcessor logic app
received email

Downloaded the attachment and following is what we expected

final output






Related Post 

Tuesday, March 17, 2020

Batching(Aggregating) messages in Logic App

Introduction

In last couple of posts Debatching(Splitting) XML Message in Logic Apps - ForEach and SplitOn  and Debatching(Splitting) JSON Message in Logic Apps - ForEach and SplitOn we understood about Splitter pattern and how to use it to split/debatch messages.

Batching or aggregation is opposite process of debatching/splitting where we group various individual message and send them as a single composed message for further processing.
aggregator pattern
If you have worked on BizTalk, you can relate to the Parallel convoy - Which is used to receive multiple messages and act upon them based on certain condition. 

Likewise , here the Aggregator/Batch Processor gets started as soon as it receives individual message and keeps on waiting and receiving for other individual messages till the time the Release condition is met.

Release conditions are to set boundary to the compose message, like if size of composed message is reached to certain limit or if count of messages grouped reached to limit or batched messages have to be sent at a scheduled time etc.

When done with grouping it is further sent for processing as one Batch this certainly increases throughput, performance, cost etc - as the number of calls to destination system is reduced.

In this post we will see how to batch/aggregate message, for this we would create two logic app 1. BatchProcessor 2. BatchProcessCaller


Steps to implement Aggregator/Batching Pattern in Logic App


1. Create a Batch Processor Logic App

From the Azure Portal select Create a resource -> search of “Logic App” -> click Create button


Create BatchProcessor Logic App

After instance is created, go to logic app designer and to add trigger - search for Batch. Select the Batch message trigger and configure batch process trigger .

  • Batch Mode 
Here we have two option

- Inline: For defining release criteria inside the batch trigger itself (for this post am using this - it's kind of hard coding). 

IMO This should be opted if you don't have integration account or you have limited batch processing requirement.

- Integration Account: Ideally, this is should be preferred, as it helps for defining multiple release criteria configurations and also you don't have to make updates in Logic app every time release criteria is changed . It is like a configuration file sitting in an integration account,so you can maintain these configurations all in one place rather than in separate logic apps
  • Batch Name
The name for your batch, which is "CommonBatch" in this example  and applies only to Inline batch mode
  • Release Criteria

As we have used Inline batch mode, we need to select the criteria to meet before processing each batch (we can select multiple criteria also):
- Message count based: Release the batch based on the number of messages collected by the batch.
- Size based: Release the batch based on the total size in bytes for all messages collected by that batch.
- Schedule: Release the batch based on a recurrence schedule, which specifies an interval and frequency. In the advanced options, you can also select a time zone and provide a start date and time.
- Select all: Use all the specified criteria.

Here Message count is selected with count as 3

configuring batch process trigger in logic app

Now, what you want to do with Batched Message? you can send it to the required destination/do further processing, for the sake of this post, I have used send email action to send the batched message
send email action


Note : Batch trigger is a special trigger meant for collecting messages and it only starts the instance once it's release condition is met - so you won't see it's instance in run history till it reaches it's release point. 

Also you would not be able to call it outside of your subscription or from different region.


2. Create a caller Logic App

From the Azure Portal select Create a resource -> search of “Logic App” -> click Create button.

In designer select http tigger and in next action search for Batch, you should see all the logic apps which have Batchtrigger, select the one created in above step.


call batch process logic app
As soon as you select it, Trigger Name and Workflow will auto populate, against Batch Name give the same name used in step 1 and for Message content provide triggerbody.

Configure batch process logic app

That's it, ready to use.


Testing


Using Postman, sent 3 request(JSON) having three two properties in it in it to the BatchProcessCaller logic app
request to logic app


And below is the email received - which is sent by BatchProcessor logic app

The message contains some additional fields apart from content(highlighted in yellow), which are details about the Batch .

We saw how we can batch json messages, in following post have covered about XML messages batching  -    XML Batching(Aggregation) in Logic App





Related Post 

Sunday, March 8, 2020

Removing ns0: prefix from xml output from BizTalk/Logic app XSLT map

Introduction

Xml, which stands for stands for EXtensible Markup Language was designed to describe data and to do that you use tags. It's basic purpose is to focus on what data is  - self decsribing. 

And as we have a provision to add our own tags, it becomes Extensible.

Having this properties makes it very suitable for defining a message (Records, elements and attributes) and exchange it across, also data can be stored in it and can be refered/updated as an when needed (most of the configurations are xml based).

In case of where XML is involved in messaging, it becomes vital to have Namespace as it helps in avoiding element name clashes/conflicts.

A Namespace is a set of unique names. Namespace is a mechanisms by which element and attribute name can be assigned to a group. The Namespace is identified by URI(Uniform Resource Identifiers).

An XML instance may contain element or attribute names from more than one XML vocabulary. If each vocabulary is given a namespace, the ambiguity between identically named elements or attributes can be resolved.

For example, you are dealing with OrderXML and you are receiving multiple OrderXML from different vendors - so how do you differentiate which xml(same structure) is from whom - It is with namespace.


To define xml you use xsd and when you create xsd in BizTalk/Logic app Project(using visual studio) few namespaces automatically gets added, also gets added is Target namespace(identifier - that created xsd belongs to this project ) to say that this xsd belong to it.


default namespaces in xsd

The elements (schema, element, complexType, sequence, …) of the schema itself belong to the namespace http://www.w3.org/2001/XMLSchema.

The namespaces which gets added automatically are actually to tell our schema editor that we are referring to definitions in them and it is done with help of namespace prefix - xs and b in above image.  Note that for Target Namespace there is no prefix .

The target namespace is the place were your declared elements and attributes in your schema belong to (you can refer them as Local).

Consider Target Namespace is to schema is what a namespace is to .Net Object. 

Read more about Schema -- BizTalk Server : Deep dive in schema design

How ns0 gets added in output of map using above schema as destination


Now when we do map from one schema to another, we create new xml and to which namespace it
belongs to it gets added and thus you see namespace and it's prefix

 ns0 added in map result

If you see above ns0 gets added in test map result, although there was no prefix associated with TragetNamespace in xsd -- It is the mapper which does it and ns0 is the default prefix.

So if I use xsd having prefix already associated with TargetNamespace  -- Still mapper will add the default prefix ns0.


How to remove default prefix ns0 and have prefix as per our preference


To do this we need to bypass the mapper and for that we make changes in xslt generated
by the mapper.

Right click the map, select debug and you should see the path where xslt is generated,Copy the path.

Now edit the xslt and add preferred prefix and save it
change prefix ns0
You can use saved xslt in Logic app or in BizTalk by setting the custom xslt path and get desired output as below-

As can be seen , edi is now added as prefix appears with Root node.

What if you are required to have this prefix associated with all the local elements and local attributes?

To do that we use following properties Element FormDefault and Attribute FormDefault
This properties are only relevant if a target namespace(it is optional) is used and only for local declared attributes and elements.

Qualified means: “belongs to a target namespace”.
Unqualified means: “not belonging to a target namespace”.

When Default or Unqualified is used -- No Prefix is associated and for
Qualified -- Prefix is associated








Related Post 

Saturday, March 7, 2020

Logic Apps : Fetching ISA and GS Segment Values From Interchange Envelope and Mapping

Introduction


When we work with B2B using EDI standards in logic apps we use Decode component which does work of validation and decodes the EDI(flat file) and based on the settings we specify in Agreement, the Transaction sets are splitted (one edi document comprise of segments contained in ST to SE) and is available in Payload of Decode component for further use. 

EDI X12 messages are wrapped with envelope, at header it has ISA segment (Interchange control header), GS (Functional group) and ST (Transaction Set) and at the trailer it has IEA, GE and SE. Enveloping segments work in pairs. ISA-IEA represents an interchange. GS-GE is a functional group inside of the interchange and ST-SE is a transaction inside the group.

Most of the time, we need to map the individual edi documents(ST-SE) to destination system's expected format.

Read example about it -  Getting Started with Logic Apps - EDI X12 to XML where following map was used

EDI 850 to XML Order Map
If you see the destination schema gets all that it needs from the EDI document transaction set (segments between ST and SE).

What if destination system also want some info from Interchange headers or functional headers? But our Payload has only data segments and no Envelope segments. So how we do it?


Following are post which talk about how we did it in BizTalk

Now let's see how we do it in Logic Apps

Scenario


Say there is a company Techfindings and it wants to receive EDI X12 850 from its business partner and after getting it, it needs to convert it XML Purchase Order format which is expected by it's inhouse application for further processing.

EDI X12 850 files are received and data from it has to be mapped to the destination structure, also Interchange Control number, Interchange Sender ID and Functional Group Header Code values.

So we need to create a solution which will accept EDI X12 850 message and convert it to XML Purchase Order along with the Envelope (Interchange/functional header) values and forward it to Blob location for Inhouse application.

Steps in creating solution

    
    You can refer to previous blog  -  Getting Started with Logic Apps - EDI X12 to XML , which has same solution implemented and explained except for the new mapping  requirement  which I cover here.


   1. Destination schema with Interchange and Functional Header(XML_Orders_WithHeaders)

destination schema with headers


2.Map between EDI X12 850 and Destination schema (XML_Orders_WithHeaders)

map between EDI and destination schema with headers

If you see there is no mapping done for header nodes , it is because we don't have values needed for them in source schema.


3.Where to get Interchange and Functional Header values from

To get the ISA/GS header values in map, first we need to understand the output generated by Decode_X12_message component

Decode_X12_message Output

The output does has a Interchange/functional header values made available as a whole and as an individual elements apart from the actual data (Payload).

And this values can be accessed by any Action followed after Decode_X12_message action.


4.How to make this values available in Map


  Once mapping is done (step 2 above), we need to get the xslt file , for that right click on map file -->Select Debug and in output window you can see the xslt path

xslt path


  make note of path where xslt is stored,as it is this which is to be uploaded to integration account (.btm file is not supported). 
  


  Following is the xslt

<<?xml version="1.0" encoding="UTF-16"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:msxsl="urn:schemas-microsoft-com:xslt" xmlns:var="http://schemas.microsoft.com/BizTalk/2003/var" exclude-result-prefixes="msxsl var s0" version="1.0" xmlns:ns0="http://EDIToXMLDemo.XML_Orders" xmlns:s0="http://schemas.microsoft.com/BizTalk/EDI/X12/2006">
  <xsl:output omit-xml-declaration="yes" method="xml" version="1.0" />
  <xsl:template match="/">
    <xsl:apply-templates select="/s0:X12_00504_850" />
  </xsl:template>
  <xsl:template match="/s0:X12_00504_850">
    <ns0:Orders>
      <Order>
        <Header>
          <PONumber>
            <xsl:value-of select="s0:BEG/BEG03/text()" />
          </PONumber>
          <PODate>
            <xsl:value-of select="s0:BEG/BEG05/text()" />
          </PODate>
        </Header>
        <xsl:for-each select="s0:PO1Loop1">
          <LineItems>
            <xsl:if test="s0:PO1/PO101">
              <ItemId>
                <xsl:value-of select="s0:PO1/PO101/text()" />
              </ItemId>
            </xsl:if>
            <xsl:if test="s0:PO1/PO102">
              <Quantity>
                <xsl:value-of select="s0:PO1/PO102/text()" />
              </Quantity>
            </xsl:if>
            <xsl:if test="s0:PO1/PO104">
              <UnitPrice>
                <xsl:value-of select="s0:PO1/PO104/text()" />
              </UnitPrice>
            </xsl:if>
            <xsl:if test="s0:PO1/PO107">
              <ItemDescription>
                <xsl:value-of select="s0:PO1/PO107/text()" />
              </ItemDescription>
            </xsl:if>
          </LineItems>
        </xsl:for-each>
      </Order>
    </ns0:Orders>
  </xsl:template>
</xsl:stylesheet>

  Now we make changes in above xslt to enable us to get the header values, and
for that we add parameters and assign them to respective nodes in destination
(marked in red)

<?xml version="1.0" encoding="UTF-16"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:msxsl="urn:schemas-microsoft-com:xslt" xmlns:var="http://schemas.microsoft.com/BizTalk/2003/var" exclude-result-prefixes="msxsl var s0" version="1.0" xmlns:ns0="http://EDIToXMLDemo.XML_Orders" xmlns:s0="http://schemas.microsoft.com/BizTalk/EDI/X12/2006">
  <xsl:output omit-xml-declaration="yes" method="xml" version="1.0" />
  <xsl:param name="SenderId"/>
  <xsl:param name="ControlNumber"/>
  <xsl:param name="Code"/>
  <xsl:template match="/">
    <xsl:apply-templates select="/s0:X12_00504_850" />
  </xsl:template>
  <xsl:template match="/s0:X12_00504_850">
    <ns0:Orders>
      <Order>
        <Header>
          <PONumber>
            <xsl:value-of select="s0:BEG/BEG03/text()" />
          </PONumber>
          <PODate>
            <xsl:value-of select="s0:BEG/BEG05/text()" />
          </PODate>
        </Header>
        <xsl:for-each select="s0:PO1Loop1">
          <LineItems>
            <xsl:if test="s0:PO1/PO101">
              <ItemId>
                <xsl:value-of select="s0:PO1/PO101/text()" />
              </ItemId>
            </xsl:if>
            <xsl:if test="s0:PO1/PO102">
              <Quantity>
                <xsl:value-of select="s0:PO1/PO102/text()" />
              </Quantity>
            </xsl:if>
            <xsl:if test="s0:PO1/PO104">
              <UnitPrice>
                <xsl:value-of select="s0:PO1/PO104/text()" />
              </UnitPrice>
            </xsl:if>
            <xsl:if test="s0:PO1/PO107">
              <ItemDescription>
                <xsl:value-of select="s0:PO1/PO107/text()" />
              </ItemDescription>
            </xsl:if>
          </LineItems>
        </xsl:for-each>
      </Order>
  <InterchangeHeader>
        <ISA04>
          <xsl:value-of select="$SenderId" />
        </ISA04>
        <ISA13>
          <xsl:value-of select="$ControlNumber" />
        </ISA13>
      </InterchangeHeader>
      <FunctionalHeader>
        <GS01>
          <xsl:value-of select="$Code" />
        </GS01>
      </FunctionalHeader>
    </ns0:Orders>
  </xsl:template>

</xsl:stylesheet>


Save the modified xslt and upload it to integration account.

Go to the integration account created earlier in step1  and select the Maps tiles and click on add button. 
   Give name EDI850_toXMLOrderWithheaders and browse to xslt path(saved in step 3) and save.

 5. Use above XSLT in Logic App 


In Transform XML action


Against content add following expression -- xml(base64ToBinary(item()?['Payload']))

Against map select EDI850_toXMLOrderWithheaders and as soon as you select it
Parameters which you added in xslt will be should diplayed
xslt parameter in logic app

So now we need to pass the values to the parameters, against SenderId -- ISA06 , ControlNumber -- ISA13 and for Code -- GS01

Passing values to parameter

That's it , we are done with all required to do in Transform XML, complete the logic app design and save it and let's test.




Testing


Below is EDI X12 850 test input posted using ARC to logic app



Test input

Logic app receives EDI 850 messager over http, then it Decodes EDI X12 message  and then  transforms it  in xml Order  with help of Transform XML action and finally places the converted file in blob container.

Below is xml file created in blob container and its content


final output










Related Post 



ServerLess360