Monday, October 21, 2019

Getting Started with Logic Apps - What happened to the Request?

What happened to the request  or what happened to the order with Order No #xyz or why the particular order is not processed etc etc – I am sure most of us have faced or atleast heard of it and it is most common thing if you are working in Integration domain 😊 .


Offcourse it is well known situation and way to handle it is to identify the key business entities and have them logged in database/monitoring application and run query against it to do the findings. If custom Integration application is build then it should be definitely considered  while designing.

If you are to use someone one else’s product or services  then do check if there is provision for tracking or any out of box monitoring capability available and you can search or query on that tracked data based on the needs.  If you know BizTalk, then you know that there is out of box feature available i.e. Business Activity Monitoring (BAM) .

Now let’s get to the topic of blog post, tracking key business entities in logic app. We do have provision in Logic App wherein the run history of each logic app is maintained and the happenings around each action can be seen (inputs and outputs) – I personally think this is very powerful and helps a lot in troubleshooting the issues but does this solve the above mentioned issue “What happened to the order with Order No #xyz”.

Upto certain extent yes it is helpful, say if you know the time when request was processed then you traversed down to that request processed time or near to it and check few logic apps one by one. Now what if there are hundreds or thousands of requests processed during same time?  - Still it can be done by going through them one by one but the amount of time needed would be too much and the frustration too 😏.

Logic App Product team should really consider this and add a search capability on top of run history.

I am sure down the line in the near future it will be provisioned, but how do we cater the situation now?

For now we can have a function written to log the details to either custom database or application insights and call that function from the logic app and we can search/query the database/application Insight thus saving lot of time and efforts. But note we are to put efforts to write code and efforts in managing (code and database).

So is there a better Way


Yes, there is and it is with the help of Log Analytics service, it is azure monitor log service which collects  Log data (from any Azure resources or any external resource) and  stores it in a Log Analytics workspace (which can be further analyzed by the Log Analytics service through the Log analytics portal).

And provision to integrate Logic App with Log Analytics workspace is through linking them while creating logic app instance and mentioning the properties to track wile developing logic App. In our Logic App, we can add tracked properties onto different actions to track our custom data(e.g, OrderID or MsgID etc)
in diagnostics data.

Let's see how to do it 

1.Create your Log Analytics Workspace


create log Analytics Workspace


 In Azure Portal-->All services -->Monitor and select Log Analytics workspace

configure log analytics workspace

Give the desired Name,select resource group or create new, select subscription and location. Pricing tier - I would suggest to use Free one as the data is retained for 7 days and it's good enough to  get started .  If you need more days of data retention then choose accordingly 

Note: The first 5 GB of data ingested per customer to the Azure Log Analytics service every month is offered free


2.Create your Logic App



Order Processing Logic App


While creating instance of Logic App, switch On the Log Analytics and then select from drop down list the workspace created in above step.

When you do this you are telling Logic App runtime engine that it should now start sending Diagnostics Logs to the selected Log Analytics workspace. Details such as status, execution time , Start/end of each run, Start/end for each action, resubmission status,Run ids, Resource group, resource name  and correlation IDs.

Behind the scene , all the above data is added to AzureDiagnostics Table. 


For the sake of simplicity, a simple logic app is created here which will accept Purchase Order (xml) and will send email to the Bill to contact person

Order Process Logic App Design


Next step is to add Tracked Properties, and to add those select the action (not supported in trigger Action) where you want tracking to happened. 


action settings

You can either edit Logic app in code view and add tracked properties or you can use designer to do by clicking on ellipsis of action and selecting the Settings



Add Tracked Properties in Logic App

Give the name of property being tracked and provide the expression to extract the value, here I want to track only orderID of the purchase Order thus below expression

                  json(xml(triggerBody())).purchaseOrder.orderID

                                          

But you can add more than one property if needed. And below is how it looks in code view


logic App code view



What happens when we add Tracked Properties 

When you add tracked properties, a column trackedProperties_Propertyname_s gets added to AzureDiagnostics table of the workspace you created and at runtime only after the particular action on which tracked properties are added is executed then entry is made in this column. So here if SendEmail action is executed then only orderID will be added else nothing will be seen in that column.



Testing 


To test you can use tool like Postman, ARC etc. I have used ARC 

ARC

And following XML as message body

<purchaseOrder>
    <orderID>PO0025</orderID>
<orderDate>2019-10-20</orderDate>
<description>Discount applied on the Order</description>
    <shipTo>
        <name>Mike Taylor</name>
        <street>MG Road</street>
        <city>Pune</city>
        <state>MH</state>
        <pin>411001</pin>
<country>India</country>
    </shipTo>
    <billTo>
        <name>Mike Taylor</name>
        <street>MG Road</street>
        <city>Pune</city>
        <state>MH</state>
        <pin>411001</pin>
<country>India</country>
<emailID>maheshkumar.tiwari@emtecinc.com</emailID>
    </billTo>
    <items>
        <item>
    <productID>ABC123</productID>
            <productName>ABC product</productName>
            <quantity>1</quantity>
            <price>148.95</price>
    <shipDate>2019-10-21</shipDate>
            <comment>Confirm this is electric</comment>
        </item>
        <item>
    <productID>XYZ123</productID>
            <productName>XYZ product</productName>
            <quantity>1</quantity>
            <price>39.98</price>
            <shipDate>2019-10-22</shipDate>
        </item>
    </items>
</purchaseOrder>

Check the logs

checking Logic App tracked properties in logs

Now to check if all went as desired go to Log Workpace created and click on logs and run query on AzureDiagnostics Table. Say to check about all the orders having orderID having PO002 in it ,following query is fired 

AzureDiagnostics
| where trackedProperties_OrderID__s contains "PO002"


Microsoft is on its way to provide Logic App management solutions on top of AzureDiagnostics.It is still in preview but looks promising 

logic app monitoring solution

Click on configure monitoring solutions and you should see all details presented to you as below

Logic App Monitoring Solution

Apart from grouping the requests, there is feature to apply filters and narrow down the search based on our requirement and another very good feature is the abiltiy to  Resubmit one or more logic apps runs that failed, succeeded, or are still running, like in Biztalk we have ESB Portal.

Conclusion


So if I have to search any Order say having OrderId as PO0031 then I have two options


1. Query the Diagnostics log

In Query tab fire following query

AzureDiagnostics
| where trackedProperties_OrderID__s == "PO0031"

And the result would be as below

result1


2. Search it in Logic App management solution

Click on Logic App runs tile and apply filter on Tracked Properties column

filter on tracked properties

 And the result would be as below

result


Note : It takes time for the tracked data to get synched in Log Analytics. I experienced around 5-10 minutes lag.



If you know any better way then do share !!!

Related Post 


Sunday, September 22, 2019

Getting Started with Logic Apps - AS2

What is AS2?

Every enterprise requires some kind of product, service or counselling from another enterprises thus there happens a B2B(business to business) communications, some of the communication is unique whereas most of the communications are common for most of the enterprises e.g., Purchase Order. But every enterprise has there own way(format) of sending the messages, leading to difficult management when number of partners increased. 

Considering this the Enterprise leaders decided to have some standards/rules defined across the communication leading to B2B protocol standards which provide guidelines for trading partners to follow when conducting business between enterprises. EDI X12, EDIFACT, TRADACOM etc.

AS2 stands for Applicability Statement 2 and is a B2B messaging protocol used to transmit Electronic Data Interchange (EDI)/Business documents from one organization to another. So EDI standards define how to format data and AS2 specifies how to securely transport data via the Internet using HTTP/S (hypertext transport protocol secure). AS2 uses the HTTP POST operation to send business data. In response to an AS2 message, a message disposition notification (MDN) is returned as an acknowledgment, provided the sender of the AS2 messages expects an MDN.


Note: Although AS2 is majorly associated with EDI Standards, however AS2 is an open standard for secure, payload-agnostic exchange of B2B documents.



How exactly it provides Security




AS2 uses the S/MIME(Secure/Multipurpose Internet Mail Extensions) protocol to wrap EDI/business data in a secure "envelope" . Files are encoded as "attachments" in a standardized S/MIME message, so the basic structure of an AS2 message consists of MIME format inside a HTTP message with additional AS2-specific header and send it over the internet, also it allows to makes use of digital certificates for following (not mandatory but recommended as it adds security). 
Digital Signing: For signing the data the sender uses the private key to ensure the sender's identity as the creator of the document, on receiver side the signature on the document is verified using the sender's public key to verify the sender's identity.

Encryption: To encrypt data sender uses the receiver's public encryption key, so only the proper recipient will be able to extract the document, the prepared document is decrypted using the receiver's private key.
Thus above two ensure the integrity of data transmitted.
AS2 also supports compression of the data using a compression algorithm to reduce the size of the transported data.

Another thing is MDN(Message Disposition Notification), which is sent to the sender and it denotes that message from sender is received, validated and decrypted successfully. MDN can be send synchronously(over the same HTTP connection message was received) or Asynchronously (later on given url or via email). 

AS2 Architecture

AS2 Architecture

Whenever two organization decide to go with AS2, they need to agree upon following things and thus an Agreement is saved at each partner and further communications happens based on it.
i. What will be the data format
ii. Which signing algorithm is to be used
iii. If encryption is required
iv. If compression is required
v. If MDN is required 

IF signing and encryption is chosen then each of the partner have to share the public key of digital certificate which will be used for signing and encryption.

The AS2 protocol relies on X.509 certificates for signing and encryption.A .pfx includes both the public and private key for the associated certificate which should never be shared outside your organization, it can be used for TLS/SSL on web site, for digitally signing messages or authorization tokens, or for authenticating to a partner system. A .cer file only has the public key - this is what you share with integration partners, also it can be used to verify tokens or client authentication requests.

AS2 and Logic App together


When we decide to use AS2 for B2B communication, then we need systems/services which understand AS2 and support. Earlier Microsoft had only BizTalk Server(on-premises) which was capable to support AS2 and now we have cloud service too which supports AS2 i.e. Logic Apps.

With Enterprise Integration Pack features required for B2B are made available to Logic Apps,the enterprise B2B capabilities like AS2 and EDI standards support and set of XML capabilities like XML Validation, XSLT Transformation and Flat file to XML encode/decode etc (this all are present in BizTalk server already) .

Both the partners need to procure certificate and share the public certificate with each other.

Steps to enable Logic App for AS2

1. Add  Private certificate in Key Vault

We need to store the Private Certificate in Key vault, and provide the access to the  services which would need the certificates via Access Policies of Key Vault, here it would be Logic Apps.

2. Integration Account

With Integration account you get a container wherein you can store schemas, maps, partners, agreements, certificates etc required for Integration project. Also you get access to following connectors 
Integration Account connectors




Note: Both your integration account and logic app must exist in the same location or region.

3. Add certificates in Integration Account

Click on certificates and add Partner1(sender) public certificate, give it a Name , from Certificate Type drop-down select Public and select a certificate by browsing to the certificate location.



Add public certificate in Integration account

and Partner2(receiver)’s private certificate, give it a Name , from Certificate Type drop-down select Private and select a Resource Group from drop-down which has the Key Vault and Key name in which private certificate is added in step1.

Add private certificate in Integration account
Note: Private certificate has to be stored in KeyVault and has to be referenced whereas Public certificate can be stored locally and uploaded.


4. Add Partner

Click on Partners and add Partner1 the sender and Partner2 the receiver(host), against Qualifier select AS2Identity and against Value Partner1 and Partner2 respectively.
Add partner1 in Integration accountAdd partner2 in Integration account
5. Add Agreement
                                
After adding partners next is to add Agreement, give it a Name , from Agreement Type drop-down select AS2 , Host Partner is Partner2 , Guest Partner is Partner1 and select identity respectively.
                         
                                 Add agreement between two partners in Integration Account

Agreement Receive setting - here we decide how the host partner would receive the AS2 message

i.Override Message Properties - If selected then the settings for validation and MDN which is defined below in Agreement is used else info from Message header is used.

ii.Message Should be signed - If selected then the sender partner has to sign the message with his private certificate and to validate it's public certificate has to be used at receiver s - Select the public certificate of the partner from the drop-down.

ii.Message Should be encrypted - If selected then the sender partner has to encrypt the message with receiver's public certificate and to decrypt receivers private certificate has to be used at receiver partner - Select the private certificate of the partner from the drop-down.

iii.Message Should be compressed - If selected all incoming messages must be compressed. Non-compressed messages are rejected.

iv.Disallow Message ID duplicates - It is to specify whether to allow messages with duplicate IDs. If you disallow duplicate IDs, select the number of days between checks. You can also choose whether to suspend duplicates.

v.MDN Text - Specifies the default message disposition notification (MDN) that you want sent to the message sender.

vi.Send MDN - Whether to send synchronous MDNs for received messages.

vii.Send signed MDN - Whether to send signed MDNs for received messages. If you require signing, from the MIC Algorithm list, select the algorithm to use for signing messages.

viii. Send asynchronous MDN - Whether to send MDNs asynchronously. If you select asynchronous MDNs, in the URL box, specify the URL for where to send the MDNs



Agreement Receive setting



Agreement  Send Setting - here we decide how the host partner would send the AS2 message

i. Enable message signing - Should all outgoing messages must be digitally signed. If you require signing, select these values:
- From the Signing Algorithm list, select the algorithm to use for signing messages.
- From the Certificate list, select an existing host partner private certificate for signing messages. 

ii.Enable message encryptionShould all outgoing messages must be encrypted. If you require encryption, select these values:
- From the Encryption Algorithm list, select the guest partner public certificate algorithm to use for encrypting messages.
- From the Certificate list, select an existing guest partner private certificate for encrypting outgoing messages.

iii. Enable message compressionShould all outgoing messages must be compressed.

iv.Unfold HTTP headers - Puts the HTTP content-type header onto a single line.

v.Transmit file name in MIME header - Whether to include the file name in the MIME header.

vi.Request MDN Should message disposition notifications (MDNs) to be received for all outgoing messages.

vii.Request signed MDNShould receive signed MDNs for all outgoing messages. If you require signing, from the MIC Algorithm list, select the algorithm to use for signing messages.

viii.Request asynchronous MDNShould receive MDNs asynchronously. If you select asynchronous MDNs, in the URL box, specify the URL for where to send the MDNs.

ix.Enable NRRShould require non-repudiation receipt (NRR). This communication attribute provides evidence that the data was received as addressed.

x.SHA2 Algorithm format - Specifies the MIC algorithm format to use for signing in the headers for the outgoing AS2 messages or MDN
Agreement  Send Setting

Create Logic App to receive AS2 Message and send MDN back

The first step is to Link Integration Account with Logic App, go to Settings --> Workflow Settings and select the Integration Account to 

linking integration Account


On designer Add Http request action to receive the AS2 message over the HTTP endpoint

AS2 receive logic app

Next is to Add Decode AS2 action here we first need to create a AS2 connection, an api connection to Integration Account (this is how the Encode/Decode AS2 connector gets access to the Partners/Agreement/Certificate which is required for them to perform expected operations i.e. signing/validation/decryption/encryption/compresssion)


Decode AS2 Message action

Then after provide input, i.e. the trigger body and header (header has http headers + AS2 headers which contains AS2 specific info )  to Decode AS2 Message

"body": "@triggerBody()",
"headers": "@triggerOutputs()['headers']",

Last step is to add Response Action to Send MDN (the same http connection will be used thus synchronous)  and input to it will be 
                     "body": "@base64ToString(body('Decode_AS2_message')?['OutgoingMdn']?['Content'])",
"headers": "@body('Decode_AS2_message')?['OutgoingMdn']?['OutboundHeaders']",
"statusCode": 200

AS2 messages are base64 encoded, thus to send MDN in string form @base64ToString function is used.

Note: OutgoingMdn is created by the AS2 Decoder, a task apart from validation and decryption. And AS2 decoder can use settings as per the agreement or it can use AS2 header of the incoming message .



Tuesday, August 27, 2019

The template validation failed: 'The action(s) 'xxx' referenced by 'inputs' in action 'xxx' are not defined in the template.'.

Currently working on a Logic App to accept AS2 message,decode it and do further processing and if any error encountered then notify the stakeholders by sending an email.

And for that I used email action and tried to pass on the error message as a body to the email, but I was not allowed to save the logic app for following reason


Action not defined in template



Error

The template validation failed: 'The action(s) 'Decode AS2 message' referenced by 'inputs' in action 'Failure_Alert' are not defined in the template.'.



Why it happened




I am using following expression to set the body of alert email which will execute only if Decode AS2 message action fails


actions('Decode AS2 message').outputs.body.errormessage


It looks absolutely fine on designer but still wasn't able to save and the reason is how Logic App renders in background i.e. at code level. Upon switching to code view found that Decode AS2 message gets converted to Decode_AS2_ message



What to do


I just did the way logic app needs to provide the action name in the expression as below and all worked fine.

actions('Decode_AS2_message').outputs.body.errormessage