Wednesday, January 22, 2020

Operation on target Copy data failed: Failure happened on 'Source' side

A simple Pipeline created to fetch data by querying the Rest Api and dumping the data received in blob storage, failed with following error:


error happened at source end





Error


Operation on target Copy data_From API failed: Failure happened on 'Source' side. ErrorCode=UserErrorHttpStatusCodeIndicatingFailure,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The HttpStatusCode 400 indicates failure. {"errors":{"detail":"Internal server error"}},Source=Microsoft.DataTransfer.ClientLibrary,'


Why it happened



As the error says, something went wrong at source side however, I have tested the Rest API using postman and was able to get the response back with data.
request using postman

And I have reflected same in Source setting of Copy Data activity while creating the pipeline i.e. query, headers – content type and authorization etc
Azure ADF Source Settings

Still getting the error, I rechecked again the keys,content type etc - no clue.

Took help from colleague and she pointed out the issue, it was with the way the query was posted. The rest api expects Graphql query, she asked me to check code view in postman and the issue was found.

postman code view


What to do


So now I added the query in the expected format against Request Body  i.e.

                        {"query": "{clients {id}}"}


correct source settings




That's it, all worked fine then after.









Related Post 


ServerLess360


Friday, January 17, 2020

Using Managed Identity in Logic Apps for Calling Active Directory Secured Function App

Introduction

In last posts we explored as to how AD authentication works -Understanding Azure Active Directory Authentication and Authorization using Oauth

But there was couple of challenges - creation and management of the AD app, it's credentials and credentials getting exposed/becoming part of the code. 

So we explored a better approach to call AD secured service using Managed Identity - Understanding Managed Identity and using it to Call Secured function App

In this post we will see how to use Managed Identity in Logic Apps for Calling Active Directory Secured Function App.


How to do it


1. Create Target Function app

Here we are going to call the same function app created in following post - Securing Function App with Azure Active Directory authentication

for that we would grab its url and AudienceID (client ID of AD app created to secure function App)
AD App client ID



2. Create Logic App and Enable Identity on it


Create Logic App instance and click on Identity and then switch on the Status. That's it, an identity for your logic app is created - behind the scene, an AD app gets created in AD with the same name as that of the logic app .

So whenever this logic app has to call any AD secured Azure service, the above created Identity would be leveraged.

Enable Identity on Logic Apps

3. Design Logic App

Create a simple http triggered logic app which will get triggered upon receiving http request and then calls the secured function app and sends back response . In response it will add whatever received from secured function app.

logic app to call secured function app

In second step note that we are using HTTP action rather than using Azure Functions action and it is because Azure Functions action does not support adding authentication details, whereas Http connector is flexible with it.
Logic App Managed Identity

After adding http action, Select POST as method and provide the url of secured function app. In Authentication select  Managed Identity and Click on Add new parameter
add new parameter

And  provide the Audience ID we grabbed in step 1 as shown below 
add audience ID


Now add http response action and against body - select the body returned from above step



That's it logic app is ready to use.


Testing


The target function expects input in following form 

{"name":"Input"}

and it returns result as "Reply from target app,+input it received"
test request


Use any rest client and make call to logic app and pass on input in expected format as above and below is the result
test response


How it works

If you noted, we haven't provided any credentials of Logic app and just provided the url of the function to call and Audience ID and still Logic app was able to get response back -- it is because under the hood logic app runtime uses the Identity created in step 2 to authenticate itself with Active directory and gets the access token to make a call. 

Thus with help of Managed Identity, overhead of managing AD app and credentials is removed.




Wednesday, January 15, 2020

Understanding Managed Identity and using it to Call Secured function App

What is Managed Service Identity / Managed Identity

In last post we saw how to secure a Function App with Active directory and how to make call to it from another function App.

And for doing it from function we used libraries provided by Microsoft i.e., Microsoft.IdentityModel.Clients.ActiveDirectory;

So we needed to authenticate first with AD and then request for access token. For that we create an AD app, create secret/add certificate , manage secrets expiry and storing the info somewhere so as to use in our code

With this approach there are two problem areas
1. We have to create/manage the AD app and the  certificates/secrets associated with it. 
2. We have to use credentials in our code, although from app settings (they are visible to all)  or keyvault - but to access those you need credential to authenticate to it.

To avoid this, we can use Managed Service identity (MSI)/Managed Identity feature, and the Azure will do this for us automatically. To use MI/MSI, turn on Identity of the respective Azure Service(here function app) from the Azure Portal.


Note: MSI/MI feature is not available for all services. See Services that support managed identities for Azure resources

What happens when Identity is switched on ? -- When you switch on Identity of any Azure service, Azure itself creates an AD app with same name of the service thus creating an identity in Azure Active Directory.

And you don't have to worry about Tenant ID(Directory ID),
Client ID and Client Secret - as it is taken care(managed) by Azure thus the name Managed Identities.

Using Managed Identity we can authenticate to any service that supports Azure AD authentication without having credentials in your code. But we would need access token for the target app in order to access it. 

Microsoft has provided a library Microsoft.Azure.Services.AppAuthentication  which has azureServiceTokenProvider.GetAccessTokenAsync method which can be used to get access token by just providing Audience ID.

In some cases, we need to explicitly provide access to an identity apart from token e.g., in Key Vault we have to add identity in it's access policy. The access mechanism can be different for each Azure Service.

I personally feel, the access mechanism should be unified across all Azure service.

How to do it


1. Create Function App and Enable Identity (Caller AD App)


Go to Portal and create an instance on Function App and once created select it and go to Platform features and select Identity


select Identity on function App


Upon selecting Identity you will be presented with two option System assigned and User assigned - Select System assigned.

Identity On
 Make Status as On and save

identityOn Confirmation


What happens when we switch on Identity?

As mentioned earlier, an AD app gets created with same name as that of the azure service in Active directory, you can check it in Enterprise Application
check AD app in Enterprise application


Difference between System assigned and user assigned Identity -

The lifecycle of a system-assigned identity is directly tied to the Azure service instance that it's enabled on (function app here). So if this function app is deleted, Azure automatically cleans up the credentials and the identity in Azure AD

user-assigned managed identity is created as a standalone Azure resource i.e.not tied to any service.So it is same as explicitly creating AD app and can be shared by any number of service.


2. Create Function



In Visual Studio, create a project using Azure functions Template and add a http triggered function .

Now we create a function which will use the above identity to call the AD secured function app by getting access token from Active Directory and pass it in header along with request.

However here no details other than AudienceID and TargetURL is required, this we store in App settings of function app (on local machine it is to be kept in local.settings.json file). As rest is taken care by azureServiceTokenProvider.GetAccessTokenAsync method and to use it we need to install   Microsoft.Azure.Services.AppAuthentication nuget package.Microsoft has provided this library to work with Managed Identity.

Following code is used to get the access token

var azureServiceTokenProvider = new AzureServiceTokenProvider();
string accessToken= awaitazureServiceTokenProvider.GetAccessTokenAsync(Environment.GetEnvironmentVariable("AudienceID"));

And this access token is passed with request in header  as AuthenticationHeaderValue

httpClient.DefaultRequestHeaders.Authorization = newSystem.Net.Http.Headers.AuthenticationHeaderValue("Bearer", accessToken);
var content = new StringContent(body, System.Text.Encoding.UTF8, "application/json");
response = httpClient.PostAsync(Environment.GetEnvironmentVariable("TargetURL"),   content).Result;

And below is the full code

using System;
using System.IO;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.Logging;
using Newtonsoft.Json;
using System.Net.Http;
using Microsoft.Azure.Services.AppAuthentication;

namespace DevT2
{
    public static class CallerFunctionMSI
    {
        [FunctionName("CallerFunctionMSI")]
        public static async Task<IActionResult> Run(
            [HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = null)] HttpRequest req,
            ILogger log)
        {
            log.LogInformation("C# HTTP trigger function processed a request.");

            string requestBody = await new StreamReader(req.Body).ReadToEndAsync();

            HttpResponseMessage Targetresp = await PostMessage(requestBody);
            return Targetresp != null
                ? (ActionResult)new OkObjectResult($"Hello, {Targetresp.Content.ReadAsStringAsync().Result}")
                : new BadRequestObjectResult("Please pass a name on the query string or in the request body");
        }
        public static async Task<HttpResponseMessage> PostMessage(string body)
        {
            HttpResponseMessage response;
            var azureServiceTokenProvider = new AzureServiceTokenProvider();
            string accessToken = await azureServiceTokenProvider.GetAccessTokenAsync(Environment.GetEnvironmentVariable("AudienceID"));

            using (var httpClient = new HttpClient())
            {
                httpClient.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Bearer", accessToken);
                var content = new StringContent(body, System.Text.Encoding.UTF8, "application/json");
                response = httpClient.PostAsync(Environment.GetEnvironmentVariable("TargetURL"), content).Result;
            }
            return response;
        }

    }
}


Build it and deploy it to Function App created in step1. And do not forget to add the TargetUrl and AudienceID in app settings.





Testing


The target function expects input in following form 

{"name":"Input"}

and it returns result as "Hello,Reply from target app,+input it received"
testing evidence


Grab the url of caller function, use any rest client and make call to caller function app and pass on input in expected format as above and check the response.


How it works


GetAccessTokenAsync method automatically manages the authentication part before raising an access token request to authority.And to get the details required it makes  a call internally to Managed Identity rest api. 

Thus this process eliminates the overhead of managing the app, it's credentials and helps in avoiding the credentials in code.



Monday, January 13, 2020

Calling Active Directory Secured Function App from Logic Apps

Introduction

In last post we saw how to secure a Function App with Active directory and how to make call to it from another function App.

And for doing it from function we used libraries provided by Microsoft i.e., Microsoft.IdentityModel.Clients.ActiveDirectory;

So how do we do this in logic app? How do we add the above library to logic app and where do we write code? – No code required as Logic app already supports Active Directory OAuth out of box . We just need to provide the  following info

Tenant ID(Directory ID) -- It is the authority which grants the token
AudienceID - For whom the token will be created i.e.Target AD App's client ID
Client ID and Client Secret - It is caller AD app's details, this proves client identity

Thus other steps remains same, in order to call AD secured app you(client/calling app) need to get access token from AD and for doing that you need to prove your identity with AD. And to create identity we need to register an App with AD. And then you use this identity to request access token.


How to do it


So first we need to get access token, and to request token we need

1. Register an Application with Active directory (Caller AD App)


caller AD app

As this app will be used to get token, we need to make note of application(client) ID and also need to create a secret
create client secret


 Remember to copy and save the secret as it won't be visible later
client secret

2. Create Logic App

Create a simple http triggered logic app which will get triggered upon receiving http request and then calls the secured function app and sends back response . In response it will add whatever received from secured function app.

logic app to call secured function app

In second step note that we are using HTTP action rather than using Azure Functions action and it is because Azure Functions action does not support adding authentication details, whereas Http action is flexible with it.
select Active Directory OAuth

After adding http action, Select POST as method and provide the url of secured function app. In Authentication select Active Directory OAuth and provide the other details as shown below 

Here I personally think, having an application settings for logic Apps would be good to have feature. However we can use Key Vault here to store the info and fetch values at runtime but for the sake of post, have done it here itself.
oauth config

Now add http response action and against body - select the body returned from above step



That's it logic app is ready to use.


Testing


The target function expects input in following form 

{"name":"Input"}

and it returns result as "Reply from target app,+input it received"


Testing request
Use any rest client and make call to logic app and pass on input in expected format as above and below is the result
Testing result


How it works


When calling from function app we had to explicitly call the endpoint (https://login.microsoftonline.com/) alongwith the details like TenantId, AudienceID, ClienID and Secret using the provided library, however in case of logic app you just need to configure the details and rest is taken care by http connector.