A few months back, I wrote a guide for installing and locking down OpenFaaS in a Docker Swarm running on Google Cloud Platform virtual machines. Today I want to share a step-by-step guide that shows how to install OpenFaaS on a new Azure Kubernetes Service (AKS) cluster using an Nginx ingress controller to lock it down with basic authentication and free Let's Encrypt TLS certificates.

Before we begin

If you just want to do a quick deployment of OpenFaaS on AKS, there's a guide in the official AKS documentation here, however it does not show how to implement TLS encryption or authentication.

All the Azure configuration I'll show today is done via the command line, so if you don't already have the Azure CLI installed on your local system, install it from here. You can do it through the Azure portal, but it's much faster to do with the CLI.

You will also need Git command-line tools installed so you can pull down the latest version of OpenFaaS from its repository.

Finally, in order to secure your OpenFaaS installation with TLS, you will need a domain and access to your DNS provider so you can point a hostname to your cluster's public IP address

Ready? Let's get started.

Basic Azure configuration

1. From the command line, log in to Azure using az login. Follow the prompts to complete your authentication.
2. If you don't have an existing resource group you want to use for your AKS cluster, create a new one with az group create -l REGIONNAME -n RESOURCEGROUP. Replace REGIONNAME and RESOURCEGROUP with appropriate values, but make sure you use a region where AKS is currently available.
3. Create a new AKS cluster with az aks create -g RESOURCEGROUP -n CLUSTERNAME --generate-ssh-keys. The RESOURCEGROUP value is the same as before, and CLUSTERNAME is whatever you want it to be called. Note that the default virtual machine size for your cluster is Standard_DS1_v2. You can change this by setting the --node-vm-size, and I am personally using burstable Standard_B2s VMs for my AKS cluster.
4. Once the AKS cluster creation completes, use this command to get the credentials you need to manage the cluster with the Kubernetes CLI az aks get-credentials --resource-group RESOURCEGROUP --name CLUSTERNAME.
5. Install the Kubernetes CLI (kubectl) with az aks install-cli.
6. Get the name of the node resource group that was created for your AKS cluster with this command az resource show --resource-group RESOURCEGROUP --name CLUSTERNAME --resource-type Microsoft.ContainerService/managedClusters --query properties.nodeResourceGroup -o tsv. You should get an output that looks like MC_resourcegroup_clustername_regionname. You will use this return value in the next step.
7. Create a public IP address in the node resource groupe with this command az network public-ip create --resource-group MC_RESOURCEGROUP --name IPADDRESSNAME --allocation-method static. You will get a JSON response that contains a "publicIp" object. Copy its "ipAddress" value and save it for later.
   Note: you might be tempted to create a DNS name label for this IP address so you can avoid using a custom domain name, but *.cloudapp.azure.com host names don't work with Let's Encrypt.
8. Go to your DNS provider and register a new A record for your a hostname that points to the external IP you reserved in the previous step (mine is akskube.alexanderdevelopment.net). This will be the hostname you use to access OpenFaaS. You may also need to create a new CAA record to explicitly allow Let's Encrypt to issue certificates for your domain.

Basic cluster configuration

Once the basic Azure configuration work is done, it's time to configure the AKS cluster.

1. If you don't already have the Helm client installed on your local system, install it by following the directions here. I am using a Windows dev workstation, so I installed Chocolatey and then installed the Helm client with choco install kubernetes-helm.
2. Install Helm components on your AKS cluster with helm init --upgrade --service-account default.
3. Install the Nginx ingress controller on your AKS cluster with helm install stable/nginx-ingress --namespace kube-system --set rbac.create=false --set rbac.createRole=false --set rbac.createClusterRole=false --set controller.service.loadBalancerIP=STATICIPADDRESS. Replace STATICIPADDRESS with the public IP address you created previously.
4. Install cert-manager to request and manage your TLS certificates helm install --name cert-manager --namespace kube-system stable/cert-manager --set rbac.create=false.

Installing OpenFaas

It's relatively easy to install OpenFaaS on your AKS cluster using Helm, and a detailed readme is available here. Basically you need to download the faas-netes Git repository to your local system, create a couple of namespaces on the AKS cluser and use the Helm chart in the repo you downloaded. Here's how I set it up on my AKS cluster.

kubectl create ns openfaas
kubectl create ns openfaas-fn

git clone https://github.com/openfaas/faas-netes
cd faas-netes

helm install --namespace openfaas -n openfaas --set functionNamespace=openfaas-fn --set ingress.enabled=true --set rbac=false chart/openfaas/

Once OpenFaaS is installed, you need to create ingress resources to make it available externally.

Creating the ingress resources

Before creating your ingress resources, you need to create certificate issuer resources to get TLS certificates. Here's the YAML for a Let's Encrypt staging issuer:

apiVersion: certmanager.k8s.io/v1alpha1
kind: Issuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    # The ACME server URL
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: EMAILADDRESS
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-staging
    # Enable the HTTP-01 challenge provider
    http01: {}

Copy it, replace EMAILADDRESS with your email address and save it as faas-staging-issuer.yml. Then run kubectl apply -f faas-staging-issuer.yml -n openfaas.

Here's a corresponding production issuer:

apiVersion: certmanager.k8s.io/v1alpha1
kind: Issuer
metadata:
  name: letsencrypt-production
spec:
  acme:
    # The ACME server URL
    server: https://acme-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: EMAILADDRESS
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-production
    # Enable the HTTP-01 challenge provider
    http01: {}

Copy it, replace EMAILADDRESS with your email address and save it as faas-production-issuer.yml. Then run kubectl apply -f faas-production-issuer.yml -n openfaas.

Next you need to create a password file to implement basic authentication. If you are working on a system with apache2-utils installed, you can just use the htpasswd command. Otherwise, you can use a tool like http://aspirine.org/htpasswd_en.html to generate your htpasswd content. Once you have your htpasswd content generated, save it in a file named "auth" and run the following command kubectl create secret generic basic-auth --from-file=auth -n openfaas

Now you can use the following YAML to create an ingress resource that exposes your OpenFaaS instance:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: faas-ingress
  annotations:
    nginx.ingress.kubernetes.io/auth-realm: "Authentication Required"
    nginx.ingress.kubernetes.io/auth-secret: basic-auth
    nginx.ingress.kubernetes.io/auth-type: basic
    kubernetes.io/tls-acme: "true"
    certmanager.k8s.io/issuer: letsencrypt-staging
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  tls:
  - hosts:
    - HOSTNAME
    secretName: faas-letsencrypt-staging
  rules:
  - host: HOSTNAME
    http:
      paths:
      - path: /faas-admin
        backend:
          serviceName: gateway
          servicePort: 8080

Copy it, replace both instances of HOSTNAME with the hostname you created earlier and save it as faas-ingress.yml. Deploy it to your cluser with this command kubectl apply -f faas-ingress.yml -n openfaas.

As the ingress starts up, it will request a staging certificate from Let's Encrypt, and then it will start listening for requests. It may take a few minutes, so now might be a good time to take a short break. Once everything is complete, you will be able to access your OpenFaaS UI from https://HOSTNAME/faas-admin/ui/, and once you deploy functions, they will be available at https://HOSTNAME/faas-admin/functions/FUNCTIONNAME. You should get a browser warning about the certificate because it's using a Let's Encrypt production certificate, but that's OK for now. You should also be prompted for basic authentication credentials, which will be the username and password you created earlier.

If everything looks good, you can switch over to using a production TLS certificate. Take the faas-ingress YAML and replace the "letsencrypt-staging" in the secretname and certmanager.k8s.io issuer values with "letsencrypt-production" instead. Save it and deploy the update with kubectl apply -f faas-ingress.yml -n openfaas. Like before, the ingress will take a few minutes to restart and request a production TLS certificate from Let's Encrypt. Once that's done, you can access the your OpenFaaS UI via the same URL, but now you should not get a warning about an invalid certificate.

At this point you have a locked-down OpenFaaS installation, but you might not want to use basic authentication to restrict access to your OpenFaaS functions. If that's the case you can create another ingress resource that exposes them outside the "/faas-admin" path. Here's the YAML for that resource:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: faas-function-ingress
  annotations:
    kubernetes.io/tls-acme: "true"
    certmanager.k8s.io/issuer: letsencrypt-production
    nginx.ingress.kubernetes.io/rewrite-target: /functions
spec:
  tls:
  - hosts:
    - HOSTNAME
    secretName: faas-letsencrypt-production
  rules:
  - host: HOSTNAME
    http:
      paths:
      - path: /functions
        backend:
          serviceName: gateway
          servicePort: 8080

Copy it, replace both instances of HOSTNAME with your DNS hostname from earlier and save it as faas-function-ingress.yml. Deploy it to your cluser with this command kubectl apply -f faas-function-ingress.yml -n openfaas.

Once the ingress starts up and applies the TLS certificate, you will be able to access your functions at https://HOSTNAME/functions/FUNCTIONNAME without authenticating.

Wrapping up

A few closing thoughts:

1. I am still extremely new to AKS and Kubernetes, and I've tried to simplify this guide as much as possible for other newbies. In figuring out how to set this up, I relied heavily on the official AKS docs, and I encourage you to take a look at them if you want to dig in deeper.
2. This configuration does not expose the OpenFaaS Prometheus montitoring service. If you want to set that up, you will need to create a different DNS entry (either an A record or CNAME record) and create another ingress resource in the openfaas namespace for that host name that points to the "prometheus" service on service port 9090.
3. The Nginx ingress controller configuration I showed here is extremely simple. If you want to use a more sophisticated configurations to enable advanced features like rate limiting, for example, take a look at https://github.com/kubernetes/charts/tree/master/stable/nginx-ingress#configuration and https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/configmap.md.

This is the final post in my series about building a service relay for Dynamics 365 CE with RabbitMQ and Python. In my previous post in this series, I showed the Python code to make the service relay work. In today's post, I will show how you can use Azure Functions to make a consumer service proxy using C# so client applications don't have to access to your RabbitMQ broker directly, and I will also discuss some general thoughts on security and scalability for this service relay architecture.

Although this simple service relay allows external consumers to get data from Dynamics 365 CE without needing to connect directly, the examples I've shown so far require that they can connect to a RabbitMQ broker. This may be problematic for a variety of reasons, so you would probably want external consumers to connect to a web service proxy that would write requests to and read responses from the RabbitMQ broker.

Building a service proxy function

You can build an Azure Functions service proxy with Python, but I don't recommend it for three reasons:

1. Azure Functions Python support is still considered experimental.
2. Python scripts that use external libraries can run exceedingly slow.
3. Getting the environment set up is a bit of a hassle.

On the other hand, building a service proxy function with C# was so much easier, and it performed much better than a comparable Python function (~.5 seconds for C# compared to 5+ seconds for Python).

Here are the steps I took to build my C# service proxy function:

1. Create a C# HTTP trigger function.
2. Create and upload a project.json file with a dependency on the RabbitMQ client (see below).
3. Take the "RpcClient" class from the RabbitMQ .Net RPC tutorial and call it from within my function.

Here's my project.json file:

{
  "frameworks": {
    "net46":{
      "dependencies": {
        "RabbitMQ.Client": "5.0.1"
      }
    }
   }
}

And here's my run.csx file:

using System.Net;
using System;
using System.Collections.Concurrent;
using System.Text;
using RabbitMQ.Client;
using RabbitMQ.Client.Events;

public static async Task<HttpResponseMessage> Run(HttpRequestMessage req, TraceWriter log)
{
    log.Info("Processing request");

    // parse query parameter
    string query = req.GetQueryNameValuePairs()
        .FirstOrDefault(q => string.Compare(q.Key, "query", true) == 0)
        .Value;

    // Get request body
    dynamic data = await req.Content.ReadAsAsync<object>();

    // Set name to query string or body data
    query = query ?? data?.query;

    var rpcClient = new RpcClient();
    
    log.Info(string.Format(" [.] query start time {0}", DateTime.Now.ToString("MM/dd/yyyy hh:mm:ss.fff tt")));
    var response = rpcClient.Call(query);

    log.Info(string.Format(" [.] query end time {0}", DateTime.Now.ToString("MM/dd/yyyy hh:mm:ss.fff tt")));
    rpcClient.Close();

    return req.CreateResponse(HttpStatusCode.OK, response);
}

public class RpcClient
{
    private readonly IConnection connection;
    private readonly IModel channel;
    private readonly string replyQueueName;
    private readonly EventingBasicConsumer consumer;
    private readonly BlockingCollection<string> respQueue = new BlockingCollection<string>();
    private readonly IBasicProperties props;

    public RpcClient()
    {
        var factory = new ConnectionFactory() { HostName = "RABBITHOST", UserName="RABBITUSER", Password="RABBITUSERPASS"  };

        connection = factory.CreateConnection();
        channel = connection.CreateModel();
        replyQueueName = channel.QueueDeclare().QueueName;
        consumer = new EventingBasicConsumer(channel);

        props = channel.CreateBasicProperties();
        var correlationId = Guid.NewGuid().ToString();
        props.CorrelationId = correlationId;
        props.ReplyTo = replyQueueName;

        consumer.Received += (model, ea) =>
        {
            var body = ea.Body;
            var response = Encoding.UTF8.GetString(body);
            if (ea.BasicProperties.CorrelationId == correlationId)
            {
                respQueue.Add(response);
            }
        };
    }

    public string Call(string message)
    {
        var messageBytes = Encoding.UTF8.GetBytes(message);
        channel.BasicPublish(
            exchange: "",
            routingKey: "rpc_queue",
            basicProperties: props,
            body: messageBytes);

        channel.BasicConsume(
            consumer: consumer,
            queue: replyQueueName,
            autoAck: true);

        return respQueue.Take(); ;
    }

    public void Close()
    {
        connection.Close();
    }
}

Here's a screenshot showing me calling the C# function with Postman.

Because I did actually build a Python function, I will go ahead and share how I did it if you're interested. Here are the steps I took:

1. Create a Python HTTP trigger function.
2. Install Python 3.6 via site extensions (see steps 2.1-2.4 here).
3. Install the necessary libraries using pip via KUDU.

Here's the Python function code:

import os
import sys
import json
import pika
import uuid
import datetime

class CrmRpcClient(object):
    def __init__(self):
        #RabbitMQ connection details
        self.rabbituser = 'RABBITUSERNAME'
        self.rabbitpass = 'RABBITUSERPASS'
        self.rabbithost = 'RABBITHOST' 
        self.rabbitport = 5672
        self.rabbitqueue = 'rpc_queue'
        rabbitcredentials = pika.PlainCredentials(self.rabbituser, self.rabbitpass)
        rabbitparameters = pika.ConnectionParameters(host=self.rabbithost,
                                    port=self.rabbitport,
                                    virtual_host='/',
                                    credentials=rabbitcredentials)

        self.rabbitconn = pika.BlockingConnection(rabbitparameters)

        self.channel = self.rabbitconn.channel()

        #create an anonymous exclusive callback queue
        result = self.channel.queue_declare(exclusive=True)
        self.callback_queue = result.method.queue

        self.channel.basic_consume(self.on_response, no_ack=True,
                                   queue=self.callback_queue)

    #callback method for when a response is received - note the check for correlation id
    def on_response(self, ch, method, props, body):
        if self.corr_id == props.correlation_id:
            self.response = body

    #method to make the initial request
    def call(self, n):
        self.response = None
        #generate a new correlation id
        self.corr_id = str(uuid.uuid4())

        #publish the message to the rpc_queue - note the reply_to property is set to the callback queue from above
        self.channel.basic_publish(exchange='',
                                   routing_key=self.rabbitqueue,
                                   properties=pika.BasicProperties(
                                         reply_to = self.callback_queue,
                                         correlation_id = self.corr_id,
                                         ),
                                   body=n)
        while self.response is None:
            self.rabbitconn.process_data_events()
        return self.response

print(" [.] query start time %r" % str(datetime.datetime.now()))
#instantiate an rpc client
crm_rpc = CrmRpcClient()

postreqdata = json.loads(open(os.environ['req']).read())
query = postreqdata['query']

crm_rpc = CrmRpcClient()
print(" [.] query start time %r" % str(datetime.datetime.now()))
queryresponse = crm_rpc.call(query)
print(" [.] query end time %r" % str(datetime.datetime.now()))
response = open(os.environ['res'], 'w')
response.write(queryresponse.decode())
response.close()

Here's a screenshot showing me calling the Python function with Postman.

Note the difference in time between the two functions - 5.62 seconds for Python and .46 seconds for C#!

Security and scalability

If you decide to use this approach in production, I'd suggest you carefully consider both security and scalability. Obviously the overall solution will only be as secure as your RabbitMQ broker and communications between the broker and its clients, so you'll want to look at best practices for access control and securing the communications with TLS. Here are some links for further reading on those subjects:

• TLS - https://www.rabbitmq.com/ssl.html
• Access control - https://www.rabbitmq.com/access-control.html

As for scalability, the approach I've shown creates a separate response queue for each consumer, but it can have problems scaling, especially if you are using a RabbitMQ cluster. You may want to look at the "direct reply-to" approach instead. For an interesting real-world overview of using direct reply-to, take a look at this blog post..

Wrapping up

I hope you've enjoyed this series and that it has given you some ideas about how to implement service relays in your Dynamics 365 CE projects. As I worked through the examples, I certainly learned a few new things, especially when I created my Python service proxy in Azure Functions.

Here are links to all the previous posts in this series.

1. Part 1 - Series introduction
2. Part 2 - Solution prerequisites
3. Part 3 - Python code for the consumer and listener processes

What do you think about this approach? Is it something you think you'd use in production? Let us know in the comments!

My Dynamics 365 Configuration Data Mover utility allows you to run synchronization jobs from an interactive GUI tool or the command line, but the actual data synchronization logic is contained in a separate AlexanderDevelopment.ConfigDataMover.Lib.dll file that can be included in other applications. In today's post I will show how you can set up an Azure Function to execute a Configuration Data Mover job file to sync data between two Dynamics 365 organizations.

Setting up your Azure Function

First create a new Azure Function. I created an HTTP trigger function with the default "function" authorization level so that I could post parameters to it, but if you could modify the code sample later in this post to use it with a different type of trigger.

Next open your function's Kudu console so you can create a "bin" directory to upload the AlexanderDevelopment.ConfigDataMover.Lib.dll file.

Navigate to the correct directory and create a "bin" directory inside it.

After that, go back to the main function editor interface to see the newly created "bin" directory in the "view files" area.

Look in the local directory where you have installed the Configuration Data Mover and find the "AlexanderDevelopment.ConfigDataMover.Lib.dll" file. Upload it to the "bin" directory you just created.

Navigate back to the main function directory and create a project.json file to pull in dependencies via NuGet.

Here is the content of the file so you can copy and paste:

{
  "frameworks": {
    "net46":{
      "dependencies": {
        "Microsoft.CrmSdk.CoreAssemblies": "8.2.0",
        "Microsoft.CrmSdk.XrmTooling.CoreAssembly": "8.2.0",
        "log4net": "2.0.8"
      }
    }
  }
}

Finally, you need to update the function code in the run.csx file.

Here's the code you can use:

#r "NewtonSoft.Json"
#r "AlexanderDevelopment.ConfigDataMover.Lib.dll"
using System.Net;
using System.Collections.Generic;
using AlexanderDevelopment.ConfigDataMover.Lib;
using System.Xml;

static string _sourceString = null;
static string _targetString = null;
static bool _mapBaseBu = false;
static bool _mapBaseCurrency = false;
static List<GuidMapping> _guidMappings = new List<GuidMapping>();
static List<JobStep> _jobSteps = new List<JobStep>();

public static async Task<HttpResponseMessage> Run(HttpRequestMessage req, TraceWriter log)
{
    log.Info("C# HTTP trigger function processed a request.");

    // parse query parameters
    string jobdata = req.GetQueryNameValuePairs()
        .FirstOrDefault(q => string.Compare(q.Key, "jobdata", true) == 0)
        .Value;
    string sourceparam = req.GetQueryNameValuePairs()
        .FirstOrDefault(q => string.Compare(q.Key, "source", true) == 0)
        .Value;
    string targetparam = req.GetQueryNameValuePairs()
        .FirstOrDefault(q => string.Compare(q.Key, "target", true) == 0)
        .Value;

    if(jobdata == null)
    {
        return req.CreateResponse(HttpStatusCode.BadRequest, "Please post jobdata");
    }
    else
    {
        ParseConfig(jobdata);
		
		//use source and target values if provided in the POST
		if(!string.IsNullOrEmpty(targetparam))
		{
			_targetString = targetparam;
		}
		if(!string.IsNullOrEmpty(sourceparam))
		{
			_sourceString = sourceparam;
		}
		
		//do some basic validations
		if (string.IsNullOrEmpty(_sourceString))
		{
			return req.CreateResponse(HttpStatusCode.BadRequest,"no source connection specified - exiting");
		}
		if (string.IsNullOrEmpty(_targetString))
		{
			return req.CreateResponse(HttpStatusCode.BadRequest,"no target connection specified - exiting");
		}
		if (!(_jobSteps.Count > 0))
		{
			return req.CreateResponse(HttpStatusCode.BadRequest,"no steps in job - exiting");
		}

		Importer importer = new Importer();
		importer.GuidMappings = _guidMappings;
		importer.JobSteps = _jobSteps;
		importer.SourceString = _sourceString; 
		importer.TargetString = _targetString; 
		importer.MapBaseBu = _mapBaseBu;
		importer.MapBaseCurrency = _mapBaseCurrency;
		importer.Process();

		int errorCount = importer.ErrorCount;

		importer = null;
		
		//show a message to the user
		if (errorCount == 0)
		{
			return req.CreateResponse(HttpStatusCode.OK,"Job finished with no errors.");
		}
		else
		{
			return req.CreateResponse(HttpStatusCode.BadRequest,"Job finished with errors.");
		}
    }
}

public static void ParseConfig(string jobdata)
{
    XmlDocument xml = new XmlDocument();
	xml.LoadXml(jobdata);
	_jobSteps.Clear();
	_guidMappings.Clear();

	XmlNodeList stepList = xml.GetElementsByTagName("Step");
	foreach (XmlNode xn in stepList)
	{
		JobStep step = new JobStep();
		step.StepName = xn.SelectSingleNode("Name").InnerText;
		step.StepFetch = xn.SelectSingleNode("Fetch").InnerText;
		step.UpdateOnly = false;
		if(xn.Attributes["updateOnly"]!=null)
			step.UpdateOnly = Convert.ToBoolean(xn.Attributes["updateOnly"].Value);

		step.CreateOnly = false;
		if (xn.Attributes["createOnly"] != null)
			step.CreateOnly = Convert.ToBoolean(xn.Attributes["createOnly"].Value);

		_jobSteps.Add(step);
	}

	XmlNodeList configData = xml.GetElementsByTagName("JobConfig");
	_mapBaseBu = Convert.ToBoolean(configData[0].Attributes["mapBuGuid"].Value);
	_mapBaseCurrency = Convert.ToBoolean(configData[0].Attributes["mapCurrencyGuid"].Value);

	XmlNodeList mappingList = xml.GetElementsByTagName("GuidMapping");
	foreach (XmlNode xn in mappingList)
	{
		Guid sourceGuid = new Guid(xn.Attributes["source"].Value);
		Guid targetGuid = new Guid(xn.Attributes["target"].Value);
		_guidMappings.Add(new GuidMapping { sourceId = sourceGuid, targetId = targetGuid });
	}
	XmlNodeList connectionNodes = xml.GetElementsByTagName("ConnectionDetails");
	if (connectionNodes.Count > 0)
	{
		_sourceString = connectionNodes[0].Attributes["source"].Value;
		_targetString = connectionNodes[0].Attributes["target"].Value;
	}
}

Executing your Azure Function

To execute the function, you'll need to copy the function key so that you can send it as a POST parameter.

Once you have that, you can build a POST request. Here's what it looks like in Postman:

The only required parameters are "code" and "jobdata." "Code" is the function key you copied earlier. "Jobdata" is the content of a Configuration Data Mover job XML file. If you do not have the source/target connection details saved in your "jobdata" XML, you will need to also include "source" and "target" connection string parameters in your request. If you do have connection details saved in the "jobdata" XML and you also supply "source" and "target" connection string parameters, they will be used instead of what is in the "jobdata" XML.

Assuming everything goes as expected, you should get a "job finished with no errors" response like this:

If you still have the function editor interface open, you'll see new log entries like this:

Caveats

1. I originally built the Configuration Data Mover to use Apache log4net for detailed logging. The AlexanderDevelopment.ConfigDataMover.Lib library here is still trying to use log4net, but those log entries aren't going anywhere. I'll probably look into modifying the logging approach in the library so that it works better in situations where the library used like this. The upshot is that what I've outlined here will give you only a basic success/failure message.
2. Currently the AlexanderDevelopment.ConfigDataMover.Lib library can either connect to a live CRM instance or read configuration data from a JSON file that it can open. I have a future enhancement in mind that would allow for configuration data to be passed to the library to make it more flexible.

The only Dynamics 365 actions that Microsoft Flow offers right now are "create a new record" and "list records," but with just a bit of additional effort it's possible to access all the capabilities of the Web API. Today I will show how to create a Microsoft Flow that queries a set of accounts and executes a workflow for each one.

My Flow comprises five separate actions, which you can see in the screenshot below:

1. The request trigger - In this case I am triggering my Flow by making an empty HTTP POST request, but you could use any supported trigger.
2. A "getauthtoken" HTTP request - Flow knows how to query a list of CRM records without requiring any additional authentication, but in order to work with the Web API directly, it's necessary to get an OAuth2 token from Active Directory like I described in my recent "Dynamics 365 and Node.js integration using the Web API" post.
3. A Dynamics 365 list records action - This action queries Dynamics 365 for the accounts on which the workflow will be executed.
4. An "apply to each" loop - This loops through the returned accounts from the previous step and executes the workflow for each one.
5. A response step - This action sends back a success message regardless of what happens in the previous steps. Ideally I would handle errors appropriately here.

Let's take a closer look at each action.

Triggering the Flow

As I mentioned above, my Flow is triggered from an empty POST request, but you could use a different trigger.

Getting the OAuth token

In order to get an OAuth token, my Flow uses an HTTP request action to post the following parameters to the AD token endpoint:

1. client_id - registered client id from AD
2. resource - root path to your Dynamics 365 org (https://XXXXX.crm.dynamics.com, for example)
3. username - Dynamics 365 username
4. password - Dynamics 365 user password
5. grant_type = "password"

The AD token endpoint expects data to be sent as application/x-www-form-urlencoded content, so you have to set the request header and manually build the POST string. The POST string should look like this:

client_id=REGISTERED_CLIENT_ID&resource=https://CRMORG.crm.dynamics.com&username=USERNAME&password=PASSWORD&grant_type=password

Interestingly enough, you do not have to URI encode your parameters to escape special characters in general, though I suspect it would be necessary if your password contains "&" or "="
characters.

Here's what my configured action looks like:

(It would make more sense store all these string configuration settings as variables using Flow "compose" actions, but I chose not to do that for this post to make the overall sequence easier to follow.)

Retrieving records

The next step in my Flow queries Dynamics 365 for accounts using the native "list records" functionality. This action does not require the OAuth token retrieved in the previous step.

Looping through retrieved records

Next my Flow use an "apply to each" step to loop through the retrieved accounts and then execute the workflow.

Calling the workflow requires POSTing a JSON object containing the accountid value to the Dynamics 365 Web API endpoint (note the workflow id is included in the endpoint here), and the Web API expects the OAuth token from earlier to be included in the request headers. Flow does not automatically know how to parse the JSON response from the OAuth token retrieval request, so it won't show up in the dynamic content menu, but you can access it using this format @{body('HTTP REQUEST ACTION NAME').access_token}.

Here's my request headers object:

{
"Authorization": "Bearer @{body('getauthtoken').access_token}",
"OData-MaxVersion": "4.0",
"OData-Version": "4.0",
"Accept": "application/json",
"Content-Type": "application/json; charset=utf-8"
}

Once you save the Flow, the UI will replace the @{body('HTTP REQUEST ACTION NAME').access_token} with something that looks like a dynamic content placeholder as you can see in the screenshot below.

Sending a response

Finally my Flow uses a response action to send a message back to the client. As I mentioned above, I should do some error handling here, but for now I am just sending back "success" no matter what happens earlier in the Flow.

When I call this Flow via Postman, I get this output:

Checking the Flow run log shows me this visual with all green checks.

And I can expand the individual actions to see what happened in each step. Here you can see the loop of execute workflow calls.

Wrapping up

Now that I've shown how you could call a Dynamics 365 workflow from Microsoft Flow, it's worth asking whether you actually should call a Dynamics 365 workflow from Microsoft Flow.

Given the state of Microsoft Flow's native Dynamics 365 support today, I would say no. At this point Flow's visual designer makes it a lot harder to build this functionality than it is if you use Azure Functions as I discussed in these earlier posts:

1. Scheduling Dynamics 365 workflows with Azure Functions and Node.js
2. Scheduling Dynamics 365 workflows with Azure Functions and Python
3. Scheduling Dynamics 365 workflows with Azure Functions and C#

For more considerations on when to use Flow or something else, here is an interesting post from Microsoft that compares and contrasts its various "serverless" computing offerings: https://docs.microsoft.com/en-us/azure/azure-functions/functions-compare-logic-apps-ms-flow-webjobs.

Although I think Flow isn't a great tool for this particular use case, I can envision some good uses in the Dynamics 365 space, and I will certainly be paying attention to see how it improves in the future. Thanks for reading!

Over the past few days, I've shared two approaches for scheduling Dynamics 365 workflows using Azure Functions and the Dynamics 365 Web API. One uses Node.js, and the other uses Python. Because most Dynamics CRM developers are probably more familiar with C# than Node.js or Python, I also created an equivalent C# version. Just like with my previous examples, this version calls the Web API directly instead of using any SDK assemblies.

Here's my code. It does the following:

1. Request an OAuth token using a username and password.
2. Query the Dynamics 365 Web API for accounts with names that start with the letter "F."
3. Execute a workflow for each record that was retrieved in the previous step. The workflow that I am executing is the same workflow I used in my Node.js example to create a note on an account.

#r "Newtonsoft.Json"

using System;
using System.Net;
using System.IO;
using Newtonsoft.Json;

//set these values to retrieve the oauth token
static string crmorg = "https://CRMORG.crm.dynamics.com";
static string clientid = "00000000-0000-0000-0000-000000000000";
static string username = "xxxxxx@xxxxxxxx";
static string userpassword = "xxxxxxxx";
static string tokenendpoint = "https://login.microsoftonline.com/00000000-0000-0000-0000-000000000000/oauth2/token";

//set these values to query your crm data
static string crmwebapihost = "https://CRMORG.api.crm.dynamics.com/api/data/v8.2";
static string crmwebapipath = "/accounts?$select=name,accountid&$filter=startswith(name,'F')";

static string workflowid = "DC8519EC-F3CE-4BC9-BB79-DF2AD70217A1";

public static void Run(TimerInfo myTimer, TraceWriter log)
{
	//build the authorization request
	var reqstring = "client_id=" + clientid;
	reqstring += "&resource=" + Uri.EscapeUriString(crmorg);
	reqstring += "&username=" + Uri.EscapeUriString(username);
	reqstring += "&password=" + Uri.EscapeUriString(userpassword);
	reqstring += "&grant_type=password";

	WebRequest req = WebRequest.Create(tokenendpoint);
	req.ContentType = "application/x-www-form-urlencoded";
	req.Method = "POST";
	byte[] bytes = System.Text.Encoding.ASCII.GetBytes(reqstring);
	req.ContentLength = bytes.Length;
	System.IO.Stream os = req.GetRequestStream();
	os.Write(bytes, 0, bytes.Length);
	os.Close();

	HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
	StreamReader tokenreader = new StreamReader(resp.GetResponseStream());
	string responseBody = tokenreader.ReadToEnd();
	tokenreader.Close();
	var tokenresponse = JsonConvert.DeserializeObject<Newtonsoft.Json.Linq.JObject>(responseBody);
	var token = tokenresponse["access_token"];
	log.Info("got token");

	WebRequest crmreq = WebRequest.Create(crmwebapihost + crmwebapipath);
	crmreq.Headers = new WebHeaderCollection();
	crmreq.Headers.Add("Authorization", "Bearer " + token);
	crmreq.Headers.Add("OData-MaxVersion", "4.0");
	crmreq.Headers.Add("OData-Version", "4.0");
	crmreq.Headers.Add("Prefer", "odata.maxpagesize=500");
	crmreq.Headers.Add("Prefer", "odata.include-annotations=OData.Community.Display.V1.FormattedValue");
	crmreq.ContentType = "application/json; charset=utf-8";
	crmreq.Method = "GET";

	HttpWebResponse crmresp = (HttpWebResponse)crmreq.GetResponse();
	StreamReader crmreader = new StreamReader(crmresp.GetResponseStream());
	string crmresponseBody = crmreader.ReadToEnd();
	crmreader.Close();
	var crmresponseobj = JsonConvert.DeserializeObject<Newtonsoft.Json.Linq.JObject>(crmresponseBody);
	foreach(var row in crmresponseobj["value"].Children())
	{
		log.Info(row["name"].ToString());
		runWorkflow(token.ToString(), new Guid(row["accountid"].ToString()), log);

	}

	Console.ReadLine();
}

static void runWorkflow(string token, Guid entityid, TraceWriter log)
{
	var crmwebapiworkflowpath = "/workflows(" + workflowid + ")/Microsoft.Dynamics.CRM.ExecuteWorkflow";
	WebRequest req = WebRequest.Create(crmwebapihost + crmwebapiworkflowpath);

	log.Info("  calling workflow for " + entityid);

	string reqobject = "{ \"EntityId\": \"" + entityid + "\"}";
    
	req.Headers.Add("Authorization", "Bearer " + token);
	req.Headers.Add("OData-MaxVersion", "4.0");
	req.Headers.Add("OData-Version", "4.0");
	req.Headers.Add("Prefer", "odata.maxpagesize=500");
	req.Headers.Add("Prefer", "odata.include-annotations=OData.Community.Display.V1.FormattedValue");
	req.ContentType = "application/json; charset=utf-8";
    req.Method = "POST";
	
	byte[] bytes = System.Text.Encoding.ASCII.GetBytes(reqobject);
	req.ContentLength = bytes.Length;
	System.IO.Stream os = req.GetRequestStream();
	os.Write(bytes, 0, bytes.Length);
	os.Close();

	HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
	StreamReader reader = new StreamReader(resp.GetResponseStream());
	string responseBody = reader.ReadToEnd();
	reader.Close();
	
    var responseobj = JsonConvert.DeserializeObject<Newtonsoft.Json.Linq.JObject>(responseBody);
	if(resp.StatusCode == HttpStatusCode.OK)
	{
		log.Info("    success " + entityid.ToString());
	}
	else
	{
		log.Info("    error " + entityid.ToString());
	}
}

To set this up in your Azure tenant, set up a Functions App and a new C# timer trigger function like I described in the Node.js example. Copy the C# code from above and paste it into the function editor window. Set any specifics relative to your Dynamics 365 organization and click save. That's all there is to it.

*If you're wondering about that #r "Newtonsoft.Json" at the top of the C# code, take a look here.

Last week I shared a solution for Scheduling Dynamics 365 workflows with Azure Functions and Node.js. In this post, I will show how to achieve equivalent functionality using Python. The actual Python code is simpler than my Node.js example, but the Azure Functions configuration is much more complicated.

First, here's the Python script I am using. It does this following:

1. Request an OAuth token using a username and password.
2. Query the Dynamics 365 Web API for accounts with names that start with the letter "F."
3. Execute a workflow for each record that was retrieved in the previous step. The workflow that I am executing is the same workflow I used in my Node.js example to create a note on an account.

import os
import sys
cwd = os.getcwd()
sitepackage = cwd + "\site-packages"
sys.path.append(sitepackage)

import requests
import json

#set these values to retrieve the oauth token
crmorg = 'https://CRMORG.crm.dynamics.com' #base url for crm org  
clientid = '00000000-0000-0000-0000-000000000000' #application client id  
username = 'xxxxxx@xxxxxxxx' #username  
userpassword = 'xxxxxxxx' #password  
tokenendpoint = 'https://login.microsoftonline.com/00000000-0000-0000-0000-000000000000/oauth2/token' #oauth token endpoint

#set these values to query your crm data
crmwebapi = 'https://lucasdemo01.api.crm.dynamics.com/api/data/v8.2'
crmwebapiquery = "/accounts?$select=name,accountid&$filter=startswith(name,'f')"

workflowid = 'DC8519EC-F3CE-4BC9-BB79-DF2AD70217A1'; #guid for the workflow you want to execute

def start():
    #build the authorization request
    tokenpost = {
        'client_id':clientid,
        'resource':crmorg,
        'username':username,
        'password':userpassword,
        'grant_type':'password'
    }

    #make the token request
    print('requesting token . . .')
    tokenres = requests.post(tokenendpoint, data=tokenpost)
    print('token response received. . .')

    accesstoken = ''

    #extract the access token
    try:
        print('parsing token response . . .')
        accesstoken = tokenres.json()['access_token']
        #print('accesstoken is - ' + accesstoken)

    except(KeyError):
        print('Could not get access token')

    if(accesstoken!=''):
        crmrequestheaders = {
            'Authorization': 'Bearer ' + accesstoken,
            'OData-MaxVersion': '4.0',
            'OData-Version': '4.0',
            'Accept': 'application/json',
            'Content-Type': 'application/json; charset=utf-8',
            'Prefer': 'odata.maxpagesize=500',
            'Prefer': 'odata.include-annotations=OData.Community.Display.V1.FormattedValue'
        }

        print('making crm account request . . .')
        crmres = requests.get(crmwebapi+crmwebapiquery, headers=crmrequestheaders)
        print('crm account response received . . .')
        try:
            print('parsing crm account response . . .')
            crmresults = crmres.json()
            for x in crmresults['value']:
                print (x['name'] + ' - ' + x['accountid'])
                runWorkflow(accesstoken, x['accountid'])
        except KeyError:
            print('Could not parse CRM account results')

        
def runWorkflow(token, entityid):
    crmwebapiworkflowpath = "/workflows("+workflowid+")/Microsoft.Dynamics.CRM.ExecuteWorkflow"

    #set the web api request headers
    requestheaders = { 
        'Authorization': 'Bearer ' + token,
        'OData-MaxVersion': '4.0',
        'OData-Version': '4.0',
        'Accept': 'application/json',
        'Content-Type': 'application/json; charset=utf-8'
    };
    
    reqobj = {'EntityId': entityid}
    
    print('  calling workflow for ' + entityid)
	
    crmres = requests.post(crmwebapi+crmwebapiworkflowpath, headers=requestheaders, data=json.dumps(reqobj))
    if(crmres.status_code == requests.codes.ok):
        print('    success ' + entityid)
    else:
        print('    error ' + entityid)

start()

To set it up as an Azure Function, after you have either created a new Function app or opened an existing Function app, do the following:

1. Create a new function. Select "create your own custom function" at the bottom.

2. Set language to "Python" and scenario to "Experimental." Currently there is no timer trigger template for Python, but you can work around that by selecting the "QueueTrigger-Python" template in this step and manually changing the trigger later.

3. Give your function a name and click create. Ignore the queue name and storage account connection values.

4. Once the function is created, select the "integrate" tab from the menu on the left and delete the Azure Queue Storage trigger.

5. Click "new trigger" at the top, and then select "timer" from the list that appears.

6. Give it a name (or leave the default) and set the schedule. The 0 */5 * * * * value here will execute every five minutes, just like in my previous Node.js example.

7. After the function is configured to run on a timer trigger, you have to upload the Python Requests library that is used to make the web service calls. Open a new browser window to download the Requests library code from GitHub at https://github.com/kennethreitz/requests/releases/. Select the most recent .zip file.

8. After it downloads, open it and extract the entire "requests" directory. It should contain a directory called "packages."

9. Zip up just the "requests" directory in a separate "requests.zip" file.

10. Back in the Azure portal Function App blade, select "function app settings" from the menu on the left and then click "Go to Kudu."

11. In the window that opens, click the "site" link at the top.

12. Then click the "wwwroot" link.

13. Then click the name of your function. In this case it is "CrmWorkflowTimerPython."

14. Go to the cmd prompt at the bottom of the page and type mkdir site-packages. Click enter. You should see a new "site-packages" directory get created.

15. Click on the new "site-packages" link to enter that directory.

16. Drag the "requests" .zip file you created in step 9 above into the "site-packages" directory like is shown.

17. Once it is uploaded, extract it by typing unzip requests.zip at the cmd prompt and clicking enter. You should see something like this when it's complete.

18. At this point, you can close the Kudu window.

19. From the Function App blade, select your Python function and click "develop." Highlight everything and delete it.

20. Paste the Python code from the beginning of this post. The highlighted lines in the image are how the script knows how to find the Requests library you uploaded earlier. Set any specifics relative to your Dynamics 365 organization, and click save. At this point it should just start running on the schedule you set earlier.

21. You can see all the function invocation logs on the monitor tab.

22. Here are the notes that the workflow created in my Dynamics 365 online org.

A few notes/caveats:

1. My Python code has hardly any error handling right now. If the workflow execution call returns an error, the Python code will not recognize it as an error.
2. My CRM record retrieval is set to retrieve a maximum of 500 records. You would need to modify the Web API request logic to handle more.
3. Per the "Best Practices for Azure Functions" guide:

Assume your function could encounter an exception at any time. Design your functions with the ability to continue from a previous fail point during the next execution.

This means you should put logic in your workflow to make sure that duplicate executions are avoided (unless that's what you intend to happen).

That's it for today. Until next time, happy coding!

Earlier this week I showed an easy way to integrate a Node.js application with Dynamics 365 using the Web API. Building on that example, I have created a scheduled workflow runner using Node.js and Azure Functions. Here's how I did it.

First, I created a workflow in Dynamics 365 that creates a note on an account record. The screenshots below shows what it looks like:

Next, I wrote Node.js code to do the following in an Azure Function.

1. Request an OAuth token using a username and password.
2. Query the Dynamics 365 Web API for accounts with names that start with the letter "F."
3. Execute a workflow for each record that was retrieved in the previous step.

Most of this is regular Node.js, but there are a couple of nuances specific to Azure Functions. See the
"Azure Functions NodeJS developer reference" for more information.

var https = require('https');

//set these values to retrieve the oauth token
//see http://alexanderdevelopment.net/post/2016/11/23/dynamics-365-and-node-js-integration-using-the-web-api/ for more details
var _crmorg = 'https://CRMORG...dynamics.crom';  
var _clientid = 'OAUTH CLIENT ID';  
var _username = 'CRM USERNAME';  
var _userpassword = 'CRM PASSWORD';  
var _tokenendpoint = 'OAUTH TOKEN ENDPOINT FROM EARLIER';

//set these values to query your crm data
var _apipath = '/api/data/v8.2'; //web api version
var _workflowid = 'DC8519EC-F3CE-4BC9-BB79-DF2AD70217A1'; //guid for the workflow you want to execute
var _crmwebapihost = 'XXXX.api.crm.dynamics.com'; //crm api url (without https://)
var _crmwebapiquerypath = "/accounts?$select=name,accountid&$filter=startswith(name,'f')"; //web api query

var _counter = 0; //variable to keep track of how many records retrieved and workflows started

module.exports = function (context, myTimer) {
	//remove https from _tokenendpoint url
	_tokenendpoint = _tokenendpoint.toLowerCase().replace('https://','');

	//get the authorization endpoint host name
	var authhost = _tokenendpoint.split('/')[0];

	//get the authorization endpoint path
	var authpath = '/' + _tokenendpoint.split('/').slice(1).join('/');

	//build the authorization request
	var reqstring = 'client_id='+_clientid;
	reqstring+='&resource='+encodeURIComponent(_crmorg);
	reqstring+='&username='+encodeURIComponent(_username);
	reqstring+='&password='+encodeURIComponent(_userpassword);
	reqstring+='&grant_type=password';

	//set the token request parameters
	var tokenrequestoptions = {
		host: authhost,
		path: authpath,
		method: 'POST',
		headers: {
			'Content-Type': 'application/x-www-form-urlencoded',
			'Content-Length': Buffer.byteLength(reqstring)
		}
	};

	//make the token request
	context.log('starting token request');
	var tokenrequest = https.request(tokenrequestoptions, function(response) {
		//make an array to hold the response parts if we get multiple parts
		var responseparts = [];
		response.setEncoding('utf8');
		response.on('data', function(chunk) {
			//add each response chunk to the responseparts array for later
			responseparts.push(chunk);		
		});
		response.on('end', function(){
			//once we have all the response parts, concatenate the parts into a single string
			var completeresponse = responseparts.join('');
			//context.log('Response: ' + completeresponse);
			context.log('token response retrieved');
			
			//parse the response JSON
			var tokenresponse = JSON.parse(completeresponse);
			
			//extract the token
			var token = tokenresponse.access_token;
			//context.log(token);
			
			//pass the token to our data retrieval function
			getData(context, token);
		});
	});
	tokenrequest.on('error', function(e) {
		context.error(e);
		context.done();
	});

	//post the token request data
	tokenrequest.write(reqstring);

	//close the token request
	tokenrequest.end();
}

function getData(context, token){
	//set the web api request headers
	var requestheaders = { 
		'Authorization': 'Bearer ' + token,
		'OData-MaxVersion': '4.0',
		'OData-Version': '4.0',
		'Accept': 'application/json',
		'Content-Type': 'application/json; charset=utf-8',
		'Prefer': 'odata.maxpagesize=500',
		'Prefer': 'odata.include-annotations=OData.Community.Display.V1.FormattedValue'
	};
	
	//set the crm request parameters
	var crmrequestoptions = {
		host: _crmwebapihost,
		path: _apipath+_crmwebapiquerypath,
		method: 'GET',
		headers: requestheaders
	};
	
	//make the web api request
	context.log('starting data request');
	var crmrequest = https.request(crmrequestoptions, function(response) {
		//make an array to hold the response parts if we get multiple parts
		var responseparts = [];
		response.setEncoding('utf8');
		response.on('data', function(chunk) {
			//add each response chunk to the responseparts array for later
			responseparts.push(chunk);		
		});
		response.on('end', function(){
			//once we have all the response parts, concatenate the parts into a single string
			var completeresponse = responseparts.join('');
			
			//parse the response JSON
			var collection = JSON.parse(completeresponse).value;
			
			//set counter length = number of records
			_counter = collection.length;

			//loop through the results and call the workflow for each one
			collection.forEach(function (row, i) {
				callWorkflow(context, token, row['accountid']);
			});
		});
	});
	crmrequest.on('error', function(e) {
		context.error(e);
		context.done();
	});
	//close the web api request
	crmrequest.end();
}

function callWorkflow(context, token, entityid){
	var crmwebapiworkflowpath = _apipath + "/workflows("+_workflowid+")/Microsoft.Dynamics.CRM.ExecuteWorkflow";

	//set the web api request headers
	var requestheaders = { 
		'Authorization': 'Bearer ' + token,
		'OData-MaxVersion': '4.0',
		'OData-Version': '4.0',
		'Accept': 'application/json',
		'Content-Type': 'application/json; charset=utf-8'
	};
	
	//set the crm request parameters
	var crmrequestoptions = {
		host: _crmwebapihost,
		path: crmwebapiworkflowpath,
		method: 'POST',
		headers: requestheaders
	};

	//create an object to post to the executeworkflow action
	var reqobj = {};
	reqobj["EntityId"] = entityid;
	
	//turn it into a string
	var reqjson = JSON.stringify(reqobj);
	
	//calculate the length to set the content-length header
	crmrequestoptions.headers['Content-Length'] = Buffer.byteLength(reqjson);
	
	//make the web api request
	context.log('starting workflow request for ' + entityid);
	var crmrequest = https.request(crmrequestoptions, function(response) {
		//make an array to hold the response parts if we get multiple parts
		var responseparts = [];
		response.setEncoding('utf8');
		response.on('data', function(chunk) {
			//add each response chunk to the responseparts array for later
			responseparts.push(chunk);		
		});
		response.on('end', function(){
			//once we have all the response parts, concatenate the parts into a single string
			var completeresponse = responseparts.join('');
			context.log('success ' + entityid);
			
			//decrement the counter
			_counter = _counter-1;
			
			//if nothing is left to start, we are done
			if(_counter==0){
				context.log('all workflows started');
				context.done();
			}
		});
	});
	crmrequest.on('error', function(e) {
		context.error(e);
		context.done();
	});
	crmrequest.write(reqjson);

	//close the web api request
	crmrequest.end();
}

Then in the Azure Portal, I configured an Azure Function app to query accounts and execute the workflow every five minutes. Here are the detailed steps to replicate that.

1. Create a new Function app via New->Compute->Function App.
2. Set the app name, resource group, etc.
3. Once the new Function app is provisioned, open it.
4. Select "new function" on the left.
5. Set language to "JavaScript" and scenario to "Core." Find the "TimerTrigger-JavaScript" template and select it.
6. Give your function a name and set the schedule options. The schedule value is a CRON expression that includes six fields: {second} {minute} {hour} {day} {month} {day of the week}. You can accept the default value of every five minutes and change it later. Click "create" to create the new function.
7. Copy the Node.js code from above and paste it into the editor window. Set any specifics relative to your Dynamics 365 organization, and click save. (You can also use Git for deploying your code, but that's beyond the scope of today's post.)
8. On the "integrate" tab, you can modify the timer schedule. The schedule shown (0 */5 * * * *) will execute the function every five minutes.
9. The function will automatically execute at the next fifth minute, and the invocation log is available on the monitor tab. Selecting a specific invocation row shows detailed logging output on the right.
10. This screenshot shows the process sessions for when the workflow was executed in Dynamics 365.
11. This screenshot shows the note records that were created by the workflow.

A few notes/caveats:

1. My Node.js code has hardly any error handling right now. If the workflow execution call returns an error, the Node.js code will not recognize it as an error.
2. My CRM record retrieval is set to retrieve a maximum of 500 records. You would need to modify the Web API request logic to handle more.
3. Per the "Best Practices for Azure Functions" guide:

Assume your function could encounter an exception at any time. Design your functions with the ability to continue from a previous fail point during the next execution.

This means you should put logic in your workflow to make sure that duplicate executions are avoided (unless that's what you intend to happen).

This sample just scratches the surface of what's possible with Azure Functions and Dynamics 365, and I'm looking forward to working with Azure Functions more in the future. Have you looked at Azure Functions yet? What do you think? Please let me know in the comments.

For the last several months I've been working on an enterprise Dynamics CRM project where one of our goals is to minimize the amount of custom code we write by using North52's Business Process Activities. I had not been exposed to North52 before working on this project, but I have been pleasantly surprised with how much it has allowed our mostly functional resources to achieve without needing technical assistance.

While looking through North52's documentation a while back, I noticed it could be used to call a REST web service. This got me thinking about how I could rework my Sentiment analysis in Dynamics CRM using Azure Text Analytics sample using North52 instead of a custom workflow activity.

I found I was able to replace the custom workflow activity with a North52 Process Genie. It executes a Smart Flow to call the Azure Machine Learning Text Analytics API and then return the result to the calling CRM dialog.

Here's the formula I used in my Process Genie:

SmartFlow(
  SetVar('jsoninput', CreateJObject( 
     CreateJProperty('Inputs', 
           CreateJArray(CreateJObject(
                           CreateJProperty('Id', '1'),  
                           CreateJProperty('Text', [account.texttoanalyze]) 
                              )
                        )
                     )
               )
          ),

  CallRestAPI(
      SetRequestBaseURL('https://api.datamarket.azure.com/data.ashx/amla/text-analytics/v1'),
      SetRequestResource('/GetSentimentBatch'),
      SetRequestDetails('POST'),
      SetRequestHeaders(),
      SetRequestParams('RawContentTextJSON',GetVar('jsoninput')),
      SetRequestAuthenticationBasic('AccountKey','YOUR_AZURE_ML_API_KEY_HERE'),
      SetRequestFiles(),
      SetRequestExpected('OK'),
      SetRequestActionPass(SetVar('result', GetVarJsonValue('SentimentBatch{0}.Score'))),
      SetRequestActionFail(SetVar('result', 'ERROR' + GetVarJsonValue('Errors{0}.Message')))
    ), 

   SmartFlowReturn(GetVar('result'))
)

Here's a screenshot of my CRM dialog:

To execute the Process Genie, you use a North52 N52 Process Genie step like this:

The Formula ShortCode value is the short code of the North52 Process Genie. The Formula Parameter Xml value contains the text to analyze from the dialog input in the format expected by the formula:

<account><texttoanalyze>{Response Text(Get text)}</texttoanalyze></account>

If were going to use this in real solution, I would not hardcore the Azure ML API key directly in the formula. Other than that I think this is a production-ready approach.

What do you think? Would you consider using something like this as an alternative to writing your own custom code?

On Monday, April 11, at 12 p.m. EDT, I will be presenting a webcast at MSDynamicsWorld.com that will show how a custom integration with Microsoft Azure Machine Learning can be used to perform sentiment analysis on any data stored in Dynamics CRM.

Custom sentiment analysis integrations can enable a number of interesting processes in Dynamics CRM including:

• Routing emails to queues based on sentiment
• Dynamically load agent scripts
• Reporting on interactions

This webinar will include an introduction to Azure Machine Learning, an overview of a sample solution and a code deep dive. Programming knowledge will be helpful during the code deep dive, but it is not required for most of the discussion.

You can register for the session here.

Earlier this year I wrote a post that showed how to perform sentiment analysis in Dynamics CRM using Microsoft Azure Text Analytics. Azure Text Analytics makes it incredibly easy to use sentiment analysis (with English text only), but the full Azure Machine Learning offering is much more powerful. In today's post I will show how to create a custom predictive web service in Azure ML and make predictions with it in Dynamics CRM.

Why am I doing this?

One of the exciting announcements about Dynamics CRM 2016 is that it includes some sort of integration with Azure ML, so what's the point of this blog post? Actually, here are two reasons:

1. It's not clear (as of late November 2016) what capabilities the standard CRM-Azure ML integration will have. The approach demonstrated here allows for total customization.
2. The approach below can be used right now in any version of CRM (online or on-premise) from CRM 2011 onward.

The demo data set

For this demonstration I am using data from the AdventureWorks data warehouse sample database to build a model to predict whether a contact in CRM is likely to be a bicycle buyer. I created a flat file of contacts that includes several independent variables and a yes/no flag for whether they purchased a bicycle. That sample file is included in my solution source linked at the bottom of this post.

You can also download the full AdventureWorks database from CodePlex here.

Getting started with Azure ML

As I was trying to get the hang of Azure ML, I found the Bluewater SQL blog tremendously helpful. If you're just getting started with Azure ML, I suggest you take a look at both of these posts before continuing. (These two posts also use the AdventureWorks data warehouse sample database, though I had already decided to use it for my demo before I found this site. I guess the AdventureWorks data is an obvious choice for this sort of thing.)

1. https://bluewatersql.wordpress.com/2014/07/31/azure-machine-learning-first-look/
2. https://bluewatersql.wordpress.com/2014/08/01/azure-machine-learning-a-deeper-look/

Creating the predictive web service

Now that the introductory bits are out of the way, here's an overview of how I set up my predictive web service in Azure ML. I've also shared my work in the Cortana Analytics Gallery, so if you just want to see how the experiments are set up without going through all the steps yourself, skip to the end of this post for the gallery links.

1. Export your data set from CRM. I use a simple KingswaySoft SSIS package to do this. It's included in my sample project files.
2. Upload your data set to Azure ML Studio. I am using the Azure ML free tier, which requires me to manually upload the data set for the initial upload and any subsequent updates. If I were using the standard tier, it would be possible to upload the data set to Azure blob storage as part of my SSIS package. This would be useful for periodically retraining the model with updated data in the future.
3. Create and run a new experiment to classify bike buyers. I basically followed the steps outlined in the two Bluewater SQL posts, so take a look at those if you want to see a more detailed explanation.
4. Save the trained model from this experiment to use in your predictive web service.
5. Create a new experiment that will serve as the foundation for the predictive web service
6. Add the following items to the new experiment canvas:
7. Data set
8. Trained model from the earlier experiment
9. Score model component
10. Project columns transformation between the data set and score model component to exclude the lpa_bikerbuyername column
11. Project columns transformation after score model component to only include score labels
12. Run the experiment.
13. Create a predictive web service.
14. Run the experiment.
15. Deploy the web service.
16. Open the web service and copy the API key to use later.
17. Open the request/response page.
18. Copy the post URL to use later. Leave off the "&details=true" at the end.
19. Scroll down to find the JSON request/response samples and copy them for review later.

Making predictions from CRM

Once your predictive web service is set up, using it in Dynamics CRM is as simple as posting a JSON request to the web service and then parsing the JSON response. A few years ago, I wrote a code sample showing an easy way to make JSON requests from CRM custom assemblies that serves as the foundation for the workflow activity I'm using in this demo. There are only a few changes required to that sample:

1. Add an authorization header that includes the bearer API key you copied earlier.
2. Add input/output parameters that match the inputs and outputs for your predictive web service. I am also passing the API key and service endpoint as input parameters instead of hardcoding them.
3. Modify the JSON request/response classes to allow for serialization to/deserialization from the request and response messages. Because of how Azure ML passes the inputs/outputs, these are actually fairly generic, so you can probably use mine with no or minimal changes.
4. Send the correct request parameters and correctly handle the response parameters.

To demonstrate the custom workflow activity, I created a sample CRM dialog.

Here's where I supply the input parameters.

Here's what the dialog looks like when I run it from a contact record.

The code

You can get all the code for this demonstration from my GitHub repository here. The sample code includes the following:

1. CRM solution containing:
2. Contact entity with custom fields added and a system view for data export for Azure ML
3. Sample dialog (you would need to update the parameters with your own API key and endpoint details)
4. Compiled custom workflow activity to call the web service
5. Contact data ready for import to CRM
6. SSIS package solution for data export
7. Custom workflow activity source code

I've also shared my training and prediction experiments in the Cortana Analytics Gallery. Here are the links where you can open and import them into your own workspace:

1. Training experiment
2. Prediction experiment

What do you think about this approach? Does it seem like it would be useful to you in a real-world situation? Let me know in the comments!

Last year I created a proof-of-concept solution that showed how to integrate Dynamics CRM with HP Haven OnDemand (then called HP IDOL OnDemand) to perform sentiment analysis and index records to support "find similar" queries. While I was working through the AzureCon challenge a few weeks ago, I thought it would be an interesting exercise to update my sentiment analysis code to work with the Text Analytics offering from the Microsoft Azure Marketplace.

The approach

As with my Haven OnDemand solution, the approach I'm using with Azure relies on a custom workflow activity that does the following:

1. Parse a supplied text input and strip any HTML tags using a helper function.
2. Create a JSON sentiment analysis request and post it to Azure Text Analytics with an HttpWebRequest.
3. Deserialize the JSON respsonse returned by Azure Text Analytics to a custom class object using a DataContractJsonSerializer.
4. Return the sentiment score to the calling process.

There are two main differences with the my Azure Text Analytics solution:

1. The Azure service only returns a sentiment score, so this custom workflow activity doesn't return a positive/negative string value.
2. Instead of embedding an access key in the custom workflow activity code, I've made it a parameter, which means you can take the solution straight from GitHub and start using it in your CRM organization as soon as you sign up for the Text Analytics service.

The solution in action

Here's a sample dialog I've created to demonstrate the use of the custom workflow activity.

Here's how the process works for the sample text "I hate you."

Wrapping it up on a more positive note, here's the same dialog with "I love you" instead.

As you can see, the score for "I hate you" is about .06, and the score for "I love you" is about .91, which makes sense as scores closer to one are more positive, and scores closer to zero are more negative.

The code

You can download all the custom code and a CRM solution extract from my Crm-Sample-Code repository on GitHub. Let me know what you think in the comments!