Over the past several months, I've been doing a lot of work with OpenFaaS in my spare time, and in today's post I will show how you can use it to easily build and deploy a custom web service interface for data in a Dynamics 365 Customer Engagement online tenant.

OpenFaaS

If you're not familiar with OpenFaaS, it's basically a serverless functions platform like Azure Functions or AWS Lambda, but you run it on Kubernetes or Docker Swarm on your own servers or in the cloud. What I particularly like about OpenFaaS compared to the various commercial serverless platforms is that in addition to offering more control over how/where it's deployed, OpenFaaS supports a wider variety of languages for writing serverless functions.

OpenFaaS (Functions as a Service) is a framework for building serverless functions with Docker and Kubernetes which has first class support for metrics. Any process can be packaged as a function enabling you to consume a range of web events without repetitive boiler-plate coding.

To follow along with the samples in this post, you'll need access to a cluster with OpenFaaS deployed, so if you don't already have one, now would be an excellent time to look at the OpenFaaS deployment docs or maybe even work through the hands-on workshop. I've also previously written about how to securely deploy OpenFaaS on a free Google Cloud VM with Docker Swarm or on an Azure Kubernetes Service cluster.

Preparing to build the interface function

As soon as you have OpenFaaS running, it's time to look at the actual custom interface function.

My demo C# function does the following:

1. Parse a JSON object sent in the client request for an access key and optional query filter
2. Validate the client-supplied access key to authorize or reject the request
3. Retrieve a Dynamics 365 OAuth access token using my Azure AD OAuth 2 helper microservice
4. Execute a query for contacts against the Dynamics 365 Web API
5. Return the Web API query results to the client in an array as part of a JSON object

Because the OpenFaaS function uses my OAuth helper microservice instead of requesting an OAuth access token directly from Azure Active Directory, you need to deploy that microservice to your cluster before moving forward.

If you're using Kubernetes, you can create the deployment and corresponding service using the following YAML. You'll need to set the RESOURCE environment variable to the FQDN for your Dynamics 365 CE organization, but you can leave the CLIENTID and TOKEN_ENDPOINT values alone. (While I used to think you needed to register a separate client application for every Dynamics 365 org to use OAuth authentication, I recently learned via a Twitter conversation that there is a "universal" CRM client id you can use instead.)

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: azuread-oauth2-helper
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: azuread-oauth2-helper
    spec:
      containers:
      - name: azuread-oauth2-helper
        image: lucasalexander/azuread-oauth2-helper
        ports:
        - containerPort: 5000
        env:
        - name: RESOURCE
          value: "https://XXXXXXXX.crm.dynamics.com"
        - name: CLIENTID
          value: "2ad88395-b77d-4561-9441-d0e40824f9bc"
        - name: TOKEN_ENDPOINT
          value: "https://login.microsoftonline.com/common/oauth2/token"
---
apiVersion: v1
kind: Service
metadata:
  name: azuread-oauth2-helper
spec:
  ports:
  - port: 5000
  selector:
    app: azuread-oauth2-helper

Once you've deployed the microservice, here's the definition for a Kubernetes ingress. In this case my microservice is accessible on the same host as OpenFaaS (akskube.alexanderdevelopment.net), and it is secured with the same Let's Encrypt certificate. You'll want to update your configuration with the appropriate values for your specific situation.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: azuread-oauth2-helper-ingress
  annotations:
    kubernetes.io/tls-acme: "true"
    certmanager.k8s.io/issuer: letsencrypt-production
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  tls:
  - hosts:
    - akskube.alexanderdevelopment.net
    secretName: faas-letsencrypt-production
  rules:
  - host: akskube.alexanderdevelopment.net
    http:
      paths:
      - path: /oauthhelper
        backend:
          serviceName: azuread-oauth2-helper
          servicePort: 5000

After the OAuth helper microservice is deployed, you should validate that you can get a token returned for a valid username/password combination. Here's what that looks like in Postman.

Building the interface function

If you've made it to this point, building and deploying the function is easy!

First the function gets its configuration data from environment variables that are set when the function is deployed. If you were actually using this function in production, it would be better to store sensitive values like the access key and the Dynamics 365 password as secrets, but I've used environment variables here to keep this overview as simple as possible.

//get configuration from env variables        
var username = Environment.GetEnvironmentVariable("USERNAME");
var userpassword = Environment.GetEnvironmentVariable("USERPASS");
var tokenendpoint = Environment.GetEnvironmentVariable("TOKENENDPOINT");
var accesskey = Environment.GetEnvironmentVariable("ACCESSKEY");
var crmwebapi = Environment.GetEnvironmentVariable("CRMAPI");

After the function gets its configuration data, it deserializes the client request using Json.Net to extract a client-supplied access key and an optional query filter. The client-supplied key is validated against the stored key value, and if they don't match, an error response is returned.

var queryrequest = JsonConvert.DeserializeObject<QueryRequest>(input);

if(accesskey!=queryrequest.AccessKey)
{
    JObject outputobject = new JObject();
    outputobject.Add("error", "Invalid access key");
    Console.WriteLine(outputobject.ToString());
    return;
}

After the access key is validated, the function then makes a request to the authentication helper microservice to get an access token.

var token = GetToken(username, userpassword, tokenendpoint);

...
...
...

string GetToken(string username, string userpassword, string tokenendpoint){
    try
    {
        JObject tokencredentials = new JObject();
        tokencredentials.Add("username", username);
        tokencredentials.Add("password",userpassword);
        var reqcontent = new StringContent(tokencredentials.ToString(), Encoding.UTF8, "application/json");
        var result = _client.PostAsync(tokenendpoint, reqcontent).Result;
        var tokenobj = JsonConvert.DeserializeObject<Newtonsoft.Json.Linq.JObject>(
            result.Content.ReadAsStringAsync().Result);
        var token = tokenobj["accesstoken"];
        return token.ToString();
    }
    catch(Exception ex)
    {
        return string.Format("Error: {0}", ex.Message);
    }
}

Once the token is returned from the microservice, the function executes the Web API query. The query is just a hardcoded OData query in the form of /contacts?$select=fullname,contactid plus any filter supplied by the client. The function expects that the filter will also be provided in supported Dynamics 365 OData format like startswith(fullname,'y').

var crmreq = new HttpRequestMessage(HttpMethod.Get, crmwebapi + crmwebapiquery);
crmreq.Headers.Add("Authorization", "Bearer " + token);
crmreq.Headers.Add("OData-MaxVersion", "4.0");
crmreq.Headers.Add("OData-Version", "4.0");
crmreq.Headers.Add("Prefer", "odata.maxpagesize=500");
crmreq.Headers.Add("Prefer", "odata.include-annotations=OData.Community.Display.V1.FormattedValue");
crmreq.Content = new StringContent(string.Empty.ToString(), Encoding.UTF8, "application/json");
var crmres = _client.SendAsync(crmreq).Result;

var crmresponse = crmres.Content.ReadAsStringAsync().Result;

var crmresponseobj = JsonConvert.DeserializeObject<Newtonsoft.Json.Linq.JObject>(crmresponse);

Finally results are returned to the client in an array as part of a JSON object.

JArray outputarray = new JArray();
foreach(var row in crmresponseobj["value"].Children())
{
    JObject record = new JObject();
    record.Add("id", row["contactid"]);
    record.Add("fullname", row["fullname"]);
    outputarray.Add(record);
}
JObject outputobject = new JObject();
outputobject.Add("contacts", outputarray);
Console.WriteLine(outputobject.ToString());

Here's the complete function.

using System;
using System.Text;
using System.Net;
using System.Net.Http;
using System.Net.Http.Headers;
using System.IO;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using System.Collections.Generic;

namespace Function
{
    public class FunctionHandler
    {
        private static HttpClient _client = new HttpClient();

        public void Handle(string input) {
            //get configuration from env variables        
            var username = Environment.GetEnvironmentVariable("USERNAME");
            var userpassword = Environment.GetEnvironmentVariable("USERPASS");
            var tokenendpoint = Environment.GetEnvironmentVariable("TOKENENDPOINT");
            var accesskey = Environment.GetEnvironmentVariable("ACCESSKEY");
            var crmwebapi = Environment.GetEnvironmentVariable("CRMAPI");
            
            //deserialize the client request
            var queryrequest = JsonConvert.DeserializeObject<QueryRequest>(input);
            
            //validate the client access key
            if(accesskey!=queryrequest.AccessKey)
            {
                JObject outputobject = new JObject();
                outputobject.Add("error", "Invalid access key");
                Console.WriteLine(outputobject.ToString());
                return;
            }

            //get the oauth token
            var token = GetToken(username, userpassword, tokenendpoint);
            
            if(!token.ToUpper().StartsWith("ERROR:"))
            {
                //set the base odata query
                var crmwebapiquery = "/contacts?$select=fullname,contactid";
                
                //add a filter if the client included one in the request
                if(!string.IsNullOrEmpty(queryrequest.Filter))
                    crmwebapiquery+="&$filter="+queryrequest.Filter;
                try
                {
                    //make the request to d365
                    var crmreq = new HttpRequestMessage(HttpMethod.Get, crmwebapi + crmwebapiquery);
                    crmreq.Headers.Add("Authorization", "Bearer " + token);
                    crmreq.Headers.Add("OData-MaxVersion", "4.0");
                    crmreq.Headers.Add("OData-Version", "4.0");
                    crmreq.Headers.Add("Prefer", "odata.maxpagesize=500");
                    crmreq.Headers.Add("Prefer", "odata.include-annotations=OData.Community.Display.V1.FormattedValue");
                    crmreq.Content = new StringContent(string.Empty.ToString(), Encoding.UTF8, "application/json");
                    var crmres = _client.SendAsync(crmreq).Result;
                    
                    //handle the d365 response
                    var crmresponse = crmres.Content.ReadAsStringAsync().Result;

                    var crmresponseobj = JsonConvert.DeserializeObject<Newtonsoft.Json.Linq.JObject>(crmresponse);
                    
                    try
                    {
                        //build the function response
                        JArray outputarray = new JArray();
                        foreach(var row in crmresponseobj["value"].Children())
                        {
                            JObject record = new JObject();
                            record.Add("id", row["contactid"]);
                            record.Add("fullname", row["fullname"]);
                            outputarray.Add(record);
                        }
                        JObject outputobject = new JObject();
                        outputobject.Add("contacts", outputarray);
                        
                        //return the response to the client
                        Console.WriteLine(outputobject.ToString());
                    }
                    catch(Exception ex)
                    {
                        JObject outputobject = new JObject();
                        outputobject.Add("error", string.Format("Could not parse query response: {0}", ex.Message));
                        Console.WriteLine(outputobject.ToString());
                    }
                }
                catch(Exception ex)
                {
                    JObject outputobject = new JObject();
                    outputobject.Add("error", string.Format("Could not query data: {0}", ex.Message));
                    Console.WriteLine(outputobject.ToString());
                }
            }
            else
            {
                JObject outputobject = new JObject();
                outputobject.Add("error", "Could not get token");
                Console.WriteLine(outputobject.ToString());
            }
        }

        string GetToken(string username, string userpassword, string tokenendpoint){
            try
            {
                JObject tokencredentials = new JObject();
                tokencredentials.Add("username", username);
                tokencredentials.Add("password",userpassword);
                var reqcontent = new StringContent(tokencredentials.ToString(), Encoding.UTF8, "application/json");
                var result = _client.PostAsync(tokenendpoint, reqcontent).Result;
                var tokenobj = JsonConvert.DeserializeObject<Newtonsoft.Json.Linq.JObject>(
                    result.Content.ReadAsStringAsync().Result);
                var token = tokenobj["accesstoken"];
                return token.ToString();
            }
            catch(Exception ex)
            {
                return string.Format("Error: {0}", ex.Message);
            }
        }
    }

    public class QueryRequest
    {
        public string AccessKey {get;set;}
        public string Filter{get;set;}
    }
}

Because the function relies on Json.Net, you need to add a reference to it in your .csproj file before you build the function.

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>netstandard2.0</TargetFramework>
  </PropertyGroup>
  <PropertyGroup>
    <GenerateAssemblyInfo>false</GenerateAssemblyInfo>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="newtonsoft.json" Version="11.0.2" />
  </ItemGroup>
</Project>

Here is my function definition YAML file with enviroment variables included. You will need to update them with your appropriate values, and you will also need to change the image name if you're building your own function instead of just deploying mine from Docker Hub.

provider:
  name: faas
  gateway: http://localhost:8080

functions:
  demo-crm-function:
    lang: csharp
    handler: ./demo-crm-function
    image: lucasalexander/faas-demo-crm-function
    environment:
      USERNAME: XXXXXX@XXXXXX.onmicrosoft.com
      USERPASS: XXXXXX
      TOKENENDPOINT: https://akskube.alexanderdevelopment.net/oauthhelper/requesttoken
      CRMAPI: https://lucastest20.api.crm.dynamics.com/api/data/v9.0
      ACCESSKEY: MYACCESSKEY

Once the function is deployed, you can execute it either through the OpenFaaS admin UI or another tool that makes HTTP requests like Curl or Postman. Here's what an unfiltered query in Postman looks like for a Dynamics 365 org with sample data installed.

And here's a query with a filter included.

Wrapping up

Once I got OpenFaaS running, writing and deploying the actual function only took about an hour. Obviously writing a more complex data interface to support real-world requirements would take longer, but using a serverless functions platform like OpenFaaS is definitely a significant accelerator for custom Dynamics 365 integration development.

What do you think about this approach? Are you using serverless functions with your Dynamics 365 projects? What do you think about OpenFaaS vs Azure Functions or AWS Lambda? Let us know in the comments!

I was recently asked to be a guest on the third-anniversary episode of the CRM Audio podcast. While I was there George Doubinski challenged me to create a plugin in one Dynamics 365 organization to retrieve records from another Dynamics 365 organization so they could be displayed as virtual entities. I was promised adulation on Dynamics CRM Tip of the Day and fame beyond my wildest dreams, so naturally I accepted.

To address the challenge, I wrote a simple Dynamics 365 plugin that calls the Web API in a different Dynamics 365 organization to retrieve records and return them to a virtual entity data provider. From there, configuration of the Dynamics 365 virtual entity is simple. Let's take a look at how I did it.

The plugin

First you need to create a plugin to retrieve the data from the "external" Dynamics 365 org. Because this code connects directly to the Web API, you'll need to get an access token from Azure AD before you can make the request to Dynamics 365. Just like I showed in my "Scheduling Dynamics 365 workflows with Azure Functions and C#" post back in 2016, my sample code does not use ADAL to get the access token, but rather it issues a request directly to the Azure AD OAuth 2 token endpoint.

Here's the code for the plugin. There are some configuration values you'll need to set for your Dynamics 365 organization and whatever query you want to run. It's not a best practice to have any of this actually hardcoded in your plugin, but I've done it this way so it's easier to see how things work.

using Microsoft.Xrm.Sdk;
using Microsoft.Xrm.Sdk.Data.Exceptions;
using Microsoft.Xrm.Sdk.Extensions;
using Microsoft.Xrm.Sdk.Query;
using System;
using System.IO;
using System.Net;
using System.Threading.Tasks;
using Newtonsoft.Json;

namespace VirtualEntityProvider
{
    public class RetrieveOtherOrgData : IPlugin
    {
        //set these values for your D365 instance, user credentials and Azure AD clientid/token endpoint
        string crmorg = "https://XXXXX.crm.dynamics.com";
        string clientid = "XXXXXXXXX";
        string username = "lucasalexander@XXXXXX.onmicrosoft.com";
        string userpassword = "XXXXXXXXXXXX";
        string tokenendpoint = "https://login.microsoftonline.com/XXXXXXXXXXX/oauth2/token";

        //relative path to web api endpoint
        string crmwebapi = "/api/data/v8.2";

        //web api query to execute - in this case all accounts that start with "F"
        string crmwebapipath = "/accounts?$select=name,accountid&$filter=startswith(name,'F')";

        public void Execute(IServiceProvider serviceProvider)
        {
            //basic plugin set-up stuff
            IPluginExecutionContext context = (IPluginExecutionContext)serviceProvider.GetService(typeof(IPluginExecutionContext));
            IOrganizationServiceFactory servicefactory = (IOrganizationServiceFactory)serviceProvider.GetService(typeof(IOrganizationServiceFactory));
            IOrganizationService service = servicefactory.CreateOrganizationService(context.UserId);
            ITracingService tracingService = (ITracingService)serviceProvider.GetService(typeof(ITracingService));

            try
            {
                //instantiate a new entity collection to hold the records we'll return later
                EntityCollection results = new EntityCollection();

                //build the authorization request for Azure AD
                var reqstring = "client_id=" + clientid;
                reqstring += "&resource=" + Uri.EscapeUriString(crmorg);
                reqstring += "&username=" + Uri.EscapeUriString(username);
                reqstring += "&password=" + Uri.EscapeUriString(userpassword);
                reqstring += "&grant_type=password";

                //make the Azure AD authentication request
                WebRequest req = WebRequest.Create(tokenendpoint);
                req.ContentType = "application/x-www-form-urlencoded";
                req.Method = "POST";
                byte[] bytes = System.Text.Encoding.ASCII.GetBytes(reqstring);
                req.ContentLength = bytes.Length;
                System.IO.Stream os = req.GetRequestStream();
                os.Write(bytes, 0, bytes.Length);
                os.Close();

                HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
                StreamReader tokenreader = new StreamReader(resp.GetResponseStream());
                string responseBody = tokenreader.ReadToEnd();
                tokenreader.Close();

                //deserialize the Azure AD token response and get the access token to supply with the web api query
                var tokenresponse = JsonConvert.DeserializeObject<Newtonsoft.Json.Linq.JObject>(responseBody);
                var token = tokenresponse["access_token"];

                //make the web api query
                WebRequest crmreq = WebRequest.Create(crmorg+crmwebapi+crmwebapipath);
                crmreq.Headers = new WebHeaderCollection();

                //use the access token from earlier as the authorization header bearer value
                crmreq.Headers.Add("Authorization", "Bearer " + token);
                crmreq.Headers.Add("OData-MaxVersion", "4.0");
                crmreq.Headers.Add("OData-Version", "4.0");
                crmreq.Headers.Add("Prefer", "odata.maxpagesize=500");
                crmreq.Headers.Add("Prefer", "odata.include-annotations=OData.Community.Display.V1.FormattedValue");
                crmreq.ContentType = "application/json; charset=utf-8";
                crmreq.Method = "GET";

                HttpWebResponse crmresp = (HttpWebResponse)crmreq.GetResponse();
                StreamReader crmreader = new StreamReader(crmresp.GetResponseStream());
                string crmresponseBody = crmreader.ReadToEnd();
                crmreader.Close();

                //deserialize the response
                var crmresponseobj = JsonConvert.DeserializeObject<Newtonsoft.Json.Linq.JObject>(crmresponseBody);

                //loop through the response values
                foreach (var row in crmresponseobj["value"].Children())
                {
                    //create a new virtual entity of type lpa_demove
                    Entity verow = new Entity("lpa_otheraccount");
                    //verow["lpa_otheraccountid"] = Guid.NewGuid();
                    //verow["lpa_name"] = ((Newtonsoft.Json.Linq.JValue)row["name"]).Value.ToString();
                    verow["lpa_otheraccountid"] = (Guid)row["accountid"];
                    verow["lpa_name"] = (string)row["name"];

                    //add it to the collection
                    results.Entities.Add(verow);
                }

                //return the results
                context.OutputParameters["BusinessEntityCollection"] = results;
            }
            catch (Exception e)
            {
                tracingService.Trace($"{e.Message} {e.StackTrace}");
                if (e.InnerException != null)
                    tracingService.Trace($"{e.InnerException.Message} {e.InnerException.StackTrace}");

                throw new InvalidPluginExecutionException(e.Message);
            }
        }
    }
}

Because the plugin uses JSON.Net, you'll need to use ILMerge to bundle the Newtonsoft.Json.dll assembly with your compiled plugin before you deploy it to Dynamics 365.

Setting up the virtual entity

After you've deployed the plugin using the plugin registration tool, register a new data provider. When the data provider registration window opens, first create a new data source entity.

Complete the details for the data source and save it.

Complete the rest of the details for the data provider and save it.

You should now see a new data provider and data source.

Open the Dynamics 365 web UI, and go to settings->administration->virtual entity data sources.

Click the "new" button to create a new virtual entity data source.

In the window that pops up, select the data provider you created earlier.

Give your new virtual entity data source a name and save it.

Open your solution and create a new entity.

Configure your entity as a virtual entity that uses the virtual entity data source you created previously.

Once you save and publish the virtual entity, you can open an advanced find view that will retrieve data from your other Dynamics 365 organization and display it.

If you export this data to Excel and unhide the id column, you will see that the GUIDs match the records in the external system.

And that's all there is to it. Happy entity virtualizing!

Last week at its annual Build conference, Microsoft announced ML.NET, an "open source and cross-platform machine learning framework" that runs in .NET Core. I took a look at the getting started samples and realized ML.NET would be a great tool to use in OpenFaas functions.

I decided to write a proof-of-concept function based on the ML.NET sentiment analysis sample. Because the function needs a trained model before it can run, you actually need to use a separate application to generate the model and save it as a file. Then you can include the model as part of your function deployment.

Here's a screenshot of my function in action.

You can get the code for my OpenFaas sentiment analysis function here, and the code for the application that generates the model is available here.

This is the final post in my series about building a service relay for Dynamics 365 CE with RabbitMQ and Python. In my previous post in this series, I showed the Python code to make the service relay work. In today's post, I will show how you can use Azure Functions to make a consumer service proxy using C# so client applications don't have to access to your RabbitMQ broker directly, and I will also discuss some general thoughts on security and scalability for this service relay architecture.

Although this simple service relay allows external consumers to get data from Dynamics 365 CE without needing to connect directly, the examples I've shown so far require that they can connect to a RabbitMQ broker. This may be problematic for a variety of reasons, so you would probably want external consumers to connect to a web service proxy that would write requests to and read responses from the RabbitMQ broker.

Building a service proxy function

You can build an Azure Functions service proxy with Python, but I don't recommend it for three reasons:

1. Azure Functions Python support is still considered experimental.
2. Python scripts that use external libraries can run exceedingly slow.
3. Getting the environment set up is a bit of a hassle.

On the other hand, building a service proxy function with C# was so much easier, and it performed much better than a comparable Python function (~.5 seconds for C# compared to 5+ seconds for Python).

Here are the steps I took to build my C# service proxy function:

1. Create a C# HTTP trigger function.
2. Create and upload a project.json file with a dependency on the RabbitMQ client (see below).
3. Take the "RpcClient" class from the RabbitMQ .Net RPC tutorial and call it from within my function.

Here's my project.json file:

{
  "frameworks": {
    "net46":{
      "dependencies": {
        "RabbitMQ.Client": "5.0.1"
      }
    }
   }
}

And here's my run.csx file:

using System.Net;
using System;
using System.Collections.Concurrent;
using System.Text;
using RabbitMQ.Client;
using RabbitMQ.Client.Events;

public static async Task<HttpResponseMessage> Run(HttpRequestMessage req, TraceWriter log)
{
    log.Info("Processing request");

    // parse query parameter
    string query = req.GetQueryNameValuePairs()
        .FirstOrDefault(q => string.Compare(q.Key, "query", true) == 0)
        .Value;

    // Get request body
    dynamic data = await req.Content.ReadAsAsync<object>();

    // Set name to query string or body data
    query = query ?? data?.query;

    var rpcClient = new RpcClient();
    
    log.Info(string.Format(" [.] query start time {0}", DateTime.Now.ToString("MM/dd/yyyy hh:mm:ss.fff tt")));
    var response = rpcClient.Call(query);

    log.Info(string.Format(" [.] query end time {0}", DateTime.Now.ToString("MM/dd/yyyy hh:mm:ss.fff tt")));
    rpcClient.Close();

    return req.CreateResponse(HttpStatusCode.OK, response);
}

public class RpcClient
{
    private readonly IConnection connection;
    private readonly IModel channel;
    private readonly string replyQueueName;
    private readonly EventingBasicConsumer consumer;
    private readonly BlockingCollection<string> respQueue = new BlockingCollection<string>();
    private readonly IBasicProperties props;

    public RpcClient()
    {
        var factory = new ConnectionFactory() { HostName = "RABBITHOST", UserName="RABBITUSER", Password="RABBITUSERPASS"  };

        connection = factory.CreateConnection();
        channel = connection.CreateModel();
        replyQueueName = channel.QueueDeclare().QueueName;
        consumer = new EventingBasicConsumer(channel);

        props = channel.CreateBasicProperties();
        var correlationId = Guid.NewGuid().ToString();
        props.CorrelationId = correlationId;
        props.ReplyTo = replyQueueName;

        consumer.Received += (model, ea) =>
        {
            var body = ea.Body;
            var response = Encoding.UTF8.GetString(body);
            if (ea.BasicProperties.CorrelationId == correlationId)
            {
                respQueue.Add(response);
            }
        };
    }

    public string Call(string message)
    {
        var messageBytes = Encoding.UTF8.GetBytes(message);
        channel.BasicPublish(
            exchange: "",
            routingKey: "rpc_queue",
            basicProperties: props,
            body: messageBytes);

        channel.BasicConsume(
            consumer: consumer,
            queue: replyQueueName,
            autoAck: true);

        return respQueue.Take(); ;
    }

    public void Close()
    {
        connection.Close();
    }
}

Here's a screenshot showing me calling the C# function with Postman.

Because I did actually build a Python function, I will go ahead and share how I did it if you're interested. Here are the steps I took:

1. Create a Python HTTP trigger function.
2. Install Python 3.6 via site extensions (see steps 2.1-2.4 here).
3. Install the necessary libraries using pip via KUDU.

Here's the Python function code:

import os
import sys
import json
import pika
import uuid
import datetime

class CrmRpcClient(object):
    def __init__(self):
        #RabbitMQ connection details
        self.rabbituser = 'RABBITUSERNAME'
        self.rabbitpass = 'RABBITUSERPASS'
        self.rabbithost = 'RABBITHOST' 
        self.rabbitport = 5672
        self.rabbitqueue = 'rpc_queue'
        rabbitcredentials = pika.PlainCredentials(self.rabbituser, self.rabbitpass)
        rabbitparameters = pika.ConnectionParameters(host=self.rabbithost,
                                    port=self.rabbitport,
                                    virtual_host='/',
                                    credentials=rabbitcredentials)

        self.rabbitconn = pika.BlockingConnection(rabbitparameters)

        self.channel = self.rabbitconn.channel()

        #create an anonymous exclusive callback queue
        result = self.channel.queue_declare(exclusive=True)
        self.callback_queue = result.method.queue

        self.channel.basic_consume(self.on_response, no_ack=True,
                                   queue=self.callback_queue)

    #callback method for when a response is received - note the check for correlation id
    def on_response(self, ch, method, props, body):
        if self.corr_id == props.correlation_id:
            self.response = body

    #method to make the initial request
    def call(self, n):
        self.response = None
        #generate a new correlation id
        self.corr_id = str(uuid.uuid4())

        #publish the message to the rpc_queue - note the reply_to property is set to the callback queue from above
        self.channel.basic_publish(exchange='',
                                   routing_key=self.rabbitqueue,
                                   properties=pika.BasicProperties(
                                         reply_to = self.callback_queue,
                                         correlation_id = self.corr_id,
                                         ),
                                   body=n)
        while self.response is None:
            self.rabbitconn.process_data_events()
        return self.response

print(" [.] query start time %r" % str(datetime.datetime.now()))
#instantiate an rpc client
crm_rpc = CrmRpcClient()

postreqdata = json.loads(open(os.environ['req']).read())
query = postreqdata['query']

crm_rpc = CrmRpcClient()
print(" [.] query start time %r" % str(datetime.datetime.now()))
queryresponse = crm_rpc.call(query)
print(" [.] query end time %r" % str(datetime.datetime.now()))
response = open(os.environ['res'], 'w')
response.write(queryresponse.decode())
response.close()

Here's a screenshot showing me calling the Python function with Postman.

Note the difference in time between the two functions - 5.62 seconds for Python and .46 seconds for C#!

Security and scalability

If you decide to use this approach in production, I'd suggest you carefully consider both security and scalability. Obviously the overall solution will only be as secure as your RabbitMQ broker and communications between the broker and its clients, so you'll want to look at best practices for access control and securing the communications with TLS. Here are some links for further reading on those subjects:

• TLS - https://www.rabbitmq.com/ssl.html
• Access control - https://www.rabbitmq.com/access-control.html

As for scalability, the approach I've shown creates a separate response queue for each consumer, but it can have problems scaling, especially if you are using a RabbitMQ cluster. You may want to look at the "direct reply-to" approach instead. For an interesting real-world overview of using direct reply-to, take a look at this blog post..

Wrapping up

I hope you've enjoyed this series and that it has given you some ideas about how to implement service relays in your Dynamics 365 CE projects. As I worked through the examples, I certainly learned a few new things, especially when I created my Python service proxy in Azure Functions.

Here are links to all the previous posts in this series.

1. Part 1 - Series introduction
2. Part 2 - Solution prerequisites
3. Part 3 - Python code for the consumer and listener processes

What do you think about this approach? Is it something you think you'd use in production? Let us know in the comments!

Over the past few days, I've shared two approaches for scheduling Dynamics 365 workflows using Azure Functions and the Dynamics 365 Web API. One uses Node.js, and the other uses Python. Because most Dynamics CRM developers are probably more familiar with C# than Node.js or Python, I also created an equivalent C# version. Just like with my previous examples, this version calls the Web API directly instead of using any SDK assemblies.

Here's my code. It does the following:

1. Request an OAuth token using a username and password.
2. Query the Dynamics 365 Web API for accounts with names that start with the letter "F."
3. Execute a workflow for each record that was retrieved in the previous step. The workflow that I am executing is the same workflow I used in my Node.js example to create a note on an account.

#r "Newtonsoft.Json"

using System;
using System.Net;
using System.IO;
using Newtonsoft.Json;

//set these values to retrieve the oauth token
static string crmorg = "https://CRMORG.crm.dynamics.com";
static string clientid = "00000000-0000-0000-0000-000000000000";
static string username = "xxxxxx@xxxxxxxx";
static string userpassword = "xxxxxxxx";
static string tokenendpoint = "https://login.microsoftonline.com/00000000-0000-0000-0000-000000000000/oauth2/token";

//set these values to query your crm data
static string crmwebapihost = "https://CRMORG.api.crm.dynamics.com/api/data/v8.2";
static string crmwebapipath = "/accounts?$select=name,accountid&$filter=startswith(name,'F')";

static string workflowid = "DC8519EC-F3CE-4BC9-BB79-DF2AD70217A1";

public static void Run(TimerInfo myTimer, TraceWriter log)
{
	//build the authorization request
	var reqstring = "client_id=" + clientid;
	reqstring += "&resource=" + Uri.EscapeUriString(crmorg);
	reqstring += "&username=" + Uri.EscapeUriString(username);
	reqstring += "&password=" + Uri.EscapeUriString(userpassword);
	reqstring += "&grant_type=password";

	WebRequest req = WebRequest.Create(tokenendpoint);
	req.ContentType = "application/x-www-form-urlencoded";
	req.Method = "POST";
	byte[] bytes = System.Text.Encoding.ASCII.GetBytes(reqstring);
	req.ContentLength = bytes.Length;
	System.IO.Stream os = req.GetRequestStream();
	os.Write(bytes, 0, bytes.Length);
	os.Close();

	HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
	StreamReader tokenreader = new StreamReader(resp.GetResponseStream());
	string responseBody = tokenreader.ReadToEnd();
	tokenreader.Close();
	var tokenresponse = JsonConvert.DeserializeObject<Newtonsoft.Json.Linq.JObject>(responseBody);
	var token = tokenresponse["access_token"];
	log.Info("got token");

	WebRequest crmreq = WebRequest.Create(crmwebapihost + crmwebapipath);
	crmreq.Headers = new WebHeaderCollection();
	crmreq.Headers.Add("Authorization", "Bearer " + token);
	crmreq.Headers.Add("OData-MaxVersion", "4.0");
	crmreq.Headers.Add("OData-Version", "4.0");
	crmreq.Headers.Add("Prefer", "odata.maxpagesize=500");
	crmreq.Headers.Add("Prefer", "odata.include-annotations=OData.Community.Display.V1.FormattedValue");
	crmreq.ContentType = "application/json; charset=utf-8";
	crmreq.Method = "GET";

	HttpWebResponse crmresp = (HttpWebResponse)crmreq.GetResponse();
	StreamReader crmreader = new StreamReader(crmresp.GetResponseStream());
	string crmresponseBody = crmreader.ReadToEnd();
	crmreader.Close();
	var crmresponseobj = JsonConvert.DeserializeObject<Newtonsoft.Json.Linq.JObject>(crmresponseBody);
	foreach(var row in crmresponseobj["value"].Children())
	{
		log.Info(row["name"].ToString());
		runWorkflow(token.ToString(), new Guid(row["accountid"].ToString()), log);

	}

	Console.ReadLine();
}

static void runWorkflow(string token, Guid entityid, TraceWriter log)
{
	var crmwebapiworkflowpath = "/workflows(" + workflowid + ")/Microsoft.Dynamics.CRM.ExecuteWorkflow";
	WebRequest req = WebRequest.Create(crmwebapihost + crmwebapiworkflowpath);

	log.Info("  calling workflow for " + entityid);

	string reqobject = "{ \"EntityId\": \"" + entityid + "\"}";
    
	req.Headers.Add("Authorization", "Bearer " + token);
	req.Headers.Add("OData-MaxVersion", "4.0");
	req.Headers.Add("OData-Version", "4.0");
	req.Headers.Add("Prefer", "odata.maxpagesize=500");
	req.Headers.Add("Prefer", "odata.include-annotations=OData.Community.Display.V1.FormattedValue");
	req.ContentType = "application/json; charset=utf-8";
    req.Method = "POST";
	
	byte[] bytes = System.Text.Encoding.ASCII.GetBytes(reqobject);
	req.ContentLength = bytes.Length;
	System.IO.Stream os = req.GetRequestStream();
	os.Write(bytes, 0, bytes.Length);
	os.Close();

	HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
	StreamReader reader = new StreamReader(resp.GetResponseStream());
	string responseBody = reader.ReadToEnd();
	reader.Close();
	
    var responseobj = JsonConvert.DeserializeObject<Newtonsoft.Json.Linq.JObject>(responseBody);
	if(resp.StatusCode == HttpStatusCode.OK)
	{
		log.Info("    success " + entityid.ToString());
	}
	else
	{
		log.Info("    error " + entityid.ToString());
	}
}

To set this up in your Azure tenant, set up a Functions App and a new C# timer trigger function like I described in the Node.js example. Copy the C# code from above and paste it into the function editor window. Set any specifics relative to your Dynamics 365 organization and click save. That's all there is to it.

*If you're wondering about that #r "Newtonsoft.Json" at the top of the C# code, take a look here.

About three years ago I released an open source Dynamics CRM solution for scheduling and executing recurring workflows. My solution would execute a FetchXML query to return a set of records and then start a workflow for each of those records without requiring any external processes or tools. This is a generalized approach to solving a class of problems that includes the following scenarios:

1. The birthday greetings problem: How can you, on a daily basis, send an e-mail to every contact with a birthday = today (where the date value for today is obviously different every day)?
2. The monthly update problem: How can you, on a monthly basis, generate an activity for every account with status reason = X (where it's important that the process only runs on a certain day of the month based on status reason values as of that exact date)?

Of all the CRM sample code and solutions I've ever shared, I think this was probably the most popular, but it had one glaring flaw. My original solution would only retrieve a maximum of 5,000 records per run because it didn't include any result paging code. Today I have released an updated version of my solution for CRM 2016 that does include result paging, and I've also moved hosting for the solution source code to GitHub.

How it works

In case you're unfamiliar with the previous version of my solution, my approach requires three components (the names have changed in the updated version):

1. A custom workflow activity (AlexanderDevelopment.WorkflowScheduler) that can execute a supplied FetchXML query and initiate the workflow for each retrieved record.
2. A custom entity (Recurring process) to hold the FetchXML query and scheduling details.
3. A workflow (Recurring workflow runner) to run the AlexanderDevelopment.WorkflowScheduler activity on a recurring schedule.

A "Recurring process" record is created, which starts a corresponding "Recurring workflow runner" workflow in a timeout state. When the next run date == the current time, the "Recurring workflow runner" workflow advances the next run date, and then it initiates the AlexanderDevelopment.WorkflowScheduler activity with the FetchXML query and workflow lookup from the "Recurring process" record. The AlexanderDevelopment.WorkflowScheduler executes the FetchXML, loops through the results and starts the specified workflow from the lookup for each record. A newly started "Recurring workflow runner" workflow then waits for the next run date to start the process again.

Here's what a "Recurring process" record looks like:

Here's the "Recurring workflow runner" workflow:

The default result page size is 1,000, but you change it in "Recurring workflow runner" workflow definition:

Putting it all together

As mentioned above, the source code for the solution is available in a GitHub repository here. You can also download a CRM solution ready to load into your system directly from the repository releases area.

Happy workflow scheduling!

Dynamics CRM offers sophisticated tools for working with cases and service queues, but sometimes users just want a quick and simple way to get the next case to work. In today's post, I'll share an easy way to implement this functionality in your Dynamics CRM organization.

There are three components to my approach:

1. First, there is a custom workflow activity that searches a queue for unassigned cases and assigns the one that entered the queue earliest to a user. This activity returns the assigned case id and case number as output parameters.
2. Second, there is a CRM action that that acts as a wrapper for the custom workflow activity.
3. Finally, there is a web resource that displays a button to call the CRM action and generate a link to the assigned case (or a message if there are no cases in the queue to assign).

Let's look at each of these pieces in more detail. If you'd rather get right to the code, scroll to the bottom of this post for a link.

The custom workflow activity

The custom workflow activity has the following parameters:

1. Queue - input - Reference to the queue that contains the case
2. User - input - Reference to the user who will be assigned the case
3. Assign - input - Flag for whether to assign the case or just return the case details
4. Remove from queue - input - Flag for whether to completely remove the case from the queue when it is assigned
5. Case id - output - Case GUID as a string (string because it makes working with non-existing cases easier in the custom action)
6. Case number - output - Case ticket number value
7. Case found - output - Flag for whether a case was found
8. Case title - output - Case title value

The activity then executes a FetchXML query to retrieve unassigned cases in ascending order of the date they entered the queue. If the "assign" flag is set to true, the first case is assigned to the user, and then the details are returned as output parameters. If the flag is set to false, the case details are returned as output parameters, but the case is not updated.

The CRM action

The action is the simplest part of the solution. It just takes input parameters to pass to the custom activity and then returns the output parameters.

The web resource

Finally the web resource uses JavaScript to call the action and display a link to the assigned case. If no "next" case can be found, then a message is displayed to the user. In my solution, the queue id and assign/remove from queue flags are supplied as encoded query string parameters.

The image below shows the output after the get next button is clicked:

The web resource approach allows you to embed the button in a dashboard or CRM form through an iframe, but you could also trigger the action from a ribbon button using similar JavaScript.

The code

You can download all the custom code and a CRM solution extract from my Crm-Sample-Code repository on GitHub. Happy case management!

This the final post in my five-part series on creating loosely coupled data interfaces for Dynamics CRM using RabbitMQ. In part 3 and part 4 I showed two approaches for building a Dynamics CRM plug-in that publishes notification messages to a RabbitMQ exchange. In today’s post I will show how to create a Windows console application that reads messages from a queue and writes the data to Dynamics CRM. The code for this application is available on GitHub in the LeadWriterSample project under the LucasCrmMessageQueueTools solution.

The approach

This application is extraordinarily simple. On startup it prompts the user to supply connection information for the RabbitMQ queue that it will monitor as well as a Dynamics CRM connection string. It then monitors the queue for new JSON-formatted messages. When new messages arrive, it attempts to deserialize them into a lightweight "leadtype" object, and then it creates new lead records in CRM. Once a message is successfully processed and a lead is created, the application then sends a confirmation back to RabbitMQ so that the message can be removed from the queue.

The following code shows what happens after a connection to the RabbitMQ is established:

//wait for some messages

var consumer = new QueueingBasicConsumer(channel);

channel.BasicConsume(_queue, false, consumer);

 

Console.WriteLine(" [*] Waiting for messages. To exit press CTRL+C");

 

//instantiate crm org service

using (OrganizationService service = new OrganizationService(_targetConn))

{

   while (true)

   {

     //get the message from the queue

     var ea = (BasicDeliverEventArgs)consumer.Queue.Dequeue();

 

     var body = ea.Body;

     var message = Encoding.UTF8.GetString(body);

 

     try

     {

       //deserialize message json to object

       LeadType lead = JsonConvert.DeserializeObject<LeadType>(message);

 

       try

       {

         //create record in crm

         Entity entity = new Entity("lead");

         entity["firstname"] = lead.FirstName;

         entity["lastname"] = lead.LastName;

         entity["subject"] = lead.Topic;

         entity["companyname"] = lead.Company;

         service.Create(entity);

 

         //write success message to cli

         Console.WriteLine("Created lead: {0} {1}", lead.FirstName, lead.LastName);

 

         //IMPORTANT - tell the queue the message was processed successfully so it doesn't get requeued

         channel.BasicAck(ea.DeliveryTag, false);

       }

       catch (FaultException<Microsoft.Xrm.Sdk.OrganizationServiceFault> ex)

       {

         //return error - note no confirmation is sent to the queue, so the message will be requeued

         Console.WriteLine("Could not create lead: {0} {1}", lead.FirstName, lead.LastName);

         Console.WriteLine("Error: {0}", ex.Message);

       }

     }

     catch(Exception ex)

     {

       //return error - note no confirmation is sent to the queue, so the message will be requeued

       Console.WriteLine("Could not process message from queue");

       Console.WriteLine("Error: {0}", ex.Message);

     }

   }

}

If this were to be used in production, I would have created a Windows service instead of a console application, but I wanted to make it easy to try out different connection parameters.

Verifying the application

The queuewriter.js application in the node-app directory in the GitHub repository contains a sample web page that can be used to publish lead data to the CRM-Leads queue. If the application is running, you can access the web page at http://<YOUR_SERVER_NAME>:3000/leadform. When the form’s submit button is clicked, an AJAX call posts a JSON object to the Node.js POST endpoint I showed in my previous post. If the LeadWriterSample console application is running, it will take the message from the queue and you will see a new lead record created in CRM. The screenshots below show each piece working.

The lead has been submitted via the web form, and a success message has been received from the Node.js endpoint.

The lead has landed in the CRM-Leads queue and is ready to be retrieved.

The console application has retrieved and processed the submitted lead message.

The lead record has been created in CRM.

One caveat about the demo lead form is that it has the RabbitMQ credentials embedded in the HTML source, so this code should not be used in production. My approach was originally formulated with the thought that a server-side process would build the JSON message to post to Node.js, so sensitive information would not be exposed. If you decide to use an AJAX post operation like is shown here, you would want to modify the queuewriter.js application to contain the credentials so they do not need to be passed from the end user’s web browser.

Wrapping up

That does it for this series, but I’ve just barely explored the capabilities of RabbitMQ. There’s so much more you can do with it than what I’ve shown here, and I hope I’ve piqued your interest about how you can use RabbitMQ or any other message broker in your Dynamics CRM projects. If you have any questions or want to continue the discussion, please share your thoughts in the comments.

A version of this post was originally published on the HP Enterprise Services Application Services blog.

Welcome back to my five-part series on creating loosely coupled data interfaces for Dynamics CRM using RabbitMQ. In my last post I showed how to build a Dynamics CRM plug-in that publishes notification messages to a RabbitMQ exchange using the official RabbitMQ .Net client library. Unfortunately, that plug-in can’t successfully communicate with a RabbitMQ server if it’s executed inside the Dynamics CRM sandbox, so in today’s post I will show how to achieve the same results with a sandboxed plug-in. The code for this plug-in is available on GitHub in the MessageQueueSandboxPlugin project under the LucasCrmMessageQueueTools solution.

The approach

As I mentioned in my previous post, last month I wrote a series of blog posts about how to create a near real-time streaming API using plug-ins and Node.js. That plug-in worked fine in the Dynamics CRM sandbox, and Node.js can easily publish messages to a RabbitMQ exchange, so today’s plug-in will post a JSON-formatted message to a Node.js application, and then that Node.js application will do the actual publishing to RabbitMQ. As a result, I only need to make a couple of minor modifications to my earlier Node.js message-posting plug-in so that it can pass the RabbitMQ connection parameters to my Node.js application. Additionally, the Node.js application that I described in my earlier series only needs a few changes to publish the message to a RabbitMQ exchange instead of sending it to Socket.IO clients.

The plug-in

The plug-in is registered for an operation (create, update, delete, etc.) with a FetchXML query in its unsecure configuration. When the plug-in step is triggered, its associated FetchXML query is executed, and then the resulting fields are serialized into a JSON object, which is then sent to a Node.js application called queuewriter.js via an HTTP POST request. The JSON object also needs to contain RabbitMQ connection details, so I pass them as part of the plug-in step’s unsecure configuration. Here’s the configuration XML fragment to enable case notifications:

<nodeendpoint>http://lucas-ajax.cloudapp.net:3000/rabbit_post_endpoint</nodeendpoint>
<endpoint>lucas-ajax.cloudapp.net</endpoint>
<exchange>CRM</exchange>
<routingkey>Case</routingkey>
<user>rabbituser</user>
<password>PASSWORDHERE</password>
<query><![CDATA[
<fetch mapping='logical'>
<entity name='incident'>
 <attribute name='ownerid'/>
 <attribute name='modifiedby'/>
 <attribute name='createdby'/>
 <attribute name='title'/>
 <attribute name='incidentid'/>
 <attribute name='ticketnumber'/>
 <attribute name='createdon'/>
 <attribute name='modifiedon'/>
 <filter type='and'>
  <condition attribute='incidentid' operator='eq' value='{0}' />
 </filter>
</entity>
</fetch>
]]>
</query>
</config>

Just like in my earlier Node.js plug-in, the FetchXML is extracted from the configuration XML, and the query is executed against Dynamics CRM. The results are then serialized to JSON using Json.NET just like before, except the serialized CRM data is included as a "message" object that is part of a parent JSON object that includes the RabbitMQ connection parameters. Here’s an example of the structure:

{

   "endpoint":"lucas-ajax.cloudapp.net",

   "username":"rabbituser",

   "password":"XXXXXXXX",

   "exchange":"CRM",

   "routingkey":"Lead",

   "message":{

     "property1":"value 1",

     "property2":"value 2",

     "property3":"value 3"

   }

}

Because this plug-in uses the Json.NET client library, it has to be merged with the plug-in assembly before registering it in Dynamics CRM. I’ve included a batch script called ilmerge.bat in the project directory on GitHub.

The Node.js application

The Node.js application (queuewriter.js) waits to receive JSON messages via HTTP POST from a client. When it receives a POST request, it checks whether the message is valid JSON. If it is, the RabbitMQ connection parameters are extracted and then the notification "message" object is published to the RabbitMQ exchange. If everything is successful, it sends "success" back as a response to the client. If any errors are encountered, it sends back a descriptive error message. I am using the node-amqp library for communicating with the RabbitMQ server, but the behavior isn’t that different from a .Net client. Here’s an extract with the relevant code:

if (request.method == 'POST') {

   request.on('data', function(chunk) {

     //check if received data is valid json

     if(IsJsonString(chunk.toString())){

       //convert message to json object

       var requestobject = JSON.parse(chunk.toString());

      

       //connect to rabbitmq

       var connection = amqp.createConnection({ host: requestobject.endpoint

       , port: 5672 //assumes default port

       , login: requestobject.username

       , password: requestobject.password

       , connectionTimeout: 0

       , authMechanism: 'AMQPLAIN'

       , vhost: '/' //assumes default vhost

       });

      

       //when connection is ready

       connection.on('ready', function () {

          //get the "message" property of the supplied request

          var message = JSON.stringify(requestobject.message);

         

          //post it to the exchange with the supplied routing key

          connection.exchange = connection.exchange(requestobject.exchange, {passive: true, confirm: true }, function(exchange) {

            exchange.publish(requestobject.routingkey, message, {mandatory: true, deliveryMode: 2}, function () {

              //if successful, write message to console

              console.log('Message published: ' + message);

             

              //send "success" back in response

              response.write('success');

             

              //close the rabbitmq connection and end the response

              connection.end();

              response.end();

            });

          });

       });

      

       //if an error occurs with rabbitmq

       connection.on('error', function () {

          //send error message back in response and end it

          response.write('failure writing message to exchange');

          response.end();

       });

     }

     else {

       //if request contains invalid json

       //send error message back in response and end it

       response.write("invalid JSON");

       response.end();

     }

   });

}

The complete queuewriter.js application is contained in the node-app directory in the GitHub repository.

Wrapping up

In addition to registering the plugin and registering a step to publish a notification message to RabbitMQ, you need to deploy and start the queuewriter.js application to publish messages. Once that’s done, you can verify everything is working as expected either by looking at the Queues tab in the RabbitMQ management web UI or running the CliConsumer sample application I showed in part 2.

Obviously using queuewriter.js as message proxy adds an extra layer of complexity, and you have to make sure that the application is up and running in order to process message, but it also offers a couple of advantages. First, by using queuewriter.js instead of a direct connection, you can easily use this same plug-in with different message brokers like Apache ActiveMQ and Microsoft’s Azure Service Bus. Second, the queuewriter.js application isn’t limited to just handling messages outbound from Dynamics CRM. You can also use it to process inbound messages without any changes. You just have to configure a client application to read messages from the queue and process them accordingly. A good example of this would be writing data submitted through a web form to Dynamics CRM via a RabbitMQ queue, and I will show that exact scenario in my next post!

A version of this post was originally published on the HP Enterprise Services Application Services blog.

This is the third post of a five-part series on creating loosely coupled data interfaces for Dynamics CRM using RabbitMQ.
Last time I showed how to install and configure a RabbitMQ server to support passing messages to and from Dynamics CRM. Today I will show how to build a Dynamics CRM plug-in that publishes notification messages to a RabbitMQ exchange using the official RabbitMQ .Net client library. The code for this plug-in is available on GitHub in the MessageQueuePlugin project under the LucasCrmMessageQueueTools solution.

Before going any further, let’s get some bad news out of the way. Plug-ins that execute in the Dynamics CRM sandbox cannot use RabbitMQ .Net client library to publish messages to a RabbitMQ server, so you can’t use today’s plug-in approach from a CRM Online organization. In my next post, I will be showing an alternate mechanism for publishing messages that you can use from a sandboxed plug-in, but today I want to focus on the most direct integration method. Now that we’re clear on the limitations of this approach, let’s get started!

The approach

Last month I wrote a series of blog posts about how to create a near real-time streaming API using plug-ins and Node.js. For this plug-in I’m going to basically copy the logic I used for the plug-in in that series.

This post outlines the approach in detail, but if you don’t want to read the entire thing, the basic idea was to create a plug-in that is registered for an operation (create, update, delete, etc.) with a FetchXML query in its unsecure configuration. When the plug-in step is triggered, its associated FetchXML query is executed, and then the resulting fields are serialized into a JSON object, which is then sent to the Node.js application via an HTTP POST request. Today’s plug-in operates in the exact same way, except instead of sending the JSON object to a Node.js endpoint, the JSON object will be published as a message to a RabbitMQ exchange.

Configuring the plug-in

To make the plug-in easily useable in any organization without needing to be recompiled, all the RabbitMQ connection parameters are stored in the unsecure configuration along with the FetchXML query for the data to retrieve. Here’s the configuration XML fragment to enable case notifications:

<config>
<endpoint>lucas-ajax.cloudapp.net</endpoint>
<exchange>CRM</exchange>
<routingkey>Case</routingkey>
<user>rabbituser</user>
<password>PASSWORDHERE</password>
<query><![CDATA[
<fetch mapping='logical'>
<entity name='incident'>
 <attribute name='ownerid'/>
 <attribute name='modifiedby'/>
 <attribute name='createdby'/>
 <attribute name='title'/>
 <attribute name='incidentid'/>
 <attribute name='ticketnumber'/>
 <attribute name='createdon'/>
 <attribute name='modifiedon'/>
 <filter type='and'>
  <condition attribute='incidentid' operator='eq' value='{0}' />
 </filter>
</entity>
</fetch>
]]>
</query>
</config>

Generating the notification message

Just like in my Node.js plug-in, the FetchXML is extracted from the configuration XML, and the query is executed against Dynamics CRM. The results are then serialized to JSON using Json.NET.

Publishing the message

The endpoint, exchange name, RabbitMQ user, RabbitMQ password and routing key values from the configuration XML are then used to establish a connection to RabbitMQ and publish the notification message to the exchange like so:

try
{
     //connect to rabbitmq
     var factory = new ConnectionFactory();
     factory.UserName = \_brokerUser;
     factory.Password = \_brokerPassword;
     factory.VirtualHost = "/";
     factory.Protocol = Protocols.DefaultProtocol;
     factory.HostName = \_brokerEndpoint;
     factory.Port = AmqpTcpEndpoint.UseDefaultPort;
     IConnection conn = factory.CreateConnection();
     using (var connection = factory.CreateConnection())
     {
         using (var channel = connection.CreateModel())
         {
             //tell rabbitmq to send confirmation when messages are successfully published
             channel.ConfirmSelect();
             channel.WaitForConfirmsOrDie();
            
             //prepare message to write to queue
             var body = Encoding.UTF8.GetBytes(jsonMsg);
 
             var properties = channel.CreateBasicProperties();
             properties.SetPersistent(true);
            
             //publish the message to the exchange with the supplied routing key
             channel.BasicPublish(_exchange, _routingKey, properties, body);
         }
     }
}
catch (Exception e)
{
     tracingService.Trace("Exception: {0}", e.ToString());
     throw;
}

If any errors are encountered, the message is captured via the tracing service, and then an exception is thrown.

Because this plug-in uses both the RabbitMQ .Net and Json.NET client libraries, they have to be merged with the plug-in assembly before registering it in Dynamics CRM. I’ve included a batch script called ilmerge.bat in the project directory on GitHub.

Wrapping up

After you register the plugin and register a step to publish a notification message to RabbitMQ, you can verify everything is working as expected either by looking at the Queues tab in the RabbitMQ management web UI or running the CliConsumer sample application I showed in
part 2.

A version of this post was originally published on the HP Enterprise Services Application Services blog.

Welcome back to this five-part series on creating loosely coupled data interfaces for Dynamics CRM using RabbitMQ. In my last post I discussed why you would want to incorporate a message broker into your Dynamics CRM data interfaces, and today I will show how to install and configure RabbitMQ to support the examples I’ll be presenting in the rest of the series.

Installation

First, you’ll need to download the installation files from here: http://www.rabbitmq.com/download.html. The RabbitMQ server runs on Windows, Linux, UNIX and Mac OS X, and there are installation guides for each supported platform. Because RabbitMQ is written in Erlang, you will need to install an Erlang VM before you can install RabbitMQ, but there is a download link provided in the installation guide. I set up my RabbitMQ server on a Windows 2012 server, and I was up and running in less than 10 minutes.

Once you’ve installed RabbitMQ and started the server, the easiest way to manage it is via the web-based management interface that’s included with the server distribution. You can enable the management interface with the rabbitmq-plugins tool. Run the following command to enable it: rabbitmq-plugins enable rabbitmq_management.

After the management plugin is enabled, you can access the web management UI from your server at http://localhost:15672. The default username is "guest" with "guest" as the password.

You’ll also need to configure any firewall rules necessary to allow access to your RabbitMQ server if it’s running on a server separate from your Dynamics CRM server. The default port is 5672, but that can be changed if you like. This page discusses RabbitMQ configuration in great detail.

Setting up users, queues and exchanges

The first thing you should do after the install is complete is change your default guest user password via the management UI. Then you can add additional users as necessary. For the examples in the rest of this series, you’ll need a user with full permissions on the default "/" virtual host. Here is what my "rabbituser" user account looks like:

Next you need to create the entities required to broker the messages between publishers and consumers. Before continuing, I recommend you take a moment to skim this Advanced Message Queuing Protocol (AMQP) overview document. If nothing else, at least read through the "hello, world" example section because it’s a great introduction to concepts that will be important throughout the rest of this series.

Queues
In the management UI, navigate to the Queues tab, and create two new durable queues named CRM-Cases and CRM-Leads. (You can create any queues you want, but my examples in the rest of this series use queues with those names.) The screenshot below shows the queues in my system.

Exchanges
After your queues are created, you can create an exchange and bindings to your queues so messages get routed correctly. Navigate to the Exchanges tab and create a new, durable exchange named CRM. After your CRM exchange is created, you should see something like the screenshot below.

Next, click on the name of the CRM exchange to open its edit screen. Scroll to the "add binding" section toward the bottom of the page and add a binding to the CRM-Cases queue for a routing key value of "Case" live in the following picture and click "bind."

Do the same for the CRM-Leads queue with "Lead" as the routing key. You should then see the two queues bound to the exchange.

Checking the configuration

At this point you should have everything in place to start publishing and consuming messages. You can verify your configuration works with the CliProvider and CliConsumer sample applications included in my GitHub repository as part of the LucasCrmMessageQueueTools solution.

First, build and run the CliProvider application. You will be prompted to supply basic connection details, and then you can type a message to publish to your RabbitMQ server.

Once the message has been published, you can verify there’s a message waiting in the correct queue on the Queues tab of the management UI.

Next, build and run the CliConsumer application. Once it connects to the CRM-Cases queue, the message will be retrieved and displayed.

When the CliConsumer application processes a message, it sends a confirmation back to queue that triggers removal of the message from the queue. You can check the Queues tab in the management UI to verify that the CRM-Cases queue is empty.

Wrapping up

That’s it for today. Your RabbitMQ server is now fully configured and ready for use with the examples in the rest of this series. Next time I will show how to send messages to a RabbitMQ exchange from a plug-in using the RabbitMQ .Net client library. See you then!

A version of this post was originally published on the HP Enterprise Services Application Services blog.

One of the things I love about Dynamics CRM is how easy it is to create data interfaces to enable integration with other systems. If you’ve worked with Dynamics CRM for any length of time, you’ve probably seen multiple web service integrations that enable interoperability with other line-of-business and legacy systems. A typical pair of inbound and outbound integrations might look like the picture below.

Using a tightly coupled connection between the source and target systems is usually the easiest (thus the quickest and cheapest) way to establish an integration, but this is often a bad idea. Consider the inbound scenario in which an external application is sending data to Dynamics CRM. What happens if the calling application misbehaves and starts sending thousands of requests per second? This has the potential to overload your CRM server and make it completely unusable. Now consider the outbound scenario in which a CRM plug-in calls an external web service. If the destination application’s web service is offline for a few minutes, the update from the CRM plug-in will not get received unless there’s some sort of error handling and retry logic built into the plug-in

An alternate approach

For these reasons, and lots of others (logging, security, scalability, just to name a few), it’s considered a best practice to create loosely coupled integrations that rely on a message broker that sits between the source and destination systems. Though the formal definition is more complicated, for our purposes a message broker can be thought of as a collection of queues that hold messages. Publishers write messages to queues, and then consumers pick up the messages and process them appropriately. Additionally, the message broker can be configured to keep messages in their queues until the consumers provide confirmation of successful processing.

Here’s an example of what the integrations I showed earlier would look like with a message broker.

For the outbound call from the CRM plug-in, the plug-in writes the message to a broker. The message is routed to a queue where it waits to be processed. A separate processing service application retrieves the message from the queue and sends it to the destination application. For the inbound call to CRM, the process works exactly the same, except the source and destination applications are reversed.

Why is a message broker better?

In the inbound call scenario, an effective message broker would typically be expected to handle a larger volume of inbound messages than Dynamics CRM because all it’s doing is receiving and routing the data without any additional processing. The processing service can then process the messages in the queue at a speed that doesn’t overload the Dynamics CRM server. In the case of the outbound call, the combination of a message broker and processing service can enable complex retry logic and custom logging without having to store it in the plugin layer. As an added bonus to either scenario, a message broker can provide a guarantee that messages don’t get lost between the source and destination systems as long as the message is successfully published to the broker.

Where do we go from here?

Over the course of my next four blog posts, I will show how to use RabbitMQ as a message broker in your Dynamics CRM data interfaces. I chose RabbitMQ for this series for several reasons:

1. It’s open source.
2. It runs on multiple platforms.
3. It’s easy to install and configure.
4. It’s fast at processing messages.

If you already have a different message broker in place in your organization or you would like to try a different message broker like Apache ActiveMQ or Microsoft’s Azure Service Bus, most of the approaches and a lot of the code I’m going to show in this series will still be applicable, with the notable exception of the post that discusses how to install and configure RabbitMQ.

Here’s the roadmap for the rest of the series:

• Part 2 – basic installation and configuration of a RabbitMQ
• Part 3 – creating a Dynamics CRM plug-in that publishes messages using the RabbitMQ .Net client library
• Part 4 – creating a sandboxed Dynamics CRM plug-in that publishes messages to RabbitMQ via Node.js
• Part 5 – reading messages from a queue and writing them to Dynamics CRM

If you just can’t wait to dig into the code, I’ve already posted everything to my repository on GitHub, so you can go ahead and take a look.

See you next time!

A version of this post was originally published on the HP Enterprise Services Application Services blog.

When Dynamics CRM 2013 was released, I thought access teams were the new killer feature in that version, and I even developed custom workflow activity code to make managing access team membership easier by using connection records. I have thus far not had an opportunity to use access teams in a real project, so I was disappointed to read this blog post by Ben Hosking (AKA "The Hosk") about how Microsoft doesn't provide any out-of-the-box capabilities for moving access team templates between Dynamics CRM organizations. In that post, the Hosk says, "It’s possible someone could build a console app to import the access team templates but as yet no one has created it." Challenge accepted.

My CRM Access Team Mover tool is available for download here, and the source code is available on GitHub.

To copy/update access team templates from one organization to another using my tool, do the following:

2. Execute the tool from the command line.
3. When prompted to enter the the source connection string, supply a complete Dynamics CRM simplified connection string for the source organization.
4. When prompted to enter the the target connection string, supply a complete Dynamics CRM simplified connection string for the target organization.
5. The tool will attempt to update existing team templates based on their ids. If any source records don't already exist in the target environment, they will be created (with the identical id).
6. Any failures will be reported by the tool. Errors will be encountered if the target schema doesn't match the source schema for the relevant entities.

Here's a screenshot of the tool being executed:

And the image below shows access teams in a source and target system after I ran the tool. The source system has two access team templates for entities that don't exist in the target system, so they were not created.

The approach in detail

Access team templates are just regular Dynamics CRM records, and they can be accessed through the Dynamics CRM organization web service like most other records. (You can see a list of the messages and methods available for 2013 here). Because of this, all I needed to do was query a source CRM organization for access teams, and then loop through the results to recreate each one in the target organization.

At first I tried to retrieve the teamtemplate records using a RetrieveMultiple query for all attributes, but that resulted in a strange service fault involving the issystem attribute that looks like a bug in CRM. I then decided to use a FetchXML query instead. To make it easy on myself, I first make a metadata request to retrieve all the teamtemplate attributes, and then I dynamically build a FetchXML query for everything except the issystem field. That query then gets executed. The code for all the source organization operations is below:

using (OrganizationService service = new OrganizationService(sourceConn))  
{  
 try  
 {  
 //attributes to exclude from the query  
 List<string> IgnoredAttributes = new List<string> { "issystem" };  
  
 Console.WriteLine("Retrieving entity metadata . . .");  
 RetrieveEntityRequest entityreq = new RetrieveEntityRequest  
 {  
 LogicalName = "teamtemplate",  
 EntityFilters = Microsoft.Xrm.Sdk.Metadata.EntityFilters.Attributes  
 };  
 RetrieveEntityResponse entityres = (RetrieveEntityResponse)service.Execute(entityreq);  
 string fetchXml = "<fetch version='1.0' output-format='xml-platform' mapping='logical' distinct='false'>";  
 fetchXml += "<entity name='teamtemplate'>";  
  
 foreach (AttributeMetadata amd in entityres.EntityMetadata.Attributes)  
 {  
 if (!IgnoredAttributes.Contains(amd.LogicalName))  
 {  
 fetchXml += "<attribute name='" + amd.LogicalName + "' />";  
 //Console.WriteLine(amd.LogicalName);  
 }  
 }  
 fetchXml += "</entity></fetch>";  
  
 Console.WriteLine("");  
 Console.WriteLine("Exporting data . . .");  
 exported = service.RetrieveMultiple(new FetchExpression(fetchXml));  
 }  
 catch (FaultException<Microsoft.Xrm.Sdk.OrganizationServiceFault> ex)  
 {  
 Console.WriteLine("Could not export data: {0}", ex.Message);  
 return;  
 }  
}

Once the teamtemplate records have been retrieved, they are then created in the target organization:

using (OrganizationService service = new OrganizationService(targetConn))  
{  
 if (exported.Entities.Count > 0)  
 {  
 foreach (Entity entity in exported.Entities)  
 {  
 try  
 {  
 //try to update first  
 try  
 {  
 service.Update(entity);  
 }  
 catch (FaultException<Microsoft.Xrm.Sdk.OrganizationServiceFault>)  
 {  
 //if update fails, then create  
 service.Create(entity);  
 }  
 }  
 catch (FaultException<Microsoft.Xrm.Sdk.OrganizationServiceFault> ex)  
 {  
 //if everything fails, return error  
 Console.WriteLine("Error: {0} - {1}", entity.Id, entity["teamtemplatename"]);  
 }  
 }  
 }  
}

You'll note this code not only creates new records, but it also tries to update existing records, so if you have a teamtemplate that's changed, this will handle it if the record was created with the same GUID in the target system as in the source system.

What do you think about this approach? Would you have done anything differently?

This is the final post in my four-part series about creating a near real-time streaming interface for Microsoft Dynamics CRM using Node.js and Socket.IO. In my last post I showed how to write the plug-in code to send messages from CRM to the Node.js application. In today’s post I will show how to configure a client to receive and process notifications from the Node.js application, and I’ll also discuss some general considerations related to this solution.

My first post in this series included a video that showed two clients connected to the Node.js application via Socket.IO. One was a web page that displayed notifications using JavaScript, and the other was a simple C# console application. You can find the code for both of the clients from the video in the “client-src” directory in the solution on GitHub.

Web pages

Creating a web page to display notifications received from the Node.js application is incredibly simple. For the web page I demonstrated in the introductory video I first load JavaScript libraries for Socket.IO and JQuery:

<script src="https://cdn.socket.io/socket.io-1.2.0.js"></script>

<script src="http://code.jquery.com/jquery-1.11.1.js"></script>

Then I connect to the Socket.IO endpoint and register a callback function to append the notification text to an element on the page with JQuery:

var socket = io("http://lucas-ajax.cloudapp.net:3000");

socket.on('message', function(msg){

var obj = jQuery.parseJSON( msg );

$('#records').append($('<li>').text(msg));

});

Of course you’re not limited to displaying just raw text. Once you parse the JSON message, it’s a fully-fledged object that you can work with as you like. For example you can create a web page client that lists case updates by displaying the case numbers hyperlinked to the record in Dynamics CRM.

The code for this is only marginally more complicated than the raw stream example above.

var socket = io("http://lucas-ajax.cloudapp.net:3000");

socket.on('message', function(msg){

var obj = jQuery.parseJSON( msg );

if(obj.entity==="incident"){

$('#records').append($('<li>'+obj.operation + ' - <a href="https://lucas-ajax.cloudapp.net/Lucas01/main.aspx?etc=112&pagetype=entityrecord&id='+obj.id+'" target="_blank">'+obj.ticketnumber+'</a>'));

}

});

As before, first the page creates a connection to the Socket.IO endpoint, and then it registers a callback function. This time, however, the callback function includes a check for incident entities, and the append step creates a hyperlinked case number. The code for this example and another one for contact updates is included in the solution source code on GitHub.

Other clients

The C# console application I showed in the introductory video was also incredibly simple to create. First I needed a way to communicate with the Socket.IO endpoint. I used the Socket.IO Client Library for .Net (also available from Nuget -> Install-Package SocketIoClientDotNet). Once I included the library in the project, the code ended up looking a lot like the JavaScript examples above.

var socket = IO.Socket("http://lucas-ajax.cloudapp.net:3000/");

socket.On(Socket.EVENT_CONNECT, () =>

{

socket.On("message", (data) =>

{

Console.WriteLine(data);

//socket.Disconnect();

});

});

In addition to C#, you can create Socket.IO clients in other languages, too. The Socket.IO FAQ has links to client libraries for Java and iOS clients.

Security considerations

As I mentioned in the second post in this series, this solution lacks any mechanism for authenticating or authorizing clients, but I said it was possible. There are two typical approaches to securing Socket.IO interfaces, cookie-based and token-based. This article on "Token-based authentication with Socket.IO" gives a good overview of the problems with the cookie-based approach and goes into some detail about how to implement token-based authentication. In addition to securing the Socket.IO endpoint, you’d also want to consider securing the endpoint where Dynamics CRM posts notifications. You could use the same token-based approach for that, too.

I would also suggest that depending on how you’ve deployed Dynamics CRM and how you need to grant client access, you might not need to worry about security at all. For example, in the case notification example above the only information exposed via the interface is the case number and whether it was created or updated. To see anything else, the end user actually has to open the record, and regular Dynamics CRM security will do the rest.

Wrapping up

I hope you’ve enjoyed this series, and I hope I’ve given you some ideas about how you could implement and use a near real-time streaming API for Dynamics CRM in your own projects. If you have any questions or want to continue the discussion, please share your thoughts in the comments.

A version of this post was originally published on the HP Enterprise Services Application Services blog.