Every year my family hangs a large decorative spider over our doorway for Halloween. This year I decided to make it flash red LED eyes and play a random assortment of spooky sounds (including, but not limited to, bats, chains and the Vincent Price laugh from "Thriller") when trick-or-treaters come to the door. In today's blog post, I'll show how I did it.

The approach

A few years ago, I wrote a post about setting up a motion-activated webcam to trigger license plate recognition with a Raspberry Pi, so I already had a Raspberry Pi and an HC-SR04 ultrasonic rangefinder I could use to detect motion. I initially thought I would only need the Pi for this project, but as I started evaluating the physical constraints of my porch, I realized it would be difficult to mount the Pi in a secure location where it could handle motion detection and also flash the LED eyes. I decided a better approach would be to use an Arduino to detect motion and flash the LEDs, while the Pi would communicate with the Arduino over a serial connection and play Halloween sound effect MP3s.

Setting up the Arduino

I have two red LEDs taped over the spider's eyes that are connected in series to pin 12 on an Arduino UNO. The ultrasonic rangefinder is taped to the side of my porch so that trick-or-treaters have to pass by it before they can knock on my door, and it is connected to pin 9 for the trigger and pin 10 for the echo. The Arduino listens for serial commands from the Pi to measure distance or to flash the LEDs if the Pi determines the reported distance measurement means someone has approached.

Here's my sketch:

//set the pin numbers
const int triggerPin = 9;
const int echoPin = 10;
const int ledPin = 12;

void setup() 
{
  Serial.begin(9600); // Starts the serial communication
  pinMode(triggerPin, OUTPUT); // Sets the triggerPin as an Output
  pinMode(echoPin, INPUT); // Sets the echoPin as an Input
  pinMode(ledPin, OUTPUT);
}

void loop() 
{
  if (Serial.available() > 0) { 
    int controlcode = Serial.parseInt();
    if(controlcode==1)
    {
      //clear the trigger pin
      digitalWrite(triggerPin, LOW);
      delayMicroseconds(2);

      //send a 10 microsecond pulse
      digitalWrite(triggerPin, HIGH);
      delayMicroseconds(10);
      digitalWrite(triggerPin, LOW);

      //read the echo pin to get the duration in microseconds
      long duration = pulseIn(echoPin, HIGH);
      
      //calculate the distance in centimeters
      int distance = duration*0.034/2;
      
      //return the distance to the serial monitor
      Serial.println(distance);
    }
    if(controlcode==2)
    {
      flashEyes();
    }
  }
}

void flashEyes()
{
  //flash 50 times
  for(int i=0;i<50;i++)
  {
    //turn the eyes on
    digitalWrite(ledPin, HIGH);
    
    //wait for 150ms
    delay(150);
    
    //turn the eyes off
    digitalWrite(ledPin, LOW);
    
    //wait for 100ms
    delay(100);
  }
}

Setting up the Raspberry Pi

My Raspberry Pi 2 Model B is connected to the Arduino over a standard USB A/B cable, and it's also connected to a Bluetooth speaker mounted behind the spider. To run things, I have a Python script that sends a serial command to the Arduino to measure the distance every 50 milliseconds. If the Arduino returns a value between 20 and 120 centimeters, the Python script sends a serial command to the Arduino to flash the LEDs, and uses mpg123 to play a random MP3 file.

Here's the script:

import serial
import time
import subprocess
import random
port = "/dev/ttyACM0"

def playsound():
    i = 0
    screamnumber = str(random.randint(1,5))
    screamfile = "/home/pi/halloween/X.mp3".replace("X",screamnumber) 
    subprocess.Popen(["mpg123", screamfile])
 
if __name__ == '__main__':
    s1 = serial.Serial(
      port=port,\
      baudrate=9600)
    s1.flushInput()
    
    #wait 5 seconds before telling the arduino look for motion
    time.sleep(5)
    try:
        while True:
          if s1.in_waiting==0:
            #print('sent request')
            s1.write('1\n')
            time.sleep(.05)
          if s1.in_waiting>0:
            #print('received response')
            inputValue = s1.readline()
            #print('Distance: ' + inputValue)
            if int(inputValue) > 20 and int(inputValue) < 120:
              print('Distance: ' + inputValue)
              s1.write('2\n')
              playsound()
              time.sleep(15)
            else:
              time.sleep(.05)
 
    #quit with ctrl + c
    except KeyboardInterrupt:
        print("script stopped by user")

To enable serial communication, I am using the pySerial module. I should also note the logic for playing a random MP3 assumes that there are five files in the same directory as the script that are named 1.mp3, 2.mp3, etc. Using a different number of files would require changing the upper bound value on line 9.

Happy Halloween!

One of the biggest trends in systems architecture these days is the use of "serverless" functions like Azure Functions, Amazon Lambda and OpenFaas. Because these functions are stateless, if you want to use a purely serverless approach to work with resources secured using Azure Active Directory like Dynamics 365 online, a new token will have to be requested every time a function executes. This is inefficient, and it requires the function to fully understand OAuth 2 authentication, which could be handled better elsewhere.

To address this problem, I've written a microservice in Python that can be used to request OAuth 2 tokens from Azure Active Directory, and it also handles refreshing them as needed. I've containerized it as Docker image so you can easily run it without needing to build anything.

How it works

When a request containing a username and password arrives for the first time, the microservice retrieves an OAuth2 access token from Azure AD and returns it to the requester. The microservice also caches an object that contains the access token, refresh token, username, password and expiration time.

When subsequent requests arrive, the microservice checks its cache for an existing token that matches the username and password. If it finds one, it checks if the token has expired or needs to be refreshed.

If the existing token has expired, a new one is requested. If the existing token has not expired, but it will expire within a specified period of time (10 minutes is the default value), the microservice will execute a refresh request to Azure AD, cache the updated token and return it to the requester. If there's an unexpired existing token that doesn't need to be refreshed, the cached access token will be returned to the requester.

Here's what a raw token request to and response from the microservice looks like in Postman:

Back in 2016 I shared some sample Python code that showed how to authenticate to Azure AD and query the Dynamics 365 (then called Dynamics CRM) Web API. Here is an updated version of that sample code that uses this new microservice to acquire access tokens:

import requests
import json

#set these values to retrieve the oauth token
username = 'lucasalexander@xxxxxx.onmicrosoft.com'
userpassword = 'xxxxxx'
tokenendpoint = 'http://localhost:5000/requesttoken'

#set these values to query your crm data
crmwebapi = 'https://xxxxxx.api.crm.dynamics.com/api/data/v8.2'
crmwebapiquery = '/contacts?$select=fullname,contactid'

#build the authorization request
tokenpost = {
    'username':username,
    'password':userpassword
}

#make the token request
print('requesting token . . .')
tokenres = requests.post(tokenendpoint, json=tokenpost)
print('token response received. . .')

accesstoken = ''

#extract the access token
try:
    print('parsing token response . . .')
    print(tokenres)
    accesstoken = tokenres.json()['accesstoken']

except(KeyError):
    print('Could not get access token')

if(accesstoken!=''):
    crmrequestheaders = {
        'Authorization': 'Bearer ' + accesstoken,
        'OData-MaxVersion': '4.0',
        'OData-Version': '4.0',
        'Accept': 'application/json',
        'Content-Type': 'application/json; charset=utf-8',
        'Prefer': 'odata.maxpagesize=500',
        'Prefer': 'odata.include-annotations=OData.Community.Display.V1.FormattedValue'
    }

    print('making crm request . . .')
    crmres = requests.get(crmwebapi+crmwebapiquery, headers=crmrequestheaders)
    print('crm response received . . .')
    try:
        print('parsing crm response . . .')
        crmresults = crmres.json()
        for x in crmresults['value']:
            print (x['fullname'] + ' - ' + x['contactid'])
    except KeyError:
        print('Could not parse CRM results')

Here's the output when I run the sample against my demo environment:

Running the microservice

Pull the image from Docker Hub: docker pull lucasalexander/azuread-oauth2-helper:latest

Required environment variables

1. RESOURCE - The URL of the service that is going to be accessed
2. CLIENTID - The Azure AD application client ID
3. TOKEN_ENDPOINT - The OAuth2 token endpoint from the Azure AD application

Run the image with the following command (replacing the environment variables with your own).

docker run -d -p 5000:5000 -e RESOURCE=https://XXXXXX.crm.dynamics.com -e CLIENTID=XXXXXX -e TOKEN_ENDPOINT=https://login.microsoftonline.com/XXXXXX/oauth2/token --name oauthhelper lucasalexander/azuread-oauth2-helper:latest

You can also optionally supply an additional "REFRESH_THRESHOLD" environment variable that sets the time remaining (in seconds) before a token's expiration time when it will be refreshed. The default value is 600 seconds.

A note on security

Because the microservice is caching usernames, passwords and access tokens in memory, this approach is vulnerable to heap inspection attacks, so you'll want to make sure your environment is appropriately locked down. Also you'll want to make sure any communication between your code that requests tokens and the microservice is encrypted.

Wrapping up

Although I wrote this with Dynamics 365 in mind, it should work for any resource that is secured by Azure AD. If you'd like to take a closer look at the code, it's available on GitHub here.

This is the final post in my series about building a service relay for Dynamics 365 CE with RabbitMQ and Python. In my previous post in this series, I showed the Python code to make the service relay work. In today's post, I will show how you can use Azure Functions to make a consumer service proxy using C# so client applications don't have to access to your RabbitMQ broker directly, and I will also discuss some general thoughts on security and scalability for this service relay architecture.

Although this simple service relay allows external consumers to get data from Dynamics 365 CE without needing to connect directly, the examples I've shown so far require that they can connect to a RabbitMQ broker. This may be problematic for a variety of reasons, so you would probably want external consumers to connect to a web service proxy that would write requests to and read responses from the RabbitMQ broker.

Building a service proxy function

You can build an Azure Functions service proxy with Python, but I don't recommend it for three reasons:

1. Azure Functions Python support is still considered experimental.
2. Python scripts that use external libraries can run exceedingly slow.
3. Getting the environment set up is a bit of a hassle.

On the other hand, building a service proxy function with C# was so much easier, and it performed much better than a comparable Python function (~.5 seconds for C# compared to 5+ seconds for Python).

Here are the steps I took to build my C# service proxy function:

1. Create a C# HTTP trigger function.
2. Create and upload a project.json file with a dependency on the RabbitMQ client (see below).
3. Take the "RpcClient" class from the RabbitMQ .Net RPC tutorial and call it from within my function.

Here's my project.json file:

{
  "frameworks": {
    "net46":{
      "dependencies": {
        "RabbitMQ.Client": "5.0.1"
      }
    }
   }
}

And here's my run.csx file:

using System.Net;
using System;
using System.Collections.Concurrent;
using System.Text;
using RabbitMQ.Client;
using RabbitMQ.Client.Events;

public static async Task<HttpResponseMessage> Run(HttpRequestMessage req, TraceWriter log)
{
    log.Info("Processing request");

    // parse query parameter
    string query = req.GetQueryNameValuePairs()
        .FirstOrDefault(q => string.Compare(q.Key, "query", true) == 0)
        .Value;

    // Get request body
    dynamic data = await req.Content.ReadAsAsync<object>();

    // Set name to query string or body data
    query = query ?? data?.query;

    var rpcClient = new RpcClient();
    
    log.Info(string.Format(" [.] query start time {0}", DateTime.Now.ToString("MM/dd/yyyy hh:mm:ss.fff tt")));
    var response = rpcClient.Call(query);

    log.Info(string.Format(" [.] query end time {0}", DateTime.Now.ToString("MM/dd/yyyy hh:mm:ss.fff tt")));
    rpcClient.Close();

    return req.CreateResponse(HttpStatusCode.OK, response);
}

public class RpcClient
{
    private readonly IConnection connection;
    private readonly IModel channel;
    private readonly string replyQueueName;
    private readonly EventingBasicConsumer consumer;
    private readonly BlockingCollection<string> respQueue = new BlockingCollection<string>();
    private readonly IBasicProperties props;

    public RpcClient()
    {
        var factory = new ConnectionFactory() { HostName = "RABBITHOST", UserName="RABBITUSER", Password="RABBITUSERPASS"  };

        connection = factory.CreateConnection();
        channel = connection.CreateModel();
        replyQueueName = channel.QueueDeclare().QueueName;
        consumer = new EventingBasicConsumer(channel);

        props = channel.CreateBasicProperties();
        var correlationId = Guid.NewGuid().ToString();
        props.CorrelationId = correlationId;
        props.ReplyTo = replyQueueName;

        consumer.Received += (model, ea) =>
        {
            var body = ea.Body;
            var response = Encoding.UTF8.GetString(body);
            if (ea.BasicProperties.CorrelationId == correlationId)
            {
                respQueue.Add(response);
            }
        };
    }

    public string Call(string message)
    {
        var messageBytes = Encoding.UTF8.GetBytes(message);
        channel.BasicPublish(
            exchange: "",
            routingKey: "rpc_queue",
            basicProperties: props,
            body: messageBytes);

        channel.BasicConsume(
            consumer: consumer,
            queue: replyQueueName,
            autoAck: true);

        return respQueue.Take(); ;
    }

    public void Close()
    {
        connection.Close();
    }
}

Here's a screenshot showing me calling the C# function with Postman.

Because I did actually build a Python function, I will go ahead and share how I did it if you're interested. Here are the steps I took:

1. Create a Python HTTP trigger function.
2. Install Python 3.6 via site extensions (see steps 2.1-2.4 here).
3. Install the necessary libraries using pip via KUDU.

Here's the Python function code:

import os
import sys
import json
import pika
import uuid
import datetime

class CrmRpcClient(object):
    def __init__(self):
        #RabbitMQ connection details
        self.rabbituser = 'RABBITUSERNAME'
        self.rabbitpass = 'RABBITUSERPASS'
        self.rabbithost = 'RABBITHOST' 
        self.rabbitport = 5672
        self.rabbitqueue = 'rpc_queue'
        rabbitcredentials = pika.PlainCredentials(self.rabbituser, self.rabbitpass)
        rabbitparameters = pika.ConnectionParameters(host=self.rabbithost,
                                    port=self.rabbitport,
                                    virtual_host='/',
                                    credentials=rabbitcredentials)

        self.rabbitconn = pika.BlockingConnection(rabbitparameters)

        self.channel = self.rabbitconn.channel()

        #create an anonymous exclusive callback queue
        result = self.channel.queue_declare(exclusive=True)
        self.callback_queue = result.method.queue

        self.channel.basic_consume(self.on_response, no_ack=True,
                                   queue=self.callback_queue)

    #callback method for when a response is received - note the check for correlation id
    def on_response(self, ch, method, props, body):
        if self.corr_id == props.correlation_id:
            self.response = body

    #method to make the initial request
    def call(self, n):
        self.response = None
        #generate a new correlation id
        self.corr_id = str(uuid.uuid4())

        #publish the message to the rpc_queue - note the reply_to property is set to the callback queue from above
        self.channel.basic_publish(exchange='',
                                   routing_key=self.rabbitqueue,
                                   properties=pika.BasicProperties(
                                         reply_to = self.callback_queue,
                                         correlation_id = self.corr_id,
                                         ),
                                   body=n)
        while self.response is None:
            self.rabbitconn.process_data_events()
        return self.response

print(" [.] query start time %r" % str(datetime.datetime.now()))
#instantiate an rpc client
crm_rpc = CrmRpcClient()

postreqdata = json.loads(open(os.environ['req']).read())
query = postreqdata['query']

crm_rpc = CrmRpcClient()
print(" [.] query start time %r" % str(datetime.datetime.now()))
queryresponse = crm_rpc.call(query)
print(" [.] query end time %r" % str(datetime.datetime.now()))
response = open(os.environ['res'], 'w')
response.write(queryresponse.decode())
response.close()

Here's a screenshot showing me calling the Python function with Postman.

Note the difference in time between the two functions - 5.62 seconds for Python and .46 seconds for C#!

Security and scalability

If you decide to use this approach in production, I'd suggest you carefully consider both security and scalability. Obviously the overall solution will only be as secure as your RabbitMQ broker and communications between the broker and its clients, so you'll want to look at best practices for access control and securing the communications with TLS. Here are some links for further reading on those subjects:

• TLS - https://www.rabbitmq.com/ssl.html
• Access control - https://www.rabbitmq.com/access-control.html

As for scalability, the approach I've shown creates a separate response queue for each consumer, but it can have problems scaling, especially if you are using a RabbitMQ cluster. You may want to look at the "direct reply-to" approach instead. For an interesting real-world overview of using direct reply-to, take a look at this blog post..

Wrapping up

I hope you've enjoyed this series and that it has given you some ideas about how to implement service relays in your Dynamics 365 CE projects. As I worked through the examples, I certainly learned a few new things, especially when I created my Python service proxy in Azure Functions.

Here are links to all the previous posts in this series.

1. Part 1 - Series introduction
2. Part 2 - Solution prerequisites
3. Part 3 - Python code for the consumer and listener processes

What do you think about this approach? Is it something you think you'd use in production? Let us know in the comments!

In my last post in this series, I walked through the prerequisites for building a simple service relay for Dynamics 365 CE with RabbitMQ and Python. In today's post I will show the Python code to make the service relay work.

As I described in the first post in this series, this approach relies on a consumer process and a queue listener process that can both access a RabbitMQ message broker.

A consumer writes a request to a cloud-hosted RabbitMQ request queue (either directly or through a proxy service) and starts waiting for a response. On the other end, a Python script monitors the request queue for inbound requests. When it sees a new one, it executes the appropriate request through the Dynamics 365 Web API and writes the response back to a client-specific RabbitMQ response queue. The consumer then picks up the response from the queue.

This solution is based on the remote procedure call (RPC) approach shown here. The main difference is that I have added logic to the queue monitoring script to query the Dynamics 365 Web API based on the inbound request from the consumer.

Consumer sample

The consumer does the following:

1. Read the text of the request to write to the queue from a command-line argument.
2. Establish a connection to the RabbitMQ broker.
3. Create a new anonymous, exclusive callback queue.
4. Write a request message a queue called "rpc_queue." This message will include the callback queue as its "reply_to" property.
5. Monitor the callback queue for a response.

There's no validation in this sample, so if you run it without a command-line argument, it will just throw an error and exit.

import sys
import pika
import uuid
import datetime

class CrmRpcClient(object):
    def __init__(self):
        #RabbitMQ connection details
        self.rabbituser = 'crmuser'
        self.rabbitpass = 'crmpass'
        self.rabbithost = '127.0.0.1' 
        self.rabbitport = 5672
        self.rabbitqueue = 'rpc_queue'
        rabbitcredentials = pika.PlainCredentials(self.rabbituser, self.rabbitpass)
        rabbitparameters = pika.ConnectionParameters(host=self.rabbithost,
                                    port=self.rabbitport,
                                    virtual_host='/',
                                    credentials=rabbitcredentials)

                self.rabbitconn = pika.BlockingConnection(rabbitparameters)

        self.channel = self.rabbitconn.channel()

        #create an anonymous exclusive callback queue
        result = self.channel.queue_declare(exclusive=True)
        self.callback_queue = result.method.queue

        self.channel.basic_consume(self.on_response, no_ack=True,
                                   queue=self.callback_queue)

    #callback method for when a response is received - note the check for correlation id
    def on_response(self, ch, method, props, body):
        if self.corr_id == props.correlation_id:
            self.response = body

    #method to make the initial request
    def call(self, n):
        self.response = None
        #generate a new correlation id
        self.corr_id = str(uuid.uuid4())

        #publish the message to the rpc_queue - note the reply_to property is set to the callback queue from above
        self.channel.basic_publish(exchange='',
                                   routing_key=self.rabbitqueue,
                                   properties=pika.BasicProperties(
                                         reply_to = self.callback_queue,
                                         correlation_id = self.corr_id,
                                         ),
                                   body=n)
        while self.response is None:
            self.rabbitconn.process_data_events()
        return self.response

#instantiate an rpc client
crm_rpc = CrmRpcClient()

#read the request from the command line
request = sys.argv[1]

#make the request and get the response
print(" [x] Requesting crm data("+request+")")
print(" [.] Start time %s" % str(datetime.datetime.now()))
response = crm_rpc.call(request)

#convert the response message body from the queue to a string 
decoderesponse = response.decode()

#print the output
print(" [.] Received response: %s" % decoderesponse)
print(" [.] End time %s" % str(datetime.datetime.now()))

Queue listener sample

The queue listener does the following:

1. Establish a connection to the RabbitMQ broker
2. Monitor "rpc_queue" queue.
3. When a new message from the "rpc_queue" queue is delivered, decode the message body as a string, and determine what Web API query to execute. Note: This sample can return a list of contacts or accounts from Dynamics 365 CE based on the request the consumer sends ("getcontacts" or "getaccounts"). If any other request is received, the listener will return an error message to the consumer callback queue.
4. Execute the appropriate query against the Dynamics 365 Web API and write the response to the callback queue the client established originally.

import pika
import requests
from requests_ntlm import HttpNtlmAuth
import json

#NTLM credentials to access on-prem Dynamics 365 Web API
username = 'DOMAIN\\USERNAME'
userpassword = 'PASSWORD'

#full path to Web API
crmwebapi = 'http://33.0.0.16/lucastest02/api/data/v8.1'

#RabbitMQ connection details
rabbituser = 'crmuser'
rabbitpass = 'crmpass'
rabbithost = '127.0.0.1' 
rabbitport = 5672

#method to execute a Web API query based on the client request
def processquery(query):
    #set the Web API request headers
    crmrequestheaders = {
        'OData-MaxVersion': '4.0',
        'OData-Version': '4.0',
        'Accept': 'application/json',
        'Content-Type': 'application/json; charset=utf-8',
        'Prefer': 'odata.maxpagesize=500',
        'Prefer': 'odata.include-annotations=OData.Community.Display.V1.FormattedValue'
    }

    #determine which Web API query to execute
    if query == 'getcontacts':
        crmwebapiquery = '/contacts?$select=fullname,contactid'
    elif query == 'getaccounts':
        crmwebapiquery = '/accounts?$select=name,accountid'
    else:
        #only handle 'getcontacts' or 'getaccounts' requests
        return 'Operation not supported'

    #execute the query
    crmres = requests.get(crmwebapi+crmwebapiquery, headers=crmrequestheaders,auth=HttpNtlmAuth(username,userpassword))
    
    #get the results json
    crmjson = crmres.json()

    #return the json
    return crmjson

#method to handle new inbound requests
def on_request(ch, method, props, body):
    #convert the message body from the queue to a string
    decodebody = body.decode('utf-8')

    #print the request
    print(" [.] Received request: '%s'" % decodebody)

    #process the request query
    response = processquery(decodebody)

    #publish the response back to 'reply-to' queue from the request message and set the correlation id
    ch.basic_publish(exchange='',
                     routing_key=props.reply_to,
                     properties=pika.BasicProperties(correlation_id = \
                                                         props.correlation_id),
                     body=str(response).encode(encoding="utf-8", errors="strict"))
    ch.basic_ack(delivery_tag = method.delivery_tag)

print(" [x] Awaiting RPC requests")

#connect to RabbitMQ broker
rabbitcredentials = pika.PlainCredentials(rabbituser, rabbitpass)
rabbitparameters = pika.ConnectionParameters(host=rabbithost,
                               port=rabbitport,
                               virtual_host='/',
                               credentials=rabbitcredentials)
rabbitconn = pika.BlockingConnection(rabbitparameters)
channel = rabbitconn.channel()

#declare the 'rpc_queue' queue
channel.queue_declare(queue='rpc_queue')

#set qos settings for the channel
channel.basic_qos(prefetch_count=1)

#assign the 'on_request' method as a callback for when new messages delivered from the 'rpc_queue' queue
channel.basic_consume(on_request, queue='rpc_queue')

#start listening for requests
channel.start_consuming()

Trying it out

As I mentioned in my last post, I initially wrote my code to use a RabbitMQ broker running on my local PC, so that's why the connections in the samples show 127.0.0.1 as the host. For a demo, I've spun up a copy of RabbitMQ in a Docker container in the cloud and updated my connection parameters accordingly, but I am still running my queue listener and consumer processes on my local PC.

When the listener first starts, it displays a simple status message.

Then I execute a "getcontacts" request from the consumer in a separate window.

From the timestamps before and after the request, you can see the round-trip time is less than .2 seconds, which includes two round trips between my local PC and the cloud-based RabbitMQ broker plus the actual query processing time in my local Dynamics 365 CE org.

Then I execute a "getaccounts" request.

This request was also fulfilled in less than .2 seconds.

Finally I execute an invalid request to show what the error response looks like.

You'll note the total time from request to response is only about .05 seconds less than the total time for the valid queries. That indicates most of the time used in these samples is being spent on the round trips between my local PC and the RabbitMQ broker, which is not surprising.

Meanwhile, the queue listener wrote a simple status update for every request it received. If I were using this in production, I would use more sophisticated logging.

Wrapping up

That's it for now. In my next (and final) post in this series, I will show how you can use Azure Functions to make a consumer service proxy so consuming applications don't have to access to your RabbitMQ broker directly, and I will also discuss some general thoughts on security and scalability for the service .

In my last post in this series, I outlined an approach for building a simple service relay with RabbitMQ and Python to easily expose an on-premises Dynamics 365 Customer Engagement organization to external consumers. In this post I will walk through the prerequisites for building this out. I'm assuming you have access to a Dynamics 365 CE organization, so I'm going to skip the setup for that and focus on just RabbitMQ and Python today.

Setting up RabbitMQ

Back in 2015 When I first blogged about RabbitMQ and Dynamics 365, I wrote a detailed post showing how to install and configure RabbitMQ. Since then I have discovered the joys of Docker, which makes the process significantly easier. If you have access to Docker, I highly recommend using it. Once you have Docker running, you can use one of the official RabbitMQ images. For this project, I initially used the rabbitmq:3-management image in Docker for Windows running on my local PC. After I got the basic functionality working, I then moved to an instance of Docker running in the cloud on a Digital Ocean VPS.

If don't want to use Docker, you can use a full RabbitMQ install like I showed previously. The main thing to remember is that no matter how you set up your RabbitMQ server, if it is not accessible from the public internet, you will not be able to use it as a service relay between an on-premises Dynamics 365 org and external consumers.

Setting up Python

I'm assuming if you've gotten this far, you have a functional Python development environment (if not, give Visual Studio Code a try), and the code I have written works in Python versions 2.7 or 3.x. In order to connect to both RabbitMQ and Dynamics 365, you will need a few additional packages. To connect to RabbitMQ, Pika is the RabbitMQ team's recommended Python client, and you can get it using pip.

To communicate with Dynamics 365, you'll need to use the Web API, but authentication will be handled differently depending on whether you connect to an on-premises org or an online / IFD org. For online or IFD orgs, you can either use ADAL or this alternate approach I described back in 2016. If you have an on-premises org, you can authenticate using the requests_ntlm package like I showed here. As with the Pika client, all the packages you need to connect to Dynamics 365 are also available via pip.

Wrapping up

That's it for today. In my next post in this series I will show the Python code you need to make this service relay work.

Integrating with external systems is a common requirement in Dynamics 365 Customer Engagement projects, but when the project involves an on-premises instance of Dynamics 365, routing requests from external systems through your firewall can present an additional challenge. Over the course of the next few posts, I will show you can easily build a simple service relay with RabbitMQ and Python to handle inbound requests from external data interface consumers.

Here's how my approach works. A consumer writes a request to a cloud-hosted RabbitMQ request queue (either directly or through a proxy service) and starts waiting for a response. On the other end, a Python script monitors the request queue for inbound requests. When it sees a new one, it executes the appropriate request through the Dynamics 365 Web API and writes the response back to a client-specific RabbitMQ response queue. The consumer then picks up the response from the queue. This way the consumer doesn't need to know anything other than how to write the initial request, and no extra inbound firewall ports need to be opened.

This diagram shows an overview of the process.

Although my original goal was to accelerate the deployment of data interfaces for on-premises Dynamics 365 CE instances, a simple service relay like this could also be useful for IFD or Dynamics 365 online deployments if you don't want to allow direct access to your organization. Because the queue monitoring process is single-threaded, it's an easy way to throttle requests, but you can run multiple instances of the queue monitor script if you want to increase the number of concurrent requests the relay can process.

Why use this approach?

There are lots message brokers and service bus offerings (Azure Service Bus, IBM MQ, Amazon SQS, etc.) you could use to build a service relay. In fact there's even an Azure offering called Azure Relay that aims to solve exactly the same problem that my approach does, but not just for Dynamics 365, so "why use this?" is a great question.

First, I think RabbitMQ is just a great tool, and I previously wrote a five-part series about using RabbitMQ with Dynamics 365 (back when it was still called Dynamics CRM). Second, using RabbitMQ instead of a cloud-specific service bus offering gives you maximum flexibility in where you host your request and response queues and how you chose to scale. For example, my RabbitMQ broker runs in a Docker container on a Digital Ocean VPS. If I ever decide to move off of Digital Ocean, I can easily switch to any IaaS or VPS provider. I can also configure a RabbitMQ cluster to achieve significantly faster throughput.

As for why I'm using Python instead of C#, which is probably more familiar to most Dynamics 365 developers, Python also makes this approach more flexible. Using Python means I'm not tied to the Dynamics 365 SDK client libraries or a Windows host for running my queue monitoring process, and I can easily package my monitoring process in a Docker image. (Although I highly recommend Python, there are RabbitMQ clients for .Net, and you can also find RabbitMQ tutorials for other languages including Java, Ruby and JavaScript here.)

Wrapping up

That's it for now. In my next post in this series I will walk through the prerequisites for building the simple service relay.

How have you handled inbound data interfaces for on-premises Dynamics 365 CE organizations? Let us know in the comments!

I've previously showed how to access online and IFD instances of Dynamics 365 Customer Engagement from Python code. Because that sample code authenticated to the Web API using OAuth, it won't work with on-premises instances. Recently I've been doing some work with Python and an on-premises Dynamics 365 organization, so I thought I'd share a sample that shows how to authenticate to the Web API using NTLM.

import requests
from requests_ntlm import HttpNtlmAuth
import json

username = 'companyx\\administrator'
userpassword = 'PASSWORD GOES HERE'

#set these values to query your crm data
crmwebapi = 'http://33.0.0.16/lucastest02/api/data/v8.1'
crmwebapiquery = '/contacts?$select=fullname,contactid'

crmrequestheaders = {
    'OData-MaxVersion': '4.0',
    'OData-Version': '4.0',
    'Accept': 'application/json',
    'Content-Type': 'application/json; charset=utf-8',
    'Prefer': 'odata.maxpagesize=500',
    'Prefer': 'odata.include-annotations=OData.Community.Display.V1.FormattedValue'
}

print('making crm request . . .')
crmres = requests.get(crmwebapi+crmwebapiquery, headers=crmrequestheaders,auth=HttpNtlmAuth(username,userpassword))
print('crm response received . . .')
try:
    print('parsing crm response . . .')
    crmresults = crmres.json()
    for x in crmresults['value']:
        print (x['fullname'] + ' - ' + x['contactid'])
except KeyError:
    print('Could not parse CRM results')

As you can see, this code doesn't retrieve an OAuth token before calling the Dynamics 365 Web API, but rather it uses the requests-ntlm package to authenticate directly to the Web API using a username and password. Other than that small change, everything else works the same as in my previous examples.

Last week I shared a solution for Scheduling Dynamics 365 workflows with Azure Functions and Node.js. In this post, I will show how to achieve equivalent functionality using Python. The actual Python code is simpler than my Node.js example, but the Azure Functions configuration is much more complicated.

First, here's the Python script I am using. It does this following:

1. Request an OAuth token using a username and password.
2. Query the Dynamics 365 Web API for accounts with names that start with the letter "F."
3. Execute a workflow for each record that was retrieved in the previous step. The workflow that I am executing is the same workflow I used in my Node.js example to create a note on an account.

import os
import sys
cwd = os.getcwd()
sitepackage = cwd + "\site-packages"
sys.path.append(sitepackage)

import requests
import json

#set these values to retrieve the oauth token
crmorg = 'https://CRMORG.crm.dynamics.com' #base url for crm org  
clientid = '00000000-0000-0000-0000-000000000000' #application client id  
username = 'xxxxxx@xxxxxxxx' #username  
userpassword = 'xxxxxxxx' #password  
tokenendpoint = 'https://login.microsoftonline.com/00000000-0000-0000-0000-000000000000/oauth2/token' #oauth token endpoint

#set these values to query your crm data
crmwebapi = 'https://lucasdemo01.api.crm.dynamics.com/api/data/v8.2'
crmwebapiquery = "/accounts?$select=name,accountid&$filter=startswith(name,'f')"

workflowid = 'DC8519EC-F3CE-4BC9-BB79-DF2AD70217A1'; #guid for the workflow you want to execute

def start():
    #build the authorization request
    tokenpost = {
        'client_id':clientid,
        'resource':crmorg,
        'username':username,
        'password':userpassword,
        'grant_type':'password'
    }

    #make the token request
    print('requesting token . . .')
    tokenres = requests.post(tokenendpoint, data=tokenpost)
    print('token response received. . .')

    accesstoken = ''

    #extract the access token
    try:
        print('parsing token response . . .')
        accesstoken = tokenres.json()['access_token']
        #print('accesstoken is - ' + accesstoken)

    except(KeyError):
        print('Could not get access token')

    if(accesstoken!=''):
        crmrequestheaders = {
            'Authorization': 'Bearer ' + accesstoken,
            'OData-MaxVersion': '4.0',
            'OData-Version': '4.0',
            'Accept': 'application/json',
            'Content-Type': 'application/json; charset=utf-8',
            'Prefer': 'odata.maxpagesize=500',
            'Prefer': 'odata.include-annotations=OData.Community.Display.V1.FormattedValue'
        }

        print('making crm account request . . .')
        crmres = requests.get(crmwebapi+crmwebapiquery, headers=crmrequestheaders)
        print('crm account response received . . .')
        try:
            print('parsing crm account response . . .')
            crmresults = crmres.json()
            for x in crmresults['value']:
                print (x['name'] + ' - ' + x['accountid'])
                runWorkflow(accesstoken, x['accountid'])
        except KeyError:
            print('Could not parse CRM account results')

        
def runWorkflow(token, entityid):
    crmwebapiworkflowpath = "/workflows("+workflowid+")/Microsoft.Dynamics.CRM.ExecuteWorkflow"

    #set the web api request headers
    requestheaders = { 
        'Authorization': 'Bearer ' + token,
        'OData-MaxVersion': '4.0',
        'OData-Version': '4.0',
        'Accept': 'application/json',
        'Content-Type': 'application/json; charset=utf-8'
    };
    
    reqobj = {'EntityId': entityid}
    
    print('  calling workflow for ' + entityid)
	
    crmres = requests.post(crmwebapi+crmwebapiworkflowpath, headers=requestheaders, data=json.dumps(reqobj))
    if(crmres.status_code == requests.codes.ok):
        print('    success ' + entityid)
    else:
        print('    error ' + entityid)

start()

To set it up as an Azure Function, after you have either created a new Function app or opened an existing Function app, do the following:

1. Create a new function. Select "create your own custom function" at the bottom.

2. Set language to "Python" and scenario to "Experimental." Currently there is no timer trigger template for Python, but you can work around that by selecting the "QueueTrigger-Python" template in this step and manually changing the trigger later.

3. Give your function a name and click create. Ignore the queue name and storage account connection values.

4. Once the function is created, select the "integrate" tab from the menu on the left and delete the Azure Queue Storage trigger.

5. Click "new trigger" at the top, and then select "timer" from the list that appears.

6. Give it a name (or leave the default) and set the schedule. The 0 */5 * * * * value here will execute every five minutes, just like in my previous Node.js example.

7. After the function is configured to run on a timer trigger, you have to upload the Python Requests library that is used to make the web service calls. Open a new browser window to download the Requests library code from GitHub at https://github.com/kennethreitz/requests/releases/. Select the most recent .zip file.

8. After it downloads, open it and extract the entire "requests" directory. It should contain a directory called "packages."

9. Zip up just the "requests" directory in a separate "requests.zip" file.

10. Back in the Azure portal Function App blade, select "function app settings" from the menu on the left and then click "Go to Kudu."

11. In the window that opens, click the "site" link at the top.

12. Then click the "wwwroot" link.

13. Then click the name of your function. In this case it is "CrmWorkflowTimerPython."

14. Go to the cmd prompt at the bottom of the page and type mkdir site-packages. Click enter. You should see a new "site-packages" directory get created.

15. Click on the new "site-packages" link to enter that directory.

16. Drag the "requests" .zip file you created in step 9 above into the "site-packages" directory like is shown.

17. Once it is uploaded, extract it by typing unzip requests.zip at the cmd prompt and clicking enter. You should see something like this when it's complete.

18. At this point, you can close the Kudu window.

19. From the Function App blade, select your Python function and click "develop." Highlight everything and delete it.

20. Paste the Python code from the beginning of this post. The highlighted lines in the image are how the script knows how to find the Requests library you uploaded earlier. Set any specifics relative to your Dynamics 365 organization, and click save. At this point it should just start running on the schedule you set earlier.

21. You can see all the function invocation logs on the monitor tab.

22. Here are the notes that the workflow created in my Dynamics 365 online org.

A few notes/caveats:

1. My Python code has hardly any error handling right now. If the workflow execution call returns an error, the Python code will not recognize it as an error.
2. My CRM record retrieval is set to retrieve a maximum of 500 records. You would need to modify the Web API request logic to handle more.
3. Per the "Best Practices for Azure Functions" guide:

Assume your function could encounter an exception at any time. Design your functions with the ability to continue from a previous fail point during the next execution.

This means you should put logic in your workflow to make sure that duplicate executions are avoided (unless that's what you intend to happen).

That's it for today. Until next time, happy coding!

A few days back I wrote a post that showed an easy way to set up Dynamics 365 and Node.js integration using the Web API. Here is Python code that demonstrates equivalent functionality to query contacts and display their information:

import requests
import json

#set these values to retrieve the oauth token
crmorg = 'https://CRMORG.crm.dynamics.com' #base url for crm org
clientid = '00000000-0000-0000-0000-000000000000' #application client id
username = 'xxxxxx@xxxxxxxx' #username
userpassword = 'xxxxxxxx' #password
tokenendpoint = 'https://login.microsoftonline.com/00000000-0000-0000-0000-000000000000/oauth2/token' #oauth token endpoint

#set these values to query your crm data
crmwebapi = 'https://CRMORG.api.crm.dynamics.com/api/data/v8.2' #full path to web api endpoint
crmwebapiquery = '/contacts?$select=fullname,contactid' #web api query (include leading /)

#build the authorization token request
tokenpost = {
    'client_id':clientid,
    'resource':crmorg,
    'username':username,
    'password':userpassword,
    'grant_type':'password'
}

#make the token request
tokenres = requests.post(tokenendpoint, data=tokenpost)

#set accesstoken variable to empty string
accesstoken = ''

#extract the access token
try:
    accesstoken = tokenres.json()['access_token']
except(KeyError):
    #handle any missing key errors
    print('Could not get access token')

#if we have an accesstoken
if(accesstoken!=''):
    #prepare the crm request headers
    crmrequestheaders = {
        'Authorization': 'Bearer ' + accesstoken,
        'OData-MaxVersion': '4.0',
        'OData-Version': '4.0',
        'Accept': 'application/json',
        'Content-Type': 'application/json; charset=utf-8',
        'Prefer': 'odata.maxpagesize=500',
        'Prefer': 'odata.include-annotations=OData.Community.Display.V1.FormattedValue'
    }

    #make the crm request
    crmres = requests.get(crmwebapi+crmwebapiquery, headers=crmrequestheaders)

    try:
        #get the response json
        crmresults = crmres.json()

        #loop through it
        for x in crmresults['value']:
            print (x['fullname'] + ' - ' + x['contactid'])
    except KeyError:
        #handle any missing key errors
        print('Could not parse CRM results')

This assumes you have configured an application in Azure Active Directory and know its client id and the Azure AD tenant id just like in my earlier Node.js example.

When I run this from my local PC against a my Dynamics 365 org with sample data installed, I get this output:

Like my earlier Node.js example, this Python version isn't fancy, but it shows how easy it is to authenticate to Dynamics 365 and retrieve data without requiring special libraries.