Running Dynamics 365 Configuration Data Mover jobs in Azure Functions

by Lucas Alexander

My Dynamics 365 Configuration Data Mover utility allows you to run synchronization jobs from an interactive GUI tool or the command line, but the actual data synchronization logic is contained in a separate AlexanderDevelopment.ConfigDataMover.Lib.dll file that can be included in other applications. In today's post I will show how you can set up an Azure Function to execute a Configuration Data Mover job file to sync data between two Dynamics 365 organizations.

Setting up your Azure Function

First create a new Azure Function. I created an HTTP trigger function with the default "function" authorization level so that I could post parameters to it, but if you could modify the code sample later in this post to use it with a different type of trigger.

Next open your function's Kudu console so you can create a "bin" directory to upload the AlexanderDevelopment.ConfigDataMover.Lib.dll file.

Navigate to the correct directory and create a "bin" directory inside it.

After that, go back to the main function editor interface to see the newly created "bin" directory in the "view files" area.

Look in the local directory where you have installed the Configuration Data Mover and find the "AlexanderDevelopment.ConfigDataMover.Lib.dll" file. Upload it to the "bin" directory you just created.

Navigate back to the main function directory and create a project.json file to pull in dependencies via NuGet.

Here is the content of the file so you can copy and paste:

{
  "frameworks": {
    "net46":{
      "dependencies": {
        "Microsoft.CrmSdk.CoreAssemblies": "8.2.0",
        "Microsoft.CrmSdk.XrmTooling.CoreAssembly": "8.2.0",
        "log4net": "2.0.8"
      }
    }
  }
}

Finally, you need to update the function code in the run.csx file.

Here's the code you can use:

#r "NewtonSoft.Json"
#r "AlexanderDevelopment.ConfigDataMover.Lib.dll"
using System.Net;
using System.Collections.Generic;
using AlexanderDevelopment.ConfigDataMover.Lib;
using System.Xml;

static string _sourceString = null;
static string _targetString = null;
static bool _mapBaseBu = false;
static bool _mapBaseCurrency = false;
static List<GuidMapping> _guidMappings = new List<GuidMapping>();
static List<JobStep> _jobSteps = new List<JobStep>();

public static async Task<HttpResponseMessage> Run(HttpRequestMessage req, TraceWriter log)
{
    log.Info("C# HTTP trigger function processed a request.");

    // parse query parameters
    string jobdata = req.GetQueryNameValuePairs()
        .FirstOrDefault(q => string.Compare(q.Key, "jobdata", true) == 0)
        .Value;
    string sourceparam = req.GetQueryNameValuePairs()
        .FirstOrDefault(q => string.Compare(q.Key, "source", true) == 0)
        .Value;
    string targetparam = req.GetQueryNameValuePairs()
        .FirstOrDefault(q => string.Compare(q.Key, "target", true) == 0)
        .Value;

    if(jobdata == null)
    {
        return req.CreateResponse(HttpStatusCode.BadRequest, "Please post jobdata");
    }
    else
    {
        ParseConfig(jobdata);
		
		//use source and target values if provided in the POST
		if(!string.IsNullOrEmpty(targetparam))
		{
			_targetString = targetparam;
		}
		if(!string.IsNullOrEmpty(sourceparam))
		{
			_sourceString = sourceparam;
		}
		
		//do some basic validations
		if (string.IsNullOrEmpty(_sourceString))
		{
			return req.CreateResponse(HttpStatusCode.BadRequest,"no source connection specified - exiting");
		}
		if (string.IsNullOrEmpty(_targetString))
		{
			return req.CreateResponse(HttpStatusCode.BadRequest,"no target connection specified - exiting");
		}
		if (!(_jobSteps.Count > 0))
		{
			return req.CreateResponse(HttpStatusCode.BadRequest,"no steps in job - exiting");
		}

		Importer importer = new Importer();
		importer.GuidMappings = _guidMappings;
		importer.JobSteps = _jobSteps;
		importer.SourceString = _sourceString; 
		importer.TargetString = _targetString; 
		importer.MapBaseBu = _mapBaseBu;
		importer.MapBaseCurrency = _mapBaseCurrency;
		importer.Process();

		int errorCount = importer.ErrorCount;

		importer = null;
		
		//show a message to the user
		if (errorCount == 0)
		{
			return req.CreateResponse(HttpStatusCode.OK,"Job finished with no errors.");
		}
		else
		{
			return req.CreateResponse(HttpStatusCode.BadRequest,"Job finished with errors.");
		}
    }
}

public static void ParseConfig(string jobdata)
{
    XmlDocument xml = new XmlDocument();
	xml.LoadXml(jobdata);
	_jobSteps.Clear();
	_guidMappings.Clear();

	XmlNodeList stepList = xml.GetElementsByTagName("Step");
	foreach (XmlNode xn in stepList)
	{
		JobStep step = new JobStep();
		step.StepName = xn.SelectSingleNode("Name").InnerText;
		step.StepFetch = xn.SelectSingleNode("Fetch").InnerText;
		step.UpdateOnly = false;
		if(xn.Attributes["updateOnly"]!=null)
			step.UpdateOnly = Convert.ToBoolean(xn.Attributes["updateOnly"].Value);

		step.CreateOnly = false;
		if (xn.Attributes["createOnly"] != null)
			step.CreateOnly = Convert.ToBoolean(xn.Attributes["createOnly"].Value);

		_jobSteps.Add(step);
	}

	XmlNodeList configData = xml.GetElementsByTagName("JobConfig");
	_mapBaseBu = Convert.ToBoolean(configData[0].Attributes["mapBuGuid"].Value);
	_mapBaseCurrency = Convert.ToBoolean(configData[0].Attributes["mapCurrencyGuid"].Value);

	XmlNodeList mappingList = xml.GetElementsByTagName("GuidMapping");
	foreach (XmlNode xn in mappingList)
	{
		Guid sourceGuid = new Guid(xn.Attributes["source"].Value);
		Guid targetGuid = new Guid(xn.Attributes["target"].Value);
		_guidMappings.Add(new GuidMapping { sourceId = sourceGuid, targetId = targetGuid });
	}
	XmlNodeList connectionNodes = xml.GetElementsByTagName("ConnectionDetails");
	if (connectionNodes.Count > 0)
	{
		_sourceString = connectionNodes[0].Attributes["source"].Value;
		_targetString = connectionNodes[0].Attributes["target"].Value;
	}
}

Executing your Azure Function

To execute the function, you'll need to copy the function key so that you can send it as a POST parameter.

Once you have that, you can build a POST request. Here's what it looks like in Postman:

The only required parameters are "code" and "jobdata." "Code" is the function key you copied earlier. "Jobdata" is the content of a Configuration Data Mover job XML file. If you do not have the source/target connection details saved in your "jobdata" XML, you will need to also include "source" and "target" connection string parameters in your request. If you do have connection details saved in the "jobdata" XML and you also supply "source" and "target" connection string parameters, they will be used instead of what is in the "jobdata" XML.

Assuming everything goes as expected, you should get a "job finished with no errors" response like this:

If you still have the function editor interface open, you'll see new log entries like this:

Caveats

  1. I originally built the Configuration Data Mover to use Apache log4net for detailed logging. The AlexanderDevelopment.ConfigDataMover.Lib library here is still trying to use log4net, but those log entries aren't going anywhere. I'll probably look into modifying the logging approach in the library so that it works better in situations where the library used like this. The upshot is that what I've outlined here will give you only a basic success/failure message.
  2. Currently the AlexanderDevelopment.ConfigDataMover.Lib library can either connect to a live CRM instance or read configuration data from a JSON file that it can open. I have a future enhancement in mind that would allow for configuration data to be passed to the library to make it more flexible.