I've released an updated version of my popular Dynamics 365 Configuration Data Mover utility that includes the ability to create many-to-many associations in the target system. This upgrade is fully compatible with existing job files.

To create a many-to-many job step in the GUI, select the new "many to many" step type and input a FetchXML query for the relationship entity (relationship entity name on the many-to-many relationship form) that includes the GUID fields for each entity. The relationship entities cannot be queried in the advanced find builder, so you must write the FetchXML manually or use a separate query builder.

One thing to keep in mind is that a many-to-many job step will create many-to-many record associations in the target system, but it will not delete any existing N:N associations that have been removed in the source.

Getting the Dynamics 365 Configuration Data Mover

The source code is available in my GitHub repository here.

A compiled version can be downloaded here.

In today's post I will show how to use my Dynamics 365 Configuration Data Mover utility for synchronizing configuration data between CRM orgs as part of a Visual Studio Team Services build.

1. Download the latest version of the Configuration Data Mover utility's CLI tool from my repository on GitHub here: https://github.com/lucasalexander/AlexanderDevelopment.ConfigDataMover/releases.
   Any version from 2.2 onward should work, and make sure you get the "AlexanderDevelopment.ConfigDataMover.Cli.zip" file.
2. Extract everything from the zip archive and add the entire directory to your source code repository. I have historically used a "tools" directory that contains various utilities, so I keep the Data Mover CLI tool in its own subdirectory there.
3. Add a directory to your source code repository to hold configuration data mover job files and JSON data extracts. I call mine "data."
4. Set up a data mover job using the GUI tool. You can create a job that syncs data between two CRM orgs or a job that reads from a file source. I prefer to use file sources for deployments because then I don't have a dependency on the development org, and also the configuration data can be kept under source control. You can also save your connection parameters in the job file or pass them to the Data Mover at run time.
5. After you create and test your data mover job, save a copy of the XML job file to your source control "data" directory.
6. (This step is only required for file sources when you are saving the connection parameters in the job file.) Open the XML file in a text editor and find the <ConnectionDetails> node toward the bottom. There will be a "source" attribute that you need to make sure is set to a relative path to the JSON data file from the location of your AlexanderDevelopment.ConfigDataMover.Cli.exe assembly. Because of how I have my directory structure set up in source control, the a JSON file named "alm-test.json" has "....\data\alm-test.json" as its relative path. Here's a screenshot of my directory structure for reference:
   
7. Commit your changes to your VSTS instance. I am using Git, but this should work for TFS, too.
8. Create a new build and add a batch script step.
9. Configure the batch script step to run the AlexanderDevelopment.ConfigDataMover.Cli.exe assembly you pushed to source control in step #7.
   
10. Set the arguments to include the relative path to your job configuration file. For this example, it is -c ....\data\alm-test-import.xml
    
11. (This step is only required if you are not saving the connection parameters in the job file.) Add source and target arguments, too. The source argument is -s, and the target argument is -t. The values will look just like the source/target attributes from a job configuration file where you have saved the values.
12. Set the working folder value to the directory that contains the AlexanderDevelopment.ConfigDataMover.Cli.exe assembly. You can see that setting in the screenshot above.

At this point you should be able to save and queue the build. Assuming everything is set up properly, you should get a nice result like this:

If you do get an error, it's probably something simple like the paths not being set correctly, and a quick review of the output logs should point to the solution.

That's all there is to it. Are you using automated builds in VSTS for your Dynamics 365 projects? Do see value in managing configuration data like this? Share your thoughts in the comments!

I've released an updated version of my popular Dynamics 365 Configuration Data Mover utility that includes the following features:

• Path to source and target JSON data files can now be absolute or relative to current working directory. This is particularly useful for running the CLI tool in an automated fashion.
• Location of job configuration file can be passed to CLI tool as absolution or relative path.
• Import tool can now read configuration data directly as a JSON string instead of having to read it from a file. The syntax for supplying the the data to the CLI tool (or importer library DLL) is RAWJSON={YOUR_EXPORT_JSON_HERE}

This upgrade is fully compatible with existing job files.

Getting the Dynamics 365 Configuration Data Mover

The source code is available in my GitHub repository here.

A compiled version can be downloaded here.

My Dynamics 365 Configuration Data Mover utility allows you to run synchronization jobs from an interactive GUI tool or the command line, but the actual data synchronization logic is contained in a separate AlexanderDevelopment.ConfigDataMover.Lib.dll file that can be included in other applications. In today's post I will show how you can set up an Azure Function to execute a Configuration Data Mover job file to sync data between two Dynamics 365 organizations.

Setting up your Azure Function

First create a new Azure Function. I created an HTTP trigger function with the default "function" authorization level so that I could post parameters to it, but if you could modify the code sample later in this post to use it with a different type of trigger.

Next open your function's Kudu console so you can create a "bin" directory to upload the AlexanderDevelopment.ConfigDataMover.Lib.dll file.

Navigate to the correct directory and create a "bin" directory inside it.

After that, go back to the main function editor interface to see the newly created "bin" directory in the "view files" area.

Look in the local directory where you have installed the Configuration Data Mover and find the "AlexanderDevelopment.ConfigDataMover.Lib.dll" file. Upload it to the "bin" directory you just created.

Navigate back to the main function directory and create a project.json file to pull in dependencies via NuGet.

Here is the content of the file so you can copy and paste:

{
  "frameworks": {
    "net46":{
      "dependencies": {
        "Microsoft.CrmSdk.CoreAssemblies": "8.2.0",
        "Microsoft.CrmSdk.XrmTooling.CoreAssembly": "8.2.0",
        "log4net": "2.0.8"
      }
    }
  }
}

Finally, you need to update the function code in the run.csx file.

Here's the code you can use:

#r "NewtonSoft.Json"
#r "AlexanderDevelopment.ConfigDataMover.Lib.dll"
using System.Net;
using System.Collections.Generic;
using AlexanderDevelopment.ConfigDataMover.Lib;
using System.Xml;

static string _sourceString = null;
static string _targetString = null;
static bool _mapBaseBu = false;
static bool _mapBaseCurrency = false;
static List<GuidMapping> _guidMappings = new List<GuidMapping>();
static List<JobStep> _jobSteps = new List<JobStep>();

public static async Task<HttpResponseMessage> Run(HttpRequestMessage req, TraceWriter log)
{
    log.Info("C# HTTP trigger function processed a request.");

    // parse query parameters
    string jobdata = req.GetQueryNameValuePairs()
        .FirstOrDefault(q => string.Compare(q.Key, "jobdata", true) == 0)
        .Value;
    string sourceparam = req.GetQueryNameValuePairs()
        .FirstOrDefault(q => string.Compare(q.Key, "source", true) == 0)
        .Value;
    string targetparam = req.GetQueryNameValuePairs()
        .FirstOrDefault(q => string.Compare(q.Key, "target", true) == 0)
        .Value;

    if(jobdata == null)
    {
        return req.CreateResponse(HttpStatusCode.BadRequest, "Please post jobdata");
    }
    else
    {
        ParseConfig(jobdata);
		
		//use source and target values if provided in the POST
		if(!string.IsNullOrEmpty(targetparam))
		{
			_targetString = targetparam;
		}
		if(!string.IsNullOrEmpty(sourceparam))
		{
			_sourceString = sourceparam;
		}
		
		//do some basic validations
		if (string.IsNullOrEmpty(_sourceString))
		{
			return req.CreateResponse(HttpStatusCode.BadRequest,"no source connection specified - exiting");
		}
		if (string.IsNullOrEmpty(_targetString))
		{
			return req.CreateResponse(HttpStatusCode.BadRequest,"no target connection specified - exiting");
		}
		if (!(_jobSteps.Count > 0))
		{
			return req.CreateResponse(HttpStatusCode.BadRequest,"no steps in job - exiting");
		}

		Importer importer = new Importer();
		importer.GuidMappings = _guidMappings;
		importer.JobSteps = _jobSteps;
		importer.SourceString = _sourceString; 
		importer.TargetString = _targetString; 
		importer.MapBaseBu = _mapBaseBu;
		importer.MapBaseCurrency = _mapBaseCurrency;
		importer.Process();

		int errorCount = importer.ErrorCount;

		importer = null;
		
		//show a message to the user
		if (errorCount == 0)
		{
			return req.CreateResponse(HttpStatusCode.OK,"Job finished with no errors.");
		}
		else
		{
			return req.CreateResponse(HttpStatusCode.BadRequest,"Job finished with errors.");
		}
    }
}

public static void ParseConfig(string jobdata)
{
    XmlDocument xml = new XmlDocument();
	xml.LoadXml(jobdata);
	_jobSteps.Clear();
	_guidMappings.Clear();

	XmlNodeList stepList = xml.GetElementsByTagName("Step");
	foreach (XmlNode xn in stepList)
	{
		JobStep step = new JobStep();
		step.StepName = xn.SelectSingleNode("Name").InnerText;
		step.StepFetch = xn.SelectSingleNode("Fetch").InnerText;
		step.UpdateOnly = false;
		if(xn.Attributes["updateOnly"]!=null)
			step.UpdateOnly = Convert.ToBoolean(xn.Attributes["updateOnly"].Value);

		step.CreateOnly = false;
		if (xn.Attributes["createOnly"] != null)
			step.CreateOnly = Convert.ToBoolean(xn.Attributes["createOnly"].Value);

		_jobSteps.Add(step);
	}

	XmlNodeList configData = xml.GetElementsByTagName("JobConfig");
	_mapBaseBu = Convert.ToBoolean(configData[0].Attributes["mapBuGuid"].Value);
	_mapBaseCurrency = Convert.ToBoolean(configData[0].Attributes["mapCurrencyGuid"].Value);

	XmlNodeList mappingList = xml.GetElementsByTagName("GuidMapping");
	foreach (XmlNode xn in mappingList)
	{
		Guid sourceGuid = new Guid(xn.Attributes["source"].Value);
		Guid targetGuid = new Guid(xn.Attributes["target"].Value);
		_guidMappings.Add(new GuidMapping { sourceId = sourceGuid, targetId = targetGuid });
	}
	XmlNodeList connectionNodes = xml.GetElementsByTagName("ConnectionDetails");
	if (connectionNodes.Count > 0)
	{
		_sourceString = connectionNodes[0].Attributes["source"].Value;
		_targetString = connectionNodes[0].Attributes["target"].Value;
	}
}

Executing your Azure Function

To execute the function, you'll need to copy the function key so that you can send it as a POST parameter.

Once you have that, you can build a POST request. Here's what it looks like in Postman:

The only required parameters are "code" and "jobdata." "Code" is the function key you copied earlier. "Jobdata" is the content of a Configuration Data Mover job XML file. If you do not have the source/target connection details saved in your "jobdata" XML, you will need to also include "source" and "target" connection string parameters in your request. If you do have connection details saved in the "jobdata" XML and you also supply "source" and "target" connection string parameters, they will be used instead of what is in the "jobdata" XML.

Assuming everything goes as expected, you should get a "job finished with no errors" response like this:

If you still have the function editor interface open, you'll see new log entries like this:

Caveats

1. I originally built the Configuration Data Mover to use Apache log4net for detailed logging. The AlexanderDevelopment.ConfigDataMover.Lib library here is still trying to use log4net, but those log entries aren't going anywhere. I'll probably look into modifying the logging approach in the library so that it works better in situations where the library used like this. The upshot is that what I've outlined here will give you only a basic success/failure message.
2. Currently the AlexanderDevelopment.ConfigDataMover.Lib library can either connect to a live CRM instance or read configuration data from a JSON file that it can open. I have a future enhancement in mind that would allow for configuration data to be passed to the library to make it more flexible.

I've released an updated version of my popular CRM Configuration Data Mover utility that includes the following features:

• Steps can now be set as create-only, update-only or create+update.
• You can browse to data files using a standard file open dialog window.

This upgrade is fully compatible with existing job files.

Getting the Dynamics 365 Configuration Data Mover

The source code is available in my GitHub repository here.

A compiled version can be downloaded here.

I've released an updated version of my popular CRM Configuration Data Mover utility. This version now supports Dynamics 365 and has an updated GUI built with Windows Presentation Foundation that replaces the previous Windows Forms GUI.

Jobs that were created with earlier versions of the tool will work with this version, but you will have to update your CRM connection parameters to specify the authorization type - Active Directory, IFD or Office 365.

The other main change in the client is the way GUID mappings are entered. GUIDs are now entered in a text box with each mapping on a separate line. A "->" is used to separate source and target. The screenshot below shows the new UI.

Getting the Dynamics CRM Configuration Data Mover

The source code is available in my GitHub repository here.

A compiled version can be downloaded here.

Please let me know what you think of this new version in the comments section below.

I've released an updated version of my Dynamics CRM Configuration Data Mover tool that includes the following enhancements:

• Validation for FetchXML queries against the Dynamics CRM 2016 FetchXML XSD when jobs are saved through the GUI client and at the start of a job before any data is processed
• Better error reporting in the GUI client
• GUID mapping for the root business unit's default team when the option to map the root business unit is selected

Getting the Dynamics CRM Configuration Data Mover

The source code is available in my GitHub repository here.

A compiled version can be downloaded here.

I've released an updated version of my Dynamics CRM Configuration Data Mover tool that adds the ability to synchronize entityimage attributes stored in flat-file extracts. Previous versions of the tool already allowed for entityimage synchronization when running a job directly between source and target CRM systems.

Getting the Dynamics CRM Configuration Data Mover

The source code is available in my GitHub repository here.

A compiled version can be downloaded here.

I've released an updated version of my Dynamics CRM Configuration Data Mover tool that includes the following enhancements:

• Setting statecode and statuscode values on create or update is now supported. The tool will not set statuscode unless statecode is also specified.
• Record ownership can now be changed on updates.

Getting the Dynamics CRM Configuration Data Mover

The source code is available in my GitHub repository here.

A compiled version can be downloaded here.

Last month a colleague of mine asked if it would be possible to synchronize Dynamics CRM service calendars and holiday calendars using my Configuration Data Mover tool. At first it looked like it would be complicated, but after trying a few different approaches, it turned out to be incredibly simple. Before getting to the solution, let's look at how Dynamics CRM stores calendar data.

(If you don't care about the details of how it all works, skip to the end of this post for a link to download a sample job file.)

Dynamics CRM calendar entities

There are only two types of calendar entities in Dynamics CRM - calendars and calendarrules.

A calendar describes the availability of a service or a resource. Calendars are related to calendarrule records, which include details about the duration, start and end times, and recurring patterns of events included in the calendar.

Although there's only one calendar entity in Dynamics CRM, there are four types of calendars that are differentiated by their "type" attribute:

1. Default (type 0) - All calendars that are not customer service, holiday schedule, or inner calendars.
2. Customer Service (type 1) - Service calendars for customer service.
3. Holiday Schedule (type 2) - Holiday schedule calendars for customer service.
4. Inner Calendar type (type -1) - Inner Calendars are used by other calendars to build a graph of time slots available for customer service or service scheduling to be performed.

Calendar records also have a self-referencing relationship. This is how a customer service calendar can have a linked holiday schedule calendar.

Here's a FetchXML query to retrieve all the customer service calendars in an organization:

<fetch distinct="false" mapping="logical" output-format="xml-platform" version="1.0" >
    <entity name="calendar" >
        <attribute name="name" />
        <attribute name="typename" />
        <attribute name="type" />
        <attribute name="primaryuserid" />
        <attribute name="isshared" />
        <attribute name="holidayschedulecalendaridname" />
        <attribute name="holidayschedulecalendarid" />
        <attribute name="description" />
        <attribute name="businessunitid" />
        <order descending="true" attribute="type" />
		<filter type="and">
			<condition attribute="type" operator="in">
				<value>1</value>
			</condition>
		</filter>
    </entity>
</fetch>

As for the calendarrule records, they cannot be directly modified using the SDK, but you can access them through their parent calendar records. Although the query above does not explicitly request the calendarrules, CRM does return them in the response. This allows the Configuration Data Mover to copy the calendars with their associated rules and closures to a destination organization relatively easily.

Copying the data

Copying service and holiday calendars from one CRM organization to another with the Configuration Data Mover requires three steps:

1. Copy inner calendars
2. Copy holiday calendars
3. Copy customer service calendars

Here's a link to a sample job file that is already set up to do that: calendars-closures-noconnection.xml.

One thing to keep in mind is that the schema I was using to serialize output data to a flat-file destination up to version 1.7.1.1 does not support a nested entity structure like CRM uses to store calendar rules, so you need to make sure you are using that version or later if you want to use a flat-file source/target jobs. Otherwise you will get an error when attempting to import the calendar data from a flat-file source.

Getting the Dynamics CRM Configuration Data Mover

The source code is available in my GitHub repository here.

A compiled version can be downloaded here.

I've released an updated version of my Dynamics CRM Configuration Data Mover tool that includes the following GUI enhancements:

• Password masking in connection form (saved passwords are still stored as plain text)
• Improved record error logging to show reason for import failure
• Improved connection error handling for import jobs

Thanks to GitHub user btull89 for suggesting the first two items and submitting some code for them. I'm always looking for feedback and contributions, so if anyone else would like to suggest a feature or write some code, please let me know!

Getting the Dynamics CRM Configuration Data Mover

The source code is available in my GitHub repository here.

A compiled version can be downloaded here.