In my last post I provided a detailed walkthrough of how to perform sentiment analysis on incoming emails received in Microsoft Dynamics CRM by parsing them with HP IDOL OnDemand’s sentiment analysis API and then storing the calculated sentiment values on the email records. In today’s post, I will show a similar, but slightly more complex integration that enables "find similar" functionality for emails stored in Dynamics CRM.
Continuing with the scenario from my previous post, let’s assume you manage a Microsoft Dynamics CRM system (online or on-premise) for an organization that provides customer service and support via a variety of channels including phone, email and live chat. In order to help agents resolve inbound email cases more quickly, the operations team has asked you to add an embedded display on the email form that shows conceptually similar email records so the agents can easily view them. Just as before, we’re in luck because IDOL OnDemand has a find similar API.
The solution approach
Unlike the sentiment analysis solution I discussed previously where we used a workflow to retrieve a sentiment score and write it directly to the email record, the find similar email display will require two different calls to the IDOL OnDemand service because to find similar items, IDOL OnDemand needs to have an index of items to search. We’ll need to populate this index with emails as they’re received, and then we’ll run the find similar search against the index when agents view an email record. Thus, the solution I am showing today has two pieces:
- First, there is a custom workflow activity that adds inbound emails to an IDOL OnDemand text index using the add to text index API.
- Second, there is an HTML web resource that can be embedded on the email form that uses an AJAX call to the find similar API to retrieve similar emails from the index and then display summary data and a hyperlink to the actual CRM record for each result to the end user.
Creating the index
Before we can populate an index with data, we need to have an index that can be populated. Although IDOL OnDemand offers APIs that can be used to manage indexes programmatically, it’s easier in this case to use the index management tools that are available on the main account screen. On that screen, click the "Create, Manage, Index and Search your own Text Indexes" link, then click the "create text index" button on the next page. Give your index a name, and then just accept the default options for each of the following prompts in the wizard.* Remember the name you selected because you’ll need to use it later. My index name is "lucas-test."
The indexing process
IDOL OnDemand can index JSON documents that contain both standard and custom fields, so we’re going to create a JSON object that contains the email body, the email subject line and the email activity id as the index reference (basically a primary key) field. Because an IDOL OnDemand index has a standard field called title that is, per the index developer documentation, considered "highly relevant," we’ll also set that to the value of the email subject. Our logical mappings will look like this:
- Email "body" (description) -> Index "content"
- Email "subject" -> Index "title"
- Email "subject" -> Index "subject"
- Email "activityid" -> Index "reference"
You can, of course, index lots of other fields like "to," "from," "date," etc., and the more complete your index, the better your search results will be.
In order to populate the index with the fields above, the custom workflow activity needs input parameters for the email body, subject and activity id. Here's how we define them:
[Input("Content")]
public InArgument<String> Content { get; set; }
[Input("Subject")]
public InArgument<String> Subject { get; set; }
[Input("Email")]
[ReferenceTarget("email")]
public InArgument<EntityReference> Email { get; set; }We also need to set up three variables to help us call IDOL OnDemand. They are the URL for the find similar API, the API key to authorize our requests and the name of the text index that will hold the email data:
//address of the service to which you will post your json messages
private string _webAddress = "https://api.idolondemand.com/1/api/sync/addtotextindex/v1";
//name of the text index
private string _indexName = "lucas-test";
//idol ondemand api key
private string _apiKey = "XXXXXXXXXXXXXXXX";Now we’re ready to index the inbound email using the following steps:
- Check whether the email description field contains text.
- If it does, strip HTML tags using a helper function.
- Serialize the index request to a JSON object.
- Pass the index request object to IDOL OnDemand with an HttpWebRequest. (In order to keep this example simple, we’re going to ignore the response from IDOL OnDemand, but you might want to do something with the response if you try this out on your own.)
Here's the code that does all of this:
string inputText = Content.Get(executionContext);
if (inputText != string.Empty)
{
inputText = HtmlTools.StripHTML(inputText);
IndexDocument myDoc = new IndexDocument
{
Content = inputText,
Reference = (Email.Get(executionContext)).Id.ToString(),
Subject = Subject.Get(executionContext),
Title = Subject.Get(executionContext)
};
DocumentWrapper myWrapper = new DocumentWrapper();
myWrapper.Document = new List<IndexDocument>();
myWrapper.Document.Add(myDoc);
//serialize the myjsonrequest to json
System.Runtime.Serialization.Json.DataContractJsonSerializer serializer = new System.Runtime.Serialization.Json.DataContractJsonSerializer(myWrapper.GetType());
MemoryStream ms = new MemoryStream();
serializer.WriteObject(ms, myWrapper);
string jsonMsg = Encoding.Default.GetString(ms.ToArray());
//create the webrequest object and execute it (and post jsonmsg to it)
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(\_webAddress);
//set request content type so it is treated as a regular form post
req.ContentType = "application/x-www-form-urlencoded";
//set method to post
req.Method = "POST";
StringBuilder postData = new StringBuilder();
//HttpUtility.UrlEncode
//set the apikey request value
postData.Append("apikey=" + System.Uri.EscapeDataString(\_apiKey) + "&");
//postData.Append("apikey=" + \_apiKey + "&");
//set the json request value
postData.Append("json=" + jsonMsg + "&");
//set the index name request value
postData.Append("index=" + _indexName);
//create a stream
byte[] bytes = System.Text.Encoding.ASCII.GetBytes(postData.ToString());
req.ContentLength = bytes.Length;
System.IO.Stream os = req.GetRequestStream();
os.Write(bytes, 0, bytes.Length);
os.Close();
//get the response
System.Net.WebResponse resp = req.GetResponse();
//deserialize the response to a ResponseBody object
ResponseBody myResponse = new ResponseBody();
System.Runtime.Serialization.Json.DataContractJsonSerializer deserializer = new System.Runtime.Serialization.Json.DataContractJsonSerializer(myResponse.GetType());
myResponse = deserializer.ReadObject(resp.GetResponseStream()) as ResponseBody;
}The StripHTML function is the same function from CodeProject that used in my sentiment analysis post, and the code is included in the download at the end of this post.
Here are the classes used for serializing the JSON request and deserializing the JSON response:
//DataContract decoration necessary for serialization/deserialization to work properly
[DataContract]
public class DocumentWrapper
{
//datamember name value indicates name of json field to which data will be serialized/from which data will be deserialized
[DataMember(Name = "document")]
public List<IndexDocument> Document { get; set; }
}
//DataContract decoration necessary for serialization/deserialization to work properly
[DataContract]
public class IndexDocument
{
[DataMember(Name = "title")]
public string Title { get; set; }
[DataMember(Name = "reference")]
public string Reference { get; set; }
[DataMember(Name = "subject")]
public string Subject { get; set; }
[DataMember(Name = "content")]
public string Content { get; set; }
}
//DataContract decoration necessary for serialization/deserialization to work properly
[DataContract]
public class ResponseBody
{
[DataMember(Name = "index")]
public string Index { get; set; }
[DataMember(Name = "references")]
public List<ResponseReference> References { get; set; }
}
//DataContract decoration necessary for serialization/deserialization to work properly
[DataContract]
public class ResponseReference
{
[DataMember(Name = "reference")]
public string Reference { get; set; }
[DataMember(Name = "id")]
public int Id { get; set; }
}The results display
The code above takes care of getting the email data into our IDOL OnDemand text index, but now we need an easy way to execute the find similar queries from a web resource on the email form. This is actually incredibly easy to do with a little bit of JavaScript and jQuery.
To keep things simple, we’re going to make an assumption that all CRM emails will be in the IDOL OnDemand index by the time an agent opens one, so we can tell IDOL OnDemand to find similar records using the email’s activityid field, which we are populating as the index reference field in the code above. This also means we can embed the web resource on the email form without having to use any custom code because the Dynamics CRM form designer lets us specify that an iframe will be passed the record id when the form loads.
If turned out that we couldn’t count on the emails always being indexed before the agents viewed them, we could search the index using the relevant fields from an email record instead of using an index reference, but this would require some changes to what I’m showing today.
Our web resource will do the following:
- Load the jQuery library (in my sample, I’m using the Google CDN).
- Parse the record id from the value supplied in the query string that is passed to the iframe.
- Execute a jQuery "getJSON" call to IDOL OnDemand’s find similar API.
- Loop through the output results and write them to the screen.
Let’s take a look at the code in detail.
First, we load jQuery:
<script src="//ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>Next we need a function that can extract query string values:
function getParameterByName(name) {
name = name.replace(/[[]/, "\[").replace(/[]]/, "\]");
var regex = new RegExp("[\?&]" + name + "=([^&#]*)"),
results = regex.exec(location.search);
return results == null ? "" : decodeURIComponent(results[1].replace(/+/g, " "));
}Using the getParameterByName function, we parse the email’s activity id and prepare if for querying IDOL OnDemand like so:
var entityId = getParameterByName('id');
//strip brackets because we indexed a string value of the guid without them
entityId = entityId.replace("{","").replace("}","");Finally we send the find similar request and process the results:
var indexName = "lucas-test";
var apiKey = "f90f3fc7-0e36-40f0-b508-734f45fe9033";
$.getJSON("https://api.idolondemand.com/1/api/sync/findsimilar/v1?indexes="+indexName+"&apikey="+apiKey+"& summary=context&index\_reference="+entityId,function(data){
var content = "";
$(content).text("");
$.each(data.documents, function(i,document){
content += '<p><a href="/main.aspx?etn=email&id={' + document.reference + '}&newWindow=true&pagetype=entityrecord" target="_blank">' + document.title + '</a>';
content += '<br />Score: ' + document.weight + '';
content += '<br />' + document.summary + '</p>';
});
$(content).appendTo("#results");
});This writes the results to a div element on the page with an id of "results." You’ll see that each result shows the email subject (stored as "title" in our index) as a hyperlink to the actual email record, a similarity score value (weight) and a summary of the similar text.
So that’s it for getting started with IDOL OnDemand’s find similar API for Microsoft Dynamics CRM data. Full code for this is in my CRM-IdolOnDemand-Tools repository on GitHub. Does this look like something you could use in your organization? If so, let us know your thoughts in the comments!
* If you want to learn more about IDOL OnDemand text indexes, I suggest you look at the text indexes section of the developer documentation.
A version of this post was originally published on the HP Enterprise Services Application Services blog.