In my last post I provided a detailed walkthrough of how to perform sentiment analysis on incoming emails received in Microsoft Dynamics CRM by parsing them with HP IDOL OnDemand’s sentiment analysis API and then storing the calculated sentiment values on the email records. In today’s post, I will show a similar, but slightly more complex integration that enables "find similar" functionality for emails stored in Dynamics CRM.

Continuing with the scenario from my previous post, let’s assume you manage a Microsoft Dynamics CRM system (online or on-premise) for an organization that provides customer service and support via a variety of channels including phone, email and live chat. In order to help agents resolve inbound email cases more quickly, the operations team has asked you to add an embedded display on the email form that shows conceptually similar email records so the agents can easily view them. Just as before, we’re in luck because IDOL OnDemand has a find similar API.

The solution approach

Unlike the sentiment analysis solution I discussed previously where we used a workflow to retrieve a sentiment score and write it directly to the email record, the find similar email display will require two different calls to the IDOL OnDemand service because to find similar items, IDOL OnDemand needs to have an index of items to search. We’ll need to populate this index with emails as they’re received, and then we’ll run the find similar search against the index when agents view an email record. Thus, the solution I am showing today has two pieces:

1. First, there is a custom workflow activity that adds inbound emails to an IDOL OnDemand text index using the add to text index API.
2. Second, there is an HTML web resource that can be embedded on the email form that uses an AJAX call to the find similar API to retrieve similar emails from the index and then display summary data and a hyperlink to the actual CRM record for each result to the end user.

Creating the index

Before we can populate an index with data, we need to have an index that can be populated. Although IDOL OnDemand offers APIs that can be used to manage indexes programmatically, it’s easier in this case to use the index management tools that are available on the main account screen. On that screen, click the "Create, Manage, Index and Search your own Text Indexes" link, then click the "create text index" button on the next page. Give your index a name, and then just accept the default options for each of the following prompts in the wizard.* Remember the name you selected because you’ll need to use it later. My index name is "lucas-test."

The indexing process

IDOL OnDemand can index JSON documents that contain both standard and custom fields, so we’re going to create a JSON object that contains the email body, the email subject line and the email activity id as the index reference (basically a primary key) field. Because an IDOL OnDemand index has a standard field called title that is, per the index developer documentation, considered "highly relevant," we’ll also set that to the value of the email subject. Our logical mappings will look like this:

• Email "body" (description) -> Index "content"
• Email "subject" -> Index "title"
• Email "subject" -> Index "subject"
• Email "activityid" -> Index "reference"

You can, of course, index lots of other fields like "to," "from," "date," etc., and the more complete your index, the better your search results will be.

In order to populate the index with the fields above, the custom workflow activity needs input parameters for the email body, subject and activity id. Here's how we define them:

[Input("Content")]

public InArgument<String> Content { get; set; }

[Input("Subject")]

public InArgument<String> Subject { get; set; }

[Input("Email")]

[ReferenceTarget("email")]

public InArgument<EntityReference> Email { get; set; }

We also need to set up three variables to help us call IDOL OnDemand. They are the URL for the find similar API, the API key to authorize our requests and the name of the text index that will hold the email data:

//address of the service to which you will post your json messages

private string _webAddress = "https://api.idolondemand.com/1/api/sync/addtotextindex/v1";

//name of the text index

private string _indexName = "lucas-test";

//idol ondemand api key

private string _apiKey = "XXXXXXXXXXXXXXXX";

Now we’re ready to index the inbound email using the following steps:

1. Check whether the email description field contains text.
2. If it does, strip HTML tags using a helper function.
3. Serialize the index request to a JSON object.
4. Pass the index request object to IDOL OnDemand with an HttpWebRequest. (In order to keep this example simple, we’re going to ignore the response from IDOL OnDemand, but you might want to do something with the response if you try this out on your own.)

Here's the code that does all of this:

string inputText = Content.Get(executionContext);
if (inputText != string.Empty)
{
	inputText = HtmlTools.StripHTML(inputText);
	IndexDocument myDoc = new IndexDocument
	{
	 Content = inputText,
	  Reference = (Email.Get(executionContext)).Id.ToString(),
	  Subject = Subject.Get(executionContext),
	  Title = Subject.Get(executionContext)
	};
	DocumentWrapper myWrapper = new DocumentWrapper();
	myWrapper.Document = new List<IndexDocument>();
	myWrapper.Document.Add(myDoc);
	//serialize the myjsonrequest to json
	System.Runtime.Serialization.Json.DataContractJsonSerializer serializer = new System.Runtime.Serialization.Json.DataContractJsonSerializer(myWrapper.GetType());
	MemoryStream ms = new MemoryStream();
	serializer.WriteObject(ms, myWrapper);
	string jsonMsg = Encoding.Default.GetString(ms.ToArray());
	//create the webrequest object and execute it (and post jsonmsg to it)
	HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(\_webAddress);
	//set request content type so it is treated as a regular form post
	req.ContentType = "application/x-www-form-urlencoded";
	//set method to post
	req.Method = "POST";
	StringBuilder postData = new StringBuilder();
	//HttpUtility.UrlEncode
	//set the apikey request value
	postData.Append("apikey=" + System.Uri.EscapeDataString(\_apiKey) + "&");
	//postData.Append("apikey=" + \_apiKey + "&");
	//set the json request value
	postData.Append("json=" + jsonMsg + "&");
	//set the index name request value
	postData.Append("index=" + _indexName);
	//create a stream
	byte[] bytes = System.Text.Encoding.ASCII.GetBytes(postData.ToString());
	req.ContentLength = bytes.Length;
	System.IO.Stream os = req.GetRequestStream();
	os.Write(bytes, 0, bytes.Length);
	os.Close();
	//get the response
	System.Net.WebResponse resp = req.GetResponse();
	//deserialize the response to a ResponseBody object
	ResponseBody myResponse = new ResponseBody();
	System.Runtime.Serialization.Json.DataContractJsonSerializer deserializer = new System.Runtime.Serialization.Json.DataContractJsonSerializer(myResponse.GetType());
	myResponse = deserializer.ReadObject(resp.GetResponseStream()) as ResponseBody;
}

The StripHTML function is the same function from CodeProject that used in my sentiment analysis post, and the code is included in the download at the end of this post.

Here are the classes used for serializing the JSON request and deserializing the JSON response:

//DataContract decoration necessary for serialization/deserialization to work properly

[DataContract]

public class DocumentWrapper

{

//datamember name value indicates name of json field to which data will be serialized/from which data will be deserialized

[DataMember(Name = "document")]

public List<IndexDocument> Document { get; set; }

}

//DataContract decoration necessary for serialization/deserialization to work properly

[DataContract]

public class IndexDocument

{

[DataMember(Name = "title")]

public string Title { get; set; }

[DataMember(Name = "reference")]

public string Reference { get; set; }

[DataMember(Name = "subject")]

public string Subject { get; set; }

[DataMember(Name = "content")]

public string Content { get; set; }

}

//DataContract decoration necessary for serialization/deserialization to work properly

[DataContract]

public class ResponseBody

{

[DataMember(Name = "index")]

public string Index { get; set; }

[DataMember(Name = "references")]

public List<ResponseReference> References { get; set; }

}

//DataContract decoration necessary for serialization/deserialization to work properly

[DataContract]

public class ResponseReference

{

[DataMember(Name = "reference")]

public string Reference { get; set; }

[DataMember(Name = "id")]

public int Id { get; set; }

}

The results display

The code above takes care of getting the email data into our IDOL OnDemand text index, but now we need an easy way to execute the find similar queries from a web resource on the email form. This is actually incredibly easy to do with a little bit of JavaScript and jQuery.

To keep things simple, we’re going to make an assumption that all CRM emails will be in the IDOL OnDemand index by the time an agent opens one, so we can tell IDOL OnDemand to find similar records using the email’s activityid field, which we are populating as the index reference field in the code above. This also means we can embed the web resource on the email form without having to use any custom code because the Dynamics CRM form designer lets us specify that an iframe will be passed the record id when the form loads.

If turned out that we couldn’t count on the emails always being indexed before the agents viewed them, we could search the index using the relevant fields from an email record instead of using an index reference, but this would require some changes to what I’m showing today.

Our web resource will do the following:

1. Load the jQuery library (in my sample, I’m using the Google CDN).
2. Parse the record id from the value supplied in the query string that is passed to the iframe.
3. Execute a jQuery "getJSON" call to IDOL OnDemand’s find similar API.
4. Loop through the output results and write them to the screen.

Let’s take a look at the code in detail.

First, we load jQuery:

<script src="//ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>

Next we need a function that can extract query string values:

function getParameterByName(name) {

name = name.replace(/[[]/, "\[").replace(/[]]/, "\]");

var regex = new RegExp("[\?&]" + name + "=([^&#]*)"),

results = regex.exec(location.search);

return results == null ? "" : decodeURIComponent(results[1].replace(/+/g, " "));

}

Using the getParameterByName function, we parse the email’s activity id and prepare if for querying IDOL OnDemand like so:

var entityId = getParameterByName('id');

//strip brackets because we indexed a string value of the guid without them

entityId = entityId.replace("{","").replace("}","");

Finally we send the find similar request and process the results:

var indexName = "lucas-test";
var apiKey = "f90f3fc7-0e36-40f0-b508-734f45fe9033";
$.getJSON("https://api.idolondemand.com/1/api/sync/findsimilar/v1?indexes="+indexName+"&apikey="+apiKey+"& summary=context&index\_reference="+entityId,function(data){
	var content = "";
	$(content).text("");
	$.each(data.documents, function(i,document){
		content += '<p><a href="/main.aspx?etn=email&id={' + document.reference + '}&newWindow=true&pagetype=entityrecord" target="_blank">' + document.title + '</a>';
		content += '<br />Score: ' + document.weight + '';
		content += '<br />' + document.summary + '</p>';
	});
	$(content).appendTo("#results");
});

This writes the results to a div element on the page with an id of "results." You’ll see that each result shows the email subject (stored as "title" in our index) as a hyperlink to the actual email record, a similarity score value (weight) and a summary of the similar text.

So that’s it for getting started with IDOL OnDemand’s find similar API for Microsoft Dynamics CRM data. Full code for this is in my CRM-IdolOnDemand-Tools repository on GitHub. Does this look like something you could use in your organization? If so, let us know your thoughts in the comments!

* If you want to learn more about IDOL OnDemand text indexes, I suggest you look at the text indexes section of the developer documentation.

A version of this post was originally published on the HP Enterprise Services Application Services blog.

In my last post I provided an overview of HP IDOL OnDemand and discussed a couple of things you can do with it to process and analyze data stored in a Microsoft Dynamics CRM instance. Today I'll provide a detailed walkthrough of how perform sentiment analysis on incoming emails using IDOL OnDemand. First let's consider a typical scenario for sentiment analysis.

Assume you manage a Microsoft Dynamics CRM system (online or on-premise) for an organization that provides customer service and support via a variety of channels including phone, email and live chat. The business manager has asked you to set up a mechanism that will categorize the sentiment of incoming emails sent to the main "support" address so that they can be assigned to different queues for handling by agents. Fortunately for you, IDOL OnDemand offers a sentiment analysis API!

This API parses a body of text and returns a text value for sentiment (positive, neutral or negative) and a decimal sentiment score between 1 and -1 (larger positive scores indicating positivity and larger negative scores indicating negativity)*. Let's look at how we can make use of that API inside Dynamics CRM.

The solution approach

Communicating with IDOL OnDemand from Dynamics CRM obviously requires custom code, but whether that code is called from a plug-in or workflow doesn't materially affect the code. For the purpose of this example, I'll be showing an approach that uses a custom workflow activity called from an asynchronous workflow. When an email is received, the workflow will pass the text of the email to the custom workflow activity, which will then call out to IDOL OnDemand's sentiment analysis API to retrieve the sentiment and score. The workflow will then write these values to custom fields on the email record.

The code

The custom workflow activity needs one input parameter for email text and two output parameters for sentiment and score. Here's how we define them:

[Input("Text Input")]

public InArgument<String> TextInput { get; set; }

[Output("Sentiment")]

public OutArgument<String> Sentiment { get; set; }

[Output("Score")]

public OutArgument<Decimal> Score { get; set; }

We also need to set up two variables to help us call IDOL OnDemand. The first is the URL for the sentiment analysis API:

//address of the service to which you will post your json messages

private string _webAddress = "https://api.idolondemand.com/1/api/sync/analyzesentiment/v1";

The second is the API key to authorize our requests:

//idol ondemand api key

private string _apiKey = "XXXXXXXXXXXXXXXX";

Now that all the inputs and configuration parameters are set, you are ready to process the inbound email. We're going to do the following:

1. Check whether the email description field contains text.
2. If it does, strip HTML tags using a helper function.
3. Pass the "clean" text to IDOL OnDemand with an HttpWebRequest.
4. Deserialize the JSON respsonse returned by IDOL OnDemand to a custom class object (SentimentResponse) using a DataContractJsonSerializer.
5. Return the sentiment value and score to the calling workflow.

Here's the code that does all of this:

//get the text we want to analyze

string inputText = TextInput.Get(executionContext);

if (inputText != string.Empty)

{

//use a helper function to strip out HTML tags

inputText = HtmlTools.StripHTML(inputText);

//create the webrequest object and execute it (and post jsonmsg to it)

HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(_webAddress);

//set request content type so it is treated as a regular form post

req.ContentType = "application/x-www-form-urlencoded";

//set method to post

req.Method = "POST";

//create a stringbuilder to build our request

StringBuilder postData = new StringBuilder();

//set the apikey request value

postData.Append("apikey=" + System.Uri.EscapeDataString(_apiKey) + "&");

//set the text request value

postData.Append("text=" + System.Uri.EscapeDataString(inputText));

//create a stream

byte[] bytes = System.Text.Encoding.ASCII.GetBytes(postData.ToString());

req.ContentLength = bytes.Length;

System.IO.Stream os = req.GetRequestStream();

os.Write(bytes, 0, bytes.Length);

os.Close();

//get the response

System.Net.WebResponse resp = req.GetResponse();

//deserialize the response to a SentimentResponse object

SentimentResponse myResponse = new SentimentResponse();

System.Runtime.Serialization.Json.DataContractJsonSerializer deserializer = new System.Runtime.Serialization.Json.DataContractJsonSerializer(myResponse.GetType());

myResponse = deserializer.ReadObject(resp.GetResponseStream()) as SentimentResponse;

//set output values from the fields of the deserialzed myjsonresponse object

Score.Set(executionContext, myResponse.Aggregate.Score);

Sentiment.Set(executionContext, myResponse.Aggregate.Sentiment);

}

The StripHTML function is taken from a sample on CodeProject.

Here are the classes used for deserializing the JSON response:

//DataContract decoration necessary for serialization/deserialization to work properly

[DataContract]

public class SentimentResponse

{

//datamember name value indicates name of json field to which data will be serialized/from which data will be deserialized

[DataMember(Name = "positive")]

public List<SentimentEntity> Positive { get; set; }

[DataMember(Name = "negative")]

public List<SentimentEntity> Negative { get; set; }

[DataMember(Name = "aggregate")]

public SentimentAggregate Aggregate { get; set; }

public SentimentResponse() { }

}

//DataContract decoration necessary for serialization/deserialization to work properly

[DataContract]

public class SentimentAggregate

{

[DataMember(Name = "sentiment")]

public string Sentiment { get; set; }

[DataMember(Name = "score")]

public decimal Score { get; set; }

}

//DataContract decoration necessary for serialization/deserialization to work properly

[DataContract]

public class SentimentEntity

{

[DataMember(Name = "sentiment")]

public string Sentiment { get; set; }

[DataMember(Name = "topic")]

public string Topic { get; set; }

[DataMember(Name = "score")]

public decimal Score { get; set; }

[DataMember(Name = "original_text")]

public string OriginalText { get; set; }

[DataMember(Name = "normalized_text")]

public string NormalizedText { get; set; }

[DataMember(Name = "original_length")]

public int OriginalLength { get; set; }

[DataMember(Name = "normalized_length")]

public int NormalizedLength { get; set; }

}

Although the scenario for this example mentions email, you'll note that nothing in the code is specific to emails, so you can use this same code without modifications to perform sentiment analysis on any record in your Dynamics CRM system. Full code for this is in my CRM-IdolOnDemand-Tools repository on GitHub.

That's all for now. Join me next time to see how you can index Dynamics CRM records and execute "find similar" queries using IDOL OnDemand.

* Actually the sentiment analysis API also returns separate sentiment and score values for different phrases in the text, but we're going to disregard those here and only work with the overall sentiment and score values.

A version of this post was originally published on the HP Enterprise Services Application Services blog.

Inside any organization's Microsoft Dynamics CRM system, there is a wealth of raw data that can be turned into actionable intelligence if it can be effectively analyzed. This data can be found in emails, case notes, attachments and other records that are created and stored in the course of day-to-day business. In this three-part series, I will show how you can unlock this data using HP IDOL OnDemand's powerful text analysis capabilities. Specifically I will show how you can perform sentiment analysis on incoming emails and also how you can index Dynamics CRM records so that agents can easily see similar, relevant records. Before we get into the technical details, let's take a closer look at IDOL OnDemand.

What is HP IDOL OnDemand?

IDOL OnDemand is a collection of web service APIs that expose a subset of the functionality offered by HP Autonomy. These APIs allow developers to write code that can "understand and engage with human information, including text, images, social media, and app data." Currently IDOL OnDemand offers more than 20 different APIs including sentiment analysis, image recognition, face detection and text indexing/searching. A full list of IDOL OnDemand APIs is available here: https://www.idolondemand.com/developer/apis. Despite the breadth of functionality available in IDOL OnDemand, there are definite differences between it and the on-premise HP IDOL version. You can read more about those differences here.

How do we connect Dynamics CRM and IDOL OnDemand?

Because all IDOL OnDemand APIs are RESTful web services that return responses as JSON-formatted messages, custom workflow activities or plug-ins can communicate with IDOL OnDemand via the System.Net.HttpWebRequest class, but first you need to sign up for an IDOL OnDemand developer account at https://www.idolondemand.com/signup.html. Once you sign up for a developer account, you'll be given an API key that grants access to run IDOL OnDemand APIs. I'll be showing detailed code examples in parts two and three of this series, but in the meantime I suggest you go ahead and take a look at the IDOL OnDemand getting started documentation.

That's all for today. In my next post, I'll show how to analyze inbound emails for customer sentiment. See you then!

A version of this post was originally published on the HP Enterprise Services Application Services blog.