Using IDOL OnDemand for text analysis in Dynamics CRM - part 2

by Lucas Alexander

In my last post I provided an overview of HP IDOL OnDemand and discussed a couple of things you can do with it to process and analyze data stored in a Microsoft Dynamics CRM instance. Today I'll provide a detailed walkthrough of how perform sentiment analysis on incoming emails using IDOL OnDemand. First let's consider a typical scenario for sentiment analysis.

Assume you manage a Microsoft Dynamics CRM system (online or on-premise) for an organization that provides customer service and support via a variety of channels including phone, email and live chat. The business manager has asked you to set up a mechanism that will categorize the sentiment of incoming emails sent to the main "support" address so that they can be assigned to different queues for handling by agents. Fortunately for you, IDOL OnDemand offers a sentiment analysis API!

This API parses a body of text and returns a text value for sentiment (positive, neutral or negative) and a decimal sentiment score between 1 and -1 (larger positive scores indicating positivity and larger negative scores indicating negativity)*. Let's look at how we can make use of that API inside Dynamics CRM.

The solution approach

Communicating with IDOL OnDemand from Dynamics CRM obviously requires custom code, but whether that code is called from a plug-in or workflow doesn't materially affect the code. For the purpose of this example, I'll be showing an approach that uses a custom workflow activity called from an asynchronous workflow. When an email is received, the workflow will pass the text of the email to the custom workflow activity, which will then call out to IDOL OnDemand's sentiment analysis API to retrieve the sentiment and score. The workflow will then write these values to custom fields on the email record.

The code

The custom workflow activity needs one input parameter for email text and two output parameters for sentiment and score. Here's how we define them:

[Input("Text Input")]


public InArgument<String> TextInput { get; set; }


[Output("Sentiment")]


public OutArgument<String> Sentiment { get; set; }


[Output("Score")]


public OutArgument<Decimal> Score { get; set; }

We also need to set up two variables to help us call IDOL OnDemand. The first is the URL for the sentiment analysis API:

//address of the service to which you will post your json messages


private string _webAddress = "https://api.idolondemand.com/1/api/sync/analyzesentiment/v1";

The second is the API key to authorize our requests:

//idol ondemand api key


private string _apiKey = "XXXXXXXXXXXXXXXX";

Now that all the inputs and configuration parameters are set, you are ready to process the inbound email. We're going to do the following:

  1. Check whether the email description field contains text.
  2. If it does, strip HTML tags using a helper function.
  3. Pass the "clean" text to IDOL OnDemand with an HttpWebRequest.
  4. Deserialize the JSON respsonse returned by IDOL OnDemand to a custom class object (SentimentResponse) using a DataContractJsonSerializer.
  5. Return the sentiment value and score to the calling workflow.

Here's the code that does all of this:

//get the text we want to analyze


string inputText = TextInput.Get(executionContext);


if (inputText != string.Empty)


{


//use a helper function to strip out HTML tags


inputText = HtmlTools.StripHTML(inputText);


//create the webrequest object and execute it (and post jsonmsg to it)


HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(_webAddress);


//set request content type so it is treated as a regular form post


req.ContentType = "application/x-www-form-urlencoded";


//set method to post


req.Method = "POST";


//create a stringbuilder to build our request


StringBuilder postData = new StringBuilder();


//set the apikey request value


postData.Append("apikey=" + System.Uri.EscapeDataString(_apiKey) + "&");


//set the text request value


postData.Append("text=" + System.Uri.EscapeDataString(inputText));


//create a stream


byte[] bytes = System.Text.Encoding.ASCII.GetBytes(postData.ToString());


req.ContentLength = bytes.Length;


System.IO.Stream os = req.GetRequestStream();


os.Write(bytes, 0, bytes.Length);


os.Close();


//get the response


System.Net.WebResponse resp = req.GetResponse();


//deserialize the response to a SentimentResponse object


SentimentResponse myResponse = new SentimentResponse();


System.Runtime.Serialization.Json.DataContractJsonSerializer deserializer = new System.Runtime.Serialization.Json.DataContractJsonSerializer(myResponse.GetType());


myResponse = deserializer.ReadObject(resp.GetResponseStream()) as SentimentResponse;


//set output values from the fields of the deserialzed myjsonresponse object


Score.Set(executionContext, myResponse.Aggregate.Score);


Sentiment.Set(executionContext, myResponse.Aggregate.Sentiment);


}

The StripHTML function is taken from a sample on CodeProject.

Here are the classes used for deserializing the JSON response:

//DataContract decoration necessary for serialization/deserialization to work properly


[DataContract]


public class SentimentResponse


{


//datamember name value indicates name of json field to which data will be serialized/from which data will be deserialized


[DataMember(Name = "positive")]


public List<SentimentEntity> Positive { get; set; }


[DataMember(Name = "negative")]


public List<SentimentEntity> Negative { get; set; }


[DataMember(Name = "aggregate")]


public SentimentAggregate Aggregate { get; set; }


public SentimentResponse() { }


}


//DataContract decoration necessary for serialization/deserialization to work properly


[DataContract]


public class SentimentAggregate


{


[DataMember(Name = "sentiment")]


public string Sentiment { get; set; }


[DataMember(Name = "score")]


public decimal Score { get; set; }


}


//DataContract decoration necessary for serialization/deserialization to work properly


[DataContract]


public class SentimentEntity


{


[DataMember(Name = "sentiment")]


public string Sentiment { get; set; }


[DataMember(Name = "topic")]


public string Topic { get; set; }


[DataMember(Name = "score")]


public decimal Score { get; set; }


[DataMember(Name = "original_text")]


public string OriginalText { get; set; }


[DataMember(Name = "normalized_text")]


public string NormalizedText { get; set; }


[DataMember(Name = "original_length")]


public int OriginalLength { get; set; }


[DataMember(Name = "normalized_length")]


public int NormalizedLength { get; set; }


}

Although the scenario for this example mentions email, you'll note that nothing in the code is specific to emails, so you can use this same code without modifications to perform sentiment analysis on any record in your Dynamics CRM system. Full code for this is in my CRM-IdolOnDemand-Tools repository on GitHub.

That's all for now. Join me next time to see how you can index Dynamics CRM records and execute "find similar" queries using IDOL OnDemand.

* Actually the sentiment analysis API also returns separate sentiment and score values for different phrases in the text, but we're going to disregard those here and only work with the overall sentiment and score values.

A version of this post was originally published on the HP Enterprise Services Application Services blog.