celedonpartners.com

Light-weight approach for increasing request throughput in Dynamics CRM

by | Jan 5, 2016 | Blog, Microsoft Dynamics | 1 comment

Performance should always be considered while developing a migration or integration. Generally speaking, the faster you can move the data the better. However, you must take into account the limitations of the platform in order to achieve the highest throughput. In the case of CRM, that means calling the Organization Service as efficiently as possible.

This post isn’t a comprehensive list of all the things you should consider while tuning the performance of your migration/integration. Instead, this post highlights a light-weight approach for running requests that you can use and build upon to enhance performance.

For this post we will be using the ExecuteMultipleRequest from CRM for batching and C# tasks for multi-threading. The ExecuteMultipleRequest does have the limitation of being throttled in CRM Online, so you should not combine ExecuteMultipleRequests with multi-threading if you are targeting CRM Online as you will receive concurrent fault errors.

C# Tasks – https://msdn.microsoft.com/en-us/library/system.threading.tasks.task(v=vs.110).aspx

Execute Multiple Request – https://msdn.microsoft.com/en-us/library/jj863631.aspx

In our sample we are going to create a simple console application that calls the Organization Service using different combinations of threading and batching. In order to run this sample you will need access to an instance of CRM 2013+. Make sure that you target a dev or sandbox environment. Also, it should be noted that this sample does not handle errors or write logs of any kind.

To get started, you will need to download the CRM SDK appropriate for your version of CRM – https://msdn.microsoft.com/en-us/library/hh547453(v=crm.8).aspx

Next, create a console application targeting at least the .NET 4.5 framework. Visual Studio 2012 or higher is recommended.

You will need to add the following references –

  • Xrm.Client (from SDK)
  • Xrm.Sdk (from SDK)
  • Runtime.Serialization (.NET)

In order to take advantage of the multi-threading piece you will need to update your console application’s app.config and increase the allowed concurrent web connections. Add this section inside the <configuration> node –

<system.net>
<connectionManagement>
<add address=”*” maxconnection=”1000″/>
</connectionManagement>
</system.net>

Now replace the standard Program.cs with the following code –

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using Microsoft.Xrm.Client;
using Microsoft.Xrm.Client.Services;
using Microsoft.Xrm.Sdk;
using Microsoft.Xrm.Sdk.Messages;
using Microsoft.Xrm.Sdk.Query;namespace BulkAccountUpload
{
class BulkTester
{
private const string SERVER_URL = “[SERVER_URL]”;
private const string USER_NAME = “[USER_NAME]”;
private const string PASSWORD = “[PASSWORD]”;public const int BATCH_SIZE = 100;// Controls how many items get passed into an ExecuteMultipleRequest
public const int LIST_BATCH_SIZE = 200; // Controls how many items get passed into a thread

static void Main(string[] args)
{
Console.WriteLine(“Bulk Upload testing started.”);
Console.WriteLine();

using (OrganizationService orgService = new OrganizationService(CrmConnection.Parse(string.Format(“Url={0};Username={1};Password={2}”, SERVER_URL, USER_NAME, PASSWORD))))
{
// 1: Single threaded with no execute multiple
CreateAccounts(orgService, false, false);
DeleteAccounts(orgService, false, false);

Console.WriteLine();

// 2: Multi-threaded with no execute multiple
CreateAccounts(orgService, false, true);
DeleteAccounts(orgService, false, true);

Console.WriteLine();

// 3: Single threaded with execute multiple
CreateAccounts(orgService, true, false);
DeleteAccounts(orgService, true, false);

Console.WriteLine();

// 4: Multi-threaded with execute multiple
CreateAccounts(orgService, true, true);
DeleteAccounts(orgService, true, true);

Console.WriteLine();
}

Console.WriteLine(“Bulk Upload testing complete. Press Enter key to exit.”);

Console.Read();
}

/// <summary>
/// Create 1000 dummy accounts – name starts with BulkTestUploadAccount
/// </summary>
/// <param name=”orgService”>Organization Service instance</param>
/// <param name=”useExecuteMultiple”>Specify whether to use ExecuteMultipleRequest to batch requests</param>
/// <param name=”useMultiThread”>Specify whether to run requests in multi-thread mode</param>
public static void CreateAccounts(IOrganizationService orgService, bool useExecuteMultiple, bool useMultiThread)
{
DateTime startTime = DateTime.Now;

Console.WriteLine(“Creating Accounts – StartTime: {0} UseExecuteMultiple:{1} UseMultiThread:{2}”, startTime, useExecuteMultiple, useMultiThread);

List<OrganizationRequest> requests = new List<OrganizationRequest>();

for (int i = 0; i < 1000; i++)
{
Entity account = new Entity(“account”);

// Account name will be something like BulkTestUploadAccount 0123
account[“name”] = “BulkTestUploadAccount ” + i.ToString().PadLeft(4, ‘0’);

CreateRequest request = new CreateRequest();
request.Target = account;

requests.Add(request);
}

RunRequests(orgService, requests, useExecuteMultiple, useMultiThread);

DateTime endTime = DateTime.Now;

Console.WriteLine(“Created Accounts – EndTime: {0} TotalTime: {1}”, endTime, endTime – startTime);
}

/// <summary>
/// Deletes all dummy accounts where name starts with BulkTestUploadAccount
/// </summary>
/// <param name=”orgService”>Organization Service instance</param>
/// <param name=”useExecuteMultiple”>Specify whether to use ExecuteMultipleRequest to batch requests</param>
/// <param name=”useMultiThread”>Specify whether to run requests in multi-thread mode</param>
public static void DeleteAccounts(IOrganizationService orgService, bool useExecuteMultiple, bool useMultiThread)
{
DateTime startTime = DateTime.Now;

Console.WriteLine(“Deleting Accounts – StartTime: {0} UseExecuteMultiple:{1} UseMultiThread:{2}”, startTime, useExecuteMultiple, useMultiThread);

QueryExpression deleteQuery = new QueryExpression(“account”);
deleteQuery.Criteria.AddCondition(“name”, ConditionOperator.BeginsWith, “BulkTestUploadAccount”);

DataCollection<Entity> entitiesToDelete = orgService.RetrieveMultiple(deleteQuery).Entities;

List<OrganizationRequest> requests = new List<OrganizationRequest>();

foreach (Entity entityToDelete in entitiesToDelete)
{
DeleteRequest request = new DeleteRequest();
request.Target = entityToDelete.ToEntityReference();

requests.Add(request);
}

RunRequests(orgService, requests, useMultiThread, useExecuteMultiple);

DateTime endTime = DateTime.Now;

Console.WriteLine(“Deleted Accounts – EndTime: {0} TotalTime: {1}”, endTime, endTime – startTime);
}

/// <summary>
/// Runs requests using parameters for batching and threading
/// </summary>
/// <param name=”orgService”>Organization Service instance</param>
/// <param name=”requests”>Requests to run</param>
/// <param name=”useExecuteMultiple”>Specify whether to use ExecuteMultipleRequest to batch requests</param>
/// <param name=”useMultiThread”>Specify whether to run requests in multi-thread mode</param>
public static void RunRequests(IOrganizationService orgService, List<OrganizationRequest> requests, bool useExecuteMultiple, bool useMultiThread)
{
// If we are using multi-threading we need to call MultiThreadRequests – that method will take care of the batching
if (useMultiThread && useExecuteMultiple)
{
MultiThreadRequests(orgService, requests, true);
}
else if (useMultiThread)
{
MultiThreadRequests(orgService, requests, false);
}
// Do batching with no threading
else if (useExecuteMultiple)
{
ExecuteMultipleRequests(orgService, requests);
}
// Just loop through all requests
else
{
ExecuteRequests(orgService, requests);
}
}

/// <summary>
/// Sets up and runs requests in multi-threaded mode
/// </summary>
/// <param name=”orgService”>Organization Service instance</param>
/// <param name=”requests”>Requests to run</param>
/// <param name=”useExecuteMultiple”>Specify whether to use ExecuteMultipleRequest to batch requests</param>
public static void MultiThreadRequests(IOrganizationService orgService, List<OrganizationRequest> requests, bool useExecuteMultiple)
{
// Split all requests into a multi-dimensional list – this is a list of lists containing OrganizationRequest items
// We want to hand off a list of requests to each thread so they can take care of larger chunks
List<List<OrganizationRequest>> splitRequestLists = SplitRequestList(requests);

// Create a list of function calls – these functions will be used by Tasks in RunTasks()
// We use bool so we specify if the list was processed
List<Func<bool>> requestActions = new List<Func<bool>>();

foreach (List<OrganizationRequest> requestList in splitRequestLists)
{
// Change method call for the list depending on batching parameter
if (useExecuteMultiple)
{
requestActions.Add(() => ExecuteMultipleRequests(orgService, requestList));
}
else
{
requestActions.Add(() => ExecuteRequests(orgService, requestList));
}
}

RunTasks(requestActions);
}

/// <summary>
/// Executes requests with no batching
/// </summary>
/// <param name=”orgService”>Organization Service instance</param>
/// <param name=”requests”>Requests to run</param>
/// <returns>Always true – operation was completed</returns>
public static bool ExecuteRequests(IOrganizationService orgService, List<OrganizationRequest> requests)
{
foreach (OrganizationRequest request in requests)
{
orgService.Execute(request);
}

return true;
}

/// <summary>
/// Executes requests with batching
/// </summary>
/// <param name=”orgService”>Organization Service instance</param>
/// <param name=”requests”>Requests to run</param>
/// <returns>Always true – operation was completed</returns>
public static bool ExecuteMultipleRequests(IOrganizationService orgService, List<OrganizationRequest> requests)
{
int requestCount = 0; // Current request count – if we fill up an ExecuteMultiple we want to run it
int batchCount = 1; // Current batch count – can be used for logging

// Get a refresh ExecuteMultiplRequest
ExecuteMultipleRequest executeMultipleRequest = CreateExecuteMultipleRequest();

if (requests.Count > 0)
{
// Go through all the requests from the list
// The requests will be executed once enough requests fill up the ExecuteMultipleRequest or we’ve run out of requests
foreach (OrganizationRequest request in requests)
{
executeMultipleRequest.Requests.Add(request);

requestCount++;

// Have we reached our batch size or did we already hit the total count?
// Only fire if there are actually requests
if ((executeMultipleRequest.Requests.Count == BATCH_SIZE || requestCount == requests.Count)
&& executeMultipleRequest.Requests.Count > 0)
{
// Run all requests
ExecuteMultipleResponse response = orgService.Execute(executeMultipleRequest) as ExecuteMultipleResponse;

// This response will be faulted if there is at least one error
if (response.IsFaulted)
{
// TODO: Handle errors
}

// Reset ExecuteMultiple for next batch
executeMultipleRequest = CreateExecuteMultipleRequest();

batchCount++;
}
}
}

return true;
}

/// <summary>
/// Generic function that runs a list of methods in multi-threaded mode. Waits for all methods to finish before returning.
/// </summary>
/// <param name=”actions”>Functions to call</param>
public static void RunTasks(List<Func<bool>> actions)
{
Task[] tasks = new Task[actions.Count];

// Loop through all functions and create a new Task for them
for (int taskIndex = 0; taskIndex < tasks.Length; taskIndex++)
{
int localIndex = taskIndex;

tasks[localIndex] = Task.Factory.StartNew(() => actions[localIndex]());
}

List<AggregateException> exceptions = new List<AggregateException>();

try
{
// Function will only exit once all Tasks are complete
Task.WaitAll(tasks);
}
catch (AggregateException ex)
{
exceptions.Add(ex);
}

// TODO: Handle errors in exceptions list
}

/// <summary>
/// Creates a fresh ExecuteMultiplRequest with standard defaults – ContinueOnError and ReturnResponses
/// </summary>
/// <returns>ExecuteMultipleRequest ready for use</returns>
public static ExecuteMultipleRequest CreateExecuteMultipleRequest()
{
ExecuteMultipleRequest executeMultipleRequest = new ExecuteMultipleRequest();
executeMultipleRequest.Requests = new OrganizationRequestCollection();

executeMultipleRequest.Settings = new ExecuteMultipleSettings()
{
ContinueOnError = true,
ReturnResponses = true
};

return executeMultipleRequest;
}

/// <summary>
/// Splits up a large list of OrganizationRequest items into lists that will be used for batching
/// </summary>
/// <param name=”requests”>Requests to run</param>
/// <returns>List of Lists of OrganizationRequest items</returns>
public static List<List<OrganizationRequest>> SplitRequestList(List<OrganizationRequest> requests)
{
List<List<OrganizationRequest>> splitRequestList = new List<List<OrganizationRequest>>();

// We always round up to the next whole number to ensure all requests are accounted for
int listCount = (int)Math.Ceiling((double)requests.Count / LIST_BATCH_SIZE);

int currentListIndex = 0;

// Add empty lists
for (int i = 0; i < listCount; i++)
{
splitRequestList.Add(new List<OrganizationRequest>());
}

// Fill empty lists with requests up to the threshold
foreach (OrganizationRequest request in requests)
{
// Increment the listIndex if the list is full
if (splitRequestList[currentListIndex].Count == LIST_BATCH_SIZE)
{
currentListIndex++;
}

splitRequestList[currentListIndex].Add(request);
}

return splitRequestList;
}
}
}

You will want to format the document after pasting in the code for readability. Also, you will notice that the Organization Service used here is the Xrm.Client version. Make sure to change the SERVER_URL, USER_NAME, and PASSWORD constants before running the application.

Running the application will create and delete 1000 accounts using a combination of multi-threading and batching. At the end of each operation it will output the total time it took for each operation.

The beginning of the operation is always the same – a list of requests will be created that are then fed into the other methods. Depending on the parameters for batching/multi-threading, the list will be cut up into smaller lists and then distributed amongst multiple threads. These requests will then be batched depending on if the ExecuteMultipleRequest is being used.

You will notice that the result times improve when multi-threading and batching are added to the equation. However, you will eventually hit a max throughput that will be dependent on various factors including – the amount of fields per record, network speed, CRM Online vs On-Premise, machine resources (RAM/CPU), etc.

In general, you will see a noticeable increase in performance with just a few changes to how you call the Organization Service. However, you will still want to make sure you understand the implications of how the addition of batching and multi-threading affects your data. For example, if you are importing accounts, where the Parent Account field is filled out, you will need to do it two passes with creates first followed by updates.

For CRM On-Premise the best option is probably to use a combination of batching and multi-threading. For CRM Online you will either want to use batching on a single thread, making sure that the operation runs at a time when it’s unlikely for another ExecuteMultipleRequest to run at the same time, or do multi-threading without batching.

 

1 Comment

  1. Sean on 01/05/2016 at 8:12 pm

    Hi, a couple of tips
    1. Organization Service proxy is *not* thread-safe, however you can cache and re-use IServiceManagement (but you’ll have to create a new service proxy per thread, and using the same IServiceManagement will help speed that up – you can also keep that proxy around on the same thread. It just won’t work on two threads at the same time).

    2. for easier multi-threading extension methods check out the PFE Core, we also have a pattern for how to handle multi-threading that could be looked at/borrowed:
    – https://pfexrmcore.codeplex.com/
    – https://www.nuget.org/packages/Microsoft.Pfe.Xrm.CoreV7