Load testing a PollingDuplexHttp WCF service-Part 1

One of the challenges of writing services that are designed to be consumed by hundreds (or thousands) of clients at once is testing under any sort of load.  Sure, you can test every method to see how it responds one at a time, and maybe you’ve got three or four testers hitting it at once, but how do you test the real world load (or even the extreme load) of a whole pile of clients using the service at the same time?  The simple solution would appear to be to just create a simple load test client that creates multiple threads and uses each of these threads to create connections to the service.  Sure, the scenario isn’t completely accurate, since you’re going to create some degree of latency on the client with a multithreaded app, but it’s probably going to give you a good idea of where the bottlenecks exist.

This was the approach I took with a PollingDuplexHttp service, and since the binding was only available in Silverlight, the load test client needed to be a silverlight app as well.  The structure of a basic load test looked something like this:

image

The test application would start a whole pile of threads, connect to the service, and then time how long it takes to invoke a method.  This basic idea worked, but it turns out that there’s a throttle that gets in the way.

MaxSessionsPerAddress

The MaxSessionsPerAddress throttle will come up if you enable tracing on your service (which is well documented here), and it caps out at 10 sessions.  I couldn’t find this in any of the documentation, but after digging a little deeper, I found that it’s a non-configurable throttle designed to prevent DOS attacks.  In normal usage, this makes sense, but in this case, it makes things more complicated.

Load tester v2

To get around this problem, I morphed the load tester into something that looks more like this:

image

The test controller launches N Silverlight applications, which each run T threads.  Tests are loaded from a test library DLL, and results are recorded in a database using another simple service.  This way, I can run hundreds of clients from one controller, and run multiple controllers on multiple machines to generate higher load.  Each of the Silverlight projects can take everything as URI parameters to initialize the tests (test name,number of threads, synchronized start time etc), so it’s pretty simple to seed this to multiple machines to start at a predetermined time.  Yes, it’s not a full on enterprise app architecture, but it’s something I put together in a day, and it did its job.

Within the load tester, I’ve created a load test library that collects all the test methods.  My load test object looks like this:

public class LoadTest
{

    public string TestName { get; set; }
    public DateTime TestStartTime { get { return m_testStartTime; } set { m_testStartTime = value; } }
    private DateTime m_testStartTime = new DateTime();
    public DateTime TestEndTime { get { return m_testEndTime; } set { m_testEndTime = value; } }
    private DateTime m_testEndTime;
    public String ResultString { get { return m_resultString; } set { m_resultString = value; } }
    private string m_resultString;
    public TimeSpan Duration
    {
        get { return TestEndTime - TestStartTime; }
    }

    public int RetryCount { get; set; }
    public int MaxRetries { get; set; }

    public ManualResetEvent DoneEvent = new ManualResetEvent(false);

    public Func<LoadTest,string> TestFn;
    public DateTime StartTime { get { return m_startTime; } set { m_startTime = value; } }
    private DateTime m_startTime = new DateTime();  //used to coordinate start across N threads
    public LoadTest(Func<LoadTest,string> fn)
    {
        TestFn = fn;
        m_thread = new Thread(new ThreadStart(ThreadFn));
        m_thread.Start();
        StartTime = DateTime.Now;
    }

    public string AddRetryStatus(string msg)
    {
        RetryCount++;
        return "\r\ngot error-**retrying(" + RetryCount + "):" + msg;
    }

    public ManualResetEvent TestDone = new ManualResetEvent(false);

    public static iceService.IiceServiceClient CreateProxy()
    {
        EndpointAddress address = new EndpointAddress("https://chrislaptop.corp.computer-talk.com:8085/iceWCFService");
        PollingDuplexHttpBinding b = new PollingDuplexHttpBinding(PollingDuplexHttpSecurityMode.TransportWithMessageCredential, System.ServiceModel.Channels.PollingDuplexMode.MultipleMessagesPerPoll);

        b.InactivityTimeout = new TimeSpan(0, 10, 0);
        b.CloseTimeout = new TimeSpan(0, 10, 0);
        b.SendTimeout = new TimeSpan(0, 10, 0);
        b.ReceiveTimeout = new TimeSpan(0, 10, 0);

        b.MaxReceivedMessageSize = long.MaxValue;

        //need to register to use the client HTTP stack, and not the browser stack
        //as per http://blogs.msdn.com/b/silverlightws/archive/2010/12/15/pollingduplex-using-multiplemessagesperpoll-issue-in-latest-sl4-gdrs.aspx
        //since there's a Silverlight bug...

        bool httpResult = WebRequest.RegisterPrefix("http://", WebRequestCreator.ClientHttp);
        bool httpsResult = WebRequest.RegisterPrefix("https://", WebRequestCreator.ClientHttp);

        service.IServiceClient proxy = new service.IServiceClient(b, address);
        return proxy;
    }

    private void ThreadFn()
    {
        TestStartTime = DateTime.Now;
        ResultString = TestFn.Invoke(this);  //do whatever the test method says
        TestEndTime = DateTime.Now;
        TestDone.Set();
    }
    private System.Threading.Thread m_thread;

To use this class, I construct objects with delegates like this:

public static LoadTest DemoTest()
{
    LoadTest t1 = new LoadTest((t) =>
    {
        try
        {
            string returnText = "";
            service.IServiceClient c = LoadTest.CreateProxy();

            c.TestMethodCompleted += (sndr, ea) =>
            {
                if (ea.Error != null)
                {
                    returnText += t.AddRetryStatus(ea.Error.Message);
                    Thread.Sleep(1000);
                    c.TestMethodAsync();
                }
                else
                {
                    if (ea.Result != null)
                        returnText += "\r\n Got Result " + ea.Result.ToString();
                    else
                        returnText += "\r\n return is null!";
                    t.DoneEvent.Set();
                }
            };

            c.TestMethodAsync();
            t.DoneEvent.WaitOne();
            return returnText;
        }
        catch (Exception ex)
        {
            return "Got Exception: " + ex.ToString();
        }
    });
    t1.TestName = "DemoTest";
    return t1;
}

The framework then takes care of starting threads, submitting results, and all the other plumbing that a tester like this would need.

While this testing approach is convenient, it does have some drawbacks:

1. Having such a large number of requests initiating from the same machine will cause load on the client side as well. 10 test runners each running 10 threads will likely not respond as well on the client side as 100 individual clients. This may cause execution times to be higher than actual performance.

2. Creating the proxy for the first time is a longer operation than using an existing proxy. If a test in the library creates a proxy, runs a single method, and then closes the proxy, the establishment time will be a factor in the test. This is demonstrated by the test results.

3. This framework does not take into account or control for network conditions between the client and the service being tested.

4. The probability that under normal operation that many clients would be creating sessions at once is low (likely only in a case where a network connection is broken).

5.Probably something else I didn’t think of…

Of course, even with these in mind, I’m still able to get a worst case scenario for the service, and see how different scenarios stack up relative to each other.

Obviously, I’ve left out a lot of details about how to build a load tester, but this should give you a basic idea of what I put together.  I’ll post a follow up to this later in the week on some of the results that I came away with.

Advertisements
This entry was posted in Silverlight, WCF. Bookmark the permalink.

One Response to Load testing a PollingDuplexHttp WCF service-Part 1

  1. Pingback: Binding customization and throttles-Load testing a PollingDuplexHttp WCF service-Part 2 | Chris Bardon

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s