Methodology

Load testing was performed against our preview environment using scrubbed data from real interactions with the chatbot. All conversations included at least five guest interactions to simulate actual engagement. All client/server interactions were accounted for and reproduced faithfully, including non-critical functionality such as rating responses.

Differences from Production

The most notable difference with load testing was using a zero second interval between polling calls. Normally polling is performed in one second intervals, though we decided to remove the delay between calls so the entire test could run within time limits. Being faithful to the source data, the same number of polling calls are made, however, these are now back-to-back with now pause between. This - in our opinion - will produce equal, if not greater, stress on the server.

Another challenge was scaling the application in a natural manner. The chatbot server, being hosted in Google App Engine, has built in scaling capabilities. This scaling handles normal web traffic well, where traffic will ramp up over a period of time. The testing performed, in contrast, spiked load in a less-than-natural manner, resulting in "hard starts". To counteract the spikes, we tuned the scaling parameters so that we had more idle instances to "smooth out" the initial spike. Some of the changes are candidates for inclusion in the mainline code base.

We also encountered some issues with our Dialogflow projects. All our environments were configured for Dialogflow Essentials Edition, which had an upgraded quota of 3,000 text queries per minute, which surpassed. As such, we submitted a request to have this quota increased to 6,000 per minute which allowed us to proceed without the results being tainted by errors.

Lastly, we increased capacity of the database in our preview environment. This was to determine the role the database played in scalability. The result was favourable and will be very likely implemented in production before any new chatbots are added to the platform. 

Results

Overview

During our test, we were able to simulate about 11,000 concurrent users

The primary optimization, adding client-side caching to OPTIONS calls, reduced the number of requests and thus the number of instances required

With 2.1, we had 1,713 created instances and 344 active instances. The optimizations slashed these number significantly.

As a direct result, we reduce the number of active database connections from about 2,600 to 660. 

The overall CPU utilization was well in the safe zone at 33%, meaning we should be able to scale even higher than observed.

Data

 Latency and Pending values are in seconds

Analysis

By caching the results of CORS Pre-flight requests, we were able to significantly decrease overhead. This, in turn, increased our concurrent user support by nearly 15%. Additionally, we've also reduced demands on the database as a side effect.

One thing we did notice is that this wasn't a 100% successful test. There was one session that had failed to establish as the /v1/start call had been aborted as it took too long to warmup an instance to handle it. I believe this was the result of the servlet taking more than ten seconds to start, and as such, addressed the start up time in a future release.

Given that we were unable to cause any apparent bottleneck in the system (eg: database CPU utilization), one can assume that we've hit the limitation of our current load testing framework. We will need to determine a better approach for future load tests.