Saturday, May 25, 2024
HomeBig DataHow you can Do Load Testing with Rockset

How you can Do Load Testing with Rockset

What’s load testing and why does it matter?


Load testing is a important course of for any database or information service, together with Rockset. By doing load testing, we intention to evaluate the system’s conduct beneath each regular and peak situations. This course of helps in evaluating vital metrics like Queries Per Second (QPS), concurrency, and question latency. Understanding these metrics is important for sizing your compute assets accurately, and guaranteeing that they’ll deal with the anticipated load. This, in flip, helps in attaining Service Degree Agreements (SLAs) and ensures a easy, uninterrupted consumer expertise. That is particularly vital for customer-facing use instances, the place finish customers count on a quick consumer expertise. Load testing is usually additionally referred to as efficiency or stress testing.

“53% of visits are more likely to be deserted if pages take longer than 3 seconds to load” — Google

Rockset compute assets (referred to as digital cases or VIs) come in numerous sizes, starting from Small to 16XL, and every measurement has a predefined variety of vCPUs and reminiscence accessible. Selecting an acceptable measurement is dependent upon your question complexity, dataset measurement and selectivity of your queries, variety of queries which can be anticipated to run concurrently and goal question efficiency latency. Moreover, in case your VI can be used for ingestion, it is best to consider assets wanted to deal with ingestion and indexing in parallel to question execution. Fortunately, we provide two options that may assist with this:

  • Auto-scaling – with this function, Rockset will robotically scale the VI up and down relying on the present load. That is vital when you have some variability in your load and/or use your VI to do each ingestion and querying.
  • Compute-compute separation – that is helpful as a result of you possibly can create VIs which can be devoted solely for operating queries and this ensures that the entire accessible assets are geared in direction of executing these queries effectively. This implies you possibly can isolate queries from ingest or isolate completely different apps on completely different VIs to make sure scalability and efficiency.

We suggest doing load testing on at the least two digital cases – with ingestion operating on the primary VI and on a separate question VI. This helps with deciding on a single or multi-VI structure.

Load testing helps us determine the boundaries of the chosen VI for our specific use case and helps us choose an acceptable VI measurement to deal with our desired load.

Instruments for load testing

With regards to load testing instruments, a number of well-liked choices are JMeter, k6, Gatling and Locust. Every of those instruments has its strengths and weaknesses:

  • JMeter: A flexible and user-friendly instrument with a GUI, excellent for varied forms of load testing, however will be resource-intensive.
  • k6: Optimized for prime efficiency and cloud environments, utilizing JavaScript for scripting, appropriate for builders and CI/CD workflows.
  • Gatling: Excessive-performance instrument utilizing Scala, greatest for complicated, superior scripting situations.
  • Locust: Python-based, providing simplicity and fast script growth, nice for simple testing wants.

Every instrument gives a singular set of options, and the selection is dependent upon the precise necessities of the load check being performed. Whichever instrument you utilize, you’ll want to learn by way of the documentation and perceive the way it works and the way it measures the latencies/response occasions. One other good tip is to not combine and match instruments in your testing – if you’re load testing a use case with JMeter, keep it up to get reproducible and reliable outcomes which you can share along with your staff or stakeholders.

Rockset has a REST API that can be utilized to execute queries, and all instruments listed above can be utilized to load check REST API endpoints. For this weblog, I’ll deal with load testing Rockset with Locust, however I’ll present some helpful assets for JMeter, k6 and Gatling as nicely.

Establishing Rockset and Locust for load testing

Let’s say we’ve got a pattern SQL question that we need to check and our information is ingested into Rockset. The very first thing we often do is convert that question right into a Question Lambda – this makes it very simple to check that SQL question as a REST endpoint. It may be parametrized and the SQL will be versioned and saved in a single place, as an alternative of going backwards and forwards and altering your load testing scripts each time you might want to change one thing within the question.

Step 1 – Determine the question you need to load check

In our situation, we need to discover the preferred product on our webshop for a selected day. That is what our SQL question appears like (be aware that :date is a parameter which we are able to provide when executing the question):

--top product for a selected day
    MAX_BY(p.ProductName, s.Depend) AS ProductName,
    MAX(s.Depend) AS NumberOfClicks
    "Demo-Ecommerce".ProductStatsAlias s
    INNER JOIN "Demo-Ecommerce".ProductsAlias p ON s.ProductID = CAST(p._id AS INT)
    s.Date = :date
    1 DESC;


Step 2 – Save your question as a Question Lambda

We’ll save this question as a question lambda referred to as LoadTestQueryLambda which can then be accessible as a REST endpoint:

curl --request POST 
-H "Authorization: ApiKey $ROCKSET_APIKEY" 
-H 'Content material-Kind: utility/json' 
  -d '{
    "parameters": [
        "name": "days",
        "type": "int",
        "value": "1"
      "virtual_instance_id": "<your digital occasion ID>"
 | python -m json.instrument

Step 3 – Generate your API key

Now we have to generate an API key, which we’ll use as a means for our Locust script to authenticate itself to Rockset and run the check. You possibly can create an API key simply by way of our console or by way of the API.

Step 4 – Create a digital occasion for load testing

Subsequent, we want the ID of the digital occasion we need to load check. In our situation, we need to run a load check in opposition to a Rockset digital occasion that’s devoted solely to querying. We spin up a further Medium digital occasion for this:


As soon as the VI is created, we are able to get its ID from the console:


Step 5 – Set up Locust

Subsequent, we’ll set up and arrange Locust. You are able to do this in your native machine or a devoted occasion (assume EC2 in AWS).

$ pip set up locust

Step 6 – Create your Locust check script

As soon as that’s executed, we’ll create a Python script for the Locust load check (be aware that it expects a ROCKSET_APIKEY surroundings variable to be set which is our API key from step 3).

We will use the script under as a template:

import os
from locust import HttpUser, process, tag
from random import randrange

class query_runner(HttpUser):
    ROCKSET_APIKEY = os.getenv('ROCKSET_APIKEY') # API key's an surroundings variable

    header = {"authorization": "ApiKey " + ROCKSET_APIKEY}

    def on_start(self):
        self.headers = {
            "Authorization": "ApiKey " + self.ROCKSET_APIKEY,
            "Content material-Kind": "utility/json"
        self.consumer.headers = self.headers"" # substitute this along with your area's URI
        self.consumer.base_url =
        self.vi_id = '<your digital occasion ID>' # substitute this along with your VI ID

    def LoadTestQueryLambda(self):
        # utilizing default params for now
        information = {
            "virtual_instance_id": self.vi_id
        target_service="/ws/sandbox/lambdas/LoadTestQueryLambda/tags/newest" # substitute this along with your question lambda
        consequence = self.consumer.publish(

Step 7 – Run the load check

As soon as we set the API key surroundings variable, we are able to run the Locust surroundings:

export ROCKSET_APIKEY=<your api key>
locust -f --host

And navigate to: http://localhost:8089 the place we are able to begin our Locust load check:


Let’s discover what occurs as soon as we hit the Begin swarming button:

  1. Initialization of simulated customers: Locust begins creating digital customers (as much as the quantity you specified) on the charge you outlined (the spawn charge). These customers are cases of the consumer class outlined in your Locust script. In our case, we’re beginning with a single consumer however we are going to then manually improve it to five and 10 customers, after which go down to five and 1 once more.
  2. Job execution: Every digital consumer begins executing the duties outlined within the script. In Locust, duties are sometimes HTTP requests, however they are often any Python code. The duties are picked randomly or primarily based on the weights assigned to them (if any). We have now only one question that we’re executing (our LoadTestQueryLambda).
  3. Efficiency metrics assortment: Because the digital customers carry out duties, Locust collects and calculates efficiency metrics. These metrics embody the variety of requests made, the variety of requests per second, response occasions, and the variety of failures.
  4. Actual-time statistics replace: The Locust internet interface updates in real-time, displaying these statistics. This contains the variety of customers at the moment swarming, the request charge, failure charge, and response occasions.
  5. Take a look at scalability: Locust will proceed to spawn customers till it reaches the whole quantity specified. It ensures the load is elevated step by step as per the desired spawn charge, permitting you to watch how the system efficiency adjustments because the load will increase. You possibly can see this within the graph under the place the variety of customers begins to develop to five and 10 after which go down once more.
  6. Person conduct simulation: Digital customers will watch for a random time between duties, as outlined by the wait_time within the script. This simulates extra lifelike consumer conduct. We didn’t do that in our case however you are able to do this and extra superior issues in Locust like customized load shapes, and so forth.
  7. Steady check execution: The check will proceed operating till you determine to cease it, or till it reaches a predefined length in case you’ve set one.
  8. Useful resource utilization: Throughout this course of, Locust makes use of your machine’s assets to simulate the customers and make requests. It is vital to notice that the efficiency of the Locust check can even rely on the assets of the machine it is operating on.

Let’s now interpret the outcomes we’re seeing.

Deciphering and validating load testing outcomes

Deciphering outcomes from a Locust run includes understanding key metrics and what they point out in regards to the efficiency of the system beneath check. Listed below are a few of the essential metrics offered by Locust and the right way to interpret them:

  • Variety of customers: The whole variety of simulated customers at any given level within the check. This helps you perceive the load degree in your system. You possibly can correlate system efficiency with the variety of customers to find out at what level efficiency degrades.
  • Requests per second (RPS): The variety of requests (queries) made to your system per second. The next RPS signifies a better load. Evaluate this with response occasions and error charges to evaluate if the system can deal with concurrency and excessive visitors easily.
  • Response time: Normally displayed as common, median, and percentile (e.g., ninetieth and 99th percentile) response occasions. You’ll doubtless take a look at median and the 90/99 percentile as this offers you the expertise for “most” customers – solely 10 or 1 p.c could have worse expertise.
  • Failure charge: The share or variety of requests that resulted in an error. A excessive failure charge signifies issues with the system beneath check. It is essential to investigate the character of those errors.

Under you possibly can see the whole RPS and response occasions we achieved beneath completely different masses for our load check, going from a single consumer to 10 customers after which down once more.


Our RPS went as much as about 20 whereas sustaining median question latency under 300 milliseconds and P99 of 700 milliseconds.


We will now correlate these information factors with the accessible digital occasion metrics in Rockset. Under, you possibly can see how the digital occasion handles the load when it comes to CPU, reminiscence and question latency. There’s a correlation between variety of customers from Locust and the peaks we see on the VI utilization graphs. You can too see the question latency beginning to rise and see the concurrency (requests or queries per second) go up. The CPU is under 75% on the height and reminiscence utilization appears secure. We additionally don’t see any important queueing taking place in Rockset.


Other than viewing these metrics within the Rockset console or by way of our metrics endpoint, it’s also possible to interpret and analyze the precise SQL queries that had been operating, what was their particular person efficiency, queue time, and so forth. To do that, we should first allow question logs after which we are able to do issues like this to determine our median run and queue occasions:

    COUNT(*) as rely,
    ARRAY_SORT(ARRAY_AGG(runtime_ms)) [(COUNT(*) + 1) / 2] as median_runtime,
    ARRAY_SORT(ARRAY_AGG(queued_time_ms)) [(COUNT(*) + 1) / 2] as median_queue_time
    vi_id = '<your digital occasion ID>'
    AND _event_time > TIMESTAMP '2023-11-24 09:40:00'

We will repeat this load check on the primary VI as nicely, to see how the system performs ingestion and runs queries beneath load. The method could be the identical, we might simply use a distinct VI identifier in our Locust script in Step 6.


In abstract, load testing is a crucial a part of guaranteeing the reliability and efficiency of any database answer, together with Rockset. By choosing the best load testing instrument and organising Rockset appropriately for load testing, you possibly can acquire useful insights into how your system will carry out beneath varied situations.

Locust is straightforward sufficient to get began with rapidly, however as a result of Rockset has REST API help for executing queries and question lambdas, it’s simple to hook up any load testing instrument.

Bear in mind, the objective of load testing isn’t just to determine the utmost load your system can deal with, but in addition to know the way it behaves beneath completely different stress ranges and to make sure that it meets the required efficiency requirements.

Fast load testing ideas earlier than we finish the weblog:

  • At all times load check your system earlier than going to manufacturing
  • Use question lambdas in Rockset to simply parametrize, version-control and expose your queries as REST endpoints
  • Use compute-compute separation to carry out load testing on a digital occasion devoted for queries, in addition to in your essential (ingestion) VI
  • Allow question logs in Rockset to maintain statistics of executed queries
  • Analyze the outcomes you’re getting and evaluate them in opposition to your SLAs – in case you want higher efficiency, there are a number of methods on the right way to deal with this, and we’ll undergo these in a future weblog.

Have enjoyable testing 💪

Helpful assets

Listed below are some helpful assets for JMeter, Gatling and k6. The method is similar to what we’re doing with Locust: you might want to have an API key and authenticate in opposition to Rockset after which hit the question lambda REST endpoint for a selected digital occasion.



Please enter your comment!
Please enter your name here

Most Popular

Recent Comments