Points load tester should consider

From last 6-7 months, I was working closely on load testing of a couple of application. While working on it I realized few capabilities a responsible person should have.

1) One should have a good understanding of the application domain, even more than a functional tester. If one needs to craft the load test plan then he should be aware of how the user does use it. He should clarify his assumptions of how the system will be used. Recommended way is to observe few users, to understand better about application usage.

2) one should have a better understanding of the application architecture, how components interact? What is the difference between non-prod and prod system from an infrastructure perspective? What will be the network bandwidth? What will be the average think time? etc.

3) One should be technically strong and creative enough to gather the load test data or way to generate the load test data. This one is very critical to building independent, low maintenance, automated load test. One can have a utility which will run prior to the Load test to generate test data. A process of test data management is the key to having successful continues delivery with zero human interaction.

4) One should be able to identify the peak load details based on historical data. If the historical data is not available then one needs to come up with the approximate numbers. As load test may run for 1-2 hour, one needs to have a good combination of different transactions. NOTE: once you baseline the performance of each transaction through the full test (1-2 hours), in CD you can have tested for lesser numbers just to verify if the build is not impacting the baseline numbers. For major builds, one can run the full test.

5) And the key and important one, expert knowledge of the load testing tool, protocols, command-line or configuration options, how the tool works etc.

Points load tester should consider

Web App performance benchmarking – Realtime and End-to-End

OK, last night was thinking of benchmark performance of each component of web app, e.g. client, app server and data services.
There are great APM available in the market which gives you a good amount of information to start with, but still, I was thinking something is missing and want more. APM I tried and used is AppDynamics.

So now what is more we want, AppDynamics provides the analytics capabilities but it comes with the cost.😁 So what all other open-source tools can help.

ELK: this is the really good centralized log analysis tool which you can configure to meet your expectations. It has an inbuilt search engine(elastic search) and data visualization (Kibana), and to get the logs or data you have LogStash which supports multiple technologies to receive data. Using ELK one can get the high-level health of the application, e.g. success rate of the transaction, average response time, slow performing requests, application usage details etc. With ELK you are covered for application server and data services performance.

In ELK we can push the Apache web access logs which give you visibility to usage and performance of the application.

Using MDC filters one can push the service/method performance details to ELK and yes exception details.

OK all this is configured and available, what next? So we don’t have to keep monitoring logs n data we are capturing we can configure alerts (email) n dashboards. But my recommendation (if you ask 😎) monitor the logs for at least a week to see is your new setup is capturing the details you are expecting n then tweak the configuration accordingly.

Now the challenge is how you can monitor client performance and do we really need to monitor it real time?

My thought is, at least monitor it for pilot release to see how the user is adaptive to your new application and if there are any issues. Even I feel it’s more critical than server performance, as most of your testing, all kind of is done with a controlled environment (machines, network, internet speed, browser types and even user behavior). So to get an answer to the question, how the actual user is using your app n what challenges he is facing? Real time Browser performance metric/log will be a real help.


\\TODO
Thoughts on real-time client component performance and monitoring of web application.

Web App performance benchmarking – Realtime and End-to-End

Redis PROD Setup – Part 1

Recently worked on analysis to use Redis as a cache for REST services. Performed the basic configuration of Redis and ran the benchmark test on it and results were amazing. I have used the redis-benchmark to perform benchmarking. https://redis.io/topics/benchmarks

First test where I used 568 bytes data size for Get/Set

redis-benchmark -q -n 100000 -c 50 -P 12 -r 16 -d 568
PING_INLINE: 337837.84 requests per second
PING_BULK: 331125.84 requests per second
SET: 284090.91 requests per second
GET: 318471.31 requests per second
INCR: 444444.47 requests per second
LPUSH: 349650.34 requests per second
RPUSH: 352112.66 requests per second
LPOP: 392156.88 requests per second
RPOP: 390624.97 requests per second
SADD: 425531.91 requests per second
SPOP: 401606.44 requests per second
LPUSH (needed to benchmark LRANGE): 346020.75 requests per second
LRANGE_100 (first 100 elements): 354609.94 requests per second
LRANGE_300 (first 300 elements): 337837.84 requests per second
LRANGE_500 (first 450 elements): 343642.59 requests per second
LRANGE_600 (first 600 elements): 317460.31 requests per second
MSET (10 keys): 62227.75 requests per second

Second test where I used 1000 bytes data size for Get/Set, still there no huge decline in the throughput.

redis-benchmark -q -n 100000 -c 50 -P 12 -r 16 -d 1000
PING_INLINE: 369003.69 requests per second
PING_BULK: 416666.69 requests per second
SET: 277777.78 requests per second
GET: 367647.03 requests per second
INCR: 423728.81 requests per second
LPUSH: 277777.78 requests per second
RPUSH: 277777.78 requests per second
LPOP: 462962.94 requests per second
RPOP: 432900.41 requests per second
SADD: 373134.31 requests per second
SPOP: 403225.81 requests per second
LPUSH (needed to benchmark LRANGE): 251889.16 requests per second
LRANGE_100 (first 100 elements): 318471.31 requests per second
LRANGE_300 (first 300 elements): 317460.31 requests per second
LRANGE_500 (first 450 elements): 335570.47 requests per second
LRANGE_600 (first 600 elements): 325732.88 requests per second
MSET (10 keys): 41666.66 requests per second

My Redis setup is, 1 Master and 4 Slaves and I have configured the Sentinel to monitor the instances and on failure select the Master.

Redis.conf

    • Custom IP address bind configuration, used the private IP address instead on Public one.
    • Changed the default to port to the custom one.
    • One needs to select the Timeout setting carefully, this setting tells the Redis server when to disconnect the client if it’s ideal for N seconds. If you are considering this setup to use with Spring Cache then align this setting with your Jedis connection pool setting.
    • Log level for PROD instance needs to be kept at the lower side, as you may face server issues if your logs occupy more disk space than what is necessary.
    • Limit the maximum database your server wants to handle.
    • Snapshotting section is the critical one, here if you have the number of slaves then need to select the snapshot frequency accordingly. Redis does asynchronous sync with the slaves, but you need to consider a good balance between CPU and Memory usage and the time for the eventually consistent state. One can have multiple conditions to trigger the snapshot, e.g.
    • save 900 1 (after 900 seconds if there is one record changed)
    • save 300 10 (after 300 seconds if there are 10 records changed)
    • save 60 10000 (after 60 seconds if there are 1000 records changed)
  • Configure the Slave Master using “slaveof” and “masterauth” settings
  • slave-priority is another key configuration you need to consider if you are planning to use Sentinel. If the master is down then based on this configuration sentinel process identifies the next master. Lower priority one will be considered as next master. NOTE: Do not set it to 0, the value indicates that this Redis instance will not be promoted to Master, it will be always considered as Slave.
  • If you have separate IPs, Private and Public then based on challenges I faced, always use the private IP for binding to Redis server, and configure the announce-IP configuration to announce IP of Redis server which will have the Public IP address.
  • Always set the maxclient configuration.
  • Another tow very important configuration once should consider while configuring PROD server. We found this very useful when we performed load test on the Redis server to see how Redis performs with huge data sets. “maxmemory” this tells what is the max memory allocation for the Redis, maxmemory-policy configuration tells the Redis what to do when maxmemory threshold is reached, Redis has provided different strategies you can choose from.

NEXT : Detailed configuration of Redis Master, Slave, and Sentinel

Redis PROD Setup – Part 1