Redis PROD Setup – Part 1

Recently worked on analysis to use Redis as a cache for REST services. Performed the basic configuration of Redis and ran the benchmark test on it and results were amazing. I have used the redis-benchmark to perform benchmarking. https://redis.io/topics/benchmarks

First test where I used 568 bytes data size for Get/Set

redis-benchmark -q -n 100000 -c 50 -P 12 -r 16 -d 568
PING_INLINE: 337837.84 requests per second
PING_BULK: 331125.84 requests per second
SET: 284090.91 requests per second
GET: 318471.31 requests per second
INCR: 444444.47 requests per second
LPUSH: 349650.34 requests per second
RPUSH: 352112.66 requests per second
LPOP: 392156.88 requests per second
RPOP: 390624.97 requests per second
SADD: 425531.91 requests per second
SPOP: 401606.44 requests per second
LPUSH (needed to benchmark LRANGE): 346020.75 requests per second
LRANGE_100 (first 100 elements): 354609.94 requests per second
LRANGE_300 (first 300 elements): 337837.84 requests per second
LRANGE_500 (first 450 elements): 343642.59 requests per second
LRANGE_600 (first 600 elements): 317460.31 requests per second
MSET (10 keys): 62227.75 requests per second

Second test where I used 1000 bytes data size for Get/Set, still there no huge decline in the throughput.

redis-benchmark -q -n 100000 -c 50 -P 12 -r 16 -d 1000
PING_INLINE: 369003.69 requests per second
PING_BULK: 416666.69 requests per second
SET: 277777.78 requests per second
GET: 367647.03 requests per second
INCR: 423728.81 requests per second
LPUSH: 277777.78 requests per second
RPUSH: 277777.78 requests per second
LPOP: 462962.94 requests per second
RPOP: 432900.41 requests per second
SADD: 373134.31 requests per second
SPOP: 403225.81 requests per second
LPUSH (needed to benchmark LRANGE): 251889.16 requests per second
LRANGE_100 (first 100 elements): 318471.31 requests per second
LRANGE_300 (first 300 elements): 317460.31 requests per second
LRANGE_500 (first 450 elements): 335570.47 requests per second
LRANGE_600 (first 600 elements): 325732.88 requests per second
MSET (10 keys): 41666.66 requests per second

My Redis setup is, 1 Master and 4 Slaves and I have configured the Sentinel to monitor the instances and on failure select the Master.

Redis.conf

    • Custom IP address bind configuration, used the private IP address instead on Public one.
    • Changed the default to port to the custom one.
    • One needs to select the Timeout setting carefully, this setting tells the Redis server when to disconnect the client if it’s ideal for N seconds. If you are considering this setup to use with Spring Cache then align this setting with your Jedis connection pool setting.
    • Log level for PROD instance needs to be kept at the lower side, as you may face server issues if your logs occupy more disk space than what is necessary.
    • Limit the maximum database your server wants to handle.
    • Snapshotting section is the critical one, here if you have the number of slaves then need to select the snapshot frequency accordingly. Redis does asynchronous sync with the slaves, but you need to consider a good balance between CPU and Memory usage and the time for the eventually consistent state. One can have multiple conditions to trigger the snapshot, e.g.
    • save 900 1 (after 900 seconds if there is one record changed)
    • save 300 10 (after 300 seconds if there are 10 records changed)
    • save 60 10000 (after 60 seconds if there are 1000 records changed)
  • Configure the Slave Master using “slaveof” and “masterauth” settings
  • slave-priority is another key configuration you need to consider if you are planning to use Sentinel. If the master is down then based on this configuration sentinel process identifies the next master. Lower priority one will be considered as next master. NOTE: Do not set it to 0, the value indicates that this Redis instance will not be promoted to Master, it will be always considered as Slave.
  • If you have separate IPs, Private and Public then based on challenges I faced, always use the private IP for binding to Redis server, and configure the announce-IP configuration to announce IP of Redis server which will have the Public IP address.
  • Always set the maxclient configuration.
  • Another tow very important configuration once should consider while configuring PROD server. We found this very useful when we performed load test on the Redis server to see how Redis performs with huge data sets. “maxmemory” this tells what is the max memory allocation for the Redis, maxmemory-policy configuration tells the Redis what to do when maxmemory threshold is reached, Redis has provided different strategies you can choose from.

NEXT : Detailed configuration of Redis Master, Slave, and Sentinel

Redis PROD Setup – Part 1

Maven – Quick Start

Many of my friends were facing similar kind of problem while configuring the maven, so thought to cover this topic in simple form

Disclaimer : Please consider this as a quick start guide, its not the in details one, and I tried to cover the same topic in different way.

How I look at Maven?

Maven is the tool which helps me to download/manage all required API for my project.

In old days, if you need to build a simple application which handles the MS Office files (excel, word), you have to search for the API, then on the API home page you need to go to the page which details out the all required jars to use that API, and yes few of API provider also has a ZIP file which contents all the dependent jar files, if there is any other jar dependency you need to download it from their site, and many of time you will face compilation or runtime issues as some of the jar is missing or its version is not compatible.

To fix this Maven and other build tools came into the picture, where you tell the tool the details of the repository, and the API you want to use. API provider add the details of the dependencies to use their API.

Repository : Its the server which stores the all required jar files, one of them is NEXUS, which stores the required files with their binary, source and versions. Nexus provides facility to add, publish new dependencies or jar files or any dependent files which will be used by projects. You can publish your jar files on this server so that it can be reused in other applications.

POM file : POM files has the details of the dependencies to run/use API or project. This file can also has details about project, jvm version, team members etc. POM also has the Plugins configurations, there are multiple plugins available to build, report, measure code quality, analyse dependencies and deploy.

Configuration and Getting the dependencies

Settings.xml : This file holds the configurations which is common for all Maven projects, like proxy details, repository server details, authentication details. So when you run the mvn command to build the project, maven search for the settings file in Maven Folder/conf/ and <user dir>/.m2/,  If both files exists, their contents gets merged, with the user-specific settings.xml being dominant.

Now maven know how to connect to server/internet (if required) and download the required dependencies, plugins etc.

POM.xml : In this file one can add the dependency details which looks like as

<dependency>
 <groupId>org.apache.tomcat</groupId>
 <artifactId>tomcat-jdbc</artifactId>
 <version>7.0.42</version>
 <scope>runtime</scope>
 </dependency>

GroupId, ArtifactID and version identifies the dependency uniquely, and based on these details  required file is get downloaded to .m2 directory which is your local repository, and from where the file is referred in your project.

What is SCOPE in dependency tag?

Most of the time I fund the people face problems as they havent tried to understand what this scope is for.

compile – this is the default scope, used if none is specified. Compile dependencies are available in all classpaths. Furthermore, those dependencies are propagated to dependent projects.

provided – this is much like compile, but indicates you expect the JDK or a container to provide it at runtime. It is only available on the compilation and test classpath, and is not transitive.

runtime – this scope indicates that the dependency is not required for compilation, but is for execution. It is in the runtime and test classpaths, but not the compile classpath.

test – this scope indicates that the dependency is not required for normal use of the application, and is only available for the test compilation and execution phases.

system – this scope is similar to provided except that you have to provide the JAR which contains it explicitly. The artifact is always available and is not looked up in a repository.

So based on your scope, jar or package files are referred in build process. e.g. provided and test scoped dependencies wont be added to your WAR file.

for more deatils : http://maven.apache.org/pom.html

You can exclude few of the dependencies which you wont needs to be downloaded/included from the child dependencies.

Basic commands

mvn clean install

mvn eclipse:configure-workspace
is used to add the classpath variable M2_REPO to Eclipse which points to your local repository and optional to configure other workspace features.

mvn eclipse:eclipse
generates the Eclipse configuration files.

mvn eclipse:clean
is used to delete the files used by the Eclipse IDE.

Few more commands

mvn dependency:tree -Dverbose -Dincludes=commons-collections

mvn verify

mvn dependency:analyze-only verify

mvn dependency:analyze-duplicate

mvn dependency:analyze-report

mvn site

Quick Start Guide: http://maven.apache.org/guides/getting-started/maven-in-five-minutes.html

Maven – Quick Start

Spring’s Device Detection (Spring Boot) Vs WURFL Device detection

I liked the Spring Boot idea very much, very impressive. Excellent use of Embedded servers, auto configs and the best thing with less code you can complete the functionality.

When I was going through the basic guides of Spring boot, I tried the Device Detection guide, which detects the the device from which the request is made, its basic implementation which tells you if request is made form Mobile, Tab or PC.

Spring Device has used the “User-Agent”, “x-wap-profile”, “Accept” HTTP headers to detect the device type, if all of the listed headers fails to identify the device type, in the last code iterate through all the headers to see if the request is from “Opera Mini” browser, which mostly used by many mobile users.

Spring Device has used basic algorithm used in WordPress’s Mobile pack, which works for large number of mobile browsers.

If you need to know full capabilities of the phone, like OS type, touch screen support, browser type, XHTML-MP supprted or not etc then WURFL API is a good option.

Only thing is WURFL API is not updated for the Spring Boot and uses old version of commons-lang and other dependent libraries. As source code is provided I modified a bit and I was able to use the WURFL API for device detection.

WURFL has commercial license available and also has cloud based service if you want to try.

You can download the code from the WURFL’s repository, to create GeneralWURFLEngine class as spring componant I added @Component annotation

@Component(value="WURFLEngine")public class GeneralWURFLEngine implements WURFLEngine, WurflWebConstants {

and it dosent have default constructor, I added one with following lines, where wurfl.zip file has the XML file which has all data related to mobile devices.

 static URL filePath = GeneralWURFLEngine.class.getClassLoader().getResource("wurfl.zip");
  
 public GeneralWURFLEngine() {
 this(new XMLResource(filePath.getPath()));
 }

and when I started the Spring Boot Application, I added the package of the GeneralWURFLEngine class to component scan path, and one more major change, as Maven repository for WURFL is only available if you license, I added jar in /lib folder.

Below is the code changed made to Spring Boot guide for device detection.

package com.ykshinde.controller;
 
import javax.annotation.Resource;
import javax.servlet.http.HttpServletRequest;
 
import net.sourceforge.wurfl.core.WURFLEngine;
 
import org.springframework.mobile.device.Device;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.ResponseBody;
 
@Controller
public class DeviceDetectionController {
 
    @Resource(name="WURFLEngine")
    WURFLEngine engine;
     
    @RequestMapping("/detect-device")
    public @ResponseBody String detectDevice(Device device, HttpServletRequest request) {
         
        net.sourceforge.wurfl.core.Device device2 = engine.getDeviceForRequest(request);
         
        StringBuffer deviceCapabilities = new StringBuffer();
 
        deviceCapabilities.append(" DEVICE_ID : ").append(device2.getId()).append("<br>")
        .append(" DEVICE_OS : ").append(device2.getCapability("device_os")).append("<br>")
        .append(" DEVICE_OS_VERSION : ").append(device2.getCapability("device_os_version")).append("<br>")
        .append(" IS_TABLET : ").append(device2.getCapability("is_tablet")).append("<br>")
        .append(" IS_WIRELESS_DEVICE : ").append(device2.getCapability("is_wireless_device")).append("<br>")
        .append(" MOBILE_BROWSER : ").append(device2.getCapability("mobile_browser")).append("<br>")
        .append(" MOBILE_BROWSER_VERSION : ").append(device2.getCapability("mobile_browser_version")).append("<br>")
        .append(" POINTING_METHOD : ").append(device2.getCapability("pointing_method")).append("<br>")
        .append(" PREFERRED_MARKUP : ").append(device2.getCapability("preferred_markup")).append("<br>")
        .append(" RESOLUTION_HEIGHT : ").append(device2.getCapability("resolution_height")).append("<br>")
        .append(" RESOLUTION_WIDTH : ").append(device2.getCapability("resolution_width")).append("<br>")
        .append(" UX_FULL_DESKTOP : ").append(device2.getCapability("ux_full_desktop")).append("<br>")
        .append(" XHTML_SUPPORT_LEVEL : ").append(device2.getCapability("xhtml_support_level")).append("<br>");
         
         
        String deviceType = "unknown";
        if (device.isNormal()) {
            deviceType = "normal";
        } else if (device.isMobile()) {
            deviceType = "mobile";
        } else if (device.isTablet()) {
            deviceType = "tablet";
        }
         
        deviceCapabilities.append(" DEVICE TYPE (SPRING BOOT) : ").append(deviceType);
         
        return deviceCapabilities.toString();
    }
 
}

And below is the response displayed when request emulated as from “Samsung Tab ”


DEVICE_ID : samsung_galaxy_tab_ver1_subschi800
DEVICE_OS : Android
DEVICE_OS_VERSION : 2.2
IS_TABLET : true
IS_WIRELESS_DEVICE : true
MOBILE_BROWSER : Android Webkit
MOBILE_BROWSER_VERSION :
POINTING_METHOD : touchscreen
PREFERRED_MARKUP : html_web_4_0
RESOLUTION_HEIGHT : 1024
RESOLUTION_WIDTH : 600
UX_FULL_DESKTOP : false
XHTML_SUPPORT_LEVEL : 4
DEVICE TYPE (SPRING BOOT) : mobile

NOTE : If you are going to use WURFL Api commercially, please do check the licensing part of it.

 Download Code

Spring’s Device Detection (Spring Boot) Vs WURFL Device detection

Cross-site request and Sever configuration

While developing REST web service, we came across an error “No ‘Access-Control-Allow-Origin’ header is present on the requested resource” with Http Status code 403, after exploring more on it, found the details of CORS and how browser supports it. Obviously as a user (mostly developer) I can disable the security and can make the request, but lets understand the ideal way of handling it.

What is CORS?

This document defines a mechanism to enable client-side cross-origin requests. Specifications that enable an API to make cross-origin requests to resources can use the algorithms defined by this specification. If such an API is used on http://example.org resources, a resource on http://hello-world.example can opt in using the mechanism described by this specification (e.g., specifying Access-Control-Allow-Origin: http://example.org as response header), which would allow that resource to be fetched cross-origin from http://example.org.

You can find more details on : http://www.w3.org/TR/cors/

So if you are making request to same server but the URL is different, the request will be treated as the cross-domain, e.g. http://localhost and http://120.0.0.1 will be treated as the cross-domain. Specification is implemented by the browser to support same-origin policy and security. When browser makes a request, it check for the origin and the URL of the request made to, if it wont match (protocol+domain+port number should be same) then a pre-flight request is made to the server, which is nothing but a same request with HTTP method OPTION. e.g. if you are making a call from localhost:8080/app/index.html to 127.0.0.1:8080/app/hello then request is made to 127.0.0.1:8080/app/hello with OPTION http method.

Pre-flight Request 

OPTIONS /app/home HTTP/1.1 
Host: 127.0.0.1:8080 
Connection: keep-alive 
Cache-Control: max-age=0 
Access-Control-Request-Method: GET 
Origin: http://localhost:8080 
Access-Control-Request-Headers: accept, content-type 
Accept: */* 
Referer: http://localhost:8080/app/index.html 
  

Response

HTTP/1.1 200 OK 
Access-Control-Allow-Origin: http://localhost:8080 
Access-Control-Allow-Credentials: true 
Access-Control-Max-Age: 1800 
Access-Control-Allow-Methods: GET 
Access-Control-Allow-Headers: content-type,access-control-request-headers,access-control-request-method,accept,origin,x-requested-with 
Content-Length: 0
  

 

If your server returns the Access-Control-Allow-Origin headers then only client can access the targeted site content, when the preflight request is successful then browser makes the original request. Once the request is successful, browser cache the details of the origin, url (request made to), max-age and header details, so for subsequent requests to same URL will be served directly and in that case preflight request will be not sent.

On Server Side, How I can support the cross-site request

We do use Tomcat, from Tomact version 7.0.40 by default CorsFilter is included, you just need to enable it for your application, the minimal configuration is

	<filter>
		<filter-name>CorsFilter</filter-name>
		<filter-class>org.apache.catalina.filters.CorsFilter</filter-class>
	</filter>
	<filter-mapping>
		<filter-name>CorsFilter</filter-name>
		<url-pattern>/*</url-pattern>
	</filter-mapping>
 

you can specify the optional parameters by adding init parameters, follow this link for more details.

e.g.

		<init-param>
			<param-name>cors.allowed.origins</param-name>
			<param-value>http://localhost:8080, http://127.0.0.1:8080</param-value>
		</init-param>
		<init-param>
			<param-name>cors.allowed.methods</param-name>
			<param-value>GET,POST,HEAD,OPTIONS,PUT</param-value>
		</init-param>

If you have Apache or any other HTTP server between client and you application server, then make sure you have implemented cross-site support on those servers.
if your application is used or open for limited applications, it is recommended to use specific domain names list for “Allowed Origin”, if you have a global web API then you can use *.
More on OPTION method

This method allows the client to determine the options and/or requirements associated with a resource, or the capabilities of a server, without implying a resource action or initiating a resource retrieval. Mostly option do not have the body, but specification is open and in future option body may be supported to make detailed query on server.

BTW : WebSocket won’t fall under same-origin or cross-site policy, so if you create a WebSocket to different URL it will work.

Download code from GIT

Cross-site request and Sever configuration

Configuring SonarQube

SonarQube is all in one code quality tool, it covers following main aspects of code quality
1) Complexity
2) Code coverage
3) Potential bugs
4) Architecture and Design

It is easy to use and configurable as per your need, SonarQube reuses the tools like FindBug, PMD, Cobertura, JUnits etc. Sonar also supports multiple languages. You can configure your own rules, rules can be configured project specific. You can integrate Sonar with Jenkins, so once your project is built all required rules will be analysed and published on SonarQube.

Shopizer Project On SonarQube

This is a quick start tutorial to setup a SonarQube.
You need JRE 6 or above to run the 4.5 SonarQube distribution.

Download the SonarQube distribution from the site, I used 4.5 version. Download from SonarQube 4.5

Extract the content of the file to a folder.

I used MySQL as backend to store the analysis data. to configure MySQL, run following queries on MySQL

 
CREATE DATABASE SonarDBName CHARACTER SET utf8 COLLATE utf8_general_ci;

CREATE USER 'sonar' IDENTIFIED BY 'sonar';
GRANT ALL ON SonarDBName.* TO 'sonar'@'%' IDENTIFIED BY 'sonar';
GRANT ALL ON SonarDBName.* TO 'sonar'@'localhost' IDENTIFIED BY 'sonar';
FLUSH PRIVILEGES; 

Change the SonarDBName to whatever you prefer, if you have your mySQL running on different server, change the value of ‘localhost’ to the specific host.

Configure Sonar to use MySQL as your default DB,Open the file conf/sonar.properties to edit, search for “sonar.jdbc.url”

 
sonar.jdbc.url=jdbc:mysql://localhost:3306/sonarQube?useUnicode=true&amp;amp;characterEncoding=utf8&amp;amp;rewriteBatchedStatements=true&amp;amp;useConfigs=maxPerformance

In above snippet sonarQube is the Sonar DB name.

Now start your SonarQube /bin/windows-x86-32 (if you are running it on different platform, choose appropriate directory), in initial start up Sonar will download two plugins by default java and FindBug. if you need additional plugins you can download it from Sonar plugin site and copy it to /extensions/plugins. This is manual method.

I have used following plugins
sonar-checkstyle-plugin-2.1.1.jar
sonar-cobertura-plugin-1.6.3.jar
sonar-findbugs-plugin-2.4.jar
sonar-java-plugin-2.5.jar
sonar-javascript-plugin-2.1.jar
sonar-pmd-plugin-2.2.jar
sonar-scm-activity-plugin-1.8.jar
sonar-scm-stats-plugin-0.3.1.jar
sonar-web-plugin-2.3.jar

Your default password for admin user is admin, login to the SonarQube instance by hitting localhost:9000 and change the default admin password, if required you can add more users and user groups. plugins also available to synch with Active Directory.

Configuring SonarQube

Running Embedded Tomcat

While searching for Tomcat 8, I came across a term called “Embedded Tomcat”, so I was wondering what it’s all about. With Embedded Tomcat, one can write a standalone application which can have a embedded web server, so when you run the standalone java application, tomcat embedded server will also start with it. I was surprised to see it has all the features which are available with Tomcat, including WebSocket. Spring Boot is using this API for starting the Spring Boot application. Even you can start your Spring web application by registering Springs DispatcherServlet in below example (but why need to reinvent the wheel, Spring Boot has done that for us).

Maven :

 <dependency> 
  <groupId>org.apache.tomcat.embed</groupId> 
  <artifactId>tomcat-embed-core</artifactId> 
  <version>8.0.14</version>
</dependency> 

<dependency> 
  <groupId>org.apache.tomcat.embed</groupId> 
  <artifactId>tomcat-embed-logging-juli</artifactId> 
  <version>8.0.14</version> 
</dependency>      

Java Code :

		Tomcat tomcat = new Tomcat();
		tomcat.setPort(8080);
		File base = new File(&amp;amp;quot;./staticContent&amp;amp;quot;);
		
		Context rootCtx = tomcat.addWebapp(&amp;amp;quot;/app&amp;amp;quot;, base.getAbsolutePath());
		
		Tomcat.addServlet(rootCtx, &amp;amp;quot;home&amp;amp;quot;, new SimpleServlet());
		rootCtx.addServletMapping(&amp;amp;quot;/home&amp;amp;quot;, &amp;amp;quot;home&amp;amp;quot;);
		
		tomcat.start();
		tomcat.getServer().await();

In above code, setPort sets the port to listen on for embedded Tomcat, I have added a folder “staticContent”, which has my html and to simplify the tomcat configuration I have added simpleWEB-INF/ web.xml in same folder. you can call it from main method.

tomcat.addWebApp method is called with two arguments, first is the context root, and second is the folder path.

if you observer addServlet is Static method of Tomcat class, which registers the servlet class. SimpleServlet is a simple servlet class.

tomcat.start will start the server. That’s all and your embedded Tomcat server is started. you can access your resources by hitting http://localhost:8080/app/home

One of the possible use (Other than running it as tomcat server) I can see is, if you have a web based tool or utility and you want user to quick start and try your product, with minimal setup i.e. just unzipping the package to folder, user can start exploring functionality.

Download Code

Running Embedded Tomcat