Writing Data From REST to a File

private static void doHttpGet(HttpGet request, String inputFile) {
OutputStream outputStream = null;
File file = null;
InputStream content = null;
CloseableHttpClient httpClient = null;
try {


httpClient = getHTTPClient();

HttpResponse response = httpClient.execute(request);
HttpEntity entity = response.getEntity();
content = entity.getContent();

if (content != null) {
file = new File(inputFile);
System.out.print(file.getAbsolutePath());
outputStream = new FileOutputStream(file);

int read = 0;
byte[] bytes = new byte[1024];

while ((read = content.read(bytes)) != -1) {
outputStream.write(bytes, 0, read);
}
}
// Ensure it is fully consumed
EntityUtils.consume(entity);

} catch (IOException e) {
e.printStackTrace();
} finally {
if (outputStream != null) {
try {
outputStream.close();
outputStream = null;
content.close();
httpClient.close();
httpClient = null;
content = null;
} catch (IOException e) {
e.printStackTrace();
}

}
}
}




private static CloseableHttpClient  getHTTPClient() {
RequestConfig.Builder requestBuilder = RequestConfig.custom();
requestBuilder = requestBuilder.setConnectTimeout(180000); //3 * 60 * 1000
   requestBuilder = requestBuilder.setConnectionRequestTimeout(180000);//3 * 60 * 1000
   requestBuilder = requestBuilder.setSocketTimeout(180000);//3 * 60 * 1000
 

HttpClientBuilder  builder = HttpClientBuilder.create();     
builder.setDefaultRequestConfig(requestBuilder.build());

return builder.build();
}

How to use dynamic sql and use its return value

SET @sql = NULL;
SET @cout = '';
SELECT CONCAT('SELECT CONCAT(',GROUP_CONCAT(c.COLUMN_NAME, ',","'),') as v1 INTO @cout FROM 
   student where enrolment_month =given_month() and enrolment_year=given_year()')
INTO @sql
FROM INFORMATION_SCHEMA.COLUMNS c
WHERE c.TABLE_NAME IN ('client_group_types') ;
PREPARE sql_statement FROM @sql;
EXECUTE sql_statement;
DEALLOCATE PREPARE sql_statement; 
SET @Param2 = @cout;
SELECT @Param2;

Can use @Param2 as return type of a function

Activity Streams for Social Media

Today most of the social media data is stored in No SQL databases as Activity Streams. Below given is some of useful activity streams I found over the internet

Discussion
{
"_id" : "4d19fb9ddhd4000005",
"_t" : "Discussion",
"created" : "Tue, 28 Dec 2010 07:00:35 GMT",
"GroupID" : 129242,
"eIndividualID" : "22872F9",
"firstName" : "Danial",
"lastName" : "Micheal",
"discussionStart" : "How is the meeting.",
"comments": [
{
"_id" : "4d19fe329aceye400000a",
"eIndividualID" : "FFE4251",
"created" : "2010-12-28T15:11:46.6760000Z",
"firstName" : "Robin",
"lastName" : "Tally",
"commentText" : "Went alright. Thank you"
},
{
"_id" : "4d19feyey027400000a",
"eIndividualID" : "FFE441",
"created" : "2010-12-28T15:11:46.6760000Z",
"firstName" : "Robin",
"lastName" : "Tally",
"commentText" : "That is good"
},
{
"_id" : "4d1deheh000003",
... } ]
}

Live Chat 
{
"_id": "4cd261dsdfshsd000006",
"senderEIndividualID": "1B0D6E8",
"senderFirstName": "Danial",
"senderLastName": "Tally",
"sendDate": "Thu, 04 Nov 2010 00:31:32 GMT",
"groupID": 119420,
"message": "way faster then "
}
{
"_id": "4cd261a69ac0900c2400000a",
"senderEIndividualID": "B656641",
"senderFirstName": "Steve",
"senderLastName": "Chen",
"sendDate": "Thu, 04 Nov 2010 00:32:54 GMT",
"groupID": 119420,
"message": "ohhhhh"
} 
{
"_id": "4cd2621b9ac0900c2400001f",
"senderEIndividualID": "1B0D6E8",
"senderFirstName": "Danial",
"senderLastName": "Tally",
"sendDate": "Thu, 04 Nov 2010 00:34:51 GMT",
"groupID": 119420,
"message": "I think it made"
}


User Data 
{ "_id": 4252992,
 "events" : {
 "all_ids": [ 116706, 179487, 16389, 827496 ],
 "curr_ids": [ 827496 ],
 },
 "nns" : [
 [ 2816442, 0.2 ],
 [ 1615962, 0.047619047619047616 ],
 ],
 "facebook" : {
 "_id" : 4808871, 
 "name" : "Danial Tally",
 "location" : "Melbourne, Victoria",
 "mutualfriends" : [ 56341525, 262659792 ],
 "allfriends" : [ 56341525, 562526567, 262659792 ],
 },
}

Performance Tuning on MongoDB

If your writing huge amount of data to MongoDB it is really important to have a proper performance tuning method with that. I will discuss one of the performance tuning method, but this might not be the only one

The explain() method returns a document that describes the process used to return the query results. If the mango has a collection name inventory and inventory is having 1500000 records, and this how it explain

db.inventory.find().explain()

{
    "cursor" : "BasicCursor",
    "isMultiKey" : false,
    "n" : 1500000,
    "nscannedObjects" : 1500000,
    "nscanned" : 1500000,
    "nscannedObjectsAllPlans" : 1500000,
    "nscannedAllPlans" : 1500000,
    "scanAndOrder" : false,
    "indexOnly" : false,
    "nYields" : 0,
    "nChunkSkips" : 0,
    "millis" : 22,
    "indexBounds" : {},
    "server" : "myServerIp"
}

cursor displays BasicCursor to indicate a collection scan. n displays 1500000 to indicate that the query matches and returns three document.
nscanned and nscannedObjects display 1500000 to indicate that MongoDB had to scan ten documents 

db.inventory.find({"verb":"getInventoryPrice"}).explain()

{
    "cursor" : "BasicCursor",
    "isMultiKey" : false,
    "n" : 302,
    "nscannedObjects" : 1500000,
    "nscanned" : 1500000,
    "nscannedObjectsAllPlans" : 1500000,
    "nscannedAllPlans" : 1500000,
    "scanAndOrder" : false,
    "indexOnly" : false,
    "nYields" : 0,
    "nChunkSkips" : 0,
    "millis" : 65,
    "indexBounds" : {},
    "server" : "myServerIp"
}

The difference between the number of matching documents and the number documents scanned may suggest that, to improve efficiency, the query might benefit from the use of an index.

Adding Indexes
Good performance starts with Indexing
Indexes can improve the performance by 2 to 3 times or 1000 ms query down to <1ms, but that is for good indexing

Key commands on indexing

db.inventory.ensureIndex( { verb: 1 } )
db.inventory.dropIndex( { verb: 1 } )

ascending 1
descending -1


db.inventory.getIndexKeys()
{
    "0" : {
        "_id" : 1
    },
    "1" : {
        "verb" : 1
    }
}
db.inventory.getIndexes()
{
    "0" : {
        "v" : 1,
        "key" : {
            "_id" : 1
        },
        "ns" : "inventory.inventory",
        "name" : "_id_"
    },
    "1" : {
        "v" : 1,
        "key" : {
            "verb" : 1
        },
        "ns" : "inventory.inventory",
        "name" : "verb_1"
    }
}

scanAndOrder is a very bad thing in MongoDB
MongoDB sorts documents in-memory is very very expensive and with out an index large result sets can be rejected with an error

Explain After the above change

{
    "cursor" : "BtreeCursor verb_1",
    "isMultiKey" : false,
    "n" : 302,
    "nscannedObjects" : 302,
    "nscanned" : 302,
    "nscannedObjectsAllPlans" : 302,
    "nscannedAllPlans" : 302,
    "scanAndOrder" : false,
    "indexOnly" : false,
    "nYields" : 0,
    "nChunkSkips" : 0,
    "millis" : 2,
    "indexBounds" : {
        "verb" : [ 
            [ 
                "getInventoryPrice", 
                "getInventoryPrice"
            ]
        ]
    },
    "server" : "myServerIp"
}
BtreeCursor indicates that the query used an index. The cursor includes name of the index. When a query uses an index, the output of explain() includes indexBounds details.

Compare the Performance
To manually compare the performance of a query using more than one index, you can use the hint() method in conjunction with the explain() method.
db.inventory.find({"verb":"getInventoryPrice"}).explain().hint({"verb": 1})
{
    "cursor" : "BtreeCursor verb_1",
    "isMultiKey" : false,
    "n" : 302,
    "nscannedObjects" : 302,
    "nscanned" : 302,
    "nscannedObjectsAllPlans" : 302,
    "nscannedAllPlans" : 302,
    "scanAndOrder" : false,
    "indexOnly" : false,
    "nYields" : 0,
    "nChunkSkips" : 0,
    "millis" : 0,
    "indexBounds" : {
        "verb" : [ 
            [ 
                "getInventoryPrice", 
                "getInventoryPrice"
            ]
        ]
    },
    "server" : "myServerIp"
}

The ordering of fields in an Index should be
fields where extract exact values
fields where will sort
fields where query for range vallues like $in, $gt, $lt...Etc

Need to be very careful if you have
Many query patterns
Use mongoDb like RDBMS
Have many indexes for each collection

How to identify problematic quires by profiling
// Will profile all queries that take 100 ms
db.setProfilingLevel(1, 100);

// Will profile all queries
db.setProfilingLevel(2);

// Will disable the profiler
db.setProfilingLevel(0);

// Find the most recent profile entries
db.system.profile.find().sort({$natural:-1});
    
// Find all queries that took more than 5ms
db.system.profile.find( { millis : { $gt : 5 } } );
    
// Find only the slowest queries
db.system.profile.find().sort({millis:-1});

db.system.profile.find({op:{$in: ["query", "update", "command"]}})


So far we have discussed how to do indexing in manual methods. Now lets discuss about some easy ways
DEX- Dex is a MongoDB performance tuning tool that compares queries to the available indexes in the queried collection(s) and generates index suggestions based on simple heuristics. Currently you must provide a connection URI for your database.
https://github.com/mongolab/dex/blob/master/README.md

Identify Indexing with DEX
A Indexed query helps performance by several ways. The trick is to identify an ideal indexes. Even for experts, hand-crawling through the MongoDB logs for slow queries is a laborious process.
Dex helps you in this. Dex is a tool that looks at your slow queries and tells you exactly which indexes you need to make those queries fast.

Running Dex is easy! For up-to-date instructions, refer to our README on GitHub. However, what you do with Dex's recommendations depends on your situation.

Install Dex
>pip install dex # in sudo mode or use sudo pip install dex

How to find the Location on EC2

> vi /etc/mongod.conf

Location of mongodb log can be found with logpath=/var/log/mongo/mongod.log

dex -f /var/log/mongo/mongod.log mongodb://ec2-<myec2ip>.ap-southeast-2.compute.amazonaws.com:<mongo port>/inventory

> dex -f /var/log/mongo/mongod.log mongodb://ec2-<myec2ip>.ap-southeast-2.compute.amazonaws.com:<mongo port>/inventory

{
    'runStats': {
        'linesRecommended': 150,
        'linesProcessed': 169,
        'linesPassed': 66594
    },
    'results': [
        {
            'queryMask': '{"$query":{"verb":"<val>"}}',
            'namespace': 'mydb.inventory',
            'recommendation': {
                'index': '{"date": 1}',
                'namespace': 'mydb.inventory',
                'shellCommand': 'db["inventory"].ensureIndex({"date": 1}, {"background": true})'
            },
            'details': {
                'count': 95,
                'totalTimeMillis': 10676,
                'avgTimeMillis': 112
            }
        },
        ...............
    ]
}


Filter by query time (millis)
dex -f /var/log/mongo/mongod.log mongodb://ec2-<myec2ip>.ap-southeast-2.compute.amazonaws.com:<mongo port>/inventory -s 400

In more details 
dex -f /var/log/mongo/mongod.log mongodb://ec2-<myec2ip>.ap-southeast-2.compute.amazonaws.com:<mongo port>/inventory -v

Even Dex guide to find the required Indexes final decision in on you to find the required one



References
http://mongolab.org/dex/
http://blog.mongolab.com/2012/06/cardinal-ins/

Adding MIME Types for Fonts in EC2 using Python

By Default EC2 does not include mime types for fonts except svg in  /etc/mime.types

So need to add them to the system

Below code example tells you how to do that in python

import mimetypes
..
mimetypes.init()
mimetypes.add_type('font/ttf', '.ttf')
mimetypes.add_type('font/opentype', '.otf')
mimetypes.add_type('application/font-woff', '.woff')
mimetypes.add_type('application/vnd.ms-fontobject', '.eot')

For more Information please visit https://docs.python.org/2/library/mimetypes.html

CSV format from MySQL View

With the latest Technologies and Tools available its very easy to export data select from database to CSV format.

But most of them are not able to to configure as you want.

Here is a sample for MySql for Student Data from client_marks_view View


set @given_from:='2014-01-01';
set @given_to:='2014-06-01';

SELECT CONCAT(' Students Marks Summary for the period ',`given_from`(), ' - ',`given_to`()) AS `Students Marks Summary`
UNION ALL
SELECT '' AS ``
UNION ALL
SELECT CONCAT('"Student Name','",','"Total Attendance','",','"Total Marks','"') AS `CONCAT('"Student','",','"Resource Name','",','"Total Attendance','",','"Total Marks')`
UNION ALL
(SELECT CONCAT('"',ifnull(`a`.`studnet_name`,''),'","',ifnull(`a`.`total_attendance`,' '),'","$',round(ifnull(`a`.`total_marks`,0),2),'",') AS `CONCAT('"',
IFNULL(a.student_name, ""),
'","',
IFNULL(a.total_attendance," "),
'","',
IFNULL(a.total_marks, "")
ROUN` from `student_marks_view` `a`) ;


How to create a direct view from that

CREATE OR REPLACE VIEW students_view
AS

   SELECT CONCAT(' Student Marks Summary for the period ',`given_from`(), ' - ',`given_to`()), ',,,,,,'
     UNION ALL
   SELECT "", "" FROM DUAL
     UNION ALL
   SELECT CONCAT('"Student Name','",','"Total Attendance','",','"Total Marks'),'",,,,,,'
     UNION ALL
 
   (SELECT CONCAT('"',
IFNULL(a.student_name, ""),
'","',
IFNULL(a.total_attendance," "),
'","$',
ROUND(IFNULL(a.total_marks, 0),2),
'",'), ',,,,,,' 

FROM student_marks_view  AS a )

Basic Web Testing with Selenium WebDriver With Ruby 2.0.0

1. Install Node and Ruby on windows

D:\Selinium Tests\1>ruby -v
ruby 2.0.0p481 (2014-05-08) [x64-mingw32]

2. Install Selinium Web Driver 
gem install selenium-webdriver

If no web driver installed you will get bellow error on windows
C:/Ruby200-x64/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- selenium-webdriver (LoadError)
        from C:/Ruby200-x64/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from 1.rb:2:in `<main>'


3. Install Chrome & IEDriverServer Driver
Copy the chromedriver.exe to  C:/Ruby200-x64/bin or any classpath location
Copy the IEDriverServer.exe to   C:/Ruby200-x64/bin or any classpath location

4. Install HTTP server locally and create a folder name ruby under htdocs
5. Copy All the *.rb and *.html files to above folder in (4)
6. Execute each ruby file as 
>ruby xxx.rb

Download the example file from Examples

Using Cookie as RewriteRule variable in Apache ReverseProxy

If have a cookie named myServerName, and want to check for its availability and use its content for building a URL

Code:
RewriteCond %{REQUEST_URI} ^/
RewriteCond %{HTTP_COOKIE} ^.*myServerName=([^;]+)
RewriteRule /(.*) http://%1.test.com.au/$1 [P]

Re Write to get the content from AWS S3 Bucket

RewriteEngine On
RewriteCond %{HTTP_COOKIE} ^.*mycookie.*$ [NC] 
RequestHeader set Host 's3-ap-southeast-2.amazonaws.com'
RewriteRule (.*)/files/(.*) http://s3-ap-southeast-2.amazonaws.com/mybucket/$2 [P]
ProxyPassReverse /files/ http://s3-ap-southeast-2.amazonaws.com/mybucket/

#If no Cookies Found
RewriteRule (.*)/files/(.*) http://s3-ap-southeast-2.amazonaws.com/mybucket//error_url/error.html [P]
ProxyPassReverse /files/ http://s3-ap-southeast-2.amazonaws.com/mybucket/

Easy Table Sorting and Searching with JQuery Plugings

<link rel="stylesheet" href="........../themes/blue/style.css" />

<script type="text/javascript" src="...../jquery.tablesorter.min.js"></script>
<script type="text/javascript" src=".../jquery.searcher.min.js"></script>


            <input id="tablesearchinput" />
            <table id="student-list" class="tablesorter">
                <thead>
                    <tr>
                        <th class="id">Student Id</th>
                        <th class="name">Student Name</th>
                    </tr>
                </thead>
                <tbody>
                </tbody>
            </table>

$("#student-list").tablesorter();
$("#student-list").searcher({
inputSelector: "#tablesearchinput"
});

Improved data source for Spring

Why and what about C3p0

c3p0 is an easy-to-use library for making traditional JDBC drivers "enterprise-ready" by augmenting them with functionality defined by the jdbc3 spec and the optional extensions to jdbc2. In particular, c3p0 provides several useful services:

Classes which adapt traditional DriverManager-based JDBC drivers to the newer javax.sql.DataSource scheme for acquiring database Connections.

Transparent pooling of Connection and PreparedStatements behind DataSources which can "wrap" around traditional drivers or arbitrary unpooled DataSources.

The library tries hard to get the details right:

c3p0 DataSources are both Referenceable and Serializable, and are thus suitable for binding to a wide-variety of JNDI-based naming services.

Statement and ResultSets are carefully cleaned up when pooled Connections and Statements are checked in, to prevent resource- exhaustion when clients use the lazy but common resource-management strategy of only cleaning up their Connections....

The library adopts the approach defined by the JDBC 2 and 3 specification (even where these conflict with the library author's preferences). DataSources are written in the JavaBean style, offering all the required and most of the optional properties (as well as some non-standard ones), and no-arg constructors. All JDBC-defined internal interfaces are implemented (ConnectionPoolDataSource, PooledConnection, ConnectionEvent-generating Connections, etc.) You can mix c3p0 classes with compliant third-party implementations (although not all c3p0 features will work with external implementations).

c3p0 now fully supports the JDBC4 specification.

c3p0 hopes to provide DataSource implementations more than suitable for use by high-volume "J2EE enterprise applications".

Change to maven
                  <dependency>
<groupId>com.mchange</groupId>
<artifactId>c3p0</artifactId>
<version>0.9.2.1</version>
</dependency>


Change to Spring config where data source is created

<bean id="dataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource" destroy-method="close">
<property name="driverClass" value="com.mysql.jdbc.Driver" />
<property name="jdbcUrl" value="${LIBRARY_JDBC_CONNECTION_STRING}" />
<property name="user" value="${LIBRARY_DATASOURCE_USER_NAME}" />
<property name="password" value="${LIBRARY_DATASOURCE_USER_PASSWORD}" />

<property name="idleConnectionTestPeriod" value="${C3P0_POOL_IDLE_CONNECTION_TEST_PERIOD}"/> 
<property name="preferredTestQuery" value="select 1"/>

<!-- performance improvements and configuration so that you don't get above error from: http://javatech.org/2007/11/c3p0-connectionpool-configuration-rules-of-thumb/ -->
<property name="acquireIncrement" value="${C3P0_POOL_ACQUIRE_INCREMENT}"/>
<property name="maxIdleTime" value="${C3P0_POOL_MAX_IDLE_TIME}"/>
<property name="maxIdleTimeExcessConnections" value="${C3P0_POOL_MAX_IDLE_TIME_EXCESS_CONNECTIONS}"/>
<property name="maxPoolSize" value="${C3P0_POOL_MAX_POOL_SIZE}"/>
<property name="minPoolSize" value="${C3P0_POOL_MIN_POOL_SIZE}"/>
<property name="numHelperThreads" value="${C3P0_POOL_NUM_HELPER_THREADS}"/>
<property name="unreturnedConnectionTimeout" value="${C3P0_POOL_UNRETURNED_CONNECTION_TIMEOUT}"/>  
</bean>

Example  values

C3P0_POOL_ACQUIRE_INCREMENT : 1 
C3P0_POOL_IDLE_CONNECTION_TEST_PERIOD : 300 
C3P0_POOL_MAX_IDLE_TIME : 3600 
C3P0_POOL_MAX_IDLE_TIME_EXCESS_CONNECTIONS : 2 
C3P0_POOL_MAX_POOL_SIZE : 4 
C3P0_POOL_MIN_POOL_SIZE : 2 
C3P0_POOL_NUM_HELPER_THREADS :2 
C3P0_POOL_UNRETURNED_CONNECTION_TIMEOUT : 3600

References
http://www.mchange.com/projects/c3p0/

Best way to Identifying integer value from String

Best way to Identifying integer value from String


if(stringVariable.trim().matches("^\\d*$")){
// true this is a integer
}

To Decimal number like 1.23

if(("1.23".matches("^\\d+\\.\\d{2}$")){
// this will be truw
}

Building a Complex Regular Expressions

I follow this way to build a very complex regular expression. But this has to be start from very small

• /v1/user  -  /(\\w*)/(\\w*)
• v1/203i03 -  /(\\w*)/(\\w*)
• v1 -   /(\\w*)/*(\\w*)

• /v1/user?name  -  /(\\w*)/(\\w*)(\\?*)(\\w*)
• /v1/user?name=don  -  /(\\w*)/(\\w*)(\\?*)(\\w*)(\\=*)(\\w*)

Java Code to Test the regular expression

private static void getEndPointUrl(String restUrlPattern,String urlToMatch) {
Pattern pattern = Pattern.compile(restUrlPattern);
Matcher matcher = pattern.matcher(urlToMatch);
System.out.println(restUrlPattern + " -- " +urlToMatch);
if (matcher.matches()) {
System.out.println("Matches");;
} else {
System.out.println("Not Matches");;
} }

Get a JSON Object from HTML form

Found this great JS library which can be used to serialize an HTML Form to a JavaScript Object

Example HTML form
<form id="my-form">
  <input type="text" name="name"              value="david" />

  <!-- object -->
  <input type="text" name="address[city]"         value="Melbourne" />
  <input type="text" name="address[state][name]"  value="Victoria" />
  <input type="text" name="address[state][abbr]"  value="VIC" />
</form>

Java Script
var jsonData = $('#my-form').serializeJSON();

Return value
{
  name: "david",

  address: {
    city: "Melbourne",
    state: {
      name: "Victoria",
      abbr: "VIC"
    }
  }
}

Install

Download the jquery.serializeJSON.min.js(https://raw.github.com/marioizquierdo/jquery.serializeJSON/master/jquery.serializeJSON.min.js) script and include in your page after jQuery, for example:

<script type="text/javascript" src="jquery.min.js"></script>
<script type="text/javascript" src="jquery.serializeJSON.min.js"></script>

Reference :https://github.com/marioizquierdo/jquery.serializeJSON