Java Performance, Lack Thereof

I am having an interesting issue with the performance of a Java program. The job of this program is to suck a lot of data out of a database and dump it to a file (with some formatting). Easy enough but for some reason it is really slow an defies speeding up.

The first issue is that the program manages about 25-50 queries/second against the MySQL server (as measured buy innotop). I know the MySQL server can manage about 2,500 queries/second because I logged all the queries emitted by the dump process (using ‘log’ in my.cnf,) extracted all queries (the log includes tracking information,) and ran that using the ‘mysql’ command line tool.

What is odder is that when I check the state of the machine itself using ‘atop’ I can see that the Java process is using about 5-10% of one CPU, and MySQL is using about 2%-5% of one CPU, so the machine is basically doing very little. Yet the process just does not want to speed up.

A colleague suggested running multiple threads in the program which I tried but I still get the same crummy performance, actually the performance got a little worse.

This is more than a little frustrating as I come from the C side of things where a program will pretty much us all the CPU power it can get if you let it, and I like that ‘pedal-to-the-metal unless restrained’ approach. Java seems to take a ‘restrained at all costs’ approach for some reason.

It looks like I am going to have to rethink my approach completely here.