Java Answers Forum - Most Efficent method to transfer Data from Result Set object to flat file

Articles |
News |
Weblogs |
Books |
Forums

Artima Forums | Articles | Weblogs | Java Answers | News

Sponsored Link •

Java Answers Forum
Most Efficent method to transfer Data from Result Set object to flat file

4 replies on 1 page. Most recent reply: Nov 16, 2005 6:08 AM by Kondwani Mkandawire

Welcome Guest
Sign In

Back to Topic List

Reply to this Topic

Search Forum

Threaded View


Previous Topic		Next Topic

Flat View: This topic has 4 replies on 1 page

Jigar Shah

Posts: 2
Nickname: liveinjava
Registered: Nov, 2005

Most Efficent method to transfer Data from Result Set object to flat file

Posted: Oct 31, 2005 10:48 PM

I have a huge database in oracle of about 800MB
i get result set object i need to apply some
formatting on that data and store it in the
flat file.
I have tryed with JAVA IO to write in file
it takes about 38 min.
If i can achive within 20 min. it would be
helpful.
Can any one suggest some solution may be in
NIO or oracle specific but not any native lib.

Thanks.

Kondwani Mkandawire

Posts: 530
Nickname: spike
Registered: Aug, 2004

Re: Most Efficent method to transfer Data from Result Set object to flat fi

Posted: Nov 1, 2005 3:43 AM

> I have a huge database in oracle of about 800MB
> i get result set object i need to apply some
> formatting on that data and store it in the
> flat file.

800Megs?? Wooh hoo, that's a harsh one...
Naturally the Result Set even on a Large Database
won't be as Large as 800Megs though. Unless you
mean the Table carries 800 megs of data on a much
larger database and you are performing:

select * from this_800_meg_talbe;

How much time are you taking up on the connection
and data-fetching. That might be where the bulk
of your time is spent, especially if you are not
Pooling your connections whilst dealing with something
that large.

> I have tryed with JAVA IO to write in file
> it takes about 38 min.
> If i can achive within 20 min. it would be
> helpful.
> Can any one suggest some solution may be in
> NIO or oracle specific but not any native lib.
>
> Thanks.

Kondwani Mkandawire

Posts: 530
Nickname: spike
Registered: Aug, 2004

Re: Most Efficent method to transfer Data from Result Set object to flat fi

Posted: Nov 1, 2005 3:48 AM

Sorry a much simplified version of my above blabbering would
be the following questions.

Is the Result Set itself 800Megs or the Database is
totalling 800Megs? Reason I ask is that your connection
algorithms may be coming into play here.

Another question would be, are you using connection
pooling?

One last one would be are your using the Statement
class or the PreparedStatement class to attain your
ResultSet? (I read on one of the Java sites that the
former is not as efficient as the latter).

Collectively, these might help in improving on speed.

Jigar Shah

Posts: 2
Nickname: liveinjava
Registered: Nov, 2005

Re: Most Efficent method to transfer Data from Result Set object to flat file

Posted: Nov 1, 2005 6:33 AM

Thanks

Acually according to the worst case the
database tables which i need to write in to flat files
is 800 mb.

I am using DAO Factory pattern and i am getting
result set from the database connection pooling
will be there which might be managed by the
container once i have a result set i have a provision
that i might get chunks of data in arraylist.

the use of prepared statment is a good suggetion
which was the first thing i did still so performace
gain i could get.

According to one of my test case, once i am getting
resultset which may be huge i need to write it to
file that time should as less as possible.

The whole process of selecting table rows and dumping
it to flat files takes about 45 mins.

I need to find a solution to reduce file IO once i get
resultset object or a collection.

Kondwani Mkandawire

Posts: 530
Nickname: spike
Registered: Aug, 2004

Re: Most Efficent method to transfer Data from Result Set object to flat fi

Posted: Nov 16, 2005 6:08 AM

> I need to find a solution to reduce file IO once i get
> resultset object or a collection.

Apparently if you write away data without buffering it, it
involves writing it away byte by byte. Apparently using
BufferedOutputStream should improve the performance.

http://www.precisejava.com/javaperf/j2se/IO.htm

Flat View: This topic has 4 replies on 1 page

Previous Topic

Next Topic


	Web Artima.com