The Artima Developer Community
Articles | News | Weblogs | Books | Forums
Artima Forums | Articles | Weblogs | Java Answers | News
Sponsored Link •

Java Answers Forum
Handling files efficiently

5 replies on 1 page. Most recent reply: Aug 7, 2006 11:53 PM by mohammedkazim hussain

Welcome Guest
  Sign In

Go back to the topic listing  Back to Topic List Click to reply to this topic  Reply to this Topic Click to search messages in this forum  Search Forum Click for a threaded view of the topic  Threaded View   
Previous Topic   Next Topic
Flat View: This topic has 5 replies on 1 page
mohammedkazim hussain

Posts: 3
Nickname: kazimh
Registered: Jul, 2006

Handling files efficiently Posted: Jul 29, 2006 8:53 AM
Reply to this message Reply
Advertisement
Hi,

I need to read and parse files of large size, maximum of 4-5GB
within less time using java.io.
Which is the best approach to apply.
Please help, as it is urgent.


Thanks,


Matthias Neumair

Posts: 660
Nickname: neumi
Registered: Sep, 2003

Re: Handling files efficiently Posted: Jul 31, 2006 12:56 AM
Reply to this message Reply
You have to give us more information than that.

What sort of data does the file contain?

What do you need to to with it (explain 'handle')?

Why does java.io not work fo you?

mohammedkazim hussain

Posts: 3
Nickname: kazimh
Registered: Jul, 2006

Re: Handling files efficiently Posted: Aug 2, 2006 5:24 AM
Reply to this message Reply
Actually I need to read , parse the flat files then insert the required data from the files to the database(oracle).

As the files size is huge.Which process would be efficient.

Thanks,

Matthias Neumair

Posts: 660
Nickname: neumi
Registered: Sep, 2003

Re: Handling files efficiently Posted: Aug 2, 2006 11:40 PM
Reply to this message Reply
I don't think you can do it any faster than using the standard buffered reader.

It could improve the efficiency if you don't parse and store after reading each line, but to collect for example a thousand lines, then parse them and store them with a prepared statement.

Werner Schulz

Posts: 18
Nickname: werner
Registered: May, 2005

Re: Handling files efficiently Posted: Aug 4, 2006 12:16 PM
Reply to this message Reply
What kind of parsing do you need to do?

If the file is already in CSV format, Oracle can read them using bulk loading operations, no need to use Java (or any other language). May need to place the column names at top of the file.

More questions:
- Is it a one-off operation?
- CVS or similar spreadsheet format?
- Can data be read in blocks (or even just lines)?

Again, more details are needed before anyone can advise on a good approach. The best approach? Only testing would show but I don't think you have time for that.

mohammedkazim hussain

Posts: 3
Nickname: kazimh
Registered: Jul, 2006

Re: Handling files efficiently Posted: Aug 7, 2006 11:53 PM
Reply to this message Reply
Thank you for your reply.
Yes,it is one-off operation.
Actually the funtionality of application is to collect the delivered files and store into database.
Files format would be CSV,text files.No need of keeping all the file contents into database only required values has to be stored into database.

File size may be 1-2 GB.
As it should be automated,on one operation files should be processed.

Flat View: This topic has 5 replies on 1 page
Topic: java.util.Collection type casting Previous Topic   Next Topic Topic: JRE & JVM
Sponsored Links


Google
  Web Artima.com   

Copyright © 1996-2019 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use