CSV Parser method

Share business templates, ideas, experiences, etc with fellow Servoy developers here

Re: CSV Parser method

Postby patrick » Thu Sep 22, 2011 1:30 pm

rogel wrote:We are doing some testing this moment because we see that the foundset filling speed (rows added per second) drops at a certain (memory ?) moment. We save the data during the import to the database at a regular interval of 500 records. (Yes, autosave is off)


Are you using a transaction? Are you clearing the foundset after your 500 records? I think both would speed up things and prevent memory issues...
Patrick Ruhsert
Servoy DACH
patrick
 
Posts: 3703
Joined: Wed Jun 11, 2003 10:33 am
Location: Munich, Germany

Re: CSV Parser method

Postby rogel » Wed Dec 07, 2011 9:36 am

hi!

i am importing a large amount of data, using smart client and it very much slows down averaging on 2 records per second using developer. :(

here is my logic.
read csv file per line using the parser
for each line
1. create the foundset records and related foundsets(its 5 tables deep)
2. databaseManager.saveData()

is there any suggestions on how to speed things?

i am thinking of using a headlessclient. would it perform any difference if testing in developer?

thanks!
rogel
 
Posts: 264
Joined: Mon Jul 04, 2011 9:09 am

Re: CSV Parser method

Postby rgansevles » Wed Dec 07, 2011 11:38 am

Rogel,

I would not call saveData for each record.
Try saving every 100th record or so.

Rob
Rob Gansevles
Servoy
User avatar
rgansevles
 
Posts: 1927
Joined: Wed Nov 15, 2006 6:17 pm
Location: Amersfoort, NL

Re: CSV Parser method

Postby rogel » Thu Jan 19, 2012 7:38 am

patrick wrote:
rogel wrote:We are doing some testing this moment because we see that the foundset filling speed (rows added per second) drops at a certain (memory ?) moment. We save the data during the import to the database at a regular interval of 500 records. (Yes, autosave is off)


Are you using a transaction? Are you clearing the foundset after your 500 records? I think both would speed up things and prevent memory issues...


What do you mean by "clearing the foundset"?
rogel
 
Posts: 264
Joined: Mon Jul 04, 2011 9:09 am

Re: CSV Parser method

Postby Harjo » Thu Jan 19, 2012 10:56 am

if you add many records to a foundset, also once in while do: foundset.clear()
that will keep the speed up, and the memory low,
Harjo Kompagnie
ServoyCamp
Servoy Certified Developer
Servoy Valued Professional
SAN Developer
Harjo
 
Posts: 4321
Joined: Fri Apr 25, 2003 11:42 pm
Location: DEN HAM OV, The Netherlands

Re: CSV Parser method

Postby rogel » Mon Jan 23, 2012 3:05 am

Harjo wrote:if you add many records to a foundset, also once in while do: foundset.clear()
that will keep the speed up, and the memory low,


when do you perform the foundset.clear? would it be like the snippet below?

Code: Select all
for (...) {
fs.newRecord();
if (index % 100 == 0) {
databaseManager.saveData(fs);
fs.clear();
}
}
rogel
 
Posts: 264
Joined: Mon Jul 04, 2011 9:09 am

Previous

Return to Sharing Central

Who is online

Users browsing this forum: No registered users and 5 guests