Page 2 of 2

Re: CSV Parser method

PostPosted: Thu Sep 22, 2011 1:30 pm
by patrick
rogel wrote:We are doing some testing this moment because we see that the foundset filling speed (rows added per second) drops at a certain (memory ?) moment. We save the data during the import to the database at a regular interval of 500 records. (Yes, autosave is off)


Are you using a transaction? Are you clearing the foundset after your 500 records? I think both would speed up things and prevent memory issues...

Re: CSV Parser method

PostPosted: Wed Dec 07, 2011 9:36 am
by rogel
hi!

i am importing a large amount of data, using smart client and it very much slows down averaging on 2 records per second using developer. :(

here is my logic.
read csv file per line using the parser
for each line
1. create the foundset records and related foundsets(its 5 tables deep)
2. databaseManager.saveData()

is there any suggestions on how to speed things?

i am thinking of using a headlessclient. would it perform any difference if testing in developer?

thanks!

Re: CSV Parser method

PostPosted: Wed Dec 07, 2011 11:38 am
by rgansevles
Rogel,

I would not call saveData for each record.
Try saving every 100th record or so.

Rob

Re: CSV Parser method

PostPosted: Thu Jan 19, 2012 7:38 am
by rogel
patrick wrote:
rogel wrote:We are doing some testing this moment because we see that the foundset filling speed (rows added per second) drops at a certain (memory ?) moment. We save the data during the import to the database at a regular interval of 500 records. (Yes, autosave is off)


Are you using a transaction? Are you clearing the foundset after your 500 records? I think both would speed up things and prevent memory issues...


What do you mean by "clearing the foundset"?

Re: CSV Parser method

PostPosted: Thu Jan 19, 2012 10:56 am
by Harjo
if you add many records to a foundset, also once in while do: foundset.clear()
that will keep the speed up, and the memory low,

Re: CSV Parser method

PostPosted: Mon Jan 23, 2012 3:05 am
by rogel
Harjo wrote:if you add many records to a foundset, also once in while do: foundset.clear()
that will keep the speed up, and the memory low,


when do you perform the foundset.clear? would it be like the snippet below?

Code: Select all
for (...) {
fs.newRecord();
if (index % 100 == 0) {
databaseManager.saveData(fs);
fs.clear();
}
}