I am currently exporting csv file using several ways among the which the fastest seems to be:
var dataset = databaseManager.convertToDataSet(foundset,new Array(‘field1’,‘fiedl2’,…,‘fieldn’);
data = dataset.getAsText(“;”, “\r\n”, null, true);
var $result = plugins.file.writeTXTFile($file,data);
but that goes slow over 20k lines, and lasts for ages with 200k lines (more than 1 hour…). any improvement? or can I write to a file by appending to break the process in several steps (as I can do in php)?
beyond my understanding this…
there are commands I have never seen. for java fluent people I guess.
sometimes I regret my filemaker…
so no easy solution inside servoy then.
lesouef:
so no easy solution inside servoy then
Nope
lesouef:
beyond my understanding this…
there are commands I have never seen. for java fluent people I guess.
I think you should try to understand it. It’s quite straightforward.
I’m not at all a Java programmer myself, but these are just a few commands just to handle the append stuff.
If you look at the code I posted I guess you can almost copy it 1-on-1 and have it work.
You won’t regret the effort, because it performs great!!!
just tried the code, does write to file. I’ll try what speed I get with this on the real db 2morrow (I am back home), and let you know.
do you recommand writing line by line or 1k records at a time?
what makes me upset is why is this missing from the IDE commands? do they expect me to read the whole forum?
and what is boring too is, while data is in memory for the current foundset, I still have to create a new dataset (more memory!) to put it in a file, and that takes ages too and servoy seems to reach a limit aound 700MB of memory beyond which it stays frozen.
I may also try to exporting using SQL OUTFILE if mssql supports thats as mysql does; I use db2 most of the time and this is not supported.
thanks for the advice anyway, I’ll let you know
lesouef:
do you recommand writing line by line or 1k records at a time?
if you are in memory trouble already I’d recommend to keep the size as small as possible.
As I had memory issues using the ‘writeTxtFile’ function, I came across this code.
Even with very large datasets it works great (although fetching the data from the foundset still takes time…)
lesouef:
what makes me upset is why is this missing from the IDE commands?
Maybe it’s possible to add this to the current file plugin.
Unbelievable, magic effect, 154s to export 247k lines all included, dataset loading and writing file, flushing every 1k lines.
And this was over 4 hours (I never waited untill the end actually).
This does not really needs a plug-in, just to be mentionned somewhere, I losted weeks exporting big files in the past months.
Figures:
154s overvall, 94s for dataset loading and 60s to save the file.
the 60s can be reduced to 30s with a flush every 10k, resulting file is 14.5MB.
I never chatted with you before, but I already owe you a good beer.