Hi
In reading in quite a big file, i. e. 160 MB, I get this error:
org.mozilla.javascript.JavaScriptException: java.lang.OutOfMemoryError: Java heap space
I need to read in this file for several preprocessing steps before inserting it into a table (programmatically, not by Servoy import).
Where can I assign JavaScript more memory?
Thanks for help, Robert
When you do this in developer you can find the settings either on the forum or in the help files (I think).
In general you need to look for the .plist in a os x package or the batch file on windows.
When it is the client you should use servoy-admin to add memory to a client.
And if you are using Servoy Client you can set the memory usage on the Admin Pages (servoy.maxClientHeap), remember to redownload the client JNLP after changing this.
When it is the client you should use servoy-admin to add memory to a client.
Oops…
Didn’t notice that…
Well, two is better than one, no?
Hi Marcel
I am using for this job the Servoy Developer. I couldn’t find something in the forum to the Developer, may be someone can point me to the correct package or .plist - a servoy.plist doesn’t exist (or I did not find one at least .-)
Best regards, Robert
PS: I am using Macs - of course .-)
IT2Be:
When you do this in developer you can find the settings either on the forum or in the help files (I think).In general you need to look for the .plist in a os x package or the batch file on windows.
When it is the client you should use servoy-admin to add memory to a client.
Looking at this from a different perspective: Are you importing a file with many rows?
Might be usefull looking into reading the file per record and if needed for preprocessing: insert each row into a temp table.
Have a look at http://forum.servoy.com/viewtopic.php?t=6391
That tpoic is about writing a file, but in Java the following also exists:
FileReader and BufferedReader.
Just a thought…
Paul
Hi Paul
Yeah, quite many rows, around one and a half millions
I am already reading row by row and inserting it as such into the db. I don’t think I need a temp table as there is no problem with the table, it’s a problem with the memory heap in JavaScript.
Can I set the allocated memory to JavaScript or is it dynamically allocating memory?
By the way, another problem (same category) is that we tried to initialize an array with an index of 8’000’000’000 (8 millon). But that seems too much for Servoy JavaScript .-) but only one array element filled. Now it seems that it’s trying to allocate statically all the leading array elements. Isn’t it allocating memory dynamically and only eserving memory for the one element currently assigned to the array?
Any insights appreciated as we should (have) already delivered results but now we are stuck at memory problems.
Best regards, Robert
pbakker:
Looking at this from a different perspective: Are you importing a file with many rows?Might be usefull looking into reading the file per record and if needed for preprocessing: insert each row into a temp table.
Have a look at http://forum.servoy.com/viewtopic.php?t=6391
That tpoic is about writing a file, but in Java the following also exists:
FileReader and BufferedReader.Just a thought…
Paul
Hi,
dunno how you read the file currently line by line, but with the technique I suggest, if I’m correct, you only read 1 line into memory, not the whole file, so memory consumption is way lower.
As for initializing an array with an index of 8 million: if you initialize an array with a certain lenght, Java will reserve the memoryspace allready. Al least that is my understanding of it. You can see that for yourself: look at the momory usage before you initialize the array and straight after.
Paul
Robert Huber:
Where can I assign JavaScript more memory?
Well, you can’t assign more memory to Javascript specifically, but you can assign more to Servoy.
Here’s how: open your servoy_developer.command (or .sh or .bat) file. In mine, there’s a line that looks like this:
java -XX:MaxPermSize=128m -Xmx384m -Xms64m -jar servoy_developer.jar
The number after “Xmx” is the maximum amount of heap memory in megabytes (hence the trailing “m”) that Servoy is allowed to use. That’s what you need to increase. For instance, for 800 megabytes, the line should look like this:
java -XX:MaxPermSize=128m -Xmx800m -Xms64m -jar servoy_developer.jar
Don’t overdo it though. You should keep it well below the physical amount of memory in your system. If you get the number too high, your system will swap like crazy when it uses a lot of memory - effectively grinding it to a halt.
Hi Sandor
Thanks for the tip. Unfortunatly it doesn’t influence the error (org.mozilla.javascript.JavaScriptException: java.lang.OutOfMemoryError: Java heap space) in a positive way.
I very much hope we find another idea of how to assign more memory to JavaScript somehow, as we absolutly have to parse this file (and fill up tables with the parsed data). Soem time ago we developed a parser in FileMaker which works (but quite slow). Now, as we sold Servoy telling the customer that everything is better .-) how should we tell him we can’t open and parse a large file in Servoy??? Huchhh. I hope it’s not true otherwise we come into quite some troubles, as I never thought this could be a problem with Servoy.
In hope for a solution with best regards, Robert
PS: Is there anyone out there working with big files in the range of 1 - 2 GB?
svroemisse:
Robert Huber:
Where can I assign JavaScript more memory?Well, you can’t assign more memory to Javascript specifically, but you can assign more to Servoy.
Here’s how: open your servoy_developer.command (or .sh or .bat) file. In mine, there’s a line that looks like this:
java -XX:MaxPermSize=128m -Xmx384m -Xms64m -jar servoy_developer.jarThe number after “Xmx” is the maximum amount of heap memory in megabytes (hence the trailing “m”) that Servoy is allowed to use. That’s what you need to increase. For instance, for 800 megabytes, the line should look like this:
java -XX:MaxPermSize=128m -Xmx800m -Xms64m -jar servoy_developer.jarDon’t overdo it though. You should keep it well below the physical amount of memory in your system. If you get the number too high, your system will swap like crazy when it uses a lot of memory - effectively grinding it to a halt.
Hi Robert, I think your issue is more related HOW you do it.
So please send us your code, so we can look at it.
(as said before, you can stream(read) a file, line by line, this will give you no memory-issues.)
Hello Harjo
HJK:
Hi Robert, I think your issue is more related HOW you do it.
I am glad if it is so
HJK:
So please send us your code, so we can look at it.
(as said before, you can stream(read) a file, line by line, this will give you no memory-issues.)
Attached is the code. The problem seems to be opening the file!?
Thanks for help!
Best regards, Robert
readTXTfile doesn’t only open the file but reads in memory in FULL…at least it tries to.
Paul Bakker’s solution will apply here
Hi Robert
The easiest would be to allocate enough memory, wouldn’t it? Isn’t that possible?
ROCLASI:
readTXTfile doesn’t only open the file but reads in memory in FULL…at least it tries to.
Paul Bakker’s solution will apply here
Do you mean Jan Blocks suggestion for writing a log file?
I assume the code then would be something like:
// br is a global (type media) — correct?
var file = new Packages.java.io.FileReader(‘/tmp/xyz.csv’);
global.br = new Packages.java.io.BufferedReader(file);
Would reading a line be like this?
global.br.read(file);
How to position the point to the first line of the file? Is this implicit?
Repeating the read command reads the next line?
Thanks, Robert
Google on “java.io.BufferedReader”
use global.readLine()
Paul
