java heap space error

When I re-start an import after trying it at least once, I get a web page error (“internal error”) instructing the user to return to the home page. In the server log, there is a message

com.servoy.j2db.util.Debug Error rendering the page [Page class = com.servoy.j2db.server.headlessclient.MediaUploadPage, id = 13, version = 0]

Looking part way down in the red section below this message, I see the following items,

Caused by: java.lang.reflect.InvocationTargetException
Caused by: java.lang.OutOfMemoryError: Java heap space

There was some discussion here of java heap space, but I don’t see where to modify it. Is this what needs to be done?

Thank you,
Don

depends what or how you exactly start
but you need to set the -Xmx1280m of the java that you start (to a value that works for you)
see servoy_server.bat/sh or the wrapper.conf if you start the server as a service

Here is what is in that file:

#!/bin/sh

while true
do
java -Djava.awt.headless=true -Xmx1280m -Xms64m -XX:MaxPermSize=128m -classpath .:lib/activation.jar:lib/antlr.jar:lib/apache-mime4j.jar:lib/BrowserLauncher2.jar:lib/commons-codec.jar:lib/commons-collections.jar:lib/commons-dbcp.jar:lib/commons-fileupload.jar:lib/commons-io.jar:lib/commons-logging.jar:lib/commons-pool.jar:lib/dom4j.jar:lib/hibernate3.jar:lib/httpclient.jar:lib/httpclient-cache.jar:lib/httpcore.jar:lib/httpmime.jar:lib/j2db.jar:lib/j2dbdev.jar:lib/jabsorb.jar:lib/javassist.jar:lib/jcifs.jar:lib/joda-time.jar:lib/js.jar:lib/jta.jar:lib/jug.jar:lib/log4j.jar:lib/mail.jar:lib/MRJAdapter.jar:lib/networktnl.jar:lib/rmitnl.jar:lib/server-bootstrap.jar:lib/servlet-api.jar:lib/slf4j-api.jar:lib/slf4j-log4j.jar:lib/wicket.jar:lib/wicket-calendar.jar:lib/wicket-extentions.jar com.servoy.j2db.server.main.ApplicationServer “$@” 1>> server.log 2>> server.log
EXITCODE=$?
if [ “$EXITCODE” != 99 ]; then exit $EXITCODE; fi
done

I see where it indicates “Xmx1280m”, but I don’t see a value…?

1280 is the vallue…
That is already quite large. Do you start the app server by running that .sh file?
How big is the file that you upload??

All I am doing is clicking on the button to start the test web browser, from Servoy Developer (6.0.eight). This is the green, right-arrow button with the yellow bow tie, which is located in the toolbar.

The file that I am importing is a text file of approximately 43 MB. This is actually only a moderate-sized file, for this operation.

Thank you,
Don

ok so your are testing this in the developer, then i guess your developer must have enough memory
this can be controlled in the servoy.ini file just besides the servoy.exe file that you start. in there you also have a Xmx value

djlapin:
The file that I am importing is a text file of approximately 43 MB. This is actually only a moderate-sized file, for this operation.

Don, beside the problem you are having you should consider reading the file line by line, it will be more efficient.

I just did a Spotlight search for a servoy.ini file, and found nothing. There is a config.ini file in the developer/configuration folder, but it doesn’t have any “xmx” text.

Is this an optional file?

Hi Nicola,

I am reading the file line-by-line from a global variable. The user uploads the file from the web browser to the server, so it’s already stored in memory, right? After the import, I set the variable equal to blank (“”). Once the variable is cleared (after the import), shouldn’t the memory be recovered? Do you suggest doing something else with it once it arrives on the server?

Thank you,
Don

How do you upload to the server?
I would use plugins.file.streamFilesToServer() to create a file on the server filesystem, in the callback method I would read it line by line without loading it entirely in memory and at the end I would delete the server file.

Reading it line by line can be done like this:

var _fileReader = new Packages.java.io.FileReader('/uploadFolder/uploadedFile.txt'); 

var _bufferedReader = new Packages.java.io.BufferedReader(_fileReader); 
for(.......)
{
   var _myLine = br.readLine(); 
   ...
}

_bufferedReader.close();//do NOT forget this close! to prevent mem leaks, even better to use try/finally here

Hi Nicola,

I am using the web browser select file dialog,

plugins.file.showFileOpenDialog(null, null, false, globals.import_SELPA_central, 'Select the import file')

…which then gets the text as a callback,

function import_SELPA_central(callbackArray) {
     ...
     importFileText = plugins.file.readTXTFile(callbackArray[0]);	
     if(importFileText) {
          ...
     }
}

I will need to figure out how to send the file to disk on the server, without placing it in memory first – is that correct? I will look at that streamFilesToServer command.

Thank you,
Don

Correct, it should be the cheapest way.

Hi Nicola,

When I try this,

var fileName = plugins.file.showFileOpenDialog(1, null, false, null, null, 'Select the import file');
if(fileName) {
	plugins.file.streamFilesToServer(fileName, globals.import_SELPA_central);
}

…I get an error,

Failed to execute the method of context browse_form and name browse_import on the solution Selpa_Mgr_SQL
Wrapped java.lang.RuntimeException: Function callback not set for webclient (/Users/djlapin/servoy_workspace_20120313/Selpa_Mgr_SQL/forms/browse_form.js#7679)
at /Users/djlapin/servoy_workspace_20120313/Selpa_Mgr_SQL/forms/browse_form.js:7679 (browse_import)

From what I can divine from reading other threads, the streamFilesToServer command doesn’t work with webclient.

So I went back to the original arguments in showOpenFileDialog. I then tried this on the server end, in an attempt to replace the “plugins.file.readTXTFile(callbackArray[0])”:

var vFile = plugins.file.convertToJSFile(callbackArray[0]);
var _fileReader = new Packages.java.io.FileReader(vFile.getName());
var _bufferedReader = new Packages.java.io.BufferedReader(_fileReader);
var _myLine = _bufferedReader.readLine();

I also tried it this way, based on some earlier feedback I received,

var fileObject = callbackArray[0];
var serverFile = plugins.file.createFile(fileObject);
plugins.file.writeFile(serverFile,serverFile.getBytes())
		
var _oFR = new Packages.java.io.FileInputStream(serverFile),
_oIR = new Packages.java.io.InputStreamReader(_oFR, "UTF8"),
_oBR = new Packages.java.io.BufferedReader(_oIR),
_sLine = "dummy",
_nReadLine = 0;

…but in both cases I got an error that the file doesn’t exist. What am I missing?

Where does readLine() stop? This text file does not have the normal carriage return/linefeed delimiters; I parse it based on specific high-ascii characters.

Thank you,
Don

on the mac it is i think somewhere in that package file, but a mac user should be able to tell you more.

if you call this: plugins.file.writeFile(serverFile,serverFile.getBytes()) then all bytes will be transfered to memory first.

the error you get about the call back is not about streamToServer, but to your call:
var fileName = plugins.file.showFileOpenDialog(1, null, false, null, null, ‘Select the import file’);

this can’t be done in the webclient like that, you have to give the last callback argument… And in that argument you get 1 or an array of files
and you could use stream to server for that by giving it.
Yes it doesn’t really do the actual background streaming from 1 client to the server, because you are already on the server.
But for you the large JSFile you get is then streamed to disk so it is a good choice to use streamToServer because then not the whole file has to be in memory.
There is special support for this situation build in in the FilePlugin

Johan,

What change(s) do I need to make in the code snippet to make it work?

Thank you,
Don

var fileObject = callbackArray[0];
var serverFile = plugins.file.createFile(fileObject);
plugins.file.writeFile(serverFile,serverFile.getBytes())
      
var _oFR = new Packages.java.io.FileInputStream(serverFile),
_oIR = new Packages.java.io.InputStreamReader(_oFR, "UTF8"),
_oBR = new Packages.java.io.BufferedReader(_oIR),
_sLine = "dummy",
_nReadLine = 0;

no dont go that route at least not that way

plugins.file.showFileOpenDialog(1, null, false, null, null, 'Select the import file', mycallbackfunction);

function mycallbackfunction(file) {
   plugins.file.streamFilesToServer(file, globals.import_SELPA_central);
}

something like that.

Hi Johan,

As soon as the showFileOpenDialog line is executed, I get this error in the console,

Failed to execute the method of context browse_form and name browse_import on the solution Selpa_Mgr_SQL
Wrapped java.lang.RuntimeException: Function callback not set for webclient (/Users/djlapin/servoy_workspace_20120313/Selpa_Mgr_SQL/forms/browse_form.js#7680)
at /Users/djlapin/servoy_workspace_20120313/Selpa_Mgr_SQL/forms/browse_form.js:7680 (browse_import)

So it doesn’t even get to the streamFilesToServer. There is also a yellow warning symbol next to the showFileOpenDialog line,

The method showFileOpenDialog([?],[?],[?],[?],[?],[?]) in the type Plugin is not applicable for the
arguments (Number,null,Boolean,null,null,String,Function)

Thank you,
Don

Hi Don,

Johan’s code has some errors in them. Too many arguments and the last 2 arguments are switched.
The syntax is:

showFileOpenDialog(selectionMode, startDirectory, multiselect, filter, callbackfunction, title)

So Johan’s code:

plugins.file.showFileOpenDialog(1, null, false, null, null, 'Select the import file', mycallbackfunction);

Should be:

plugins.file.showFileOpenDialog(1, null, false, null, mycallbackfunction, 'Select the import file');

Anyway, here is some (tested) code that does what you are looking for I think.
The first method ‘btnUpload’ shows the upload dialog that calls ‘uploadCallbackFunction’ when the user triggers the upload.
This method will stream the file to the filesystem of the server from where it gets imported line by line by the callback function ‘doImportFile’.

/**
 * @param {JSEvent} event the event that triggered the action
 */
function btnUpload(event) {
    // show the upload dialog and call the callback function when the user his Upload
    plugins.file.showFileOpenDialog(1, null, false, null, uploadCallbackFunction, 'Select the import file');
}

/**
 * @param {plugins.file.JSFile} _oFile
 */
function uploadCallbackFunction(_oFile) {
    // Streaming the file to the server and call the callback method when this is done
    plugins.file.streamFilesToServer(_oFile, doImportFile);
}

/**
 * @param {plugins.file.JSFile} _oFile
 */
function doImportFile(_oFile) {
    // We need to add the upload path defined in the Servoy-Admin pages to the filename 
    _oFile = plugins.file.getDefaultUploadLocation() + "/" + _oFile.getName();

    //
    // Use BufferedReader so we don't have to read the whole file into memory
    //
    var _oFR = new Packages.java.io.FileInputStream(_oFile),
        _oIR = new Packages.java.io.InputStreamReader(_oFR, "UTF8"),
        _oBR = new Packages.java.io.BufferedReader(_oIR),
        _sLine = "dummy",
        _nReadLine = 0;

    // using a database transaction (might/will) speed things up
    databaseManager.startTransaction();

    try {
        while (_sLine) {
            _sLine = _oBR.readLine();
            _nReadLine++;

            if (_sLine) {

                // Put your processing code here
            }
        }
        // Save any unsaved data
        databaseManager.saveData();

        //
        //do NOT forget this close! to prevent memory leaks
        //
        _oBR.close();

        // Close the database transaction
        databaseManager.commitTransaction();

    } catch (_oErr) {
        _oBR.close();
        application.output("ERROR: " + _oFile.getName() + " at row " + _nReadLine, LOGGINGLEVEL.ERROR);
        application.output("ERROR: " + _oErr, LOGGINGLEVEL.ERROR);
        databaseManager.rollbackTransaction();
    } finally {
        //
        // garbage collection
        //
        _oFR = null;
        _oIR = null;
        _oBR = null;
    }
}

Hope this helps.

Hi Robert,

I am now able to see contents in _sLine.

Servoy presents a warning when I use “@param {plugins.file.JSFile} _oFile”, but seems okay with using {Array} as the parameter type. Does this matter?

There’s some additional complexity: I read (and then remove) a header from the top of the file, and present the information from the header for the user to confirm (continue) with the import.

In order to be able to trace/debug, I use globals.DIALOGS.showFormInModalDialog rather than mod_dialog to display the confirm dialog, and the OK button on the dialog calls the next import method. It’s in this next method that I am parsing the incoming text, and reducing the size of _sLine.

So the code needs to be more like,

 try {
       if(_sLine) {
            // parse header from _sLine and present a dialog for user to confirm that import should proceed;
            // from dialog OK button somehow continue with this loop
       }
        // this needs to be in the next (called) import method(?) --
        while (_sLine) {
            _sLine = _oBR.readLine();
            _nReadLine++;
            if (_sLine) {
               // parse text out of _sLine and reduce the size of _sLine              
            }
        }
         _oBR.close();
        databaseManager.commitTransaction();
    } catch (_oErr) {
    ....

How can I do this? It seems like the while loop won’t work because it needs to be in both this method and the next method (the one that actually parses the import).

Also, because I don’t have the entire file in _sLine, is there a way to find the size of the entire file, so that I can keep track of the percentage of the file that I have read?

Thanks so much for your reply,
Don

Hi Don,

djlapin:
Servoy presents a warning when I use “@param {plugins.file.JSFile} _oFile”, but seems okay with using {Array} as the parameter type. Does this matter?

I guess you pass an array of JSFile objects then. You can use {Array} or {Array<plugins.file.JSFile>} or {plugins.file.JSFile}. They all denote an array but the last 2 notations tell Servoy what the array holds as well.

djlapin:
There’s some additional complexity: I read (and then remove) a header from the top of the file, and present the information from the header for the user to confirm (continue) with the import.

In order to be able to trace/debug, I use globals.DIALOGS.showFormInModalDialog rather than mod_dialog to display the confirm dialog, and the OK button on the dialog calls the next import method. It’s in this next method that I am parsing the incoming text, and reducing the size of _sLine.

I guess you can have 2 methods where one reads the header and the second imports the file. Of course with second method you want to skip the lines of the header. This could be a parameter in the function call for instance.
Or you could even make the import method more ‘pluggable’ by also passing the function that will process the _sLine for you so your method can be used for all kinds of file read actions.

In any case you need to keep track of your file object and the last position the import method was at. You could use some global variables for this.

djlapin:
Also, because I don’t have the entire file in _sLine, is there a way to find the size of the entire file, so that I can keep track of the percentage of the file that I have read?

You can get the actual file-size from the JSFile object using the .getSize() function. This will return the size in bytes.
With this info and by keeping track how much you have read from the file you could calculate the progress.

Hope this helps.