Servoy Forum

50MB is pretty big. You can try playing around with the maxClientHeap setting in Servoy Admin.

Note that Servoy's XML is a DOM based API. The advantage is that you can randomly access anything in the xml object. The disadvantage is that you need to read the entire xml document in memory first.

When using xml to backup and restore data, typically a streaming xml api implementation is used: http://en.wikipedia.org/wiki/StAX. There are a number of Java implementations out there. With a little work you could roll one of them up into a Servoy plugin.

But I would think it would be much easier to use your SQL vendor's backup and restore tools for this kind of job (no logic needed in the archive or restore process). If you need to automate from within Servoy there is always application.executeProgram(). Trigger via the headless client plugin or batch processor so it runs on the server.

Thanks for your reply david. Changing maxClientHeap size is not really a good option because the table data might of any size and in some point later I'll face the memory issue again when the data size gets bigger and bigger. As for the direct SQL backup/restore, I can't use that method as I need to read the content of the backup when I'm restoring the data (the backup might be an old backup and to restore that backup I need to manipulate the data to match the current structure of the database and restore it that way) and even if I can manipulate it, I might face this memory issue again.

writing a plugin is a good option but I'm not sure how much work is it for me to do it so I was thinking of checking the file size during the backup procedure and don't let it get bigger than an acceptable amount of size (let's say 500K which is easy to load in RAM) and basically instead of storing the whole table data in one xml file, I break it down to a few, this way reading the files are not going to be an issue (Hopefully!). I'm going to read some documents about xml streaming API's implemented in java to see if it is going to be easy for me to write a servoy plugin using one of those implementations.

Example code that might help:

Code: Select all

function xmlStreamTest() {
   
   importPackage(javax.xml.stream)
   importPackage(java.io)

   // input
   var xml =   <xml>
               <fruit>
                  <title>orange</title>
                  <color>orangish</color>
                  <taste>tangy</taste>
               </fruit>
               <fruit>
                  <title>apple</title>
                  <color>red</color>
                  <taste>refreshing</taste>
               </fruit>
               <fruit>
                  <title>banana</title>
                  <color>yellow</color>
                  <taste>energizing</taste>
               </fruit>
            </xml>
            
   // output
   var jsonFruit = []
   
   // pseudo file
   var tempFile = plugins.file.createTempFile('xmlFile', '.xml');
   plugins.file.writeTXTFile(tempFile, xml);
   
   // initialize XMLStreamReader
   var factory = XMLInputFactory.newInstance()
   var reader   = factory.createXMLStreamReader('testFile', new FileInputStream(tempFile.getAbsolutePath()))
   
   // process xml and transform to json object
   var fruitItem = new Object
   while ( reader.hasNext() ) {   
      reader.next();
      try {
         if ( reader.isStartElement() ) {
            var element = reader.getLocalName()
            switch (element) {
               case "title":
                     fruitItem["title"] = reader.getElementText()
                     break;
               case "color":
                     fruitItem["color"] = reader.getElementText()
                     break;
               case "taste":
                     fruitItem["taste"] = reader.getElementText()
                     break;
            }
         }
         else if ( reader.isEndElement() && reader.getLocalName() == "fruit" ) {
            jsonFruit.push(fruitItem)
            fruitItem = new Object
         }
      } catch (e) {
         application.output(e.message)
      }
   }
}

I'm not sure but doing Java like this in Servoy...probably need to explicitly close stuff down when you're done. A point for the Java gurus to weigh in on.

Yes, as-is your code will leave the stream opened, and lock the file, so you want to add a finally clause with a reader.close();

Thanks David, I have temporarily solved the issue by braking down the xml files into smaller size within the code, so when I read them in the memory they are not going to be big. But I'll give your solution a try soon.