How to import data from an XML file?

Find out how to get things done with Servoy. Post how YOU get things done with Servoy

Re: How to import data from an XML file?

Postby david » Tue Jun 18, 2013 4:01 pm

50MB is pretty big. You can try playing around with the maxClientHeap setting in Servoy Admin.

Note that Servoy's XML is a DOM based API. The advantage is that you can randomly access anything in the xml object. The disadvantage is that you need to read the entire xml document in memory first.

When using xml to backup and restore data, typically a streaming xml api implementation is used: http://en.wikipedia.org/wiki/StAX. There are a number of Java implementations out there. With a little work you could roll one of them up into a Servoy plugin.

But I would think it would be much easier to use your SQL vendor's backup and restore tools for this kind of job (no logic needed in the archive or restore process). If you need to automate from within Servoy there is always application.executeProgram(). Trigger via the headless client plugin or batch processor so it runs on the server.
David Workman, Kabootit

Image
Everything you need to build great apps with Servoy
User avatar
david
 
Posts: 1727
Joined: Thu Apr 24, 2003 4:18 pm
Location: Washington, D.C.

Re: How to import data from an XML file?

Postby siavash.roozbehan » Wed Jun 19, 2013 4:08 am

Thanks for your reply david. Changing maxClientHeap size is not really a good option because the table data might of any size and in some point later I'll face the memory issue again when the data size gets bigger and bigger. As for the direct SQL backup/restore, I can't use that method as I need to read the content of the backup when I'm restoring the data (the backup might be an old backup and to restore that backup I need to manipulate the data to match the current structure of the database and restore it that way) and even if I can manipulate it, I might face this memory issue again.

writing a plugin is a good option but I'm not sure how much work is it for me to do it so I was thinking of checking the file size during the backup procedure and don't let it get bigger than an acceptable amount of size (let's say 500K which is easy to load in RAM) and basically instead of storing the whole table data in one xml file, I break it down to a few, this way reading the files are not going to be an issue (Hopefully!). I'm going to read some documents about xml streaming API's implemented in java to see if it is going to be easy for me to write a servoy plugin using one of those implementations.
Siavash

TSM (The Service Manager)
Australia
siavash.roozbehan
 
Posts: 97
Joined: Wed Apr 11, 2012 2:21 am

Re: How to import data from an XML file?

Postby david » Wed Jun 19, 2013 5:55 pm

Example code that might help:

Code: Select all
function xmlStreamTest() {
   
   importPackage(javax.xml.stream)
   importPackage(java.io)

   // input
   var xml =   <xml>
               <fruit>
                  <title>orange</title>
                  <color>orangish</color>
                  <taste>tangy</taste>
               </fruit>
               <fruit>
                  <title>apple</title>
                  <color>red</color>
                  <taste>refreshing</taste>
               </fruit>
               <fruit>
                  <title>banana</title>
                  <color>yellow</color>
                  <taste>energizing</taste>
               </fruit>
            </xml>
            
   // output
   var jsonFruit = []
   
   // pseudo file
   var tempFile = plugins.file.createTempFile('xmlFile', '.xml');
   plugins.file.writeTXTFile(tempFile, xml);
   
   // initialize XMLStreamReader
   var factory = XMLInputFactory.newInstance()
   var reader   = factory.createXMLStreamReader('testFile', new FileInputStream(tempFile.getAbsolutePath()))
   
   // process xml and transform to json object
   var fruitItem = new Object
   while ( reader.hasNext() ) {   
      reader.next();
      try {
         if ( reader.isStartElement() ) {
            var element = reader.getLocalName()
            switch (element) {
               case "title":
                     fruitItem["title"] = reader.getElementText()
                     break;
               case "color":
                     fruitItem["color"] = reader.getElementText()
                     break;
               case "taste":
                     fruitItem["taste"] = reader.getElementText()
                     break;
            }
         }
         else if ( reader.isEndElement() && reader.getLocalName() == "fruit" ) {
            jsonFruit.push(fruitItem)
            fruitItem = new Object
         }
      } catch (e) {
         application.output(e.message)
      }
   }
}


I'm not sure but doing Java like this in Servoy...probably need to explicitly close stuff down when you're done. A point for the Java gurus to weigh in on.
David Workman, Kabootit

Image
Everything you need to build great apps with Servoy
User avatar
david
 
Posts: 1727
Joined: Thu Apr 24, 2003 4:18 pm
Location: Washington, D.C.

Re: How to import data from an XML file?

Postby ptalbot » Wed Jun 19, 2013 6:33 pm

Yes, as-is your code will leave the stream opened, and lock the file, so you want to add a finally clause with a reader.close();
Patrick Talbot
Freelance - Open Source - Servoy Valued Professional
https://www.servoyforge.net
Velocity rules! If you don't use it, you don't know what you're missing!
User avatar
ptalbot
 
Posts: 1654
Joined: Wed Mar 11, 2009 5:13 am
Location: Montreal, QC

Re: How to import data from an XML file?

Postby siavash.roozbehan » Tue Jun 25, 2013 1:10 am

Thanks David, I have temporarily solved the issue by braking down the xml files into smaller size within the code, so when I read them in the memory they are not going to be big. But I'll give your solution a try soon.
Siavash

TSM (The Service Manager)
Australia
siavash.roozbehan
 
Posts: 97
Joined: Wed Apr 11, 2012 2:21 am

Previous

Return to How To

Who is online

Users browsing this forum: No registered users and 8 guests

cron