copyFile Size Limit?

[irennie]

Hi,
We have a solution that does some file handling. In some places in Smart Client, Web Client and Headless Client we use the plugins.file.copyFile command to copy files from one location to another. Normally files are managable size but recently we tried to copy a file that was 660MB and the copy failed with an OutOfMemoryError in the log:

Code: Select all

2011-02-02 13:05:57,543 ERROR [http-8080-5] com.servoy.j2db.util.Debug - Throwable
java.io.IOException: Map failed
   at sun.nio.ch.FileChannelImpl.map(Unknown Source)
   at sun.nio.ch.FileChannelImpl.transferFromFileChannel(Unknown Source)
   at sun.nio.ch.FileChannelImpl.transferFrom(Unknown Source)
   at com.servoy.extensions.plugins.file.FileProvider.js_copyFile(Unknown Source)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
   at java.lang.reflect.Method.invoke(Unknown Source)
   at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:179)
   at org.mozilla.javascript.NativeJavaMethod.call(NativeJavaMethod.java:353)
   at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:3666)
   at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:2680)
   at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:166)
   at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:387)
   at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3127)
   at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:165)
   at com.servoy.j2db.scripting.ScriptEngine.executeFunction(ScriptEngine.java:458)
   at com.servoy.j2db.debug.RemoteDebugScriptEngine.executeFunction(RemoteDebugScriptEngine.java:382)
   at com.servoy.j2db.FormController.executeFunction(FormController.java:3977)
   at com.servoy.j2db.FormController.executeFunction(FormController.java:3865)
   at com.servoy.j2db.FormController.executeFunction(FormController.java:3787)
   at com.servoy.j2db.FormController$ScriptExecuter.executeFunction(FormController.java:3642)
   at com.servoy.j2db.ui.BaseEventExecutor.fireEventCommand(BaseEventExecutor.java:272)
   at com.servoy.j2db.ui.BaseEventExecutor.fireActionCommand(BaseEventExecutor.java:217)
   at com.servoy.j2db.server.headlessclient.dataui.WebEventExecutor.onEvent(WebEventExecutor.java:362)
   at com.servoy.j2db.server.headlessclient.dataui.WebEventExecutor$2.onEvent(WebEventExecutor.java:166)
   at org.apache.wicket.ajax.AjaxEventBehavior.respond(AjaxEventBehavior.java:177)
   at org.apache.wicket.ajax.AbstractDefaultAjaxBehavior.onRequest(AbstractDefaultAjaxBehavior.java:300)
   at org.apache.wicket.request.target.component.listener.BehaviorRequestTarget.processEvents(BehaviorRequestTarget.java:157)
   at org.apache.wicket.request.AbstractRequestCycleProcessor.processEvents(AbstractRequestCycleProcessor.java:92)
   at org.apache.wicket.RequestCycle.processEventsAndRespond(RequestCycle.java:1250)
   at org.apache.wicket.RequestCycle.step(RequestCycle.java:1329)
   at org.apache.wicket.RequestCycle.steps(RequestCycle.java:1436)
   at org.apache.wicket.RequestCycle.request(RequestCycle.java:545)
   at org.apache.wicket.protocol.http.WicketFilter.doGet(WicketFilter.java:486)
   at com.servoy.j2db.server.servlets.Zl.doGet(Zl.java:11)
   at org.apache.wicket.protocol.http.WicketServlet.doGet(WicketServlet.java:138)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
   at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
   at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:433)
   at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:433)
   at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:433)
   at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:433)
   at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:433)
   at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
   at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
   at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:567)
   at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
   at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
   at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849)
   at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
   at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454)
   at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.OutOfMemoryError: Map failed
   at sun.nio.ch.FileChannelImpl.map0(Native Method)
   ... 57 more

Is there a limit to the file size that can be copied with the file plugin, and if so is there a way to copy larger files?

[lvostinar]

you don't have enough memory left to load that whole file in the memory; also the copyfile method is not very efficient, you can open a case in our system so that we don't load all file into memory but in chunks

[irennie]

For the case where I am copying files between locations on the server using the web client and headless client, would it be possible to use the plugins.file.streamFilesToServer method instead of the copyFiles method? Does this work on the web client and headless client to just copy files between server locations but without the file size memory problems, or do the same memory limitations apply for very large files?

[jcarlos]

you don't have enough memory left to load that whole file in the memory; also the copyfile method is not very efficient, you can open a case in our system so that we don't load all file into memory but in chunks

Yep! Files should be streamed by chunks to the server - meaning, they should be sent in packet of less than 64k at a time until the transfer is done (to avoid memory problems).

This is what we did in the SmartDoc plugin, which Patrick is presenting in ServoyWorld. He used RMI (Java Remote Method Invocation) in combination of byte[] buffer in a for loop with Threading.

See Patrick for details . 

JC

[lwjwillemsen]

For the case where I am copying files between locations on the server using the web client and headless client, would it be possible to use the plugins.file.streamFilesToServer method instead of the copyFiles method? Does this work on the web client and headless client to just copy files between server locations but without the file size memory problems, or do the same memory limitations apply for very large files?

An answer please to the question of irenny...
I would think streaming is always in chunks of data.

Regards,

[jcarlos]

You are right. I think that's what the file plugin does too. Try plugins.file.streamFilesToServer

JC