how to download file in Web Client without loading it in mem

[rossent]

Hi all,

I am looking for a way to provide file download options in the web client but the files could be quite big so the server should not load them in memory but "serve" them in chunks.
Basically what I am looking for is similar to using the plugins.file.writeFile('suggested_file_name.ext', <JSFile or server file path>) where the browser shows its standard fileSave dialog and does the saving of the local file but the file does not have to be loaded all in memory on the server side - instead I would like to provide only the path (or use a JSFile) of the file on the server.

Is something like this possible currently in Servoy and what is the best way of implementing it?

[jcompagner]

i am not 100% sure but i think if you just let them upload to the server
Then over a specific size it is cached to disk (FileUpload project)

Then if you use the streaming "streamFilesToServer" of the plugin that will not ask for the bytes through the getBytes() method
but will ask the getInputStream() method and then stream them in blocks to a file on disk..

[rossent]

I am not sure if I am getting your answer correctly. Let me rephrase my question:

How can I send a file from the server to the web client (so the user can save a local copy of the file) without having to load the whole file in memory on the server. For something smaller, we can use the following code in the web client:

Code: Select all

//read the file data in memory
var fileData = ...get the file data as byte[] from database or file;

//send the result back to the browser where the standard Save Download dialog will be displayed
plugins.file.writeFile('suggested_file_name.ext', fileData);


I would like to avoid the first part of loading the file in memory and instead use something like:

Code: Select all

//get the server file info - does not read/load the file in memory
var serverJSFile = plugins.file.convertToJSFile(<path to the file on the server>);

//send the result back to the browser where the standard Save Download dialog will be displayed without loading the file in memory on the server
plugins.file.writeFile('suggested_file_name.ext', serverJSFile); //ideally, this should read from the server file and stream the file data to the client in blocks/chunks which can be specified...


Does this make sense?

As a side note, everything which does reading and/or writing of binary data to/from files or database MEDIA columns should provide options for doing this in blocks/chunks. I am not sure how Servoy currently reads and writes MEDIA columns in the database - does it load the whole thing in memory?

[jcompagner]

ah ok its the other way so from server to client.

That doesn't work currently with the file plugin because it expects bytes
We could add support that you can say stream this (temp) file to the user.

But streaming colunm, so database content, is another matter, currently if you touch a blob column that column will be completely loaded, we currently have no way of streaming that from the database to a client (and then stream that through to a file on the clients hd)
For servoy itself this would result in quite a big change, because you have to have an open result set and connection to the database from the client until it is streamed, and you need some handle to it that represents this output stream..
I guess something like this could be made by a plugin that has some method like getOutputStream(server,table,pk,column) and that output stream needs to be given then to an enhanced file plugin.

[rossent]

ah ok its the other way so from server to client.

That doesn't work currently with the file plugin because it expects bytes
We could add support that you can say stream this (temp) file to the user.

But streaming colunm, so database content, is another matter, currently if you touch a blob column that column will be completely loaded, we currently have no way of streaming that from the database to a client (and then stream that through to a file on the clients hd)
For servoy itself this would result in quite a big change, because you have to have an open result set and connection to the database from the client until it is streamed, and you need some handle to it that represents this output stream..
I guess something like this could be made by a plugin that has some method like getOutputStream(server,table,pk,column) and that output stream needs to be given then to an enhanced file plugin.

Streaming from a file on the server to the client (both Smart and Web) should definitely be in the File plugin. I guess the same applies for "streaming" data from the client to a file on the server.

I agree with you that for streaming data to/from the database in a media column a separate plugin (or perhaps new methods in the current File plugin) will work best. In the case of streaming a file from the web client all the way to the database will involve two "hops" - the first one to stream the data in chunks from the client to a temp file on the app server and then streaming the temp file from the app server to the database.

I understand that this type of implementation is a bit more complex but when it comes to dealing with files, I believe it is the only proper way of handling them since we have no control on how big files the end user will be pushing around.

Let me know if I need to create a feature request for this.

[patrick]

The UserManager plugin can stream directly from and to a database column...

[rossent]

I created a feature request for this: https://support.servoy.com/browse/SVY-1104

Since many new applications require interacting with larger files/data, adding streaming support all the way through (smart client, web client, database media fields) will be a very useful feature to have and will provide competitive advantage of the Servoy platform.

[rossent]

Are there any updates on this? Do you have plans if/when this functionality will be added to Servoy?

[jcompagner]

as far as i know this can already be done.

in the callback of showFileOpenDialog where you get an array of JSFile objects.
if you give those to streamToServer() then it will really not load the whole thing in memory.

We use FileUpload library and big files (there is a certain threshold) are first cached by a DiskItemFile so its stored in a temp file.

Then if you give it to the streamToServer() the input stream is taken from that file and that is send to the "server" (which is for the webclient of course the same thing/vm)
And that goes in chunks. The file is never completely in memory.

Streaming it directly to a column in the db, then i guess the user manager plugin can be used by passing in the same JSFile (if the usermanager plugin uses the right api and calls getInputStream() on it)

[rossent]

as far as i know this can already be done.

in the callback of showFileOpenDialog where you get an array of JSFile objects.
if you give those to streamToServer() then it will really not load the whole thing in memory.

We use FileUpload library and big files (there is a certain threshold) are first cached by a DiskItemFile so its stored in a temp file.

Then if you give it to the streamToServer() the input stream is taken from that file and that is send to the "server" (which is for the webclient of course the same thing/vm)
And that goes in chunks. The file is never completely in memory.

Streaming it directly to a column in the db, then i guess the user manager plugin can be used by passing in the same JSFile (if the usermanager plugin uses the right api and calls getInputStream() on it)

For uploading to the server that will be OK, but what about for the other direction? (when the client needs to download from the server a big file) The streamFilesFromServer really can't be used for Web Client, can it?

[jcompagner]

no we don't have support for that.
But i think for this the quickest way (and also the most configurable way) would be to do this in the web_client_utils plugin, that one already has a JSON or XML behavior, they could add a BinairyBehavior quite easy in the current setup
you just have to supply for example the filename thats on the system, give it maybe already the mimetype (so that doesn't have to be checked by the plugin) give it options how it should open it in the browser (content disposition)
and you get an url back to that file..

[Harjo]

HI Johan,

we have the same issue, we have to stream/download files of 100MB and larger, from server (file-system) to servoy-webclient. We do not want them to be loaded into memory. (serverside)
If I read your answer carefully, you say the webclient utils plugin must be changed for this?

Or is this already possible?

[jcompagner]

no the fastest way is to have support build in in the webclient utils.

[ptalbot]

You could use Velocity put a form with a vr_getContext(request) in it, define a route for it in config.json and return like this:
plugins.Velocity.createFileResponse(file, [mimeType], [deleteAfterServe]);