Streaming feedback with Servoy's file plugin

Classic Servoy
1

We’ve been hammering on the new streaming functionality of the file plugin past couple of weeks. The core streaming functionality works very well but the associated callback and monitor functionalities definitely have a few quirks that have sent us on all kinds of sleuthing journeys.

Some of it is pretty complicated so just wondering if I should list all the feedback here or in the interest of efficiency establish a direct connection. I’m all queued up with notes and code so either way works for me.

Ok, I’m done now with Servoy for 2010. See everyone next year :)

2

Hi David,

yes please, post your feedback. The monitoring mechanism is effectively not perfect nor will it ever be, because you have to setup a java Timer instance that repeatedly calls back Servoy.
The timer itself has his own specific Thread of course but it cannot be accurate (it tends to be more and more accurate the more file you have), and to callback Servoy it has to do so using an Executor which is queuing the callbacks, so there’s no way to guarantee any accuracy in terms of time. Which is also why you have to keep track of the list of files you have send and check the JProgressMonitor against it, and why you have a first callback which is also called whenever a file is done streaming. One thing is that each file will be processed in sequence inside one Thread, so the guarantee you have is that the order of files you send is going to be the order of files that is processed.
Don’t know if I’m clear enough here, but tell us about your difficulties and we might find the best way to do it…

Happy 2011!

3

thanks patrick! wasn’t 100% sure that you coded this part. very nice addition to servoy.

it turns out all the crazy issues i was having with the monitor and callback was user error: i was calling streamFilesFromServer() for each file in a directory instead of passing in an array of files to the function and calling it once. creating a situation where i had many streams going at the same time. not so good :)

one question i haven’t been able to figure out: can you stream directly to a variable? currently i’m creating temp files, streaming to those, then doing readTXTFile() on them.

other random notes so far:

  • if you pass a sub folder that doesn’t exist to getRemoteFolderContents(), it returns the name of the folder you pass in. should probably return either nothing or an error code representing sub folder does not exist.
  • it’s not readily clear that the default uploads folder (“/uploads/”) is in the ROOT directory from the info on the admin page.
  • setting a default folder on the admin page has to be a fully qualified path from the hard drive root. could be made more clear in the admin info.
  • could use a function to return the servoy install directory on the server. gives a way to programmatically ensure that you are always working within the servoy directory.
  • hidden directories are returned with plugins.file.streamFilesFromServer(option 2 for folders). not sure if this is a good or bad.

tip for people: when checking for file existence when streaming from the server (based on a path of type string), make sure to use the remote-convert-to-js file instead of the normal convert-to-js file. Since both functions return the same value when working in developer it’s worth noting:

plugins.file.convertToRemoteJSFile(directory + themesArray*.getName() + "/description.txt").exists()*
*```*
4

david:
thanks patrick! wasn’t 100% sure that you coded this part. very nice addition to servoy.

it turns out all the crazy issues i was having with the monitor and callback was user error: i was calling streamFilesFromServer() for each file in a directory instead of passing in an array of files to the function and calling it once. creating a situation where i had many streams going at the same time. not so good :)

Yes, I guess the monitoring works best if you are using an array of files.
That being said, it’s true that you can use more than one thread by issuing more than one streamFilesToServer or streamFilesFromServer - if you are not interested in the feedback that’s good to know.

david:
one question i haven’t been able to figure out: can you stream directly to a variable? currently i’m creating temp files, streaming to those, then doing readTXTFile() on them.

No you can’t.
The whole deal of streaming files to and from server is to be able to stream files that wouldn’t fit into memory (think video files for example), the idea being that only a small buffer is needed to transfer big files. So streaming to a variable makes no sense with that use case in mind.
Maybe readTXTFile and readFile could accept a JSFile of type RemoteFile and read directly into a variable? - right now they don’t.

david:

  • if you pass a sub folder that doesn’t exist to getRemoteFolderContents(), it returns the name of the folder you pass in. should probably return either nothing or an error code representing sub folder does not exist.

Yes, you are probably right. Maybe it should throw an Exception.

david:

  • it’s not readily clear that the default uploads folder (“/uploads/”) is in the ROOT directory from the info on the admin page.
  • setting a default folder on the admin page has to be a fully qualified path from the hard drive root. could be made more clear in the admin info.

Will see how I can make the info for the property clearer.

david:

  • could use a function to return the servoy install directory on the server. gives a way to programmatically ensure that you are always working within the servoy directory.

There is getDefaultUploadLocation() (since 5.2.4) that will give you the path used by the plugin as the root of all uploads (“/” in relative notation)
Returning the servoy install directory won’t give you much more, in any case you cannot set the default upload location programmatically (for security reasons).

david:

  • hidden directories are returned with plugins.file.streamFilesFromServer(option 2 for folders). not sure if this is a good or bad.

Not sure either. Right now the function doesn’t filter anything in the root upload location, it’s up to you whether you want to give access to your users or not.

david:
tip for people: when checking for file existence when streaming from the server (based on a path of type string), make sure to use the remote-convert-to-js file instead of the normal convert-to-js file. Since both functions return the same value when working in developer it’s worth noting:

plugins.file.convertToRemoteJSFile(directory + themesArray*.getName() + "/description.txt").exists()*
*```[/quote]*
*Yes, people need to understand that the JSFile that is returned by this function is really a different kind of JSFile than the one returned by convertToJSFile (which is why there is a different function).*
*Internally they are represented by a RemoteFile or a LocalFile class.*
5

ptalbot:

david:
one question i haven’t been able to figure out: can you stream directly to a variable? currently i’m creating temp files, streaming to those, then doing readTXTFile() on them.

No you can’t.
The whole deal of streaming files to and from server is to be able to stream files that wouldn’t fit into memory (think video files for example), the idea being that only a small buffer is needed to transfer big files. So streaming to a variable makes no sense with that use case in mind.
Maybe readTXTFile and readFile could accept a JSFile of type RemoteFile and read directly into a variable? - right now they don’t.

Figures my first time trying all this stuff out is to do something entirely different :) Using text files on the server to automate entering meta data on the client.