restrictions of getAsText() function of JSDataSet?

Classic Servoy
21

HI marc, wonderfull! :-)

could you explain this a little: ? (this is new to me… ;-)

 $cols = $cols.filter(function(item){ return !(/password/.test(item)); });
         $cols = $cols.filter(function(item){ return !(/k_owner_id/.test(item)); });
         $cols = $cols.filter(function(item)   { return $table.getColumn(item).getLength() < 100000; });
         $cols.sort();
22 
$cols = $cols.filter(function(item){ return !(/password/.test(item)); });

means:

  • the $cols array gets filtered based on the boolean which is returned by the function (true = passed).
  • the function receives each position from the array as a parameter (I called it ‘item’)
    If you would output ‘item’ you’d get the value for each position in the array.
  • /password/.test(item) is short notation for a reg ex. This case I look for the string ‘password’ in every item (=columnname)
    /password/.test(item) will return true for the column ‘password_exchange’, ‘master_password’, ‘login_password_user’, etc, etc.
    so: !(/password/.test(item)) will return false when the columnname contains password, true if it doesn’t.

All the rules above result in a array filter function which leaves out all columnnames containing ‘password’

Does this clarify all code you quoted?

23

ahh, thanks, yes! much appreciated! :-)

24

The best way to do this is through a plugin that uses foundsets to dump…
But also the above pure javascript version can be improved by using foundset instead of getDataSetByQuery()
Because if the result of that dataset by query is many many rows then you still are getting it all into the client.

So something like:

var foundset = databaseManager.getFoundset(server,table)
foundset.find()
foundset.owner_id=xxx
foundset.search()

and then do the loop by creating the row data through the foundset (by using the column dataproviders)

And if you want to be really fast, run that code in a HeadlessClient through the HeadlessClient plugin.
And dump it in a file that can be accesses by an url.
Or use another plugin that can stream stuff to the client.

25

jcompagner:
So something like:

var foundset = databaseManager.getFoundset(server,table)
foundset.find()
foundset.owner_id=xxx
foundset.search()

and then do the loop by creating the row data through the foundset (by using the column dataproviders)

Johan, I have a question about this.
Why is this better? because if you loop over the foundset, at the end, you have also all data in memory right? (at least the whole foundset!)

26

no you only have all the PK values in memory
Not all the rows. We only cache a certain amount (around the selection and around the last ask row)

27

oke wait, so if i have a foundset of 100.000 records and I loop through all them(by using the foundset) and ask for every record the value of every column, that is not the same in memory as loading a whole dataset?

28

Harjo:
oke wait, so if i have a foundset of 100.000 records and I loop through all them(by using the foundset) and ask for every record the value of every column, that is not the same in memory as loading a whole dataset?

don’t forget the code I posted does an append to the file.
So you don’t have to store all data in 1 var first…

Just have 1 var that you overwrite every iteration and append the content to the file on each iteration as well.

29

in your code, you ARE storing ALL data in 1 variable:

$dataset = databaseManager.getDataSetByQuery($server, $query, [‘’], -1);

right?

30

Harjo:
oke wait, so if i have a foundset of 100.000 records and I loop through all them(by using the foundset) and ask for every record the value of every column, that is not the same in memory as loading a whole dataset?

no it is not the same, a dataset will have really everything in memory
a foundset will not do that. (you could check the source now ;) )

31

jcompagner:
[(you could check the source now ;) )

yeah right! :lol: :lol:

32

Harjo:
in your code, you ARE storing ALL data in 1 variable:

$dataset = databaseManager.getDataSetByQuery($server, $query, [‘’], -1);

right?

correct, but that’s not wat I posted.
I replied to your question asking if looping through records was not the same as loading the whole dataset in memory.

I mentioned my code as this does the append trick and YES I know that I’m stil storing the dataset. (on purpose, 'cause sometimes not all my data is in just 1 foundset…)
So nothing for me to check. ;-) [EDIT] sorry this was about Servoy being OS… :oops:

But as we are on this subject: Johan is this what you had in mind for the iteration?

var $rec;
var $fs = forms.myForm.foundset;
for (var i = 1; i <= $fs.getSize(); i++)
	{
		$rec = $fs.getRecord(i)
                etc.
                etc.
	}
33

mboegem:
But as we are on this subject: Johan is this what you had in mind for the iteration?

var $rec;

var $fs = forms.myForm.foundset;
for (var i = 1; i <= $fs.getSize(); i++)
{
$rec = $fs.getRecord(i)
etc.
etc.
}

yes