databaseManager.recalcualte(record) in HC, calcs in module

[Robert Huber]

Around 500 MB is the lowest value it drops to. But it dosn't behave like it does with 2048 MB when I set memory to 1024 or 1536, and that's something I have no explanation. Why shouldn't the value drop to around 500 MB with the lower settings like it does with the 2048 setting?

does it really drop to 500 in the middle of doing stuff?
Then i guess it just needs memory to first being able to clear stuff up.
(i have seen that before that java takes more memory when it garbage collects and then dropping way below the memory it had before, it needs some working mem)

It really does drop to around 500 MB in the middle of calculating! It continually raises then to near 2 GB (the max set value) and drops again to 500 MB after a short while. And exactly this behaviour must be speeding up the whole calculation process by factors! When I set the max. value to 1036 MB, I can NOT see this process coming into action!? It must have something to do with how the current Java garbage collector is programmed. And, there is surely a reason they are going to introduce a new garbage collector called G1 GC for Java 7,isn't it?

Regards,

[jcompagner]

yes java 7 will get G1 (in java 6 you also have this already but is not enabled by default)

[jbader]

Hello, this post got a bit off topic, and I was hoping to hear from some community members about the original topic. Please see below...

...I assume that there are plenty of Servoy developers that recalcualte calculations in modules from within HC right? Can someone confirm that this works consistently for them? And by this I mean:

Calcualtions are in a module
HC includes the module
HC runs databaseManager.recalculate on records that have calculations that are in the module, and not in the HC solution itself.

This must be common enough. Who has this working from the community?

Thanks

[david]

If we see a databaseManager.recalculate() in code it always raises a red flag. If it affects anything more than UI display or light business rules we will redo that task so as to not need to do databaseManager.recalculate(). Update tracking fields at the transactional level, compute stuff in SQL at the point when needed, use of merge tables, run an update statement and flush the clients, etc. This way we know that long term we are avoiding issues with data integrity, performance and burying business logic that too many calculations eventually leads to.

Our general view is that there is nothing elegant about an application with tons of calcs based on calcs, calcs with relations in them, calcs with globals in them, calcs getting summed, calcs not failing gracefully, etc. Finding that one business rule that you need to change or debug is a pain, you can't be sure if your data is coming out the end of a long calculation chain is solid, and the solution will be comparatively slow.

We basically never use databaseManager.recalculate() here. It's a situational function in my opinion and some of the uses it's being applied to in this thread sound scary to me.

[jbader]

Hi David,

Thanks for the reply.

Calculated columns are common enough in models inside and outside of the Servoy ecosphere. But if the idea with Servoy calcs is don't use them and/or or don't count on them b/c they are too implicit and/or flaky, then that seems unreasonable to me (although I don't doubt everything you say is true!).

I don't want to do in the database, or in a controller, what can be done in the model layer. If I do, then what's the point? My assumption has always been that Hibernate is in use somewhere in Servoy (and I believe it has been said that it is), although I was surprised to see it didn't surface when the code went public. But irregardless, every major ORM supports calculated columns and they can be counted on and are used frequently in applications that implement those ORMs.

If calculations don't work I'd like to know what doesn't work. Up until now I have seen nothing in the documentation to suggest that they don't work, although in my own experience, only very trivial calculations work consistently.

Thanks David...I'm now pretty frightened by my implementation; relying on features that are supposed to work, but apparently do not.

[Robert Huber]

Hi David

If we see a databaseManager.recalculate() in code it always raises a red flag. If it affects anything more than UI display or light business rules we will redo that task so as to not need to do databaseManager.recalculate(). Update tracking fields at the transactional level, compute stuff in SQL at the point when needed, use of merge tables, run an update statement and flush the clients, etc. This way we know that long term we are avoiding issues with data integrity, performance and burying business logic that too many calculations eventually leads to.

I seems to me that raising a red flag just for the general existence of a calculation is a bit a unspecific statement. Or do you mean it's discussable if showing the multiplication of atribute a and attribute b as a calc is a sincere solution?
And, there are just more complex requirements that are asking for a solution. Anyone used to analyse requirements, design and model software including data, is aware that the various possible solutions at hand have to be weighted against each other - with it's advantages and disadvantages. There is often not right or wrong.
By the way, if we are at it, there is, in my opinion, still one of the best topic dealt with requirement analysis (Problem Frames) to be started with here
http://en.wikipedia.org/wiki/Problem_Frames_Approach
or
http://www.jacksonworkbench.co.uk/steve ... rames.html

I don't think your proposed "solutions" (compute in SQL, merge tables, ... are in principle better as other solutions - it would be nice to see standard problems solved with different patterns though - showing the advantages as well as disadvantages.

Our general view is that there is nothing elegant about an application with tons of calcs based on calcs, calcs with relations in them, calcs with globals in them, calcs getting summed, calcs not failing gracefully, etc. Finding that one business rule that you need to change or debug is a pain, you can't be sure if your data is coming out the end of a long calculation chain is solid, and the solution will be comparatively slow.

We basically never use databaseManager.recalculate() here. It's a situational function in my opinion and some of the uses it's being applied to in this thread sound scary to me.

I don't understand what exactly is your statement here? Tons of calcs is a unit I can't assign to software engineerig. I could get an idea if you would say a blinking warning sign should appear if for example calcs exceed the number of 10% of the given attributes in a entity relationship model, or any other measurement which is based on mathematics.
I agree that the question of how solid can we expect the result to be with a not simple calc chain. I really would like to hear from all of you! But the same question arises when the implementation is done "programmatically". Did I catch really all occuring cases, exceptions etc. Calcs can not easily solve that because there is (again) program code behind the scene to make them work.
And yes, we too (as lot's of developer colleagues) are very hard at work to find the most simple, most robust and most logical understandable solution to a given requirement - in other words the "best" one, sometimes with, sometimes without using calcs

Regards, Robert

[david]

Thanks David...I'm now pretty frightened by my implementation; relying on features that are supposed to work, but apparently do not.

In my view the problem with calculations is not that they don't work -- it's that they work too well. Consequently, they can easily be abused. The whole task that calculations make easier comes with ramifications that is impossible for the calculation engine to overcome. You can't have easy and no overhead with any platform -- Servoy or whatever.

Calculations are wonderful if used appropriate to the situation. What I wrote above is our highly-developed-algorithm-based-on-many-solutions-plus-some-voodoo-magic approach to where we the draw the line when it comes to calculations. The simple version is that if databaseManager.recalculate() has to be used in a method to get things to work right for anything but a UI calc, then we've probably crossed the line into calc hell (also a very precise measurement).

We recently completed an almost entirely transactional based app (with very few calcs) and the speed is astounding. Overkill to use this approach on a LAN only app but if you're looking to get speeds in smart client to be as fast as a terminal session over 2,000 miles of wire then you have to realize the limitations of calculations and design accordingly. And these kinds of design decisions are part of any programming environment.

[jbader]

I don't agree that they work "too well," in fact they frequently do not work for me at all, in my use scenario, which is why I posted this in the first place. I guess the problem with Servoy's features not behaving for me could very well just be me, but if that were true I would expect at least one person in the community to raise their hand and say, yeah we do what you do and that works great for us, but that doesn't seem to happen.

I'm less concerned about the performance of the calcs as we use a messaging system (that we built in Servoy,and that runs in HC) to handle record inserts and subsequent calculations. We defer everything we do to the server. Our clients only create records in queues, which servers then pick up and process further. This has been the only way we can make a WAN solution work with the amount of data we create, considering the speed we need our workflows to function. But our calcs frequently don't fire!

[jcompagner]

we are talking here about stored calcs.. unstored cacls shouldn't have any problems and are way faster and easier to use

databaseManager.recalculate() shouldnt be used that often on stored calcs, That call only should be needed when data is changing from outside of servoy (this is logical ofcourse)
Or your calcs are nested quite deep and are depending on data from other tables that are changed even when the record where the calc is on is not in memory at all.
These kind of constructs should be avoided if possible.. Because yes then you can't expect that all data is consistent at all times according the the calc..
for example if you have orders -> order_lines
and in orders you have a calc that is the order total so it just loops over the order_lines and adds the end price of every line into a total price.

But if you then also have a situation that you really only show the order_lines for example product_to_order_lines and through that relation you change the end price of an order line
then the total price of orders is out of sync. And yes you need to use recalculate on that one, because if orders where not loaded at all then servoy didn't have any clue.

The good thing is that if you dont rely on the stored calc with queries or aggregates it doesn't matter, because then it is just an UI thing and at the moment a user will display that order it will be updated.

These kind of situations are very difficult to handle right in what ever ORM library you use that has stored calc support.

[jbader]

But if you then also have a situation that you really only show the order_lines for example product_to_order_lines and through that relation you change the end price of an order line
then the total price of orders is out of sync. And yes you need to use recalculate on that one, because if orders where not loaded at all then servoy didn't have any clue.

What is the preferred way to do this in HC? Is it to use recalculate(record) on the order table's record? My experience is that this does not work. The only thing that does seem to work is "looking" at the record.

Recalculate returns void by the way so I have no idea what it did or didn't do.

[jcompagner]

yes recalculate should be called with an orders record (that has the calc) or with a foundset of the orders table (so that all records in that foundset are recalculated)

there is nothing to return, because there is no state of "it didnt recaclulate the calcs"
it should always just recalculate all the calcs of that record.
what it does it it flags all calcs for recalculation, and that asks the for the value so that it is recalculated.

What do you see? when you call databaseManager.recalculate(record) are the calcs really not called? If you output some debug code in the calc do you really see that some are not called?

[ryanparrish]

Johan,

In my and Jeff's case it does seem as if the calculations are executed after calling .recalculate(), but the data never gets written back to the database when the recalculate() is executed within a HC batch job. We have a block of code that runs in HC and takes care of recalculating everything that is needed and after which calls an application.output() on the calculations so we can verify the data in the logs. The logs shows the correct recalculated values but when we look at the database we can see it has not been written. I've tried doing the recalcs in a transaction, out of a transaction, locking records, not locking records, autoSave(true|false), and nearly every combination of those with no different results.

I have my db (MySQL 5.0.14) logging deadlocks to make sure that is not preventing the data from being written, and those are not coming up in this case (I know the logging works as I have used it to diagnose other issues). The only thing that really seems to trigger the data to be recalculated and written to the DB is to visit a form that has the values on it, then they get recalculated.

Just throwing out a random idea...Some of these calcs can take more than two seconds to execute within smart client over WAN in some cases, could there be some kind of timing issue because it's excuted so much faster when done on the server? Like is recalculate() non-blocking and the saveData() gets called before the recalculate has truly finished?

[jcompagner]

which version of servoy are you using?

If the calc is done then an insert statement should happen, but there where some problems which i think jeff did encounter because jeff has i think default value column in his tables
and there where problems with that especially in combination with a transaction.
Maybe it is related somehow.

but if you dont use a transaction then the call recalculate(record) before it is finished the insert statement should already happened.
If you use database transactions then it also should have happened (you can see that in the performance log) but the actual result is of course only really in the db after you commit.

If you use in memory transactions (saveData(false)) then the insert is only done when you say saveData(true) (and then a whole bunch if you did it for more records)

Just monitor also the insert statements in the performance log

[jbader]

In our solutions nothing is done outside of a transaction. We always set autosave to false when we start a transaction and before we commit, then set it to true again when /after saving. Also all of our database tables have database managed auto enter column defaults.

So where does that leave us? Do you test stored calcs under these conditions?

[Robert Huber]

Hi Jeff

I don't agree that they work "too well," in fact they frequently do not work for me at all, in my use scenario, which is why I posted this in the first place. I guess the problem with Servoy's features not behaving for me could very well just be me, but if that were true I would expect at least one person in the community to raise their hand and say, yeah we do what you do and that works great for us, but that doesn't seem to happen.

Neither do I, how can something in informatic work too well??? Did anyone you ever see that? You are always the first and lonely one

But our calcs frequently don't fire!

So do ours, but nobody believes it because it is not handily reproducable in a complex solution where you often just can't make a sample in a useful time (paid by who?).

Best regards,