databaseManager.recalcualte(record) in HC, calcs in module

[jcompagner]

yes recalculate should be called with an orders record (that has the calc) or with a foundset of the orders table (so that all records in that foundset are recalculated)

there is nothing to return, because there is no state of “it didnt recaclulate the calcs”
it should always just recalculate all the calcs of that record.
what it does it it flags all calcs for recalculation, and that asks the for the value so that it is recalculated.

What do you see? when you call databaseManager.recalculate(record) are the calcs really not called? If you output some debug code in the calc do you really see that some are not called?

[ryanparrish]

Johan,

In my and Jeff’s case it does seem as if the calculations are executed after calling .recalculate(), but the data never gets written back to the database when the recalculate() is executed within a HC batch job. We have a block of code that runs in HC and takes care of recalculating everything that is needed and after which calls an application.output() on the calculations so we can verify the data in the logs. The logs shows the correct recalculated values but when we look at the database we can see it has not been written. I’ve tried doing the recalcs in a transaction, out of a transaction, locking records, not locking records, autoSave(true|false), and nearly every combination of those with no different results.

I have my db (MySQL 5.0.14) logging deadlocks to make sure that is not preventing the data from being written, and those are not coming up in this case (I know the logging works as I have used it to diagnose other issues). The only thing that really seems to trigger the data to be recalculated and written to the DB is to visit a form that has the values on it, then they get recalculated.

Just throwing out a random idea…Some of these calcs can take more than two seconds to execute within smart client over WAN in some cases, could there be some kind of timing issue because it’s excuted so much faster when done on the server? Like is recalculate() non-blocking and the saveData() gets called before the recalculate has truly finished?

[jcompagner]

which version of servoy are you using?

If the calc is done then an insert statement should happen, but there where some problems which i think jeff did encounter because jeff has i think default value column in his tables
and there where problems with that especially in combination with a transaction.
Maybe it is related somehow.

but if you dont use a transaction then the call recalculate(record) before it is finished the insert statement should already happened.
If you use database transactions then it also should have happened (you can see that in the performance log) but the actual result is of course only really in the db after you commit.

If you use in memory transactions (saveData(false)) then the insert is only done when you say saveData(true) (and then a whole bunch if you did it for more records)

Just monitor also the insert statements in the performance log

[jbader]

In our solutions nothing is done outside of a transaction. We always set autosave to false when we start a transaction and before we commit, then set it to true again when /after saving. Also all of our database tables have database managed auto enter column defaults.

So where does that leave us? Do you test stored calcs under these conditions?

[Robert_Huber]

Hi Jeff

jbader:
I don’t agree that they work “too well,” in fact they frequently do not work for me at all, in my use scenario, which is why I posted this in the first place. I guess the problem with Servoy’s features not behaving for me could very well just be me, but if that were true I would expect at least one person in the community to raise their hand and say, yeah we do what you do and that works great for us, but that doesn’t seem to happen.

Neither do I, how can something in informatic work too well??? Did anyone you ever see that? You are always the first and lonely one

jbader:
But our calcs frequently don’t fire!

So do ours, but nobody believes it because it is not handily reproducable in a complex solution where you often just can’t make a sample in a useful time (paid by who?).

Best regards,

[Robert_Huber]

jcompagner:
we are talking here about stored calcs… unstored cacls shouldn’t have any problems and are way faster and easier to use

I’m not sooo sure about that.

jcompagner:
The good thing is that if you dont rely on the stored calc with queries or aggregates it doesn’t matter, because then it is just an UI thing and at the moment a user will display that order it will be updated.

You just forget that there are in some applications quite a lot of reports and statistics, and then it’s NOT a HI thing, but a serious problem!

Regards,

[jbader]

So do ours, but nobody believes it because it is not handily reproducable in a complex solution where you often just can’t make a sample in a useful time (paid by who?).

Actually over the years this have become a major sore point for me. I fee like if I can’t reproduce an issue in a simple, concise and convenient sample solution then I don’t get support. Sample solutions are a major time sink, and seeing as this is a commercial RAD tool, with the “R” of course standing for “Rapid” then if I spend all my time building sample solutions that attempt to reproduce issues that no one acknowledges otherwise, the tool loses value quickly. But this of course is a topic for another post.

Johan, you certainly know more than anyone about under what circumstances calcs do and do not work/fire/run updates etc then anyone else. I have provided my use case, and I really need some help understanding where the problem is. [please]

[Robert_Huber]

jbader:
Actually over the years this have become a major sore point for me. I fee like if I can’t reproduce an issue in a simple, concise and convenient sample solution then I don’t get support. Sample solutions are a major time sink, and seeing as this is a commercial RAD tool, with the “R” of course standing for “Rapid” then if I spend all my time building sample solutions that attempt to reproduce issues that no one acknowledges otherwise, the tool loses value quickly. But this of course is a topic for another post.

Jeff, you express what we have found (over the years) as well.

Regards,

[pbakker]

Hi Jeff (and others),

Rereading this entire thread, which got hijacked somewhere along the way, I read that you would create a simple sample. Did you ever do this? Also, which version of Servoy are you using (asked by Johan, no answer yet). This is important to us.

As for getting support/the time sink creating sample solutions: we do our best trying to reproduce reported issues, but as there can be many factors that play a role, we cannot always reproduce the reported behavior. Therefor, unfortunately, we rely on the person reporting the issue to make a reproducable case.

As for the red flag: up to a certain point I agree with David: when I see a databaseManager.recalculate(…) in code, I wonder what is wrong, because it shouldn’t be necessary to perform recalculations, except for stored calculations and then only in the following scenario: if you require stored values in the database to be the correct value for external access (reporting tools for example) or when you extract their values from the database through databaseManager.getDatasetByQuery(…).

Why? Because in all other scenario’s, as soon as Servoy loads a record, the calcs of the record will recalculate themselves and from the moment onwards that they will keep themselves up to date until the record gets unloaded. When a calc gets fired, it “records” which data is accessed in the calculation up to the point where a value is returned in the calc code. From that moment onwards, the calc “listens” to changes made to the data the previous calculation was dependant on.

So, as long as you are just within a Servoy Client and you don’t show the values of stored calculations using a getDatasetByQuery(…), there is no reason to call databaseManager.recalculate().

The databaseManager.recalculate(…) function is basically just a convenient function to loop over an entire foundset (containing more records that initially cached in the FoundSet), touching each record, which triggers all the stored calcs which update themselves in the database if required.

If you’re in the situation where external insert/updates/deletes occur on data used by Servoy, you shouldn’t call recalculate, but call refreshRecordFromDatabase() (or plugins.rawSQL.flushAllClientsCache(…) depending on how far you want to take it). Off course it would be even better to get that external process to notify Servoy of the dataChange, but that is another story.

Paul

[jcompagner]

Robert Huber:

jcompagner:
The good thing is that if you dont rely on the stored calc with queries or aggregates it doesn’t matter, because then it is just an UI thing and at the moment a user will display that order it will be updated.

You just forget that there are in some applications quite a lot of reports and statistics, and then it’s NOT a HI thing, but a serious problem!

i didn’t forget that, i did mention that " if you dont rely on the stored calc with queries or aggregates"

so yes if you do that then you need to have calcs up to date.

[jcompagner]

jbader:
Johan, you certainly know more than anyone about under what circumstances calcs do and do not work/fire/run updates etc then anyone else. I have provided my use case, and I really need some help understanding where the problem is. [please]

jeff, you really need and i do stress this, you need to go to 5.2.4
Because you use db transactions and columns with default values, there where problems with that in 5.2.2 and lower when columns with default values didn’t update related foundsets (what you also reported in another thread)
that was fixed in 5.2.3 but then default values and db transactions but surfaced this is fixed in 5.2.4.
So in your case you really need to be on 5.2.4

These 2 things don’t really have to do anything specific with calcs. Stored calcs are just column values that are changed and updated to the database when coming to storing it to the database (there is no difference in that way)

Then calcs just should work, besides the cases that paul and i described in this thread.
databaseManager.recalculate() should really always update the stored column values of the records that are walked over. I don’t know any way in 5.2.4 why that would not work for you.

[Robert_Huber]

I am glad you noticed it, just didn’t interpret your line correctly

[ryanparrish]

Johan and Paul,

Jeff and I are using version 5.2.2 -build 1002, we tried 5.2.3 when it came out but all the regressions meant we had to roll back to 5.2.2 and subsequently hold off on 5.2.4 until we have the time to really test it well.

To answer both Paul and Johan, yes the reason we do use recalculate() is because of exactly both of those three situations (external reports, .getDatasetByQuery(), aggregates). We utilize jasper reports (and soon jasper server) quite heavily here and rely on queries embedded within the report, and their sub-reports. Also, we never insert or update data outside the servoy data model, we went down that trap filled road in the previous version of our software and vowed to keep everything ‘pure’ servoy in this version. Also keep in mind that I previously said that our solution does have an application.output() right after the recalc in HC and the data that is logged is correct, it just doesn’t ever get written to the database when it is done within a HC batch job.

I think the main questions Jeff really wants to know is this…
*Does there exist unit tests at Servoy for things like verifying stored calcs work the same in smart client and HC?
*Are there unit tests to make sure stored calcs based on aggregates work the same in smart client and HC?
*Are there unit tests to make sure stored calcs work the same when called within a transaction, not within a transaction, with .autoSave() set to true or false (making the assumption that the user knows what they are doing and is calling things in the sequence and syntax).
*Are there unit tests to make sure auto enter DP’s that are defined by the database execute correctly inside and outside a transaction?
*And are these unit tests run with each release?

We totally understand the nature of trying to debug someone else’s code via snippets on a forum and the reasoning behind the request for sample solutions, we are programmers also here. But as everyone else here I’m sure can attest, bugs and feature requests don’t stop coming from users during the weeks that you need to spend to build out a near duplicate of your entire code base to demonstrate the specific behavior deep within your solution that is being seen. So we would really like to know if the servoy methods and features are always tested to verify correct functionality prior to release, as a sanity check on our own side. We are fine if everything is broken because we are messing up, we just need to know for sure that the tools we rely on are performing as we expect them to with every release (that auto enter within a txn bug seemed like a major oversight to us, as a programer that didn’t come from the 4GL world I can’t imagine doing things any other way).

[jcompagner]

the code for calcs is exactly the same for smart or headless client
the calculation engine, row manager,foundsets thats all the same code there is no difference in that
Client changes are in the UI itself and the interaction with the UI.

But calling databaseManager.recalculate(foundsetOrRecord) touches the exact same code for any client that we have in Servoy.

then if the stored calc is recalculated (so if that is triggered) then there is no change what so ever if that where a normal column… Then it is just a column change that will get saved to the database.

We have all kinds of test, but there are many many combinations. And yes some will be slipped through. For example some of the test can only be tested with 2 clients and then it is not really a unit test anymore (its to big)
All our unit test are run after any check in.

But you guys really must go to 5.2.4, you use a combination of “settings” that only work correctly in 5.2.4

[ryanparrish]

jcompagner:
the code for calcs is exactly the same for smart or headless client
the calculation engine, row manager,foundsets thats all the same code there is no difference in that
Client changes are in the UI itself and the interaction with the UI.

But calling databaseManager.recalculate(foundsetOrRecord) touches the exact same code for any client that we have in Servoy.

That I know, I’ve looked at the code quite a bit now that it’s open. But one programer to another we both know we often use the answer “things are exactly the same, they should work the same” only to find something that we didn’t think of in the way it’s called or combinations of the object state that really isn’t the same that is causing the apparent issue.

jcompagner:
But you guys really must go to 5.2.4, you use a combination of “settings” that only work correctly in 5.2.4

If that’s the case then it is something we are going to work towards now, and hope that it solves our long standing issue.

[jbader]

But you guys really must go to 5.2.4, you use a combination of “settings” that only work correctly in 5.2.4

Johan,

Can you be specific? What combination of settings is this exactly and why is it that if something is broken in 5.2.2 that is not documented anywhere. Banging away on a forum post for a month to find out there is indeed some unknown problem with calculations, that I would have never otherwise discovered without asking is pretty frustrating.

Please tell me what is broken in 5.2.2.

Thanks a lot and happy new year

[jcompagner]

this is for you broken in 5.2.2:

viewtopic.php?f=5&t=14943&p=81747#p79693

that is because you use db default column values, then it could be that when you update a column X on a record then you don’t see that change in your related foundset.
The databroadcast was not working in this situation. So this could also result in you not seeing calc updates when you think the data is changed… (in another client)

This is fixed in 5.2.4

This fix doesn’t directly have anything to do with calcs (that could just be a side effect)

[jbader]

But what does broadcasting have to do with our calcs not being written to the database? Do you see some connection that I am missing?

You think that by updating to 5.2.4 that our calcs will be written to the DB and our databroadcasting issues will go away?

[jcompagner]

i dont know exactly what goes wrong for you, but you use default column values and transactions, both that had some changes in the 2 releases that where done after the release you use.

again if the calc code is called and you return a different value then before, then it has nothing to do with being a calc. then it is exactly the same as you just did record.value = x
because the stored calc’s value is exactly the the same as a normal record value.
So if you are sure about that the calc is called, then we just have to look why the insert statement doesnt happen (for in mem transactions when you call saveData(true) and directly when you only use db transactions)

[jbader]

if you are sure about that the calc is called, then we just have to look why the insert statement doesnt happen (for in mem transactions when you call saveData(true) and directly when you only use db transactions)

Meaning you/your team are going to look into testing updates/inserts of records with stored calcs when the table has auto enter DB managed data, when in a transaction? Or meaning you need me to do something further?

Please advise!