Page 1 of 1

Batchprocessor - increase in CPU usage

PostPosted: Fri Nov 25, 2011 9:16 am
by mihaela
Hello,

I am using Servoy 5.2.10 - build 1021. It seems that when the batchprocessor is running a lot more cpu is used, even after the cron jobs are finished (we can see in the logs that they are finished).
Do you know why this can happen?

The OS information is:
os.name=Linux
os.version=2.6.38-bpo.2-amd64
os.arch=i386

JVM information:
java.vm.name=Java HotSpot(TM) Server VM
java.version=1.6.0_26
java.vm.info=mixed mode

Thank you.

Re: Batchprocessor - increase in CPU usage

PostPosted: Mon Nov 28, 2011 12:49 pm
by jcompagner
when you see high cpu usage when you don't expect it you could through the admin pages dump a few stacktraces (every 5 seconds or so)
zip 5 of them and attach it here

Re: Batchprocessor - increase in CPU usage

PostPosted: Wed Nov 30, 2011 12:47 pm
by mihaela
thank you for your answer.
can you please explain how i can do that?

Re: Batchprocessor - increase in CPU usage

PostPosted: Wed Nov 30, 2011 3:43 pm
by jcompagner
just go to the admin page and press "Dump the current stack/systeminfo" every 5 or so seconds?
and copy past this into a file..

Re: Batchprocessor - increase in CPU usage

PostPosted: Wed Mar 14, 2012 7:41 pm
by mihaela
Is there a way to limit the memory that the batchprocessor can use? Like the servoy.maxClientHeap property is for the smart client.

Re: Batchprocessor - increase in CPU usage

PostPosted: Wed Mar 14, 2012 8:09 pm
by ROCLASI
Why would you want to limit it ? It will only use what it needs.
So if you want to control the memory usage you need to optimize your code.
And if you limit it and it's running a method that runs out of memory the method will just die. No error no nothing (at least that's how it was a couple of versions ago).

Actually that last bit only counts for Smart Client. The batchprocessor runs inside the Server memory space.

Re: Batchprocessor - increase in CPU usage

PostPosted: Wed Mar 14, 2012 8:36 pm
by mihaela
Thanks for your reply Robert.

I want to limit it because it seems there is no way to release it. I saw in developer that the heap memory bounces between certain values, never exceeding the max heap value. Looks to me like the GC comes and cleans, and then the memory is re-used.
In BP it only goes up. No matter how much memory it needs, it never gets released.
Or is there a way to release it?

Re: Batchprocessor - increase in CPU usage

PostPosted: Wed Mar 14, 2012 9:09 pm
by ROCLASI
Hi Mihaela,

I guess the question is what are you doing in your code. Perhap you are leaving references that the the GC can't clean up.
Or otherwise there might be an issue in Servoy's code. But in either case limiting the client heap will not solve this issue, it will only add a new one.

Re: Batchprocessor - increase in CPU usage

PostPosted: Wed Mar 14, 2012 9:29 pm
by mihaela
Hi Robert,

I ran the same method that runs in the BP: in developer: http://screencast.com/t/4npnQ3Yp
Since I did nothing in developer after running the method, looks to me that after a while, the memory is released..
I was wonderring if something like this can be triggered in the BP?

Re: Batchprocessor - increase in CPU usage

PostPosted: Wed Mar 14, 2012 9:44 pm
by ROCLASI
Hi Mihaela,

Well the JVM should take of that for you like it did in Developer. At what interval is this method run in the batchprocess?
Perhaps the method is triggered again before the previous one ended?

Re: Batchprocessor - increase in CPU usage

PostPosted: Wed Mar 14, 2012 10:19 pm
by mihaela
Hi Robert,
There are 4 methods planned to run. Each one runs once per day, at a separate hour in the night. They are all finished long before the next run, in the next night. But the memory that the BP uses will remain at similar values from day to day. Looks cached.. It feels like the GC is not triggered at all in the BP..

Re: Batchprocessor - increase in CPU usage

PostPosted: Wed Mar 14, 2012 11:41 pm
by ptalbot
So what do you do in your methods then?
Do you use plugins or inline Java, is there a memory leak in there? Do you keep references to files, big chunk of data as globals?
What is it that make the GB seeing your objects as referenced and not garbage collectible?

Re: Batchprocessor - increase in CPU usage

PostPosted: Fri Mar 16, 2012 7:28 pm
by mihaela
Hi Patrick,
In the methods I load a foundset, I change fields values, i save and do a recalculation. All this I log using the Log plugin.
Can you please tell me when is the GC triggered?

Re: Batchprocessor - increase in CPU usage

PostPosted: Fri Mar 16, 2012 7:35 pm
by ptalbot
The GC is triggered whenever it's needed and when it can :)
What is this Log plugin? Maybe there's a memory leak in there?

Re: Batchprocessor - increase in CPU usage

PostPosted: Tue Mar 20, 2012 9:57 am
by jcompagner
mihaela wrote:Hi Robert,

I ran the same method that runs in the BP: in developer: http://screencast.com/t/4npnQ3Yp
Since I did nothing in developer after running the method, looks to me that after a while, the memory is released..
I was wonderring if something like this can be triggered in the BP?


i see the line of the memory jump completely down (yes after you point this is the end but that is of no relation to when java gc does its job)

Its not then when a method/function ends that all resources are released right away, the gc is only doing some incremental quick stuff
When you look further you see that it hits the top line (i guess that 250MB) and then it really does a full gc and releases pretty much everything.

But looking at these memory usage tables doesn't say much about memory leaks, then you really have to know when also full gc's happens and over time you slowly see even with full gc's the min used memory creeping up.


Setting that client property makes no sense, it runs in the server, so it uses what you give the server. Is that really 250MB? thats a bit low for a server.. And thats why the cpu does a lot of stuff when i guess your batch jobs run
because it is really doing a log of Garbage Collecting because it is so tied in memory..