Batchprocessor - increase in CPU usage

The forum to discuss the Headless version of Servoy. Web, Java and Servlet development questions can all be posted here.

Batchprocessor - increase in CPU usage

Postby mihaela » Fri Nov 25, 2011 9:16 am

Hello,

I am using Servoy 5.2.10 - build 1021. It seems that when the batchprocessor is running a lot more cpu is used, even after the cron jobs are finished (we can see in the logs that they are finished).
Do you know why this can happen?

The OS information is:
os.name=Linux
os.version=2.6.38-bpo.2-amd64
os.arch=i386

JVM information:
java.vm.name=Java HotSpot(TM) Server VM
java.version=1.6.0_26
java.vm.info=mixed mode

Thank you.
Mihaela Rotaru
Stb Software Development
SAN Partner

Stb Software Development - http://www.stb.nl
User avatar
mihaela
 
Posts: 14
Joined: Mon Feb 21, 2011 11:14 am

Re: Batchprocessor - increase in CPU usage

Postby jcompagner » Mon Nov 28, 2011 12:49 pm

when you see high cpu usage when you don't expect it you could through the admin pages dump a few stacktraces (every 5 seconds or so)
zip 5 of them and attach it here
Johan Compagner
Servoy
User avatar
jcompagner
 
Posts: 8829
Joined: Tue May 27, 2003 7:26 pm
Location: The Internet

Re: Batchprocessor - increase in CPU usage

Postby mihaela » Wed Nov 30, 2011 12:47 pm

thank you for your answer.
can you please explain how i can do that?
Mihaela Rotaru
Stb Software Development
SAN Partner

Stb Software Development - http://www.stb.nl
User avatar
mihaela
 
Posts: 14
Joined: Mon Feb 21, 2011 11:14 am

Re: Batchprocessor - increase in CPU usage

Postby jcompagner » Wed Nov 30, 2011 3:43 pm

just go to the admin page and press "Dump the current stack/systeminfo" every 5 or so seconds?
and copy past this into a file..
Johan Compagner
Servoy
User avatar
jcompagner
 
Posts: 8829
Joined: Tue May 27, 2003 7:26 pm
Location: The Internet

Re: Batchprocessor - increase in CPU usage

Postby mihaela » Wed Mar 14, 2012 7:41 pm

Is there a way to limit the memory that the batchprocessor can use? Like the servoy.maxClientHeap property is for the smart client.
Mihaela Rotaru
Stb Software Development
SAN Partner

Stb Software Development - http://www.stb.nl
User avatar
mihaela
 
Posts: 14
Joined: Mon Feb 21, 2011 11:14 am

Re: Batchprocessor - increase in CPU usage

Postby ROCLASI » Wed Mar 14, 2012 8:09 pm

Why would you want to limit it ? It will only use what it needs.
So if you want to control the memory usage you need to optimize your code.
And if you limit it and it's running a method that runs out of memory the method will just die. No error no nothing (at least that's how it was a couple of versions ago).

Actually that last bit only counts for Smart Client. The batchprocessor runs inside the Server memory space.
Robert Ivens
SAN Developer / Servoy Valued Professional / Servoy Certified Developer

ROCLASI Software Solutions / JBS Group, Partner
Mastodon: @roclasi
--
ServoyForge - Building Open Source Software.
PostgreSQL - The world's most advanced open source database.
User avatar
ROCLASI
Servoy Expert
 
Posts: 5438
Joined: Thu Oct 02, 2003 9:49 am
Location: Netherlands/Belgium

Re: Batchprocessor - increase in CPU usage

Postby mihaela » Wed Mar 14, 2012 8:36 pm

Thanks for your reply Robert.

I want to limit it because it seems there is no way to release it. I saw in developer that the heap memory bounces between certain values, never exceeding the max heap value. Looks to me like the GC comes and cleans, and then the memory is re-used.
In BP it only goes up. No matter how much memory it needs, it never gets released.
Or is there a way to release it?
Mihaela Rotaru
Stb Software Development
SAN Partner

Stb Software Development - http://www.stb.nl
User avatar
mihaela
 
Posts: 14
Joined: Mon Feb 21, 2011 11:14 am

Re: Batchprocessor - increase in CPU usage

Postby ROCLASI » Wed Mar 14, 2012 9:09 pm

Hi Mihaela,

I guess the question is what are you doing in your code. Perhap you are leaving references that the the GC can't clean up.
Or otherwise there might be an issue in Servoy's code. But in either case limiting the client heap will not solve this issue, it will only add a new one.
Robert Ivens
SAN Developer / Servoy Valued Professional / Servoy Certified Developer

ROCLASI Software Solutions / JBS Group, Partner
Mastodon: @roclasi
--
ServoyForge - Building Open Source Software.
PostgreSQL - The world's most advanced open source database.
User avatar
ROCLASI
Servoy Expert
 
Posts: 5438
Joined: Thu Oct 02, 2003 9:49 am
Location: Netherlands/Belgium

Re: Batchprocessor - increase in CPU usage

Postby mihaela » Wed Mar 14, 2012 9:29 pm

Hi Robert,

I ran the same method that runs in the BP: in developer: http://screencast.com/t/4npnQ3Yp
Since I did nothing in developer after running the method, looks to me that after a while, the memory is released..
I was wonderring if something like this can be triggered in the BP?
Mihaela Rotaru
Stb Software Development
SAN Partner

Stb Software Development - http://www.stb.nl
User avatar
mihaela
 
Posts: 14
Joined: Mon Feb 21, 2011 11:14 am

Re: Batchprocessor - increase in CPU usage

Postby ROCLASI » Wed Mar 14, 2012 9:44 pm

Hi Mihaela,

Well the JVM should take of that for you like it did in Developer. At what interval is this method run in the batchprocess?
Perhaps the method is triggered again before the previous one ended?
Robert Ivens
SAN Developer / Servoy Valued Professional / Servoy Certified Developer

ROCLASI Software Solutions / JBS Group, Partner
Mastodon: @roclasi
--
ServoyForge - Building Open Source Software.
PostgreSQL - The world's most advanced open source database.
User avatar
ROCLASI
Servoy Expert
 
Posts: 5438
Joined: Thu Oct 02, 2003 9:49 am
Location: Netherlands/Belgium

Re: Batchprocessor - increase in CPU usage

Postby mihaela » Wed Mar 14, 2012 10:19 pm

Hi Robert,
There are 4 methods planned to run. Each one runs once per day, at a separate hour in the night. They are all finished long before the next run, in the next night. But the memory that the BP uses will remain at similar values from day to day. Looks cached.. It feels like the GC is not triggered at all in the BP..
Mihaela Rotaru
Stb Software Development
SAN Partner

Stb Software Development - http://www.stb.nl
User avatar
mihaela
 
Posts: 14
Joined: Mon Feb 21, 2011 11:14 am

Re: Batchprocessor - increase in CPU usage

Postby ptalbot » Wed Mar 14, 2012 11:41 pm

So what do you do in your methods then?
Do you use plugins or inline Java, is there a memory leak in there? Do you keep references to files, big chunk of data as globals?
What is it that make the GB seeing your objects as referenced and not garbage collectible?
Patrick Talbot
Freelance - Open Source - Servoy Valued Professional
https://www.servoyforge.net
Velocity rules! If you don't use it, you don't know what you're missing!
User avatar
ptalbot
 
Posts: 1654
Joined: Wed Mar 11, 2009 5:13 am
Location: Montreal, QC

Re: Batchprocessor - increase in CPU usage

Postby mihaela » Fri Mar 16, 2012 7:28 pm

Hi Patrick,
In the methods I load a foundset, I change fields values, i save and do a recalculation. All this I log using the Log plugin.
Can you please tell me when is the GC triggered?
Mihaela Rotaru
Stb Software Development
SAN Partner

Stb Software Development - http://www.stb.nl
User avatar
mihaela
 
Posts: 14
Joined: Mon Feb 21, 2011 11:14 am

Re: Batchprocessor - increase in CPU usage

Postby ptalbot » Fri Mar 16, 2012 7:35 pm

The GC is triggered whenever it's needed and when it can :)
What is this Log plugin? Maybe there's a memory leak in there?
Patrick Talbot
Freelance - Open Source - Servoy Valued Professional
https://www.servoyforge.net
Velocity rules! If you don't use it, you don't know what you're missing!
User avatar
ptalbot
 
Posts: 1654
Joined: Wed Mar 11, 2009 5:13 am
Location: Montreal, QC

Re: Batchprocessor - increase in CPU usage

Postby jcompagner » Tue Mar 20, 2012 9:57 am

mihaela wrote:Hi Robert,

I ran the same method that runs in the BP: in developer: http://screencast.com/t/4npnQ3Yp
Since I did nothing in developer after running the method, looks to me that after a while, the memory is released..
I was wonderring if something like this can be triggered in the BP?


i see the line of the memory jump completely down (yes after you point this is the end but that is of no relation to when java gc does its job)

Its not then when a method/function ends that all resources are released right away, the gc is only doing some incremental quick stuff
When you look further you see that it hits the top line (i guess that 250MB) and then it really does a full gc and releases pretty much everything.

But looking at these memory usage tables doesn't say much about memory leaks, then you really have to know when also full gc's happens and over time you slowly see even with full gc's the min used memory creeping up.


Setting that client property makes no sense, it runs in the server, so it uses what you give the server. Is that really 250MB? thats a bit low for a server.. And thats why the cpu does a lot of stuff when i guess your batch jobs run
because it is really doing a log of Garbage Collecting because it is so tied in memory..
Johan Compagner
Servoy
User avatar
jcompagner
 
Posts: 8829
Joined: Tue May 27, 2003 7:26 pm
Location: The Internet


Return to Servoy Headless Client

Who is online

Users browsing this forum: No registered users and 5 guests