Jenkins jsunit testing

Questions and answers on designing your Servoy solutions, database modelling and other 'how do I do this' that don't fit in any of the other categories

Jenkins jsunit testing

Postby jozef.kopanicak » Mon Jun 20, 2016 6:43 pm

Hello everyone,

I'm trying to setup a Jenkins Server to run our JSunit tests automatically as described on the wiki https://wiki.servoy.com/display/Serv7/Using+Jenkins+As+Continuous+Build+Software

Jenkins works wonderfully, but the unit tests do not run. I get the following exception when the ant build commes to the "run_smart_client_tests_if_exported_ok" part:

Code: Select all
run_smart_client_tests_if_exported_ok:
    [junit] Running com.servoy.automation.jsunit.smartclient.ServoyJSUnitTestRunner
    [junit] Testsuite: com.servoy.automation.jsunit.smartclient.ServoyJSUnitTestRunner
    [junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0 sec
    [junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0 sec
    [junit]
    [junit] Null Test:    Caused an ERROR
    [junit] null
    [junit] java.lang.reflect.InvocationTargetException
    [junit] Caused by: java.lang.IllegalStateException
    [junit]    at com.servoy.j2db.ClientState.<init>(ClientState.java:151)
    [junit]    at com.servoy.j2db.smart.J2DBClient.<init>(J2DBClient.java:733)
    [junit]    at com.servoy.j2db.smart.J2DBClient.<init>(J2DBClient.java:725)
    [junit]    at com.servoy.automation.Za.Za.<init>(Za.java:44)
    [junit]    at com.servoy.automation.Za.Zb.<init>(Zb.java:7)
    [junit]    at com.servoy.automation.Za.Zb.getInstance(Zb.java:20)
    [junit]    at com.servoy.automation.Za.Zb.access$000(Zb.java:14)
    [junit]    at com.servoy.automation.Za.Zf.Za(Zf.java:1)
    [junit]    at com.servoy.automation.Za.Zf.Zb(Zf.java:48)
    [junit]    at com.servoy.automation.jsunit.smartclient.Za.<init>(Za.java:80)
    [junit]    at com.servoy.automation.jsunit.smartclient.ServoyJSUnitTestRunner.suite(ServoyJSUnitTestRunner.java:10)
    [junit]
    [junit]
    [junit] Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
    [junit] Test com.servoy.automation.jsunit.smartclient.ServoyJSUnitTestRunner FAILED


Has anybody else experienced this?

I'm trying to find something in the logs, but no luck. The JSUnit tests run fine when run in the developer, but i cannot get them to work with the ServoyJSUnitTestRunner class.

The solution is exported without problems and it resides in the directory defined in smart.test.exports.dir in the export_and_test.properties file
Jozef Kopanicak
Mirus Software AG
jozef.kopanicak
 
Posts: 11
Joined: Wed Aug 29, 2012 10:17 am

Re: Jenkins jsunit testing

Postby Gordon McLean » Wed Jun 22, 2016 1:20 pm

Hi Jozef

I don't have an answer for your question sorry, but can I ask some related basic questions:

Your aiming to use Jenkins to unit test, what are the advantages of this over running the tests within developer ?

Do you test every single function or only key functions within your application ?

Possibly related to the question above do your tests check a specific function or do you write one test that in turn tests a number of other functions in one process ?

Sorry for the rookie level questions, I suspect many like me have a vague idea about testing but in reality are not actively doing it. Its good to see Jenkins being actively attempted within the community.

Gordon
Gordon McLean
Clickdigital.com
Gordon McLean
 
Posts: 253
Joined: Wed Aug 03, 2005 12:24 pm
Location: UK

Re: Jenkins jsunit testing

Postby paronne » Wed Jun 22, 2016 1:39 pm

Hi Josef,

have you tried to start the very same developer with the same workspace used by Jenkins to run the unit test ? what happens then ?
Also can you check if is trying to run the test on the correct solution ? do you have a log of the solution name used to run the tests ?
paronne
 
Posts: 202
Joined: Fri Nov 02, 2012 3:21 pm

Re: Jenkins jsunit testing

Postby jozef.kopanicak » Wed Jun 22, 2016 3:00 pm

@paronne

When I run the tests manually in the same developer jenkins uses, the test run ok and are all green. I also see in the log, that the same solution is used. As I said, up until (and including) the export of the solutin, everything works. The developer is also a fresh install as the wiki suggests. As far as I can tell, I've configured everything according to the wiki. I've checked multiple times. I even tried creating a very simple solution with just 1 function and 1 test to rule out that the problem is caused because of the solution I want to unittest.

There's a file with the name workspace_exporter_app_server_log.txt is empty, so I'm guesing, that the application server isn't even started. I've checked all files modified after the ant job run to see if there's any logfile, that might help me find the error, but I found nothing except the exception

@Gordon

I don't know if I understand your question correctly. Jenkins runs the same unit tests as you would in the developer. In fact, the jenkins job is configured to start a developer (without the GUI) that in succession should run the jsunit tests in the developer. It just does it automatically, so that after every commited change, these tests are run automatically to ensure everything runs as it should. We also do not run jenkins instead of the developer tests, but in addition to.

We are in the process of adopting unittesting, so just a very small part of the application has tests, so my answers come from limited experience, but I've tried to research what would work best for us.

Do you test every single function or only key functions within your application ?


The goal is to test 100% of the application. That said, we will never reach this goal a we also do not strive to. It's not feasible to test everything. Try to test as much as you can without going to extremes.

We already have a big codebase, all of which was previously not unittested. And that's going to stay so. It would a be nearly impossible to write tests for every existing function, so we ignore them for now. We just try to write tests for new functions, but not every single one, just what we feel is needed (you have to start somewhere :). We will add tests to the existing functionality only if the code needs to be redesigned, refactored or completely rewritten.

do your tests check a specific function or do you write one test that in turn tests a number of other functions in one process ?


Every unittest only tests a specific function with specific inputs. So there are usually multiple tests for one function. We bundle these in separate scopes, so every unittested function has it's own scope. If a function is dependent on another function, we mock the values returned from the other function (i.e. replace the other function with a dummy function that gives a predefined result back) to isolate the function, that is being tested
Jozef Kopanicak
Mirus Software AG
jozef.kopanicak
 
Posts: 11
Joined: Wed Aug 29, 2012 10:17 am

Re: Jenkins jsunit testing

Postby Gordon McLean » Wed Jun 22, 2016 3:14 pm

Jozef
Thank you for your answers I am pleased to see you have a large code base that has not been previously unit tested, I have the same. I also agree with your view on the basis you have to start somewhere so from X point on is a good approach. I had heard of Jenkins but has not appreciated that it ran on commit, I will have to look hard at this as there is no doubt we should be including unit testing into the workflow as a matter of some urgency.
Gordon McLean
Clickdigital.com
Gordon McLean
 
Posts: 253
Joined: Wed Aug 03, 2005 12:24 pm
Location: UK

Re: Jenkins jsunit testing

Postby Andrei Costescu » Wed Jun 22, 2016 3:19 pm

What version of Servoy do you use there?
Did you try it with a simple tests solution first? (only one test method in it, no onOpen or other things)

Note: Jenkins will run unit tests in a modified smart-client that auto-imports the exported .servoy solution before running the tests. So it's not starting a developer behind the scenes. But it should work the same.

We just have to see what is going wrong.
Andrei Costescu
Servoy
Andrei Costescu
 
Posts: 1018
Joined: Tue Jun 26, 2007 3:14 pm

Re: Jenkins jsunit testing

Postby Andrei Costescu » Wed Jun 22, 2016 3:25 pm

Btw. for Servoy 8 we have (I hope) improved those scripts and their usage https://wiki.servoy.com/display/public/ ... ng+Jenkins
That also provides some Servoy 8 specific features for automated tests. Just in case someone working on 8 reads this thread.
Andrei Costescu
Servoy
Andrei Costescu
 
Posts: 1018
Joined: Tue Jun 26, 2007 3:14 pm

Re: Jenkins jsunit testing

Postby Andrei Costescu » Wed Jun 22, 2016 3:47 pm

From what I can tell the problem here is that the signing is different for some jars compared to others.
Did you custom-sign jars in the developer that is running those tests?

I think that the two jars with different signing are:
  • application_server/lib/j2db.jar
  • developer/plugins/com.servoy.eclipse.jsunit_8.0.2.3024.jar/j2db_test.jar
Andrei Costescu
Servoy
Andrei Costescu
 
Posts: 1018
Joined: Tue Jun 26, 2007 3:14 pm

Re: Jenkins jsunit testing

Postby jozef.kopanicak » Wed Jun 22, 2016 4:24 pm

Hello Andrei

I'm using 7.4.4 developer. It's a fresh install, I have not re-signed the jars.

I tried the simple solution (1 function, 1 unittest on this function, all in the same scope, no forms, no modules, no tables, no onOpen). Same results.

I also tried the newest https://github.com/Servoy/software_factory_build_scripts/tree/master/servoyAntScripts

I'm running it on Java 1.8.0_77 64-bit version, Windows 7. The tests run fine, when run from the developer in the "servoy.install.dir" that's used in the ant build

Edit: sorry, i was a little quick there. i did custom sign the application server jars, including the j2db.jar. I have now signed the com.servoy.eclipse.jsunit_7.4.4.2039.jar with the same certificate, but the result is the same
Jozef Kopanicak
Mirus Software AG
jozef.kopanicak
 
Posts: 11
Joined: Wed Aug 29, 2012 10:17 am

Re: Jenkins jsunit testing

Postby jozef.kopanicak » Wed Jun 22, 2016 4:50 pm

Hello Andrei

I finally have managed to run the tests. Thank you for your tip to look at the jar signing. I have signed the com.servoy.eclipse.jsunit_7.4.4.2039.jar, but what i really needed to do was to sign the *.jar files, that are in the com.servoy.eclipse.jsunit_7.4.4.2039.jar itself

I think it would be helpful, to include a custom jar signing section to the wiki https://wiki.servoy.com/display/public/DOCS/Servoy+Software+Factory+Example+using+Jenkins

Thank you for your help
Jozef Kopanicak
Mirus Software AG
jozef.kopanicak
 
Posts: 11
Joined: Wed Aug 29, 2012 10:17 am

Re: Jenkins jsunit testing

Postby Andrei Costescu » Wed Jun 22, 2016 7:55 pm

Great. You are welcome. A note about this could be added in the wiki indeed.
But this will not happen with 8.0 as that code has changed meannwhile.
Andrei Costescu
Servoy
Andrei Costescu
 
Posts: 1018
Joined: Tue Jun 26, 2007 3:14 pm

Re: Jenkins jsunit testing

Postby pbakker » Fri Jun 24, 2016 8:27 am

If you find the Servoy supplied configuration files for running Unit Tests through Jenkins not understandable or feel that the effort involved in setting it all up is too cumbersome, you might be interested in this:l https://github.com/TheOrangeDots/Servoy ... uildConfig

Currently doesn't support Mobile not NGClient, but comes with way better configuration options, more sensible defaults, code Coverage and Code Analysis integrated etc.

If you're interested, contact me directly.

Paul

Note that currently you're not allowed to modify not distribute any of the scripts Servoy provides, as they are published under a license that doesn't allow you to do so
pbakker
 
Posts: 2822
Joined: Wed Oct 01, 2003 8:12 pm
Location: Amsterdam, the Netherlands

Re: Jenkins jsunit testing

Postby Andrei Costescu » Fri Jun 24, 2016 10:57 am

You were allowed to modify and distribute but under the terms of AGPL 3. So there were some conditions (keep source public, keep same license, ...).
So Paul was right to not like the used license for those build scripts.

We changed it to Apache as there was no real need for AGPL 3 on those repos (they only contained Apache 3rd party software). So now all the annoying restrictions are gone.
Andrei Costescu
Servoy
Andrei Costescu
 
Posts: 1018
Joined: Tue Jun 26, 2007 3:14 pm


Return to Programming with Servoy

Who is online

Users browsing this forum: No registered users and 12 guests

cron