plugins.VelocityReport.fromJSON fails on UTF-8 file

[Bernd.N]

We created an interface where we get a JSON file from SAP SuccessFactors EmployeeCentral when employee data got changed there.

The first lines of such a JSON look like this:

Code: Select all

{
  "S136" : {
    "e_all_in_contract": null,
    "e_cost_center": "10 3800",
...


We read the JSON into an object variable with

Code: Select all

oEC_EIM_persons = plugins.VelocityReport.fromJSON( plugins.file.readTXTFile( jsFile.getAbsolutePath() ));

Everything works fine when the JSON is ANSI coded.
However when I switch the JSON to UTF-8 coded, Servoy prompts this strange error:
JAVASCRIPT ERROR
net.stuff.plugin.velocityreport.org.json.JSONException: Expected a ':' after a key at 3 [character 4 line 1]

Does someone know why?
I also tried to give plugins.file.readTXTFile() the second argument "UTF-8" as codepage, but that did not change anything.

[Bernd.N]

I found the reason here:
https://en.wikipedia.org/wiki/UTF-8#Byte_order_mark

There are UTF-8 files with and without a BOM (Byte Order Mark) at the beginning.
That BOM is the sequence EF BB BF.
When saving my JSON file as "UTF-8 without BOM" (can be done easily with Notepad++ in menu 'Coding / Convert to UTF-8 without BOM'), everything works well.

So to be able to process also "UTF-8 with BOM", VelocityReport.fromJSON() could check if the BOM is there, and then ignore or strip it before processing.
As a workaround, one has to make sure to process always UTF-8 without BOM.

I created a ticket on Servoyforge.

[Bernd.N]

To make everything work fine in the end, it is also important to add the charset parameter on the call of plugins.file.readTXTFile() :

oEC_EIM_persons = plugins.VelocityReport.fromJSON( plugins.file.readTXTFile( jsFile.getAbsolutePath(), "UTF-8" ));

Same is true when using plugins.file.writeTXTFile() in this context.