plugins.VelocityReport.fromJSON fails on UTF-8 file

Classic Servoy
1

We created an interface where we get a JSON file from SAP SuccessFactors EmployeeCentral when employee data got changed there.

The first lines of such a JSON look like this:

{
  "S136" : {
    "e_all_in_contract": null,
    "e_cost_center": "10 3800",
...

We read the JSON into an object variable with

oEC_EIM_persons = plugins.VelocityReport.fromJSON( plugins.file.readTXTFile( jsFile.getAbsolutePath() ));

Everything works fine when the JSON is ANSI coded.
However when I switch the JSON to UTF-8 coded, Servoy prompts this strange error:
JAVASCRIPT ERROR
net.stuff.plugin.velocityreport.org.json.JSONException: Expected a ‘:’ after a key at 3 [character 4 line 1]

Does someone know why?
I also tried to give plugins.file.readTXTFile() the second argument “UTF-8” as codepage, but that did not change anything.

2

I found the reason here:

There are UTF-8 files with and without a BOM (Byte Order Mark) at the beginning.
That BOM is the sequence EF BB BF.
When saving my JSON file as “UTF-8 without BOM” (can be done easily with Notepad++ in menu ‘Coding / Convert to UTF-8 without BOM’), everything works well.

So to be able to process also “UTF-8 with BOM”, VelocityReport.fromJSON() could check if the BOM is there, and then ignore or strip it before processing.
As a workaround, one has to make sure to process always UTF-8 without BOM.

I created a ticket on Servoyforge.

3

To make everything work fine in the end, it is also important to add the charset parameter on the call of plugins.file.readTXTFile() :

oEC_EIM_persons = plugins.VelocityReport.fromJSON( plugins.file.readTXTFile( jsFile.getAbsolutePath(), “UTF-8” ));

Same is true when using plugins.file.writeTXTFile() in this context.