Chinese characters in OS X storing to db as ???

Classic Servoy
21

It was fine when I checked www.google.co.jp at work I’m sure, plus I use it all the time. Whats weird is I just noticed on my home computer thats its not ok. I use Japanese Windows XP at work and at home I’m using English Windows XP. I am using the new plugin here and work. I use the one at work with Shift-JIS all the time and its ok. I know Japanese Windows XP encodes files as Shift-JIS so maybe thats why it works there. Or maybe an international/different Java runtime? Humm interesting.

Ill check and investigate again tomorrow at work.

Just curious what computer OS are you on?

Steve

22

Please see the below link. This is my test server.

http://fckeditor.testpilotweb.com/index … _data_id=8

Here I am outputting the Chinese data as UTF-8.
If you view this data in the browser the Chinese is displayed in the browser properly. But the text is corrupted when using http plugin.

Please help!!!

Thanks

23

But the text is corrupted when using http plugin.

This probably means that the display of that encoded data fails - not the way it is read from the server.
It is interesting what you discovered, Steve. So there is either something with the JVM language support, or with the regional settings of the OS.

24

Just tested www.google.co.jp at work and the result is different from English and Japanese Windows XP. Everything but < is ok on Japanese Windows XP. It’s changing < to this &#60 . I was just pulling text out so I didn’t really notice before.

A selection from the source page (www.google.co.jp) from Firefox for comparison:

書籍

From Servoy:

書籍&#60/a>

English Windows XP using Servoy scrambles the Japanese Characters when I tested www.google.co.jp.

As for the Java Runtime, on a Japanese Windows XP machine I have to go to the Japanese site and download it from there. The version reported in Servoy says this: Java version 10.0-b19 (Windows XP)

Steve

25

If I copy/pasted text from this link to servoy then the chinese is fine inside the servoy field.

http://fckeditor.testpilotweb.com/index … _data_id=8

So I guess it is something with how http plugin reads from the server.

26

I have the same issue when I post something (utf8 stuff) to the server using http plugin.

characters are converted to ??? (question marks)

27

OK. I still think it is an issue of how the fields interpret the encoded data (taking into consideration what Steve discovered), not an issue of the get itself. Anyway this is still an issue to be solved. As for the post part, I didn’t check.

Please add this to the support system. The issue should contain:

  • a small sample solution that demonstrates both get and post - we need this to see how exactly you are using data from get / how you are posting data and what data you are posting
  • an exact description of what you are doing

Thank you.

28

The issues are corrected in Servoy 4.0 rc2 and Servoy 3.5.7.

POST:
The default encoding used by the poster did not support Japanese/Chinese characters (and probably others as well).
So now you can specify the encoding you want to use for posting - using poster.setEncoding(…). If you do not specify an encoding it will use the new default one (UTF-8).

GET PAGE DATA:
Added a plugins.http.getLastPageEncoding() function that will allow you to find out what encoding the page had when received (although the data is already decoded, you might need the initial encoding in some cases - see sample).
There was also a problem with getting page data without using a http client - the data was not decoded correctly in some cases. This is fixed too.

29

Here is a link to the new http.jar - make a backup copy of the old jar (from your 3.5 install) and replace it with this one.
http://www.servoy.com/downloads/downloads/bean_plugin/http.jar
Tell me if you still experience problems. :wink:

30

Renamed the functions from poster.setEncoding(…) to poster.setCharset(…) and from plugins.http.getLastPageEncoding() to plugins.http.getLastPageCharset() in order to avoid some confusion that might appear - as encoding is not only used for charsets in http headers. The old functions only appear in Servoy 4.0 rc2, and are deprecated in rc3.

The new ones will appear in Servoy 4.0 rc3 and Servoy 3.5.7.

31

Servoy v 4.1.0 - build 651
MySQL 5.1.39
OS X 10.6.1

I have an issue which is similar to the initial problem in this post

I have characters which are storing to a MySQL backend as ‘?’ (question marks)

The characters being input are high ascii characters entered via the keyboard using ‘Alt-z’, ‘Alt-x’, ‘Alt-v’, ‘Alt-b’, ‘Alt-<’ or ‘Alt->’ combinations and are those characters which are 2 byte unicode chars (forgive me if I am not explaining this clearly - not big on fonts or char sets) and which are written to the backend as ‘?’

So I found this post and it gave very good advice on what to do to resolve this

I ensured that the DB was set to UTF8 & UTF8_unicode_ci

I followed the advice to the point where I added this string to my server connection url: ‘?useUnicode=yes&characterEncoding=UTF-8’ as suggested by Steve

This was accepted and I quit and restarted and tried again to enter data

The characters can be entered into the field but on commit Servoy shows me an error dialogue with ‘Cannot save form data’

Console tells me that the error is:
'Cannot save form data

com.servoy.j2db.dataprocessing.DataException: Incorrect string value: ‘\xCE\xA9…’ for column ‘item_description’ at row 1’

If I enter any non-2 byte characters into the field then it accepts and writes them without question

So I wonder whether anyone out there can offer me any further guidance on this problem

Thanks in advance
Harry

p.s. I followed the same advice and actions as above using Servoy version 3.5.10 and it worked in that version !