Chinese characters in OS X storing to db as ???

Classic Servoy
1

Hi there,

The chinese character I input in the servoy forms displays correct when I enter them. But when I checked the database the values are stored as ???. After I restarted the servoy the forms also display the chinese chars as ???

Servoy Developer
Version 3.5.1-build 514
Java version 1.5.0_13-121 (Mac OS X)

MySQL 5.0.41 (character set for the table is UTF-8 Unicode (utf8))

My windows machine does not have similar problem.

Please help!

Thanks

2

Sounds like a mysql driver bug to me. Does it occur with other databases eg iAnywhere?

3

i will test with other database and also will try to update the mysql driver to the latest one

4

Make fully sure that the database, table and fields are actually set to utf-8 in Mysql.

This is what I have set in the file my.ini and it works for Japanese:

[mysql]

default-character-set=utf8

[mysqld]
default-character-set=utf8

collation_server=utf8_unicode_ci
character_set_server=utf8

init-connect=‘SET NAMES utf8’

And check the mysql jdbc connection to your database in servoy.properties has something simiilar to this (can find more out about it doing a google search)

server.4.URL=jdbc:mysql://localhost/user_data?useUnicode=yes&characterEncoding=UTF-8

I’ve been using Macs/PCs with the Mysql/Servoy/utf8 setup for a long time now and if I had a problem with ??? or something else it’s always been a setting I missed somewhere.

If it still doesn’t work let me know.

Steve

5

Thanks a lot Steve,

I have just to changed my db connection URL to something like you specified.

server.4.URL=jdbc:mysql://localhost/user_data?useUnicode=yes&characterEncoding=UTF-8

Now servoy can store the Chinese chars to the db properly.

Thanks!

6

I have another similar problem. I am using http plugin to get the data from the web in servoy. When there are some chinese chars in the web table I get as ??? into the servoy.

Any ideas please??

Thanks a lot

7

We solved a similar problem with the Http plugin a few months ago… I will check to see what release contains that fix - may be after 3.5.1.

8

Can you please send me a link to download the plugin?
At the moment I am not going to update my servoy shortly.

Thanks

9

The issue should be corrected in Servoy 3.5.4.
The same problem - here is how the solved issue sounded like:

The plugin http.getPageData will not get text from a page encoded in Shift-JIS or other Asian languages. It gets the page but the Shift-JIS text is scrambled. I suppose it won’t do other complex encodings in other countries too.

10

The Servoy 3.5.5 version of the http plugin can be downloaded from:

http://www.servoy.com/downloads/downloa … n/http.jar

Put this in your Servoy plugins directory and restart Servoy.

11

Thanks a lot. I will try.

12

Yes, that was fixed, works for me now.

Steve

13

I still have the issue. Please see my code below:

var url = "http://www.google.com.hk/";


var frmObj = forms.broadcast$detail;
var pageData = plugins.http.getPageData(url);

frmObj.body = pageData;

The Chinese text comes as garbage characters. Any idea please?

14

I checked that site out using the plugin and it doesn’t look right to me either. I can’t find text between Servoy and what Firefox reports viewing source. I can with www.google.jp just checked.

Probably something with the plugin and Big5 charset is wrong. They fixed the Shift-JIS encoding last time.

They usually have it fixed pretty fast.

Steve

15

I will have a look at the content of that page - to see what is happening.

16

This is a very tricky page. For example when I accessed that page from IE it didn’t show me the Chinese version. When accessed the first time from IE it will give a page translated to your current locale. If you then press on the “show in Chinese” it will show Chinese.

If you do the same with safari for windows, you get directly the page in Chinese. Saving the Chinese looking page content from the browser also gives different results depending on the browser you are using and the way you reached that page (directly or through the “show in Chinese” link).

So I did the same thing in Servoy - got the page data and saved it to disc. Surprise - the file was exactly the same as when I saved it from Safari using “Download Linked File As” … So Servoy does get the correct page contents, but the google server is very sensitive to the information it can collect from your browser.

So what you get using Servoy is probably the default contents of that page if google cannot identify browser/location information - and it is encoded in Big 5.

If I open the file saved with Servoy it shows the correct Chinese content in the browser… So what do you need to do with the contents of that page? The content is correct, but I think the using part is more complicated due to the encoding of the data.

17

Here is my experience accessing http://www.google.com.hk/

Using IE 7 I get the page correctly directly accessed and any other way. The page says it’s encoded unicode (UTF8).

Firefox says exactly same thing unicode (UTF8) and displays fine version 2.0.0.13

When I view the source in Firefox the page says at the top “content=“text/html; charset=UTF-8”>”

When I view the source in Servoy it says charset=Big5. Could this be the problem? The Chinese characters are scrambled in the debugger and when trying to save to the database. They aren’t for other pages like www.google.jp.

Hope this helps.

Steve

18

Yes, in IE and FF you get the UTF-8 encoded page back - but this is because the google server decides what to give back (probably) based on some info it gets from the browser.

You get the Big 5 encoded page when asking for it from Servoy or when you choose “Download Linked File As” from Safari (make a small HTML page that has a href to that address, and right click on that) - so this is really what the google server returns (probably because it cannot get the browser/location information received from FF or IE).

So what I am saying is that it is really the content returned by the server - and it is stranger because it is encoded in Big5 - but I think it is correct because if you save it to disk and open it in a browser it shows the right way…

That is why I asked what this page is used for in Servoy - probably working with Big 5 encoded data is not that easy…

19

Ok thanks, I understand now. I agree Big5 is probably going to be a pain.

Steve

20

Hi Steve,

I tried the following code:

var url = "http://www.google.co.jp/";
var fckData = plugins.http.getPageData(url);

application.output(fckData)
forms.broadcast$detail.body = fckData;

I still received messed up japanese data. May be because the charset I received is Shift_JIS. I wonder how you can get japanese right? I can send you a screen shot if you need it.

Thanks
Hameed