UTF8 Character Encoding

[Daniel Woods]

Hi Everyone,

I realize that this issue has been brought up several times, but I
definitely need some help... I have tried everything that is
recommended in all the docs and lists and previous emails from this
list on the subject.

I have a mysql database that has been created with the charset=utf8. I
have ensured that /etc/my.cnf is properly configured with utf8 and
utf8_general_ci values on the "default-character-set" and
"default-collation" directives respectively. My MySQL JDBC URL string
contains the "useUnicode=true" and "characterEncoding=utf8"
parameters. My Config.groovy contains the "grails.views.gsp.encoding"
and "grails.converters.encoding" parameters with both set to "UTF-8".
I have confirmed that the data in the database is UTF-8 encoded data
and is thereby not dirty. I've done this by 1) performing a simple
command-line dump of a column containing UTF-8 encoded characters into
a file; and 2) attached the old Ruby on Rails app to the database (I
am converting the app from RoR to Grails), and it renders the UTF-8
characters to the browser just fine. I can definitely confirm that
Grails rendering the UTF-8 characters with Latin1 (ISO-8859-1)
encoding. The characters: æ are rendered instead of æ ... I have
tried setting the contentType and characterEncoding parameters on the
response object, both in the view and in the controller... Nothing
works. I've even taken a more Java-esque approach and tried to create
a properly encoded String object using the data I'm getting back from
the database. It is almost as though the MySQL JDBC library is not
properly respecting the useUnicode and characterEncoding parameters.

I don't know what else to try... What can I do here?

Thanks in advance.

Dan

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

[Enrique Medina Montenegro]

Are you starting up Tomcat within Grails, or deploying the Grails
application to a separated Tomcat?

If the latter, try setting "URIEncoding=UTF-8" attribute in Connector tag.

Post by Daniel Woods
Hi Everyone,
I realize that this issue has been brought up several times, but I
definitely need some help... I have tried everything that is
recommended in all the docs and lists and previous emails from this
list on the subject.
I have a mysql database that has been created with the charset=utf8. I
have ensured that /etc/my.cnf is properly configured with utf8 and
utf8_general_ci values on the "default-character-set" and
"default-collation" directives respectively. My MySQL JDBC URL string
contains the "useUnicode=true" and "characterEncoding=utf8"
parameters. My Config.groovy contains the "grails.views.gsp.encoding"
and "grails.converters.encoding" parameters with both set to "UTF-8".
I have confirmed that the data in the database is UTF-8 encoded data
and is thereby not dirty. I've done this by 1) performing a simple
command-line dump of a column containing UTF-8 encoded characters into
a file; and 2) attached the old Ruby on Rails app to the database (I
am converting the app from RoR to Grails), and it renders the UTF-8
characters to the browser just fine. I can definitely confirm that
Grails rendering the UTF-8 characters with Latin1 (ISO-8859-1)
encoding. The characters: Ê are rendered instead of Ê ... I have
tried setting the contentType and characterEncoding parameters on the
response object, both in the view and in the controller... Nothing
works. I've even taken a more Java-esque approach and tried to create
a properly encoded String object using the data I'm getting back from
the database. It is almost as though the MySQL JDBC library is not
properly respecting the useUnicode and characterEncoding parameters.
I don't know what else to try... What can I do here?
Thanks in advance.
Dan
---------------------------------------------------------------------
http://xircles.codehaus.org/manage_email

[Daniel Woods]

Post by Enrique Medina Montenegro
Are you starting up Tomcat within Grails, or deploying the Grails
application to a separated Tomcat?
If the latter, try setting "URIEncoding=UTF-8" attribute in Connector tag.

I've tried it with running Tomcat within Grails and as deployed to
Tomcat... Neither makes a difference. I'm actually using Jetty right
now, but this hasn't had an effect one way or another.

Post by Enrique Medina Montenegro

Post by Daniel Woods
Hi Everyone,
I realize that this issue has been brought up several times, but I
definitely need some help... I have tried everything that is
recommended in all the docs and lists and previous emails from this
list on the subject.
I have a mysql database that has been created with the charset=utf8. I
have ensured that /etc/my.cnf is properly configured with utf8 and
utf8_general_ci values on the "default-character-set" and
"default-collation" directives respectively. My MySQL JDBC URL string
contains the "useUnicode=true" and "characterEncoding=utf8"
parameters. My Config.groovy contains the "grails.views.gsp.encoding"
and "grails.converters.encoding" parameters with both set to "UTF-8".
I have confirmed that the data in the database is UTF-8 encoded data
and is thereby not dirty. I've done this by 1) performing a simple
command-line dump of a column containing UTF-8 encoded characters into
a file; and 2) attached the old Ruby on Rails app to the database (I
am converting the app from RoR to Grails), and it renders the UTF-8
characters to the browser just fine. I can definitely confirm that
Grails rendering the UTF-8 characters with Latin1 (ISO-8859-1)
encoding. The characters: æ are rendered instead of æ ... I have
tried setting the contentType and characterEncoding parameters on the
response object, both in the view and in the controller... Nothing
works. I've even taken a more Java-esque approach and tried to create
a properly encoded String object using the data I'm getting back from
the database. It is almost as though the MySQL JDBC library is not
properly respecting the useUnicode and characterEncoding parameters.
I don't know what else to try... What can I do here?
Thanks in advance.
Dan
---------------------------------------------------------------------
   http://xircles.codehaus.org/manage_email

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

[Wilson Akira Ikeda]

I think the problem is between the dump and the import of the data (or the data per si, there's something called signature that i don't know what it is), have you tried with a fresh app (no data import) to see if it can generate the proper characters? After you dumped the file have you checked it? eg: file --mime myfile.sql

Post by Daniel Woods

Post by Enrique Medina Montenegro
Are you starting up Tomcat within Grails, or deploying the Grails
application to a separated Tomcat?
If the latter, try setting "URIEncoding=UTF-8" attribute in Connector tag.

I've tried it with running Tomcat within Grails and as deployed to
Tomcat... Neither makes a difference. I'm actually using Jetty right
now, but this hasn't had an effect one way or another.

Post by Enrique Medina Montenegro

Post by Daniel Woods
Hi Everyone,
I realize that this issue has been brought up several times, but I
definitely need some help... I have tried everything that is
recommended in all the docs and lists and previous emails from this
list on the subject.
I have a mysql database that has been created with the charset=utf8. I
have ensured that /etc/my.cnf is properly configured with utf8 and
utf8_general_ci values on the "default-character-set" and
"default-collation" directives respectively. My MySQL JDBC URL string
contains the "useUnicode=true" and "characterEncoding=utf8"
parameters. My Config.groovy contains the "grails.views.gsp.encoding"
and "grails.converters.encoding" parameters with both set to "UTF-8".
I have confirmed that the data in the database is UTF-8 encoded data
and is thereby not dirty. I've done this by 1) performing a simple
command-line dump of a column containing UTF-8 encoded characters into
a file; and 2) attached the old Ruby on Rails app to the database (I
am converting the app from RoR to Grails), and it renders the UTF-8
characters to the browser just fine. I can definitely confirm that
Grails rendering the UTF-8 characters with Latin1 (ISO-8859-1)
encoding. The characters: æ are rendered instead of æ ... I have
tried setting the contentType and characterEncoding parameters on the
response object, both in the view and in the controller... Nothing
works. I've even taken a more Java-esque approach and tried to create
a properly encoded String object using the data I'm getting back from
the database. It is almost as though the MySQL JDBC library is not
properly respecting the useUnicode and characterEncoding parameters.
I don't know what else to try... What can I do here?
Thanks in advance.
Dan
---------------------------------------------------------------------
http://xircles.codehaus.org/manage_email

---------------------------------------------------------------------
http://xircles.codehaus.org/manage_email

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

[Daniel Woods]

Post by Wilson Akira Ikeda
I think the problem is between the dump and the import of the data (or the data per si, there's something called signature that i don't know what it is), have you tried with a fresh app (no data import) to see if it can generate the proper characters? After you dumped the file have you checked it? eg: file --mime myfile.sql

I'm not sure what "signature" is either as it pertains to character
encoding. I've not done any data import; I've simply mapped existing
database structures to new domain classes. The data, columns, tables,
and databases are all definitely UTF-8 encoded, and as I said before,
the existing Ruby on Rails app (which attaches to the same db
structure) displays the characters fine.

[***@angelinux ~]$ file --mime /tmp/foo
/tmp/foo: text/plain; charset=utf-8
[***@angelinux ~]$

As an interesting side note, when I use just straight Groovy SQL to
pull the data, it too produces a String with 8859_1 character
encoding.

Here's my datasource configuration for my grails app -- maybe
something worthwhile here?

dataSource {
pooled = true
driverClassName = "com.mysql.jdbc.Driver"
username = "xxx"
password = "xxx"
dbCreate = "update"
url = "jdbc:mysql://localhost/database?tcpKeepAlive=true&useUnicode=yes&characterEncoding=UTF-8"
autoReconnect = true
dialect = org.hibernate.dialect.MySQL5Dialect
properties {
maxActive = 50
maxIdle = 25
minIdle = 5
initialSize = 5
minEvictableIdleTimeMillis = 60000
timeBetweenEvictionRunsMillis = 60000
maxWait = 10000
validationQuery = "/* ping */"
}
}

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

[Wilson Akira Ikeda]

If you create a new app and use other db, it doesn't generate the proper characters?

Post by Daniel Woods

Post by Wilson Akira Ikeda
I think the problem is between the dump and the import of the data (or the data per si, there's something called signature that i don't know what it is), have you tried with a fresh app (no data import) to see if it can generate the proper characters? After you dumped the file have you checked it? eg: file --mime myfile.sql

I'm not sure what "signature" is either as it pertains to character
encoding. I've not done any data import; I've simply mapped existing
database structures to new domain classes. The data, columns, tables,
and databases are all definitely UTF-8 encoded, and as I said before,
the existing Ruby on Rails app (which attaches to the same db
structure) displays the characters fine.
/tmp/foo: text/plain; charset=utf-8
As an interesting side note, when I use just straight Groovy SQL to
pull the data, it too produces a String with 8859_1 character
encoding.
Here's my datasource configuration for my grails app -- maybe
something worthwhile here?
dataSource {
pooled = true
driverClassName = "com.mysql.jdbc.Driver"
username = "xxx"
password = "xxx"
dbCreate = "update"
url = "jdbc:mysql://localhost/database?tcpKeepAlive=true&useUnicode=yes&characterEncoding=UTF-8"
autoReconnect = true
dialect = org.hibernate.dialect.MySQL5Dialect
properties {
maxActive = 50
maxIdle = 25
minIdle = 5
initialSize = 5
minEvictableIdleTimeMillis = 60000
timeBetweenEvictionRunsMillis = 60000
maxWait = 10000
validationQuery = "/* ping */"
}
}
---------------------------------------------------------------------
http://xircles.codehaus.org/manage_email

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

[Daniel Woods]

Post by Wilson Akira Ikeda
If you create a new app and use other db, it doesn't generate the proper characters?

Same effect. I've also written a small Perl/Dancer app just to test
another language/framework, and it too displays the characters and
character encoding correctly. Groovy/Grails seems to be the only
technology that is having a problem with this :-(

/facepalm

..dan

Post by Wilson Akira Ikeda

Post by Daniel Woods

Post by Wilson Akira Ikeda
I think the problem is between the dump and the import of the data (or the data per si, there's something called signature that i don't know what it is), have you tried with a fresh app (no data import) to see if it can generate the proper characters? After you dumped the file have you checked it? eg: file --mime myfile.sql

I'm not sure what "signature" is either as it pertains to character
encoding. I've not done any data import; I've simply mapped existing
database structures to new domain classes. The data, columns, tables,
and databases are all definitely UTF-8 encoded, and as I said before,
the existing Ruby on Rails app (which attaches to the same db
structure) displays the characters fine.
/tmp/foo: text/plain; charset=utf-8
As an interesting side note, when I use just straight Groovy SQL to
pull the data, it too produces a String with 8859_1 character
encoding.
Here's my datasource configuration for my grails app -- maybe
something worthwhile here?
dataSource {
      pooled = true
      driverClassName = "com.mysql.jdbc.Driver"
      username = "xxx"
      password = "xxx"
      dbCreate = "update"
      url = "jdbc:mysql://localhost/database?tcpKeepAlive=true&useUnicode=yes&characterEncoding=UTF-8"
      autoReconnect = true
      dialect = org.hibernate.dialect.MySQL5Dialect
      properties {
              maxActive = 50
              maxIdle = 25
              minIdle = 5
              initialSize = 5
              minEvictableIdleTimeMillis = 60000
              timeBetweenEvictionRunsMillis = 60000
              maxWait = 10000
              validationQuery = "/* ping */"
      }
}
---------------------------------------------------------------------
   http://xircles.codehaus.org/manage_email

---------------------------------------------------------------------
   http://xircles.codehaus.org/manage_email

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

[Daniel Woods]

FWIW, what I did was go through and use groovy's
StreamingMarkupBuilder to convert all the abnormal characters to HTML
codes. This will suffice for now, but I'm still kind of annoyed that I
couldn't get it to work with properly encoded UTF-8 characters.

Oh well...

..dan

Post by Daniel Woods

Post by Wilson Akira Ikeda
If you create a new app and use other db, it doesn't generate the proper characters?

Same effect. I've also written a small Perl/Dancer app just to test
another language/framework, and it too displays the characters and
character encoding correctly. Groovy/Grails seems to be the only
technology that is having a problem with this :-(
/facepalm
..dan

Post by Wilson Akira Ikeda

Post by Daniel Woods

Post by Wilson Akira Ikeda
I think the problem is between the dump and the import of the data (or the data per si, there's something called signature that i don't know what it is), have you tried with a fresh app (no data import) to see if it can generate the proper characters? After you dumped the file have you checked it? eg: file --mime myfile.sql

I'm not sure what "signature" is either as it pertains to character
encoding. I've not done any data import; I've simply mapped existing
database structures to new domain classes. The data, columns, tables,
and databases are all definitely UTF-8 encoded, and as I said before,
the existing Ruby on Rails app (which attaches to the same db
structure) displays the characters fine.
/tmp/foo: text/plain; charset=utf-8
As an interesting side note, when I use just straight Groovy SQL to
pull the data, it too produces a String with 8859_1 character
encoding.
Here's my datasource configuration for my grails app -- maybe
something worthwhile here?
dataSource {
      pooled = true
      driverClassName = "com.mysql.jdbc.Driver"
      username = "xxx"
      password = "xxx"
      dbCreate = "update"
      url = "jdbc:mysql://localhost/database?tcpKeepAlive=true&useUnicode=yes&characterEncoding=UTF-8"
      autoReconnect = true
      dialect = org.hibernate.dialect.MySQL5Dialect
      properties {
              maxActive = 50
              maxIdle = 25
              minIdle = 5
              initialSize = 5
              minEvictableIdleTimeMillis = 60000
              timeBetweenEvictionRunsMillis = 60000
              maxWait = 10000
              validationQuery = "/* ping */"
      }
}
---------------------------------------------------------------------
   http://xircles.codehaus.org/manage_email

---------------------------------------------------------------------
   http://xircles.codehaus.org/manage_email

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

[Peter Ledbrook]

Post by Daniel Woods
Here's my datasource configuration for my grails app -- maybe
something worthwhile here?
dataSource {

...

Post by Daniel Woods
       dbCreate = "update"
       url = "jdbc:mysql://localhost/database?tcpKeepAlive=true&useUnicode=yes&characterEncoding=UTF-8"
       autoReconnect = true
       dialect = org.hibernate.dialect.MySQL5Dialect

...

One other option: restore the database to ensure all data is UTF-8,
then remove the 'dbCreate' and 'dialect' properties before running
your Grails application again.

Peter

[Al Baker]

How are you editing your GSPs and other files?

Eclipse, STS? Make sure you have UTF-8 file encodings of your GSPs and
message files and such too.

What OS are you on?

Post by Daniel Woods

Post by Daniel Woods
Here's my datasource configuration for my grails app -- maybe
something worthwhile here?
dataSource {

...

Post by Daniel Woods
dbCreate = "update"
url =

"jdbc:mysql://localhost/database?tcpKeepAlive=true&useUnicode=yes&characterEncoding=UTF-8"

Post by Daniel Woods
autoReconnect = true
dialect = org.hibernate.dialect.MySQL5Dialect

...
One other option: restore the database to ensure all data is UTF-8,
then remove the 'dbCreate' and 'dialect' properties before running
your Grails application again.
Peter
--
Peter Ledbrook
Grails Advocate
SpringSource - A Division of VMware
---------------------------------------------------------------------
http://xircles.codehaus.org/manage_email

[Daniel Woods]

Post by Al Baker
How are you editing your GSPs and other files?

STS.

Post by Al Baker
Eclipse, STS?  Make sure you have UTF-8 file encodings of your GSPs and
message files and such too.

STS; I'm not sure what you mean here... They unrendered GSPs have only
ASCII text in them and therefore the interpreted charset comes as
us-ascii. I'm unaware of a way to "force" UTF-8 encoding on a text
file. Please correct me if I'm wrong here.

For example:

[***@angelinux products]$ file --mime list.gsp
list.gsp: text/plain; charset=us-ascii
[***@angelinux products]$ convmv -f ascii -t UTF-8 --nosmart
--notest list.gsp
Ready!
[***@angelinux products]$ file --mime list.gsp
list.gsp: text/plain; charset=us-ascii

Post by Al Baker
What OS are you on?

Fedora 15

I have no idea what the problem is, but my workaround has satisfied
the client, and new data going into the db is being converted to HTML
codes, so for now I'm happy. I'm still curious about this though...
:-/

..dan

Post by Al Baker

Post by Daniel Woods
Here's my datasource configuration for my grails app -- maybe
something worthwhile here?
dataSource {

         ...

Post by Daniel Woods
       dbCreate = "update"
       url =
"jdbc:mysql://localhost/database?tcpKeepAlive=true&useUnicode=yes&characterEncoding=UTF-8"
       autoReconnect = true
       dialect = org.hibernate.dialect.MySQL5Dialect

         ...
One other option: restore the database to ensure all data is UTF-8,
then remove the 'dbCreate' and 'dialect' properties before running
your Grails application again.
Peter
--
Peter Ledbrook
Grails Advocate
SpringSource - A Division of VMware
---------------------------------------------------------------------
   http://xircles.codehaus.org/manage_email

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

[Al Baker]

Unicode is one of those things that I think we'd want to get to the bottom
of, even if it's a corner case. The internals of the JVM are UTF-16, and
most modern APIs, formats, and interfaces are UTF-8. It sounds like we've
covered all of the main APIs from the domain model down - the database, the
JDBC driver, etc.

Let's look at the APIs going up to the presentation tier - the GSP, any
servlet filters, tag libs, or other things that are having to write
information to an output stream.

Can you make sure that STS saves the GSPs in UTF-8?

Window->Preferences... and Opened General->Workspace and changed the file
encoding setting to UTF-8

Also in 'gedit' on Linux, the 'save as' dialog has encoding in it as well.
You could dump your database to a flat file, and see if it comes out as
UTF-8 using gedit.

Al

Post by Daniel Woods

Post by Al Baker
How are you editing your GSPs and other files?

STS.

Post by Al Baker
Eclipse, STS? Make sure you have UTF-8 file encodings of your GSPs and
message files and such too.

STS; I'm not sure what you mean here... They unrendered GSPs have only
ASCII text in them and therefore the interpreted charset comes as
us-ascii. I'm unaware of a way to "force" UTF-8 encoding on a text
file. Please correct me if I'm wrong here.
list.gsp: text/plain; charset=us-ascii
--notest list.gsp
Ready!
list.gsp: text/plain; charset=us-ascii

Post by Al Baker
What OS are you on?

Fedora 15
I have no idea what the problem is, but my workaround has satisfied
the client, and new data going into the db is being converted to HTML
codes, so for now I'm happy. I'm still curious about this though...
:-/
..dan

Post by Al Baker

Post by Peter Ledbrook

Post by Daniel Woods
Here's my datasource configuration for my grails app -- maybe
something worthwhile here?
dataSource {

...

Post by Daniel Woods
dbCreate = "update"
url =

"jdbc:mysql://localhost/database?tcpKeepAlive=true&useUnicode=yes&characterEncoding=UTF-8"

Post by Al Baker

Post by Peter Ledbrook

Post by Daniel Woods
autoReconnect = true
dialect = org.hibernate.dialect.MySQL5Dialect

...
One other option: restore the database to ensure all data is UTF-8,
then remove the 'dbCreate' and 'dialect' properties before running
your Grails application again.
Peter
--
Peter Ledbrook
Grails Advocate
SpringSource - A Division of VMware
---------------------------------------------------------------------
http://xircles.codehaus.org/manage_email

---------------------------------------------------------------------
http://xircles.codehaus.org/manage_email

[Daniel Woods]

Post by Peter Ledbrook
One other option: restore the database to ensure all data is UTF-8,
then remove the 'dbCreate' and 'dialect' properties before running
your Grails application again.

Done; same effect.

..dan

Post by Peter Ledbrook
Peter
--
Peter Ledbrook
Grails Advocate
SpringSource - A Division of VMware
---------------------------------------------------------------------
   http://xircles.codehaus.org/manage_email

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

[Peter Ledbrook]

Post by Daniel Woods
"default-collation" directives respectively. My MySQL JDBC URL string
contains the "useUnicode=true" and "characterEncoding=utf8"
parameters. My Config.groovy contains the "grails.views.gsp.encoding"
and "grails.converters.encoding" parameters with both set to "UTF-8".

This problem is typically a result of a database or db connection
issue. For what it's worth, the JDBC URL on grails.org is:

jdbc:mysql://...?autoReconnect=true&useUnicode=yes&characterEncoding=UTF-8

and that seems to be working fine.

Peter

[Daniel Henrique Alves Lima]

Daniel, are you saying that you've generated a *new* Grails
app and the UTF-8 characters were not properly displayed? Can you
share this app?
If you use HSQLDB (instead of MySQL) with your *new* Grails
app, how does the text look like?
I know nothing about MySQL, but reading your posts I would bet
there is something weird in your environment or in your use of the
MySQL JDBC driver.

Grails has some issues rendering ISO-8859-1, even when
everything is configured correctly, but I haven't seen anything like
this for UTF-8 (I'm assuming that every configuration is ok and every
source file is correctly encoded).


Best regards,

Daniel.

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

[Daniel Woods]

       Daniel, are you saying that you've generated a *new* Grails
app and the UTF-8 characters were not properly displayed? Can you
share this app?

Yes; It would take me a lot of work to make the application
not-confidential, and since I have a suitable workaround (and that I
agree with you that my environment is doing something weird -- I just
can't figure it out) it's probably not worth the effort for me to do
this. I will tell you though that I took and existing database with
UTF-8 encoded characters, which display correctly in other frameworks,
and mapped my domain layer on top of that to start, and all the
columns that displayed fine in the other frameworks displayed only in
8859-1 from Grails. I also tried migrating the data to a new schema
generated from my domain classes through a Groovy script using
groovy.sql.Sql with hand-written insert statements, and Groovy SQL
didn't preserve the UTF-8 encoding either.

       If you use HSQLDB (instead of MySQL) with your *new* Grails
app, how does the text look like?

This I didn't try.

       I know nothing about MySQL, but reading your posts I would bet
there is something weird in your environment or in your use of the
MySQL JDBC driver.

Agreed... I'm just not sure what. From what I can see, I've done
everything correctly.

       Grails has some issues rendering ISO-8859-1, even when
everything is configured correctly, but I haven't seen anything like
this for UTF-8 (I'm assuming that every configuration is ok and every
source file is correctly encoded).

No doubt. And all the pages are actually rendering with UTF-8
encoding, but the character set that is displayed is ISO-8859-1. I
confirmed this by using one of the many online UTF-8 to Latin1
converters to convert the proper UTF-8 character to Latin1 format, and
the result was what the GSPs were rendering.

So, in short, I'm sure that it's something that I did wrong along the
way, but even using the existing data that works from Perl and Rails,
I couldn't get Grails or Groovy to render the characters correctly...
A suitable solution was to transform the UTF-8 characters to HTML
codes and store them as such in the database, which is what the Grails
app will do going forward anyway, so I'm sure this will serve as a
long-term solution as well.

Thanks for the email and the feedback.

..dan

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

[Daniel Henrique Alves Lima]

Daniel, I'm not asking for your original/main/true/real app.
I'm just asking for a simple (and **new**) Grails app that only
selects one encoded text field in one of your tables. I think you
won't even need to sent it to us.

Try this:

1. Create a new app using grails create-app;
2. Create one domain class that refers one of your tables.
Because you will analyze just one field, I believe you can simply map
one column/property;
3. Create a controller that performs a
YourDomainObject.withTransaction{YourDomainObject.findAll()};
4. Write a simple gsp that produce a html using the values in
YourDomainObject.findAll();
5. Check the results.

Are they wrongly encoded?

Best regards,

Daniel.

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

[Daniel Woods]

       1. Create a new app using grails create-app;

Done. Call it "utf8test".

       2. Create one domain class that refers one of your tables.
Because you will analyze just one field, I believe you can simply map
one column/property;

Done. class Products { String title; String description; static
mapping = { version false } }

       3. Create a controller that performs a
YourDomainObject.withTransaction{YourDomainObject.findAll()};

grails generate-all

       4. Write a simple gsp that produce a html using the values in
YourDomainObject.findAll();

see #3

       5. Check the results.
       Are they wrongly encoded?

Yes, still.

It's a mystery to me... Thanks for the ideas.

..dan

       Best regards,
               Daniel.
---------------------------------------------------------------------
   http://xircles.codehaus.org/manage_email

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

[Daniel Henrique Alves Lima]

I don't think this is a mystery :-)

Using your test results, we can conclude that

1. Your db is wrongly encoded;
2. Your JDBC driver is unable to properly convert the encoded data;

Can you answer the following questions?

1. Which OS are you using? If you're using some *nix, execute

uname -a

2. Which JRE/JDK are you using?

which java ; java -version ; echo $JAVA_HOME

3. Which version of Grails are you using?

which grails ; grails ; echo $GRAILS_HOME

4. What are the default language and encoding in use?

echo $LANG

5. Are you changing the default language or encoding of your JVM?

Add

println System.properties

to your BootStrap.groovy and send the results to us (if you can)

6. Which version of MySQL are you using?


7. Which version of the MySQL JDBC driver are you using?


8. Can you share the test application that you've created? Just
replace the database user/password/server and send it to us (or to me in
private), if you can.

Post by Daniel Woods

Post by Daniel Henrique Alves Lima
1. Create a new app using grails create-app;

Done. Call it "utf8test".

Post by Daniel Henrique Alves Lima
2. Create one domain class that refers one of your tables.
Because you will analyze just one field, I believe you can simply map
one column/property;

Done. class Products { String title; String description; static
mapping = { version false } }

Post by Daniel Henrique Alves Lima
3. Create a controller that performs a
YourDomainObject.withTransaction{YourDomainObject.findAll()};

grails generate-all

Post by Daniel Henrique Alves Lima
4. Write a simple gsp that produce a html using the values in
YourDomainObject.findAll();

see #3

Post by Daniel Henrique Alves Lima
5. Check the results.
Are they wrongly encoded?

Yes, still.
It's a mystery to me... Thanks for the ideas.
..dan

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email