BUG: UTF-8 pages not displaying correctly (suggested solution included)

flyingdutchie

There are various thread dealing with improper displaying of international characters (just search for UTF-8 on the support forums).

1. Smugmug does correctly generate pages with this meta tag:
<meta http-equiv="content-type" content="text/html; charset=UTF-8" />
2. However, the HTTP response-header "Content-type" is incorrect! When i do a 'graburl' to my site, i get this http-response:
HTTP/1.1 200 OK
Date: Sat, 08 Oct 2005 16:07:01 GMT
Server: Apache
X-Powered-By: smugmug/1.2.0
Set-Cookie: SMSESS=026e577c53333abbcfcfd9247f880068; path=/; domain=.smugmug.com
Cache-Control: private, max-age=1, must-revalidate
Pragma:
X-Extra: 0.26807594299316
ETag: sm-a54c5ea41ed6f8c1233cdecdc2405552-sm
Content-Length: 29046
Connection: close
Content-Type: text/html; charset=ISO-8859-1

The incorrect Content-Type header causes the browsers to select ISO-8859-1 (Western European (ISO)) instead. Can this be fixed?

The last line (in red) is incorrect. It should read:
Content-Type: text/html; charset=UTF-8


Related threads:
http://www.dgrin.com/showthread.php?p=175958#post175958
http://www.dgrin.com/showthread.php?t=18290

Addition: I just saw that Luke Church did suggest the same solution:
http://www.dgrin.com/showpost.php?p=162986&postcount=12

Scala

The people at smugmug must be well aware of all this, but here we go again:

In addition to the suggested fix above the content that was once saved as ISO-8859-1 (and is still saved as such if you type in text using the customize gallery screen, for example) will need to be converted to UTF-8. If not, my guess is that a lot of currently working international characters will end up displaying incorrectly.

Here's an example. Go to http://scala.smugmug.com and select UTF-8 as the content encoding. There will be characters in my bio text displaying incorrectly or not at all.

flyingdutchie

Scala wrote:

The people at smugmug must be well aware of all this, but here we go again:

In addition to the suggested fix above the content that was once saved as ISO-8859-1 (and is still saved as such if you type in text using the customize gallery screen, for example) will need to be converted to UTF-8. If not, my guess is that a lot of currently working international characters will end up displaying incorrectly.

Here's an example. Go to http://scala.smugmug.com and select UTF-8 as the content encoding. There will be characters in my bio text displaying incorrectly or not at all.

I see,
but Smugmug should choose one or the other, since the meta tag in the pages looks like this:
<meta http-equiv="content-type" content="text/html; charset=UTF-8" />

So, the http response-header tells the browser to use one encoding, the <meta> tag to use another... it is still a bug.
-- Anton.

Scala

flyingdutchie wrote:

So, the http response-header tells the browser to use one encoding, the <meta> tag to use another... it is still a bug.
-- Anton.

Indeed. And the fix is long overdue.

{JT}

So this should be fixed, would love feedback from someone. I don't think previous entries are fixed yet - some might be though.

Scala wrote:

Indeed. And the fix is long overdue.

Scala

{JT} wrote:

So this should be fixed, would love feedback from someone. I don't think previous entries are fixed yet - some might be though.

I tried entering some Finnish and Vietnamese text in comments and gallery description and it seems to work. Thanks!

There's still the issue of the META tag implying UTF-8 but the browser selecting ISO-8859-1 but I guess that's how you want it. Either way there's something advanced going on because I thought ISO-8859-1 did not include all those strange Vietnamese characters

Previous entries are still wrong.

And the new "Updating..." notification does not go away if you delete the last (only) gallery comment. :

The end user experience is a lot better now internationally, thanks again!

rainforest1155

{JT} wrote:

So this should be fixed, would love feedback from someone. I don't think previous entries are fixed yet - some might be though.

I successfully added a comment containing these characters:
á Á à À â Â é É è È ê Ê í Í ì Ì î Î ó Ó ò Ò ô Ô ú Ú ù Ù û Û
ä Ä ö Ö ü Ü ß

Just the €uro isn't working, yet.

The comments didn't change and are still wrong (here - ö = ö and here ü = ü)

Sebastian

Scala

rainforest1155 wrote:

Just the €uro isn't working, yet.

I tried the €uro in a comment and it worked for me in IE6 and Firefox 1.0.7. But I agree that in theory it should not work since ISO-8859-1 doesn't have the euro character

{JT}

As longs as it works for now, I am happy

Scala wrote:

I tried the €uro in a comment and it worked for me in IE6 and Firefox 1.0.7. But I agree that in theory it should not work since ISO-8859-1 doesn't have the euro character

rainforest1155

{JT} wrote:

As longs as it works for now, I am happy

Yeah, works now for me, but before it clearly didn't. Did some finetuning, didn't we? Nice!

Sebastian

flyingdutchie

{JT} wrote:

As longs as it works for now, I am happy

{JT}, as a fellow software engineer, I'm just wondering...
Why is it so tricky to send over UTF-8 instead of ISO-8859-1? Has it to do with the existing text-data in your databases?

The http-response headers still say it is ISO-8859-1 and the <META> tags declare it to be UTF-8.. still a contradiction here.

-- Anton

{JT}

flyingdutchie wrote:

Has it to do with the existing text-data in your databases?

Bingo.

flyingdutchie

{JT} wrote:

Bingo.

Aha... migration is the issue here...!

But still, what about the contradicting http-response header and <META> tag? Shouldn't the <META> tag specify the ISO-8859-1 charset as well?

{JT}

Maybe As I stated earlier - right now everything is working, so I am hesitant to change anyting

flyingdutchie wrote:

Aha... migration is the issue here...!

But still, what about the contradicting http-response header and <META> tag? Shouldn't the <META> tag specify the ISO-8859-1 charset as well?

DmitryS

{JT} wrote:

Maybe As I stated earlier - right now everything is working, so I am hesitant to change anyting

Russian characters works now too.
However:
1. When I change description of the gallery using "Edit" button, I can start new lines. If I go to the gallery desription, new line symbols can't be used.
2. The limit for number of non-english characters in the description is very small, about 25 symbols. For enlish - it's much larger.
Not a big deal, though

rainforest1155

{JT} wrote:

Maybe As I stated earlier - right now everything is working, so I am hesitant to change anyting

I did some more testing and found out that it depends where the €uro symbol stands for it to work or not.
This doesn't work:

€ á 
€ á

but that does:

€ á 
€ á

EDIT: In fact I boiled it down to this - if there's at least one space before the €uro, it won't work. It only works when the €uro is the first character. If there are more €uros in the comment it depends on a €uro being the first character of the comment or not for all others!

Works:

€
Test123 €
Another testline that is fine €

but if you add a space before the first €uro resulting in it not being the first character of the comment it's broken:

€ 
Test123 € 
Another testline that is fine €

Any ideas on this?
Sebastian

{JT}

This happens to the euro only though, correct?

I might an idea as to why, but have to check first ...

rainforest1155 wrote:

I did some more testing and found out that it depends where the €uro symbol stands for it to work or not.
This doesn't work:

€ á 
€ á

but that does:

€ á 
€ á

EDIT: In fact I boiled it down to this - if there's at least one space before the €uro, it won't work. It only works when the €uro is the first character. If there are more €uros in the comment it depends on a €uro being the first character of the comment or not for all others!

Works:

€
Test123 €
Another testline that is fine €

but if you add a space before the first €uro resulting in it not being the first character of the comment it's broken:

€ 
Test123 € 
Another testline that is fine €

Any ideas on this?
Sebastian