Header encoding/charset issue in 1.4.23-svn

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Header encoding/charset issue in 1.4.23-svn

Paul Lesniewski
NOTE:  SourceForge is currently having problems with their mailing
lists.  This is a message originally from Juergen Nickelsen sent on
Thu, 13 Nov 2014 14:13:49 +0100
=====

Hello all,

background: I run two SquirrelMail instances, one for our university
members, the other for an academic government institute. Both are
currently 1.4.22 (with a few, but different, local changes each) on
Debian "Squeeze". Impending OS upgrade to "Wheezy" implies PHP 5.4,
which AFAIK means I have to move to 1.4.23-svn.

Rebasing our local changes on the squirrelmail-20141105_0200-SVN.stable
snapshot seemed to be successful, except for one thing:

Header lines (e.g. From, Subject) that are encoded in a charset (test
case: iso-8859-1) that is not SquirrelMail's $default_charset (here:
utf-8) are not decoded correctly. The error message in the log is "PHP
Warning:  htmlspecialchars(): Invalid multibyte sequence in argument in
/home/webmail/src/squirrel/functions/strings.php on line 1512".
SquirrelMail then does not display the corresponding header contents in
the message list or the message display, rather "(unknown)".

This issue is not only present in our locally patched versions, but also
in the version in the Debian package of SquirrelMail for "Wheezy", which
is claimed to be 2:1.4.23~svn20120406-2, so apparently 2.5 years older,
as well as in a very sparingly-configured unpatched installation of
today's 1.4.23-svn snapshot.

The decisive point seems to be the value of $default_charset versus the
header encoding -- only when I set it to 'utf-8', the problem appears
with iso-8859-1 headers, although not with utf-8 headers.

I tracked the issue down to functions/i18n.php:charset_encode(), where
the charset used in the header line is not passed to
sm_encode_html_special_chars(), so htmlspecialchars() is then called
with the default encoding.

This patch seems to fix the problem:

diff --git a/functions/i18n.php b/functions/i18n.php
index ec19c25..edbc6d6 100644
--- a/functions/i18n.php
+++ b/functions/i18n.php
@@ -184,7 +184,7 @@ function charset_decode ($charset, $string,
$force_decode=3Dfalse, $save_html=3Dfals
     }

     /* All HTML special characters are 7 bit and can be replaced first *=
/
-    if (! $save_html) $string =3D sm_encode_html_special_chars ($string)=
;
+    if (! $save_html) $string =3D sm_encode_html_special_chars ($string,=

ENT_COMPAT | ENT_HTML401, $charset);
     $charset =3D strtolower($charset);

     set_my_charset();

I have attached a message that I have used to reproduce the problem. The
>From and Subject header lines are encoded in iso-8859-1, while
SquirrelMail's configured charset is utf-8. (I have also changed the
language charset for the languages we use (de_DE, en_US) to utf-8.
Apparently this is irrelevant, though.)

Perhaps I am "doing it wrong", but we need UTF-8, as the university
members and their correspondents come in all shapes, sizes, colors, and
languages.

------------------------------------------------------------------------------
New Year. New Location. New Benefits. New Data Center in Ashburn, VA.
GigeNET is offering a free month of service with a new server in Ashburn.
Choose from 2 high performing configs, both with 100TB of bandwidth.
Higher redundancy.Lower latency.Increased capacity.Completely compliant.
http://p.sf.net/sfu/gigenet
-----
squirrelmail-users mailing list
Posting guidelines: http://squirrelmail.org/postingguidelines
List address: [hidden email]
List archives: http://news.gmane.org/gmane.mail.squirrelmail.user
List info (subscribe/unsubscribe/change options): https://lists.sourceforge.net/lists/listinfo/squirrelmail-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Header encoding/charset issue in 1.4.23-svn

Paul Lesniewski
> Header lines (e.g. From, Subject) that are encoded in a charset (test
> case: iso-8859-1) that is not SquirrelMail's $default_charset (here:
> utf-8) are not decoded correctly. The error message in the log is "PHP
> Warning:  htmlspecialchars(): Invalid multibyte sequence in argument in
> /home/webmail/src/squirrel/functions/strings.php on line 1512".
> SquirrelMail then does not display the corresponding header contents in
> the message list or the message display, rather "(unknown)".

There is a patch available for this (make sure to get version 2) here:

https://sourceforge.net/p/squirrelmail/bugs/2806

Please report any issues or success with it so I can commit it to our
repository!

--
Paul Lesniewski
SquirrelMail Team
Please support Open Source Software by donating to SquirrelMail!
http://squirrelmail.org/donate_paul_lesniewski.php

------------------------------------------------------------------------------
New Year. New Location. New Benefits. New Data Center in Ashburn, VA.
GigeNET is offering a free month of service with a new server in Ashburn.
Choose from 2 high performing configs, both with 100TB of bandwidth.
Higher redundancy.Lower latency.Increased capacity.Completely compliant.
http://p.sf.net/sfu/gigenet
-----
squirrelmail-users mailing list
Posting guidelines: http://squirrelmail.org/postingguidelines
List address: [hidden email]
List archives: http://news.gmane.org/gmane.mail.squirrelmail.user
List info (subscribe/unsubscribe/change options): https://lists.sourceforge.net/lists/listinfo/squirrelmail-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Working OpenSUSE 13.2 config and Header encoding/charset issue in 1.4.23-svn

Niclas Arndt
This e-mail is both in response to the e-mail below and to my question
in December whether anybody could advice on a working OpenSUSE 13.2
config. (Since other users wanted me to report back.)

I have now applied the patch (v2 for 1.4.) and so far it seems good. I will monitor the behaviour over the next few days.

I had problems with missing subject line characters after (inside?)  the ISO-8859-1 markings, e.g. in this header that now prints correctly:
Subject: ONSDAG: Ericsson: "Det =?ISO-8859-1?Q?=E4r=20rimligt=20att=20vi=20f=E5r?=

     =?ISO-8859-1?Q?=20en=20r=E4ttvis=20kompensation=22?=

I also had problems with raw body text parts that would only display (at all) if I clicked "View as HTML" and that the body text sometimes wasn't quoted at all when replying. These bugs all seem to have been fixed, at least after a quick look at my inbox.

Here's my setup:
OpenSUSE 13.2 64-bit
PHP5 5.6.1-4.1
Apache2 2.4.10-12.1
Squirrelmail 1.4.23 SVN late December 2014 (do these builds have versions?)
Dovecot 2.2-2.1
Postfix 2.11.0-5.2.2 with legacy greylisting (not Postgrey)
in config.php, default language is 'sv_SE' and default charset is 'iso-8859-1'.

I migrated from OpenSUSE 12.3. Please note that there are differences both in Apache2 and in PHP5 that require modification of config files. Currently I don't know all that I did, but you should be able to google for it now that you know what to look for.

I hope this helps.  Thanks for the help for this great software (Paul and Attila).

Cheers, Niclas

----

> Date: Mon, 19 Jan 2015 10:04:55 -0800
> Subject: Re: [SM-USERS] Header encoding/charset issue in 1.4.23-svn
>
> > Header lines (e.g. From, Subject) that are encoded in a charset (test
> > case: iso-8859-1) that is not SquirrelMail's $default_charset (here:
> > utf-8) are not decoded correctly. The error message in the log is "PHP
> > Warning:  htmlspecialchars(): Invalid multibyte sequence in argument in
> > /home/webmail/src/squirrel/functions/strings.php on line 1512".
> > SquirrelMail then does not display the corresponding header contents in
> > the message list or the message display, rather "(unknown)".
>
> There is a patch available for this (make sure to get version 2) here:
> https://sourceforge.net/p/squirrelmail/bugs/2806
>
> Please report any issues or success with it so I can commit it to our
> repository!

> Paul Lesniewski

     
------------------------------------------------------------------------------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
-----
squirrelmail-users mailing list
Posting guidelines: http://squirrelmail.org/postingguidelines
List address: [hidden email]
List archives: http://news.gmane.org/gmane.mail.squirrelmail.user
List info (subscribe/unsubscribe/change options): https://lists.sourceforge.net/lists/listinfo/squirrelmail-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Issue report for patch /p/squirrelmail/bugs/2806

Niclas Arndt
Hi,

After almost a month using the patch, I can report only one problem:

Inbox listing:
     =?utf-8?Q?Vidar Wern=C3=B6e?=     Feb 20, 2015           Seminarium om RoHS2, CE-märkning och kravet på dok...

Header excerpt:
From: "=?utf-8?Q?Vidar Wern=C3=B6e?=" <[hidden email]>
Sender: "=?utf-8?Q?Vidar Wern=C3=B6e?=" <[hidden email]>

So far no known problems with the e-mail bodies.

Cheers

Niclas


> Date: Mon, 19 Jan 2015 10:04:55 -0800
> Subject: Re: [SM-USERS] Header encoding/charset issue in 1.4.23-svn
>
> > Header lines (e.g. From, Subject) that are encoded in a charset (test
> > case: iso-8859-1) that is not SquirrelMail's $default_charset (here:
> > utf-8) are not decoded correctly. The error message in the log is "PHP
> > Warning:  htmlspecialchars(): Invalid multibyte sequence in argument in
> > /home/webmail/src/squirrel/functions/strings.php on line 1512".
> > SquirrelMail then does not display the corresponding header contents in
> > the message list or the message display, rather "(unknown)".
>
> There is a patch available for this (make sure to get version 2) here:
> https://sourceforge.net/p/squirrelmail/bugs/2806
>
> Please report any issues or success with it so I can commit it to our
> repository!

> Paul Lesniewski

         
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
-----
squirrelmail-users mailing list
Posting guidelines: http://squirrelmail.org/postingguidelines
List address: [hidden email]
List archives: http://news.gmane.org/gmane.mail.squirrelmail.user
List info (subscribe/unsubscribe/change options): https://lists.sourceforge.net/lists/listinfo/squirrelmail-users
Loading...