Quantcast

Blank message body on some replies

classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Blank message body on some replies

Queued Queuedington
In a nutshell, when I reply to some messages, instead of getting quoted text in the message body, I get nothing at all.

There are no errors, there are no indications in the logs that anything is amiss; the body is simply blank.

One clue is that the few affected messages appear to be larger, but I cannot find any knobs in squirrelmail or php that seem to affect how large a message can be replied to.

I am open to any suggestions.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Blank message body on some replies

Tomas Kuliavas
Queued Queuedington wrote
In a nutshell, when I reply to some messages, instead of getting quoted text in the message body, I get nothing at all.

There are no errors, there are no indications in the logs that anything is amiss; the body is simply blank.

One clue is that the few affected messages appear to be larger, but I cannot find any knobs in squirrelmail or php that seem to affect how large a message can be replied to.

I am open to any suggestions.
Message can have plain text and html text parts. Which parts are present on those messages? If messages have both parts, are texts identical in them (excluding html tags)?

--
Tomas
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Blank message body on some replies

Queued Queuedington
Tomas Kuliavas wrote
Queued Queuedington wrote
In a nutshell, when I reply to some messages, instead of getting quoted text in the message body, I get nothing at all.

There are no errors, there are no indications in the logs that anything is amiss; the body is simply blank.

One clue is that the few affected messages appear to be larger, but I cannot find any knobs in squirrelmail or php that seem to affect how large a message can be replied to.

I am open to any suggestions.
Message can have plain text and html text parts. Which parts are present on those messages? If messages have both parts, are texts identical in them (excluding html tags)?

--
Tomas
I have a "test case" where the message has only a plain text component.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Blank message body on some replies

Tomas Kuliavas
Queued Queuedington wrote
Tomas Kuliavas wrote
Queued Queuedington wrote
In a nutshell, when I reply to some messages, instead of getting quoted text in the message body, I get nothing at all.

There are no errors, there are no indications in the logs that anything is amiss; the body is simply blank.

One clue is that the few affected messages appear to be larger, but I cannot find any knobs in squirrelmail or php that seem to affect how large a message can be replied to.

I am open to any suggestions.
Message can have plain text and html text parts. Which parts are present on those messages? If messages have both parts, are texts identical in them (excluding html tags)?

--
Tomas
I have a "test case" where the message has only a plain text component.
Which PHP version are you using in your setup?

--
Tomas
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Blank message body on some replies

Queued Queuedington

Tomas Kuliavas wrote
Queued Queuedington wrote
Tomas Kuliavas wrote
Queued Queuedington wrote
In a nutshell, when I reply to some messages, instead of getting quoted text in the message body, I get nothing at all.

There are no errors, there are no indications in the logs that anything is amiss; the body is simply blank.

One clue is that the few affected messages appear to be larger, but I cannot find any knobs in squirrelmail or php that seem to affect how large a message can be replied to.

I am open to any suggestions.
Message can have plain text and html text parts. Which parts are present on those messages? If messages have both parts, are texts identical in them (excluding html tags)?

--
Tomas
I have a "test case" where the message has only a plain text component.
Which PHP version are you using in your setup?

--
Tomas
It is version 5.4.5, which complains about strictness in a few places, but otherwise, logs no errors and has no other problems.  (I've turned on the PHP error log, but there's nothing in it.)
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Blank message body on some replies

Tomas Kuliavas
Queued Queuedington wrote
Tomas Kuliavas wrote
Queued Queuedington wrote
Tomas Kuliavas wrote
Queued Queuedington wrote
In a nutshell, when I reply to some messages, instead of getting quoted text in the message body, I get nothing at all.

There are no errors, there are no indications in the logs that anything is amiss; the body is simply blank.

One clue is that the few affected messages appear to be larger, but I cannot find any knobs in squirrelmail or php that seem to affect how large a message can be replied to.

I am open to any suggestions.
Message can have plain text and html text parts. Which parts are present on those messages? If messages have both parts, are texts identical in them (excluding html tags)?

--
Tomas
I have a "test case" where the message has only a plain text component.
Which PHP version are you using in your setup?

--
Tomas
It is version 5.4.5, which complains about strictness in a few places, but otherwise, logs no errors and has no other problems.  (I've turned on the PHP error log, but there's nothing in it.)
Could you check if your test case has any non us-ascii symbols and which character set is used in it?

Could you also check your pcre.backtrack_limit value for SquirrelMail and see if increasing it solves your problem?

--
Tomas
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Blank message body on some replies

Queued Queuedington

Tomas Kuliavas wrote
Queued Queuedington wrote
Tomas Kuliavas wrote
Queued Queuedington wrote
Tomas Kuliavas wrote
Queued Queuedington wrote
In a nutshell, when I reply to some messages, instead of getting quoted text in the message body, I get nothing at all.

There are no errors, there are no indications in the logs that anything is amiss; the body is simply blank.

One clue is that the few affected messages appear to be larger, but I cannot find any knobs in squirrelmail or php that seem to affect how large a message can be replied to.

I am open to any suggestions.
Message can have plain text and html text parts. Which parts are present on those messages? If messages have both parts, are texts identical in them (excluding html tags)?

--
Tomas
I have a "test case" where the message has only a plain text component.
Which PHP version are you using in your setup?

--
Tomas
It is version 5.4.5, which complains about strictness in a few places, but otherwise, logs no errors and has no other problems.  (I've turned on the PHP error log, but there's nothing in it.)
Could you check if your test case has any non us-ascii symbols and which character set is used in it?

Could you also check your pcre.backtrack_limit value for SquirrelMail and see if increasing it solves your problem?

--
Tomas
I don't see any non us-ascii symbols, html, or anything funky.  

Content-Type: text/plain; charset="windows-1252"
MIME-Version: 1.0

pcre.backtrack_limit was at its default of 1000000
pcre.recursion_limit was at its default of 100000

Doubling both had no discernible effect

(By the way, thank you for your help so far, this has been driving me nuts.)
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Blank message body on some replies

Tomas Kuliavas
In reply to this post by Queued Queuedington
2012.08.30 23:47 Queued Queuedington rašė:

> Tomas Kuliavas wrote:
>> Queued Queuedington wrote:
>>>
>>> In a nutshell, when I reply to some messages, instead of getting quoted
>>> text in the message body, I get nothing at all.
>>>
>>> There are no errors, there are no indications in the logs that anything
>>> is amiss; the body is simply blank.
>>>
>>> One clue is that the few affected messages appear to be larger, but I
>>> cannot find any knobs in squirrelmail or php that seem to affect how
>>> large a message can be replied to.
>>>
>>> I am open to any suggestions.
>>>
>> Message can have plain text and html text parts. Which parts are present
>> on those messages? If messages have both parts, are texts identical in
>> them (excluding html tags)?
>>
>> --
>> Tomas
>>
>
> I have a "test case" where the message has only a plain text component.

Is it possible to get a copy of that test case. You can save all required
message information with message_details plugin. Forward as attachment
might work too, but I would like to avoid additional changes in message
contents.

I am running out of ideas and I have to reproduce this problem to
understand it.

I'll also need to know your SquirrelMail version, language and default
charset settings used in your setup, enabled plugins and interface
translation selected by user.

Could you confirm that SquirrelMail configtest.php does not complain about
mbstring extension configuration to avoid confusion between used string
functions.

--
Tomas


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
-----
squirrelmail-users mailing list
Posting guidelines: http://squirrelmail.org/postingguidelines
List address: [hidden email]
List archives: http://news.gmane.org/gmane.mail.squirrelmail.user
List info (subscribe/unsubscribe/change options): https://lists.sourceforge.net/lists/listinfo/squirrelmail-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Blank message body on some replies

Queued Queuedington

Tomas Kuliavas wrote
2012.08.30 23:47 Queued Queuedington rašė:
> Tomas Kuliavas wrote:
>> Queued Queuedington wrote:
>>>
>>> In a nutshell, when I reply to some messages, instead of getting quoted
>>> text in the message body, I get nothing at all.
>>>
>>> There are no errors, there are no indications in the logs that anything
>>> is amiss; the body is simply blank.
>>>
>>> One clue is that the few affected messages appear to be larger, but I
>>> cannot find any knobs in squirrelmail or php that seem to affect how
>>> large a message can be replied to.
>>>
>>> I am open to any suggestions.
>>>
>> Message can have plain text and html text parts. Which parts are present
>> on those messages? If messages have both parts, are texts identical in
>> them (excluding html tags)?
>>
>> --
>> Tomas
>>
>
> I have a "test case" where the message has only a plain text component.

Is it possible to get a copy of that test case. You can save all required
message information with message_details plugin. Forward as attachment
might work too, but I would like to avoid additional changes in message
contents.

I am running out of ideas and I have to reproduce this problem to
understand it.

I'll also need to know your SquirrelMail version, language and default
charset settings used in your setup, enabled plugins and interface
translation selected by user.

Could you confirm that SquirrelMail configtest.php does not complain about
mbstring extension configuration to avoid confusion between used string
functions.

--
Tomas
SquirrelMail version:  1.4.22
Language:  English

From configtest.php:

Checking internationalization (i18n) settings...
     gettext - Gettext functions are available. On some systems you must have appropriate system locales compiled.
     mbstring - Mbstring functions are available.
     recode - Recode functions are unavailable.
     iconv - Iconv functions are available.
     timezone - Webmail users can change their time zone settings.
Checking database functions...
    not using database functionality.

Congratulations, your SquirrelMail setup looks fine to me!

Message details is able to read/save the message (and of course, I can read the original.)

What's the best way to get it to you?  (I'm uncomfortable posting the entire message publicly.)

Thanks!
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Blank message body on some replies

Tomas Kuliavas
Queued Queuedington wrote
Tomas Kuliavas wrote
Queued Queuedington wrote
 I have a "test case" where the message has only a plain text component.
Is it possible to get a copy of that test case. You can save all required
message information with message_details plugin. Forward as attachment
might work too, but I would like to avoid additional changes in message
contents.

I am running out of ideas and I have to reproduce this problem to
understand it.

I'll also need to know your SquirrelMail version, language and default
charset settings used in your setup, enabled plugins and interface
translation selected by user.

Could you confirm that SquirrelMail configtest.php does not complain about
mbstring extension configuration to avoid confusion between used string
functions.
SquirrelMail version:  1.4.22
Language:  English

From configtest.php:
...

What's the best way to get it to you?  (I'm uncomfortable posting the entire message publicly.)
Nabble is only front end for mailing list. If you used valid email address and was subscribed to mailing list, you should be able to see my email address.

--
Tomas
tokul@users.sourceforge.net
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Blank message body on some replies

Tomas Kuliavas
In reply to this post by Queued Queuedington
Queued Queuedington wrote
Message details is able to read/save the message (and of course, I can read the original.)

What's the best way to get it to you?  (I'm uncomfortable posting the entire message publicly.)
Problem is caused by PHP 5.4 change in htmlspecialchars defaults. In PHP 5.4 function switched from iso-8859-1 to utf-8, started testing text format and started freaking out on any malformed text. UTF-8 is multibyte character set with very strict formating rules. Any non utf8 text has very big chance of breaking utf-8 formating rules and SquirrelMail fed Windows-1252 text to that function.

Your sample test case has one :) non us-ascii symbol. You can't see it in message view as same PHP function cuts off whole chapter there. Sample email has additional chapter between 'I can understand that. ...' and 'I got concentrate from...' chapters.

Quick hack for compose is in http://pastebin.com/3RQCUEV4

Proper fix is a lot more complex as multiple places are affected by this PHP 5.4 change.
--
Tomas
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Blank message body on some replies

Queued Queuedington

Tomas Kuliavas wrote
Queued Queuedington wrote
Message details is able to read/save the message (and of course, I can read the original.)

What's the best way to get it to you?  (I'm uncomfortable posting the entire message publicly.)
Problem is caused by PHP 5.4 change in htmlspecialchars defaults. In PHP 5.4 function switched from iso-8859-1 to utf-8, started testing text format and started freaking out on any malformed text. UTF-8 is multibyte character set with very strict formating rules. Any non utf8 text has very big chance of breaking utf-8 formating rules and SquirrelMail fed Windows-1252 text to that function.

Your sample test case has one :) non us-ascii symbol. You can't see it in message view as same PHP function cuts off whole chapter there. Sample email has additional chapter between 'I can understand that. ...' and 'I got concentrate from...' chapters.

Quick hack for compose is in http://pastebin.com/3RQCUEV4

Proper fix is a lot more complex as multiple places are affected by this PHP 5.4 change.
--
Tomas
Thank you!  It hadn't occurred to me that I was [also] missing part of the message in the message view, which also explains why I didn't realize there were any non-ascii characters.

Thank you also for the fix, which seems to do the trick.

Is the "correct" thing to do, perhaps, converting everything to utf8 that's received in a different character set?

Thank you also for the fix, which I'll implement and play with.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Blank message body on some replies

Tomas Kuliavas
Queued Queuedington wrote
Is the "correct" thing to do, perhaps, converting everything to utf8 that's received in a different character set?
Charset conversions are costly. Some converters have special function dependencies or very large mapping tables. In this case application only needs to sanitize three symbols.

Correct fix is to update every htmlspecialchars function call to less picky iso-8859-1 htmlspecialchars version. Application is dealing with external inputs and these inputs can contain anything. 5.4 htmlspecialchars function is fubared. It is supposed to return sanitized text, but it freaks out any non critical format error. Any normal text viewer would follow utf-8 specs and display unknown symbol instead of those broken byte sequences.

--
Tomas
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Blank message body on some replies

Queued Queuedington

Tomas Kuliavas wrote
Queued Queuedington wrote
Is the "correct" thing to do, perhaps, converting everything to utf8 that's received in a different character set?
Charset conversions are costly. Some converters have special function dependencies or very large mapping tables. In this case application only needs to sanitize three symbols.

Correct fix is to update every htmlspecialchars function call to less picky iso-8859-1 htmlspecialchars version. Application is dealing with external inputs and these inputs can contain anything. 5.4 htmlspecialchars function is fubared. It is supposed to return sanitized text, but it freaks out any non critical format error. Any normal text viewer would follow utf-8 specs and display unknown symbol instead of those broken byte sequences.

--
Tomas
At the risk of further imposing, your fix takes care of the reply being blank, do you happen to know the fix, or the section of code to look at, to take care of the paragraph that disappears when the mail is first read?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Blank message body on some replies

Tomas Kuliavas
Queued Queuedington wrote
Tomas Kuliavas wrote
Queued Queuedington wrote
Is the "correct" thing to do, perhaps, converting everything to utf8 that's received in a different character set?
Charset conversions are costly. Some converters have special function dependencies or very large mapping tables. In this case application only needs to sanitize three symbols.

Correct fix is to update every htmlspecialchars function call to less picky iso-8859-1 htmlspecialchars version. Application is dealing with external inputs and these inputs can contain anything. 5.4 htmlspecialchars function is fubared. It is supposed to return sanitized text, but it freaks out any non critical format error. Any normal text viewer would follow utf-8 specs and display unknown symbol instead of those broken byte sequences.

--
Tomas
At the risk of further imposing, your fix takes care of the reply being blank, do you happen to know the fix, or the section of code to look at, to take care of the paragraph that disappears when the mail is first read?
http://pastebin.com/sWJTzGCL

functions/i18n.php - charset_decode() function used to convert text from different charsets.

Fix should deal with main text body conversion issues, but I would not trust any htmlspecialchars() call without third argument in PHP 5.4.

--
Tomas
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Blank message body on some replies

taca
Hi,

Someone created wrapper function to htmlspecialchars() to fix this
problem.  This patch contains "jaJP" in its filename but it isn't
Japanese specific fix at all.

http://doga.jp/~taka2/archive/squirrelmail-webmail-1.4.22-PHP54jaJP_20120705taka2.diff

Best regards.

--
Takahiro Kambe <[hidden email]>

------------------------------------------------------------------------------
Got visibility?
Most devs has no idea what their production app looks like.
Find out how fast your code is with AppDynamics Lite.
http://ad.doubleclick.net/clk;262219671;13503038;y?
http://info.appdynamics.com/FreeJavaPerformanceDownload.html
-----
squirrelmail-users mailing list
Posting guidelines: http://squirrelmail.org/postingguidelines
List address: [hidden email]
List archives: http://news.gmane.org/gmane.mail.squirrelmail.user
List info (subscribe/unsubscribe/change options): https://lists.sourceforge.net/lists/listinfo/squirrelmail-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Blank message body on some replies

Tomas Kuliavas
taca wrote
Hi,

Someone created wrapper function to htmlspecialchars() to fix this
problem.  This patch contains "jaJP" in its filename but it isn't
Japanese specific fix at all.

http://doga.jp/~taka2/archive/squirrelmail-webmail-1.4.22-PHP54jaJP_20120705taka2.diff
Instead of copying syntax from PHP manual, code should ignore idiocy written there and use only ENT_COMPAT. ENT_HTML401 introduces backwards compatibility break and does nothing.

--
Tomas
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Blank message body on some replies

taca
Hi, it is late reply.

In message <[hidden email]>
        on Fri, 14 Sep 2012 09:35:30 -0700 (PDT),
        Tomas Kuliavas <[hidden email]> wrote:

> taca wrote:
>> Someone created wrapper function to htmlspecialchars() to fix this
>> problem.  This patch contains "jaJP" in its filename but it isn't
>> Japanese specific fix at all.
>>
>> http://doga.jp/~taka2/archive/squirrelmail-webmail-1.4.22-PHP54jaJP_20120705taka2.diff
>>
>
> Instead of copying syntax from PHP manual, code should ignore idiocy written
> there and use only ENT_COMPAT. ENT_HTML401 introduces backwards
> compatibility break and does nothing.
I don't understand what you meant.  ENT_COMPAT or ENT_HTML401 change
default charset of htmlspecialchars()?

Intention of patch ie solve the default charset change of
htmlspecialchars().

* Prior to PHP 5.4: default is ISO-8859-1 and if real encoding isn't
  ISO-8859-1, htmlspecialchars() process as ISO-8859-1 without error.

* PHP 5.4 and later: default is UTF-8 and if real encoding isn't
  UTF-8, htmlspecialchars() may cause error with byte sequence that
  isn't correct as UTF-8.

Best regards.

--
Takahiro Kambe <[hidden email]>

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://ad.doubleclick.net/clk;258768047;13503038;j?
http://info.appdynamics.com/FreeJavaPerformanceDownload.html
-----
squirrelmail-users mailing list
Posting guidelines: http://squirrelmail.org/postingguidelines
List address: [hidden email]
List archives: http://news.gmane.org/gmane.mail.squirrelmail.user
List info (subscribe/unsubscribe/change options): https://lists.sourceforge.net/lists/listinfo/squirrelmail-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Blank message body on some replies

Tomas Kuliavas
taca wrote
Hi, it is late reply.

In message <34433472.post@talk.nabble.com>
        on Fri, 14 Sep 2012 09:35:30 -0700 (PDT),
        Tomas Kuliavas <tokul@users.sourceforge.net> wrote:
> taca wrote:
>> Someone created wrapper function to htmlspecialchars() to fix this
>> problem.  This patch contains "jaJP" in its filename but it isn't
>> Japanese specific fix at all.
>>
>> http://doga.jp/~taka2/archive/squirrelmail-webmail-1.4.22-PHP54jaJP_20120705taka2.diff
>>
>
> Instead of copying syntax from PHP manual, code should ignore idiocy written
> there and use only ENT_COMPAT. ENT_HTML401 introduces backwards
> compatibility break and does nothing.
I don't understand what you meant.  ENT_COMPAT or ENT_HTML401 change
default charset of htmlspecialchars()?
ENT_HTML401 constant is introduced only in PHP 5.4. Its value is 0. This constant does not change 'htmlspecialchars($string, ENT_COMPAT | ENT_HTML401, $encoding);' call results. Function will follow only ENT_COMPAT specification due to zero value set on ENT_HTML401.

Your proposed patch introduces backwards compatibility break. Constant is not available in older PHP versions. It also follows broken PHP documentation synopsis, which PHP developers refused to fix.

--
Tomas
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Blank message body on some replies

taca
In message <[hidden email]>
        on Mon, 1 Oct 2012 08:50:55 -0700 (PDT),
        Tomas Kuliavas <[hidden email]> wrote:

>>> Instead of copying syntax from PHP manual, code should ignore idiocy
>>> written
>>> there and use only ENT_COMPAT. ENT_HTML401 introduces backwards
>>> compatibility break and does nothing.
>> I don't understand what you meant.  ENT_COMPAT or ENT_HTML401 change
>> default charset of htmlspecialchars()?
>>
> ENT_HTML401 constant is introduced only in PHP 5.4. Its value is 0. This
> constant does not change 'htmlspecialchars($string, ENT_COMPAT |
> ENT_HTML401, $encoding);' call results. Function will follow only ENT_COMPAT
> specification due to zero value set on ENT_HTML401.
>
> Your proposed patch introduces backwards compatibility break. Constant is
> not available in older PHP versions. It also follows broken PHP
> documentation synopsis, which PHP developers refused to fix.
Thanks for your clarification.  Hmmmm, I'll check it later...

--
Takahiro Kambe <[hidden email]>

------------------------------------------------------------------------------
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev
-----
squirrelmail-users mailing list
Posting guidelines: http://squirrelmail.org/postingguidelines
List address: [hidden email]
List archives: http://news.gmane.org/gmane.mail.squirrelmail.user
List info (subscribe/unsubscribe/change options): https://lists.sourceforge.net/lists/listinfo/squirrelmail-users
12
Loading...