test debug

test debug

[Alviro Iskandar Setiawan]

test aja

-- Viro

Re: test debug

[Ammar Faizi]

Got your email.

On Mar 6, 2022, 5:50 PM, at 5:50 PM, Alviro Iskandar Setiawan <alviro.iskandar@gnuweeb.org> wrote:
>test aja
>
>-- Viro

Re: test debug

[Ammar Faizi]

On 3/6/22 5:49 PM, Alviro Iskandar Setiawan wrote:
> test aja

OK, it seems something went wrong when the request is piped to the
public-inbox-tg.php. The raw data looks like this:

gz + base64:
H4sIAAAAAAACA61UXXeqOBR9Lr8ij+2yKN8iLV1F/EJBLX7WWfchQsQ
ogg1B0V8/wZnb2941M0/DAwtOcvbZOydnd0h6APBwgGQD8RVLr1GSnx
FaV1MSATDJE+BBAoAGRMFQZUPVgSRIEucjmpOEH0O6NcDzv+S/sG0Bw
icUGmBT1vlDFKpqVZLVqqhoP8B9nuyT9JywuC5VparWqIqq/OOBu1tf
wFce9+M0oxtcPIAzplvQnnjT8cQCOARN3VLleltr6k/cHWP7WFItGZc
sf3Kug4rAHnA/m9oPXGvgePwERwlkEpABTqb4BKBJMshnWyip2hMITI
JiWKCwluHDMUZPIDS/8GGlMjNEG5jH9AlQU9QUTdUUQS5JrLem1Z3X7
ZW/6vsVvIj7+qJWufo9WiyWmii8hzQociXpksJ3A5NlbE0n4X10jC/8
NDV8tEEEJQHKjEm+3qGAGh12dkYLUmSwddu+/ZeVTKjQfe+C7TPa19x
8dh5MPAuOe851NXKLhhr5+2Lx/lET3eZwe95sUXOkBHlHIE0yqSt0ma
y4O8CYDN6c3pT03wfWwKWdYeU6tB2n3j8OnLQbyxX9ItfUxqx/LZat2
qnpidPBe3/o5vk46i9mV4mBWK0d3Idnkg07qD6ZdkdRZ9ezB26vnWW2
jS35XVPf9JMaFuJyVUSb5XU+zjqVt0Ph6UVgHXyPgXQW7W5/OmrZzZG
PA+dNFAr/WIHzUVNwvWNxGWZXj4qVzfjD9TKyds6u33xfSpVVRQomLj
nsagzEJYWeHlv+JVg1rUAaF0t/RIq+b6ntyUVxOsJe63t1orndnoyle
B6c1sqgWbhDdqPbdlfxFAaiJ9ru2KkvB1J7Hpkm97U/4Nm2Rl1N+Rij
8TAb7Q56sfxYSGEUTQ75W2bNuv3jrohW02XTlVo7Eo0i7e31AHFcjW7
vID2Ug/HZ5f8HcMl/Qob8DLOJq0saN90SBENG+4gDA1CUURCidR5xsw
wR3opQQg1gJSFJccggnIzvogQRdtVC3kNZBiPEOwyLkhxxnuO1+TkiG
U4TA4hVgbPThDIIfkpgkrHqfDsJ0hAnkQE+8rQEORKcULiO0a+9lyMq
qRS0dowhTp44EGwhyRA1Z9MOr3M/bz3w0TfKtykAVmk1oFN6zX/4zm1
YwM0PhK9+UL/5gcL8oC4IXNlNKz5hkgIn28MkZPsmiGJ4hgkDv61U8d
8r3wuwKQTd4ay2YBGGj2MmGriYkX2Ozof49RQhUv2W8XmcLdZxAaqy2
JBVvtHQBF4JQsTDhqLwDV0WJUkRkCj/pojrphRc0pwAVDbdlNocN/rL
nLXHm7pHoBqqAMbeI4D01/c/CDS5bxIZ1u8iWaisCs6EddHgXm5tgDv
IvXAvPA/mLIvj/gR0I3F5OwYAAA==


-- 
Ammar Faizi

Re: test debug

[Alviro Iskandar Setiawan]

test again

-- Viro

Re: test debug

[Ammar Faizi]

On 3/6/22 6:03 PM, Alviro Iskandar Setiawan wrote:
> test again

Got your email!

-- 
Ammar Faizi

Re: test debug

[Alviro Iskandar Setiawan]

On Sun, Mar 6, 2022 at 6:04 PM Ammar Faizi wrote:
> On 3/6/22 6:03 PM, Alviro Iskandar Setiawan wrote:
> > test again
>
> Got your email!
>

I think this regex is wrong sir, if the message-id is located at the
end of the header, we will not have double LF, because it's already
split by the explode.

if (!preg_match("/(?:^|\\n)message-id:\s+?(.+?)(?:\\n\S+\:|\\n\\n)/si",
$hdr, $m)) {
        $err = "Cannot get the \"message-id\" line";
        goto out;
}

maybe we should use this:

   "/(?:^|\\n)message-id:\s+?(.+?)(?:\\n\S+\:|$)/si"

-- Viro

Re: test debug

[Ammar Faizi]

On 3/6/22 6:08 PM, Alviro Iskandar Setiawan wrote:
> On Sun, Mar 6, 2022 at 6:04 PM Ammar Faizi wrote:
>> On 3/6/22 6:03 PM, Alviro Iskandar Setiawan wrote:
>>> test again
>>
>> Got your email!
>>
> 
> I think this regex is wrong sir, if the message-id is located at the
> end of the header, we will not have double LF, because it's already
> split by the explode.
> 
> if (!preg_match("/(?:^|\\n)message-id:\s+?(.+?)(?:\\n\S+\:|\\n\\n)/si",
> $hdr, $m)) {
>          $err = "Cannot get the \"message-id\" line";
>          goto out;
> }

Oh right, I forgot that we've already split it before parsing.

> maybe we should use this:
> 
>     "/(?:^|\\n)message-id:\s+?(.+?)(?:\\n\S+\:|$)/si"

That one looks good to me, we should also fix other headers too.

-- 
Ammar Faizi

Re: test debug

[Alviro Iskandar Setiawan]

On Sun, Mar 6, 2022 at 6:12 PM Ammar Faizi wrote:
> On 3/6/22 6:08 PM, Alviro Iskandar Setiawan wrote:
> > On Sun, Mar 6, 2022 at 6:04 PM Ammar Faizi wrote:
> > > On 3/6/22 6:03 PM, Alviro Iskandar Setiawan wrote:
> > > > test again
> > >
> > > Got your email!
> > >
> >
> > I think this regex is wrong sir, if the message-id is located at the
> > end of the header, we will not have double LF, because it's already
> > split by the explode.
> >
> > if (!preg_match("/(?:^|\\n)message-id:\s+?(.+?)(?:\\n\S+\:|\\n\\n)/si",
> > $hdr, $m)) {
> >          $err = "Cannot get the \"message-id\" line";
> >          goto out;
> > }
>
> Oh right, I forgot that we've already split it before parsing.
>
> > maybe we should use this:
> >
> >     "/(?:^|\\n)message-id:\s+?(.+?)(?:\\n\S+\:|$)/si"
>
> That one looks good to me, we should also fix other headers too.

I will send a patch to fix this, maybe tonight

-- Viro

Re: test debug

[Alviro Iskandar Setiawan]

Subject: [PATCH pit v1] bin/public-inbox-tg: Fix regex pattern for headers
From: Alviro Iskandar Setiawan <alviro.iskandar@gnuweeb.org>

When testing email with Faris, we found that Faris' emails don't get
processed. They appear on the lore, but don't appear on Telegram. Sir
Ammar debugged this bug and the log said: "Cannot get the \"message-id\"
line".

It turned out that the regex pattern for headers was wrong, if the
header to be matched happens to be at the end of the headers, it will
not have a double LF because previously we have already split the
double LF with explode(). Fix this by changing the pattern to match the
header if it happens to be at the end of the headers.

Cc: Faris Suryanto <onlyfaris@gnuweeb.org>
Debugged-by: Ammar Faizi <ammarfaizi2@gnuweeb.org>
Signed-off-by: Alviro Iskandar Setiawan <alviro.iskandar@gnuweeb.org>
---
 bin/public-inbox-tg.php | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/bin/public-inbox-tg.php b/bin/public-inbox-tg.php
index 9d5771f..2e611ec 100644
--- a/bin/public-inbox-tg.php
+++ b/bin/public-inbox-tg.php
@@ -145,27 +145,27 @@ function fx(string $input): int
 	}
 
 	$body = trim($hdr[1]);
-	$hdr = $hdr[0];
+	$hdr = trim($hdr[0]);
 
-	if (!preg_match("/(?:^|\\n)subject:\s+?(.+?)(?:\\n\S+\:|\\n\\n)/si", $hdr, $m)) {
+	if (!preg_match("/(?:^|\\n)subject:\s+?(.+?)(?:\\n\S+\:|$)/si", $hdr, $m)) {
 		$err = "Cannot get the \"subject\" line";
 		goto out;
 	}
 	$subject = clean_header_val($m[1]);
 
-	if (!preg_match("/(?:^|\\n)from:\s+?(.+?)(?:\\n\S+\:|\\n\\n)/si", $hdr, $m)) {
+	if (!preg_match("/(?:^|\\n)from:\s+?(.+?)(?:\\n\S+\:|$)/si", $hdr, $m)) {
 		$err = "Cannot get the \"from\" line";
 		goto out;
 	}
 	$from = clean_header_val($m[1]);
 
-	if (!preg_match("/(?:^|\\n)date:\s+?(.+?)(?:\\n\S+\:|\\n\\n)/si", $hdr, $m)) {
+	if (!preg_match("/(?:^|\\n)date:\s+?(.+?)(?:\\n\S+\:|$)/si", $hdr, $m)) {
 		$err = "Cannot get the \"date\" line";
 		goto out;
 	}
 	$date = clean_header_val($m[1]);
 
-	if (!preg_match("/(?:^|\\n)message-id:\s+?(.+?)(?:\\n\S+\:|\\n\\n)/si", $hdr, $m)) {
+	if (!preg_match("/(?:^|\\n)message-id:\s+?(.+?)(?:\\n\S+\:|$)/si", $hdr, $m)) {
 		$err = "Cannot get the \"message-id\" line";
 		goto out;
 	}
@@ -173,7 +173,7 @@ function fx(string $input): int
 	if ($msgId[0] === '<' && $msgId[strlen($msgId) - 1] === '>')
 		$msgId = substr(clean_header_val($msgId), 1, -1);
 
-	if (preg_match("/(?:^|\\n)in-reply-to:\s+?(.+?)(?:\\n\S+\:|\\n\\n)/si", $hdr, $m)) {
+	if (preg_match("/(?:^|\\n)in-reply-to:\s+?(.+?)(?:\\n\S+\:|$)/si", $hdr, $m)) {
 		$inReplyTo = $m[1];
 		if ($inReplyTo[0] === '<' && $inReplyTo[strlen($inReplyTo) - 1] === '>')
 			$inReplyTo = substr(clean_header_val($inReplyTo), 1, -1);
@@ -182,7 +182,7 @@ function fx(string $input): int
 	}
 
 	$err = "";
-	if (preg_match("/(?:^|\\n)to:\s+?(.+?)(?:\\n\S+\:|\\n\\n)/si", $hdr, $m)) {
+	if (preg_match("/(?:^|\\n)to:\s+?(.+?)(?:\\n\S+\:|$)/si", $hdr, $m)) {
 		$toList = extractList(clean_header_val($m[1]));
 		$toListStr = buildList($toList, "To");
 	} else {

base-commit: 667913decf1869fde55d8fd17d66b4a294670617
-- 
2.32.0

Re: test debug

[Ammar Faizi]

On 3/6/22 8:00 PM, Alviro Iskandar Setiawan wrote:
> Subject: [PATCH pit v1] bin/public-inbox-tg: Fix regex pattern for headers
> From: Alviro Iskandar Setiawan <alviro.iskandar@gnuweeb.org>
> 
> When testing email with Faris, we found that Faris' emails don't get
> processed. They appear on the lore, but don't appear on Telegram. Sir
> Ammar debugged this bug and the log said: "Cannot get the \"message-id\"
> line".
> 
> It turned out that the regex pattern for headers was wrong, if the
> header to be matched happens to be at the end of the headers, it will
> not have a double LF because previously we have already split the
> double LF with explode(). Fix this by changing the pattern to match the
> header if it happens to be at the end of the headers.

Applied, thanks!

-- 
Ammar Faizi

Re: test debug

[Alviro Iskandar Setiawan]

On Sun, Mar 6, 2022 at 8:06 PM Ammar Faizi wrote:
> Applied, thanks!

gud.
faris, test again, reply to this email plz

-- Viro

Re: test debug

[Faris Suryanto]

Test....... 👍

On 6 Mar 2022 20.11, at 20.11, Alviro Iskandar Setiawan <alviro.iskandar@gnuweeb.org> wrote:
>On Sun, Mar 6, 2022 at 8:06 PM Ammar Faizi wrote:
>> Applied, thanks!
>
>gud.
>faris, test again, reply to this email plz
>
>-- Viro

Re: test debug

[Alviro Iskandar Setiawan]

On Sun, Mar 6, 2022 at 8:13 PM Faris Suryanto wrote:
> Test....... 👍

mantep msuk

-- Viro

Re: test debug

[Alviro Iskandar Setiawan]

On Sun, Mar 6, 2022 at 8:14 PM Alviro Iskandar Setiawan wrote:
> On Sun, Mar 6, 2022 at 8:13 PM Faris Suryanto wrote:
> > Test....... 👍
>
> mantep msuk
>

something still goes wrong when the sender uses
"Content-Transfer-Encoding: quoted-printable"
the raw email shows =XX for unprintable chars

XX is a hex representation of the unprintable char

-- Viro

Re: test debug

[Ammar Faizi]

On 3/6/22 8:17 PM, Alviro Iskandar Setiawan wrote:
> something still goes wrong when the sender uses
> "Content-Transfer-Encoding: quoted-printable"
> the raw email shows =XX for unprintable chars
> 
> XX is a hex representation of the unprintable char
> 

This one a bit harder to do it manually, maybe we should start
using external library for this?

I don't feel strongly to write our own parser for this kind of
encoding.

-- 
Ammar Faizi

Re: test debug

[Alviro Iskandar Setiawan]

On Sun, Mar 6, 2022 at 8:20 PM Ammar Faizi wrote:
> On 3/6/22 8:17 PM, Alviro Iskandar Setiawan wrote:
> > something still goes wrong when the sender uses
> > "Content-Transfer-Encoding: quoted-printable"
> > the raw email shows =XX for unprintable chars
> >
> > XX is a hex representation of the unprintable char
> >
>
> This one a bit harder to do it manually, maybe we should start
> using external library for this?

yes, i think we better use an external library if there exists such a
library, but idk, me never parse this encoding

> I don't feel strongly to write our own parser for this kind of
> encoding.

me too, but if we have to, we should do it!!!!!!

-- Viro

Re: test debug

[Ammar Faizi]

On 3/6/22 8:25 PM, Alviro Iskandar Setiawan wrote:
> On Sun, Mar 6, 2022 at 8:20 PM Ammar Faizi wrote:
>> On 3/6/22 8:17 PM, Alviro Iskandar Setiawan wrote:
>>> something still goes wrong when the sender uses
>>> "Content-Transfer-Encoding: quoted-printable"
>>> the raw email shows =XX for unprintable chars
>>>
>>> XX is a hex representation of the unprintable char
>>>
>>
>> This one a bit harder to do it manually, maybe we should start
>> using external library for this?
> 
> yes, i think we better use an external library if there exists such a
> library, but idk, me never parse this encoding

We can defer this until we have good enough time for fixing it. I think
there are still more important works to be done. For example, finishing
the server migration, which is much more urgent than Telegram notification.

+ Added Dhiky to CC list.

>> I don't feel strongly to write our own parser for this kind of
>> encoding.
> 
> me too, but if we have to, we should do it!!!!!!

Sure we can do it, shouldn't be too hard for a competent programmer
like you :p

-- 
Ammar Faizi