[PATCH v2 0/8] Fix some bugs and add some features

[PATCH v2 0/8] Fix some bugs and add some features

[Muhammad Rizki]

Hi sir,

This is v2 revision of fix some bugs and add some features.
This series contain add if statement to ignore text/html and remove
a re.sub().

This series is to fix some bugs, improve some codes and add some new
features. These bugs should have been fixed now, and the email file
attachments should have been removed after all attachments have been
sent.

Known bugs:
1. Email payload extraction result become unicode if the email payload
   contain non-UTF8 characters like chinese, japanese, and similar
   like that.
2. remove_patch() doesn't remove all file attachments properly.

Improvements:
1. Improve fix_utf8_char() to make it stable for both Discord and
   Telegram bot.
2. Improve remove_patch() to make it all file attachments removed
   after sending them.
3. Fix `/lore {raw atom url}` to add a "telegram" onto the
   create_template() platform parameter.
4. Improvement on extract_body() to ignore whenever the email contain
   text/html content-type.

New features:
1. Add send_text_mail_interaction() for the `/lore {raw atom url}` slash
   command.
2. Add send_patch_mail_interaction() for the `/lore {raw atom url}`
   slash command.
3. Add `/lore {raw atom url}` slash command.

There are 8 patches in this series:
- Patch 1 is to add send_text_mail_interaction()
- Patch 2 is to add send_patch_mail_interaction()
- Patch 3 is to add `/lore` slash command
- Patch 4 is to improve fix_utf8_char() code
- Patch 5 is to improve remove_patch() code to make it more stable
- Patch 6 is to add manage_payload() for manage email payload extraction
- Patch 7 is to fix the Telegram `/lore` command
- Patch 8 is to improve extract_body() to ignore text/html content-type

How to use:
1. Execute the db.sql file in the daemon directory,
2. Setup .env file, the example is there with suffix .example, this
   file name must remove the suffix name .example,
3. Set up the config.py in each bot directory, such as dscord and
   telegram. The example is there with suffix .example & the file name
   must remove suffix name .example,
4. Run `pip3 install -r requirements.txt` in each bot directory,
5. STORAGE_DIR env value must `storage` to make it work fine,
6. Run the bot by `python3 dc.py` or `python3 tg.py`.

Both tested. But, I want to make sure if it's already fixed and stable.
So, don't forget to test it too, thanks.

## Changelog

v1 -> v2:
- Remove re.sub() in the fix_utf8_char() when set unescape to True
- Add if statement logic to ignore text/html in the extract_body()

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---

Muhammad Rizki (8):
  discord: Add send_text_mail_interaction()
  discord: Add send_patch_mail_interaction()
  discord: Add get lore mail slash command
  atom: Small change for fix_utf8_char()
  atom: Improve remove_patch()
  atom: add manage_payload()
  telegram: Fix get lore command
  atom: Improve extract_body()

 daemon/atom/utils.py                          | 32 +++++++++++----
 daemon/dscord/gnuweeb/client.py               | 23 +++++++++++
 .../plugins/slash_commands/__init__.py        |  2 +
 .../plugins/slash_commands/get_lore_mail.py   | 39 +++++++++++++++++++
 daemon/dscord/mailer/listener.py              |  4 +-
 daemon/telegram/mailer/listener.py            |  3 +-
 .../packages/plugins/commands/scrape.py       |  5 +--
 7 files changed, 93 insertions(+), 15 deletions(-)
 create mode 100644 daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py


base-commit: d9b20dab81202b93f48d5365ad680796e5839d80
--
Muhammad Rizki

[PATCH v2 1/8] discord: Add send_text_mail_interaction()

[Muhammad Rizki]

Add send_text_mail_interaction() for the `/atom get {url}` to get the
specific lore email message from an URL in future use.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/dscord/gnuweeb/client.py | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/daemon/dscord/gnuweeb/client.py b/daemon/dscord/gnuweeb/client.py
index 03a1b8c..15a6eee 100644
--- a/daemon/dscord/gnuweeb/client.py
+++ b/daemon/dscord/gnuweeb/client.py
@@ -4,6 +4,7 @@
 #
 
 import discord
+from discord import Interaction
 from discord.ext import commands
 from discord import Intents
 from dscord.config import ACTIVITY_NAME
@@ -76,3 +77,11 @@ class GWClient(commands.Bot):
 
 		utils.remove_patch(tmp)
 		return m
+
+
+	async def send_text_mail_interaction(self, i: "Interaction",
+					text: str, url: str = None):
+		return await i.response.send_message(
+			content=text,
+			view=models.FullMessageBtn(url)
+		)
-- 
Muhammad Rizki

[PATCH v2 2/8] discord: Add send_patch_mail_interaction()

[Muhammad Rizki]

This function is just the same as send_text_mail_interaction(), the
different between them is send_patch_mail_interaction() for sending a
lore email message with patch file attachment.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/dscord/gnuweeb/client.py | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/daemon/dscord/gnuweeb/client.py b/daemon/dscord/gnuweeb/client.py
index 15a6eee..b921f7d 100644
--- a/daemon/dscord/gnuweeb/client.py
+++ b/daemon/dscord/gnuweeb/client.py
@@ -85,3 +85,17 @@ class GWClient(commands.Bot):
 			content=text,
 			view=models.FullMessageBtn(url)
 		)
+
+
+	async def send_patch_mail_interaction(self, mail, i: "Interaction",
+						text: str, url: str = None):
+		tmp, doc, caption, url = utils.prepare_patch(
+			mail, text, url, "discord"
+		)
+		m = await i.response.send_message(
+			content=caption,
+			file=discord.File(doc),
+			view=models.FullMessageBtn(url)
+		)
+		utils.remove_patch(tmp)
+		return m
-- 
Muhammad Rizki

[PATCH v2 3/8] discord: Add get lore mail slash command

[Muhammad Rizki]

Add a `/lore {url}` slash command to get the specific lore email message
from the raw atom URL.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 .../plugins/slash_commands/__init__.py        |  2 +
 .../plugins/slash_commands/get_lore_mail.py   | 39 +++++++++++++++++++
 2 files changed, 41 insertions(+)
 create mode 100644 daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py

diff --git a/daemon/dscord/gnuweeb/plugins/slash_commands/__init__.py b/daemon/dscord/gnuweeb/plugins/slash_commands/__init__.py
index a6d913c..126af45 100644
--- a/daemon/dscord/gnuweeb/plugins/slash_commands/__init__.py
+++ b/daemon/dscord/gnuweeb/plugins/slash_commands/__init__.py
@@ -5,9 +5,11 @@
 
 from .manage_atom import ManageAtomSC
 from .manage_broadcast import ManageBroadcastSC
+from .get_lore_mail import GetLoreSC
 
 
 class SlashCommands(
+	GetLoreSC,
 	ManageAtomSC,
 	ManageBroadcastSC
 ): pass
diff --git a/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py b/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py
new file mode 100644
index 0000000..2d55d16
--- /dev/null
+++ b/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py
@@ -0,0 +1,39 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Copyright (C) 2022  Muhammad Rizki <kiizuha@gnuweeb.org>
+#
+
+import asyncio
+import discord
+from discord.ext import commands
+from discord import Interaction
+from discord import app_commands
+
+from atom import utils
+from atom import Scraper
+
+
+class GetLoreSC(commands.Cog):
+	def __init__(self, bot) -> None:
+		self.bot = bot
+
+
+	@app_commands.command(
+		name="lore",
+		description="Get lore email from raw email URL."
+	)
+	@app_commands.describe(url="Raw lore email URL")
+	async def get_lore(self, i: "Interaction", url: str):
+		s = Scraper()
+		mail = await s.get_email_from_url(url)
+		text, _, is_patch = utils.create_template(mail, "discord")
+
+		if is_patch:
+			m = await self.bot.send_patch_mail_interaction(
+				mail=mail, i=i, text=text, url=url
+			)
+		else:
+			text = "#ml\n" + text
+			m = await self.bot.send_text_mail_interaction(
+				i=i, text=text, url=url
+			)
-- 
Muhammad Rizki

[PATCH v2 4/8] atom: Small change for fix_utf8_char()

[Muhammad Rizki]

Change the parameter to unescape with boolean type and change from
html.escape to html.unescape for the Discord bot.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/atom/utils.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py
index a30d5cb..c95612e 100644
--- a/daemon/atom/utils.py
+++ b/daemon/atom/utils.py
@@ -206,7 +206,7 @@ def create_template(thread: Message, platform: str, to=None, cc=None):
 		if len(ret) >= substr:
 			ret = ret[:substr] + "..."
 
-		ret = fix_utf8_char(ret, platform == "telegram")
+		ret = fix_utf8_char(ret, platform == "discord")
 		ret += border
 
 	return ret, files, is_patch
@@ -242,10 +242,10 @@ def remove_patch(tmp):
 	shutil.rmtree(tmp)
 
 
-def fix_utf8_char(text: str, html_escape: bool = True):
+def fix_utf8_char(text: str, unescape: bool = True):
 	t = text.rstrip().replace("�"," ")
-	if html_escape:
-		t = html.escape(html.escape(text))
+	if unescape:
+		t = html.unescape(html.unescape(text))
 	return t
 
 
-- 
Muhammad Rizki

Re: [PATCH v2 4/8] atom: Small change for fix_utf8_char()

[Ammar Faizi]

On 10/20/22 3:38 PM, Muhammad Rizki wrote:
> Change the parameter to unescape with boolean type and change from
> html.escape to html.unescape for the Discord bot.
> 
> Signed-off-by: Muhammad Rizki<kiizuha@gnuweeb.org>
> ---
>   daemon/atom/utils.py | 8 ++++----
>   1 file changed, 4 insertions(+), 4 deletions(-)
> 
> diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py
> index a30d5cb..c95612e 100644
> --- a/daemon/atom/utils.py
> +++ b/daemon/atom/utils.py
> @@ -206,7 +206,7 @@ def create_template(thread: Message, platform: str, to=None, cc=None):
>   		if len(ret) >= substr:
>   			ret = ret[:substr] + "..."
>   
> -		ret = fix_utf8_char(ret, platform == "telegram")
> +		ret = fix_utf8_char(ret, platform == "discord")
>   		ret += border
>   
>   	return ret, files, is_patch
> @@ -242,10 +242,10 @@ def remove_patch(tmp):
>   	shutil.rmtree(tmp)
>   
>   
> -def fix_utf8_char(text: str, html_escape: bool = True):
> +def fix_utf8_char(text: str, unescape: bool = True):
>   	t = text.rstrip().replace("�"," ")
> -	if html_escape:
> -		t = html.escape(html.escape(text))
> +	if unescape:
> +		t = html.unescape(html.unescape(text))
>   	return t

This is broken.

I tested this series, but didn't have time to bisect it. After I
managed to bisect the issue, I found that the issue is introduced
by this patch.

See before and after below, anything inside the angle brackets is
gone. The sample is using this email:

https://lore.kernel.org/io-uring/f905c8cb-702f-6b2c-8954-1a736feb1ee7@kernel.dk/raw

-------------
Before this patch:

#ml
From: Jens Axboe <axboe@kernel.dk>
To: Ammar Faizi <ammarfaizi2@gnuweeb.org>
To: Dylan Yudaken <dylany@meta.com>
Cc: Pavel Begunkov <asml.silence@gmail.com>
Cc: GNU/Weeb Mailing List <gwml@vger.gnuweeb.org>
Cc: io-uring Mailing List <io-uring@vger.kernel.org>
Cc: Facebook Kernel Team <kernel-team@fb.com>
Cc: Dylan Yudaken <dylany@fb.com>

-------------
After this patch:

#ml
From: Jens Axboe
To: Ammar Faizi
To: Dylan Yudaken
Cc: Pavel Begunkov
Cc: GNU/Weeb Mailing List
Cc: io-uring Mailing List
Cc: Facebook Kernel Team
Cc: Dylan Yudaken

-- 
Ammar Faizi

Re: [PATCH v2 4/8] atom: Small change for fix_utf8_char()

[Muhammad Rizki]

On 21/10/2022 13.53, Ammar Faizi wrote:
> On 10/20/22 3:38 PM, Muhammad Rizki wrote:
>> Change the parameter to unescape with boolean type and change from
>> html.escape to html.unescape for the Discord bot.
>>
>> Signed-off-by: Muhammad Rizki<kiizuha@gnuweeb.org>
>> ---
>>   daemon/atom/utils.py | 8 ++++----
>>   1 file changed, 4 insertions(+), 4 deletions(-)
>>
>> diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py
>> index a30d5cb..c95612e 100644
>> --- a/daemon/atom/utils.py
>> +++ b/daemon/atom/utils.py
>> @@ -206,7 +206,7 @@ def create_template(thread: Message, platform: 
>> str, to=None, cc=None):
>>           if len(ret) >= substr:
>>               ret = ret[:substr] + "..."
>> -        ret = fix_utf8_char(ret, platform == "telegram")
>> +        ret = fix_utf8_char(ret, platform == "discord")
>>           ret += border
>>       return ret, files, is_patch
>> @@ -242,10 +242,10 @@ def remove_patch(tmp):
>>       shutil.rmtree(tmp)
>> -def fix_utf8_char(text: str, html_escape: bool = True):
>> +def fix_utf8_char(text: str, unescape: bool = True):
>>       t = text.rstrip().replace("�"," ")
>> -    if html_escape:
>> -        t = html.escape(html.escape(text))
>> +    if unescape:
>> +        t = html.unescape(html.unescape(text))
>>       return t
> 
> This is broken.
> 
> I tested this series, but didn't have time to bisect it. After I
> managed to bisect the issue, I found that the issue is introduced
> by this patch.
> 
> See before and after below, anything inside the angle brackets is
> gone. The sample is using this email:
> 
> https://lore.kernel.org/io-uring/f905c8cb-702f-6b2c-8954-1a736feb1ee7@kernel.dk/raw
> 
> -------------
> Before this patch:
> 
> #ml
> From: Jens Axboe <axboe@kernel.dk>
> To: Ammar Faizi <ammarfaizi2@gnuweeb.org>
> To: Dylan Yudaken <dylany@meta.com>
> Cc: Pavel Begunkov <asml.silence@gmail.com>
> Cc: GNU/Weeb Mailing List <gwml@vger.gnuweeb.org>
> Cc: io-uring Mailing List <io-uring@vger.kernel.org>
> Cc: Facebook Kernel Team <kernel-team@fb.com>
> Cc: Dylan Yudaken <dylany@fb.com>
> 
> -------------
> After this patch:
> 
> #ml
> From: Jens Axboe
> To: Ammar Faizi
> To: Dylan Yudaken
> Cc: Pavel Begunkov
> Cc: GNU/Weeb Mailing List
> Cc: io-uring Mailing List
> Cc: Facebook Kernel Team
> Cc: Dylan Yudaken
> 

That's very strange

https://i.ibb.co/z57n0LJ/image.png

Re: [PATCH v2 4/8] atom: Small change for fix_utf8_char()

[Muhammad Rizki]

On 21/10/2022 13.53, Ammar Faizi wrote:
> On 10/20/22 3:38 PM, Muhammad Rizki wrote:
>> Change the parameter to unescape with boolean type and change from
>> html.escape to html.unescape for the Discord bot.
>>
>> Signed-off-by: Muhammad Rizki<kiizuha@gnuweeb.org>
>> ---
>>   daemon/atom/utils.py | 8 ++++----
>>   1 file changed, 4 insertions(+), 4 deletions(-)
>>
>> diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py
>> index a30d5cb..c95612e 100644
>> --- a/daemon/atom/utils.py
>> +++ b/daemon/atom/utils.py
>> @@ -206,7 +206,7 @@ def create_template(thread: Message, platform: 
>> str, to=None, cc=None):
>>           if len(ret) >= substr:
>>               ret = ret[:substr] + "..."
>> -        ret = fix_utf8_char(ret, platform == "telegram")
>> +        ret = fix_utf8_char(ret, platform == "discord")
>>           ret += border
>>       return ret, files, is_patch
>> @@ -242,10 +242,10 @@ def remove_patch(tmp):
>>       shutil.rmtree(tmp)
>> -def fix_utf8_char(text: str, html_escape: bool = True):
>> +def fix_utf8_char(text: str, unescape: bool = True):
>>       t = text.rstrip().replace("�"," ")
>> -    if html_escape:
>> -        t = html.escape(html.escape(text))
>> +    if unescape:
>> +        t = html.unescape(html.unescape(text))
>>       return t
> 
> This is broken.
> 
> I tested this series, but didn't have time to bisect it. After I
> managed to bisect the issue, I found that the issue is introduced
> by this patch.
> 
> See before and after below, anything inside the angle brackets is
> gone. The sample is using this email:
> 
> https://lore.kernel.org/io-uring/f905c8cb-702f-6b2c-8954-1a736feb1ee7@kernel.dk/raw
> 
> -------------
> Before this patch:
> 
> #ml
> From: Jens Axboe <axboe@kernel.dk>
> To: Ammar Faizi <ammarfaizi2@gnuweeb.org>
> To: Dylan Yudaken <dylany@meta.com>
> Cc: Pavel Begunkov <asml.silence@gmail.com>
> Cc: GNU/Weeb Mailing List <gwml@vger.gnuweeb.org>
> Cc: io-uring Mailing List <io-uring@vger.kernel.org>
> Cc: Facebook Kernel Team <kernel-team@fb.com>
> Cc: Dylan Yudaken <dylany@fb.com>
> 
> -------------
> After this patch:
> 
> #ml
> From: Jens Axboe
> To: Ammar Faizi
> To: Dylan Yudaken
> Cc: Pavel Begunkov
> Cc: GNU/Weeb Mailing List
> Cc: io-uring Mailing List
> Cc: Facebook Kernel Team
> Cc: Dylan Yudaken
> 

What about this? can you explain what happened? Because I tested it and 
it has payload.

Re: [PATCH v2 4/8] atom: Small change for fix_utf8_char()

[Ammar Faizi]

On 10/21/22 3:25 PM, Muhammad Rizki wrote:
> What about this? can you explain what happened? Because I tested it and it has payload.

I didn't investigate this further. Can you print() the text of
that email before you send it to Telegram and paste it here?

-- 
Ammar Faizi

Re: [PATCH v2 4/8] atom: Small change for fix_utf8_char()

[Muhammad Rizki]

On 21/10/2022 15.27, Ammar Faizi wrote:
> On 10/21/22 3:25 PM, Muhammad Rizki wrote:
>> What about this? can you explain what happened? Because I tested it 
>> and it has payload.
> 
> I didn't investigate this further. Can you print() the text of
> that email before you send it to Telegram and paste it here?
> 

The text content that I'm printing out in the send_text_mail() in the 
telegram/packages/client.py is:

#ml
From: Jens Axboe <axboe@kernel.dk>
To: Ammar Faizi <ammarfaizi2@gnuweeb.org>
To: Dylan Yudaken <dylany@meta.com>
Cc: Pavel Begunkov <asml.silence@gmail.com>
Cc: GNU/Weeb Mailing List <gwml@vger.gnuweeb.org>
Cc: io-uring Mailing List <io-uring@vger.kernel.org>
Cc: Facebook Kernel Team <kernel-team@fb.com>
Cc: Dylan Yudaken <dylany@fb.com>
Date: Thu, 20 Oct 2022 05:52:08 -0700
Subject: Re: [PATCH liburing v1 1/3] liburing: Clean up 
`-Wshorten-64-to-32` warnings from clang

On 10/20/22 5:12 AM, Ammar Faizi wrote:
 > On 10/20/22 6:52 PM, Ammar Faizi wrote:
 >> From: Dylan Yudaken<dylany@fb.com>
 >>
 >> liburing has a couple of int shortening issues found by clang. Clean
 >> them all. This cleanup is particularly useful for build systems that
 >> include these files and run with that error enabled.
 >>
 >> 
Link:https://lore.kernel.org/io-uring/20221019145042.446477-1-dylany@meta.com
 >> Signed-off-by: Dylan Yudaken<dylany@fb.com>
 >> Co-authored-by: Ammar Faizi<ammarfaizi2@gnuweeb.org>
 >> Signed-off-by: Ammar Faizi<ammarfaizi2@gnuweeb.org>
 >
 > BTW, before it's too late. I think we should be consistent on the cast
 > style:
 >
 >    (type) a
 >
 > or
 >
 >    (type)a
 >
 > What do you think?

I tend to prefer (type) a, but sometimes if you have multiple casts
or the lines become too long, I'll do (type)a. I don't think it's
super important in terms of style, as it isn't that prevalent.

--
Jens Axboe
<code>------------------------------------------------------------------------</code>

Re: [PATCH v2 4/8] atom: Small change for fix_utf8_char()

[Ammar Faizi]

On 10/21/22 5:44 PM, Muhammad Rizki wrote:
> The text content that I'm printing out in the send_text_mail() in the telegram/packages/client.py is:

After reading the output you just sent, I think there are two
possibilities here:

1) You use the Telegram library wrong. Or ...

2) It's a bug in the Telegram library itself.

What do you think?

-- 
Ammar Faizi

Re: [PATCH v2 4/8] atom: Small change for fix_utf8_char()

[Muhammad Rizki]

On 21/10/2022 17.50, Ammar Faizi wrote:
> On 10/21/22 5:44 PM, Muhammad Rizki wrote:
>> The text content that I'm printing out in the send_text_mail() in the 
>> telegram/packages/client.py is:
> 
> After reading the output you just sent, I think there are two
> possibilities here:
> 
> 1) You use the Telegram library wrong. Or ...
> 
> 2) It's a bug in the Telegram library itself.
> 
> What do you think?
> 

Hmm, I didn't realize it's a bug. Because it is very weird when you 
tested it and the email payload is empty after sent to Telegram. But, 
mine doesn't.

Re: [PATCH v2 4/8] atom: Small change for fix_utf8_char()

[Ammar Faizi]

On 10/21/22 5:52 PM, Muhammad Rizki wrote:
> Hmm, I didn't realize it's a bug. Because it is very weird when you tested it and the email payload is empty after sent to Telegram. But, mine doesn't.

"doesn't" what?

-- 
Ammar Faizi

Re: [PATCH v2 4/8] atom: Small change for fix_utf8_char()

[Muhammad Rizki]

On 21/10/2022 17.54, Ammar Faizi wrote:
> On 10/21/22 5:52 PM, Muhammad Rizki wrote:
>> Hmm, I didn't realize it's a bug. Because it is very weird when you 
>> tested it and the email payload is empty after sent to Telegram. But, 
>> mine doesn't.
> 
> "doesn't" what?
> 

Doesn't have an empty email payload like you complaining.

Re: [PATCH v2 4/8] atom: Small change for fix_utf8_char()

[Ammar Faizi]

On 10/21/22 5:55 PM, Muhammad Rizki wrote:
> Doesn't have an empty email payload like you complaining.

I didn't complain about empty email payload. I complained
about everything inside the angle brackets are gone.

In this report, I didn't attach the email payload, but
header, you can see that the "after" is missing email
addresses.

-------------
Before this patch:

#ml
From: Jens Axboe <axboe@kernel.dk>
To: Ammar Faizi <ammarfaizi2@gnuweeb.org>
To: Dylan Yudaken <dylany@meta.com>
Cc: Pavel Begunkov <asml.silence@gmail.com>
Cc: GNU/Weeb Mailing List <gwml@vger.gnuweeb.org>
Cc: io-uring Mailing List <io-uring@vger.kernel.org>
Cc: Facebook Kernel Team <kernel-team@fb.com>
Cc: Dylan Yudaken <dylany@fb.com>

-------------
After this patch:

#ml
From: Jens Axboe
To: Ammar Faizi
To: Dylan Yudaken
Cc: Pavel Begunkov
Cc: GNU/Weeb Mailing List
Cc: io-uring Mailing List
Cc: Facebook Kernel Team
Cc: Dylan Yudaken

-- 
Ammar Faizi

Re: [PATCH v2 4/8] atom: Small change for fix_utf8_char()

[Muhammad Rizki]

On 21/10/2022 17.57, Ammar Faizi wrote:
> On 10/21/22 5:55 PM, Muhammad Rizki wrote:
>> Doesn't have an empty email payload like you complaining.
> 
> I didn't complain about empty email payload. I complained
> about everything inside the angle brackets are gone.

Oh, I see. Wait.

Re: [PATCH v2 4/8] atom: Small change for fix_utf8_char()

[Muhammad Rizki]

On 21/10/2022 17.54, Ammar Faizi wrote:
> On 10/21/22 5:52 PM, Muhammad Rizki wrote:
>> Hmm, I didn't realize it's a bug. Because it is very weird when you 
>> tested it and the email payload is empty after sent to Telegram. But, 
>> mine doesn't.
> 
> "doesn't" what?
> 

You can look into the Discord channel in the #test-bot, you can see it's 
not broken like you complaining when I tested it.

Jump to: 
https://discord.com/channels/845302963739033611/865963658726211604/1032970393113337947
Same sample: 
https://lore.kernel.org/io-uring/f905c8cb-702f-6b2c-8954-1a736feb1ee7@kernel.dk/raw

Re: [PATCH v2 4/8] atom: Small change for fix_utf8_char()

[Ammar Faizi]

On 10/21/22 5:57 PM, Muhammad Rizki wrote:
> You can look into the Discord channel in the #test-bot, you can see it's not broken like you complaining when I tested it.

Yes, it's not broken on Discord. But Telegram.

-- 
Ammar Faizi

Re: [PATCH v2 4/8] atom: Small change for fix_utf8_char()

[Muhammad Rizki]

On 21/10/2022 18.00, Ammar Faizi wrote:
> On 10/21/22 5:57 PM, Muhammad Rizki wrote:
>> You can look into the Discord channel in the #test-bot, you can see 
>> it's not broken like you complaining when I tested it.
> 
> Yes, it's not broken on Discord. But Telegram.
> 

I see what you complaining, my bad. Let's just use the old one of 
fix_utf8_char() here.

[PATCH v2 5/8] atom: Improve remove_patch()

[Muhammad Rizki]

Improvement on remove_patch(). So, after send all the email attachments
then remove them. On the previous patch, this function can't remove
them properly.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/atom/utils.py                                | 10 +++++++---
 daemon/dscord/mailer/listener.py                    |  4 ++--
 daemon/telegram/mailer/listener.py                  |  3 +--
 daemon/telegram/packages/plugins/commands/scrape.py |  3 +--
 4 files changed, 11 insertions(+), 9 deletions(-)

diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py
index c95612e..866c531 100644
--- a/daemon/atom/utils.py
+++ b/daemon/atom/utils.py
@@ -6,7 +6,7 @@
 
 from pyrogram.types import Chat, InlineKeyboardMarkup, InlineKeyboardButton
 from email.message import Message
-from typing import Dict
+from typing import Dict, Union
 from slugify import slugify
 import hashlib
 import uuid
@@ -238,8 +238,12 @@ def prepare_patch(mail, text, url, platform: str):
 	return tmp, file, caption, url
 
 
-def remove_patch(tmp):
-	shutil.rmtree(tmp)
+def remove_patch(tmp: Union[str, list]):
+	if isinstance(tmp, str):
+		return shutil.rmtree(tmp)
+
+	for d,_ in tmp:
+		shutil.rmtree(d)
 
 
 def fix_utf8_char(text: str, unescape: bool = True):
diff --git a/daemon/dscord/mailer/listener.py b/daemon/dscord/mailer/listener.py
index a280a58..cc0a9f7 100644
--- a/daemon/dscord/mailer/listener.py
+++ b/daemon/dscord/mailer/listener.py
@@ -125,10 +125,10 @@ class Listener:
 
 		for d, f in files:
 			await m.reply(file=File(f"{d}/{f}"))
-			if files.index((d,f)) == len(files)-1:
-				utils.remove_patch(d)
 			await asyncio.sleep(1)
 
+		utils.remove_patch(files)
+
 		return True
 
 
diff --git a/daemon/telegram/mailer/listener.py b/daemon/telegram/mailer/listener.py
index 208aed0..08feddd 100644
--- a/daemon/telegram/mailer/listener.py
+++ b/daemon/telegram/mailer/listener.py
@@ -118,8 +118,7 @@ class Bot():
 			await m.reply_document(f"{d}/{f}", file_name=f)
 			await asyncio.sleep(1)
 
-		if files:
-			shutil.rmtree(str(files[0][0]))
+		utils.remove_patch(files)
 
 		return True
 
diff --git a/daemon/telegram/packages/plugins/commands/scrape.py b/daemon/telegram/packages/plugins/commands/scrape.py
index d4d10a9..52ddb0b 100644
--- a/daemon/telegram/packages/plugins/commands/scrape.py
+++ b/daemon/telegram/packages/plugins/commands/scrape.py
@@ -53,5 +53,4 @@ async def scrap_email(c: DaemonClient, m: Message):
 		await m.reply_document(f"{d}/{f}", file_name=f)
 		await asyncio.sleep(1)
 
-	if files:
-		shutil.rmtree(str(files[0][0]))
+	utils.remove_patch(files)
-- 
Muhammad Rizki

[PATCH v2 6/8] atom: add manage_payload()

[Muhammad Rizki]

Add manage_payload() to handle the email decoding to utf-8. This include
a non-UTF8 character and base64 decoding.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/atom/utils.py | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py
index 866c531..e8b9d80 100644
--- a/daemon/atom/utils.py
+++ b/daemon/atom/utils.py
@@ -8,6 +8,7 @@ from pyrogram.types import Chat, InlineKeyboardMarkup, InlineKeyboardButton
 from email.message import Message
 from typing import Dict, Union
 from slugify import slugify
+from base64 import b64decode
 import hashlib
 import uuid
 import os
@@ -136,7 +137,7 @@ def gen_temp(name: str, platform: str):
 
 def extract_body(thread: Message, platform: str):
 	if not thread.is_multipart():
-		p = thread.get_payload(decode=True).decode(errors='replace')
+		p = manage_payload(thread)
 
 		if platform == "discord":
 			p = quote_reply(p)
@@ -253,6 +254,14 @@ def fix_utf8_char(text: str, unescape: bool = True):
 	return t
 
 
+def manage_payload(payload: Message):
+	p = str(payload.get_payload())
+	tf_encode = payload.get("Content-Transfer-Encoding")
+	if tf_encode != "base64":
+		return p.encode().decode("utf-8", errors="replace")
+	return b64decode(p).decode("utf-8")
+
+
 EMAIL_MSG_ID_PATTERN = r"<([^\<\>]+)>"
 def extract_email_msg_id(msg_id):
 	ret = re.search(EMAIL_MSG_ID_PATTERN, msg_id)
-- 
Muhammad Rizki

[PATCH v2 7/8] telegram: Fix get lore command

[Muhammad Rizki]

Fix the `/lore {raw lore url}` to add "telegram" on the
create_template() platform parameter.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/telegram/packages/plugins/commands/scrape.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/daemon/telegram/packages/plugins/commands/scrape.py b/daemon/telegram/packages/plugins/commands/scrape.py
index 52ddb0b..860e993 100644
--- a/daemon/telegram/packages/plugins/commands/scrape.py
+++ b/daemon/telegram/packages/plugins/commands/scrape.py
@@ -37,7 +37,7 @@ async def scrap_email(c: DaemonClient, m: Message):
 
 	s = Scraper()
 	mail = await s.get_email_from_url(url)
-	text, files, is_patch = utils.create_template(mail)
+	text, files, is_patch = utils.create_template(mail, "telegram")
 
 	if is_patch:
 		m = await c.send_patch_email(
-- 
Muhammad Rizki

[PATCH v2 8/8] atom: Improve extract_body()

[Muhammad Rizki]

Add an if statement to ignore whenever the email payload containing a
text/html content-type header.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/atom/utils.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py
index e8b9d80..cdfa5a7 100644
--- a/daemon/atom/utils.py
+++ b/daemon/atom/utils.py
@@ -150,6 +150,9 @@ def extract_body(thread: Message, platform: str):
 		fname = p.get_filename()
 		payload = p.get_payload(decode=True)
 
+		if p.get("content-type").split(";")[0] == "text/html":
+			continue
+
 		if not payload:
 			continue
 
-- 
Muhammad Rizki

Re: [PATCH v2 0/8] Fix some bugs and add some features

[Ammar Faizi]

On 10/20/22 3:38 PM, Muhammad Rizki wrote:
> Both tested. But, I want to make sure if it's already fixed and stable.
> So, don't forget to test it too, thanks.

I'll give it a try tomorrow morning.

-- 
Ammar Faizi