[PATCH v3 00/17] RESEND: Fix, improvement and implement a bot logger.

[PATCH v3 00/17] RESEND: Fix, improvement and implement a bot logger.

[Muhammad Rizki]

Hi sir, it's been a long time.

This is a v3 revision of fix, improvement and implement a bot logger.
this series is to improve create_template() caption substr/trim,
prepare_patch() for the .patch file payload, remove unrelevant changelog
commit in the commit description, use html.escape() once instead of
twice, small change in the first commit to make it clearer.

This series contains fixes in the utility file and some improvement for
it. I also implement a bot logger to log an error to the terminal and
to the log file, if error may happen it will send the log file to the
log channel either for Discord or Telegram, the LOG_CHANNEL_ID is
declared in the config.py.example, copy the channel ID from the Discord
or Telegram channel and then paste to the config.py.example.

What's new?
- Implementation of the BotLogger to log error message.
- Add a catch_err() in the atom/utils.py.
- Add a send_log_file() in each bot client.py.
- Add a variable LOG_CHANNEL_ID to configure the log channel.

Fixes:
- Fix media caption too long error for media/attachment message
- Fix .patch file payload
- Fix TO/CC list when an email doesn't contain TO/CC headers
- Fix typo on _flood_exception()

There are 17 patches in this series:
- Patch 1 is to fix a typo on _flood_exception() for the Discord bot.
- Patch 2 is to Fix .patch file payload to have a full email payload.
- Patch 3 is to add a substr/trim caption on the create_template().
- Patch 4 is to fix an extract_list() when there is no receipients.
- Patch 5 is to use the old one method, use decode=True in get_payload()
- Patch 6 is to improve fix_utf8_char(), make sure to unescape the HTML.
- Patch 7 is to add a utility catch_err() function to get error message.
- Patch 8 is to just add an initial BotLogger() to use it in next commit
- Patch 9 is to use the created BotLogger() for the Telegram bot.
- Patch 10 is to add a Telegram LOG_CHANNEL_ID for the send_log_file().
- Patch 11 is to add a send_log_file() for the Telegram bot DaemonClient
- Patch 12 is to implement the log message and send log file for TG bot.
- Patch 13 is to add a Discord LOG_CHANNEL_ID for the send_log_file().
- Patch 14 is to use the created BotLogger() for the Discord bot.
- Patch 15 is to add a send_log_file() for the Discord bot GWClient().
- Patch 16 is to implement the log message and send log file for DC bot.
- Patch 17 is to remove unused imports.

How to set log channel ID for the Telegram bot:
1. Make a channel for log in the Telegram.
2. Copy the channel ID, if you unable to see the ID you can use API or
   bot who has the feature, or use a 64gram desktop.
3. Paste the copied ID into the config.py in the telegram directory.

How to set log channel ID for the Discord bot:
1. Make a channel for log in your Discord server.
2. Copy the channel ID by right click and copy, if you unable to get the
   ID you can go to your account settings, then enable the
   `Developer mode` or use bot who has the feature.
3. Paste the copied ID into the config.py in the dscord directory.

How to use:
1. Execute the db.sql file in the daemon directory.
2. Setup .env file, the example is there with suffix .example, this
   file name must remove the suffix name .example.
3. Set up the config.py in each bot directory, such as dscord and
   telegram. The example is there with suffix .example & the file name
   must remove suffix name .example.
4. Run `pip3 install -r requirements.txt` in each bot directory.
5. STORAGE_DIR env value must `storage` to make it work fine.
6. Run the bot by `python3 dc.py` or `python3 tg.py`.

** Tested and works, please give it a test, thanks! **

Oh, about the keeping both .env and config.py we can discuss in this
series. I was asking a question for that, but you let it ignore.

## Changelog

v2 -> v3:
  - Fix prepare_patch() from media caption too long error.
  - Fix prepare_patch() for the .patch file.
  - Improve fix_utf8_char() function, don't use escape() twice and
    unecessary unescape().

v1 -> v2:
  - Improvement on the create_template() for the substr/trim caption.
  - Fix the prepare_patch() to use the full email payload instead of the
    trimmed caption taken from the `text` parameter.
  - Remove unrelevant changelog in the commit description.
  - Small change in the first commit description to make it clearer to
    read.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---

Muhammad Rizki (17):
  discord: Fix typo on _flood_exception()
  fix: utils: Fix .patch file payload
  fix: utils: add a substr for the patch media caption
  fix: utils: Fix the extract_list() utility function
  utils: Back to use decode=True for the get_payload()
  utils: Improve fix_utf8_char()
  utils: Add catch_err() for the log message
  logger: Initial work for the bot logger for future use
  telegram: Use the created BotLogger() class
  telegram: Add variable LOG_CHANNEL_ID declaration
  telegram: Add send_log_file() in the DaemonClient()
  telegram: Implement the log message for catching errors
  discord: Add variable LOG_CHANNEL_ID declaration
  discord: Use the BotLogger() to the GWClient()
  discord: Add send_log_file in the GWClient()
  discord: Implement the catch erros and logs
  Remove some unused imports

 daemon/atom/utils.py                          | 64 ++++++++++++++-----
 daemon/dc.py                                  | 10 ++-
 daemon/dscord/config.py.example               |  4 ++
 daemon/dscord/gnuweeb/client.py               | 20 ++++--
 daemon/dscord/gnuweeb/filters.py              | 10 ++-
 .../dscord/gnuweeb/plugins/events/on_ready.py |  2 +-
 .../plugins/slash_commands/get_lore_mail.py   | 12 ++--
 daemon/dscord/mailer/listener.py              | 41 ++++++------
 daemon/logger/__init__.py                     |  1 +
 daemon/logger/log.py                          | 52 +++++++++++++++
 daemon/telegram/config.py.example             |  4 ++
 daemon/telegram/mailer/listener.py            | 44 ++++++++-----
 daemon/telegram/packages/client.py            | 21 +++++-
 daemon/telegram/packages/decorator.py         | 15 ++---
 .../packages/plugins/commands/scrape.py       | 10 ++-
 daemon/tg.py                                  |  5 ++
 16 files changed, 237 insertions(+), 78 deletions(-)
 create mode 100644 daemon/logger/__init__.py
 create mode 100644 daemon/logger/log.py


base-commit: 91e86e6d1004da4526ca5514fa009bc253d5c4a3
--
Muhammad Rizki

[PATCH v3 01/17] discord: Fix typo on _flood_exception()

[Muhammad Rizki]

Fix minor typo. This function is for the Discord bot, so fix the error
message from "Telegram" to "Discord".

v2:
    - Change commit description to make it clearer.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/dscord/gnuweeb/filters.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/daemon/dscord/gnuweeb/filters.py b/daemon/dscord/gnuweeb/filters.py
index eb54a8a..b994904 100644
--- a/daemon/dscord/gnuweeb/filters.py
+++ b/daemon/dscord/gnuweeb/filters.py
@@ -49,5 +49,5 @@ def wait_on_limit(func):
 
 async def _flood_exceptions(e: "discord.errors.RateLimited"):
 	wait = e.retry_after
-	print(f"[wait_on_limit]: Sleeping for {wait} seconds due to Telegram limit")
+	print(f"[wait_on_limit]: Sleeping for {wait} seconds due to Discord limit")
 	await asyncio.sleep(wait)
-- 
Muhammad Rizki

[PATCH v3 02/17] fix: utils: Fix .patch file payload

[Muhammad Rizki]

From the previous version 59d20af, the .patch file payload uses the
prepare_patch() `text` parameter that taken from the create_template()
`caption` returned which is the caption is already trimmed. With the
current fix, this should be use the full email payload instead so its
not bothering with the trimmed payload from the create_template().

Fixes: 59d20af ("daemon: Move prepare for patch and clean up patch functions")
Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/atom/utils.py | 18 ++++++++++++++++--
 1 file changed, 16 insertions(+), 2 deletions(-)

diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py
index 03453b4..8a6c1a8 100644
--- a/daemon/atom/utils.py
+++ b/daemon/atom/utils.py
@@ -213,7 +213,7 @@ def create_template(thread: Message, platform: Platform, to=None, cc=None):
 	return ret, files, is_patch
 
 
-def prepare_patch(mail, text, url, platform: Platform):
+def prepare_patch(mail: Message, text, url, platform: Platform):
 	tmp = gen_temp(url, platform)
 	fnm = str(mail.get("subject"))
 	sch = re.search(PATCH_PATTERN, fnm, re.IGNORECASE)
@@ -230,7 +230,21 @@ def prepare_patch(mail, text, url, platform: Platform):
 	cap = text.split("\n\n")[0]
 
 	with open(file, "wb") as f:
-		f.write(bytes(text, encoding="utf8"))
+		write_payload = bytes(f"{cap}\n\n".encode())
+
+		# sometimes an email PATCH is a multipart
+		# so, we must loop the payload first
+		# then, check if each payload is a PATCH payload
+		# or no, if it's a PATCH payload then write payload
+		# to the file.
+		if mail.is_multipart():
+			for m in mail.get_payload():
+				subject = mail.get("subject")
+				if not __is_patch(subject, str(m)):
+					continue
+				write_payload += m.get_payload(decode=True)
+
+			f.write(write_payload)
 
 	caption = "#patch #ml"
 	if platform is Platform.TELEGRAM:
-- 
Muhammad Rizki

Re: [PATCH v3 02/17] fix: utils: Fix .patch file payload

[Alviro Iskandar Setiawan]

On Tue, Dec 20, 2022 at 6:57 AM Muhammad Rizki <kiizuha@gnuweeb.org> wrote:
> From the previous version 59d20af, the .patch file payload uses the
> prepare_patch() `text` parameter that taken from the create_template()
> `caption` returned which is the caption is already trimmed. With the
> current fix, this should be use the full email payload instead so its
> not bothering with the trimmed payload from the create_template().
>
> Fixes: 59d20af ("daemon: Move prepare for patch and clean up patch functions")
> Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>

Referencing a commit must use at least 12 characters of the SHA-1 commit hash.

Re: [PATCH v3 02/17] fix: utils: Fix .patch file payload

[Kiizuha]

On 20/12/2022 07.15, Alviro Iskandar Setiawan wrote:
> On Tue, Dec 20, 2022 at 6:57 AM Muhammad Rizki <kiizuha@gnuweeb.org> wrote:
>>  From the previous version 59d20af, the .patch file payload uses the
>> prepare_patch() `text` parameter that taken from the create_template()
>> `caption` returned which is the caption is already trimmed. With the
>> current fix, this should be use the full email payload instead so its
>> not bothering with the trimmed payload from the create_template().
>>
>> Fixes: 59d20af ("daemon: Move prepare for patch and clean up patch functions")
>> Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
> 
> Referencing a commit must use at least 12 characters of the SHA-1 commit hash.

Ah yes, I forget.

[PATCH v3 03/17] fix: utils: add a substr for the patch media caption

[Muhammad Rizki]

Add a substr caption for the patch email to avoid media caption too
long.

Regarding this commit bb8855bf5e, it's forget to add a substr caption
method, which is it will get media caption too long error for both
Discord and Telegram bot.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/atom/utils.py | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py
index 8a6c1a8..0ba649a 100644
--- a/daemon/atom/utils.py
+++ b/daemon/atom/utils.py
@@ -246,9 +246,20 @@ def prepare_patch(mail: Message, text, url, platform: Platform):
 
 			f.write(write_payload)
 
-	caption = "#patch #ml"
+	caption = f"#patch #ml\n{cap}"
 	if platform is Platform.TELEGRAM:
-		caption += fix_utf8_char("\n" + cap, True)
+		# Telegram media caption is limit to 1024
+		# set limit to 1021, because we will add "..."
+		if len(caption) >= 1024:
+			caption = caption[:1021] + "..."
+
+		fixed = fix_utf8_char(caption, html_escape=True)
+		return tmp, file, fixed, url
+
+	# Discord attachment caption limit about 1998 or 2000
+	# set limit to 1995, because we will add "..."
+	if len(caption) >= 1998:
+		caption = caption[:1995] + "..."
 
 	return tmp, file, caption, url
 
-- 
Muhammad Rizki

Re: [PATCH v3 03/17] fix: utils: add a substr for the patch media caption

[Alviro Iskandar Setiawan]

On Tue, Dec 20, 2022 at 6:57 AM Muhammad Rizki <kiizuha@gnuweeb.org> wrote:
> Add a substr caption for the patch email to avoid media caption too
> long.
>
> Regarding this commit bb8855bf5e, it's forget to add a substr caption
> method, which is it will get media caption too long error for both
> Discord and Telegram bot.
>
> Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>

What about changing the commit message to be something like this?

Commit bb8855bf5e3 ("Fix the storage management after the refactor was
happened") did not care with the caption length when sending an
attachment. This leads to error when the caption length hits the
limit. Add a substr() call to cut the caption if it's too long to
avoid such an error.

P.S.: <sic> at the "was happened" part.

Re: [PATCH v3 03/17] fix: utils: add a substr for the patch media caption

[Kiizuha]

On 20/12/2022 07.29, Alviro Iskandar Setiawan wrote:
> On Tue, Dec 20, 2022 at 6:57 AM Muhammad Rizki <kiizuha@gnuweeb.org> wrote:
>> Add a substr caption for the patch email to avoid media caption too
>> long.
>>
>> Regarding this commit bb8855bf5e, it's forget to add a substr caption
>> method, which is it will get media caption too long error for both
>> Discord and Telegram bot.
>>
>> Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
> 
> What about changing the commit message to be something like this?
> 
> Commit bb8855bf5e3 ("Fix the storage management after the refactor was
> happened") did not care with the caption length when sending an
> attachment. This leads to error when the caption length hits the
> limit. Add a substr() call to cut the caption if it's too long to
> avoid such an error.

I will change it in v4, thanks.

> 
> P.S.: <sic> at the "was happened" part.

Sorry, IDK what this means.

Re: [PATCH v3 03/17] fix: utils: add a substr for the patch media caption

[Alviro Iskandar Setiawan]

On Tue, Dec 20, 2022 at 9:00 AM Kiizuha <riskimuhammmad1@gmail.com> wrote:
> Sorry, IDK what this means.

https://grammar.yourdictionary.com/style-usage/what-does-sic-mean-using-it-correctly

[PATCH v3 04/17] fix: utils: Fix the extract_list() utility function

[Muhammad Rizki]

Fix the extract_list() when the `payload.get()` header function is not
retrieving the header value instead of throwing error.

Simple error log says:
AttributeError: 'Group' object has no attribute 'local_part'

Source problem: https://s.id/1n1C6 (shorted lore email URL)

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/atom/utils.py | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py
index 0ba649a..c5f6a00 100644
--- a/daemon/atom/utils.py
+++ b/daemon/atom/utils.py
@@ -86,9 +86,12 @@ def __extract_list(ss):
 	return ret
 
 
-def extract_list(key: str, content: Dict[str, str]):
-	people = content.get(key.lower())
-	if not people:
+def extract_list(key: str, thread: Message):
+	try:
+		people = thread.get(key.lower())
+		if not people:
+			return []
+	except:
 		return []
 	return __extract_list(people)
 
-- 
Muhammad Rizki

Re: [PATCH v3 04/17] fix: utils: Fix the extract_list() utility function

[Alviro Iskandar Setiawan]

On Tue, Dec 20, 2022 at 6:57 AM Muhammad Rizki <kiizuha@gnuweeb.org> wrote:
> +def extract_list(key: str, thread: Message):
> +       try:
> +               people = thread.get(key.lower())
> +               if not people:
> +                       return []
> +       except:
>                 return []
>         return __extract_list(people)

Why does this have to be a "try and except"?
And not something like:

        k = key.lower()
        if k in thread:
                return __extract_list(people)
        return []

Re: [PATCH v3 04/17] fix: utils: Fix the extract_list() utility function

[Ammar Faizi]

On 12/20/22 7:33 AM, Alviro Iskandar Setiawan wrote:
> On Tue, Dec 20, 2022 at 6:57 AM Muhammad Rizki <kiizuha@gnuweeb.org> wrote:
>> +def extract_list(key: str, thread: Message):
>> +       try:
>> +               people = thread.get(key.lower())
>> +               if not people:
>> +                       return []
>> +       except:
>>                  return []
>>          return __extract_list(people)
> 
> Why does this have to be a "try and except"?
> And not something like:
> 
>          k = key.lower()
>          if k in thread:
>                  return __extract_list(people)
>          return []

I don't know much about Python, but doesn't that only
work for dict? Or it can use `in` operator too?

`thread` is an instance of Message class object here.

-- 
Ammar Faizi

Re: [PATCH v3 04/17] fix: utils: Fix the extract_list() utility function

[Alviro Iskandar Setiawan]

On Tue, Dec 20, 2022 at 7:45 AM Ammar Faizi <ammarfaizi2@gnuweeb.org> wrote:
> I don't know much about Python, but doesn't that only
> work for dict? Or it can use `in` operator too?
>
> `thread` is an instance of Message class object here.

oh ic ic, it doesn't work for the normal class, for ex:

  class X():
          a = 1
          b = 2

  x = X()
  print("a" in x)

this error:
TypeError: argument of type 'X' is not iterable

if the Message object is iterable, maybe it can use the "in" operator
just like dict and array, but I don't know. Rizki?

Re: [PATCH v3 04/17] fix: utils: Fix the extract_list() utility function

[Kiizuha]

On 20/12/2022 08.15, Alviro Iskandar Setiawan wrote:
> On Tue, Dec 20, 2022 at 7:45 AM Ammar Faizi <ammarfaizi2@gnuweeb.org> wrote:
>> I don't know much about Python, but doesn't that only
>> work for dict? Or it can use `in` operator too?
>>
>> `thread` is an instance of Message class object here.
> 
> oh ic ic, it doesn't work for the normal class, for ex:
> 
>    class X():
>            a = 1
>            b = 2
> 
>    x = X()
>    print("a" in x)
> 
> this error:
> TypeError: argument of type 'X' is not iterable
> 
> if the Message object is iterable, maybe it can use the "in" operator
> just like dict and array, but I don't know. Rizki?


Like in the library code, the Message class has a __str__(self) method.

	def __str__(self):
         	"""Return the entire formatted message as a string.
         	"""
         	return self.as_string()

If we use it like `str(thread)` it will return a full email content.

Re: [PATCH v3 04/17] fix: utils: Fix the extract_list() utility function

[Kiizuha]

On 20/12/2022 07.45, Ammar Faizi wrote:
> On 12/20/22 7:33 AM, Alviro Iskandar Setiawan wrote:
>> On Tue, Dec 20, 2022 at 6:57 AM Muhammad Rizki <kiizuha@gnuweeb.org> 
>> wrote:
>>> +def extract_list(key: str, thread: Message):
>>> +       try:
>>> +               people = thread.get(key.lower())
>>> +               if not people:
>>> +                       return []
>>> +       except:
>>>                  return []
>>>          return __extract_list(people)
>>
>> Why does this have to be a "try and except"?
>> And not something like:
>>
>>          k = key.lower()
>>          if k in thread:
>>                  return __extract_list(people)
>>          return []
> 
> I don't know much about Python, but doesn't that only
> work for dict? Or it can use `in` operator too?

Message class has its own `get()` function.

> 
> `thread` is an instance of Message class object here.
> 

That's right, but if you use it like `str(thread)` it will return a full 
email content like email headers and email body.

Re: [PATCH v3 04/17] fix: utils: Fix the extract_list() utility function

[Kiizuha]

On 20/12/2022 07.33, Alviro Iskandar Setiawan wrote:
> On Tue, Dec 20, 2022 at 6:57 AM Muhammad Rizki <kiizuha@gnuweeb.org> wrote:
>> +def extract_list(key: str, thread: Message):
>> +       try:
>> +               people = thread.get(key.lower())
>> +               if not people:
>> +                       return []
>> +       except:
>>                  return []
>>          return __extract_list(people)
> 
> Why does this have to be a "try and except"?
> And not something like:
> 
>          k = key.lower()
>          if k in thread:
>                  return __extract_list(people)
>          return []


It supposed to be just like this:

	people = thread.get(key.lower())
	if not people:
		return []
	return __extract_list(people)



But, the library is throwing an error if an email is not adding a To 
recipients.

Logs:

Traceback (most recent call last):
   File 
"D:\Riski\Coding\Github\rushkii\lore-daemon\daemon\atom\utils.py", line 
90, in extract_list
     people = thread.get(key.lower())
   File "C:\Program Files\Python38\lib\email\message.py", line 471, in get
     return self.policy.header_fetch_parse(k, v)
   File "C:\Program Files\Python38\lib\email\policy.py", line 163, in 
header_fetch_parse
     return self.header_factory(name, value)
   File "C:\Program Files\Python38\lib\email\headerregistry.py", line 
602, in __call__
     return self[name](name, value)
   File "C:\Program Files\Python38\lib\email\headerregistry.py", line 
197, in __new__
     cls.parse(value, kwds)
   File "C:\Program Files\Python38\lib\email\headerregistry.py", line 
344, in parse
     [Address(mb.display_name or '',
   File "C:\Program Files\Python38\lib\email\headerregistry.py", line 
345, in <listcomp>
     mb.local_part or '',
AttributeError: 'Group' object has no attribute 'local_part'



But, I know what causing this error right now. This error is the problem 
from using the email policy to default.

Refer: 
https://github.com/ammarfaizi2/lore-daemon/blob/91e86e6d1004da4526ca5514fa009bc253d5c4a3/daemon/atom/scraper.py#L61


I forget why I'm using `policy=email.policy.default` but I know it has a 
reason! What do you think? Should we just remove 
`policy=email.policy.default` or just keeping it and use the try/except 
in extract_list()?

Re: [PATCH v3 04/17] fix: utils: Fix the extract_list() utility function

[Alviro Iskandar Setiawan]

On Tue, Dec 20, 2022 at 8:52 AM Kiizuha <riskimuhammmad1@gmail.com> wrote:
> I forget why I'm using `policy=email.policy.default` but I know it has a
> reason! What do you think? Should we just remove
> `policy=email.policy.default` or just keeping it and use the try/except
> in extract_list()?

I just didn't understand why we should use try and except, that's why
I asked a question, now the reason is clear. Just leave it as it is.

[PATCH v3 05/17] utils: Back to use decode=True for the get_payload()

[Muhammad Rizki]

Because at the previous version still not solving the error, this
version is use decode=True for the get_payload() in the
get_decoded_payload() just for the base64, quoted-printable, and binary
transfer encoding only.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/atom/utils.py | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)

diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py
index c5f6a00..127b0cc 100644
--- a/daemon/atom/utils.py
+++ b/daemon/atom/utils.py
@@ -10,7 +10,7 @@ from pyrogram.types import Chat, InlineKeyboardMarkup, InlineKeyboardButton
 from email.message import Message
 from typing import Dict, Union
 from slugify import slugify
-from base64 import b64decode
+import traceback
 import hashlib
 import uuid
 import os
@@ -287,11 +287,9 @@ def get_decoded_payload(payload: Message):
 	tf_encode = payload.get("Content-Transfer-Encoding")
 	charset = payload.get_content_charset("utf-8")
 
-	if tf_encode == "base64":
-		return b64decode(p).decode(charset)
-	if tf_encode == "quoted-printable":
-		quobyte = quopri.decodestring(p.encode())
-		return quobyte.decode(charset)
+	if tf_encode in ["base64", "quoted-printable", "binary"]:
+		return payload.get_payload(decode=True) \
+			.decode(errors="replace")
 
 	return p.encode().decode(charset, errors="replace")
 
-- 
Muhammad Rizki

[PATCH v3 06/17] utils: Improve fix_utf8_char()

[Muhammad Rizki]

Change on the html.escape() to use it only once. From the past
issue bb8855bf, some email message doesn't escaped correctly, so I use
the html.escape() twice. Within the current version, this issue should
be fixed and can call the html.escape() just once.

Fixes: bb8855bf ("Fix the storage management after the refactor was happened")
Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/atom/utils.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py
index 127b0cc..4fa6bcd 100644
--- a/daemon/atom/utils.py
+++ b/daemon/atom/utils.py
@@ -278,7 +278,7 @@ def remove_patch(tmp: Union[str, list]):
 def fix_utf8_char(text: str, html_escape: bool = True):
 	t = text.rstrip().replace("�"," ")
 	if html_escape:
-		t = html.escape(html.escape(text))
+		return html.escape(t)
 	return t
 
 
-- 
Muhammad Rizki

[PATCH v3 07/17] utils: Add catch_err() for the log message

[Muhammad Rizki]

Add catch_err() for the logger message in the future use.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/atom/utils.py | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py
index 4fa6bcd..b9adc4f 100644
--- a/daemon/atom/utils.py
+++ b/daemon/atom/utils.py
@@ -294,6 +294,12 @@ def get_decoded_payload(payload: Message):
 	return p.encode().decode(charset, errors="replace")
 
 
+def catch_err():
+	return traceback.format_exc() \
+	.encode("unicode_escape") \
+	.decode("utf-8")
+
+
 EMAIL_MSG_ID_PATTERN = r"<([^\<\>]+)>"
 def extract_email_msg_id(msg_id):
 	ret = re.search(EMAIL_MSG_ID_PATTERN, msg_id)
-- 
Muhammad Rizki

[PATCH v3 08/17] logger: Initial work for the bot logger for future use

[Muhammad Rizki]

Add a BotLogger() class that inherit to Logger() class from the python
logging library.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/logger/__init__.py |  1 +
 daemon/logger/log.py      | 52 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 53 insertions(+)
 create mode 100644 daemon/logger/__init__.py
 create mode 100644 daemon/logger/log.py

diff --git a/daemon/logger/__init__.py b/daemon/logger/__init__.py
new file mode 100644
index 0000000..bb3145a
--- /dev/null
+++ b/daemon/logger/__init__.py
@@ -0,0 +1 @@
+from .log import BotLogger
diff --git a/daemon/logger/log.py b/daemon/logger/log.py
new file mode 100644
index 0000000..8382404
--- /dev/null
+++ b/daemon/logger/log.py
@@ -0,0 +1,52 @@
+import logging
+import os
+from enums import Platform
+
+
+class BotLogger(logging.Logger):
+
+	def __init__(
+		self,
+		platform: Platform = Platform.TELEGRAM,
+		level: int = logging.WARNING
+	) -> None:
+
+		self.platform = str(platform.value)
+		self.level = level
+
+		plat_rep = self.platform.replace("discord", "dscord")
+		self.file_name = f"{plat_rep}/" \
+			f"{os.getenv('STORAGE_DIR', 'storage')}/" \
+			f"{self.platform}.log"
+
+		self.file_fmt = "{" \
+			"'time': '%(asctime)s'," \
+			"'func': '%(name)s: %(funcName)s'," \
+			"'path': '%(pathname)s'," \
+			"'level': '%(levelname)s'," \
+			"'msg': '%(message)s'"\
+		"}"
+
+		self.stream_fmt = "%(asctime)s | " \
+			"%(name)s: %(funcName)s | " \
+			"%(levelname)s | " \
+			"%(message)s"
+
+
+	def init(self):
+		super().__init__(self.platform, self.level)
+
+		file_handler = logging.FileHandler(self.file_name)
+		stream_handler = logging.StreamHandler()
+
+		file_fmtr = logging.Formatter(self.file_fmt)
+		stream_fmtr = logging.Formatter(self.stream_fmt)
+
+		file_handler.setFormatter(file_fmtr)
+		stream_handler.setFormatter(stream_fmtr)
+
+		file_handler.setLevel(logging.WARNING)
+		self.setLevel(logging.INFO)
+
+		self.addHandler(file_handler)
+		self.addHandler(stream_handler)
-- 
Muhammad Rizki

[PATCH v3 09/17] telegram: Use the created BotLogger() class

[Muhammad Rizki]

Use the BotLogger() class for the Telegram DaemonClient() class. This
class is to make a log message to a file within the DaemonClient() class
of the `.logger` attribute which is initialized from the BotLogger()
class.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/telegram/packages/client.py | 6 +++++-
 daemon/tg.py                       | 5 +++++
 2 files changed, 10 insertions(+), 1 deletion(-)

diff --git a/daemon/telegram/packages/client.py b/daemon/telegram/packages/client.py
index c971ea1..eb282eb 100644
--- a/daemon/telegram/packages/client.py
+++ b/daemon/telegram/packages/client.py
@@ -10,16 +10,20 @@ from typing import Union
 from email.message import Message
 from atom import utils
 from enums import Platform
+from logger import BotLogger
 from telegram.database import DB
 from .decorator import handle_flood
 
 
 class DaemonClient(Client):
 	def __init__(self, name: str, api_id: int,
-		api_hash: str, conn, **kwargs):
+		api_hash: str, conn, logger: BotLogger,
+		**kwargs
+	):
 		super().__init__(name, api_id,
 				api_hash, **kwargs)
 		self.db = DB(conn)
+		self.logger = logger
 
 
 	@handle_flood
diff --git a/daemon/tg.py b/daemon/tg.py
index 5f8c21e..8382c29 100644
--- a/daemon/tg.py
+++ b/daemon/tg.py
@@ -11,17 +11,22 @@ from atom import Scraper
 from telegram.packages import DaemonClient
 from telegram.mailer import BotMutexes
 from telegram.mailer import Bot
+from logger import BotLogger
 import os
 
 
 def main():
 	load_dotenv("telegram.env")
 
+	logger = BotLogger()
+	logger.init()
+
 	client = DaemonClient(
 		"telegram/storage/EmailScraper",
 		api_id=int(os.getenv("API_ID")),
 		api_hash=os.getenv("API_HASH"),
 		bot_token=os.getenv("BOT_TOKEN"),
+		logger=logger,
 		conn=connector.connect(
 			host=os.getenv("DB_HOST"),
 			user=os.getenv("DB_USER"),
-- 
Muhammad Rizki

[PATCH v3 10/17] telegram: Add variable LOG_CHANNEL_ID declaration

[Muhammad Rizki]

Declare a variable LOG_CHANNEL_ID in the config.py.example for the
send_log_file() in the next commit.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/telegram/config.py.example | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/daemon/telegram/config.py.example b/daemon/telegram/config.py.example
index 3818d27..221e0c1 100644
--- a/daemon/telegram/config.py.example
+++ b/daemon/telegram/config.py.example
@@ -4,6 +4,10 @@ from pyrogram.types import Message
 # Insert Telegram admin id in the list below
 ADMINS = []
 
+# Insert Telegram channel ID to send the log file
+# type (int)
+LOG_CHANNEL_ID =
+
 
 async def adm_flt(_, __, m: Message):
 	return m.from_user.id in ADMINS
-- 
Muhammad Rizki

[PATCH v3 11/17] telegram: Add send_log_file() in the DaemonClient()

[Muhammad Rizki]

Add send_log_file() to the DaemonClient() class.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/telegram/packages/client.py | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/daemon/telegram/packages/client.py b/daemon/telegram/packages/client.py
index eb282eb..6c83a5a 100644
--- a/daemon/telegram/packages/client.py
+++ b/daemon/telegram/packages/client.py
@@ -11,6 +11,7 @@ from email.message import Message
 from atom import utils
 from enums import Platform
 from logger import BotLogger
+from telegram import config
 from telegram.database import DB
 from .decorator import handle_flood
 
@@ -26,6 +27,16 @@ class DaemonClient(Client):
 		self.logger = logger
 
 
+	@handle_flood
+	async def send_log_file(self, caption: str):
+		filename = self.logger.handlers[0].baseFilename
+		await self.send_document(
+			config.LOG_CHANNEL_ID,
+			filename,
+			caption=caption
+		)
+
+
 	@handle_flood
 	async def send_text_email(
 		self,
-- 
Muhammad Rizki

[PATCH v3 12/17] telegram: Implement the log message for catching errors

[Muhammad Rizki]

After created the BotLogger() class and initialized it to the
DaemonClient() attribute, implement it in the
telegram/mailer/listener.py to log it into the file if catching errors.

v2:
    - Remove unrelevant changelog

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/telegram/mailer/listener.py            | 44 ++++++++++++-------
 daemon/telegram/packages/client.py            |  4 +-
 daemon/telegram/packages/decorator.py         | 15 +++----
 .../packages/plugins/commands/scrape.py       | 10 ++++-
 4 files changed, 45 insertions(+), 28 deletions(-)

diff --git a/daemon/telegram/mailer/listener.py b/daemon/telegram/mailer/listener.py
index 46ccf93..6fb2e44 100644
--- a/daemon/telegram/mailer/listener.py
+++ b/daemon/telegram/mailer/listener.py
@@ -13,7 +13,6 @@ from atom import utils
 from enums import Platform
 import asyncio
 import re
-import traceback
 
 
 class BotMutexes():
@@ -29,6 +28,7 @@ class Bot():
 		self.scraper = scraper
 		self.mutexes = mutexes
 		self.db = client.db
+		self.logger = client.logger
 		self.isRunnerFixed = False
 
 
@@ -49,8 +49,8 @@ class Bot():
 		# TODO(ammarfaizi2):
 		# Ideally, we also want to log and report this situation.
 		#
-		print(f"Database error: {str(e)}")
-		print("Reconnecting to the database...")
+		self.logger.error(f"Database error: {str(e)}")
+		self.logger.info("Reconnecting to the database...")
 
 		#
 		# Don't do this too often to avoid reconnect burst.
@@ -63,14 +63,17 @@ class Bot():
 
 
 	async def __run(self):
-		print("[__run]: Running...")
+		self.logger.info("Running...")
 		try:
 			for url in self.db.get_atom_urls():
 				await self.__handle_atom_url(url)
 		except (OperationalError, DatabaseError) as e:
 			await self.handle_db_error(e)
 		except:
-			print(traceback.format_exc())
+			exc_str = utils.catch_err()
+			self.logger.warning(exc_str)
+			capt = "Unkown raw lore URL, see full details in the log file."
+			await self.client.send_log_file(capt)
 
 		if not self.isRunnerFixed:
 			self.isRunnerFixed = True
@@ -86,8 +89,13 @@ class Bot():
 	async def __handle_atom_url(self, url):
 		urls = await self.scraper.get_new_threads_urls(url)
 		for url in urls:
-			mail = await self.scraper.get_email_from_url(url)
-			await self.__handle_mail(url, mail)
+			try:
+				mail = await self.scraper.get_email_from_url(url)
+				await self.__handle_mail(url, mail)
+			except:
+				exc_str = utils.catch_err()
+				self.logger.warning(exc_str)
+				await self.client.send_log_file(url)
 
 
 	async def __handle_mail(self, url, mail):
@@ -104,23 +112,27 @@ class Bot():
 	# @__must_hold(self.mutexes.send_to_tg)
 	async def __send_mail(self, url, mail, tg_chat_id):
 		email_msg_id = utils.get_email_msg_id(mail)
+
 		if not email_msg_id:
-			#
-			# It doesn't have a Message-Id.
-			# A malformed email. Skip!
-			#
+			md = "email_msg_id not detected, skipping malformed email"
+			self.logger.debug(md)
 			return False
 
 		email_id = self.__mail_id_from_db(email_msg_id,
 							tg_chat_id)
 		if not email_id:
-			#
-			# Email has already been sent to Telegram.
-			# Skip!
-			#
+			md = f"Skipping {email_id} because has already been sent to Telegram"
+			self.logger.debug(md)
 			return False
 
-		text, files, is_patch = utils.create_template(mail, Platform.TELEGRAM)
+		try:
+			text, files, is_patch = utils.create_template(mail, Platform.TELEGRAM)
+		except:
+			exc_str = utils.catch_err()
+			self.logger.warning(exc_str)
+			await self.client.send_log_file(url)
+			return
+
 		reply_to = self.get_reply(mail, tg_chat_id)
 		url = str(re.sub(r"/raw$", "", url))
 
diff --git a/daemon/telegram/packages/client.py b/daemon/telegram/packages/client.py
index 6c83a5a..58195b9 100644
--- a/daemon/telegram/packages/client.py
+++ b/daemon/telegram/packages/client.py
@@ -46,7 +46,7 @@ class DaemonClient(Client):
 		url: str = None,
 		parse_mode: ParseMode = ParseMode.HTML
 	) -> Message:
-		print("[send_text_email]")
+		self.logger.debug("[send_text_email]")
 		return await self.send_message(
 			chat_id=chat_id,
 			text=text,
@@ -71,7 +71,7 @@ class DaemonClient(Client):
 		url: str = None,
 		parse_mode: ParseMode = ParseMode.HTML
 	) -> Message:
-		print("[send_patch_email]")
+		self.logger.debug("[send_patch_email]")
 		tmp, doc, caption, url = utils.prepare_patch(
 			mail, text, url, Platform.TELEGRAM
 		)
diff --git a/daemon/telegram/packages/decorator.py b/daemon/telegram/packages/decorator.py
index c7a5f02..153fa95 100644
--- a/daemon/telegram/packages/decorator.py
+++ b/daemon/telegram/packages/decorator.py
@@ -24,20 +24,19 @@ def handle_flood(func: Callable[[T], T]) -> Callable[[T], T]:
 			try:
 				return await func(*args)
 			except FloodWait as e:
-				#
-				# Aiee... we hit our limit.
-				# Let's slow down a bit.
-				#
-				_flood_exceptions(e)
-				print("[__handle_telegram_floodwait]: Woken up from flood wait...")
+				# Calling logger attr from the DaemonClient() class
+				logger = args[0].logger
+
+				_flood_exceptions(e, logger)
+				logger.info("Woken up from flood wait...")
 	return callback
 
 
-async def _flood_exceptions(e):
+async def _flood_exceptions(e, logger):
 	x = re.search(r"A wait of (\d+) seconds is required", str(e))
 	if not x:
 		raise e
 
 	n = int(x.group(1))
-	print(f"[____handle_telegram_floodwait]: Sleeping for {n} seconds due to Telegram limit")
+	logger.info(f"Sleeping for {n} seconds due to Telegram limit")
 	await asyncio.sleep(n)
diff --git a/daemon/telegram/packages/plugins/commands/scrape.py b/daemon/telegram/packages/plugins/commands/scrape.py
index 29cc8ad..f89727d 100644
--- a/daemon/telegram/packages/plugins/commands/scrape.py
+++ b/daemon/telegram/packages/plugins/commands/scrape.py
@@ -36,8 +36,14 @@ async def scrap_email(c: DaemonClient, m: Message):
 		return
 
 	s = Scraper()
-	mail = await s.get_email_from_url(url)
-	text, files, is_patch = utils.create_template(mail, Platform.TELEGRAM)
+
+	try:
+		mail = await s.get_email_from_url(url)
+		text, files, is_patch = utils.create_template(mail, Platform.TELEGRAM)
+	except:
+		exc_str = utils.catch_err()
+		c.logger.warning(exc_str)
+		await c.send_log_file(url)
 
 	if is_patch:
 		m = await c.send_patch_email(
-- 
Muhammad Rizki

[PATCH v3 13/17] discord: Add variable LOG_CHANNEL_ID declaration

[Muhammad Rizki]

Add LOG_CHANNEL_ID into the config.py.example to use it in the future
use to send the log file to its Discord channel logs using the
LOG_CHANNEL_ID.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/dscord/config.py.example | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/daemon/dscord/config.py.example b/daemon/dscord/config.py.example
index 6dd4313..6db826c 100644
--- a/daemon/dscord/config.py.example
+++ b/daemon/dscord/config.py.example
@@ -11,3 +11,7 @@ ADMIN_ROLE_ID =
 # The activity name like "Playing Genshin Impact"
 # you set the value as "Genshin Impact"
 ACTIVITY_NAME =
+
+# Insert Discord channel ID to send the log file
+# type (int)
+LOG_CHANNEL_ID =
-- 
Muhammad Rizki

[PATCH v3 14/17] discord: Use the BotLogger() to the GWClient()

[Muhammad Rizki]

Use the BotLogger() and initialized it to the GWClient() class attribute
`.logger` to use it in the future use for log a message into the file.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/dc.py                    | 10 ++++++++--
 daemon/dscord/gnuweeb/client.py |  6 ++++--
 2 files changed, 12 insertions(+), 4 deletions(-)

diff --git a/daemon/dc.py b/daemon/dc.py
index 5028e94..f9b57e5 100644
--- a/daemon/dc.py
+++ b/daemon/dc.py
@@ -11,6 +11,8 @@ from apscheduler.schedulers.asyncio import AsyncIOScheduler
 from dscord.gnuweeb import GWClient
 from dscord.mailer import Listener
 from dscord.mailer import Mutexes
+from enums.platform import Platform
+from logger import BotLogger
 from atom import Scraper
 
 
@@ -24,13 +26,17 @@ def main():
 		}
 	)
 
+	logger = BotLogger(Platform.DISCORD)
+	logger.init()
+
 	client = GWClient(
 		db_conn=connector.connect(
 			host=os.getenv("DB_HOST"),
 			user=os.getenv("DB_USER"),
 			password=os.getenv("DB_PASS"),
 			database=os.getenv("DB_NAME")
-		)
+		),
+		logger=logger
 	)
 
 	mailer = Listener(
@@ -39,8 +45,8 @@ def main():
 		scraper=Scraper(),
 		mutexes=Mutexes()
 	)
-	client.mailer = mailer
 
+	client.mailer = mailer
 	client.run(os.getenv("DISCORD_TOKEN"), log_handler=None)
 
 
diff --git a/daemon/dscord/gnuweeb/client.py b/daemon/dscord/gnuweeb/client.py
index 82858c5..42bd405 100644
--- a/daemon/dscord/gnuweeb/client.py
+++ b/daemon/dscord/gnuweeb/client.py
@@ -7,22 +7,24 @@ import discord
 from discord import Interaction
 from discord.ext import commands
 from discord import Intents
-from dscord.config import ACTIVITY_NAME
 from typing import Union
 
 from . import filters
 from . import models
 from atom import utils
 from enums import Platform
+from logger.log import BotLogger
+from dscord.config import ACTIVITY_NAME
 from dscord.database import DB
 
 
 class GWClient(commands.Bot):
-	def __init__(self, db_conn) -> None:
+	def __init__(self, db_conn, logger: BotLogger) -> None:
 		self.db = DB(db_conn)
 		intents = Intents.default()
 		intents.message_content = True
 		self.mailer = None
+		self.logger = logger
 		super().__init__(
 			command_prefix=["$", "."],
 			description="Just a bot for receiving lore emails.",
-- 
Muhammad Rizki

[PATCH v3 15/17] discord: Add send_log_file in the GWClient()

[Muhammad Rizki]

Add send_log_file() to send a log file into the Discord channel for logs
using the `config.LOG_CHANNEL_ID` to send it to the Discord channel.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/dscord/gnuweeb/client.py | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/daemon/dscord/gnuweeb/client.py b/daemon/dscord/gnuweeb/client.py
index 42bd405..1b260ea 100644
--- a/daemon/dscord/gnuweeb/client.py
+++ b/daemon/dscord/gnuweeb/client.py
@@ -14,7 +14,7 @@ from . import models
 from atom import utils
 from enums import Platform
 from logger.log import BotLogger
-from dscord.config import ACTIVITY_NAME
+from dscord.config import ACTIVITY_NAME, LOG_CHANNEL_ID
 from dscord.database import DB
 
 
@@ -40,6 +40,16 @@ class GWClient(commands.Bot):
 		)
 
 
+	@filters.wait_on_limit
+	async def send_log_file(self, caption: str):
+		filename = self.logger.handlers[0].baseFilename
+		channel = self.get_channel(LOG_CHANNEL_ID)
+		await channel.send(
+			content=caption,
+			file=discord.File(filename)
+		)
+
+
 	@filters.wait_on_limit
 	async def send_text_email(self, guild_id: int, chat_id: int, text: str,
 				reply_to: Union[int, None] = None, url: str = None):
-- 
Muhammad Rizki

[PATCH v3 16/17] discord: Implement the catch erros and logs

[Muhammad Rizki]

Implement the catch erros and log the message into the file then send
it into the Discord log channel.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/dscord/gnuweeb/client.py               |  4 +-
 daemon/dscord/gnuweeb/filters.py              | 10 +++--
 .../dscord/gnuweeb/plugins/events/on_ready.py |  2 +-
 .../plugins/slash_commands/get_lore_mail.py   | 10 ++++-
 daemon/dscord/mailer/listener.py              | 40 +++++++++++--------
 5 files changed, 41 insertions(+), 25 deletions(-)

diff --git a/daemon/dscord/gnuweeb/client.py b/daemon/dscord/gnuweeb/client.py
index 1b260ea..406c606 100644
--- a/daemon/dscord/gnuweeb/client.py
+++ b/daemon/dscord/gnuweeb/client.py
@@ -53,7 +53,7 @@ class GWClient(commands.Bot):
 	@filters.wait_on_limit
 	async def send_text_email(self, guild_id: int, chat_id: int, text: str,
 				reply_to: Union[int, None] = None, url: str = None):
-		print("[send_text_email]")
+		self.logger.debug("[send_text_email]")
 		channel = self.get_channel(chat_id)
 
 		return await channel.send(
@@ -70,7 +70,7 @@ class GWClient(commands.Bot):
 	@filters.wait_on_limit
 	async def send_patch_email(self, mail, guild_id: int, chat_id: int, text: str,
 				reply_to: Union[int, None] = None, url: str = None):
-		print("[send_patch_email]")
+		self.logger.debug("[send_patch_email]")
 		tmp, doc, caption, url = utils.prepare_patch(
 			mail, text, url, Platform.DISCORD
 		)
diff --git a/daemon/dscord/gnuweeb/filters.py b/daemon/dscord/gnuweeb/filters.py
index b994904..1fdb70d 100644
--- a/daemon/dscord/gnuweeb/filters.py
+++ b/daemon/dscord/gnuweeb/filters.py
@@ -15,6 +15,7 @@ from discord import Interaction
 
 # gnuweeb package import
 from dscord import config
+from logger import BotLogger
 
 
 def lore_admin(func):
@@ -42,12 +43,15 @@ def wait_on_limit(func):
 			try:
 				return await func(*args)
 			except discord.errors.RateLimited as e:
+				# Calling logger attr from the GWClient() class
+				logger = args[0].logger
+
 				_flood_exceptions(e)
-				print("[wait_on_limit]: Woken up from flood wait...")
+				logger.info("Woken up from flood wait...")
 	return callback
 
 
-async def _flood_exceptions(e: "discord.errors.RateLimited"):
+async def _flood_exceptions(e: "discord.errors.RateLimited", logger: BotLogger):
 	wait = e.retry_after
-	print(f"[wait_on_limit]: Sleeping for {wait} seconds due to Discord limit")
+	logger.info(f"Sleeping for {wait} seconds due to Discord limit")
 	await asyncio.sleep(wait)
diff --git a/daemon/dscord/gnuweeb/plugins/events/on_ready.py b/daemon/dscord/gnuweeb/plugins/events/on_ready.py
index 4f25764..e7f63cd 100644
--- a/daemon/dscord/gnuweeb/plugins/events/on_ready.py
+++ b/daemon/dscord/gnuweeb/plugins/events/on_ready.py
@@ -23,4 +23,4 @@ class OnReady(commands.Cog):
 		t += f"Send `{prefix}sync` message to the Discord channel "
 		t += "where the bot is running.\n"
 
-		print(t)
+		self.bot.logger.info(t)
diff --git a/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py b/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py
index 0c67b8c..e4c171c 100644
--- a/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py
+++ b/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py
@@ -26,8 +26,14 @@ class GetLoreSC(commands.Cog):
 	@app_commands.describe(url="Raw lore email URL")
 	async def get_lore(self, i: "Interaction", url: str):
 		s = Scraper()
-		mail = await s.get_email_from_url(url)
-		text, _, is_patch = utils.create_template(mail, Platform.DISCORD)
+
+		try:
+			mail = await s.get_email_from_url(url)
+			text, _, is_patch = utils.create_template(mail, Platform.DISCORD)
+		except:
+			exc_str = utils.catch_err()
+			self.bot.logger.warning(exc_str)
+			await self.bot.send_log_file(url)
 
 		if is_patch:
 			m = await self.bot.send_patch_mail_interaction(
diff --git a/daemon/dscord/mailer/listener.py b/daemon/dscord/mailer/listener.py
index d986fbd..1764ea2 100644
--- a/daemon/dscord/mailer/listener.py
+++ b/daemon/dscord/mailer/listener.py
@@ -35,6 +35,7 @@ class Listener:
 		self.scraper = scraper
 		self.mutexes = mutexes
 		self.db = client.db
+		self.logger = client.logger
 		self.isRunnerFixed = False
 		self.runner = None
 
@@ -44,18 +45,15 @@ class Listener:
 		# Execute __run() once to avoid high latency at
 		# initilization.
 		#
-		print("Initialize listener...\n")
+		self.logger.info("Initialize listener...\n")
 		self.sched.start()
 		self.runner = self.sched.add_job(func=self.__run)
 
 
 	async def __run(self):
-		print("[__run]: Running...")
+		self.logger.info("Running...")
 		for url in self.db.get_atom_urls():
-			try:
-				await self.__handle_atom_url(url)
-			except:
-				print(traceback.format_exc())
+			await self.__handle_atom_url(url)
 
 		if not self.isRunnerFixed:
 			self.isRunnerFixed = True
@@ -71,8 +69,13 @@ class Listener:
 	async def __handle_atom_url(self, url):
 		urls = await self.scraper.get_new_threads_urls(url)
 		for url in urls:
-			mail = await self.scraper.get_email_from_url(url)
-			await self.__handle_mail(url, mail)
+			try:
+				mail = await self.scraper.get_email_from_url(url)
+				await self.__handle_mail(url, mail)
+			except:
+				exc_str = utils.catch_err()
+				self.client.logger.warning(exc_str)
+				await self.client.send_log_file(url)
 
 
 	async def __handle_mail(self, url, mail):
@@ -91,10 +94,8 @@ class Listener:
 	async def __send_to_discord(self, url, mail, dc_guild_id, dc_chat_id):
 		email_msg_id = utils.get_email_msg_id(mail)
 		if not email_msg_id:
-			#
-			# It doesn't have a Message-Id.
-			# A malformed email. Skip!
-			#
+			md = "email_msg_id not detected, skipping malformed email"
+			self.logger.debug(md)
 			return False
 
 		email_id = self.__get_email_id_sent(
@@ -102,13 +103,18 @@ class Listener:
 			dc_chat_id=dc_chat_id
 		)
 		if not email_id:
-			#
-			# Email has already been sent to Discord.
-			# Skip!
-			#
+			md = f"Skipping {email_id} because has already been sent to Discord"
+			self.logger.debug(md)
 			return False
 
-		text, files, is_patch = utils.create_template(mail, Platform.DISCORD)
+		try:
+			text, files, is_patch = utils.create_template(mail, Platform.DISCORD)
+		except:
+			exc_str = utils.catch_err()
+			self.client.logger.warning(exc_str)
+			await self.client.send_log_file(url)
+			return
+
 		reply_to = self.get_discord_reply(mail, dc_chat_id)
 		url = str(re.sub(r"/raw$", "", url))
 
-- 
Muhammad Rizki

[PATCH v3 17/17] Remove some unused imports

[Muhammad Rizki]

Remove them while it is not used again, from the previous version, they
are not removed.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/atom/utils.py                                          | 3 +--
 daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py | 2 --
 daemon/dscord/mailer/listener.py                              | 1 -
 3 files changed, 1 insertion(+), 5 deletions(-)

diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py
index b9adc4f..8810f18 100644
--- a/daemon/atom/utils.py
+++ b/daemon/atom/utils.py
@@ -8,7 +8,7 @@ from enums import Platform
 
 from pyrogram.types import Chat, InlineKeyboardMarkup, InlineKeyboardButton
 from email.message import Message
-from typing import Dict, Union
+from typing import Union
 from slugify import slugify
 import traceback
 import hashlib
@@ -18,7 +18,6 @@ import re
 import shutil
 import httpx
 import html
-import quopri
 
 
 def get_email_msg_id(mail):
diff --git a/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py b/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py
index e4c171c..56330f3 100644
--- a/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py
+++ b/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py
@@ -3,8 +3,6 @@
 # Copyright (C) 2022  Muhammad Rizki <kiizuha@gnuweeb.org>
 #
 
-import asyncio
-import discord
 from discord.ext import commands
 from discord import Interaction
 from discord import app_commands
diff --git a/daemon/dscord/mailer/listener.py b/daemon/dscord/mailer/listener.py
index 1764ea2..24f0a6c 100644
--- a/daemon/dscord/mailer/listener.py
+++ b/daemon/dscord/mailer/listener.py
@@ -5,7 +5,6 @@
 #
 
 import asyncio
-import traceback
 import re
 from apscheduler.schedulers.asyncio import AsyncIOScheduler
 from discord import File
-- 
Muhammad Rizki