[PATCH v2 00/17] Fix, improvement and implement a bot logger.

[PATCH v2 00/17] Fix, improvement and implement a bot logger.

[Muhammad Rizki]

Hi sir.

This is a v2 revision of fix, improvement and implement a bot logger.
this revision is to improve create_template() caption substr/trim,
prepare_patch() for the .patch file payload, remove unrelevant changelog
commit in the commit description, use html.escape() once instead of
twice, small change in the first commit to make it clearer.

This series contains fixes in the utility file and some improvement for
it. I also implement a bot logger to log an error to the terminal and
to the log file, if error may happen it will send the log file to the
log channel either for Discord or Telegram, the LOG_CHANNEL_ID is
declared in the config.py.example, copy the channel ID from the Discord
or Telegram channel and then paste to the config.py.example.

There are 17 patches in this series:
- Patch 1 is to fix a typo on _flood_exception() for the Discord bot.
- Patch 2 is to fix a prepare_patch() to have a full email payload.
- Patch 3 is to add a substr/trim caption on the create_template().
- Patch 4 is to fix an extract_list() when there is no receipients.
- Patch 5 is to use the old one method, use decode=True in get_payload()
- Patch 6 is to improve fix_utf8_char(), make sure to unescape the HTML.
- Patch 7 is to add a utility catch_err() function to get error message.
- Patch 8 is to just add an initial BotLogger() to use it in next commit
- Patch 9 is to use the created BotLogger() for the Telegram bot.
- Patch 10 is to add a Telegram LOG_CHANNEL_ID for the send_log_file().
- Patch 11 is to add a send_log_file() for the Telegram bot DaemonClient
- Patch 12 is to implement the log message and send log file for TG bot.
- Patch 13 is to add a Discord LOG_CHANNEL_ID for the send_log_file().
- Patch 14 is to use the created BotLogger() for the Discord bot.
- Patch 15 is to add a send_log_file() for the Discord bot GWClient().
- Patch 16 is to implement the log message and send log file for DC bot.
- Patch 17 is to remove unused imports.

What's new?
- Implementation of the BotLogger to log error message.
- Add a catch_err() in the atom/utils.py.
- Add a send_log_file() in each bot client.py.
- Add a variable LOG_CHANNEL_ID to configure the log channel.

What's changed?
- Some utility functions are fixed and improved.
- Fix typo on _flood_exception()

How to set log channel ID for the Telegram bot:
1. Make a channel for log in the Telegram.
2. Copy the channel ID, if you unable to see the ID you can use API or
   bot who has the feature, or use a 64gram desktop.
3. Paste the copied ID into the config.py in the telegram directory.

How to set log channel ID for the Discord bot:
1. Make a channel for log in your Discord server.
2. Copy the channel ID by right click and copy, if you unable to get the
   ID you can go to your account settings, then enable the
   `Developer mode` or use bot who has the feature.
3. Paste the copied ID into the config.py in the dscord directory.

How to use:
1. Execute the db.sql file in the daemon directory.
2. Setup .env file, the example is there with suffix .example, this
   file name must remove the suffix name .example.
3. Set up the config.py in each bot directory, such as dscord and
   telegram. The example is there with suffix .example & the file name
   must remove suffix name .example.
4. Run `pip3 install -r requirements.txt` in each bot directory.
5. STORAGE_DIR env value must `storage` to make it work fine.
6. Run the bot by `python3 dc.py` or `python3 tg.py`.

Tested and works, please give it a test, thanks!

## Changelog

v1 -> v2:
  - Improvement on the create_template() for the substr/trim caption.
  - Fix the prepare_patch() to use the full email payload instead of the
    trimmed caption taken from the `text` parameter.
  - Remove unrelevant changelog in the commit description.
  - Small change in the first commit description to make it clearer to
    read.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---

Muhammad Rizki (17):
  discord: Fix typo on _flood_exception()
  fix: utils: Fix .patch file payload
  fix: utils: add a substr for the patch media caption
  fix: utils: Fix the extract_list() utility function
  utils: Back to use decode=True for the get_payload()
  utils: Improve fix_utf8_char()
  utils: Add catch_err() for the log message
  logger: Initial work for the bot logger for future use
  telegram: Use the created BotLogger() class
  telegram: Add variable LOG_CHANNEL_ID declaration
  telegram: Add send_log_file() in the DaemonClient()
  telegram: Implement the log message for catching errors
  discord: Add variable LOG_CHANNEL_ID declaration
  discord: Use the BotLogger() to the GWClient()
  discord: Add send_log_file in the GWClient()
  discord: Implement the catch erros and logs
  Remove some unused imports

 daemon/atom/utils.py                          | 53 +++++++++++--------
 daemon/dc.py                                  | 10 +++-
 daemon/dscord/config.py.example               |  4 ++
 daemon/dscord/gnuweeb/client.py               | 20 +++++--
 daemon/dscord/gnuweeb/filters.py              | 10 ++--
 .../dscord/gnuweeb/plugins/events/on_ready.py |  2 +-
 .../plugins/slash_commands/get_lore_mail.py   | 12 +++--
 daemon/dscord/mailer/listener.py              | 41 +++++++-------
 daemon/logger/__init__.py                     |  1 +
 daemon/logger/log.py                          | 52 ++++++++++++++++++
 daemon/telegram/config.py.example             |  4 ++
 daemon/telegram/mailer/listener.py            | 44 +++++++++------
 daemon/telegram/packages/client.py            | 21 ++++++--
 daemon/telegram/packages/decorator.py         | 15 +++---
 .../packages/plugins/commands/scrape.py       | 10 +++-
 daemon/tg.py                                  |  5 ++
 16 files changed, 222 insertions(+), 82 deletions(-)
 create mode 100644 daemon/logger/__init__.py
 create mode 100644 daemon/logger/log.py


base-commit: 91e86e6d1004da4526ca5514fa009bc253d5c4a3
--
Muhammad Rizki

[PATCH v2 01/17] discord: Fix typo on _flood_exception()

[Muhammad Rizki]

Fix minor typo. This function is for the Discord bot, so fix the error
message from "Telegram" to "Discord".

v2:
    - Change commit description to make it clearer.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/dscord/gnuweeb/filters.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/daemon/dscord/gnuweeb/filters.py b/daemon/dscord/gnuweeb/filters.py
index eb54a8a..b994904 100644
--- a/daemon/dscord/gnuweeb/filters.py
+++ b/daemon/dscord/gnuweeb/filters.py
@@ -49,5 +49,5 @@ def wait_on_limit(func):
 
 async def _flood_exceptions(e: "discord.errors.RateLimited"):
 	wait = e.retry_after
-	print(f"[wait_on_limit]: Sleeping for {wait} seconds due to Telegram limit")
+	print(f"[wait_on_limit]: Sleeping for {wait} seconds due to Discord limit")
 	await asyncio.sleep(wait)
-- 
Muhammad Rizki

[PATCH v2 02/17] fix: utils: Fix .patch file payload

[Muhammad Rizki]

From the previous version 59d20af, the .patch file payload uses the
prepare_patch() `text` parameter that taken from the create_template()
`caption` returned which is the caption is already trimmed. With the
current fix, this should be use the full email payload instead so its
not bothering with the trimmed payload from the create_template().

Fixes: 59d20af ("daemon: Move prepare for patch and clean up patch functions")
Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/atom/utils.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py
index 03453b4..9483d26 100644
--- a/daemon/atom/utils.py
+++ b/daemon/atom/utils.py
@@ -213,7 +213,7 @@ def create_template(thread: Message, platform: Platform, to=None, cc=None):
 	return ret, files, is_patch
 
 
-def prepare_patch(mail, text, url, platform: Platform):
+def prepare_patch(mail: Message, text, url, platform: Platform):
 	tmp = gen_temp(url, platform)
 	fnm = str(mail.get("subject"))
 	sch = re.search(PATCH_PATTERN, fnm, re.IGNORECASE)
@@ -230,7 +230,7 @@ def prepare_patch(mail, text, url, platform: Platform):
 	cap = text.split("\n\n")[0]
 
 	with open(file, "wb") as f:
-		f.write(bytes(text, encoding="utf8"))
+		f.write(mail.get_payload(decode=True))
 
 	caption = "#patch #ml"
 	if platform is Platform.TELEGRAM:
-- 
Muhammad Rizki

Re: [PATCH v2 02/17] fix: utils: Fix .patch file payload

[Ammar Faizi]

On 11/9/22 9:49 AM, Muhammad Rizki wrote:
> From the previous version 59d20af, the .patch file payload uses the
> prepare_patch() `text` parameter that taken from the create_template()
> `caption` returned which is the caption is already trimmed. With the
> current fix, this should be use the full email payload instead so its
> not bothering with the trimmed payload from the create_template().
> 
> Fixes: 59d20af ("daemon: Move prepare for patch and clean up patch functions")
> Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>

If you mention a commit hash, it has to be at least 12 characters.

-- 
Ammar Faizi

Re: [PATCH v2 02/17] fix: utils: Fix .patch file payload

[Muhammad Rizki]

On 09/11/2022 15.59, Ammar Faizi wrote:
> On 11/9/22 9:49 AM, Muhammad Rizki wrote:
>> From the previous version 59d20af, the .patch file payload uses the
>> prepare_patch() `text` parameter that taken from the create_template()
>> `caption` returned which is the caption is already trimmed. With the
>> current fix, this should be use the full email payload instead so its
>> not bothering with the trimmed payload from the create_template().
>>
>> Fixes: 59d20af ("daemon: Move prepare for patch and clean up patch 
>> functions")
>> Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
> 
> If you mention a commit hash, it has to be at least 12 characters.
> 

I will keep it mind, thanks for the reminder.

Re: [PATCH v2 02/17] fix: utils: Fix .patch file payload

[Muhammad Rizki]

On 09/11/2022 15.59, Ammar Faizi wrote:
> On 11/9/22 9:49 AM, Muhammad Rizki wrote:
>> From the previous version 59d20af, the .patch file payload uses the
>> prepare_patch() `text` parameter that taken from the create_template()
>> `caption` returned which is the caption is already trimmed. With the
>> current fix, this should be use the full email payload instead so its
>> not bothering with the trimmed payload from the create_template().
>>
>> Fixes: 59d20af ("daemon: Move prepare for patch and clean up patch 
>> functions")
>> Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
> 
> If you mention a commit hash, it has to be at least 12 characters.
> 

Anyway, any comment on another commit?

Re: [PATCH v2 02/17] fix: utils: Fix .patch file payload

[Ammar Faizi]

On 11/9/22 4:44 PM, Muhammad Rizki wrote:
> Anyway, any comment on another commit?

I didn't have time to review it. Take a break. Do something else.

-- 
Ammar Faizi

Re: [PATCH v2 02/17] fix: utils: Fix .patch file payload

[Muhammad Rizki]

On 09/11/2022 16.49, Ammar Faizi wrote:
> On 11/9/22 4:44 PM, Muhammad Rizki wrote:
>> Anyway, any comment on another commit?
> 
> I didn't have time to review it. Take a break. Do something else.
> 

Alright.

Re: [PATCH v2 02/17] fix: utils: Fix .patch file payload

[Ammar Faizi]

On 11/9/22 9:49 AM, Muhammad Rizki wrote:
>  From the previous version 59d20af, the .patch file payload uses the
> prepare_patch() `text` parameter that taken from the create_template()
> `caption` returned which is the caption is already trimmed. With the
> current fix, this should be use the full email payload instead so its
> not bothering with the trimmed payload from the create_template().

I don't understand what you are talking about. Please fix your wording.

-- 
Ammar Faizi

Re: [PATCH v2 02/17] fix: utils: Fix .patch file payload

[Muhammad Rizki]

On 20/11/2022 11.49, Ammar Faizi wrote:
> On 11/9/22 9:49 AM, Muhammad Rizki wrote:
>>  From the previous version 59d20af, the .patch file payload uses the
>> prepare_patch() `text` parameter that taken from the create_template()
>> `caption` returned which is the caption is already trimmed. With the
>> current fix, this should be use the full email payload instead so its
>> not bothering with the trimmed payload from the create_template().
> 
> I don't understand what you are talking about. Please fix your wording.
> 

In the old patch, the *.patch file contents are come from the 
prepare_patch()'s text argument, which is the `text` itself are taken 
from the `caption` returned from the create_template().

Code flow example:
     	caption, files, is_patch = create_template()
     	tmp, doc, caption, url = utils.prepare_patch(caption)
	# see mailer/listener.py in each bots for more details.

As I said above, the *.patch contents are using from the 
create_template()'s caption which is already trimmed to limit the 
caption for each bots. In this patch, I change it to use email payload 
instead. Is that clear?

[PATCH v2 03/17] fix: utils: add a substr for the patch media caption

[Muhammad Rizki]

Add a substr caption for the patch email to avoid MediaCaptionTooLong
for the Telegram bot.

Regarding to commit 125111a7, its forget to add a substring to trim the
media caption for the Telegram bot, because Telegram only allow
media caption to 1024 only, so trim it to 1020.

Fixes: 125111a7 ("daemon: Full refactor everything about scraper")
Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/atom/utils.py | 17 +++++++++++------
 1 file changed, 11 insertions(+), 6 deletions(-)

diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py
index 9483d26..2e85461 100644
--- a/daemon/atom/utils.py
+++ b/daemon/atom/utils.py
@@ -186,9 +186,11 @@ def create_template(thread: Message, platform: Platform, to=None, cc=None):
 		cc = extract_list("cc", thread)
 	if platform is Platform.TELEGRAM:
 		substr = 4000
+		media_substr = 1010
 		border = f"\n<code>{'-'*72}</code>"
 	else:
 		substr = 1900
+		media_substr = None
 		border = f"\n{'-'*80}"
 
 	subject = thread.get('subject')
@@ -201,15 +203,18 @@ def create_template(thread: Message, platform: Platform, to=None, cc=None):
 
 	if is_patch:
 		ret += content
-	else:
-		ret += content.strip().replace("\t", "        ")
+		if not media_substr:
+			return ret, files, is_patch
 
-		if len(ret) >= substr:
-			ret = ret[:substr] + "..."
+		if len(ret) >= media_substr:
+			return ret[:media_substr] + "...", files, is_patch
 
-		ret = fix_utf8_char(ret, platform is Platform.TELEGRAM)
-		ret += border
+	ret += content.strip().replace("\t", " "*8)
+	if len(ret) >= substr:
+		ret = ret[:substr] + "..."
 
+	ret = fix_utf8_char(ret, platform is Platform.TELEGRAM)
+	ret += border
 	return ret, files, is_patch
 
 
-- 
Muhammad Rizki

Re: [PATCH v2 03/17] fix: utils: add a substr for the patch media caption

[Ammar Faizi]

On 11/9/22 9:49 AM, Muhammad Rizki wrote:
> +		media_substr = 1010

How did you get 1010? Explain.

-- 
Ammar Faizi

Re: [PATCH v2 03/17] fix: utils: add a substr for the patch media caption

[Muhammad Rizki]

On 20/11/2022 11.51, Ammar Faizi wrote:
> On 11/9/22 9:49 AM, Muhammad Rizki wrote:
>> +        media_substr = 1010
> 
> How did you get 1010? Explain.
>

It supposed to set to 1021 with additional "...". But, I change this 
method from implement it on the create_template() to the prepare_patch() 
instead, because on the prepare_patch() has additional "#patch #ml".


The result after substr them will be like this, for example:
	
	#patch #ml
	From: Prabhakar <prabhakar.csengg@gmail.com>
	To: Sakari Ailus <sakari.ailus@linux.intel.com>
	To: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
	To: Mauro Carvalho Chehab <mchehab@kernel.org>
	To: Rob Herring <robh+dt@kernel.org>
	To: Krzysztof Kozlowski <krzysztof.kozlowski+dt@linaro.org>
	To: Philipp Zabel <p.zabel@pengutronix.de>
	To: Jacopo Mondi <jacopo@jmondi.org>
	To: Niklas Söderlund <niklas.soderlund@ragnatech.se>
	To: Hans Verkuil <hverkuil-cisco@xs4all.nl>
	Cc: Geert Uytterhoeven <geert+renesas@glider.be>
	Cc: linux-media@vger.kernel.org
	Cc: devicetree@vger.kernel.org
	Cc: linux-renesas-soc@vger.kernel.org
	Cc: linux-kernel@vger.kernel.org
	Cc: Prabhakar <prabhakar.csengg@gmail.com>
	Cc: Biju Das <biju.das.jz@bp.renesas.com>
	Cc: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com>
	Cc: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
	Cc: Krzysztof Kozlowski <krzysztof.kozlowski@linaro.o...

The subject of the patch doesn't appear because of the limit. Since you 
looks like don't agree reducing the header TO/CC 
(https://lore.gnuweeb.org/gwml/0c2c7b6e-3b0c-db90-7202-54e468331565@gnuweeb.org/), 
I hope you agree with this change even the subject of the patch doesn't 
appear.


I THINK it should solve the limit, what do you think? please give me 
some advice if I do wrong, thanks.

[PATCH v2 04/17] fix: utils: Fix the extract_list() utility function

[Muhammad Rizki]

Fix the extract_list() when the `payload.get()` header function is not
retrieving the header value instead of throwing error.

Simple error log says:
AttributeError: 'Group' object has no attribute 'local_part'

Source problem: https://s.id/1n1C6 (shorted lore email URL)

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/atom/utils.py | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py
index 2e85461..cfcbeb4 100644
--- a/daemon/atom/utils.py
+++ b/daemon/atom/utils.py
@@ -86,9 +86,12 @@ def __extract_list(ss):
 	return ret
 
 
-def extract_list(key: str, content: Dict[str, str]):
-	people = content.get(key.lower())
-	if not people:
+def extract_list(key: str, thread: Message):
+	try:
+		people = thread.get(key.lower())
+		if not people:
+			return []
+	except:
 		return []
 	return __extract_list(people)
 
-- 
Muhammad Rizki

[PATCH v2 05/17] utils: Back to use decode=True for the get_payload()

[Muhammad Rizki]

Because at the previous version still not solving the error, this
version is use decode=True for the get_payload() in the
get_decoded_payload() just for the base64, quoted-printable, and binary
transfer encoding only.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/atom/utils.py | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)

diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py
index cfcbeb4..dd9e1a6 100644
--- a/daemon/atom/utils.py
+++ b/daemon/atom/utils.py
@@ -10,7 +10,7 @@ from pyrogram.types import Chat, InlineKeyboardMarkup, InlineKeyboardButton
 from email.message import Message
 from typing import Dict, Union
 from slugify import slugify
-from base64 import b64decode
+import traceback
 import hashlib
 import uuid
 import os
@@ -267,11 +267,9 @@ def get_decoded_payload(payload: Message):
 	tf_encode = payload.get("Content-Transfer-Encoding")
 	charset = payload.get_content_charset("utf-8")
 
-	if tf_encode == "base64":
-		return b64decode(p).decode(charset)
-	if tf_encode == "quoted-printable":
-		quobyte = quopri.decodestring(p.encode())
-		return quobyte.decode(charset)
+	if tf_encode in ["base64", "quoted-printable", "binary"]:
+		return payload.get_payload(decode=True) \
+			.decode(errors="replace")
 
 	return p.encode().decode(charset, errors="replace")
 
-- 
Muhammad Rizki

[PATCH v2 06/17] utils: Improve fix_utf8_char()

[Muhammad Rizki]

Improvement for the fix_utf8_char() to ensure the `>` will be
unescaped, because if not use the html.unescape(), the email payload
will contain `>` for the Discord bot.

Also, change on the html.escape() to use it only once. From the past
issue bb8855bf, some email message doesn't escaped correctly, so I use
the html.escape() twice. Within the current version, this issue should
be fixed and can call the html.escape() just once.

Fixes: bb8855bf ("Fix the storage management after the refactor was happened")
Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/atom/utils.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py
index dd9e1a6..c21a4b5 100644
--- a/daemon/atom/utils.py
+++ b/daemon/atom/utils.py
@@ -258,8 +258,8 @@ def remove_patch(tmp: Union[str, list]):
 def fix_utf8_char(text: str, html_escape: bool = True):
 	t = text.rstrip().replace("�"," ")
 	if html_escape:
-		t = html.escape(html.escape(text))
-	return t
+		return html.escape(text)
+	return html.unescape(t)
 
 
 def get_decoded_payload(payload: Message):
-- 
Muhammad Rizki

Re: [PATCH v2 06/17] utils: Improve fix_utf8_char()

[Ammar Faizi]

On 11/9/22 9:49 AM, Muhammad Rizki wrote:
> Improvement for the fix_utf8_char() to ensure the `>` will be
> unescaped, because if not use the html.unescape(), the email payload
> will contain `>` for the Discord bot.
> 
> Also, change on the html.escape() to use it only once. From the past
> issue bb8855bf, some email message doesn't escaped correctly, so I use
> the html.escape() twice. Within the current version, this issue should
> be fixed and can call the html.escape() just once.
> 
> Fixes: bb8855bf ("Fix the storage management after the refactor was happened")
> Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
> ---
>   daemon/atom/utils.py | 4 ++--
>   1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py
> index dd9e1a6..c21a4b5 100644
> --- a/daemon/atom/utils.py
> +++ b/daemon/atom/utils.py
> @@ -258,8 +258,8 @@ def remove_patch(tmp: Union[str, list]):
>   def fix_utf8_char(text: str, html_escape: bool = True):
>   	t = text.rstrip().replace("�"," ")
>   	if html_escape:
> -		t = html.escape(html.escape(text))
> -	return t
> +		return html.escape(text)
> +	return html.unescape(t)

Please stop trying random things to make your output looks good
without understanding what went wrong. This stupid path has been
turning on and off forever since the beginning. What is exactly
the underlying issue behind this?

I want to get a real understanding of why such an issue happens. I
will start rejecting fixes that can't be well-understood start from
now on. For this one patch, I want you:

   1. Understand went wrong from the past.

   2. Explain how did it go wrong.

   3. Explain how does this patch act as a real fix.

Double escape was just your random attempt and it didn't actually
fix the issue well enough. Why? Because your fix is not based on an
understanding, your fix is only respecting particular output and
you hacked it to make it looks good, but throw away generic cases.

You can't explain the technical reason of why you did double escape.
Just like this patch does. I don't want we work this way forever.

-- 
Ammar Faizi

Re: [PATCH v2 06/17] utils: Improve fix_utf8_char()

[Muhammad Rizki]

On 20/11/2022 12.23, Ammar Faizi wrote:
> On 11/9/22 9:49 AM, Muhammad Rizki wrote:
>> Improvement for the fix_utf8_char() to ensure the `>` will be
>> unescaped, because if not use the html.unescape(), the email payload
>> will contain `>` for the Discord bot.
>>
>> Also, change on the html.escape() to use it only once. From the past
>> issue bb8855bf, some email message doesn't escaped correctly, so I use
>> the html.escape() twice. Within the current version, this issue should
>> be fixed and can call the html.escape() just once.
>>
>> Fixes: bb8855bf ("Fix the storage management after the refactor was 
>> happened")
>> Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
>> ---
>>   daemon/atom/utils.py | 4 ++--
>>   1 file changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py
>> index dd9e1a6..c21a4b5 100644
>> --- a/daemon/atom/utils.py
>> +++ b/daemon/atom/utils.py
>> @@ -258,8 +258,8 @@ def remove_patch(tmp: Union[str, list]):
>>   def fix_utf8_char(text: str, html_escape: bool = True):
>>       t = text.rstrip().replace("�"," ")
>>       if html_escape:
>> -        t = html.escape(html.escape(text))
>> -    return t
>> +        return html.escape(text)
>> +    return html.unescape(t)
> 
> Please stop trying random things to make your output looks good
> without understanding what went wrong. This stupid path has been
> turning on and off forever since the beginning. What is exactly
> the underlying issue behind this?
> 
> I want to get a real understanding of why such an issue happens. I
> will start rejecting fixes that can't be well-understood start from
> now on. For this one patch, I want you:
> 
>    1. Understand went wrong from the past.
> 
>    2. Explain how did it go wrong.
> 
>    3. Explain how does this patch act as a real fix.
> 
> Double escape was just your random attempt and it didn't actually
> fix the issue well enough. Why? Because your fix is not based on an
> understanding, your fix is only respecting particular output and
> you hacked it to make it looks good, but throw away generic cases.
> 
> You can't explain the technical reason of why you did double escape.
> Just like this patch does. I don't want we work this way forever.
> 

Oh, my bad. Sometimes I just being dumb when I really getting tired.
adding `html.unescape()` at the end it was because I didn't focuse to 
see the change from "t = html.escape(html.escape(text))" to "return 
html.escape(text)", that's why I unescape it again from the old `t` 
variable.

I change them to:
	
	def fix_utf8_char(text: str, html_escape: bool = True):
		t = text.rstrip().replace("�"," ")
		if html_escape:
			return html.escape(t)
		return t

We only escape the text if the platform is Telegram and we don't need to 
escape nor unescape for the Discord platform, UNLESS you want me to 
force unescape if the email payload contain "&gt;". But, yeah, forcing 
unescape is not a practice way, it will break the email payload.

This should a FINAL change for html escape issue. That's my explanation, 
sorry if sometimes having a bad code, I usually don't work too seriously 
like this, please understand.

[PATCH v2 07/17] utils: Add catch_err() for the log message

[Muhammad Rizki]

Add catch_err() for the logger message in the future use.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/atom/utils.py | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py
index c21a4b5..c85d711 100644
--- a/daemon/atom/utils.py
+++ b/daemon/atom/utils.py
@@ -274,6 +274,12 @@ def get_decoded_payload(payload: Message):
 	return p.encode().decode(charset, errors="replace")
 
 
+def catch_err():
+	return traceback.format_exc() \
+	.encode("unicode_escape") \
+	.decode("utf-8")
+
+
 EMAIL_MSG_ID_PATTERN = r"<([^\<\>]+)>"
 def extract_email_msg_id(msg_id):
 	ret = re.search(EMAIL_MSG_ID_PATTERN, msg_id)
-- 
Muhammad Rizki

[PATCH v2 08/17] logger: Initial work for the bot logger for future use

[Muhammad Rizki]

Add a BotLogger() class that inherit to Logger() class from the python
logging library.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/logger/__init__.py |  1 +
 daemon/logger/log.py      | 52 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 53 insertions(+)
 create mode 100644 daemon/logger/__init__.py
 create mode 100644 daemon/logger/log.py

diff --git a/daemon/logger/__init__.py b/daemon/logger/__init__.py
new file mode 100644
index 0000000..bb3145a
--- /dev/null
+++ b/daemon/logger/__init__.py
@@ -0,0 +1 @@
+from .log import BotLogger
diff --git a/daemon/logger/log.py b/daemon/logger/log.py
new file mode 100644
index 0000000..8382404
--- /dev/null
+++ b/daemon/logger/log.py
@@ -0,0 +1,52 @@
+import logging
+import os
+from enums import Platform
+
+
+class BotLogger(logging.Logger):
+
+	def __init__(
+		self,
+		platform: Platform = Platform.TELEGRAM,
+		level: int = logging.WARNING
+	) -> None:
+
+		self.platform = str(platform.value)
+		self.level = level
+
+		plat_rep = self.platform.replace("discord", "dscord")
+		self.file_name = f"{plat_rep}/" \
+			f"{os.getenv('STORAGE_DIR', 'storage')}/" \
+			f"{self.platform}.log"
+
+		self.file_fmt = "{" \
+			"'time': '%(asctime)s'," \
+			"'func': '%(name)s: %(funcName)s'," \
+			"'path': '%(pathname)s'," \
+			"'level': '%(levelname)s'," \
+			"'msg': '%(message)s'"\
+		"}"
+
+		self.stream_fmt = "%(asctime)s | " \
+			"%(name)s: %(funcName)s | " \
+			"%(levelname)s | " \
+			"%(message)s"
+
+
+	def init(self):
+		super().__init__(self.platform, self.level)
+
+		file_handler = logging.FileHandler(self.file_name)
+		stream_handler = logging.StreamHandler()
+
+		file_fmtr = logging.Formatter(self.file_fmt)
+		stream_fmtr = logging.Formatter(self.stream_fmt)
+
+		file_handler.setFormatter(file_fmtr)
+		stream_handler.setFormatter(stream_fmtr)
+
+		file_handler.setLevel(logging.WARNING)
+		self.setLevel(logging.INFO)
+
+		self.addHandler(file_handler)
+		self.addHandler(stream_handler)
-- 
Muhammad Rizki

[PATCH v2 09/17] telegram: Use the created BotLogger() class

[Muhammad Rizki]

Use the BotLogger() class for the Telegram DaemonClient() class. This
class is to make a log message to a file within the DaemonClient() class
of the `.logger` attribute which is initialized from the BotLogger()
class.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/telegram/packages/client.py | 6 +++++-
 daemon/tg.py                       | 5 +++++
 2 files changed, 10 insertions(+), 1 deletion(-)

diff --git a/daemon/telegram/packages/client.py b/daemon/telegram/packages/client.py
index c971ea1..eb282eb 100644
--- a/daemon/telegram/packages/client.py
+++ b/daemon/telegram/packages/client.py
@@ -10,16 +10,20 @@ from typing import Union
 from email.message import Message
 from atom import utils
 from enums import Platform
+from logger import BotLogger
 from telegram.database import DB
 from .decorator import handle_flood
 
 
 class DaemonClient(Client):
 	def __init__(self, name: str, api_id: int,
-		api_hash: str, conn, **kwargs):
+		api_hash: str, conn, logger: BotLogger,
+		**kwargs
+	):
 		super().__init__(name, api_id,
 				api_hash, **kwargs)
 		self.db = DB(conn)
+		self.logger = logger
 
 
 	@handle_flood
diff --git a/daemon/tg.py b/daemon/tg.py
index 5f8c21e..8382c29 100644
--- a/daemon/tg.py
+++ b/daemon/tg.py
@@ -11,17 +11,22 @@ from atom import Scraper
 from telegram.packages import DaemonClient
 from telegram.mailer import BotMutexes
 from telegram.mailer import Bot
+from logger import BotLogger
 import os
 
 
 def main():
 	load_dotenv("telegram.env")
 
+	logger = BotLogger()
+	logger.init()
+
 	client = DaemonClient(
 		"telegram/storage/EmailScraper",
 		api_id=int(os.getenv("API_ID")),
 		api_hash=os.getenv("API_HASH"),
 		bot_token=os.getenv("BOT_TOKEN"),
+		logger=logger,
 		conn=connector.connect(
 			host=os.getenv("DB_HOST"),
 			user=os.getenv("DB_USER"),
-- 
Muhammad Rizki

[PATCH v2 10/17] telegram: Add variable LOG_CHANNEL_ID declaration

[Muhammad Rizki]

Declare a variable LOG_CHANNEL_ID in the config.py.example for the
send_log_file() in the next commit.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/telegram/config.py.example | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/daemon/telegram/config.py.example b/daemon/telegram/config.py.example
index 3818d27..221e0c1 100644
--- a/daemon/telegram/config.py.example
+++ b/daemon/telegram/config.py.example
@@ -4,6 +4,10 @@ from pyrogram.types import Message
 # Insert Telegram admin id in the list below
 ADMINS = []
 
+# Insert Telegram channel ID to send the log file
+# type (int)
+LOG_CHANNEL_ID =
+
 
 async def adm_flt(_, __, m: Message):
 	return m.from_user.id in ADMINS
-- 
Muhammad Rizki

Re: [PATCH v2 10/17] telegram: Add variable LOG_CHANNEL_ID declaration

[Ammar Faizi]

On 11/9/22 9:49 AM, Muhammad Rizki wrote:
> Declare a variable LOG_CHANNEL_ID in the config.py.example for the
> send_log_file() in the next commit.
> 
> Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>

I really don't understand why are we keep using .env file if we have
config.py. I want you to get rid of the env file and use the config.py
only. Unless you can give me a reasonable answer of why we are using
both of them.

-- 
Ammar Faizi

Re: [PATCH v2 10/17] telegram: Add variable LOG_CHANNEL_ID declaration

[Muhammad Rizki]

On 20/11/2022 11.56, Ammar Faizi wrote:
> On 11/9/22 9:49 AM, Muhammad Rizki wrote:
>> Declare a variable LOG_CHANNEL_ID in the config.py.example for the
>> send_log_file() in the next commit.
>>
>> Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
> 
> I really don't understand why are we keep using .env file if we have
> config.py. I want you to get rid of the env file and use the config.py
> only. Unless you can give me a reasonable answer of why we are using
> both of them.
> 

I used to using .env file instead of using config.py. Since you asked to 
declare a variable in the config.py (or something git-ignored) I just 
created config.py instead of keeping it in the .env file. But I just to 
remind you, if we want to use only 1 file config between config.py or 
.env will be a minor version change, choosing between config.py or .env 
file in this patch series are not related for me. Next patch series will 
be good if you agree.

You've to choose it so I can clearly what we needed, using the .env and 
load them in each bots main file and call them using 
`os.getenv("ENV_VAR")` or using the config.py and import it in each file 
we want to get LOG_CHANNEL_ID.

[PATCH v2 11/17] telegram: Add send_log_file() in the DaemonClient()

[Muhammad Rizki]

Add send_log_file() to the DaemonClient() class.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/telegram/packages/client.py | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/daemon/telegram/packages/client.py b/daemon/telegram/packages/client.py
index eb282eb..6c83a5a 100644
--- a/daemon/telegram/packages/client.py
+++ b/daemon/telegram/packages/client.py
@@ -11,6 +11,7 @@ from email.message import Message
 from atom import utils
 from enums import Platform
 from logger import BotLogger
+from telegram import config
 from telegram.database import DB
 from .decorator import handle_flood
 
@@ -26,6 +27,16 @@ class DaemonClient(Client):
 		self.logger = logger
 
 
+	@handle_flood
+	async def send_log_file(self, caption: str):
+		filename = self.logger.handlers[0].baseFilename
+		await self.send_document(
+			config.LOG_CHANNEL_ID,
+			filename,
+			caption=caption
+		)
+
+
 	@handle_flood
 	async def send_text_email(
 		self,
-- 
Muhammad Rizki

[PATCH v2 12/17] telegram: Implement the log message for catching errors

[Muhammad Rizki]

After created the BotLogger() class and initialized it to the
DaemonClient() attribute, implement it in the
telegram/mailer/listener.py to log it into the file if catching errors.

v2:
    - Remove unrelevant changelog

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/telegram/mailer/listener.py            | 44 ++++++++++++-------
 daemon/telegram/packages/client.py            |  4 +-
 daemon/telegram/packages/decorator.py         | 15 +++----
 .../packages/plugins/commands/scrape.py       | 10 ++++-
 4 files changed, 45 insertions(+), 28 deletions(-)

diff --git a/daemon/telegram/mailer/listener.py b/daemon/telegram/mailer/listener.py
index 46ccf93..6fb2e44 100644
--- a/daemon/telegram/mailer/listener.py
+++ b/daemon/telegram/mailer/listener.py
@@ -13,7 +13,6 @@ from atom import utils
 from enums import Platform
 import asyncio
 import re
-import traceback
 
 
 class BotMutexes():
@@ -29,6 +28,7 @@ class Bot():
 		self.scraper = scraper
 		self.mutexes = mutexes
 		self.db = client.db
+		self.logger = client.logger
 		self.isRunnerFixed = False
 
 
@@ -49,8 +49,8 @@ class Bot():
 		# TODO(ammarfaizi2):
 		# Ideally, we also want to log and report this situation.
 		#
-		print(f"Database error: {str(e)}")
-		print("Reconnecting to the database...")
+		self.logger.error(f"Database error: {str(e)}")
+		self.logger.info("Reconnecting to the database...")
 
 		#
 		# Don't do this too often to avoid reconnect burst.
@@ -63,14 +63,17 @@ class Bot():
 
 
 	async def __run(self):
-		print("[__run]: Running...")
+		self.logger.info("Running...")
 		try:
 			for url in self.db.get_atom_urls():
 				await self.__handle_atom_url(url)
 		except (OperationalError, DatabaseError) as e:
 			await self.handle_db_error(e)
 		except:
-			print(traceback.format_exc())
+			exc_str = utils.catch_err()
+			self.logger.warning(exc_str)
+			capt = "Unkown raw lore URL, see full details in the log file."
+			await self.client.send_log_file(capt)
 
 		if not self.isRunnerFixed:
 			self.isRunnerFixed = True
@@ -86,8 +89,13 @@ class Bot():
 	async def __handle_atom_url(self, url):
 		urls = await self.scraper.get_new_threads_urls(url)
 		for url in urls:
-			mail = await self.scraper.get_email_from_url(url)
-			await self.__handle_mail(url, mail)
+			try:
+				mail = await self.scraper.get_email_from_url(url)
+				await self.__handle_mail(url, mail)
+			except:
+				exc_str = utils.catch_err()
+				self.logger.warning(exc_str)
+				await self.client.send_log_file(url)
 
 
 	async def __handle_mail(self, url, mail):
@@ -104,23 +112,27 @@ class Bot():
 	# @__must_hold(self.mutexes.send_to_tg)
 	async def __send_mail(self, url, mail, tg_chat_id):
 		email_msg_id = utils.get_email_msg_id(mail)
+
 		if not email_msg_id:
-			#
-			# It doesn't have a Message-Id.
-			# A malformed email. Skip!
-			#
+			md = "email_msg_id not detected, skipping malformed email"
+			self.logger.debug(md)
 			return False
 
 		email_id = self.__mail_id_from_db(email_msg_id,
 							tg_chat_id)
 		if not email_id:
-			#
-			# Email has already been sent to Telegram.
-			# Skip!
-			#
+			md = f"Skipping {email_id} because has already been sent to Telegram"
+			self.logger.debug(md)
 			return False
 
-		text, files, is_patch = utils.create_template(mail, Platform.TELEGRAM)
+		try:
+			text, files, is_patch = utils.create_template(mail, Platform.TELEGRAM)
+		except:
+			exc_str = utils.catch_err()
+			self.logger.warning(exc_str)
+			await self.client.send_log_file(url)
+			return
+
 		reply_to = self.get_reply(mail, tg_chat_id)
 		url = str(re.sub(r"/raw$", "", url))
 
diff --git a/daemon/telegram/packages/client.py b/daemon/telegram/packages/client.py
index 6c83a5a..58195b9 100644
--- a/daemon/telegram/packages/client.py
+++ b/daemon/telegram/packages/client.py
@@ -46,7 +46,7 @@ class DaemonClient(Client):
 		url: str = None,
 		parse_mode: ParseMode = ParseMode.HTML
 	) -> Message:
-		print("[send_text_email]")
+		self.logger.debug("[send_text_email]")
 		return await self.send_message(
 			chat_id=chat_id,
 			text=text,
@@ -71,7 +71,7 @@ class DaemonClient(Client):
 		url: str = None,
 		parse_mode: ParseMode = ParseMode.HTML
 	) -> Message:
-		print("[send_patch_email]")
+		self.logger.debug("[send_patch_email]")
 		tmp, doc, caption, url = utils.prepare_patch(
 			mail, text, url, Platform.TELEGRAM
 		)
diff --git a/daemon/telegram/packages/decorator.py b/daemon/telegram/packages/decorator.py
index c7a5f02..153fa95 100644
--- a/daemon/telegram/packages/decorator.py
+++ b/daemon/telegram/packages/decorator.py
@@ -24,20 +24,19 @@ def handle_flood(func: Callable[[T], T]) -> Callable[[T], T]:
 			try:
 				return await func(*args)
 			except FloodWait as e:
-				#
-				# Aiee... we hit our limit.
-				# Let's slow down a bit.
-				#
-				_flood_exceptions(e)
-				print("[__handle_telegram_floodwait]: Woken up from flood wait...")
+				# Calling logger attr from the DaemonClient() class
+				logger = args[0].logger
+
+				_flood_exceptions(e, logger)
+				logger.info("Woken up from flood wait...")
 	return callback
 
 
-async def _flood_exceptions(e):
+async def _flood_exceptions(e, logger):
 	x = re.search(r"A wait of (\d+) seconds is required", str(e))
 	if not x:
 		raise e
 
 	n = int(x.group(1))
-	print(f"[____handle_telegram_floodwait]: Sleeping for {n} seconds due to Telegram limit")
+	logger.info(f"Sleeping for {n} seconds due to Telegram limit")
 	await asyncio.sleep(n)
diff --git a/daemon/telegram/packages/plugins/commands/scrape.py b/daemon/telegram/packages/plugins/commands/scrape.py
index 29cc8ad..f89727d 100644
--- a/daemon/telegram/packages/plugins/commands/scrape.py
+++ b/daemon/telegram/packages/plugins/commands/scrape.py
@@ -36,8 +36,14 @@ async def scrap_email(c: DaemonClient, m: Message):
 		return
 
 	s = Scraper()
-	mail = await s.get_email_from_url(url)
-	text, files, is_patch = utils.create_template(mail, Platform.TELEGRAM)
+
+	try:
+		mail = await s.get_email_from_url(url)
+		text, files, is_patch = utils.create_template(mail, Platform.TELEGRAM)
+	except:
+		exc_str = utils.catch_err()
+		c.logger.warning(exc_str)
+		await c.send_log_file(url)
 
 	if is_patch:
 		m = await c.send_patch_email(
-- 
Muhammad Rizki

[PATCH v2 13/17] discord: Add variable LOG_CHANNEL_ID declaration

[Muhammad Rizki]

Add LOG_CHANNEL_ID into the config.py.example to use it in the future
use to send the log file to its Discord channel logs using the
LOG_CHANNEL_ID.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/dscord/config.py.example | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/daemon/dscord/config.py.example b/daemon/dscord/config.py.example
index 6dd4313..6db826c 100644
--- a/daemon/dscord/config.py.example
+++ b/daemon/dscord/config.py.example
@@ -11,3 +11,7 @@ ADMIN_ROLE_ID =
 # The activity name like "Playing Genshin Impact"
 # you set the value as "Genshin Impact"
 ACTIVITY_NAME =
+
+# Insert Discord channel ID to send the log file
+# type (int)
+LOG_CHANNEL_ID =
-- 
Muhammad Rizki

[PATCH v2 14/17] discord: Use the BotLogger() to the GWClient()

[Muhammad Rizki]

Use the BotLogger() and initialized it to the GWClient() class attribute
`.logger` to use it in the future use for log a message into the file.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/dc.py                    | 10 ++++++++--
 daemon/dscord/gnuweeb/client.py |  6 ++++--
 2 files changed, 12 insertions(+), 4 deletions(-)

diff --git a/daemon/dc.py b/daemon/dc.py
index 5028e94..f9b57e5 100644
--- a/daemon/dc.py
+++ b/daemon/dc.py
@@ -11,6 +11,8 @@ from apscheduler.schedulers.asyncio import AsyncIOScheduler
 from dscord.gnuweeb import GWClient
 from dscord.mailer import Listener
 from dscord.mailer import Mutexes
+from enums.platform import Platform
+from logger import BotLogger
 from atom import Scraper
 
 
@@ -24,13 +26,17 @@ def main():
 		}
 	)
 
+	logger = BotLogger(Platform.DISCORD)
+	logger.init()
+
 	client = GWClient(
 		db_conn=connector.connect(
 			host=os.getenv("DB_HOST"),
 			user=os.getenv("DB_USER"),
 			password=os.getenv("DB_PASS"),
 			database=os.getenv("DB_NAME")
-		)
+		),
+		logger=logger
 	)
 
 	mailer = Listener(
@@ -39,8 +45,8 @@ def main():
 		scraper=Scraper(),
 		mutexes=Mutexes()
 	)
-	client.mailer = mailer
 
+	client.mailer = mailer
 	client.run(os.getenv("DISCORD_TOKEN"), log_handler=None)
 
 
diff --git a/daemon/dscord/gnuweeb/client.py b/daemon/dscord/gnuweeb/client.py
index 82858c5..42bd405 100644
--- a/daemon/dscord/gnuweeb/client.py
+++ b/daemon/dscord/gnuweeb/client.py
@@ -7,22 +7,24 @@ import discord
 from discord import Interaction
 from discord.ext import commands
 from discord import Intents
-from dscord.config import ACTIVITY_NAME
 from typing import Union
 
 from . import filters
 from . import models
 from atom import utils
 from enums import Platform
+from logger.log import BotLogger
+from dscord.config import ACTIVITY_NAME
 from dscord.database import DB
 
 
 class GWClient(commands.Bot):
-	def __init__(self, db_conn) -> None:
+	def __init__(self, db_conn, logger: BotLogger) -> None:
 		self.db = DB(db_conn)
 		intents = Intents.default()
 		intents.message_content = True
 		self.mailer = None
+		self.logger = logger
 		super().__init__(
 			command_prefix=["$", "."],
 			description="Just a bot for receiving lore emails.",
-- 
Muhammad Rizki

[PATCH v2 15/17] discord: Add send_log_file in the GWClient()

[Muhammad Rizki]

Add send_log_file() to send a log file into the Discord channel for logs
using the `config.LOG_CHANNEL_ID` to send it to the Discord channel.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/dscord/gnuweeb/client.py | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/daemon/dscord/gnuweeb/client.py b/daemon/dscord/gnuweeb/client.py
index 42bd405..1b260ea 100644
--- a/daemon/dscord/gnuweeb/client.py
+++ b/daemon/dscord/gnuweeb/client.py
@@ -14,7 +14,7 @@ from . import models
 from atom import utils
 from enums import Platform
 from logger.log import BotLogger
-from dscord.config import ACTIVITY_NAME
+from dscord.config import ACTIVITY_NAME, LOG_CHANNEL_ID
 from dscord.database import DB
 
 
@@ -40,6 +40,16 @@ class GWClient(commands.Bot):
 		)
 
 
+	@filters.wait_on_limit
+	async def send_log_file(self, caption: str):
+		filename = self.logger.handlers[0].baseFilename
+		channel = self.get_channel(LOG_CHANNEL_ID)
+		await channel.send(
+			content=caption,
+			file=discord.File(filename)
+		)
+
+
 	@filters.wait_on_limit
 	async def send_text_email(self, guild_id: int, chat_id: int, text: str,
 				reply_to: Union[int, None] = None, url: str = None):
-- 
Muhammad Rizki

[PATCH v2 16/17] discord: Implement the catch erros and logs

[Muhammad Rizki]

Implement the catch erros and log the message into the file then send
it into the Discord log channel.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/dscord/gnuweeb/client.py               |  4 +-
 daemon/dscord/gnuweeb/filters.py              | 10 +++--
 .../dscord/gnuweeb/plugins/events/on_ready.py |  2 +-
 .../plugins/slash_commands/get_lore_mail.py   | 10 ++++-
 daemon/dscord/mailer/listener.py              | 40 +++++++++++--------
 5 files changed, 41 insertions(+), 25 deletions(-)

diff --git a/daemon/dscord/gnuweeb/client.py b/daemon/dscord/gnuweeb/client.py
index 1b260ea..406c606 100644
--- a/daemon/dscord/gnuweeb/client.py
+++ b/daemon/dscord/gnuweeb/client.py
@@ -53,7 +53,7 @@ class GWClient(commands.Bot):
 	@filters.wait_on_limit
 	async def send_text_email(self, guild_id: int, chat_id: int, text: str,
 				reply_to: Union[int, None] = None, url: str = None):
-		print("[send_text_email]")
+		self.logger.debug("[send_text_email]")
 		channel = self.get_channel(chat_id)
 
 		return await channel.send(
@@ -70,7 +70,7 @@ class GWClient(commands.Bot):
 	@filters.wait_on_limit
 	async def send_patch_email(self, mail, guild_id: int, chat_id: int, text: str,
 				reply_to: Union[int, None] = None, url: str = None):
-		print("[send_patch_email]")
+		self.logger.debug("[send_patch_email]")
 		tmp, doc, caption, url = utils.prepare_patch(
 			mail, text, url, Platform.DISCORD
 		)
diff --git a/daemon/dscord/gnuweeb/filters.py b/daemon/dscord/gnuweeb/filters.py
index b994904..1fdb70d 100644
--- a/daemon/dscord/gnuweeb/filters.py
+++ b/daemon/dscord/gnuweeb/filters.py
@@ -15,6 +15,7 @@ from discord import Interaction
 
 # gnuweeb package import
 from dscord import config
+from logger import BotLogger
 
 
 def lore_admin(func):
@@ -42,12 +43,15 @@ def wait_on_limit(func):
 			try:
 				return await func(*args)
 			except discord.errors.RateLimited as e:
+				# Calling logger attr from the GWClient() class
+				logger = args[0].logger
+
 				_flood_exceptions(e)
-				print("[wait_on_limit]: Woken up from flood wait...")
+				logger.info("Woken up from flood wait...")
 	return callback
 
 
-async def _flood_exceptions(e: "discord.errors.RateLimited"):
+async def _flood_exceptions(e: "discord.errors.RateLimited", logger: BotLogger):
 	wait = e.retry_after
-	print(f"[wait_on_limit]: Sleeping for {wait} seconds due to Discord limit")
+	logger.info(f"Sleeping for {wait} seconds due to Discord limit")
 	await asyncio.sleep(wait)
diff --git a/daemon/dscord/gnuweeb/plugins/events/on_ready.py b/daemon/dscord/gnuweeb/plugins/events/on_ready.py
index 4f25764..e7f63cd 100644
--- a/daemon/dscord/gnuweeb/plugins/events/on_ready.py
+++ b/daemon/dscord/gnuweeb/plugins/events/on_ready.py
@@ -23,4 +23,4 @@ class OnReady(commands.Cog):
 		t += f"Send `{prefix}sync` message to the Discord channel "
 		t += "where the bot is running.\n"
 
-		print(t)
+		self.bot.logger.info(t)
diff --git a/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py b/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py
index 0c67b8c..e4c171c 100644
--- a/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py
+++ b/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py
@@ -26,8 +26,14 @@ class GetLoreSC(commands.Cog):
 	@app_commands.describe(url="Raw lore email URL")
 	async def get_lore(self, i: "Interaction", url: str):
 		s = Scraper()
-		mail = await s.get_email_from_url(url)
-		text, _, is_patch = utils.create_template(mail, Platform.DISCORD)
+
+		try:
+			mail = await s.get_email_from_url(url)
+			text, _, is_patch = utils.create_template(mail, Platform.DISCORD)
+		except:
+			exc_str = utils.catch_err()
+			self.bot.logger.warning(exc_str)
+			await self.bot.send_log_file(url)
 
 		if is_patch:
 			m = await self.bot.send_patch_mail_interaction(
diff --git a/daemon/dscord/mailer/listener.py b/daemon/dscord/mailer/listener.py
index d986fbd..1764ea2 100644
--- a/daemon/dscord/mailer/listener.py
+++ b/daemon/dscord/mailer/listener.py
@@ -35,6 +35,7 @@ class Listener:
 		self.scraper = scraper
 		self.mutexes = mutexes
 		self.db = client.db
+		self.logger = client.logger
 		self.isRunnerFixed = False
 		self.runner = None
 
@@ -44,18 +45,15 @@ class Listener:
 		# Execute __run() once to avoid high latency at
 		# initilization.
 		#
-		print("Initialize listener...\n")
+		self.logger.info("Initialize listener...\n")
 		self.sched.start()
 		self.runner = self.sched.add_job(func=self.__run)
 
 
 	async def __run(self):
-		print("[__run]: Running...")
+		self.logger.info("Running...")
 		for url in self.db.get_atom_urls():
-			try:
-				await self.__handle_atom_url(url)
-			except:
-				print(traceback.format_exc())
+			await self.__handle_atom_url(url)
 
 		if not self.isRunnerFixed:
 			self.isRunnerFixed = True
@@ -71,8 +69,13 @@ class Listener:
 	async def __handle_atom_url(self, url):
 		urls = await self.scraper.get_new_threads_urls(url)
 		for url in urls:
-			mail = await self.scraper.get_email_from_url(url)
-			await self.__handle_mail(url, mail)
+			try:
+				mail = await self.scraper.get_email_from_url(url)
+				await self.__handle_mail(url, mail)
+			except:
+				exc_str = utils.catch_err()
+				self.client.logger.warning(exc_str)
+				await self.client.send_log_file(url)
 
 
 	async def __handle_mail(self, url, mail):
@@ -91,10 +94,8 @@ class Listener:
 	async def __send_to_discord(self, url, mail, dc_guild_id, dc_chat_id):
 		email_msg_id = utils.get_email_msg_id(mail)
 		if not email_msg_id:
-			#
-			# It doesn't have a Message-Id.
-			# A malformed email. Skip!
-			#
+			md = "email_msg_id not detected, skipping malformed email"
+			self.logger.debug(md)
 			return False
 
 		email_id = self.__get_email_id_sent(
@@ -102,13 +103,18 @@ class Listener:
 			dc_chat_id=dc_chat_id
 		)
 		if not email_id:
-			#
-			# Email has already been sent to Discord.
-			# Skip!
-			#
+			md = f"Skipping {email_id} because has already been sent to Discord"
+			self.logger.debug(md)
 			return False
 
-		text, files, is_patch = utils.create_template(mail, Platform.DISCORD)
+		try:
+			text, files, is_patch = utils.create_template(mail, Platform.DISCORD)
+		except:
+			exc_str = utils.catch_err()
+			self.client.logger.warning(exc_str)
+			await self.client.send_log_file(url)
+			return
+
 		reply_to = self.get_discord_reply(mail, dc_chat_id)
 		url = str(re.sub(r"/raw$", "", url))
 
-- 
Muhammad Rizki

[PATCH v2 17/17] Remove some unused imports

[Muhammad Rizki]

Remove them while it is not used again, from the previous version, they
are not removed.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/atom/utils.py                                          | 3 +--
 daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py | 2 --
 daemon/dscord/mailer/listener.py                              | 1 -
 3 files changed, 1 insertion(+), 5 deletions(-)

diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py
index c85d711..0576c77 100644
--- a/daemon/atom/utils.py
+++ b/daemon/atom/utils.py
@@ -8,7 +8,7 @@ from enums import Platform
 
 from pyrogram.types import Chat, InlineKeyboardMarkup, InlineKeyboardButton
 from email.message import Message
-from typing import Dict, Union
+from typing import Union
 from slugify import slugify
 import traceback
 import hashlib
@@ -18,7 +18,6 @@ import re
 import shutil
 import httpx
 import html
-import quopri
 
 
 def get_email_msg_id(mail):
diff --git a/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py b/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py
index e4c171c..56330f3 100644
--- a/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py
+++ b/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py
@@ -3,8 +3,6 @@
 # Copyright (C) 2022  Muhammad Rizki <kiizuha@gnuweeb.org>
 #
 
-import asyncio
-import discord
 from discord.ext import commands
 from discord import Interaction
 from discord import app_commands
diff --git a/daemon/dscord/mailer/listener.py b/daemon/dscord/mailer/listener.py
index 1764ea2..24f0a6c 100644
--- a/daemon/dscord/mailer/listener.py
+++ b/daemon/dscord/mailer/listener.py
@@ -5,7 +5,6 @@
 #
 
 import asyncio
-import traceback
 import re
 from apscheduler.schedulers.asyncio import AsyncIOScheduler
 from discord import File
-- 
Muhammad Rizki

Re: [PATCH v2 00/17] Fix, improvement and implement a bot logger.

[Alviro Iskandar Setiawan]

On Wed, Nov 9, 2022 at 9:49 AM Muhammad Rizki wrote:
> Hi sir.
>
> This is a v2 revision of fix, improvement and implement a bot logger.
> this revision is to improve create_template() caption substr/trim,
> prepare_patch() for the .patch file payload, remove unrelevant changelog
> commit in the commit description, use html.escape() once instead of
> twice, small change in the first commit to make it clearer.

@ammarfaizi2, this has been abandoned for a while.

-- Viro

Re: [PATCH v2 00/17] Fix, improvement and implement a bot logger.

[Ammar Faizi]

On 12/7/22 9:42 PM, Alviro Iskandar Setiawan wrote:
> On Wed, Nov 9, 2022 at 9:49 AM Muhammad Rizki wrote:
>> Hi sir.
>>
>> This is a v2 revision of fix, improvement and implement a bot logger.
>> this revision is to improve create_template() caption substr/trim,
>> prepare_patch() for the .patch file payload, remove unrelevant changelog
>> commit in the commit description, use html.escape() once instead of
>> twice, small change in the first commit to make it clearer.
> 
> @ammarfaizi2, this has been abandoned for a while.

Hi Al,

Thanks for the reminder. Sorry for that...

I have other things to do recently. I don't see this as an urgent
series anyway. But I understand, waiting this long is boring.

I'll compensate Rizki this month regardless this series. JFYI, I
love the current working notification despite it's not perfect.
But of course, we will be still improving the bot through Rizki.

------------

Rizki, is your DANA account still the same?

If you want to receive the payment via another bank account or
e-money. Please give me the detailed information.

-- 
Ammar Faizi

Re: [PATCH v2 00/17] Fix, improvement and implement a bot logger.

[Kiizuha]

On 07/12/2022 21.59, Ammar Faizi wrote:
> On 12/7/22 9:42 PM, Alviro Iskandar Setiawan wrote:
>> On Wed, Nov 9, 2022 at 9:49 AM Muhammad Rizki wrote:
>>> Hi sir.
>>>
>>> This is a v2 revision of fix, improvement and implement a bot logger.
>>> this revision is to improve create_template() caption substr/trim,
>>> prepare_patch() for the .patch file payload, remove unrelevant changelog
>>> commit in the commit description, use html.escape() once instead of
>>> twice, small change in the first commit to make it clearer.
>>
>> @ammarfaizi2, this has been abandoned for a while.
> 
> Hi Al,
> 
> Thanks for the reminder. Sorry for that...
> 
> I have other things to do recently. I don't see this as an urgent
> series anyway. But I understand, waiting this long is boring.

Ah, yes. I was waiting for your replies regarding these email IDs:
- 
https://lore.gnuweeb.org/gwml/18a3f58f-d6cf-54c0-ced2-59eafec29fae@gnuweeb.org/
- 
https://lore.gnuweeb.org/gwml/b0c0f901-6349-7056-e1cb-c0ee797e6851@gnuweeb.org/

> 
> I'll compensate Rizki this month regardless this series. JFYI, I
> love the current working notification despite it's not perfect.
> But of course, we will be still improving the bot through Rizki.
> 
> ------------
> 
> Rizki, is your DANA account still the same?
> 
> If you want to receive the payment via another bank account or
> e-money. Please give me the detailed information.
> 

Oh you want to send the compensate? Actually I didn't expect this. 
Anyway, yeah my DANA is still the same.

Sorry for using my gmail account for sending this email, I really don't 
receive this email from my GWmail account.

Re: [PATCH v2 00/17] Fix, improvement and implement a bot logger.

[Kiizuha]

On 08/12/2022 23.32, Ammar Faizi wrote:
> On 12/8/22 1:41 PM, Kiizuha wrote:
>> Anyway, yeah my DANA is still the same.
> 
> Sent. Thanks!
> 

I see. Thank you so much! I really want to continue this work, but I 
still waiting for your comments about my replies recently. But, it's up 
to you, I still can wait when you are not busy.