[PATCH v1 00/16] Fix, improvement and implement a bot logger.

[PATCH v1 00/16] Fix, improvement and implement a bot logger.

[Muhammad Rizki]

Hi sir,

This series contains fixes in the utility file and some improvement for
it. I also implement a bot logger to log an error to the terminal and
to the log file, if error may happen it will send the log file to the
log channel either for Discord or Telegram, the LOG_CHANNEL_ID is
declared in the config.py.example, copy the channel ID from the Discord
or Telegram channel and then paste to the config.py.example.

There are 16 patches in this series:
- Patch 1 is to fix a typo
- Patch 2 is to fix an extract_list() when there is no receipients
- Patch 3 is to change max receipients list header like CC/TO
- Patch 4 is to use the old one method, use decode=True in get_payload()
- Patch 5 is to improve fix_utf8_char(), make sure to unescape the HTML
- Patch 6 is to add a utility catch_err() function to get error message
- Patch 7 is to just add an initial BotLogger() to use it in next commit
- Patch 8 is to use the created BotLogger() for the Telegram bot
- Patch 9 is to add a Telegram LOG_CHANNEL_ID for the send_log_file()
- Patch 10 is to add a send_log_file() for the Telegram bot DaemonClient
- Patch 11 is to implement the log message and send log file for TG bot
- Patch 12 is to add a Discord LOG_CHANNEL_ID for the send_log_file()
- Patch 13 is to use the created BotLogger() for the Discord bot
- Patch 14 is to add a send_log_file() for the Discord bot GWClient()
- Patch 15 is to implement the log message and send log file for DC bot
- Patch 16 is to remove unused imports

What's new?
- Implementation of the BotLogger to log error message.
- Add a catch_err() in the atom/utils.py.
- Add a send_log_file() in each bot client.py.
- Add a variable LOG_CHANNEL_ID to configure the log channel.

What's changed?
- Some utility functions are fixed and improved.
- Fix typo on _flood_exception()

How to set log channel ID for the Telegram bot:
1. Make a channel for log in the Telegram
2. Copy the channel ID, if you unable to see the ID you can use API or
   bot who has the feature, or use a 64gram desktop.
3. Paste the copied ID into the config.py in the telegram directory

How to set log channel ID for the Discord bot:
1. Make a channel for log in your Discord server
2. Copy the channel ID by right click and copy, if you unable to get the
   ID you can go to your account settings, then enable the
   `Developer mode` or use bot who has the feature.
3. Paste the copied ID into the config.py in the dscord directory

How to use:
1. Execute the db.sql file in the daemon directory,
2. Setup .env file, the example is there with suffix .example, this
   file name must remove the suffix name .example,
3. Set up the config.py in each bot directory, such as dscord and
   telegram. The example is there with suffix .example & the file name
   must remove suffix name .example,
4. Run `pip3 install -r requirements.txt` in each bot directory,
5. STORAGE_DIR env value must `storage` to make it work fine,
6. Run the bot by `python3 dc.py` or `python3 tg.py`.

Tested and works, please give it a test, thanks!

Muhammad Rizki (16):
  discord: Fix typo on _flood_exception()
  fix: utils: Fix the extract_list() utility function
  utils: Change on max for TO/CC header list
  utils: Back to use decode=True for the get_payload()
  utils: Improve fix_utf8_char()
  utils: Add catch_err() for the log message
  logger: Initial work for the bot logger for future use
  telegram: Use the created BotLogger() class
  telegram: Add variable LOG_CHANNEL_ID declaration
  telegram: Add send_log_file() in the DaemonClient()
  telegram: Implement the log message for catching errors
  discord: Add variable LOG_CHANNEL_ID declaration
  discord: Use the BotLogger() to the GWClient()
  discord: Add send_log_file in the GWClient()
  discord: Implement the catch erros and logs
  Remove some unused imports

 daemon/atom/utils.py                          | 50 ++++++++++++------
 daemon/dc.py                                  | 10 +++-
 daemon/dscord/config.py.example               |  4 ++
 daemon/dscord/gnuweeb/client.py               | 20 +++++--
 daemon/dscord/gnuweeb/filters.py              | 10 ++--
 .../dscord/gnuweeb/plugins/events/on_ready.py |  2 +-
 .../plugins/slash_commands/get_lore_mail.py   | 12 +++--
 daemon/dscord/mailer/listener.py              | 41 ++++++++-------
 daemon/logger/__init__.py                     |  1 +
 daemon/logger/log.py                          | 52 +++++++++++++++++++
 daemon/telegram/config.py.example             |  4 ++
 daemon/telegram/mailer/listener.py            | 44 ++++++++++------
 daemon/telegram/packages/client.py            | 21 ++++++--
 daemon/telegram/packages/decorator.py         | 15 +++---
 .../packages/plugins/commands/scrape.py       | 10 +++-
 daemon/tg.py                                  |  5 ++
 16 files changed, 223 insertions(+), 78 deletions(-)
 create mode 100644 daemon/logger/__init__.py
 create mode 100644 daemon/logger/log.py


base-commit: 91e86e6d1004da4526ca5514fa009bc253d5c4a3
--
Muhammad Rizki

[PATCH v1 01/16] discord: Fix typo on _flood_exception()

[Muhammad Rizki]

Fix minor typo. The function is for Discord bot, so fix it from
"Telegram" to "Discord".

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/dscord/gnuweeb/filters.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/daemon/dscord/gnuweeb/filters.py b/daemon/dscord/gnuweeb/filters.py
index eb54a8a..b994904 100644
--- a/daemon/dscord/gnuweeb/filters.py
+++ b/daemon/dscord/gnuweeb/filters.py
@@ -49,5 +49,5 @@ def wait_on_limit(func):
 
 async def _flood_exceptions(e: "discord.errors.RateLimited"):
 	wait = e.retry_after
-	print(f"[wait_on_limit]: Sleeping for {wait} seconds due to Telegram limit")
+	print(f"[wait_on_limit]: Sleeping for {wait} seconds due to Discord limit")
 	await asyncio.sleep(wait)
-- 
Muhammad Rizki

[PATCH v1 02/16] fix: utils: Fix the extract_list() utility function

[Muhammad Rizki]

Fix the extract_list() when the `payload.get()` header function is not
retrieving the header value instead of throwing error.

Simple error log says:
AttributeError: 'Group' object has no attribute 'local_part'

Source problem: https://s.id/1n1C6 (shorted lore email URL)

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/atom/utils.py | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py
index 03453b4..88ce128 100644
--- a/daemon/atom/utils.py
+++ b/daemon/atom/utils.py
@@ -86,9 +86,12 @@ def __extract_list(ss):
 	return ret
 
 
-def extract_list(key: str, content: Dict[str, str]):
-	people = content.get(key.lower())
-	if not people:
+def extract_list(key: str, thread: Message):
+	try:
+		people = thread.get(key.lower())
+		if not people:
+			return []
+	except:
 		return []
 	return __extract_list(people)
 
-- 
Muhammad Rizki

[PATCH v1 03/16] utils: Change on max for TO/CC header list

[Muhammad Rizki]

Telegram media caption is limited to 1024 char length, set the max list
to 15 to avoid MediaCaptionTooLong error.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/atom/utils.py | 18 ++++++++++++++----
 1 file changed, 14 insertions(+), 4 deletions(-)

diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py
index 88ce128..b807f7b 100644
--- a/daemon/atom/utils.py
+++ b/daemon/atom/utils.py
@@ -96,8 +96,17 @@ def extract_list(key: str, thread: Message):
 	return __extract_list(people)
 
 
-def consruct_to_n_cc(to: list, cc: list):
+def consruct_to_n_cc(
+	to: list,
+	cc: list,
+	is_patch: bool,
+	platform: Platform.TELEGRAM
+):
 	NR_MAX_LIST = 20
+	if platform == Platform.TELEGRAM and is_patch:
+		# Telegram caption limited to 1024
+		# so set max to 15
+		NR_MAX_LIST = 15
 
 	n = 0
 	ret = ""
@@ -195,12 +204,13 @@ def create_template(thread: Message, platform: Platform, to=None, cc=None):
 		border = f"\n{'-'*80}"
 
 	subject = thread.get('subject')
+	content, files = extract_body(thread, platform)
+	is_patch = __is_patch(subject, content)
+
 	ret = f"From: {thread.get('from')}\n"
-	ret += consruct_to_n_cc(to, cc)
+	ret += consruct_to_n_cc(to, cc, is_patch, platform)
 	ret += f"Date: {thread.get('date')}\n"
 	ret += f"Subject: {subject}\n\n"
-	content, files = extract_body(thread, platform)
-	is_patch = __is_patch(subject, content)
 
 	if is_patch:
 		ret += content
-- 
Muhammad Rizki

Re: [PATCH v1 03/16] utils: Change on max for TO/CC header list

[Ammar Faizi]

On 11/5/22 1:09 AM, Muhammad Rizki wrote:
> Telegram media caption is limited to 1024 char length, set the max list
> to 15 to avoid MediaCaptionTooLong error.
> 
> Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>

How does this guarantee to solve the issue?

It's *not* impossible to have 15 participants with a total of characters
more than 1024.

-- 
Ammar Faizi

Re: [PATCH v1 03/16] utils: Change on max for TO/CC header list

[Muhammad Rizki]

On 07/11/2022 07.55, Ammar Faizi wrote:
> On 11/5/22 1:09 AM, Muhammad Rizki wrote:
>> Telegram media caption is limited to 1024 char length, set the max list
>> to 15 to avoid MediaCaptionTooLong error.
>>
>> Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
> 
> How does this guarantee to solve the issue?
> 
> It's *not* impossible to have 15 participants with a total of characters
> more than 1024.
> 

When I tested the bot, one of the patch email causing 
MediaCaptionTooLong error, the problem is from the CC is too long, while 
I set the max to 15 it's fixed the error.

Re: [PATCH v1 03/16] utils: Change on max for TO/CC header list

[Ammar Faizi]

On 11/7/22 8:08 AM, Muhammad Rizki wrote:
> When I tested the bot, one of the patch email causing 
> MediaCaptionTooLong error, the problem is from the CC is too long, while 
> I set the max to 15 it's fixed the error.

Yes, it works with your case. But how do you guarantee that 15 participants
always yield a total of characters less than 1024?

Just because it fixes your case, it doesn't mean the underlying 1024 limit
problem is fixed.

-- 
Ammar Faizi

Re: [PATCH v1 03/16] utils: Change on max for TO/CC header list

[Muhammad Rizki]

On 07/11/2022 08.15, Ammar Faizi wrote:
> On 11/7/22 8:08 AM, Muhammad Rizki wrote:
>> When I tested the bot, one of the patch email causing 
>> MediaCaptionTooLong error, the problem is from the CC is too long, 
>> while I set the max to 15 it's fixed the error.
> 
> Yes, it works with your case. But how do you guarantee that 15 participants
> always yield a total of characters less than 1024?

Uhh, sorry my bad. I could just use this method instead, like the 
non-patch substr.

# from the atom/utils.py in create_template()

	if is_patch:
		ret += content
		if len(ret) >= substr:
			ret = ret[:substr] + "..."
	else:
		ret += content.strip().replace("\t", "        ")

		if len(ret) >= substr:
			ret = ret[:substr] + "..."

[PATCH v1 04/16] utils: Back to use decode=True for the get_payload()

[Muhammad Rizki]

Because at the previous version still not solving the error, this
version is use decode=True for the get_payload() in the
get_decoded_payload() just for the base64, quoted-printable, and binary
transfer encoding only.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/atom/utils.py | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)

diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py
index b807f7b..15679d2 100644
--- a/daemon/atom/utils.py
+++ b/daemon/atom/utils.py
@@ -10,7 +10,7 @@ from pyrogram.types import Chat, InlineKeyboardMarkup, InlineKeyboardButton
 from email.message import Message
 from typing import Dict, Union
 from slugify import slugify
-from base64 import b64decode
+import traceback
 import hashlib
 import uuid
 import os
@@ -272,11 +272,9 @@ def get_decoded_payload(payload: Message):
 	tf_encode = payload.get("Content-Transfer-Encoding")
 	charset = payload.get_content_charset("utf-8")
 
-	if tf_encode == "base64":
-		return b64decode(p).decode(charset)
-	if tf_encode == "quoted-printable":
-		quobyte = quopri.decodestring(p.encode())
-		return quobyte.decode(charset)
+	if tf_encode in ["base64", "quoted-printable", "binary"]:
+		return payload.get_payload(decode=True) \
+			.decode(errors="replace")
 
 	return p.encode().decode(charset, errors="replace")
 
-- 
Muhammad Rizki

[PATCH v1 05/16] utils: Improve fix_utf8_char()

[Muhammad Rizki]

Improvement for the fix_utf8_char() to ensure the `>` will be
unescaped, because if not use the html.unescape(), the email payload
will contain `>` for the Discord bot.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/atom/utils.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py
index 15679d2..fe3f8d2 100644
--- a/daemon/atom/utils.py
+++ b/daemon/atom/utils.py
@@ -263,8 +263,8 @@ def remove_patch(tmp: Union[str, list]):
 def fix_utf8_char(text: str, html_escape: bool = True):
 	t = text.rstrip().replace("�"," ")
 	if html_escape:
-		t = html.escape(html.escape(text))
-	return t
+		return html.escape(html.escape(text))
+	return html.unescape(t)
 
 
 def get_decoded_payload(payload: Message):
-- 
Muhammad Rizki

Re: [PATCH v1 05/16] utils: Improve fix_utf8_char()

[Ammar Faizi]

On 11/5/22 1:09 AM, Muhammad Rizki wrote:
> --- a/daemon/atom/utils.py
> +++ b/daemon/atom/utils.py
> @@ -263,8 +263,8 @@ def remove_patch(tmp: Union[str, list]):
>   def fix_utf8_char(text: str, html_escape: bool = True):
>   	t = text.rstrip().replace("�"," ")
>   	if html_escape:
> -		t = html.escape(html.escape(text))
> -	return t
> +		return html.escape(html.escape(text))
> +	return html.unescape(t)

Can you explain why we need to do the following:

    html.escape(html.escape(text))

Why does it have to be escaped twice? I still don't understand the
reason behind this mess since the beginning.

-- 
Ammar Faizi

Re: [PATCH v1 05/16] utils: Improve fix_utf8_char()

[Muhammad Rizki]

On 07/11/2022 08.01, Ammar Faizi wrote:
> On 11/5/22 1:09 AM, Muhammad Rizki wrote:
>> --- a/daemon/atom/utils.py
>> +++ b/daemon/atom/utils.py
>> @@ -263,8 +263,8 @@ def remove_patch(tmp: Union[str, list]):
>>   def fix_utf8_char(text: str, html_escape: bool = True):
>>       t = text.rstrip().replace("�"," ")
>>       if html_escape:
>> -        t = html.escape(html.escape(text))
>> -    return t
>> +        return html.escape(html.escape(text))
>> +    return html.unescape(t)
> 
> Can you explain why we need to do the following:
> 
>     html.escape(html.escape(text))
> 
> Why does it have to be escaped twice? I still don't understand the
> reason behind this mess since the beginning.
> 

 From the past patch, some email doesn't escape correctly, like it 
supposed to escaped "> >" to "> >", but the result keep "> >". 
That problem is from the past, IDK if current patch will fix that escape 
if using html.escape() 1 time.

Re: [PATCH v1 05/16] utils: Improve fix_utf8_char()

[Ammar Faizi]

On 11/7/22 8:11 AM, Muhammad Rizki wrote:
> From the past patch, some email doesn't escape correctly, like it
> supposed to escaped "> >" to "> >", but the result keep "> >".
> That problem is from the past, IDK if current patch will fix that escape
> if using html.escape() 1 time.

Any reproducer to try?

-- 
Ammar Faizi

Re: [PATCH v1 05/16] utils: Improve fix_utf8_char()

[Muhammad Rizki]

On 07/11/2022 08.26, Ammar Faizi wrote:
> On 11/7/22 8:11 AM, Muhammad Rizki wrote:
>> From the past patch, some email doesn't escape correctly, like it
>> supposed to escaped "> >" to "> >", but the result keep "> >".
>> That problem is from the past, IDK if current patch will fix that escape
>> if using html.escape() 1 time.
> 
> Any reproducer to try?
> 

Will try it in the next version.

[PATCH v1 06/16] utils: Add catch_err() for the log message

[Muhammad Rizki]

Add catch_err() for the logger message in the future use.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/atom/utils.py | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py
index fe3f8d2..60f2c76 100644
--- a/daemon/atom/utils.py
+++ b/daemon/atom/utils.py
@@ -279,6 +279,12 @@ def get_decoded_payload(payload: Message):
 	return p.encode().decode(charset, errors="replace")
 
 
+def catch_err():
+	return traceback.format_exc() \
+	.encode("unicode_escape") \
+	.decode("utf-8")
+
+
 EMAIL_MSG_ID_PATTERN = r"<([^\<\>]+)>"
 def extract_email_msg_id(msg_id):
 	ret = re.search(EMAIL_MSG_ID_PATTERN, msg_id)
-- 
Muhammad Rizki

[PATCH v1 07/16] logger: Initial work for the bot logger for future use

[Muhammad Rizki]

Add a BotLogger() class that inherit to Logger() class from the python
logging library.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/logger/__init__.py |  1 +
 daemon/logger/log.py      | 52 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 53 insertions(+)
 create mode 100644 daemon/logger/__init__.py
 create mode 100644 daemon/logger/log.py

diff --git a/daemon/logger/__init__.py b/daemon/logger/__init__.py
new file mode 100644
index 0000000..bb3145a
--- /dev/null
+++ b/daemon/logger/__init__.py
@@ -0,0 +1 @@
+from .log import BotLogger
diff --git a/daemon/logger/log.py b/daemon/logger/log.py
new file mode 100644
index 0000000..8382404
--- /dev/null
+++ b/daemon/logger/log.py
@@ -0,0 +1,52 @@
+import logging
+import os
+from enums import Platform
+
+
+class BotLogger(logging.Logger):
+
+	def __init__(
+		self,
+		platform: Platform = Platform.TELEGRAM,
+		level: int = logging.WARNING
+	) -> None:
+
+		self.platform = str(platform.value)
+		self.level = level
+
+		plat_rep = self.platform.replace("discord", "dscord")
+		self.file_name = f"{plat_rep}/" \
+			f"{os.getenv('STORAGE_DIR', 'storage')}/" \
+			f"{self.platform}.log"
+
+		self.file_fmt = "{" \
+			"'time': '%(asctime)s'," \
+			"'func': '%(name)s: %(funcName)s'," \
+			"'path': '%(pathname)s'," \
+			"'level': '%(levelname)s'," \
+			"'msg': '%(message)s'"\
+		"}"
+
+		self.stream_fmt = "%(asctime)s | " \
+			"%(name)s: %(funcName)s | " \
+			"%(levelname)s | " \
+			"%(message)s"
+
+
+	def init(self):
+		super().__init__(self.platform, self.level)
+
+		file_handler = logging.FileHandler(self.file_name)
+		stream_handler = logging.StreamHandler()
+
+		file_fmtr = logging.Formatter(self.file_fmt)
+		stream_fmtr = logging.Formatter(self.stream_fmt)
+
+		file_handler.setFormatter(file_fmtr)
+		stream_handler.setFormatter(stream_fmtr)
+
+		file_handler.setLevel(logging.WARNING)
+		self.setLevel(logging.INFO)
+
+		self.addHandler(file_handler)
+		self.addHandler(stream_handler)
-- 
Muhammad Rizki

[PATCH v1 08/16] telegram: Use the created BotLogger() class

[Muhammad Rizki]

Use the BotLogger() class for the Telegram DaemonClient() class. This
class is to make a log message to a file within the DaemonClient() class
of the `.logger` attribute which is initialized from the BotLogger()
class.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/telegram/packages/client.py | 6 +++++-
 daemon/tg.py                       | 5 +++++
 2 files changed, 10 insertions(+), 1 deletion(-)

diff --git a/daemon/telegram/packages/client.py b/daemon/telegram/packages/client.py
index c971ea1..eb282eb 100644
--- a/daemon/telegram/packages/client.py
+++ b/daemon/telegram/packages/client.py
@@ -10,16 +10,20 @@ from typing import Union
 from email.message import Message
 from atom import utils
 from enums import Platform
+from logger import BotLogger
 from telegram.database import DB
 from .decorator import handle_flood
 
 
 class DaemonClient(Client):
 	def __init__(self, name: str, api_id: int,
-		api_hash: str, conn, **kwargs):
+		api_hash: str, conn, logger: BotLogger,
+		**kwargs
+	):
 		super().__init__(name, api_id,
 				api_hash, **kwargs)
 		self.db = DB(conn)
+		self.logger = logger
 
 
 	@handle_flood
diff --git a/daemon/tg.py b/daemon/tg.py
index 5f8c21e..8382c29 100644
--- a/daemon/tg.py
+++ b/daemon/tg.py
@@ -11,17 +11,22 @@ from atom import Scraper
 from telegram.packages import DaemonClient
 from telegram.mailer import BotMutexes
 from telegram.mailer import Bot
+from logger import BotLogger
 import os
 
 
 def main():
 	load_dotenv("telegram.env")
 
+	logger = BotLogger()
+	logger.init()
+
 	client = DaemonClient(
 		"telegram/storage/EmailScraper",
 		api_id=int(os.getenv("API_ID")),
 		api_hash=os.getenv("API_HASH"),
 		bot_token=os.getenv("BOT_TOKEN"),
+		logger=logger,
 		conn=connector.connect(
 			host=os.getenv("DB_HOST"),
 			user=os.getenv("DB_USER"),
-- 
Muhammad Rizki

[PATCH v1 09/15] telegram: Add send_log_file() in the DaemonClient()

[Muhammad Rizki]

Add send_log_file() to the DaemonClient() class and declare a variable
LOG_CHANNEL_ID in the config.py.example.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/telegram/config.py.example  |  4 ++++
 daemon/telegram/packages/client.py | 11 +++++++++++
 2 files changed, 15 insertions(+)

diff --git a/daemon/telegram/config.py.example b/daemon/telegram/config.py.example
index 3818d27..221e0c1 100644
--- a/daemon/telegram/config.py.example
+++ b/daemon/telegram/config.py.example
@@ -4,6 +4,10 @@ from pyrogram.types import Message
 # Insert Telegram admin id in the list below
 ADMINS = []
 
+# Insert Telegram channel ID to send the log file
+# type (int)
+LOG_CHANNEL_ID =
+
 
 async def adm_flt(_, __, m: Message):
 	return m.from_user.id in ADMINS
diff --git a/daemon/telegram/packages/client.py b/daemon/telegram/packages/client.py
index eb282eb..6c83a5a 100644
--- a/daemon/telegram/packages/client.py
+++ b/daemon/telegram/packages/client.py
@@ -11,6 +11,7 @@ from email.message import Message
 from atom import utils
 from enums import Platform
 from logger import BotLogger
+from telegram import config
 from telegram.database import DB
 from .decorator import handle_flood
 
@@ -26,6 +27,16 @@ class DaemonClient(Client):
 		self.logger = logger
 
 
+	@handle_flood
+	async def send_log_file(self, caption: str):
+		filename = self.logger.handlers[0].baseFilename
+		await self.send_document(
+			config.LOG_CHANNEL_ID,
+			filename,
+			caption=caption
+		)
+
+
 	@handle_flood
 	async def send_text_email(
 		self,
-- 
Muhammad Rizki

[PATCH v1 09/16] telegram: Add variable LOG_CHANNEL_ID declaration

[Muhammad Rizki]

Declare a variable LOG_CHANNEL_ID in the config.py.example for the
send_log_file() in the next commit.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/telegram/config.py.example | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/daemon/telegram/config.py.example b/daemon/telegram/config.py.example
index 3818d27..221e0c1 100644
--- a/daemon/telegram/config.py.example
+++ b/daemon/telegram/config.py.example
@@ -4,6 +4,10 @@ from pyrogram.types import Message
 # Insert Telegram admin id in the list below
 ADMINS = []
 
+# Insert Telegram channel ID to send the log file
+# type (int)
+LOG_CHANNEL_ID =
+
 
 async def adm_flt(_, __, m: Message):
 	return m.from_user.id in ADMINS
-- 
Muhammad Rizki

[PATCH v1 10/16] telegram: Add send_log_file() in the DaemonClient()

[Muhammad Rizki]

Add send_log_file() to the DaemonClient() class.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/telegram/packages/client.py | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/daemon/telegram/packages/client.py b/daemon/telegram/packages/client.py
index eb282eb..6c83a5a 100644
--- a/daemon/telegram/packages/client.py
+++ b/daemon/telegram/packages/client.py
@@ -11,6 +11,7 @@ from email.message import Message
 from atom import utils
 from enums import Platform
 from logger import BotLogger
+from telegram import config
 from telegram.database import DB
 from .decorator import handle_flood
 
@@ -26,6 +27,16 @@ class DaemonClient(Client):
 		self.logger = logger
 
 
+	@handle_flood
+	async def send_log_file(self, caption: str):
+		filename = self.logger.handlers[0].baseFilename
+		await self.send_document(
+			config.LOG_CHANNEL_ID,
+			filename,
+			caption=caption
+		)
+
+
 	@handle_flood
 	async def send_text_email(
 		self,
-- 
Muhammad Rizki

[PATCH v1 10/15] telegram: Implement the log message for catching errors

[Muhammad Rizki]

After created the BotLogger() class and initialized it to the
DaemonClient() attribute, implement it in the
telegram/mailer/listener.py to log it into the file if catching errors.

v4:
  - Add logger

v3:
  - Simplify try and except statement (comment from Muhammad Rizki).

v2:
  * no changes *

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/telegram/mailer/listener.py            | 44 ++++++++++++-------
 daemon/telegram/packages/client.py            |  4 +-
 daemon/telegram/packages/decorator.py         | 15 +++----
 .../packages/plugins/commands/scrape.py       | 10 ++++-
 4 files changed, 45 insertions(+), 28 deletions(-)

diff --git a/daemon/telegram/mailer/listener.py b/daemon/telegram/mailer/listener.py
index 46ccf93..6fb2e44 100644
--- a/daemon/telegram/mailer/listener.py
+++ b/daemon/telegram/mailer/listener.py
@@ -13,7 +13,6 @@ from atom import utils
 from enums import Platform
 import asyncio
 import re
-import traceback
 
 
 class BotMutexes():
@@ -29,6 +28,7 @@ class Bot():
 		self.scraper = scraper
 		self.mutexes = mutexes
 		self.db = client.db
+		self.logger = client.logger
 		self.isRunnerFixed = False
 
 
@@ -49,8 +49,8 @@ class Bot():
 		# TODO(ammarfaizi2):
 		# Ideally, we also want to log and report this situation.
 		#
-		print(f"Database error: {str(e)}")
-		print("Reconnecting to the database...")
+		self.logger.error(f"Database error: {str(e)}")
+		self.logger.info("Reconnecting to the database...")
 
 		#
 		# Don't do this too often to avoid reconnect burst.
@@ -63,14 +63,17 @@ class Bot():
 
 
 	async def __run(self):
-		print("[__run]: Running...")
+		self.logger.info("Running...")
 		try:
 			for url in self.db.get_atom_urls():
 				await self.__handle_atom_url(url)
 		except (OperationalError, DatabaseError) as e:
 			await self.handle_db_error(e)
 		except:
-			print(traceback.format_exc())
+			exc_str = utils.catch_err()
+			self.logger.warning(exc_str)
+			capt = "Unkown raw lore URL, see full details in the log file."
+			await self.client.send_log_file(capt)
 
 		if not self.isRunnerFixed:
 			self.isRunnerFixed = True
@@ -86,8 +89,13 @@ class Bot():
 	async def __handle_atom_url(self, url):
 		urls = await self.scraper.get_new_threads_urls(url)
 		for url in urls:
-			mail = await self.scraper.get_email_from_url(url)
-			await self.__handle_mail(url, mail)
+			try:
+				mail = await self.scraper.get_email_from_url(url)
+				await self.__handle_mail(url, mail)
+			except:
+				exc_str = utils.catch_err()
+				self.logger.warning(exc_str)
+				await self.client.send_log_file(url)
 
 
 	async def __handle_mail(self, url, mail):
@@ -104,23 +112,27 @@ class Bot():
 	# @__must_hold(self.mutexes.send_to_tg)
 	async def __send_mail(self, url, mail, tg_chat_id):
 		email_msg_id = utils.get_email_msg_id(mail)
+
 		if not email_msg_id:
-			#
-			# It doesn't have a Message-Id.
-			# A malformed email. Skip!
-			#
+			md = "email_msg_id not detected, skipping malformed email"
+			self.logger.debug(md)
 			return False
 
 		email_id = self.__mail_id_from_db(email_msg_id,
 							tg_chat_id)
 		if not email_id:
-			#
-			# Email has already been sent to Telegram.
-			# Skip!
-			#
+			md = f"Skipping {email_id} because has already been sent to Telegram"
+			self.logger.debug(md)
 			return False
 
-		text, files, is_patch = utils.create_template(mail, Platform.TELEGRAM)
+		try:
+			text, files, is_patch = utils.create_template(mail, Platform.TELEGRAM)
+		except:
+			exc_str = utils.catch_err()
+			self.logger.warning(exc_str)
+			await self.client.send_log_file(url)
+			return
+
 		reply_to = self.get_reply(mail, tg_chat_id)
 		url = str(re.sub(r"/raw$", "", url))
 
diff --git a/daemon/telegram/packages/client.py b/daemon/telegram/packages/client.py
index 6c83a5a..58195b9 100644
--- a/daemon/telegram/packages/client.py
+++ b/daemon/telegram/packages/client.py
@@ -46,7 +46,7 @@ class DaemonClient(Client):
 		url: str = None,
 		parse_mode: ParseMode = ParseMode.HTML
 	) -> Message:
-		print("[send_text_email]")
+		self.logger.debug("[send_text_email]")
 		return await self.send_message(
 			chat_id=chat_id,
 			text=text,
@@ -71,7 +71,7 @@ class DaemonClient(Client):
 		url: str = None,
 		parse_mode: ParseMode = ParseMode.HTML
 	) -> Message:
-		print("[send_patch_email]")
+		self.logger.debug("[send_patch_email]")
 		tmp, doc, caption, url = utils.prepare_patch(
 			mail, text, url, Platform.TELEGRAM
 		)
diff --git a/daemon/telegram/packages/decorator.py b/daemon/telegram/packages/decorator.py
index c7a5f02..153fa95 100644
--- a/daemon/telegram/packages/decorator.py
+++ b/daemon/telegram/packages/decorator.py
@@ -24,20 +24,19 @@ def handle_flood(func: Callable[[T], T]) -> Callable[[T], T]:
 			try:
 				return await func(*args)
 			except FloodWait as e:
-				#
-				# Aiee... we hit our limit.
-				# Let's slow down a bit.
-				#
-				_flood_exceptions(e)
-				print("[__handle_telegram_floodwait]: Woken up from flood wait...")
+				# Calling logger attr from the DaemonClient() class
+				logger = args[0].logger
+
+				_flood_exceptions(e, logger)
+				logger.info("Woken up from flood wait...")
 	return callback
 
 
-async def _flood_exceptions(e):
+async def _flood_exceptions(e, logger):
 	x = re.search(r"A wait of (\d+) seconds is required", str(e))
 	if not x:
 		raise e
 
 	n = int(x.group(1))
-	print(f"[____handle_telegram_floodwait]: Sleeping for {n} seconds due to Telegram limit")
+	logger.info(f"Sleeping for {n} seconds due to Telegram limit")
 	await asyncio.sleep(n)
diff --git a/daemon/telegram/packages/plugins/commands/scrape.py b/daemon/telegram/packages/plugins/commands/scrape.py
index 29cc8ad..f89727d 100644
--- a/daemon/telegram/packages/plugins/commands/scrape.py
+++ b/daemon/telegram/packages/plugins/commands/scrape.py
@@ -36,8 +36,14 @@ async def scrap_email(c: DaemonClient, m: Message):
 		return
 
 	s = Scraper()
-	mail = await s.get_email_from_url(url)
-	text, files, is_patch = utils.create_template(mail, Platform.TELEGRAM)
+
+	try:
+		mail = await s.get_email_from_url(url)
+		text, files, is_patch = utils.create_template(mail, Platform.TELEGRAM)
+	except:
+		exc_str = utils.catch_err()
+		c.logger.warning(exc_str)
+		await c.send_log_file(url)
 
 	if is_patch:
 		m = await c.send_patch_email(
-- 
Muhammad Rizki

[PATCH v1 11/15] discord: Add variable LOG_CHANNEL_ID declaration

[Muhammad Rizki]

Add LOG_CHANNEL_ID into the config.py.example to use it in the future
use to send the log file to its Discord channel logs using the
LOG_CHANNEL_ID.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/dscord/config.py.example | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/daemon/dscord/config.py.example b/daemon/dscord/config.py.example
index 6dd4313..6db826c 100644
--- a/daemon/dscord/config.py.example
+++ b/daemon/dscord/config.py.example
@@ -11,3 +11,7 @@ ADMIN_ROLE_ID =
 # The activity name like "Playing Genshin Impact"
 # you set the value as "Genshin Impact"
 ACTIVITY_NAME =
+
+# Insert Discord channel ID to send the log file
+# type (int)
+LOG_CHANNEL_ID =
-- 
Muhammad Rizki

[PATCH v1 11/16] telegram: Implement the log message for catching errors

[Muhammad Rizki]

After created the BotLogger() class and initialized it to the
DaemonClient() attribute, implement it in the
telegram/mailer/listener.py to log it into the file if catching errors.

v4:
  - Add logger

v3:
  - Simplify try and except statement (comment from Muhammad Rizki).

v2:
  * no changes *

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/telegram/mailer/listener.py            | 44 ++++++++++++-------
 daemon/telegram/packages/client.py            |  4 +-
 daemon/telegram/packages/decorator.py         | 15 +++----
 .../packages/plugins/commands/scrape.py       | 10 ++++-
 4 files changed, 45 insertions(+), 28 deletions(-)

diff --git a/daemon/telegram/mailer/listener.py b/daemon/telegram/mailer/listener.py
index 46ccf93..6fb2e44 100644
--- a/daemon/telegram/mailer/listener.py
+++ b/daemon/telegram/mailer/listener.py
@@ -13,7 +13,6 @@ from atom import utils
 from enums import Platform
 import asyncio
 import re
-import traceback
 
 
 class BotMutexes():
@@ -29,6 +28,7 @@ class Bot():
 		self.scraper = scraper
 		self.mutexes = mutexes
 		self.db = client.db
+		self.logger = client.logger
 		self.isRunnerFixed = False
 
 
@@ -49,8 +49,8 @@ class Bot():
 		# TODO(ammarfaizi2):
 		# Ideally, we also want to log and report this situation.
 		#
-		print(f"Database error: {str(e)}")
-		print("Reconnecting to the database...")
+		self.logger.error(f"Database error: {str(e)}")
+		self.logger.info("Reconnecting to the database...")
 
 		#
 		# Don't do this too often to avoid reconnect burst.
@@ -63,14 +63,17 @@ class Bot():
 
 
 	async def __run(self):
-		print("[__run]: Running...")
+		self.logger.info("Running...")
 		try:
 			for url in self.db.get_atom_urls():
 				await self.__handle_atom_url(url)
 		except (OperationalError, DatabaseError) as e:
 			await self.handle_db_error(e)
 		except:
-			print(traceback.format_exc())
+			exc_str = utils.catch_err()
+			self.logger.warning(exc_str)
+			capt = "Unkown raw lore URL, see full details in the log file."
+			await self.client.send_log_file(capt)
 
 		if not self.isRunnerFixed:
 			self.isRunnerFixed = True
@@ -86,8 +89,13 @@ class Bot():
 	async def __handle_atom_url(self, url):
 		urls = await self.scraper.get_new_threads_urls(url)
 		for url in urls:
-			mail = await self.scraper.get_email_from_url(url)
-			await self.__handle_mail(url, mail)
+			try:
+				mail = await self.scraper.get_email_from_url(url)
+				await self.__handle_mail(url, mail)
+			except:
+				exc_str = utils.catch_err()
+				self.logger.warning(exc_str)
+				await self.client.send_log_file(url)
 
 
 	async def __handle_mail(self, url, mail):
@@ -104,23 +112,27 @@ class Bot():
 	# @__must_hold(self.mutexes.send_to_tg)
 	async def __send_mail(self, url, mail, tg_chat_id):
 		email_msg_id = utils.get_email_msg_id(mail)
+
 		if not email_msg_id:
-			#
-			# It doesn't have a Message-Id.
-			# A malformed email. Skip!
-			#
+			md = "email_msg_id not detected, skipping malformed email"
+			self.logger.debug(md)
 			return False
 
 		email_id = self.__mail_id_from_db(email_msg_id,
 							tg_chat_id)
 		if not email_id:
-			#
-			# Email has already been sent to Telegram.
-			# Skip!
-			#
+			md = f"Skipping {email_id} because has already been sent to Telegram"
+			self.logger.debug(md)
 			return False
 
-		text, files, is_patch = utils.create_template(mail, Platform.TELEGRAM)
+		try:
+			text, files, is_patch = utils.create_template(mail, Platform.TELEGRAM)
+		except:
+			exc_str = utils.catch_err()
+			self.logger.warning(exc_str)
+			await self.client.send_log_file(url)
+			return
+
 		reply_to = self.get_reply(mail, tg_chat_id)
 		url = str(re.sub(r"/raw$", "", url))
 
diff --git a/daemon/telegram/packages/client.py b/daemon/telegram/packages/client.py
index 6c83a5a..58195b9 100644
--- a/daemon/telegram/packages/client.py
+++ b/daemon/telegram/packages/client.py
@@ -46,7 +46,7 @@ class DaemonClient(Client):
 		url: str = None,
 		parse_mode: ParseMode = ParseMode.HTML
 	) -> Message:
-		print("[send_text_email]")
+		self.logger.debug("[send_text_email]")
 		return await self.send_message(
 			chat_id=chat_id,
 			text=text,
@@ -71,7 +71,7 @@ class DaemonClient(Client):
 		url: str = None,
 		parse_mode: ParseMode = ParseMode.HTML
 	) -> Message:
-		print("[send_patch_email]")
+		self.logger.debug("[send_patch_email]")
 		tmp, doc, caption, url = utils.prepare_patch(
 			mail, text, url, Platform.TELEGRAM
 		)
diff --git a/daemon/telegram/packages/decorator.py b/daemon/telegram/packages/decorator.py
index c7a5f02..153fa95 100644
--- a/daemon/telegram/packages/decorator.py
+++ b/daemon/telegram/packages/decorator.py
@@ -24,20 +24,19 @@ def handle_flood(func: Callable[[T], T]) -> Callable[[T], T]:
 			try:
 				return await func(*args)
 			except FloodWait as e:
-				#
-				# Aiee... we hit our limit.
-				# Let's slow down a bit.
-				#
-				_flood_exceptions(e)
-				print("[__handle_telegram_floodwait]: Woken up from flood wait...")
+				# Calling logger attr from the DaemonClient() class
+				logger = args[0].logger
+
+				_flood_exceptions(e, logger)
+				logger.info("Woken up from flood wait...")
 	return callback
 
 
-async def _flood_exceptions(e):
+async def _flood_exceptions(e, logger):
 	x = re.search(r"A wait of (\d+) seconds is required", str(e))
 	if not x:
 		raise e
 
 	n = int(x.group(1))
-	print(f"[____handle_telegram_floodwait]: Sleeping for {n} seconds due to Telegram limit")
+	logger.info(f"Sleeping for {n} seconds due to Telegram limit")
 	await asyncio.sleep(n)
diff --git a/daemon/telegram/packages/plugins/commands/scrape.py b/daemon/telegram/packages/plugins/commands/scrape.py
index 29cc8ad..f89727d 100644
--- a/daemon/telegram/packages/plugins/commands/scrape.py
+++ b/daemon/telegram/packages/plugins/commands/scrape.py
@@ -36,8 +36,14 @@ async def scrap_email(c: DaemonClient, m: Message):
 		return
 
 	s = Scraper()
-	mail = await s.get_email_from_url(url)
-	text, files, is_patch = utils.create_template(mail, Platform.TELEGRAM)
+
+	try:
+		mail = await s.get_email_from_url(url)
+		text, files, is_patch = utils.create_template(mail, Platform.TELEGRAM)
+	except:
+		exc_str = utils.catch_err()
+		c.logger.warning(exc_str)
+		await c.send_log_file(url)
 
 	if is_patch:
 		m = await c.send_patch_email(
-- 
Muhammad Rizki

Re: [PATCH v1 11/16] telegram: Implement the log message for catching errors

[Ammar Faizi]

On 11/5/22 1:09 AM, Muhammad Rizki wrote:
> After created the BotLogger() class and initialized it to the
> DaemonClient() attribute, implement it in the
> telegram/mailer/listener.py to log it into the file if catching errors.
> 
> v4:
>    - Add logger
> 
> v3:
>    - Simplify try and except statement (comment from Muhammad Rizki).
> 
> v2:
>    * no changes *
> 
> Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>

Why do v4, v3, v2 appear on there? I don't understand the relevance.

-- 
Ammar Faizi

Re: [PATCH v1 11/16] telegram: Implement the log message for catching errors

[Muhammad Rizki]

On 07/11/2022 08.05, Ammar Faizi wrote:
> On 11/5/22 1:09 AM, Muhammad Rizki wrote:
>> After created the BotLogger() class and initialized it to the
>> DaemonClient() attribute, implement it in the
>> telegram/mailer/listener.py to log it into the file if catching errors.
>>
>> v4:
>>    - Add logger
>>
>> v3:
>>    - Simplify try and except statement (comment from Muhammad Rizki).
>>
>> v2:
>>    * no changes *
>>
>> Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
> 
> Why do v4, v3, v2 appear on there? I don't understand the relevance.
> 

Ohh, so the changelog in the commit description MUST have relevant to 
current patch? IDK about this.

Re: [PATCH v1 11/16] telegram: Implement the log message for catching errors

[Ammar Faizi]

On 11/7/22 8:13 AM, Muhammad Rizki wrote:
> Ohh, so the changelog in the commit description MUST have relevant to current patch? IDK about this.

The changelog says about: v4, v3, v2, v1 (what are they?)

This is a v1 patch. It's the first version of the series. Where do they come?

-- 
Ammar Faizi

Re: [PATCH v1 11/16] telegram: Implement the log message for catching errors

[Muhammad Rizki]

On 07/11/2022 08.16, Ammar Faizi wrote:
> On 11/7/22 8:13 AM, Muhammad Rizki wrote:
>> Ohh, so the changelog in the commit description MUST have relevant to 
>> current patch? IDK about this.
> 
> The changelog says about: v4, v3, v2, v1 (what are they?)
> 
> This is a v1 patch. It's the first version of the series. Where do they 
> come?
> 

I understand what your talking about. IDK the rule about changelog in 
commit desc must have relevant with the "current "series". I will remove 
them as they are not relevant with the "current series".

Re: [PATCH v1 11/16] telegram: Implement the log message for catching errors

[Ammar Faizi]

On 11/7/22 8:26 AM, Muhammad Rizki wrote:
> IDK the rule about changelog in commit desc must have relevant with the
> "current "series".

I mean, I don't understand why did you write this:

v4:
   - Add logger

v3:
   - Simplify try and except statement (comment from Muhammad Rizki).

v2:
   * no changes *

What is v4?
What is v3?
What is v2?
When did the revision happen?

It makes *zero* sense to have v2 to v4 changelog in v1 patch.

-- 
Ammar Faizi

Re: [PATCH v1 11/16] telegram: Implement the log message for catching errors

[Muhammad Rizki]

On 07/11/2022 08.31, Ammar Faizi wrote:
> On 11/7/22 8:26 AM, Muhammad Rizki wrote:
>> IDK the rule about changelog in commit desc must have relevant with the
>> "current "series".
> 
> I mean, I don't understand why did you write this:
> 
> v4:
>    - Add logger
> 
> v3:
>    - Simplify try and except statement (comment from Muhammad Rizki).
> 
> v2:
>    * no changes *
> 
> What is v4?
> What is v3?
> What is v2?
> When did the revision happen?

You wrote v2 and v3, and the v4 is me. You wrote the v2 and v3 is from 
the database reconnect patch.

> 
> It makes *zero* sense to have v2 to v4 changelog in v1 patch.
> 

Yes, without further a due, let me just remove them instead :)

Re: [PATCH v1 11/16] telegram: Implement the log message for catching errors

[Ammar Faizi]

On 11/7/22 8:34 AM, Muhammad Rizki wrote:
> You wrote v2 and v3, and the v4 is me. You wrote the v2 and v3 is from the
> database reconnect patch.

I see, I understand what you were thinking about. Let me explain to you.

If you want to reference a commit that is already in the master branch, you can
use the SHA1 hash to mention the commit. Also, please use "Fixes" tag if the
patch fixes a problem introduced by a commit that is already in the master
branch.

Something like this for example:
https://github.com/torvalds/linux/commit/80ef928643c1558a0474389fcd680a5ccd6c86e6

> Yes, without further a due, let me just remove them instead :)

Thanks for the clarification.

-- 
Ammar Faizi

Re: [PATCH v1 11/16] telegram: Implement the log message for catching errors

[Muhammad Rizki]

On 07/11/2022 08.42, Ammar Faizi wrote:
> On 11/7/22 8:34 AM, Muhammad Rizki wrote:
>> You wrote v2 and v3, and the v4 is me. You wrote the v2 and v3 is from 
>> the
>> database reconnect patch.
> 
> I see, I understand what you were thinking about. Let me explain to you.
> 
> If you want to reference a commit that is already in the master branch, 
> you can
> use the SHA1 hash to mention the commit. Also, please use "Fixes" tag if 
> the
> patch fixes a problem introduced by a commit that is already in the master
> branch.
> 
> Something like this for example:
> https://github.com/torvalds/linux/commit/80ef928643c1558a0474389fcd680a5ccd6c86e6
> 

Ohh, I see now. I could just remove them instead while I just adding the 
logger attribute in that file.

>> Yes, without further a due, let me just remove them instead :)
> 
> Thanks for the clarification.
> 

Your welcome.

Re: [PATCH v1 11/16] telegram: Implement the log message for catching errors

[Muhammad Rizki]

On 07/11/2022 08.42, Ammar Faizi wrote:
> On 11/7/22 8:34 AM, Muhammad Rizki wrote:
>> You wrote v2 and v3, and the v4 is me. You wrote the v2 and v3 is from 
>> the
>> database reconnect patch.
> 
> I see, I understand what you were thinking about. Let me explain to you.
> 
> If you want to reference a commit that is already in the master branch, 
> you can
> use the SHA1 hash to mention the commit. Also, please use "Fixes" tag if 
> the
> patch fixes a problem introduced by a commit that is already in the master
> branch.
> 
> Something like this for example:
> https://github.com/torvalds/linux/commit/80ef928643c1558a0474389fcd680a5ccd6c86e6
> 

Anyway, thanks for the example. I will try like that if a file error or 
I want to mention a commit that is from the master branch.

[PATCH v1 12/16] discord: Add variable LOG_CHANNEL_ID declaration

[Muhammad Rizki]

Add LOG_CHANNEL_ID into the config.py.example to use it in the future
use to send the log file to its Discord channel logs using the
LOG_CHANNEL_ID.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/dscord/config.py.example | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/daemon/dscord/config.py.example b/daemon/dscord/config.py.example
index 6dd4313..6db826c 100644
--- a/daemon/dscord/config.py.example
+++ b/daemon/dscord/config.py.example
@@ -11,3 +11,7 @@ ADMIN_ROLE_ID =
 # The activity name like "Playing Genshin Impact"
 # you set the value as "Genshin Impact"
 ACTIVITY_NAME =
+
+# Insert Discord channel ID to send the log file
+# type (int)
+LOG_CHANNEL_ID =
-- 
Muhammad Rizki

[PATCH v1 12/15] discord: Use the BotLogger() to the GWClient()

[Muhammad Rizki]

Use the BotLogger() and initialized it to the GWClient() class attribute
`.logger` to use it in the future use for log a message into the file.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/dc.py                    | 10 ++++++++--
 daemon/dscord/gnuweeb/client.py |  6 ++++--
 2 files changed, 12 insertions(+), 4 deletions(-)

diff --git a/daemon/dc.py b/daemon/dc.py
index 5028e94..f9b57e5 100644
--- a/daemon/dc.py
+++ b/daemon/dc.py
@@ -11,6 +11,8 @@ from apscheduler.schedulers.asyncio import AsyncIOScheduler
 from dscord.gnuweeb import GWClient
 from dscord.mailer import Listener
 from dscord.mailer import Mutexes
+from enums.platform import Platform
+from logger import BotLogger
 from atom import Scraper
 
 
@@ -24,13 +26,17 @@ def main():
 		}
 	)
 
+	logger = BotLogger(Platform.DISCORD)
+	logger.init()
+
 	client = GWClient(
 		db_conn=connector.connect(
 			host=os.getenv("DB_HOST"),
 			user=os.getenv("DB_USER"),
 			password=os.getenv("DB_PASS"),
 			database=os.getenv("DB_NAME")
-		)
+		),
+		logger=logger
 	)
 
 	mailer = Listener(
@@ -39,8 +45,8 @@ def main():
 		scraper=Scraper(),
 		mutexes=Mutexes()
 	)
-	client.mailer = mailer
 
+	client.mailer = mailer
 	client.run(os.getenv("DISCORD_TOKEN"), log_handler=None)
 
 
diff --git a/daemon/dscord/gnuweeb/client.py b/daemon/dscord/gnuweeb/client.py
index 82858c5..42bd405 100644
--- a/daemon/dscord/gnuweeb/client.py
+++ b/daemon/dscord/gnuweeb/client.py
@@ -7,22 +7,24 @@ import discord
 from discord import Interaction
 from discord.ext import commands
 from discord import Intents
-from dscord.config import ACTIVITY_NAME
 from typing import Union
 
 from . import filters
 from . import models
 from atom import utils
 from enums import Platform
+from logger.log import BotLogger
+from dscord.config import ACTIVITY_NAME
 from dscord.database import DB
 
 
 class GWClient(commands.Bot):
-	def __init__(self, db_conn) -> None:
+	def __init__(self, db_conn, logger: BotLogger) -> None:
 		self.db = DB(db_conn)
 		intents = Intents.default()
 		intents.message_content = True
 		self.mailer = None
+		self.logger = logger
 		super().__init__(
 			command_prefix=["$", "."],
 			description="Just a bot for receiving lore emails.",
-- 
Muhammad Rizki

[PATCH v1 13/15] discord: Add send_log_file in the GWClient()

[Muhammad Rizki]

Add send_log_file() to send a log file into the Discord channel for logs
using the `config.LOG_CHANNEL_ID` to send it to the Discord channel.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/dscord/gnuweeb/client.py | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/daemon/dscord/gnuweeb/client.py b/daemon/dscord/gnuweeb/client.py
index 42bd405..1b260ea 100644
--- a/daemon/dscord/gnuweeb/client.py
+++ b/daemon/dscord/gnuweeb/client.py
@@ -14,7 +14,7 @@ from . import models
 from atom import utils
 from enums import Platform
 from logger.log import BotLogger
-from dscord.config import ACTIVITY_NAME
+from dscord.config import ACTIVITY_NAME, LOG_CHANNEL_ID
 from dscord.database import DB
 
 
@@ -40,6 +40,16 @@ class GWClient(commands.Bot):
 		)
 
 
+	@filters.wait_on_limit
+	async def send_log_file(self, caption: str):
+		filename = self.logger.handlers[0].baseFilename
+		channel = self.get_channel(LOG_CHANNEL_ID)
+		await channel.send(
+			content=caption,
+			file=discord.File(filename)
+		)
+
+
 	@filters.wait_on_limit
 	async def send_text_email(self, guild_id: int, chat_id: int, text: str,
 				reply_to: Union[int, None] = None, url: str = None):
-- 
Muhammad Rizki

[PATCH v1 13/16] discord: Use the BotLogger() to the GWClient()

[Muhammad Rizki]

Use the BotLogger() and initialized it to the GWClient() class attribute
`.logger` to use it in the future use for log a message into the file.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/dc.py                    | 10 ++++++++--
 daemon/dscord/gnuweeb/client.py |  6 ++++--
 2 files changed, 12 insertions(+), 4 deletions(-)

diff --git a/daemon/dc.py b/daemon/dc.py
index 5028e94..f9b57e5 100644
--- a/daemon/dc.py
+++ b/daemon/dc.py
@@ -11,6 +11,8 @@ from apscheduler.schedulers.asyncio import AsyncIOScheduler
 from dscord.gnuweeb import GWClient
 from dscord.mailer import Listener
 from dscord.mailer import Mutexes
+from enums.platform import Platform
+from logger import BotLogger
 from atom import Scraper
 
 
@@ -24,13 +26,17 @@ def main():
 		}
 	)
 
+	logger = BotLogger(Platform.DISCORD)
+	logger.init()
+
 	client = GWClient(
 		db_conn=connector.connect(
 			host=os.getenv("DB_HOST"),
 			user=os.getenv("DB_USER"),
 			password=os.getenv("DB_PASS"),
 			database=os.getenv("DB_NAME")
-		)
+		),
+		logger=logger
 	)
 
 	mailer = Listener(
@@ -39,8 +45,8 @@ def main():
 		scraper=Scraper(),
 		mutexes=Mutexes()
 	)
-	client.mailer = mailer
 
+	client.mailer = mailer
 	client.run(os.getenv("DISCORD_TOKEN"), log_handler=None)
 
 
diff --git a/daemon/dscord/gnuweeb/client.py b/daemon/dscord/gnuweeb/client.py
index 82858c5..42bd405 100644
--- a/daemon/dscord/gnuweeb/client.py
+++ b/daemon/dscord/gnuweeb/client.py
@@ -7,22 +7,24 @@ import discord
 from discord import Interaction
 from discord.ext import commands
 from discord import Intents
-from dscord.config import ACTIVITY_NAME
 from typing import Union
 
 from . import filters
 from . import models
 from atom import utils
 from enums import Platform
+from logger.log import BotLogger
+from dscord.config import ACTIVITY_NAME
 from dscord.database import DB
 
 
 class GWClient(commands.Bot):
-	def __init__(self, db_conn) -> None:
+	def __init__(self, db_conn, logger: BotLogger) -> None:
 		self.db = DB(db_conn)
 		intents = Intents.default()
 		intents.message_content = True
 		self.mailer = None
+		self.logger = logger
 		super().__init__(
 			command_prefix=["$", "."],
 			description="Just a bot for receiving lore emails.",
-- 
Muhammad Rizki

[PATCH v1 14/16] discord: Add send_log_file in the GWClient()

[Muhammad Rizki]

Add send_log_file() to send a log file into the Discord channel for logs
using the `config.LOG_CHANNEL_ID` to send it to the Discord channel.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/dscord/gnuweeb/client.py | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/daemon/dscord/gnuweeb/client.py b/daemon/dscord/gnuweeb/client.py
index 42bd405..1b260ea 100644
--- a/daemon/dscord/gnuweeb/client.py
+++ b/daemon/dscord/gnuweeb/client.py
@@ -14,7 +14,7 @@ from . import models
 from atom import utils
 from enums import Platform
 from logger.log import BotLogger
-from dscord.config import ACTIVITY_NAME
+from dscord.config import ACTIVITY_NAME, LOG_CHANNEL_ID
 from dscord.database import DB
 
 
@@ -40,6 +40,16 @@ class GWClient(commands.Bot):
 		)
 
 
+	@filters.wait_on_limit
+	async def send_log_file(self, caption: str):
+		filename = self.logger.handlers[0].baseFilename
+		channel = self.get_channel(LOG_CHANNEL_ID)
+		await channel.send(
+			content=caption,
+			file=discord.File(filename)
+		)
+
+
 	@filters.wait_on_limit
 	async def send_text_email(self, guild_id: int, chat_id: int, text: str,
 				reply_to: Union[int, None] = None, url: str = None):
-- 
Muhammad Rizki

[PATCH v1 14/15] discord: Implement the catch erros and logs

[Muhammad Rizki]

Implement the catch erros and log the message into the file then send
it into the Discord log channel.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/dscord/gnuweeb/client.py               |  4 +-
 daemon/dscord/gnuweeb/filters.py              | 10 +++--
 .../dscord/gnuweeb/plugins/events/on_ready.py |  2 +-
 .../plugins/slash_commands/get_lore_mail.py   | 10 ++++-
 daemon/dscord/mailer/listener.py              | 40 +++++++++++--------
 5 files changed, 41 insertions(+), 25 deletions(-)

diff --git a/daemon/dscord/gnuweeb/client.py b/daemon/dscord/gnuweeb/client.py
index 1b260ea..406c606 100644
--- a/daemon/dscord/gnuweeb/client.py
+++ b/daemon/dscord/gnuweeb/client.py
@@ -53,7 +53,7 @@ class GWClient(commands.Bot):
 	@filters.wait_on_limit
 	async def send_text_email(self, guild_id: int, chat_id: int, text: str,
 				reply_to: Union[int, None] = None, url: str = None):
-		print("[send_text_email]")
+		self.logger.debug("[send_text_email]")
 		channel = self.get_channel(chat_id)
 
 		return await channel.send(
@@ -70,7 +70,7 @@ class GWClient(commands.Bot):
 	@filters.wait_on_limit
 	async def send_patch_email(self, mail, guild_id: int, chat_id: int, text: str,
 				reply_to: Union[int, None] = None, url: str = None):
-		print("[send_patch_email]")
+		self.logger.debug("[send_patch_email]")
 		tmp, doc, caption, url = utils.prepare_patch(
 			mail, text, url, Platform.DISCORD
 		)
diff --git a/daemon/dscord/gnuweeb/filters.py b/daemon/dscord/gnuweeb/filters.py
index b994904..1fdb70d 100644
--- a/daemon/dscord/gnuweeb/filters.py
+++ b/daemon/dscord/gnuweeb/filters.py
@@ -15,6 +15,7 @@ from discord import Interaction
 
 # gnuweeb package import
 from dscord import config
+from logger import BotLogger
 
 
 def lore_admin(func):
@@ -42,12 +43,15 @@ def wait_on_limit(func):
 			try:
 				return await func(*args)
 			except discord.errors.RateLimited as e:
+				# Calling logger attr from the GWClient() class
+				logger = args[0].logger
+
 				_flood_exceptions(e)
-				print("[wait_on_limit]: Woken up from flood wait...")
+				logger.info("Woken up from flood wait...")
 	return callback
 
 
-async def _flood_exceptions(e: "discord.errors.RateLimited"):
+async def _flood_exceptions(e: "discord.errors.RateLimited", logger: BotLogger):
 	wait = e.retry_after
-	print(f"[wait_on_limit]: Sleeping for {wait} seconds due to Discord limit")
+	logger.info(f"Sleeping for {wait} seconds due to Discord limit")
 	await asyncio.sleep(wait)
diff --git a/daemon/dscord/gnuweeb/plugins/events/on_ready.py b/daemon/dscord/gnuweeb/plugins/events/on_ready.py
index 4f25764..e7f63cd 100644
--- a/daemon/dscord/gnuweeb/plugins/events/on_ready.py
+++ b/daemon/dscord/gnuweeb/plugins/events/on_ready.py
@@ -23,4 +23,4 @@ class OnReady(commands.Cog):
 		t += f"Send `{prefix}sync` message to the Discord channel "
 		t += "where the bot is running.\n"
 
-		print(t)
+		self.bot.logger.info(t)
diff --git a/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py b/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py
index 0c67b8c..e4c171c 100644
--- a/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py
+++ b/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py
@@ -26,8 +26,14 @@ class GetLoreSC(commands.Cog):
 	@app_commands.describe(url="Raw lore email URL")
 	async def get_lore(self, i: "Interaction", url: str):
 		s = Scraper()
-		mail = await s.get_email_from_url(url)
-		text, _, is_patch = utils.create_template(mail, Platform.DISCORD)
+
+		try:
+			mail = await s.get_email_from_url(url)
+			text, _, is_patch = utils.create_template(mail, Platform.DISCORD)
+		except:
+			exc_str = utils.catch_err()
+			self.bot.logger.warning(exc_str)
+			await self.bot.send_log_file(url)
 
 		if is_patch:
 			m = await self.bot.send_patch_mail_interaction(
diff --git a/daemon/dscord/mailer/listener.py b/daemon/dscord/mailer/listener.py
index d986fbd..1764ea2 100644
--- a/daemon/dscord/mailer/listener.py
+++ b/daemon/dscord/mailer/listener.py
@@ -35,6 +35,7 @@ class Listener:
 		self.scraper = scraper
 		self.mutexes = mutexes
 		self.db = client.db
+		self.logger = client.logger
 		self.isRunnerFixed = False
 		self.runner = None
 
@@ -44,18 +45,15 @@ class Listener:
 		# Execute __run() once to avoid high latency at
 		# initilization.
 		#
-		print("Initialize listener...\n")
+		self.logger.info("Initialize listener...\n")
 		self.sched.start()
 		self.runner = self.sched.add_job(func=self.__run)
 
 
 	async def __run(self):
-		print("[__run]: Running...")
+		self.logger.info("Running...")
 		for url in self.db.get_atom_urls():
-			try:
-				await self.__handle_atom_url(url)
-			except:
-				print(traceback.format_exc())
+			await self.__handle_atom_url(url)
 
 		if not self.isRunnerFixed:
 			self.isRunnerFixed = True
@@ -71,8 +69,13 @@ class Listener:
 	async def __handle_atom_url(self, url):
 		urls = await self.scraper.get_new_threads_urls(url)
 		for url in urls:
-			mail = await self.scraper.get_email_from_url(url)
-			await self.__handle_mail(url, mail)
+			try:
+				mail = await self.scraper.get_email_from_url(url)
+				await self.__handle_mail(url, mail)
+			except:
+				exc_str = utils.catch_err()
+				self.client.logger.warning(exc_str)
+				await self.client.send_log_file(url)
 
 
 	async def __handle_mail(self, url, mail):
@@ -91,10 +94,8 @@ class Listener:
 	async def __send_to_discord(self, url, mail, dc_guild_id, dc_chat_id):
 		email_msg_id = utils.get_email_msg_id(mail)
 		if not email_msg_id:
-			#
-			# It doesn't have a Message-Id.
-			# A malformed email. Skip!
-			#
+			md = "email_msg_id not detected, skipping malformed email"
+			self.logger.debug(md)
 			return False
 
 		email_id = self.__get_email_id_sent(
@@ -102,13 +103,18 @@ class Listener:
 			dc_chat_id=dc_chat_id
 		)
 		if not email_id:
-			#
-			# Email has already been sent to Discord.
-			# Skip!
-			#
+			md = f"Skipping {email_id} because has already been sent to Discord"
+			self.logger.debug(md)
 			return False
 
-		text, files, is_patch = utils.create_template(mail, Platform.DISCORD)
+		try:
+			text, files, is_patch = utils.create_template(mail, Platform.DISCORD)
+		except:
+			exc_str = utils.catch_err()
+			self.client.logger.warning(exc_str)
+			await self.client.send_log_file(url)
+			return
+
 		reply_to = self.get_discord_reply(mail, dc_chat_id)
 		url = str(re.sub(r"/raw$", "", url))
 
-- 
Muhammad Rizki

[PATCH v1 15/15] Remove some unused imports

[Muhammad Rizki]

Remove them while it is not used again, from the previous version, they
are not removed.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/atom/utils.py                                          | 3 +--
 daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py | 2 --
 daemon/dscord/mailer/listener.py                              | 1 -
 3 files changed, 1 insertion(+), 5 deletions(-)

diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py
index 60f2c76..1181621 100644
--- a/daemon/atom/utils.py
+++ b/daemon/atom/utils.py
@@ -8,7 +8,7 @@ from enums import Platform
 
 from pyrogram.types import Chat, InlineKeyboardMarkup, InlineKeyboardButton
 from email.message import Message
-from typing import Dict, Union
+from typing import Union
 from slugify import slugify
 import traceback
 import hashlib
@@ -18,7 +18,6 @@ import re
 import shutil
 import httpx
 import html
-import quopri
 
 
 def get_email_msg_id(mail):
diff --git a/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py b/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py
index e4c171c..56330f3 100644
--- a/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py
+++ b/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py
@@ -3,8 +3,6 @@
 # Copyright (C) 2022  Muhammad Rizki <kiizuha@gnuweeb.org>
 #
 
-import asyncio
-import discord
 from discord.ext import commands
 from discord import Interaction
 from discord import app_commands
diff --git a/daemon/dscord/mailer/listener.py b/daemon/dscord/mailer/listener.py
index 1764ea2..24f0a6c 100644
--- a/daemon/dscord/mailer/listener.py
+++ b/daemon/dscord/mailer/listener.py
@@ -5,7 +5,6 @@
 #
 
 import asyncio
-import traceback
 import re
 from apscheduler.schedulers.asyncio import AsyncIOScheduler
 from discord import File
-- 
Muhammad Rizki

[PATCH v1 15/16] discord: Implement the catch erros and logs

[Muhammad Rizki]

Implement the catch erros and log the message into the file then send
it into the Discord log channel.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/dscord/gnuweeb/client.py               |  4 +-
 daemon/dscord/gnuweeb/filters.py              | 10 +++--
 .../dscord/gnuweeb/plugins/events/on_ready.py |  2 +-
 .../plugins/slash_commands/get_lore_mail.py   | 10 ++++-
 daemon/dscord/mailer/listener.py              | 40 +++++++++++--------
 5 files changed, 41 insertions(+), 25 deletions(-)

diff --git a/daemon/dscord/gnuweeb/client.py b/daemon/dscord/gnuweeb/client.py
index 1b260ea..406c606 100644
--- a/daemon/dscord/gnuweeb/client.py
+++ b/daemon/dscord/gnuweeb/client.py
@@ -53,7 +53,7 @@ class GWClient(commands.Bot):
 	@filters.wait_on_limit
 	async def send_text_email(self, guild_id: int, chat_id: int, text: str,
 				reply_to: Union[int, None] = None, url: str = None):
-		print("[send_text_email]")
+		self.logger.debug("[send_text_email]")
 		channel = self.get_channel(chat_id)
 
 		return await channel.send(
@@ -70,7 +70,7 @@ class GWClient(commands.Bot):
 	@filters.wait_on_limit
 	async def send_patch_email(self, mail, guild_id: int, chat_id: int, text: str,
 				reply_to: Union[int, None] = None, url: str = None):
-		print("[send_patch_email]")
+		self.logger.debug("[send_patch_email]")
 		tmp, doc, caption, url = utils.prepare_patch(
 			mail, text, url, Platform.DISCORD
 		)
diff --git a/daemon/dscord/gnuweeb/filters.py b/daemon/dscord/gnuweeb/filters.py
index b994904..1fdb70d 100644
--- a/daemon/dscord/gnuweeb/filters.py
+++ b/daemon/dscord/gnuweeb/filters.py
@@ -15,6 +15,7 @@ from discord import Interaction
 
 # gnuweeb package import
 from dscord import config
+from logger import BotLogger
 
 
 def lore_admin(func):
@@ -42,12 +43,15 @@ def wait_on_limit(func):
 			try:
 				return await func(*args)
 			except discord.errors.RateLimited as e:
+				# Calling logger attr from the GWClient() class
+				logger = args[0].logger
+
 				_flood_exceptions(e)
-				print("[wait_on_limit]: Woken up from flood wait...")
+				logger.info("Woken up from flood wait...")
 	return callback
 
 
-async def _flood_exceptions(e: "discord.errors.RateLimited"):
+async def _flood_exceptions(e: "discord.errors.RateLimited", logger: BotLogger):
 	wait = e.retry_after
-	print(f"[wait_on_limit]: Sleeping for {wait} seconds due to Discord limit")
+	logger.info(f"Sleeping for {wait} seconds due to Discord limit")
 	await asyncio.sleep(wait)
diff --git a/daemon/dscord/gnuweeb/plugins/events/on_ready.py b/daemon/dscord/gnuweeb/plugins/events/on_ready.py
index 4f25764..e7f63cd 100644
--- a/daemon/dscord/gnuweeb/plugins/events/on_ready.py
+++ b/daemon/dscord/gnuweeb/plugins/events/on_ready.py
@@ -23,4 +23,4 @@ class OnReady(commands.Cog):
 		t += f"Send `{prefix}sync` message to the Discord channel "
 		t += "where the bot is running.\n"
 
-		print(t)
+		self.bot.logger.info(t)
diff --git a/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py b/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py
index 0c67b8c..e4c171c 100644
--- a/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py
+++ b/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py
@@ -26,8 +26,14 @@ class GetLoreSC(commands.Cog):
 	@app_commands.describe(url="Raw lore email URL")
 	async def get_lore(self, i: "Interaction", url: str):
 		s = Scraper()
-		mail = await s.get_email_from_url(url)
-		text, _, is_patch = utils.create_template(mail, Platform.DISCORD)
+
+		try:
+			mail = await s.get_email_from_url(url)
+			text, _, is_patch = utils.create_template(mail, Platform.DISCORD)
+		except:
+			exc_str = utils.catch_err()
+			self.bot.logger.warning(exc_str)
+			await self.bot.send_log_file(url)
 
 		if is_patch:
 			m = await self.bot.send_patch_mail_interaction(
diff --git a/daemon/dscord/mailer/listener.py b/daemon/dscord/mailer/listener.py
index d986fbd..1764ea2 100644
--- a/daemon/dscord/mailer/listener.py
+++ b/daemon/dscord/mailer/listener.py
@@ -35,6 +35,7 @@ class Listener:
 		self.scraper = scraper
 		self.mutexes = mutexes
 		self.db = client.db
+		self.logger = client.logger
 		self.isRunnerFixed = False
 		self.runner = None
 
@@ -44,18 +45,15 @@ class Listener:
 		# Execute __run() once to avoid high latency at
 		# initilization.
 		#
-		print("Initialize listener...\n")
+		self.logger.info("Initialize listener...\n")
 		self.sched.start()
 		self.runner = self.sched.add_job(func=self.__run)
 
 
 	async def __run(self):
-		print("[__run]: Running...")
+		self.logger.info("Running...")
 		for url in self.db.get_atom_urls():
-			try:
-				await self.__handle_atom_url(url)
-			except:
-				print(traceback.format_exc())
+			await self.__handle_atom_url(url)
 
 		if not self.isRunnerFixed:
 			self.isRunnerFixed = True
@@ -71,8 +69,13 @@ class Listener:
 	async def __handle_atom_url(self, url):
 		urls = await self.scraper.get_new_threads_urls(url)
 		for url in urls:
-			mail = await self.scraper.get_email_from_url(url)
-			await self.__handle_mail(url, mail)
+			try:
+				mail = await self.scraper.get_email_from_url(url)
+				await self.__handle_mail(url, mail)
+			except:
+				exc_str = utils.catch_err()
+				self.client.logger.warning(exc_str)
+				await self.client.send_log_file(url)
 
 
 	async def __handle_mail(self, url, mail):
@@ -91,10 +94,8 @@ class Listener:
 	async def __send_to_discord(self, url, mail, dc_guild_id, dc_chat_id):
 		email_msg_id = utils.get_email_msg_id(mail)
 		if not email_msg_id:
-			#
-			# It doesn't have a Message-Id.
-			# A malformed email. Skip!
-			#
+			md = "email_msg_id not detected, skipping malformed email"
+			self.logger.debug(md)
 			return False
 
 		email_id = self.__get_email_id_sent(
@@ -102,13 +103,18 @@ class Listener:
 			dc_chat_id=dc_chat_id
 		)
 		if not email_id:
-			#
-			# Email has already been sent to Discord.
-			# Skip!
-			#
+			md = f"Skipping {email_id} because has already been sent to Discord"
+			self.logger.debug(md)
 			return False
 
-		text, files, is_patch = utils.create_template(mail, Platform.DISCORD)
+		try:
+			text, files, is_patch = utils.create_template(mail, Platform.DISCORD)
+		except:
+			exc_str = utils.catch_err()
+			self.client.logger.warning(exc_str)
+			await self.client.send_log_file(url)
+			return
+
 		reply_to = self.get_discord_reply(mail, dc_chat_id)
 		url = str(re.sub(r"/raw$", "", url))
 
-- 
Muhammad Rizki

[PATCH v1 16/16] Remove some unused imports

[Muhammad Rizki]

Remove them while it is not used again, from the previous version, they
are not removed.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/atom/utils.py                                          | 3 +--
 daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py | 2 --
 daemon/dscord/mailer/listener.py                              | 1 -
 3 files changed, 1 insertion(+), 5 deletions(-)

diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py
index 60f2c76..1181621 100644
--- a/daemon/atom/utils.py
+++ b/daemon/atom/utils.py
@@ -8,7 +8,7 @@ from enums import Platform
 
 from pyrogram.types import Chat, InlineKeyboardMarkup, InlineKeyboardButton
 from email.message import Message
-from typing import Dict, Union
+from typing import Union
 from slugify import slugify
 import traceback
 import hashlib
@@ -18,7 +18,6 @@ import re
 import shutil
 import httpx
 import html
-import quopri
 
 
 def get_email_msg_id(mail):
diff --git a/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py b/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py
index e4c171c..56330f3 100644
--- a/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py
+++ b/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py
@@ -3,8 +3,6 @@
 # Copyright (C) 2022  Muhammad Rizki <kiizuha@gnuweeb.org>
 #
 
-import asyncio
-import discord
 from discord.ext import commands
 from discord import Interaction
 from discord import app_commands
diff --git a/daemon/dscord/mailer/listener.py b/daemon/dscord/mailer/listener.py
index 1764ea2..24f0a6c 100644
--- a/daemon/dscord/mailer/listener.py
+++ b/daemon/dscord/mailer/listener.py
@@ -5,7 +5,6 @@
 #
 
 import asyncio
-import traceback
 import re
 from apscheduler.schedulers.asyncio import AsyncIOScheduler
 from discord import File
-- 
Muhammad Rizki

Re: [PATCH v1 00/16] Fix, improvement and implement a bot logger.

[Ammar Faizi]

On 11/5/22 1:09 AM, Muhammad Rizki wrote:
> There are 16 patches in this series:

This series is a mess. Some patches have title:
    
     [PATCH N/16]

some other patches have title:

     [PATCH N/15]

There are also duplicates patches. This series is a disaster to apply.
Please resend.

-- 
Ammar Faizi

Re: [PATCH v1 00/16] Fix, improvement and implement a bot logger.

[Muhammad Rizki]

On 07/11/2022 08.10, Ammar Faizi wrote:
> On 11/5/22 1:09 AM, Muhammad Rizki wrote:
>> There are 16 patches in this series:
> 
> This series is a mess. Some patches have title:
>      [PATCH N/16]
> 
> some other patches have title:
> 
>      [PATCH N/15]
> 
> There are also duplicates patches. This series is a disaster to apply.
> Please resend.
> 

Uhh, yes. IDK why this is happening. It could be when I want to send the 
patch with `git send-email` it causing SMTP error, and when I want to 
send the patch the second time I made a change in the commit too. IDK 
why its sending the patch twice even the first send-email is failure.

Re: [PATCH v1 00/16] Fix, improvement and implement a bot logger.

[Muhammad Rizki]

On 07/11/2022 08.10, Ammar Faizi wrote:
> On 11/5/22 1:09 AM, Muhammad Rizki wrote:
>> There are 16 patches in this series:
> 
> This series is a mess. Some patches have title:
>      [PATCH N/16]
> 
> some other patches have title:
> 
>      [PATCH N/15]
> 
> There are also duplicates patches. This series is a disaster to apply.
> Please resend.
> 

Anyway, you want me to RESEND this series first or just continue to the v2?

Re: [PATCH v1 00/16] Fix, improvement and implement a bot logger.

[Ammar Faizi]

On 11/7/22 8:28 AM, Muhammad Rizki wrote:
> Anyway, you want me to RESEND this series first or just continue to the v2

I'll just wait for v2.

-- 
Ammar Faizi