[PATCH v1 00/15] Everything about logger changes and some fixes

[PATCH v1 00/15] Everything about logger changes and some fixes

[Muhammad Rizki]

Hi sir,
This series is about refactor logger method, fix charset issue, and some
improvements. In this series we use a simple logging method that so
easily to manage logger config using .conf file.

- What's new?
  v1:
    1) Add return type annotation for the send_text_mail() and
       send_patch_email() for the Discord bot

- Fixes:
  v1:
    1) Unknown email payload charset `unknown-8bit` that is not in the
       Python's built-in `decode()`. So, we manually using `8bit`
       charset and force it using `surrogateescape`.

    2) Fix type annotations for both Discord and Telegram decorators that
       handle flood request.

    3) Implement DaemonException() and report_err() in the Telegram's
       scrape.py and Discord's get_lore_mail.py

- Improvements:
  v1:
    1) Simplify code to get the DB_PORT
    2) Remove some unnecessary comments


There are 15 patches in this series:
- Patch 1 is to simplify to get the DB_PORT from .env for the TG bot.
- Patch 2 is to simplify to get the DB_PORT from .env for the DC bot.
- Patch 3 is to add a telegram.logger.conf.
- Patch 4 is to add a discord.logger.conf.
- Patch 5 is to initialize the logger config for the Telegram bot.
- Patch 6 is to initialize the logger config for the Discord bot.
- Patch 7 is to fix the type annotations for the Telegram's decorator.
- Patch 8 is to remove some unnecessary comments.
- Patch 9 is to fix the type annotations for the Discord's decorator.
- Patch 10 is to add return type annotations for the Discord bot.
- Patch 11 is to implement DaemonException and report_err for TG scrape.
- Patch 12 is to fix `unknown-8bit` charset when decoding email payload.
- Patch 13 is to refactor logging method in all files for Telegram bot.
- Patch 14 is to implement DaemonException and report_err for DC bot.
- Patch 15 is to refactor logging method in all files for Discord bot.

Please give it a test, thanks!

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---

Muhammad Rizki (15):
  telegram: Simplify code to get DB_PORT from env
  discord: Simplify code to get DB_PORT from env
  telegram: logger: Add a telegram.logger.conf
  discord: logger: Add a discord.logger.conf
  telegram: logger: Initialize the configuration for the Telegram
    logger.
  discord: logger: Initialize the configuration for the Discord logger.
  telegram: fix: Fix the type annoations for the decorator
  discord: cleanup: Remove some unnecessary comments
  discord: fix: Fix the type annotations for the decorator
  discord: typing: Add return type annotations
  telegram: Implement DaemonException() and report_err() in scrape.py
  utils: fix: Fix charset issue for get_decoded_payload()
  telegram: logger: Refactor all logging method
  discord: Implement DaemonException and report_err in get_lore_mail.py
  discord: logger: Refactor all logging method

 daemon/atom/utils.py                          |  3 ++
 daemon/dc.py                                  | 20 ++++-------
 daemon/dscord/discord.logger.conf             | 36 +++++++++++++++++++
 daemon/dscord/gnuweeb/client.py               | 25 +++++++------
 daemon/dscord/gnuweeb/filters.py              | 32 ++++++++---------
 .../dscord/gnuweeb/plugins/events/on_ready.py |  5 ++-
 .../plugins/slash_commands/get_lore_mail.py   |  9 +++--
 daemon/dscord/mailer/listener.py              | 17 +++++----
 daemon/telegram/mailer/listener.py            | 15 ++++----
 daemon/telegram/packages/client.py            | 21 ++++++-----
 daemon/telegram/packages/decorator.py         | 31 ++++++++--------
 .../packages/plugins/commands/scrape.py       |  9 +++--
 daemon/telegram/telegram.logger.conf          | 36 +++++++++++++++++++
 daemon/tg.py                                  | 18 ++++------
 14 files changed, 182 insertions(+), 95 deletions(-)
 create mode 100644 daemon/dscord/discord.logger.conf
 create mode 100644 daemon/telegram/telegram.logger.conf


base-commit: 1ad0c452d2f87c086a76dd9ceee7a109c8874f57
--
Muhammad Rizki

[PATCH v1 01/15] telegram: Simplify code to get DB_PORT from env

[Muhammad Rizki]

Using `int(os.environ.get("DB_PORT", 3306))` is the best practice to get
the DB_PORT rather than using if statement. This change simplifies the
code.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/tg.py | 8 +-------
 1 file changed, 1 insertion(+), 7 deletions(-)

diff --git a/daemon/tg.py b/daemon/tg.py
index 9f36e54..6b2ec20 100644
--- a/daemon/tg.py
+++ b/daemon/tg.py
@@ -22,12 +22,6 @@ def main():
 	logger = BotLogger()
 	logger.init()
 
-	port = os.getenv("DB_PORT")
-	if not port:
-		port = 3306
-	else:
-		port = int(port)
-
 	client = DaemonTelegram(
 		"telegram/storage/EmailScraper",
 		api_id=int(os.getenv("API_ID")),
@@ -37,7 +31,7 @@ def main():
 		conn=connector.connect(
 			host=os.getenv("DB_HOST"),
 			user=os.getenv("DB_USER"),
-			port=port,
+			port=int(os.environ.get("DB_PORT", 3306)),
 			password=os.getenv("DB_PASS"),
 			database=os.getenv("DB_NAME")
 		),
-- 
Muhammad Rizki

[PATCH v1 02/15] discord: Simplify code to get DB_PORT from env

[Muhammad Rizki]

Using `int(os.environ.get("DB_PORT", 3306))` is the best practice to get
the DB_PORT rather than using if statement. This change simplifies the
code.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/dc.py | 8 +-------
 1 file changed, 1 insertion(+), 7 deletions(-)

diff --git a/daemon/dc.py b/daemon/dc.py
index a9d5ebd..7d34837 100644
--- a/daemon/dc.py
+++ b/daemon/dc.py
@@ -29,17 +29,11 @@ def main():
 	logger = BotLogger(Platform.DISCORD)
 	logger.init()
 
-	port = os.getenv("DB_PORT")
-	if not port:
-		port = 3306
-	else:
-		port = int(port)
-
 	client = GWClient(
 		db_conn=connector.connect(
 			host=os.getenv("DB_HOST"),
 			user=os.getenv("DB_USER"),
-			port=port,
+			port=int(os.environ.get("DB_PORT", 3306)),
 			password=os.getenv("DB_PASS"),
 			database=os.getenv("DB_NAME")
 		),
-- 
Muhammad Rizki

[PATCH v1 03/15] telegram: logger: Add a telegram.logger.conf

[Muhammad Rizki]

Add a logger configuration file for improved configuration, debugging,
and management. This method is cleaner than before and the BotLogger()
will be removed in the future.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/telegram/telegram.logger.conf | 36 ++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)
 create mode 100644 daemon/telegram/telegram.logger.conf

diff --git a/daemon/telegram/telegram.logger.conf b/daemon/telegram/telegram.logger.conf
new file mode 100644
index 0000000..d0c0de3
--- /dev/null
+++ b/daemon/telegram/telegram.logger.conf
@@ -0,0 +1,36 @@
+[loggers]
+keys=root,telegram
+
+[handlers]
+keys=consoleHandler,fileHandler
+
+[formatters]
+keys=sampleFormatter,dictFormatter
+
+[logger_root]
+level=INFO
+handlers=consoleHandler,fileHandler
+
+[logger_telegram]
+level=INFO
+handlers=consoleHandler
+qualname=telegram
+propagate=0
+
+[handler_consoleHandler]
+class=StreamHandler
+level=INFO
+formatter=sampleFormatter
+args=(sys.stdout,)
+
+[handler_fileHandler]
+class=FileHandler
+level=WARNING
+formatter=dictFormatter
+args=("telegram/storage/telegram.log",)
+
+[formatter_sampleFormatter]
+format=%(asctime)s - %(name)s: %(funcName)s - %(levelname)s - %(message)s
+
+[formatter_dictFormatter]
+format={"time": "%(asctime)s", "func": "%(name)s: %(funcName)s", "path": "%(pathname)s", "level": "%(levelname)s", "msg": "%(message)s"}
-- 
Muhammad Rizki

[PATCH v1 04/15] discord: logger: Add a discord.logger.conf

[Muhammad Rizki]

Add a logger configuration file for improved configuration, debugging,
and management. This method is cleaner than before and the BotLogger()
will be removed in the future.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/dscord/discord.logger.conf | 36 +++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)
 create mode 100644 daemon/dscord/discord.logger.conf

diff --git a/daemon/dscord/discord.logger.conf b/daemon/dscord/discord.logger.conf
new file mode 100644
index 0000000..97b5e11
--- /dev/null
+++ b/daemon/dscord/discord.logger.conf
@@ -0,0 +1,36 @@
+[loggers]
+keys=root,dscord
+
+[handlers]
+keys=consoleHandler,fileHandler
+
+[formatters]
+keys=sampleFormatter,dictFormatter
+
+[logger_root]
+level=INFO
+handlers=consoleHandler,fileHandler
+
+[logger_dscord]
+level=INFO
+handlers=consoleHandler
+qualname=dscord
+propagate=0
+
+[handler_consoleHandler]
+class=StreamHandler
+level=INFO
+formatter=sampleFormatter
+args=(sys.stdout,)
+
+[handler_fileHandler]
+class=FileHandler
+level=WARNING
+formatter=dictFormatter
+args=("dscord/storage/discord.log",)
+
+[formatter_sampleFormatter]
+format=%(asctime)s - %(name)s: %(funcName)s - %(levelname)s - %(message)s
+
+[formatter_dictFormatter]
+format={"time": "%(asctime)s", "func": "%(name)s: %(funcName)s", "path": "%(pathname)s", "level": "%(levelname)s", "msg": "%(message)s"}
-- 
Muhammad Rizki

[PATCH v1 05/15] telegram: logger: Initialize the configuration for the Telegram logger.

[Muhammad Rizki]

This commit adds the configuration for the logger and sets the logger
level of other loggers, such as pyrogram, to the WARNING level. Changing
the level is to disable some of their log messages, so that only our bot
logs to the console.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/tg.py | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/daemon/tg.py b/daemon/tg.py
index 6b2ec20..26f8374 100644
--- a/daemon/tg.py
+++ b/daemon/tg.py
@@ -14,6 +14,8 @@ from telegram.mailer import BotMutexes
 from telegram.mailer import Bot
 from logger import BotLogger
 import os
+import logging
+import logging.config
 
 
 def main():
@@ -22,6 +24,10 @@ def main():
 	logger = BotLogger()
 	logger.init()
 
+	logging.config.fileConfig("telegram/telegram.logger.conf")
+	logging.getLogger("apscheduler").setLevel(logging.WARNING)
+	logging.getLogger("pyrogram").setLevel(logging.WARNING)
+
 	client = DaemonTelegram(
 		"telegram/storage/EmailScraper",
 		api_id=int(os.getenv("API_ID")),
-- 
Muhammad Rizki

[PATCH v1 06/15] discord: logger: Initialize the configuration for the Discord logger.

[Muhammad Rizki]

This commit adds the configuration for the logger and sets the logger
level of other loggers, such as pyrogram, to the WARNING level. Changing
the level is to disable some of their log messages, so that only our bot
logs to the console.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/dc.py | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/daemon/dc.py b/daemon/dc.py
index 7d34837..f4eeefb 100644
--- a/daemon/dc.py
+++ b/daemon/dc.py
@@ -4,6 +4,8 @@
 #
 
 import os
+import logging
+import logging.config
 from dotenv import load_dotenv
 from mysql import connector
 from apscheduler.schedulers.asyncio import AsyncIOScheduler
@@ -29,6 +31,10 @@ def main():
 	logger = BotLogger(Platform.DISCORD)
 	logger.init()
 
+	logging.config.fileConfig("dscord/discord.logger.conf")
+	logging.getLogger("apscheduler").setLevel(logging.WARNING)
+	logging.getLogger("discord").setLevel(logging.WARNING)
+
 	client = GWClient(
 		db_conn=connector.connect(
 			host=os.getenv("DB_HOST"),
-- 
Muhammad Rizki

[PATCH v1 07/15] telegram: fix: Fix the type annoations for the decorator

[Muhammad Rizki]

In the previous commit, I created a TODO to fix the typing. In this
commit, I have fixed the decorator's typing.

Fixes: ba02aa3376a4 ("daemon: Add typing in decorator")
Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/telegram/packages/decorator.py | 17 ++++++++---------
 1 file changed, 8 insertions(+), 9 deletions(-)

diff --git a/daemon/telegram/packages/decorator.py b/daemon/telegram/packages/decorator.py
index efcfc38..6bfd8bc 100644
--- a/daemon/telegram/packages/decorator.py
+++ b/daemon/telegram/packages/decorator.py
@@ -4,25 +4,24 @@
 #
 
 from pyrogram.errors.exceptions.flood_420 import FloodWait
-from pyrogram.types import Message
-from typing import Any, Callable, TypeVar
+from typing import Any, Callable, TypeVar, Coroutine
+from typing_extensions import ParamSpec, ParamSpecArgs, ParamSpecKwargs
 from functools import wraps
 import re
 import asyncio
 
 __all__ = ["handle_flood"]
 
-T = TypeVar("T", bound=Message)
+T = TypeVar("T")
+P = ParamSpec("P")
 
-#
-# TODO(Muhammad Rizki): Add more typing for @handle_flood
-#
-def handle_flood(func: Callable[[T], T]) -> Callable[[T], T]:
+
+def handle_flood(func: Callable[P, Coroutine[Any,Any,T]]) -> Callable[P, Coroutine[Any,Any,T]]:
 	@wraps(func)
-	async def callback(*args: Any) -> Any:
+	async def callback(*args: ParamSpecArgs, **kwargs: ParamSpecKwargs) -> T:
 		while True:
 			try:
-				return await func(*args)
+				return await func(*args, **kwargs)
 			except FloodWait as e:
 				# Calling logger attr from the DaemonTelegram() class
 				logger = args[0].logger
-- 
Muhammad Rizki

[PATCH v1 08/15] discord: cleanup: Remove some unnecessary comments

[Muhammad Rizki]

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/dscord/gnuweeb/filters.py | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/daemon/dscord/gnuweeb/filters.py b/daemon/dscord/gnuweeb/filters.py
index 1fdb70d..b342c7a 100644
--- a/daemon/dscord/gnuweeb/filters.py
+++ b/daemon/dscord/gnuweeb/filters.py
@@ -4,16 +4,13 @@
 #
 
 
-# built-in/dev package imports
 import asyncio
 from typing import Any
 from functools import wraps
 
-# Discord imports
 import discord
 from discord import Interaction
 
-# gnuweeb package import
 from dscord import config
 from logger import BotLogger
 
-- 
Muhammad Rizki

[PATCH v1 09/15] discord: fix: Fix the type annotations for the decorator

[Muhammad Rizki]

Previously, these decorators did not use type annotations. In this
commit, I have implemented them.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/dscord/gnuweeb/filters.py | 17 +++++++++++------
 1 file changed, 11 insertions(+), 6 deletions(-)

diff --git a/daemon/dscord/gnuweeb/filters.py b/daemon/dscord/gnuweeb/filters.py
index b342c7a..28b492c 100644
--- a/daemon/dscord/gnuweeb/filters.py
+++ b/daemon/dscord/gnuweeb/filters.py
@@ -5,7 +5,8 @@
 
 
 import asyncio
-from typing import Any
+from typing import Any, Callable, TypeVar, Coroutine
+from typing_extensions import ParamSpec, ParamSpecArgs, ParamSpecKwargs
 from functools import wraps
 
 import discord
@@ -15,9 +16,13 @@ from dscord import config
 from logger import BotLogger
 
 
-def lore_admin(func):
+T = TypeVar("T")
+P = ParamSpec("P")
+
+
+def lore_admin(func: Callable[P, Coroutine[Any,Any,T]]) -> Callable[P, Coroutine[Any,Any,T]]:
 	@wraps(func)
-	async def callback(*args: Any, **kwargs: Any) -> Any:
+	async def callback(*args: ParamSpecArgs, **kwargs: ParamSpecKwargs) -> T:
 		i: "Interaction" = args[1]
 		user_roles = [role.id for role in i.user.roles]
 
@@ -33,12 +38,12 @@ def lore_admin(func):
 	return callback
 
 
-def wait_on_limit(func):
+def wait_on_limit(func: Callable[P, Coroutine[Any,Any,T]]) -> Callable[P, Coroutine[Any,Any,T]]:
 	@wraps(func)
-	async def callback(*args: Any) -> Any:
+	async def callback(*args: ParamSpecArgs, **kwargs: ParamSpecKwargs) -> T:
 		while True:
 			try:
-				return await func(*args)
+				return await func(*args, **kwargs)
 			except discord.errors.RateLimited as e:
 				# Calling logger attr from the GWClient() class
 				logger = args[0].logger
-- 
Muhammad Rizki

[PATCH v1 10/15] discord: typing: Add return type annotations

[Muhammad Rizki]

Add return type annotations for send_text_email() & send_patch_email().
This improves code readability and helps with understanding the type of
object returned by these functions.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/dscord/gnuweeb/client.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/daemon/dscord/gnuweeb/client.py b/daemon/dscord/gnuweeb/client.py
index 78ec3f9..7d6433b 100644
--- a/daemon/dscord/gnuweeb/client.py
+++ b/daemon/dscord/gnuweeb/client.py
@@ -4,7 +4,7 @@
 #
 
 import discord
-from discord import Interaction
+from discord import Interaction, Message
 from discord.ext import commands
 from discord import Intents
 from typing import Union
@@ -60,7 +60,7 @@ class GWClient(commands.Bot):
 
 	@filters.wait_on_limit
 	async def send_text_email(self, guild_id: int, chat_id: int, text: str,
-				reply_to: Union[int, None] = None, url: str = None):
+				reply_to: Union[int, None] = None, url: str = None) -> Message:
 		self.logger.debug("[send_text_email]")
 		channel = self.get_channel(chat_id)
 
@@ -77,7 +77,7 @@ class GWClient(commands.Bot):
 
 	@filters.wait_on_limit
 	async def send_patch_email(self, mail, guild_id: int, chat_id: int, text: str,
-				reply_to: Union[int, None] = None, url: str = None):
+				reply_to: Union[int, None] = None, url: str = None) -> Message:
 		self.logger.debug("[send_patch_email]")
 		tmp, doc, caption, url = utils.prepare_patch(
 			mail, text, url, Platform.DISCORD
-- 
Muhammad Rizki

[PATCH v1 11/15] telegram: Implement DaemonException() and report_err() in scrape.py

[Muhammad Rizki]

In the last series, I didn't care so much to look into the Telegram's
scrape.py command. I've implemented them same as in the listener.py.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/telegram/packages/plugins/commands/scrape.py | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/daemon/telegram/packages/plugins/commands/scrape.py b/daemon/telegram/packages/plugins/commands/scrape.py
index 89581b6..ddcf7f1 100644
--- a/daemon/telegram/packages/plugins/commands/scrape.py
+++ b/daemon/telegram/packages/plugins/commands/scrape.py
@@ -11,6 +11,7 @@ from atom import Scraper
 from atom import utils
 from enums import Platform
 from telegram import config
+from exceptions import DaemonException
 import re
 import asyncio
 
@@ -41,9 +42,11 @@ async def scrap_email(c: DaemonTelegram, m: Message):
 		mail = await s.get_email_from_url(url)
 		text, files, is_patch = utils.create_template(mail, Platform.TELEGRAM)
 	except:
-		exc_str = utils.catch_err()
-		c.logger.warning(exc_str)
-		await c.send_log_file(url)
+		e = DaemonException()
+		e.set_thread_url(url)
+		e.set_message(utils.catch_err())
+		await c.report_err(e)
+		return
 
 	if is_patch:
 		m = await c.send_patch_email(
-- 
Muhammad Rizki

[PATCH v1 12/15] utils: fix: Fix charset issue for get_decoded_payload()

[Muhammad Rizki]

Fix the `LookupError` that is thrown by the `decode()` function. The
`unknown-8bit` charset is not found in the built-in `decode()` function
in Python, so we must manually fix it by using an if statement and
decoding using the `ascii` charset while forcing errors using
`surrogateescape`.

Sample: https://lore.kernel.org/all/Y8Am5wAxC48N12PE@quaddy.sgn/raw

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/atom/utils.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py
index 96a10a0..73c8978 100644
--- a/daemon/atom/utils.py
+++ b/daemon/atom/utils.py
@@ -323,6 +323,9 @@ def get_decoded_payload(payload: Message):
 		return payload.get_payload(decode=True) \
 			.decode(errors="replace")
 
+	if charset == "unknown-8bit":
+		return p.encode().decode("utf-8", "surrogateescape")
+
 	return p.encode().decode(charset, errors="replace")
 
 
-- 
Muhammad Rizki

[PATCH v1 13/15] telegram: logger: Refactor all logging method

[Muhammad Rizki]

We use a simple logging method by using file config and replace all
custom logger method to built-in logging method. This method is more
cleaner and less code wasteful.

The reason why we should name the logger like:

    `logging.getLogger("telegram")`

Because, we need to identify which bot platform that currently running.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/telegram/mailer/listener.py    | 15 +++++++++------
 daemon/telegram/packages/client.py    | 21 +++++++++++++--------
 daemon/telegram/packages/decorator.py | 14 +++++++-------
 daemon/tg.py                          |  6 ------
 4 files changed, 29 insertions(+), 27 deletions(-)

diff --git a/daemon/telegram/mailer/listener.py b/daemon/telegram/mailer/listener.py
index 2e8eeda..b34c74d 100644
--- a/daemon/telegram/mailer/listener.py
+++ b/daemon/telegram/mailer/listener.py
@@ -15,6 +15,10 @@ from atom import utils
 from enums import Platform
 import asyncio
 import re
+import logging
+
+
+log = logging.getLogger("telegram")
 
 
 class BotMutexes():
@@ -30,7 +34,6 @@ class Bot():
 		self.scraper = scraper
 		self.mutexes = mutexes
 		self.db = client.db
-		self.logger = client.logger
 		self.isRunnerFixed = False
 
 
@@ -57,8 +60,8 @@ class Bot():
 		# TODO(ammarfaizi2):
 		# Ideally, we also want to log and report this situation.
 		#
-		self.logger.error(f"Database error: {str(e)}")
-		self.logger.info("Reconnecting to the database...")
+		log.error(f"Database error: {str(e)}")
+		log.info("Reconnecting to the database...")
 
 		#
 		# Don't do this too often to avoid reconnect burst.
@@ -71,7 +74,7 @@ class Bot():
 
 
 	async def __run(self):
-		self.logger.info("Running...")
+		log.info("Running...")
 		url = None
 		try:
 			for url in self.db.get_atom_urls():
@@ -128,14 +131,14 @@ class Bot():
 
 		if not email_msg_id:
 			md = "email_msg_id not detected, skipping malformed email"
-			self.logger.debug(md)
+			log.debug(md)
 			return False
 
 		email_id = self.__mail_id_from_db(email_msg_id,
 							tg_chat_id)
 		if not email_id:
 			md = f"Skipping {email_id} because has already been sent to Telegram"
-			self.logger.debug(md)
+			log.debug(md)
 			return False
 
 		text, files, is_patch = utils.create_template(mail, Platform.TELEGRAM)
diff --git a/daemon/telegram/packages/client.py b/daemon/telegram/packages/client.py
index a58aaf8..7e898c5 100644
--- a/daemon/telegram/packages/client.py
+++ b/daemon/telegram/packages/client.py
@@ -3,41 +3,46 @@
 # Copyright (C) 2022  Muhammad Rizki <riskimuhammmad1@gmail.com>
 #
 
+import logging
 from pyrogram import Client
 from pyrogram.enums import ParseMode
 from pyrogram.types import Message, InlineKeyboardMarkup, InlineKeyboardButton
 from typing import Union
 from email.message import Message
+
 from atom import utils
 from enums import Platform
-from logger import BotLogger
 from telegram import config
 from telegram.database import DB
 from .decorator import handle_flood
 from exceptions import DaemonException
 
 
+log = logging.getLogger("telegram")
+
+
 class DaemonTelegram(Client):
 	def __init__(self, name: str, api_id: int,
-		api_hash: str, conn, logger: BotLogger,
-		**kwargs
+		api_hash: str, conn, **kwargs
 	):
 		super().__init__(name, api_id,
 				api_hash, **kwargs)
 		self.db = DB(conn)
-		self.logger = logger
 
 
 	async def report_err(self, e: DaemonException):
 		capt = f"Atom URL: {e.atom_url}\n"
 		capt += f"Thread URL: {e.thread_url}"
-		self.logger.warning(e.original_exception)
+		log.warning(e.original_exception)
 		await self.send_log_file(capt)
 
 
 	@handle_flood
 	async def send_log_file(self, caption: str):
-		filename = self.logger.handlers[0].baseFilename
+		for handler in log.root.handlers:
+			if isinstance(handler, logging.FileHandler):
+				filename = handler.baseFilename
+
 		await self.send_document(
 			config.LOG_CHANNEL_ID,
 			filename,
@@ -54,7 +59,7 @@ class DaemonTelegram(Client):
 		url: str = None,
 		parse_mode: ParseMode = ParseMode.HTML
 	) -> Message:
-		self.logger.debug("[send_text_email]")
+		log.debug("[send_text_email]")
 		return await self.send_message(
 			chat_id=chat_id,
 			text=text,
@@ -79,7 +84,7 @@ class DaemonTelegram(Client):
 		url: str = None,
 		parse_mode: ParseMode = ParseMode.HTML
 	) -> Message:
-		self.logger.debug("[send_patch_email]")
+		log.debug("[send_patch_email]")
 		tmp, doc, caption, url = utils.prepare_patch(
 			mail, text, url, Platform.TELEGRAM
 		)
diff --git a/daemon/telegram/packages/decorator.py b/daemon/telegram/packages/decorator.py
index 6bfd8bc..64162f6 100644
--- a/daemon/telegram/packages/decorator.py
+++ b/daemon/telegram/packages/decorator.py
@@ -9,9 +9,12 @@ from typing_extensions import ParamSpec, ParamSpecArgs, ParamSpecKwargs
 from functools import wraps
 import re
 import asyncio
+import logging
+
 
 __all__ = ["handle_flood"]
 
+log = logging.getLogger("telegram")
 T = TypeVar("T")
 P = ParamSpec("P")
 
@@ -23,19 +26,16 @@ def handle_flood(func: Callable[P, Coroutine[Any,Any,T]]) -> Callable[P, Corouti
 			try:
 				return await func(*args, **kwargs)
 			except FloodWait as e:
-				# Calling logger attr from the DaemonTelegram() class
-				logger = args[0].logger
-
-				_flood_exceptions(e, logger)
-				logger.info("Woken up from flood wait...")
+				_flood_exceptions(e)
+				log.info("Woken up from flood wait...")
 	return callback
 
 
-async def _flood_exceptions(e, logger):
+async def _flood_exceptions(e):
 	x = re.search(r"A wait of (\d+) seconds is required", str(e))
 	if not x:
 		raise e
 
 	n = int(x.group(1))
-	logger.info(f"Sleeping for {n} seconds due to Telegram limit")
+	log.info(f"Sleeping for {n} seconds due to Telegram limit")
 	await asyncio.sleep(n)
diff --git a/daemon/tg.py b/daemon/tg.py
index 26f8374..5233faa 100644
--- a/daemon/tg.py
+++ b/daemon/tg.py
@@ -7,12 +7,10 @@
 from apscheduler.schedulers.asyncio import AsyncIOScheduler
 from dotenv import load_dotenv
 from mysql import connector
-from pyrogram import idle
 from atom import Scraper
 from telegram.packages import DaemonTelegram
 from telegram.mailer import BotMutexes
 from telegram.mailer import Bot
-from logger import BotLogger
 import os
 import logging
 import logging.config
@@ -21,9 +19,6 @@ import logging.config
 def main():
 	load_dotenv("telegram.env")
 
-	logger = BotLogger()
-	logger.init()
-
 	logging.config.fileConfig("telegram/telegram.logger.conf")
 	logging.getLogger("apscheduler").setLevel(logging.WARNING)
 	logging.getLogger("pyrogram").setLevel(logging.WARNING)
@@ -33,7 +28,6 @@ def main():
 		api_id=int(os.getenv("API_ID")),
 		api_hash=os.getenv("API_HASH"),
 		bot_token=os.getenv("BOT_TOKEN"),
-		logger=logger,
 		conn=connector.connect(
 			host=os.getenv("DB_HOST"),
 			user=os.getenv("DB_USER"),
-- 
Muhammad Rizki

[PATCH v1 14/15] discord: Implement DaemonException and report_err in get_lore_mail.py

[Muhammad Rizki]

In the last series, I didn't care so much to look into the Discord's
get_lore_mail.py slash command. I've implemented them same as in the
listener.py.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 .../gnuweeb/plugins/slash_commands/get_lore_mail.py      | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py b/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py
index 56330f3..84dca83 100644
--- a/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py
+++ b/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py
@@ -10,6 +10,7 @@ from discord import app_commands
 from atom import utils
 from atom import Scraper
 from enums import Platform
+from exceptions import DaemonException
 
 
 class GetLoreSC(commands.Cog):
@@ -29,9 +30,11 @@ class GetLoreSC(commands.Cog):
 			mail = await s.get_email_from_url(url)
 			text, _, is_patch = utils.create_template(mail, Platform.DISCORD)
 		except:
-			exc_str = utils.catch_err()
-			self.bot.logger.warning(exc_str)
-			await self.bot.send_log_file(url)
+			e = DaemonException()
+			e.set_thread_url(url)
+			e.set_message(utils.catch_err())
+			await self.bot.report_err(e)
+			return
 
 		if is_patch:
 			m = await self.bot.send_patch_mail_interaction(
-- 
Muhammad Rizki

[PATCH v1 15/15] discord: logger: Refactor all logging method

[Muhammad Rizki]

Same as the telegram refactors. We use a simple logging method by using
file config and replace all custom logger method to built-in logging
method. This method is more cleaner and less code wasteful.

The reason why we should name the logger like:

    `logging.getLogger("dscord")`

Because, if we use "discord" as the logger name, it will conflict the
discord.py's logger, that's why we need to disable the discord.py's
logger not to interrupt our logger.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/dc.py                                  |  8 +-------
 daemon/dscord/gnuweeb/client.py               | 19 ++++++++++++-------
 daemon/dscord/gnuweeb/filters.py              | 12 +++++-------
 .../dscord/gnuweeb/plugins/events/on_ready.py |  5 ++++-
 daemon/dscord/mailer/listener.py              | 17 ++++++++++-------
 5 files changed, 32 insertions(+), 29 deletions(-)

diff --git a/daemon/dc.py b/daemon/dc.py
index f4eeefb..dade9f1 100644
--- a/daemon/dc.py
+++ b/daemon/dc.py
@@ -13,8 +13,6 @@ from apscheduler.schedulers.asyncio import AsyncIOScheduler
 from dscord.gnuweeb import GWClient
 from dscord.mailer import Listener
 from dscord.mailer import Mutexes
-from enums.platform import Platform
-from logger import BotLogger
 from atom import Scraper
 
 
@@ -28,9 +26,6 @@ def main():
 		}
 	)
 
-	logger = BotLogger(Platform.DISCORD)
-	logger.init()
-
 	logging.config.fileConfig("dscord/discord.logger.conf")
 	logging.getLogger("apscheduler").setLevel(logging.WARNING)
 	logging.getLogger("discord").setLevel(logging.WARNING)
@@ -42,8 +37,7 @@ def main():
 			port=int(os.environ.get("DB_PORT", 3306)),
 			password=os.getenv("DB_PASS"),
 			database=os.getenv("DB_NAME")
-		),
-		logger=logger
+		)
 	)
 
 	mailer = Listener(
diff --git a/daemon/dscord/gnuweeb/client.py b/daemon/dscord/gnuweeb/client.py
index 7d6433b..a04fd26 100644
--- a/daemon/dscord/gnuweeb/client.py
+++ b/daemon/dscord/gnuweeb/client.py
@@ -3,6 +3,7 @@
 # Copyright (C) 2022  Muhammad Rizki <kiizuha@gnuweeb.org>
 #
 
+import logging
 import discord
 from discord import Interaction, Message
 from discord.ext import commands
@@ -14,18 +15,19 @@ from . import models
 from atom import utils
 from enums import Platform
 from exceptions import DaemonException
-from logger.log import BotLogger
 from dscord.config import ACTIVITY_NAME, LOG_CHANNEL_ID
 from dscord.database import DB
 
 
+log = logging.getLogger("dscord")
+
+
 class GWClient(commands.Bot):
-	def __init__(self, db_conn, logger: BotLogger) -> None:
+	def __init__(self, db_conn) -> None:
 		self.db = DB(db_conn)
 		intents = Intents.default()
 		intents.message_content = True
 		self.mailer = None
-		self.logger = logger
 		super().__init__(
 			command_prefix=["$", "."],
 			description="Just a bot for receiving lore emails.",
@@ -42,7 +44,7 @@ class GWClient(commands.Bot):
 
 
 	async def report_err(self, e: DaemonException):
-		self.logger.warning(e.original_exception)
+		log.warning(e.original_exception)
 		capt = f"Atom URL: {e.atom_url}\n"
 		capt += f"Thread URL: {e.thread_url}"
 		await self.send_log_file(capt)
@@ -50,7 +52,10 @@ class GWClient(commands.Bot):
 
 	@filters.wait_on_limit
 	async def send_log_file(self, caption: str):
-		filename = self.logger.handlers[0].baseFilename
+		for handler in log.root.handlers:
+			if isinstance(handler, logging.FileHandler):
+				filename = handler.baseFilename
+
 		channel = self.get_channel(LOG_CHANNEL_ID)
 		await channel.send(
 			content=caption,
@@ -61,7 +66,7 @@ class GWClient(commands.Bot):
 	@filters.wait_on_limit
 	async def send_text_email(self, guild_id: int, chat_id: int, text: str,
 				reply_to: Union[int, None] = None, url: str = None) -> Message:
-		self.logger.debug("[send_text_email]")
+		log.debug("[send_text_email]")
 		channel = self.get_channel(chat_id)
 
 		return await channel.send(
@@ -78,7 +83,7 @@ class GWClient(commands.Bot):
 	@filters.wait_on_limit
 	async def send_patch_email(self, mail, guild_id: int, chat_id: int, text: str,
 				reply_to: Union[int, None] = None, url: str = None) -> Message:
-		self.logger.debug("[send_patch_email]")
+		log.debug("[send_patch_email]")
 		tmp, doc, caption, url = utils.prepare_patch(
 			mail, text, url, Platform.DISCORD
 		)
diff --git a/daemon/dscord/gnuweeb/filters.py b/daemon/dscord/gnuweeb/filters.py
index 28b492c..ccd13e7 100644
--- a/daemon/dscord/gnuweeb/filters.py
+++ b/daemon/dscord/gnuweeb/filters.py
@@ -5,6 +5,7 @@
 
 
 import asyncio
+import logging
 from typing import Any, Callable, TypeVar, Coroutine
 from typing_extensions import ParamSpec, ParamSpecArgs, ParamSpecKwargs
 from functools import wraps
@@ -13,9 +14,9 @@ import discord
 from discord import Interaction
 
 from dscord import config
-from logger import BotLogger
 
 
+log = logging.getLogger("dscord")
 T = TypeVar("T")
 P = ParamSpec("P")
 
@@ -45,15 +46,12 @@ def wait_on_limit(func: Callable[P, Coroutine[Any,Any,T]]) -> Callable[P, Corout
 			try:
 				return await func(*args, **kwargs)
 			except discord.errors.RateLimited as e:
-				# Calling logger attr from the GWClient() class
-				logger = args[0].logger
-
 				_flood_exceptions(e)
-				logger.info("Woken up from flood wait...")
+				log.info("Woken up from flood wait...")
 	return callback
 
 
-async def _flood_exceptions(e: "discord.errors.RateLimited", logger: BotLogger):
+async def _flood_exceptions(e: "discord.errors.RateLimited"):
 	wait = e.retry_after
-	logger.info(f"Sleeping for {wait} seconds due to Discord limit")
+	log.info(f"Sleeping for {wait} seconds due to Discord limit")
 	await asyncio.sleep(wait)
diff --git a/daemon/dscord/gnuweeb/plugins/events/on_ready.py b/daemon/dscord/gnuweeb/plugins/events/on_ready.py
index e7f63cd..4248aad 100644
--- a/daemon/dscord/gnuweeb/plugins/events/on_ready.py
+++ b/daemon/dscord/gnuweeb/plugins/events/on_ready.py
@@ -3,9 +3,12 @@
 # Copyright (C) 2022  Muhammad Rizki <kiizuha@gnuweeb.org>
 #
 
+import logging
 from discord.ext import commands
 
 
+log = logging.getLogger("dscord")
+
 class OnReady(commands.Cog):
 	def __init__(self, bot: "commands.Bot") -> None:
 		self.bot = bot
@@ -23,4 +26,4 @@ class OnReady(commands.Cog):
 		t += f"Send `{prefix}sync` message to the Discord channel "
 		t += "where the bot is running.\n"
 
-		self.bot.logger.info(t)
+		log.info(t)
diff --git a/daemon/dscord/mailer/listener.py b/daemon/dscord/mailer/listener.py
index fc066b7..25e5715 100644
--- a/daemon/dscord/mailer/listener.py
+++ b/daemon/dscord/mailer/listener.py
@@ -6,6 +6,7 @@
 
 import asyncio
 import re
+import logging
 from mysql.connector.errors import OperationalError, DatabaseError
 from apscheduler.schedulers.asyncio import AsyncIOScheduler
 from discord import File
@@ -18,6 +19,9 @@ from enums import Platform
 from exceptions import DaemonException
 
 
+log = logging.getLogger("dscord")
+
+
 class Mutexes:
 	def __init__(self):
 		self.lock = asyncio.Lock()
@@ -36,7 +40,6 @@ class Listener:
 		self.scraper = scraper
 		self.mutexes = mutexes
 		self.db = client.db
-		self.logger = client.logger
 		self.isRunnerFixed = False
 		self.runner = None
 
@@ -46,7 +49,7 @@ class Listener:
 		# Execute __run() once to avoid high latency at
 		# initilization.
 		#
-		self.logger.info("Initialize listener...\n")
+		log.info("Initialize listener...\n")
 		self.sched.start()
 		self.runner = self.sched.add_job(func=self.__run)
 
@@ -56,8 +59,8 @@ class Listener:
 		# TODO(ammarfaizi2):
 		# Ideally, we also want to log and report this situation.
 		#
-		self.logger.error(f"Database error: {str(e)}")
-		self.logger.info("Reconnecting to the database...")
+		log.error(f"Database error: {str(e)}")
+		log.info("Reconnecting to the database...")
 
 		#
 		# Don't do this too often to avoid reconnect burst.
@@ -70,7 +73,7 @@ class Listener:
 
 
 	async def __run(self):
-		self.logger.info("Running...")
+		log.info("Running...")
 		url = None
 
 		try:
@@ -128,7 +131,7 @@ class Listener:
 		email_msg_id = utils.get_email_msg_id(mail)
 		if not email_msg_id:
 			md = "email_msg_id not detected, skipping malformed email"
-			self.logger.debug(md)
+			log.debug(md)
 			return False
 
 		email_id = self.__get_email_id_sent(
@@ -137,7 +140,7 @@ class Listener:
 		)
 		if not email_id:
 			md = f"Skipping {email_id} because has already been sent to Discord"
-			self.logger.debug(md)
+			log.debug(md)
 			return False
 
 		text, files, is_patch = utils.create_template(mail, Platform.DISCORD)
-- 
Muhammad Rizki

Re: [PATCH v1 00/15] Everything about logger changes and some fixes

[Ammar Faizi]

On Wed, Jan 18, 2023 at 05:12:01AM +0700, Muhammad Rizki wrote:
> Hi sir,
> This series is about refactor logger method, fix charset issue, and some
> improvements. In this series we use a simple logging method that so
> easily to manage logger config using .conf file.
> 
> - What's new?
>   v1:
>     1) Add return type annotation for the send_text_mail() and
>        send_patch_email() for the Discord bot

This one breaks runtime:

    Traceback (most recent call last):
      File "/root/lore-daemon2/daemon/tg.py", line 11, in <module>
        from telegram.packages import DaemonTelegram
      File "/root/lore-daemon2/daemon/telegram/packages/__init__.py", line 1, in <module>
        from .client import DaemonTelegram
      File "/root/lore-daemon2/daemon/telegram/packages/client.py", line 17, in <module>
        from .decorator import handle_flood
      File "/root/lore-daemon2/daemon/telegram/packages/decorator.py", line 8, in <module>
        from typing_extensions import ParamSpec, ParamSpecArgs, ParamSpecKwargs
    ModuleNotFoundError: No module named 'typing_extensions'

Apparently, that happens because 'typing_extensions' module is missing.
But you didn't add that to the requirements.txt file. Apart from that,
I don't see anything immediately wrong. Overall, it looks good to me.

Please update the requirements.txt file together with that module
import addition.

-- 
Ammar Faizi

Re: [PATCH v1 00/15] Everything about logger changes and some fixes

[Muhammad Rizki]

On 18/01/2023 06.10, Ammar Faizi wrote:
> On Wed, Jan 18, 2023 at 05:12:01AM +0700, Muhammad Rizki wrote:
>> Hi sir,
>> This series is about refactor logger method, fix charset issue, and some
>> improvements. In this series we use a simple logging method that so
>> easily to manage logger config using .conf file.
>>
>> - What's new?
>>    v1:
>>      1) Add return type annotation for the send_text_mail() and
>>         send_patch_email() for the Discord bot
> 
> This one breaks runtime:
> 
>      Traceback (most recent call last):
>        File "/root/lore-daemon2/daemon/tg.py", line 11, in <module>
>          from telegram.packages import DaemonTelegram
>        File "/root/lore-daemon2/daemon/telegram/packages/__init__.py", line 1, in <module>
>          from .client import DaemonTelegram
>        File "/root/lore-daemon2/daemon/telegram/packages/client.py", line 17, in <module>
>          from .decorator import handle_flood
>        File "/root/lore-daemon2/daemon/telegram/packages/decorator.py", line 8, in <module>
>          from typing_extensions import ParamSpec, ParamSpecArgs, ParamSpecKwargs
>      ModuleNotFoundError: No module named 'typing_extensions'
> 
> Apparently, that happens because 'typing_extensions' module is missing.
> But you didn't add that to the requirements.txt file. Apart from that,
> I don't see anything immediately wrong. Overall, it looks good to me.
> 
> Please update the requirements.txt file together with that module
> import addition.
> 

My bad, I thought `typing_extensions` is a built-in library.
I realised that we only have 1 `requirements.txt` and it's in the 
`telegram` directory. Should we move it to the root `daemon` directory 
and update the `requirements.txt`?

Re: [PATCH v1 00/15] Everything about logger changes and some fixes

[Ammar Faizi]

On Wed, Jan 18, 2023 at 06:29:31AM +0700, Muhammad Rizki wrote:
> My bad, I thought `typing_extensions` is a built-in library.
> I realised that we only have 1 `requirements.txt` and it's in the `telegram`
> directory. Should we move it to the root `daemon` directory and update the
> `requirements.txt`?

Yeah, I think it's better to place it outside the telegram directory
and merge Discord dependencies into it if there are any extra
requirements later.

-- 
Ammar Faizi