GNU/Weeb Mailing List <[email protected]>
 help / color / mirror / Atom feed
* [PATCH v2 00/17] Everything about logger changes and some fixes
@ 2023-01-18 17:53 Muhammad Rizki
  2023-01-18 17:53 ` [PATCH v2 01/17] telegram: Simplify code to get DB_PORT from env Muhammad Rizki
                   ` (18 more replies)
  0 siblings, 19 replies; 21+ messages in thread
From: Muhammad Rizki @ 2023-01-18 17:53 UTC (permalink / raw)
  To: Ammar Faizi
  Cc: Muhammad Rizki, Alviro Iskandar Setiawan, GNU/Weeb Mailing List

Hi sir,
This series is about refactor logger method, fix charset issue, and some
improvements. In this series we use a simple logging method that so
easily to manage logger config using .conf file.

- What's new?
  v1:
    1) Add return type annotation for the send_te.xt_mail() and
       send_patch_email() for the Discord bot

- Fixes:
  v1:
    1) Unknown email payload charset `unknown-8bit` that is not in the
       Python's built-in `decode()`. So, we manually using `8bit`
       charset and force it using `surrogateescape`.

    2) Fix type annotations for both Discord and Telegram decorators that
       handle flood request.

    3) Implement DaemonException() and report_err() in the Telegram's
       scrape.py and Discord's get_lore_mail.py

  v2:
    1) Add `typing-extensions` into the `requirements.txt` to fix no
       no module error.

- Improvements:
  v1:
    1) Simplify code to get the DB_PORT.
    2) Remove some unnecessary comments.

  v2:
    1) Remove custom logger since it's not use anymore.

There are 17 patches in this series:
- Patch 1 is to simplify to get the DB_PORT from .env for the TG bot.
- Patch 2 is to simplify to get the DB_PORT from .env for the DC bot.
- Patch 3 is to add a telegram.logger.conf.
- Patch 4 is to add a discord.logger.conf.
- Patch 5 is to initialize the logger config for the Telegram bot.
- Patch 6 is to initialize the logger config for the Discord bot.
- Patch 7 is to fix the type annotations for the Telegram's decorator.
- Patch 8 is to remove some unnecessary comments.
- Patch 9 is to fix the type annotations for the Discord's decorator.
- Patch 10 is to add return type annotations for the Discord bot.
- Patch 11 is to implement DaemonException and report_err for TG scrape.
- Patch 12 is to fix `unknown-8bit` charset when decoding email payload.
- Patch 13 is to refactor logging method in all files for Telegram bot.
- Patch 14 is to implement DaemonException and report_err for DC bot.
- Patch 15 is to refactor logging method in all files for Discord bot.
- Patch 16 is to move requirements.txt to the daemon and update deps.
- Patch 17 is to remove custom logger since it's not used anymore.

Please give it a test, thanks!

Signed-off-by: Muhammad Rizki <[email protected]>
---

Muhammad Rizki (17):
  telegram: Simplify code to get DB_PORT from env
  discord: Simplify code to get DB_PORT from env
  telegram: logger: Add a telegram.logger.conf
  discord: logger: Add a discord.logger.conf
  telegram: logger: Initialize the configuration for the Telegram
    logger.
  discord: logger: Initialize the configuration for the Discord logger.
  telegram: fix: Fix the type annoations for the decorator
  discord: cleanup: Remove some unnecessary comments
  discord: fix: Fix the type annotations for the decorator
  discord: typing: Add return type annotations
  telegram: Implement DaemonException() and report_err() in scrape.py
  utils: fix: Fix charset issue for get_decoded_payload()
  telegram: logger: Refactor all logging method
  discord: Implement DaemonException and report_err in get_lore_mail.py
  discord: logger: Refactor all logging method
  refactor: Move requirements.txt
  cleanup: Remove the custom logger module

 daemon/atom/utils.py                          |  3 ++
 daemon/dc.py                                  | 20 +++----
 daemon/dscord/discord.logger.conf             | 36 +++++++++++++
 daemon/dscord/gnuweeb/client.py               | 25 +++++----
 daemon/dscord/gnuweeb/filters.py              | 32 ++++++------
 .../dscord/gnuweeb/plugins/events/on_ready.py |  5 +-
 .../plugins/slash_commands/get_lore_mail.py   |  9 ++--
 daemon/dscord/mailer/listener.py              | 17 +++---
 daemon/logger/__init__.py                     |  1 -
 daemon/logger/log.py                          | 52 -------------------
 daemon/{telegram => }/requirements.txt        |  2 +
 daemon/telegram/mailer/listener.py            | 15 +++---
 daemon/telegram/packages/client.py            | 21 +++++---
 daemon/telegram/packages/decorator.py         | 31 ++++++-----
 .../packages/plugins/commands/scrape.py       |  9 ++--
 daemon/telegram/telegram.logger.conf          | 36 +++++++++++++
 daemon/tg.py                                  | 18 +++----
 17 files changed, 184 insertions(+), 148 deletions(-)
 create mode 100644 daemon/dscord/discord.logger.conf
 delete mode 100644 daemon/logger/__init__.py
 delete mode 100644 daemon/logger/log.py
 rename daemon/{telegram => }/requirements.txt (81%)
 create mode 100644 daemon/telegram/telegram.logger.conf


base-commit: 1ad0c452d2f87c086a76dd9ceee7a109c8874f57
--
Muhammad Rizki

^ permalink raw reply	[flat|nested] 21+ messages in thread

* [PATCH v2 01/17] telegram: Simplify code to get DB_PORT from env
  2023-01-18 17:53 [PATCH v2 00/17] Everything about logger changes and some fixes Muhammad Rizki
@ 2023-01-18 17:53 ` Muhammad Rizki
  2023-01-18 17:53 ` [PATCH v2 02/17] discord: " Muhammad Rizki
                   ` (17 subsequent siblings)
  18 siblings, 0 replies; 21+ messages in thread
From: Muhammad Rizki @ 2023-01-18 17:53 UTC (permalink / raw)
  To: Ammar Faizi
  Cc: Muhammad Rizki, Alviro Iskandar Setiawan, GNU/Weeb Mailing List

Using `int(os.environ.get("DB_PORT", 3306))` is the best practice to get
the DB_PORT rather than using if statement. This change simplifies the
code.

Signed-off-by: Muhammad Rizki <[email protected]>
---
 daemon/tg.py | 8 +-------
 1 file changed, 1 insertion(+), 7 deletions(-)

diff --git a/daemon/tg.py b/daemon/tg.py
index 9f36e54..6b2ec20 100644
--- a/daemon/tg.py
+++ b/daemon/tg.py
@@ -22,12 +22,6 @@ def main():
 	logger = BotLogger()
 	logger.init()
 
-	port = os.getenv("DB_PORT")
-	if not port:
-		port = 3306
-	else:
-		port = int(port)
-
 	client = DaemonTelegram(
 		"telegram/storage/EmailScraper",
 		api_id=int(os.getenv("API_ID")),
@@ -37,7 +31,7 @@ def main():
 		conn=connector.connect(
 			host=os.getenv("DB_HOST"),
 			user=os.getenv("DB_USER"),
-			port=port,
+			port=int(os.environ.get("DB_PORT", 3306)),
 			password=os.getenv("DB_PASS"),
 			database=os.getenv("DB_NAME")
 		),
-- 
Muhammad Rizki


^ permalink raw reply related	[flat|nested] 21+ messages in thread

* [PATCH v2 02/17] discord: Simplify code to get DB_PORT from env
  2023-01-18 17:53 [PATCH v2 00/17] Everything about logger changes and some fixes Muhammad Rizki
  2023-01-18 17:53 ` [PATCH v2 01/17] telegram: Simplify code to get DB_PORT from env Muhammad Rizki
@ 2023-01-18 17:53 ` Muhammad Rizki
  2023-01-18 17:53 ` [PATCH v2 03/17] telegram: logger: Add a telegram.logger.conf Muhammad Rizki
                   ` (16 subsequent siblings)
  18 siblings, 0 replies; 21+ messages in thread
From: Muhammad Rizki @ 2023-01-18 17:53 UTC (permalink / raw)
  To: Ammar Faizi
  Cc: Muhammad Rizki, Alviro Iskandar Setiawan, GNU/Weeb Mailing List

Using `int(os.environ.get("DB_PORT", 3306))` is the best practice to get
the DB_PORT rather than using if statement. This change simplifies the
code.

Signed-off-by: Muhammad Rizki <[email protected]>
---
 daemon/dc.py | 8 +-------
 1 file changed, 1 insertion(+), 7 deletions(-)

diff --git a/daemon/dc.py b/daemon/dc.py
index a9d5ebd..7d34837 100644
--- a/daemon/dc.py
+++ b/daemon/dc.py
@@ -29,17 +29,11 @@ def main():
 	logger = BotLogger(Platform.DISCORD)
 	logger.init()
 
-	port = os.getenv("DB_PORT")
-	if not port:
-		port = 3306
-	else:
-		port = int(port)
-
 	client = GWClient(
 		db_conn=connector.connect(
 			host=os.getenv("DB_HOST"),
 			user=os.getenv("DB_USER"),
-			port=port,
+			port=int(os.environ.get("DB_PORT", 3306)),
 			password=os.getenv("DB_PASS"),
 			database=os.getenv("DB_NAME")
 		),
-- 
Muhammad Rizki


^ permalink raw reply related	[flat|nested] 21+ messages in thread

* [PATCH v2 03/17] telegram: logger: Add a telegram.logger.conf
  2023-01-18 17:53 [PATCH v2 00/17] Everything about logger changes and some fixes Muhammad Rizki
  2023-01-18 17:53 ` [PATCH v2 01/17] telegram: Simplify code to get DB_PORT from env Muhammad Rizki
  2023-01-18 17:53 ` [PATCH v2 02/17] discord: " Muhammad Rizki
@ 2023-01-18 17:53 ` Muhammad Rizki
  2023-01-18 17:53 ` [PATCH v2 04/17] discord: logger: Add a discord.logger.conf Muhammad Rizki
                   ` (15 subsequent siblings)
  18 siblings, 0 replies; 21+ messages in thread
From: Muhammad Rizki @ 2023-01-18 17:53 UTC (permalink / raw)
  To: Ammar Faizi
  Cc: Muhammad Rizki, Alviro Iskandar Setiawan, GNU/Weeb Mailing List

Add a logger configuration file for improved configuration, debugging,
and management. This method is cleaner than before and the BotLogger()
will be removed in the future.

Signed-off-by: Muhammad Rizki <[email protected]>
---
 daemon/telegram/telegram.logger.conf | 36 ++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)
 create mode 100644 daemon/telegram/telegram.logger.conf

diff --git a/daemon/telegram/telegram.logger.conf b/daemon/telegram/telegram.logger.conf
new file mode 100644
index 0000000..d0c0de3
--- /dev/null
+++ b/daemon/telegram/telegram.logger.conf
@@ -0,0 +1,36 @@
+[loggers]
+keys=root,telegram
+
+[handlers]
+keys=consoleHandler,fileHandler
+
+[formatters]
+keys=sampleFormatter,dictFormatter
+
+[logger_root]
+level=INFO
+handlers=consoleHandler,fileHandler
+
+[logger_telegram]
+level=INFO
+handlers=consoleHandler
+qualname=telegram
+propagate=0
+
+[handler_consoleHandler]
+class=StreamHandler
+level=INFO
+formatter=sampleFormatter
+args=(sys.stdout,)
+
+[handler_fileHandler]
+class=FileHandler
+level=WARNING
+formatter=dictFormatter
+args=("telegram/storage/telegram.log",)
+
+[formatter_sampleFormatter]
+format=%(asctime)s - %(name)s: %(funcName)s - %(levelname)s - %(message)s
+
+[formatter_dictFormatter]
+format={"time": "%(asctime)s", "func": "%(name)s: %(funcName)s", "path": "%(pathname)s", "level": "%(levelname)s", "msg": "%(message)s"}
-- 
Muhammad Rizki


^ permalink raw reply related	[flat|nested] 21+ messages in thread

* [PATCH v2 04/17] discord: logger: Add a discord.logger.conf
  2023-01-18 17:53 [PATCH v2 00/17] Everything about logger changes and some fixes Muhammad Rizki
                   ` (2 preceding siblings ...)
  2023-01-18 17:53 ` [PATCH v2 03/17] telegram: logger: Add a telegram.logger.conf Muhammad Rizki
@ 2023-01-18 17:53 ` Muhammad Rizki
  2023-01-18 17:53 ` [PATCH v2 05/17] telegram: logger: Initialize the configuration for the Telegram logger Muhammad Rizki
                   ` (14 subsequent siblings)
  18 siblings, 0 replies; 21+ messages in thread
From: Muhammad Rizki @ 2023-01-18 17:53 UTC (permalink / raw)
  To: Ammar Faizi
  Cc: Muhammad Rizki, Alviro Iskandar Setiawan, GNU/Weeb Mailing List

Add a logger configuration file for improved configuration, debugging,
and management. This method is cleaner than before and the BotLogger()
will be removed in the future.

Signed-off-by: Muhammad Rizki <[email protected]>
---
 daemon/dscord/discord.logger.conf | 36 +++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)
 create mode 100644 daemon/dscord/discord.logger.conf

diff --git a/daemon/dscord/discord.logger.conf b/daemon/dscord/discord.logger.conf
new file mode 100644
index 0000000..97b5e11
--- /dev/null
+++ b/daemon/dscord/discord.logger.conf
@@ -0,0 +1,36 @@
+[loggers]
+keys=root,dscord
+
+[handlers]
+keys=consoleHandler,fileHandler
+
+[formatters]
+keys=sampleFormatter,dictFormatter
+
+[logger_root]
+level=INFO
+handlers=consoleHandler,fileHandler
+
+[logger_dscord]
+level=INFO
+handlers=consoleHandler
+qualname=dscord
+propagate=0
+
+[handler_consoleHandler]
+class=StreamHandler
+level=INFO
+formatter=sampleFormatter
+args=(sys.stdout,)
+
+[handler_fileHandler]
+class=FileHandler
+level=WARNING
+formatter=dictFormatter
+args=("dscord/storage/discord.log",)
+
+[formatter_sampleFormatter]
+format=%(asctime)s - %(name)s: %(funcName)s - %(levelname)s - %(message)s
+
+[formatter_dictFormatter]
+format={"time": "%(asctime)s", "func": "%(name)s: %(funcName)s", "path": "%(pathname)s", "level": "%(levelname)s", "msg": "%(message)s"}
-- 
Muhammad Rizki


^ permalink raw reply related	[flat|nested] 21+ messages in thread

* [PATCH v2 05/17] telegram: logger: Initialize the configuration for the Telegram logger.
  2023-01-18 17:53 [PATCH v2 00/17] Everything about logger changes and some fixes Muhammad Rizki
                   ` (3 preceding siblings ...)
  2023-01-18 17:53 ` [PATCH v2 04/17] discord: logger: Add a discord.logger.conf Muhammad Rizki
@ 2023-01-18 17:53 ` Muhammad Rizki
  2023-01-18 17:53 ` [PATCH v2 06/17] discord: logger: Initialize the configuration for the Discord logger Muhammad Rizki
                   ` (13 subsequent siblings)
  18 siblings, 0 replies; 21+ messages in thread
From: Muhammad Rizki @ 2023-01-18 17:53 UTC (permalink / raw)
  To: Ammar Faizi
  Cc: Muhammad Rizki, Alviro Iskandar Setiawan, GNU/Weeb Mailing List

This commit adds the configuration for the logger and sets the logger
level of other loggers, such as pyrogram, to the WARNING level. Changing
the level is to disable some of their log messages, so that only our bot
logs to the console.

Signed-off-by: Muhammad Rizki <[email protected]>
---
 daemon/tg.py | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/daemon/tg.py b/daemon/tg.py
index 6b2ec20..26f8374 100644
--- a/daemon/tg.py
+++ b/daemon/tg.py
@@ -14,6 +14,8 @@ from telegram.mailer import BotMutexes
 from telegram.mailer import Bot
 from logger import BotLogger
 import os
+import logging
+import logging.config
 
 
 def main():
@@ -22,6 +24,10 @@ def main():
 	logger = BotLogger()
 	logger.init()
 
+	logging.config.fileConfig("telegram/telegram.logger.conf")
+	logging.getLogger("apscheduler").setLevel(logging.WARNING)
+	logging.getLogger("pyrogram").setLevel(logging.WARNING)
+
 	client = DaemonTelegram(
 		"telegram/storage/EmailScraper",
 		api_id=int(os.getenv("API_ID")),
-- 
Muhammad Rizki


^ permalink raw reply related	[flat|nested] 21+ messages in thread

* [PATCH v2 06/17] discord: logger: Initialize the configuration for the Discord logger.
  2023-01-18 17:53 [PATCH v2 00/17] Everything about logger changes and some fixes Muhammad Rizki
                   ` (4 preceding siblings ...)
  2023-01-18 17:53 ` [PATCH v2 05/17] telegram: logger: Initialize the configuration for the Telegram logger Muhammad Rizki
@ 2023-01-18 17:53 ` Muhammad Rizki
  2023-01-18 17:53 ` [PATCH v2 07/17] telegram: fix: Fix the type annoations for the decorator Muhammad Rizki
                   ` (12 subsequent siblings)
  18 siblings, 0 replies; 21+ messages in thread
From: Muhammad Rizki @ 2023-01-18 17:53 UTC (permalink / raw)
  To: Ammar Faizi
  Cc: Muhammad Rizki, Alviro Iskandar Setiawan, GNU/Weeb Mailing List

This commit adds the configuration for the logger and sets the logger
level of other loggers, such as pyrogram, to the WARNING level. Changing
the level is to disable some of their log messages, so that only our bot
logs to the console.

Signed-off-by: Muhammad Rizki <[email protected]>
---
 daemon/dc.py | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/daemon/dc.py b/daemon/dc.py
index 7d34837..f4eeefb 100644
--- a/daemon/dc.py
+++ b/daemon/dc.py
@@ -4,6 +4,8 @@
 #
 
 import os
+import logging
+import logging.config
 from dotenv import load_dotenv
 from mysql import connector
 from apscheduler.schedulers.asyncio import AsyncIOScheduler
@@ -29,6 +31,10 @@ def main():
 	logger = BotLogger(Platform.DISCORD)
 	logger.init()
 
+	logging.config.fileConfig("dscord/discord.logger.conf")
+	logging.getLogger("apscheduler").setLevel(logging.WARNING)
+	logging.getLogger("discord").setLevel(logging.WARNING)
+
 	client = GWClient(
 		db_conn=connector.connect(
 			host=os.getenv("DB_HOST"),
-- 
Muhammad Rizki


^ permalink raw reply related	[flat|nested] 21+ messages in thread

* [PATCH v2 07/17] telegram: fix: Fix the type annoations for the decorator
  2023-01-18 17:53 [PATCH v2 00/17] Everything about logger changes and some fixes Muhammad Rizki
                   ` (5 preceding siblings ...)
  2023-01-18 17:53 ` [PATCH v2 06/17] discord: logger: Initialize the configuration for the Discord logger Muhammad Rizki
@ 2023-01-18 17:53 ` Muhammad Rizki
  2023-01-18 17:53 ` [PATCH v2 08/17] discord: cleanup: Remove some unnecessary comments Muhammad Rizki
                   ` (11 subsequent siblings)
  18 siblings, 0 replies; 21+ messages in thread
From: Muhammad Rizki @ 2023-01-18 17:53 UTC (permalink / raw)
  To: Ammar Faizi
  Cc: Muhammad Rizki, Alviro Iskandar Setiawan, GNU/Weeb Mailing List

In the previous commit, I created a TODO to fix the typing. In this
commit, I have fixed the decorator's typing.

Fixes: ba02aa3376a4 ("daemon: Add typing in decorator")
Signed-off-by: Muhammad Rizki <[email protected]>
---
 daemon/telegram/packages/decorator.py | 17 ++++++++---------
 1 file changed, 8 insertions(+), 9 deletions(-)

diff --git a/daemon/telegram/packages/decorator.py b/daemon/telegram/packages/decorator.py
index efcfc38..6bfd8bc 100644
--- a/daemon/telegram/packages/decorator.py
+++ b/daemon/telegram/packages/decorator.py
@@ -4,25 +4,24 @@
 #
 
 from pyrogram.errors.exceptions.flood_420 import FloodWait
-from pyrogram.types import Message
-from typing import Any, Callable, TypeVar
+from typing import Any, Callable, TypeVar, Coroutine
+from typing_extensions import ParamSpec, ParamSpecArgs, ParamSpecKwargs
 from functools import wraps
 import re
 import asyncio
 
 __all__ = ["handle_flood"]
 
-T = TypeVar("T", bound=Message)
+T = TypeVar("T")
+P = ParamSpec("P")
 
-#
-# TODO(Muhammad Rizki): Add more typing for @handle_flood
-#
-def handle_flood(func: Callable[[T], T]) -> Callable[[T], T]:
+
+def handle_flood(func: Callable[P, Coroutine[Any,Any,T]]) -> Callable[P, Coroutine[Any,Any,T]]:
 	@wraps(func)
-	async def callback(*args: Any) -> Any:
+	async def callback(*args: ParamSpecArgs, **kwargs: ParamSpecKwargs) -> T:
 		while True:
 			try:
-				return await func(*args)
+				return await func(*args, **kwargs)
 			except FloodWait as e:
 				# Calling logger attr from the DaemonTelegram() class
 				logger = args[0].logger
-- 
Muhammad Rizki


^ permalink raw reply related	[flat|nested] 21+ messages in thread

* [PATCH v2 08/17] discord: cleanup: Remove some unnecessary comments
  2023-01-18 17:53 [PATCH v2 00/17] Everything about logger changes and some fixes Muhammad Rizki
                   ` (6 preceding siblings ...)
  2023-01-18 17:53 ` [PATCH v2 07/17] telegram: fix: Fix the type annoations for the decorator Muhammad Rizki
@ 2023-01-18 17:53 ` Muhammad Rizki
  2023-01-18 17:53 ` [PATCH v2 09/17] discord: fix: Fix the type annotations for the decorator Muhammad Rizki
                   ` (10 subsequent siblings)
  18 siblings, 0 replies; 21+ messages in thread
From: Muhammad Rizki @ 2023-01-18 17:53 UTC (permalink / raw)
  To: Ammar Faizi
  Cc: Muhammad Rizki, Alviro Iskandar Setiawan, GNU/Weeb Mailing List

Signed-off-by: Muhammad Rizki <[email protected]>
---
 daemon/dscord/gnuweeb/filters.py | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/daemon/dscord/gnuweeb/filters.py b/daemon/dscord/gnuweeb/filters.py
index 1fdb70d..b342c7a 100644
--- a/daemon/dscord/gnuweeb/filters.py
+++ b/daemon/dscord/gnuweeb/filters.py
@@ -4,16 +4,13 @@
 #
 
 
-# built-in/dev package imports
 import asyncio
 from typing import Any
 from functools import wraps
 
-# Discord imports
 import discord
 from discord import Interaction
 
-# gnuweeb package import
 from dscord import config
 from logger import BotLogger
 
-- 
Muhammad Rizki


^ permalink raw reply related	[flat|nested] 21+ messages in thread

* [PATCH v2 09/17] discord: fix: Fix the type annotations for the decorator
  2023-01-18 17:53 [PATCH v2 00/17] Everything about logger changes and some fixes Muhammad Rizki
                   ` (7 preceding siblings ...)
  2023-01-18 17:53 ` [PATCH v2 08/17] discord: cleanup: Remove some unnecessary comments Muhammad Rizki
@ 2023-01-18 17:53 ` Muhammad Rizki
  2023-01-18 17:53 ` [PATCH v2 10/17] discord: typing: Add return type annotations Muhammad Rizki
                   ` (9 subsequent siblings)
  18 siblings, 0 replies; 21+ messages in thread
From: Muhammad Rizki @ 2023-01-18 17:53 UTC (permalink / raw)
  To: Ammar Faizi
  Cc: Muhammad Rizki, Alviro Iskandar Setiawan, GNU/Weeb Mailing List

Previously, these decorators did not use type annotations. In this
commit, I have implemented them.

Signed-off-by: Muhammad Rizki <[email protected]>
---
 daemon/dscord/gnuweeb/filters.py | 17 +++++++++++------
 1 file changed, 11 insertions(+), 6 deletions(-)

diff --git a/daemon/dscord/gnuweeb/filters.py b/daemon/dscord/gnuweeb/filters.py
index b342c7a..28b492c 100644
--- a/daemon/dscord/gnuweeb/filters.py
+++ b/daemon/dscord/gnuweeb/filters.py
@@ -5,7 +5,8 @@
 
 
 import asyncio
-from typing import Any
+from typing import Any, Callable, TypeVar, Coroutine
+from typing_extensions import ParamSpec, ParamSpecArgs, ParamSpecKwargs
 from functools import wraps
 
 import discord
@@ -15,9 +16,13 @@ from dscord import config
 from logger import BotLogger
 
 
-def lore_admin(func):
+T = TypeVar("T")
+P = ParamSpec("P")
+
+
+def lore_admin(func: Callable[P, Coroutine[Any,Any,T]]) -> Callable[P, Coroutine[Any,Any,T]]:
 	@wraps(func)
-	async def callback(*args: Any, **kwargs: Any) -> Any:
+	async def callback(*args: ParamSpecArgs, **kwargs: ParamSpecKwargs) -> T:
 		i: "Interaction" = args[1]
 		user_roles = [role.id for role in i.user.roles]
 
@@ -33,12 +38,12 @@ def lore_admin(func):
 	return callback
 
 
-def wait_on_limit(func):
+def wait_on_limit(func: Callable[P, Coroutine[Any,Any,T]]) -> Callable[P, Coroutine[Any,Any,T]]:
 	@wraps(func)
-	async def callback(*args: Any) -> Any:
+	async def callback(*args: ParamSpecArgs, **kwargs: ParamSpecKwargs) -> T:
 		while True:
 			try:
-				return await func(*args)
+				return await func(*args, **kwargs)
 			except discord.errors.RateLimited as e:
 				# Calling logger attr from the GWClient() class
 				logger = args[0].logger
-- 
Muhammad Rizki


^ permalink raw reply related	[flat|nested] 21+ messages in thread

* [PATCH v2 10/17] discord: typing: Add return type annotations
  2023-01-18 17:53 [PATCH v2 00/17] Everything about logger changes and some fixes Muhammad Rizki
                   ` (8 preceding siblings ...)
  2023-01-18 17:53 ` [PATCH v2 09/17] discord: fix: Fix the type annotations for the decorator Muhammad Rizki
@ 2023-01-18 17:53 ` Muhammad Rizki
  2023-01-18 17:53 ` [PATCH v2 11/17] telegram: Implement DaemonException() and report_err() in scrape.py Muhammad Rizki
                   ` (8 subsequent siblings)
  18 siblings, 0 replies; 21+ messages in thread
From: Muhammad Rizki @ 2023-01-18 17:53 UTC (permalink / raw)
  To: Ammar Faizi
  Cc: Muhammad Rizki, Alviro Iskandar Setiawan, GNU/Weeb Mailing List

Add return type annotations for send_text_email() & send_patch_email().
This improves code readability and helps with understanding the type of
object returned by these functions.

Signed-off-by: Muhammad Rizki <[email protected]>
---
 daemon/dscord/gnuweeb/client.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/daemon/dscord/gnuweeb/client.py b/daemon/dscord/gnuweeb/client.py
index 78ec3f9..7d6433b 100644
--- a/daemon/dscord/gnuweeb/client.py
+++ b/daemon/dscord/gnuweeb/client.py
@@ -4,7 +4,7 @@
 #
 
 import discord
-from discord import Interaction
+from discord import Interaction, Message
 from discord.ext import commands
 from discord import Intents
 from typing import Union
@@ -60,7 +60,7 @@ class GWClient(commands.Bot):
 
 	@filters.wait_on_limit
 	async def send_text_email(self, guild_id: int, chat_id: int, text: str,
-				reply_to: Union[int, None] = None, url: str = None):
+				reply_to: Union[int, None] = None, url: str = None) -> Message:
 		self.logger.debug("[send_text_email]")
 		channel = self.get_channel(chat_id)
 
@@ -77,7 +77,7 @@ class GWClient(commands.Bot):
 
 	@filters.wait_on_limit
 	async def send_patch_email(self, mail, guild_id: int, chat_id: int, text: str,
-				reply_to: Union[int, None] = None, url: str = None):
+				reply_to: Union[int, None] = None, url: str = None) -> Message:
 		self.logger.debug("[send_patch_email]")
 		tmp, doc, caption, url = utils.prepare_patch(
 			mail, text, url, Platform.DISCORD
-- 
Muhammad Rizki


^ permalink raw reply related	[flat|nested] 21+ messages in thread

* [PATCH v2 11/17] telegram: Implement DaemonException() and report_err() in scrape.py
  2023-01-18 17:53 [PATCH v2 00/17] Everything about logger changes and some fixes Muhammad Rizki
                   ` (9 preceding siblings ...)
  2023-01-18 17:53 ` [PATCH v2 10/17] discord: typing: Add return type annotations Muhammad Rizki
@ 2023-01-18 17:53 ` Muhammad Rizki
  2023-01-18 17:53 ` [PATCH v2 12/17] utils: fix: Fix charset issue for get_decoded_payload() Muhammad Rizki
                   ` (7 subsequent siblings)
  18 siblings, 0 replies; 21+ messages in thread
From: Muhammad Rizki @ 2023-01-18 17:53 UTC (permalink / raw)
  To: Ammar Faizi
  Cc: Muhammad Rizki, Alviro Iskandar Setiawan, GNU/Weeb Mailing List

In the last series, I didn't care so much to look into the Telegram's
scrape.py command. I've implemented them same as in the listener.py.

Signed-off-by: Muhammad Rizki <[email protected]>
---
 daemon/telegram/packages/plugins/commands/scrape.py | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/daemon/telegram/packages/plugins/commands/scrape.py b/daemon/telegram/packages/plugins/commands/scrape.py
index 89581b6..ddcf7f1 100644
--- a/daemon/telegram/packages/plugins/commands/scrape.py
+++ b/daemon/telegram/packages/plugins/commands/scrape.py
@@ -11,6 +11,7 @@ from atom import Scraper
 from atom import utils
 from enums import Platform
 from telegram import config
+from exceptions import DaemonException
 import re
 import asyncio
 
@@ -41,9 +42,11 @@ async def scrap_email(c: DaemonTelegram, m: Message):
 		mail = await s.get_email_from_url(url)
 		text, files, is_patch = utils.create_template(mail, Platform.TELEGRAM)
 	except:
-		exc_str = utils.catch_err()
-		c.logger.warning(exc_str)
-		await c.send_log_file(url)
+		e = DaemonException()
+		e.set_thread_url(url)
+		e.set_message(utils.catch_err())
+		await c.report_err(e)
+		return
 
 	if is_patch:
 		m = await c.send_patch_email(
-- 
Muhammad Rizki


^ permalink raw reply related	[flat|nested] 21+ messages in thread

* [PATCH v2 12/17] utils: fix: Fix charset issue for get_decoded_payload()
  2023-01-18 17:53 [PATCH v2 00/17] Everything about logger changes and some fixes Muhammad Rizki
                   ` (10 preceding siblings ...)
  2023-01-18 17:53 ` [PATCH v2 11/17] telegram: Implement DaemonException() and report_err() in scrape.py Muhammad Rizki
@ 2023-01-18 17:53 ` Muhammad Rizki
  2023-01-18 17:53 ` [PATCH v2 13/17] telegram: logger: Refactor all logging method Muhammad Rizki
                   ` (6 subsequent siblings)
  18 siblings, 0 replies; 21+ messages in thread
From: Muhammad Rizki @ 2023-01-18 17:53 UTC (permalink / raw)
  To: Ammar Faizi
  Cc: Muhammad Rizki, Alviro Iskandar Setiawan, GNU/Weeb Mailing List

Fix the `LookupError` that is thrown by the `decode()` function. The
`unknown-8bit` charset is not found in the built-in `decode()` function
in Python, so we must manually fix it by using an if statement and
decoding using the `ascii` charset while forcing errors using
`surrogateescape`.

Sample: https://lore.kernel.org/all/[email protected]/raw

Signed-off-by: Muhammad Rizki <[email protected]>
---
 daemon/atom/utils.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py
index 96a10a0..73c8978 100644
--- a/daemon/atom/utils.py
+++ b/daemon/atom/utils.py
@@ -323,6 +323,9 @@ def get_decoded_payload(payload: Message):
 		return payload.get_payload(decode=True) \
 			.decode(errors="replace")
 
+	if charset == "unknown-8bit":
+		return p.encode().decode("utf-8", "surrogateescape")
+
 	return p.encode().decode(charset, errors="replace")
 
 
-- 
Muhammad Rizki


^ permalink raw reply related	[flat|nested] 21+ messages in thread

* [PATCH v2 13/17] telegram: logger: Refactor all logging method
  2023-01-18 17:53 [PATCH v2 00/17] Everything about logger changes and some fixes Muhammad Rizki
                   ` (11 preceding siblings ...)
  2023-01-18 17:53 ` [PATCH v2 12/17] utils: fix: Fix charset issue for get_decoded_payload() Muhammad Rizki
@ 2023-01-18 17:53 ` Muhammad Rizki
  2023-01-18 17:53 ` [PATCH v2 14/17] discord: Implement DaemonException and report_err in get_lore_mail.py Muhammad Rizki
                   ` (5 subsequent siblings)
  18 siblings, 0 replies; 21+ messages in thread
From: Muhammad Rizki @ 2023-01-18 17:53 UTC (permalink / raw)
  To: Ammar Faizi
  Cc: Muhammad Rizki, Alviro Iskandar Setiawan, GNU/Weeb Mailing List

We use a simple logging method by using file config and replace all
custom logger method to built-in logging method. This method is more
cleaner and less code wasteful.

The reason why we should name the logger like:

    `logging.getLogger("telegram")`

Because, we need to identify which bot platform that currently running.

Signed-off-by: Muhammad Rizki <[email protected]>
---
 daemon/telegram/mailer/listener.py    | 15 +++++++++------
 daemon/telegram/packages/client.py    | 21 +++++++++++++--------
 daemon/telegram/packages/decorator.py | 14 +++++++-------
 daemon/tg.py                          |  6 ------
 4 files changed, 29 insertions(+), 27 deletions(-)

diff --git a/daemon/telegram/mailer/listener.py b/daemon/telegram/mailer/listener.py
index 2e8eeda..b34c74d 100644
--- a/daemon/telegram/mailer/listener.py
+++ b/daemon/telegram/mailer/listener.py
@@ -15,6 +15,10 @@ from atom import utils
 from enums import Platform
 import asyncio
 import re
+import logging
+
+
+log = logging.getLogger("telegram")
 
 
 class BotMutexes():
@@ -30,7 +34,6 @@ class Bot():
 		self.scraper = scraper
 		self.mutexes = mutexes
 		self.db = client.db
-		self.logger = client.logger
 		self.isRunnerFixed = False
 
 
@@ -57,8 +60,8 @@ class Bot():
 		# TODO(ammarfaizi2):
 		# Ideally, we also want to log and report this situation.
 		#
-		self.logger.error(f"Database error: {str(e)}")
-		self.logger.info("Reconnecting to the database...")
+		log.error(f"Database error: {str(e)}")
+		log.info("Reconnecting to the database...")
 
 		#
 		# Don't do this too often to avoid reconnect burst.
@@ -71,7 +74,7 @@ class Bot():
 
 
 	async def __run(self):
-		self.logger.info("Running...")
+		log.info("Running...")
 		url = None
 		try:
 			for url in self.db.get_atom_urls():
@@ -128,14 +131,14 @@ class Bot():
 
 		if not email_msg_id:
 			md = "email_msg_id not detected, skipping malformed email"
-			self.logger.debug(md)
+			log.debug(md)
 			return False
 
 		email_id = self.__mail_id_from_db(email_msg_id,
 							tg_chat_id)
 		if not email_id:
 			md = f"Skipping {email_id} because has already been sent to Telegram"
-			self.logger.debug(md)
+			log.debug(md)
 			return False
 
 		text, files, is_patch = utils.create_template(mail, Platform.TELEGRAM)
diff --git a/daemon/telegram/packages/client.py b/daemon/telegram/packages/client.py
index a58aaf8..7e898c5 100644
--- a/daemon/telegram/packages/client.py
+++ b/daemon/telegram/packages/client.py
@@ -3,41 +3,46 @@
 # Copyright (C) 2022  Muhammad Rizki <[email protected]>
 #
 
+import logging
 from pyrogram import Client
 from pyrogram.enums import ParseMode
 from pyrogram.types import Message, InlineKeyboardMarkup, InlineKeyboardButton
 from typing import Union
 from email.message import Message
+
 from atom import utils
 from enums import Platform
-from logger import BotLogger
 from telegram import config
 from telegram.database import DB
 from .decorator import handle_flood
 from exceptions import DaemonException
 
 
+log = logging.getLogger("telegram")
+
+
 class DaemonTelegram(Client):
 	def __init__(self, name: str, api_id: int,
-		api_hash: str, conn, logger: BotLogger,
-		**kwargs
+		api_hash: str, conn, **kwargs
 	):
 		super().__init__(name, api_id,
 				api_hash, **kwargs)
 		self.db = DB(conn)
-		self.logger = logger
 
 
 	async def report_err(self, e: DaemonException):
 		capt = f"Atom URL: {e.atom_url}\n"
 		capt += f"Thread URL: {e.thread_url}"
-		self.logger.warning(e.original_exception)
+		log.warning(e.original_exception)
 		await self.send_log_file(capt)
 
 
 	@handle_flood
 	async def send_log_file(self, caption: str):
-		filename = self.logger.handlers[0].baseFilename
+		for handler in log.root.handlers:
+			if isinstance(handler, logging.FileHandler):
+				filename = handler.baseFilename
+
 		await self.send_document(
 			config.LOG_CHANNEL_ID,
 			filename,
@@ -54,7 +59,7 @@ class DaemonTelegram(Client):
 		url: str = None,
 		parse_mode: ParseMode = ParseMode.HTML
 	) -> Message:
-		self.logger.debug("[send_text_email]")
+		log.debug("[send_text_email]")
 		return await self.send_message(
 			chat_id=chat_id,
 			text=text,
@@ -79,7 +84,7 @@ class DaemonTelegram(Client):
 		url: str = None,
 		parse_mode: ParseMode = ParseMode.HTML
 	) -> Message:
-		self.logger.debug("[send_patch_email]")
+		log.debug("[send_patch_email]")
 		tmp, doc, caption, url = utils.prepare_patch(
 			mail, text, url, Platform.TELEGRAM
 		)
diff --git a/daemon/telegram/packages/decorator.py b/daemon/telegram/packages/decorator.py
index 6bfd8bc..64162f6 100644
--- a/daemon/telegram/packages/decorator.py
+++ b/daemon/telegram/packages/decorator.py
@@ -9,9 +9,12 @@ from typing_extensions import ParamSpec, ParamSpecArgs, ParamSpecKwargs
 from functools import wraps
 import re
 import asyncio
+import logging
+
 
 __all__ = ["handle_flood"]
 
+log = logging.getLogger("telegram")
 T = TypeVar("T")
 P = ParamSpec("P")
 
@@ -23,19 +26,16 @@ def handle_flood(func: Callable[P, Coroutine[Any,Any,T]]) -> Callable[P, Corouti
 			try:
 				return await func(*args, **kwargs)
 			except FloodWait as e:
-				# Calling logger attr from the DaemonTelegram() class
-				logger = args[0].logger
-
-				_flood_exceptions(e, logger)
-				logger.info("Woken up from flood wait...")
+				_flood_exceptions(e)
+				log.info("Woken up from flood wait...")
 	return callback
 
 
-async def _flood_exceptions(e, logger):
+async def _flood_exceptions(e):
 	x = re.search(r"A wait of (\d+) seconds is required", str(e))
 	if not x:
 		raise e
 
 	n = int(x.group(1))
-	logger.info(f"Sleeping for {n} seconds due to Telegram limit")
+	log.info(f"Sleeping for {n} seconds due to Telegram limit")
 	await asyncio.sleep(n)
diff --git a/daemon/tg.py b/daemon/tg.py
index 26f8374..5233faa 100644
--- a/daemon/tg.py
+++ b/daemon/tg.py
@@ -7,12 +7,10 @@
 from apscheduler.schedulers.asyncio import AsyncIOScheduler
 from dotenv import load_dotenv
 from mysql import connector
-from pyrogram import idle
 from atom import Scraper
 from telegram.packages import DaemonTelegram
 from telegram.mailer import BotMutexes
 from telegram.mailer import Bot
-from logger import BotLogger
 import os
 import logging
 import logging.config
@@ -21,9 +19,6 @@ import logging.config
 def main():
 	load_dotenv("telegram.env")
 
-	logger = BotLogger()
-	logger.init()
-
 	logging.config.fileConfig("telegram/telegram.logger.conf")
 	logging.getLogger("apscheduler").setLevel(logging.WARNING)
 	logging.getLogger("pyrogram").setLevel(logging.WARNING)
@@ -33,7 +28,6 @@ def main():
 		api_id=int(os.getenv("API_ID")),
 		api_hash=os.getenv("API_HASH"),
 		bot_token=os.getenv("BOT_TOKEN"),
-		logger=logger,
 		conn=connector.connect(
 			host=os.getenv("DB_HOST"),
 			user=os.getenv("DB_USER"),
-- 
Muhammad Rizki


^ permalink raw reply related	[flat|nested] 21+ messages in thread

* [PATCH v2 14/17] discord: Implement DaemonException and report_err in get_lore_mail.py
  2023-01-18 17:53 [PATCH v2 00/17] Everything about logger changes and some fixes Muhammad Rizki
                   ` (12 preceding siblings ...)
  2023-01-18 17:53 ` [PATCH v2 13/17] telegram: logger: Refactor all logging method Muhammad Rizki
@ 2023-01-18 17:53 ` Muhammad Rizki
  2023-01-18 17:53 ` [PATCH v2 15/17] discord: logger: Refactor all logging method Muhammad Rizki
                   ` (4 subsequent siblings)
  18 siblings, 0 replies; 21+ messages in thread
From: Muhammad Rizki @ 2023-01-18 17:53 UTC (permalink / raw)
  To: Ammar Faizi
  Cc: Muhammad Rizki, Alviro Iskandar Setiawan, GNU/Weeb Mailing List

In the last series, I didn't care so much to look into the Discord's
get_lore_mail.py slash command. I've implemented them same as in the
listener.py.

Signed-off-by: Muhammad Rizki <[email protected]>
---
 .../gnuweeb/plugins/slash_commands/get_lore_mail.py      | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py b/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py
index 56330f3..84dca83 100644
--- a/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py
+++ b/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py
@@ -10,6 +10,7 @@ from discord import app_commands
 from atom import utils
 from atom import Scraper
 from enums import Platform
+from exceptions import DaemonException
 
 
 class GetLoreSC(commands.Cog):
@@ -29,9 +30,11 @@ class GetLoreSC(commands.Cog):
 			mail = await s.get_email_from_url(url)
 			text, _, is_patch = utils.create_template(mail, Platform.DISCORD)
 		except:
-			exc_str = utils.catch_err()
-			self.bot.logger.warning(exc_str)
-			await self.bot.send_log_file(url)
+			e = DaemonException()
+			e.set_thread_url(url)
+			e.set_message(utils.catch_err())
+			await self.bot.report_err(e)
+			return
 
 		if is_patch:
 			m = await self.bot.send_patch_mail_interaction(
-- 
Muhammad Rizki


^ permalink raw reply related	[flat|nested] 21+ messages in thread

* [PATCH v2 15/17] discord: logger: Refactor all logging method
  2023-01-18 17:53 [PATCH v2 00/17] Everything about logger changes and some fixes Muhammad Rizki
                   ` (13 preceding siblings ...)
  2023-01-18 17:53 ` [PATCH v2 14/17] discord: Implement DaemonException and report_err in get_lore_mail.py Muhammad Rizki
@ 2023-01-18 17:53 ` Muhammad Rizki
  2023-01-18 17:53 ` [PATCH v2 16/17] refactor: Move requirements.txt Muhammad Rizki
                   ` (3 subsequent siblings)
  18 siblings, 0 replies; 21+ messages in thread
From: Muhammad Rizki @ 2023-01-18 17:53 UTC (permalink / raw)
  To: Ammar Faizi
  Cc: Muhammad Rizki, Alviro Iskandar Setiawan, GNU/Weeb Mailing List

Same as the telegram refactors. We use a simple logging method by using
file config and replace all custom logger method to built-in logging
method. This method is more cleaner and less code wasteful.

The reason why we should name the logger like:

    `logging.getLogger("dscord")`

Because, if we use "discord" as the logger name, it will conflict the
discord.py's logger, that's why we need to disable the discord.py's
logger not to interrupt our logger.

Signed-off-by: Muhammad Rizki <[email protected]>
---
 daemon/dc.py                                  |  8 +-------
 daemon/dscord/gnuweeb/client.py               | 19 ++++++++++++-------
 daemon/dscord/gnuweeb/filters.py              | 12 +++++-------
 .../dscord/gnuweeb/plugins/events/on_ready.py |  5 ++++-
 daemon/dscord/mailer/listener.py              | 17 ++++++++++-------
 5 files changed, 32 insertions(+), 29 deletions(-)

diff --git a/daemon/dc.py b/daemon/dc.py
index f4eeefb..dade9f1 100644
--- a/daemon/dc.py
+++ b/daemon/dc.py
@@ -13,8 +13,6 @@ from apscheduler.schedulers.asyncio import AsyncIOScheduler
 from dscord.gnuweeb import GWClient
 from dscord.mailer import Listener
 from dscord.mailer import Mutexes
-from enums.platform import Platform
-from logger import BotLogger
 from atom import Scraper
 
 
@@ -28,9 +26,6 @@ def main():
 		}
 	)
 
-	logger = BotLogger(Platform.DISCORD)
-	logger.init()
-
 	logging.config.fileConfig("dscord/discord.logger.conf")
 	logging.getLogger("apscheduler").setLevel(logging.WARNING)
 	logging.getLogger("discord").setLevel(logging.WARNING)
@@ -42,8 +37,7 @@ def main():
 			port=int(os.environ.get("DB_PORT", 3306)),
 			password=os.getenv("DB_PASS"),
 			database=os.getenv("DB_NAME")
-		),
-		logger=logger
+		)
 	)
 
 	mailer = Listener(
diff --git a/daemon/dscord/gnuweeb/client.py b/daemon/dscord/gnuweeb/client.py
index 7d6433b..a04fd26 100644
--- a/daemon/dscord/gnuweeb/client.py
+++ b/daemon/dscord/gnuweeb/client.py
@@ -3,6 +3,7 @@
 # Copyright (C) 2022  Muhammad Rizki <[email protected]>
 #
 
+import logging
 import discord
 from discord import Interaction, Message
 from discord.ext import commands
@@ -14,18 +15,19 @@ from . import models
 from atom import utils
 from enums import Platform
 from exceptions import DaemonException
-from logger.log import BotLogger
 from dscord.config import ACTIVITY_NAME, LOG_CHANNEL_ID
 from dscord.database import DB
 
 
+log = logging.getLogger("dscord")
+
+
 class GWClient(commands.Bot):
-	def __init__(self, db_conn, logger: BotLogger) -> None:
+	def __init__(self, db_conn) -> None:
 		self.db = DB(db_conn)
 		intents = Intents.default()
 		intents.message_content = True
 		self.mailer = None
-		self.logger = logger
 		super().__init__(
 			command_prefix=["$", "."],
 			description="Just a bot for receiving lore emails.",
@@ -42,7 +44,7 @@ class GWClient(commands.Bot):
 
 
 	async def report_err(self, e: DaemonException):
-		self.logger.warning(e.original_exception)
+		log.warning(e.original_exception)
 		capt = f"Atom URL: {e.atom_url}\n"
 		capt += f"Thread URL: {e.thread_url}"
 		await self.send_log_file(capt)
@@ -50,7 +52,10 @@ class GWClient(commands.Bot):
 
 	@filters.wait_on_limit
 	async def send_log_file(self, caption: str):
-		filename = self.logger.handlers[0].baseFilename
+		for handler in log.root.handlers:
+			if isinstance(handler, logging.FileHandler):
+				filename = handler.baseFilename
+
 		channel = self.get_channel(LOG_CHANNEL_ID)
 		await channel.send(
 			content=caption,
@@ -61,7 +66,7 @@ class GWClient(commands.Bot):
 	@filters.wait_on_limit
 	async def send_text_email(self, guild_id: int, chat_id: int, text: str,
 				reply_to: Union[int, None] = None, url: str = None) -> Message:
-		self.logger.debug("[send_text_email]")
+		log.debug("[send_text_email]")
 		channel = self.get_channel(chat_id)
 
 		return await channel.send(
@@ -78,7 +83,7 @@ class GWClient(commands.Bot):
 	@filters.wait_on_limit
 	async def send_patch_email(self, mail, guild_id: int, chat_id: int, text: str,
 				reply_to: Union[int, None] = None, url: str = None) -> Message:
-		self.logger.debug("[send_patch_email]")
+		log.debug("[send_patch_email]")
 		tmp, doc, caption, url = utils.prepare_patch(
 			mail, text, url, Platform.DISCORD
 		)
diff --git a/daemon/dscord/gnuweeb/filters.py b/daemon/dscord/gnuweeb/filters.py
index 28b492c..ccd13e7 100644
--- a/daemon/dscord/gnuweeb/filters.py
+++ b/daemon/dscord/gnuweeb/filters.py
@@ -5,6 +5,7 @@
 
 
 import asyncio
+import logging
 from typing import Any, Callable, TypeVar, Coroutine
 from typing_extensions import ParamSpec, ParamSpecArgs, ParamSpecKwargs
 from functools import wraps
@@ -13,9 +14,9 @@ import discord
 from discord import Interaction
 
 from dscord import config
-from logger import BotLogger
 
 
+log = logging.getLogger("dscord")
 T = TypeVar("T")
 P = ParamSpec("P")
 
@@ -45,15 +46,12 @@ def wait_on_limit(func: Callable[P, Coroutine[Any,Any,T]]) -> Callable[P, Corout
 			try:
 				return await func(*args, **kwargs)
 			except discord.errors.RateLimited as e:
-				# Calling logger attr from the GWClient() class
-				logger = args[0].logger
-
 				_flood_exceptions(e)
-				logger.info("Woken up from flood wait...")
+				log.info("Woken up from flood wait...")
 	return callback
 
 
-async def _flood_exceptions(e: "discord.errors.RateLimited", logger: BotLogger):
+async def _flood_exceptions(e: "discord.errors.RateLimited"):
 	wait = e.retry_after
-	logger.info(f"Sleeping for {wait} seconds due to Discord limit")
+	log.info(f"Sleeping for {wait} seconds due to Discord limit")
 	await asyncio.sleep(wait)
diff --git a/daemon/dscord/gnuweeb/plugins/events/on_ready.py b/daemon/dscord/gnuweeb/plugins/events/on_ready.py
index e7f63cd..4248aad 100644
--- a/daemon/dscord/gnuweeb/plugins/events/on_ready.py
+++ b/daemon/dscord/gnuweeb/plugins/events/on_ready.py
@@ -3,9 +3,12 @@
 # Copyright (C) 2022  Muhammad Rizki <[email protected]>
 #
 
+import logging
 from discord.ext import commands
 
 
+log = logging.getLogger("dscord")
+
 class OnReady(commands.Cog):
 	def __init__(self, bot: "commands.Bot") -> None:
 		self.bot = bot
@@ -23,4 +26,4 @@ class OnReady(commands.Cog):
 		t += f"Send `{prefix}sync` message to the Discord channel "
 		t += "where the bot is running.\n"
 
-		self.bot.logger.info(t)
+		log.info(t)
diff --git a/daemon/dscord/mailer/listener.py b/daemon/dscord/mailer/listener.py
index fc066b7..25e5715 100644
--- a/daemon/dscord/mailer/listener.py
+++ b/daemon/dscord/mailer/listener.py
@@ -6,6 +6,7 @@
 
 import asyncio
 import re
+import logging
 from mysql.connector.errors import OperationalError, DatabaseError
 from apscheduler.schedulers.asyncio import AsyncIOScheduler
 from discord import File
@@ -18,6 +19,9 @@ from enums import Platform
 from exceptions import DaemonException
 
 
+log = logging.getLogger("dscord")
+
+
 class Mutexes:
 	def __init__(self):
 		self.lock = asyncio.Lock()
@@ -36,7 +40,6 @@ class Listener:
 		self.scraper = scraper
 		self.mutexes = mutexes
 		self.db = client.db
-		self.logger = client.logger
 		self.isRunnerFixed = False
 		self.runner = None
 
@@ -46,7 +49,7 @@ class Listener:
 		# Execute __run() once to avoid high latency at
 		# initilization.
 		#
-		self.logger.info("Initialize listener...\n")
+		log.info("Initialize listener...\n")
 		self.sched.start()
 		self.runner = self.sched.add_job(func=self.__run)
 
@@ -56,8 +59,8 @@ class Listener:
 		# TODO(ammarfaizi2):
 		# Ideally, we also want to log and report this situation.
 		#
-		self.logger.error(f"Database error: {str(e)}")
-		self.logger.info("Reconnecting to the database...")
+		log.error(f"Database error: {str(e)}")
+		log.info("Reconnecting to the database...")
 
 		#
 		# Don't do this too often to avoid reconnect burst.
@@ -70,7 +73,7 @@ class Listener:
 
 
 	async def __run(self):
-		self.logger.info("Running...")
+		log.info("Running...")
 		url = None
 
 		try:
@@ -128,7 +131,7 @@ class Listener:
 		email_msg_id = utils.get_email_msg_id(mail)
 		if not email_msg_id:
 			md = "email_msg_id not detected, skipping malformed email"
-			self.logger.debug(md)
+			log.debug(md)
 			return False
 
 		email_id = self.__get_email_id_sent(
@@ -137,7 +140,7 @@ class Listener:
 		)
 		if not email_id:
 			md = f"Skipping {email_id} because has already been sent to Discord"
-			self.logger.debug(md)
+			log.debug(md)
 			return False
 
 		text, files, is_patch = utils.create_template(mail, Platform.DISCORD)
-- 
Muhammad Rizki


^ permalink raw reply related	[flat|nested] 21+ messages in thread

* [PATCH v2 16/17] refactor: Move requirements.txt
  2023-01-18 17:53 [PATCH v2 00/17] Everything about logger changes and some fixes Muhammad Rizki
                   ` (14 preceding siblings ...)
  2023-01-18 17:53 ` [PATCH v2 15/17] discord: logger: Refactor all logging method Muhammad Rizki
@ 2023-01-18 17:53 ` Muhammad Rizki
  2023-01-18 17:53 ` [PATCH v2 17/17] cleanup: Remove the custom logger module Muhammad Rizki
                   ` (2 subsequent siblings)
  18 siblings, 0 replies; 21+ messages in thread
From: Muhammad Rizki @ 2023-01-18 17:53 UTC (permalink / raw)
  To: Ammar Faizi
  Cc: Muhammad Rizki, Alviro Iskandar Setiawan, GNU/Weeb Mailing List

Move `requirements.txt` to the outside of the `telegram` directory and
update the required dependencies in it. This requirements file is for
both Telegram and Discord so we don't need to add another requirements
file in their directory.

Signed-off-by: Muhammad Rizki <[email protected]>
---
 daemon/{telegram => }/requirements.txt | 2 ++
 1 file changed, 2 insertions(+)
 rename daemon/{telegram => }/requirements.txt (81%)

diff --git a/daemon/telegram/requirements.txt b/daemon/requirements.txt
similarity index 81%
rename from daemon/telegram/requirements.txt
rename to daemon/requirements.txt
index 193cec9..3a16421 100644
--- a/daemon/telegram/requirements.txt
+++ b/daemon/requirements.txt
@@ -8,3 +8,5 @@ tgcrypto
 xmltodict
 unicode
 python-slugify
+typing-extensions
+discord.py
-- 
Muhammad Rizki


^ permalink raw reply related	[flat|nested] 21+ messages in thread

* [PATCH v2 17/17] cleanup: Remove the custom logger module
  2023-01-18 17:53 [PATCH v2 00/17] Everything about logger changes and some fixes Muhammad Rizki
                   ` (15 preceding siblings ...)
  2023-01-18 17:53 ` [PATCH v2 16/17] refactor: Move requirements.txt Muhammad Rizki
@ 2023-01-18 17:53 ` Muhammad Rizki
  2023-01-19  6:54 ` [PATCH v2 00/17] Everything about logger changes and some fixes Ammar Faizi
  2023-01-19  6:54 ` Ammar Faizi
  18 siblings, 0 replies; 21+ messages in thread
From: Muhammad Rizki @ 2023-01-18 17:53 UTC (permalink / raw)
  To: Ammar Faizi
  Cc: Muhammad Rizki, Alviro Iskandar Setiawan, GNU/Weeb Mailing List

We don't need this custom logger anymore since we already use the
built-in `logging` and use .conf file.

Signed-off-by: Muhammad Rizki <[email protected]>
---
 daemon/logger/__init__.py |  1 -
 daemon/logger/log.py      | 52 ---------------------------------------
 2 files changed, 53 deletions(-)
 delete mode 100644 daemon/logger/__init__.py
 delete mode 100644 daemon/logger/log.py

diff --git a/daemon/logger/__init__.py b/daemon/logger/__init__.py
deleted file mode 100644
index bb3145a..0000000
--- a/daemon/logger/__init__.py
+++ /dev/null
@@ -1 +0,0 @@
-from .log import BotLogger
diff --git a/daemon/logger/log.py b/daemon/logger/log.py
deleted file mode 100644
index 8382404..0000000
--- a/daemon/logger/log.py
+++ /dev/null
@@ -1,52 +0,0 @@
-import logging
-import os
-from enums import Platform
-
-
-class BotLogger(logging.Logger):
-
-	def __init__(
-		self,
-		platform: Platform = Platform.TELEGRAM,
-		level: int = logging.WARNING
-	) -> None:
-
-		self.platform = str(platform.value)
-		self.level = level
-
-		plat_rep = self.platform.replace("discord", "dscord")
-		self.file_name = f"{plat_rep}/" \
-			f"{os.getenv('STORAGE_DIR', 'storage')}/" \
-			f"{self.platform}.log"
-
-		self.file_fmt = "{" \
-			"'time': '%(asctime)s'," \
-			"'func': '%(name)s: %(funcName)s'," \
-			"'path': '%(pathname)s'," \
-			"'level': '%(levelname)s'," \
-			"'msg': '%(message)s'"\
-		"}"
-
-		self.stream_fmt = "%(asctime)s | " \
-			"%(name)s: %(funcName)s | " \
-			"%(levelname)s | " \
-			"%(message)s"
-
-
-	def init(self):
-		super().__init__(self.platform, self.level)
-
-		file_handler = logging.FileHandler(self.file_name)
-		stream_handler = logging.StreamHandler()
-
-		file_fmtr = logging.Formatter(self.file_fmt)
-		stream_fmtr = logging.Formatter(self.stream_fmt)
-
-		file_handler.setFormatter(file_fmtr)
-		stream_handler.setFormatter(stream_fmtr)
-
-		file_handler.setLevel(logging.WARNING)
-		self.setLevel(logging.INFO)
-
-		self.addHandler(file_handler)
-		self.addHandler(stream_handler)
-- 
Muhammad Rizki


^ permalink raw reply related	[flat|nested] 21+ messages in thread

* Re: [PATCH v2 00/17] Everything about logger changes and some fixes
  2023-01-18 17:53 [PATCH v2 00/17] Everything about logger changes and some fixes Muhammad Rizki
                   ` (16 preceding siblings ...)
  2023-01-18 17:53 ` [PATCH v2 17/17] cleanup: Remove the custom logger module Muhammad Rizki
@ 2023-01-19  6:54 ` Ammar Faizi
  2023-01-19 12:49   ` Muhammad Rizki
  2023-01-19  6:54 ` Ammar Faizi
  18 siblings, 1 reply; 21+ messages in thread
From: Ammar Faizi @ 2023-01-19  6:54 UTC (permalink / raw)
  To: Muhammad Rizki; +Cc: Alviro Iskandar Setiawan, GNU/Weeb Mailing List

On Thu, Jan 19, 2023 at 12:53:39AM +0700, Muhammad Rizki wrote:
> - Patch 7 is to fix the type annotations for the Telegram's decorator.

You use 'typing_extensions' extension in patch #7.

> - Patch 16 is to move requirements.txt to the daemon and update deps.

But you add 'typing_extensions' in patch #16. This honestly, doesn't
make sense from the test standpoint. You should add 'typing_extensions'
to the requirement list first, then use it later. Each patch, when
applied, should not leave a breakage in the middle.

To save time and internet bandwith, I just reordered it myself because
it's simple enough for me to fix.

-- 
Ammar Faizi


^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: [PATCH v2 00/17] Everything about logger changes and some fixes
  2023-01-18 17:53 [PATCH v2 00/17] Everything about logger changes and some fixes Muhammad Rizki
                   ` (17 preceding siblings ...)
  2023-01-19  6:54 ` [PATCH v2 00/17] Everything about logger changes and some fixes Ammar Faizi
@ 2023-01-19  6:54 ` Ammar Faizi
  18 siblings, 0 replies; 21+ messages in thread
From: Ammar Faizi @ 2023-01-19  6:54 UTC (permalink / raw)
  To: Muhammad Rizki
  Cc: Ammar Faizi, Alviro Iskandar Setiawan, GNU/Weeb Mailing List

On Thu, 19 Jan 2023 00:53:39 +0700, Muhammad Rizki wrote:
> This series is about refactor logger method, fix charset issue, and some
> improvements. In this series we use a simple logging method that so
> easily to manage logger config using .conf file.
> 
> - What's new?
>   v1:
>     1) Add return type annotation for the send_te.xt_mail() and
>        send_patch_email() for the Discord bot
> 
> [...]

Applied, thanks!

[01/17] telegram: Simplify code to get DB_PORT from env
        commit: 1701589a63270d2204d4386630e1cc35d910dd90
[02/17] discord: Simplify code to get DB_PORT from env
        commit: d9ab32d7ea83e6d08d705d3d51a5b1d35570a497
[03/17] telegram: logger: Add a telegram.logger.conf
        commit: da2ff98f54979669867bca1a69f2ca5d19d10652
[04/17] discord: logger: Add a discord.logger.conf
        commit: 5ee8bdab7d3eed174bfd06ab30358027d2d4109e
[05/17] telegram: logger: Initialize the configuration for the Telegram logger.
        commit: afc6016ce1ec177b3ae78d65a9fb157292a3d1f6
[06/17] discord: logger: Initialize the configuration for the Discord logger.
        commit: 83fa3b5cafaa7c430042404d39480b4382b6dad3
[07/17] telegram: fix: Fix the type annoations for the decorator
        commit: 3b420794c629b1adea071ecd6b3d4d421620f43e
[08/17] discord: cleanup: Remove some unnecessary comments
        commit: 556c16b605e64746dd174d3a78aefe3ccafd1075
[09/17] discord: fix: Fix the type annotations for the decorator
        commit: 57a705431e73b815f209cf340d4a102c5abf13f1
[10/17] discord: typing: Add return type annotations
        commit: dd2909aa6ae90b848561d02c8cdf1cb35d3a0420
[11/17] telegram: Implement DaemonException() and report_err() in scrape.py
        commit: 16431f083520918a6e29e9fe1e4391a626e7bfe5
[12/17] utils: fix: Fix charset issue for get_decoded_payload()
        commit: 858a25d1516d6bef22389d6bbd3577f4c1bc9fc4
[13/17] telegram: logger: Refactor all logging method
        commit: 332ffd0c5d197077dd77b4d72d323211eed7b485
[14/17] discord: Implement DaemonException and report_err in get_lore_mail.py
        commit: aae8fbb4dcc5a3c1827df70a6ed3763d9dc73619
[15/17] discord: logger: Refactor all logging method
        commit: df3476d67c90418710ceaeaff0413e0418d54041
[16/17] refactor: Move requirements.txt
        (no commit info)
[17/17] cleanup: Remove the custom logger module
        commit: bf2fe963caf19a5002f585cd07541400c5df945b

Best regards,
-- 
Ammar Faizi <[email protected]>

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: [PATCH v2 00/17] Everything about logger changes and some fixes
  2023-01-19  6:54 ` [PATCH v2 00/17] Everything about logger changes and some fixes Ammar Faizi
@ 2023-01-19 12:49   ` Muhammad Rizki
  0 siblings, 0 replies; 21+ messages in thread
From: Muhammad Rizki @ 2023-01-19 12:49 UTC (permalink / raw)
  To: Ammar Faizi; +Cc: Alviro Iskandar Setiawan, GNU/Weeb Mailing List

On 19/01/2023 13.54, Ammar Faizi wrote:
> On Thu, Jan 19, 2023 at 12:53:39AM +0700, Muhammad Rizki wrote:
>> - Patch 7 is to fix the type annotations for the Telegram's decorator.
> 
> You use 'typing_extensions' extension in patch #7.
> 
>> - Patch 16 is to move requirements.txt to the daemon and update deps.
> 
> But you add 'typing_extensions' in patch #16. This honestly, doesn't
> make sense from the test standpoint. You should add 'typing_extensions'
> to the requirement list first, then use it later. Each patch, when
> applied, should not leave a breakage in the middle.
> 
> To save time and internet bandwith, I just reordered it myself because
> it's simple enough for me to fix.
> 

Ah yes my bad, thanks anyway!

^ permalink raw reply	[flat|nested] 21+ messages in thread

end of thread, other threads:[~2023-01-19 12:49 UTC | newest]

Thread overview: 21+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-01-18 17:53 [PATCH v2 00/17] Everything about logger changes and some fixes Muhammad Rizki
2023-01-18 17:53 ` [PATCH v2 01/17] telegram: Simplify code to get DB_PORT from env Muhammad Rizki
2023-01-18 17:53 ` [PATCH v2 02/17] discord: " Muhammad Rizki
2023-01-18 17:53 ` [PATCH v2 03/17] telegram: logger: Add a telegram.logger.conf Muhammad Rizki
2023-01-18 17:53 ` [PATCH v2 04/17] discord: logger: Add a discord.logger.conf Muhammad Rizki
2023-01-18 17:53 ` [PATCH v2 05/17] telegram: logger: Initialize the configuration for the Telegram logger Muhammad Rizki
2023-01-18 17:53 ` [PATCH v2 06/17] discord: logger: Initialize the configuration for the Discord logger Muhammad Rizki
2023-01-18 17:53 ` [PATCH v2 07/17] telegram: fix: Fix the type annoations for the decorator Muhammad Rizki
2023-01-18 17:53 ` [PATCH v2 08/17] discord: cleanup: Remove some unnecessary comments Muhammad Rizki
2023-01-18 17:53 ` [PATCH v2 09/17] discord: fix: Fix the type annotations for the decorator Muhammad Rizki
2023-01-18 17:53 ` [PATCH v2 10/17] discord: typing: Add return type annotations Muhammad Rizki
2023-01-18 17:53 ` [PATCH v2 11/17] telegram: Implement DaemonException() and report_err() in scrape.py Muhammad Rizki
2023-01-18 17:53 ` [PATCH v2 12/17] utils: fix: Fix charset issue for get_decoded_payload() Muhammad Rizki
2023-01-18 17:53 ` [PATCH v2 13/17] telegram: logger: Refactor all logging method Muhammad Rizki
2023-01-18 17:53 ` [PATCH v2 14/17] discord: Implement DaemonException and report_err in get_lore_mail.py Muhammad Rizki
2023-01-18 17:53 ` [PATCH v2 15/17] discord: logger: Refactor all logging method Muhammad Rizki
2023-01-18 17:53 ` [PATCH v2 16/17] refactor: Move requirements.txt Muhammad Rizki
2023-01-18 17:53 ` [PATCH v2 17/17] cleanup: Remove the custom logger module Muhammad Rizki
2023-01-19  6:54 ` [PATCH v2 00/17] Everything about logger changes and some fixes Ammar Faizi
2023-01-19 12:49   ` Muhammad Rizki
2023-01-19  6:54 ` Ammar Faizi

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox