[PATCH v4 00/17] Fix, improvement and implement a bot logger.

[PATCH v4 00/17] Fix, improvement and implement a bot logger.

[Muhammad Rizki]

Hi sir, it's been a long time.

This is a v4 revision of fix, improvement and implement a bot logger.
this series is to improve create_template() caption substr/trim,
prepare_patch() for the .patch file payload, remove unrelevant changelog
commit in the commit description, use html.escape() once instead of
twice, small change in the first commit to make it clearer.

This series contains fixes in the utility file and some improvement for
it. I also implement a bot logger to log an error to the terminal and
to the log file, if error may happen it will send the log file to the
log channel either for Discord or Telegram, the LOG_CHANNEL_ID is
declared in the config.py.example, copy the channel ID from the Discord
or Telegram channel and then paste to the config.py.example.

What's new?
- Implementation of the BotLogger to log error message.
- Add a catch_err() in the atom/utils.py.
- Add a send_log_file() in each bot client.py.
- Add a variable LOG_CHANNEL_ID to configure the log channel.

Fixes:
- Fix media caption too long error for media/attachment message
- Fix .patch file payload
- Fix TO/CC list when an email doesn't contain TO/CC headers
- Fix typo on _flood_exception()

There are 17 patches in this series:
- Patch 1 is to fix a typo on _flood_exception() for the Discord bot.
- Patch 2 is to Fix .patch file payload to have a full email payload.
- Patch 3 is to add a substr/trim caption on the create_template().
- Patch 4 is to fix an extract_list() when there is no receipients.
- Patch 5 is to use the old one method, use decode=True in get_payload()
- Patch 6 is to improve fix_utf8_char(), make sure to unescape the HTML.
- Patch 7 is to add a utility catch_err() function to get error message.
- Patch 8 is to just add an initial BotLogger() to use it in next commit
- Patch 9 is to use the created BotLogger() for the Telegram bot.
- Patch 10 is to add a Telegram LOG_CHANNEL_ID for the send_log_file().
- Patch 11 is to add a send_log_file() for the Telegram bot DaemonClient
- Patch 12 is to implement the log message and send log file for TG bot.
- Patch 13 is to add a Discord LOG_CHANNEL_ID for the send_log_file().
- Patch 14 is to use the created BotLogger() for the Discord bot.
- Patch 15 is to add a send_log_file() for the Discord bot GWClient().
- Patch 16 is to implement the log message and send log file for DC bot.
- Patch 17 is to remove unused imports.

How to set log channel ID for the Telegram bot:
1. Make a channel for log in the Telegram.
2. Copy the channel ID, if you unable to see the ID you can use API or
   bot who has the feature, or use a 64gram desktop.
3. Paste the copied ID into the config.py in the telegram directory.

How to set log channel ID for the Discord bot:
1. Make a channel for log in your Discord server.
2. Copy the channel ID by right click and copy, if you unable to get the
   ID you can go to your account settings, then enable the
   `Developer mode` or use bot who has the feature.
3. Paste the copied ID into the config.py in the dscord directory.

How to use:
1. Execute the db.sql file in the daemon directory.
2. Setup .env file, the example is there with suffix .example, this
   file name must remove the suffix name .example.
3. Set up the config.py in each bot directory, such as dscord and
   telegram. The example is there with suffix .example & the file name
   must remove suffix name .example.
4. Run `pip3 install -r requirements.txt` in each bot directory.
5. STORAGE_DIR env value must `storage` to make it work fine.
6. Run the bot by `python3 dc.py` or `python3 tg.py`.

** Tested and works, please give it a test, thanks! **

Oh, about the keeping both .env and config.py we can discuss in this
series. I was asking a question for that, but you let it ignore.

## Changelog

v3 -> v4:
  - Change commit description

v2 -> v3:
  - Fix prepare_patch() from media caption too long error.
  - Fix prepare_patch() for the .patch file.
  - Improve fix_utf8_char() function, don't use escape() twice and
    unecessary unescape().

v1 -> v2:
  - Improvement on the create_template() for the substr/trim caption.
  - Fix the prepare_patch() to use the full email payload instead of the
    trimmed caption taken from the `text` parameter.
  - Remove unrelevant changelog in the commit description.
  - Small change in the first commit description to make it clearer to
    read.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---

Muhammad Rizki (17):
  discord: Fix typo on _flood_exception()
  fix: utils: Fix .patch file payload
  fix: utils: add a substr for the patch media caption
  fix: utils: Fix the extract_list() utility function
  utils: Back to use decode=True for the get_payload()
  utils: Improve fix_utf8_char()
  utils: Add catch_err() for the log message
  logger: Initial work for the bot logger for future use
  telegram: Use the created BotLogger() class
  telegram: Add variable LOG_CHANNEL_ID declaration
  telegram: Add send_log_file() in the DaemonClient()
  telegram: Implement the log message for catching errors
  discord: Add variable LOG_CHANNEL_ID declaration
  discord: Use the BotLogger() to the GWClient()
  discord: Add send_log_file in the GWClient()
  discord: Implement the catch erros and logs
  Remove some unused imports

 daemon/atom/utils.py                          | 67 ++++++++++++++-----
 daemon/dc.py                                  | 10 ++-
 daemon/dscord/config.py.example               |  4 ++
 daemon/dscord/gnuweeb/client.py               | 20 ++++--
 daemon/dscord/gnuweeb/filters.py              | 10 ++-
 .../dscord/gnuweeb/plugins/events/on_ready.py |  2 +-
 .../plugins/slash_commands/get_lore_mail.py   | 12 ++--
 daemon/dscord/mailer/listener.py              | 41 +++++++-----
 daemon/logger/__init__.py                     |  1 +
 daemon/logger/log.py                          | 52 ++++++++++++++
 daemon/telegram/config.py.example             |  4 ++
 daemon/telegram/mailer/listener.py            | 44 +++++++-----
 daemon/telegram/packages/client.py            | 21 +++++-
 daemon/telegram/packages/decorator.py         | 15 ++---
 .../packages/plugins/commands/scrape.py       | 10 ++-
 daemon/tg.py                                  |  5 ++
 16 files changed, 240 insertions(+), 78 deletions(-)
 create mode 100644 daemon/logger/__init__.py
 create mode 100644 daemon/logger/log.py


base-commit: 91e86e6d1004da4526ca5514fa009bc253d5c4a3
--
Muhammad Rizki

[PATCH v4 01/17] discord: Fix typo on _flood_exception()

[Muhammad Rizki]

Fix minor typo. This function is for the Discord bot, so fix the error
message from "Telegram" to "Discord".

v2:
    - Change commit description to make it clearer.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/dscord/gnuweeb/filters.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/daemon/dscord/gnuweeb/filters.py b/daemon/dscord/gnuweeb/filters.py
index eb54a8a..b994904 100644
--- a/daemon/dscord/gnuweeb/filters.py
+++ b/daemon/dscord/gnuweeb/filters.py
@@ -49,5 +49,5 @@ def wait_on_limit(func):
 
 async def _flood_exceptions(e: "discord.errors.RateLimited"):
 	wait = e.retry_after
-	print(f"[wait_on_limit]: Sleeping for {wait} seconds due to Telegram limit")
+	print(f"[wait_on_limit]: Sleeping for {wait} seconds due to Discord limit")
 	await asyncio.sleep(wait)
-- 
Muhammad Rizki

[PATCH v4 02/17] fix: utils: Fix .patch file payload

[Muhammad Rizki]

From the previous version 59d20af68a05, the .patch file payload uses the
prepare_patch() `text` parameter that taken from the create_template()
`caption` returned which is the caption is already trimmed. With the
current fix, this should be use the full email payload instead so its
not bothering with the trimmed payload from the create_template().

Fixes: 59d20af68a05 ("daemon: Move prepare for patch and clean up patch functions")
Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/atom/utils.py | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py
index 03453b4..6597c41 100644
--- a/daemon/atom/utils.py
+++ b/daemon/atom/utils.py
@@ -213,7 +213,7 @@ def create_template(thread: Message, platform: Platform, to=None, cc=None):
 	return ret, files, is_patch
 
 
-def prepare_patch(mail, text, url, platform: Platform):
+def prepare_patch(mail: Message, text, url, platform: Platform):
 	tmp = gen_temp(url, platform)
 	fnm = str(mail.get("subject"))
 	sch = re.search(PATCH_PATTERN, fnm, re.IGNORECASE)
@@ -230,7 +230,23 @@ def prepare_patch(mail, text, url, platform: Platform):
 	cap = text.split("\n\n")[0]
 
 	with open(file, "wb") as f:
-		f.write(bytes(text, encoding="utf8"))
+		write_payload = bytes(f"{cap}\n\n".encode())
+
+		# sometimes an email PATCH is a multipart
+		# so, we must loop the payload first
+		# then, check if each payload is a PATCH payload
+		# or no, if it's a PATCH payload then write payload
+		# to the file.
+		if mail.is_multipart():
+			for m in mail.get_payload():
+				subject = mail.get("subject")
+				if not __is_patch(subject, str(m)):
+					continue
+				write_payload += m.get_payload(decode=True)
+		else:
+			write_payload += mail.get_payload(decode=True)
+
+		f.write(write_payload)
 
 	caption = "#patch #ml"
 	if platform is Platform.TELEGRAM:
-- 
Muhammad Rizki

[PATCH v4 03/17] fix: utils: add a substr for the patch media caption

[Muhammad Rizki]

Commit bb8855bf5e344 ("Fix the storage management after the refactor was
happened") did not care with the caption length when sending an
attachment. This leads to error when the caption length hits the
limit. Add a substr() call to cut the caption if it's too long to
avoid such an error.

P.S.: <sic> at the "was happened" part.

Corrected-by: Alviro Iskandar Setiawan <alviro.iskandar@gnuweeb.org>
Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/atom/utils.py | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py
index 6597c41..ed59812 100644
--- a/daemon/atom/utils.py
+++ b/daemon/atom/utils.py
@@ -248,9 +248,20 @@ def prepare_patch(mail: Message, text, url, platform: Platform):
 
 		f.write(write_payload)
 
-	caption = "#patch #ml"
+	caption = f"#patch #ml\n{cap}"
 	if platform is Platform.TELEGRAM:
-		caption += fix_utf8_char("\n" + cap, True)
+		# Telegram media caption is limit to 1024
+		# set limit to 1021, because we will add "..."
+		if len(caption) >= 1024:
+			caption = caption[:1021] + "..."
+
+		fixed = fix_utf8_char(caption, html_escape=True)
+		return tmp, file, fixed, url
+
+	# Discord attachment caption limit about 1998 or 2000
+	# set limit to 1995, because we will add "..."
+	if len(caption) >= 1998:
+		caption = caption[:1995] + "..."
 
 	return tmp, file, caption, url
 
-- 
Muhammad Rizki

[PATCH v4 04/17] fix: utils: Fix the extract_list() utility function

[Muhammad Rizki]

Fix the extract_list() when the `payload.get()` header function is not
retrieving the header value instead of throwing error.

Simple error log says:
AttributeError: 'Group' object has no attribute 'local_part'

Source problem: https://s.id/1n1C6 (shorted lore email URL)

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/atom/utils.py | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py
index ed59812..4f9b9b5 100644
--- a/daemon/atom/utils.py
+++ b/daemon/atom/utils.py
@@ -86,9 +86,12 @@ def __extract_list(ss):
 	return ret
 
 
-def extract_list(key: str, content: Dict[str, str]):
-	people = content.get(key.lower())
-	if not people:
+def extract_list(key: str, thread: Message):
+	try:
+		people = thread.get(key.lower())
+		if not people:
+			return []
+	except:
 		return []
 	return __extract_list(people)
 
-- 
Muhammad Rizki

[PATCH v4 05/17] utils: Back to use decode=True for the get_payload()

[Muhammad Rizki]

Because at the previous version still not solving the error, this
version is use decode=True for the get_payload() in the
get_decoded_payload() just for the base64, quoted-printable, and binary
transfer encoding only.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/atom/utils.py | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)

diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py
index 4f9b9b5..c08a7c1 100644
--- a/daemon/atom/utils.py
+++ b/daemon/atom/utils.py
@@ -10,7 +10,7 @@ from pyrogram.types import Chat, InlineKeyboardMarkup, InlineKeyboardButton
 from email.message import Message
 from typing import Dict, Union
 from slugify import slugify
-from base64 import b64decode
+import traceback
 import hashlib
 import uuid
 import os
@@ -289,11 +289,9 @@ def get_decoded_payload(payload: Message):
 	tf_encode = payload.get("Content-Transfer-Encoding")
 	charset = payload.get_content_charset("utf-8")
 
-	if tf_encode == "base64":
-		return b64decode(p).decode(charset)
-	if tf_encode == "quoted-printable":
-		quobyte = quopri.decodestring(p.encode())
-		return quobyte.decode(charset)
+	if tf_encode in ["base64", "quoted-printable", "binary"]:
+		return payload.get_payload(decode=True) \
+			.decode(errors="replace")
 
 	return p.encode().decode(charset, errors="replace")
 
-- 
Muhammad Rizki

[PATCH v4 06/17] utils: Improve fix_utf8_char()

[Muhammad Rizki]

Change on the html.escape() to use it only once. From the past
issue bb8855bf, some email message doesn't escaped correctly, so I use
the html.escape() twice. Within the current version, this issue should
be fixed and can call the html.escape() just once.

Fixes: bb8855bf ("Fix the storage management after the refactor was happened")
Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/atom/utils.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py
index c08a7c1..1502687 100644
--- a/daemon/atom/utils.py
+++ b/daemon/atom/utils.py
@@ -280,7 +280,7 @@ def remove_patch(tmp: Union[str, list]):
 def fix_utf8_char(text: str, html_escape: bool = True):
 	t = text.rstrip().replace("�"," ")
 	if html_escape:
-		t = html.escape(html.escape(text))
+		return html.escape(t)
 	return t
 
 
-- 
Muhammad Rizki

[PATCH v4 07/17] utils: Add catch_err() for the log message

[Muhammad Rizki]

Add catch_err() for the logger message in the future use.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/atom/utils.py | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py
index 1502687..7e41674 100644
--- a/daemon/atom/utils.py
+++ b/daemon/atom/utils.py
@@ -296,6 +296,12 @@ def get_decoded_payload(payload: Message):
 	return p.encode().decode(charset, errors="replace")
 
 
+def catch_err():
+	return traceback.format_exc() \
+	.encode("unicode_escape") \
+	.decode("utf-8")
+
+
 EMAIL_MSG_ID_PATTERN = r"<([^\<\>]+)>"
 def extract_email_msg_id(msg_id):
 	ret = re.search(EMAIL_MSG_ID_PATTERN, msg_id)
-- 
Muhammad Rizki

[PATCH v4 08/17] logger: Initial work for the bot logger for future use

[Muhammad Rizki]

Add a BotLogger() class that inherit to Logger() class from the python
logging library.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/logger/__init__.py |  1 +
 daemon/logger/log.py      | 52 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 53 insertions(+)
 create mode 100644 daemon/logger/__init__.py
 create mode 100644 daemon/logger/log.py

diff --git a/daemon/logger/__init__.py b/daemon/logger/__init__.py
new file mode 100644
index 0000000..bb3145a
--- /dev/null
+++ b/daemon/logger/__init__.py
@@ -0,0 +1 @@
+from .log import BotLogger
diff --git a/daemon/logger/log.py b/daemon/logger/log.py
new file mode 100644
index 0000000..8382404
--- /dev/null
+++ b/daemon/logger/log.py
@@ -0,0 +1,52 @@
+import logging
+import os
+from enums import Platform
+
+
+class BotLogger(logging.Logger):
+
+	def __init__(
+		self,
+		platform: Platform = Platform.TELEGRAM,
+		level: int = logging.WARNING
+	) -> None:
+
+		self.platform = str(platform.value)
+		self.level = level
+
+		plat_rep = self.platform.replace("discord", "dscord")
+		self.file_name = f"{plat_rep}/" \
+			f"{os.getenv('STORAGE_DIR', 'storage')}/" \
+			f"{self.platform}.log"
+
+		self.file_fmt = "{" \
+			"'time': '%(asctime)s'," \
+			"'func': '%(name)s: %(funcName)s'," \
+			"'path': '%(pathname)s'," \
+			"'level': '%(levelname)s'," \
+			"'msg': '%(message)s'"\
+		"}"
+
+		self.stream_fmt = "%(asctime)s | " \
+			"%(name)s: %(funcName)s | " \
+			"%(levelname)s | " \
+			"%(message)s"
+
+
+	def init(self):
+		super().__init__(self.platform, self.level)
+
+		file_handler = logging.FileHandler(self.file_name)
+		stream_handler = logging.StreamHandler()
+
+		file_fmtr = logging.Formatter(self.file_fmt)
+		stream_fmtr = logging.Formatter(self.stream_fmt)
+
+		file_handler.setFormatter(file_fmtr)
+		stream_handler.setFormatter(stream_fmtr)
+
+		file_handler.setLevel(logging.WARNING)
+		self.setLevel(logging.INFO)
+
+		self.addHandler(file_handler)
+		self.addHandler(stream_handler)
-- 
Muhammad Rizki

[PATCH v4 09/17] telegram: Use the created BotLogger() class

[Muhammad Rizki]

Use the BotLogger() class for the Telegram DaemonClient() class. This
class is to make a log message to a file within the DaemonClient() class
of the `.logger` attribute which is initialized from the BotLogger()
class.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/telegram/packages/client.py | 6 +++++-
 daemon/tg.py                       | 5 +++++
 2 files changed, 10 insertions(+), 1 deletion(-)

diff --git a/daemon/telegram/packages/client.py b/daemon/telegram/packages/client.py
index c971ea1..eb282eb 100644
--- a/daemon/telegram/packages/client.py
+++ b/daemon/telegram/packages/client.py
@@ -10,16 +10,20 @@ from typing import Union
 from email.message import Message
 from atom import utils
 from enums import Platform
+from logger import BotLogger
 from telegram.database import DB
 from .decorator import handle_flood
 
 
 class DaemonClient(Client):
 	def __init__(self, name: str, api_id: int,
-		api_hash: str, conn, **kwargs):
+		api_hash: str, conn, logger: BotLogger,
+		**kwargs
+	):
 		super().__init__(name, api_id,
 				api_hash, **kwargs)
 		self.db = DB(conn)
+		self.logger = logger
 
 
 	@handle_flood
diff --git a/daemon/tg.py b/daemon/tg.py
index 5f8c21e..8382c29 100644
--- a/daemon/tg.py
+++ b/daemon/tg.py
@@ -11,17 +11,22 @@ from atom import Scraper
 from telegram.packages import DaemonClient
 from telegram.mailer import BotMutexes
 from telegram.mailer import Bot
+from logger import BotLogger
 import os
 
 
 def main():
 	load_dotenv("telegram.env")
 
+	logger = BotLogger()
+	logger.init()
+
 	client = DaemonClient(
 		"telegram/storage/EmailScraper",
 		api_id=int(os.getenv("API_ID")),
 		api_hash=os.getenv("API_HASH"),
 		bot_token=os.getenv("BOT_TOKEN"),
+		logger=logger,
 		conn=connector.connect(
 			host=os.getenv("DB_HOST"),
 			user=os.getenv("DB_USER"),
-- 
Muhammad Rizki

[PATCH v4 10/17] telegram: Add variable LOG_CHANNEL_ID declaration

[Muhammad Rizki]

Declare a variable LOG_CHANNEL_ID in the config.py.example for the
send_log_file() in the next commit.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/telegram/config.py.example | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/daemon/telegram/config.py.example b/daemon/telegram/config.py.example
index 3818d27..221e0c1 100644
--- a/daemon/telegram/config.py.example
+++ b/daemon/telegram/config.py.example
@@ -4,6 +4,10 @@ from pyrogram.types import Message
 # Insert Telegram admin id in the list below
 ADMINS = []
 
+# Insert Telegram channel ID to send the log file
+# type (int)
+LOG_CHANNEL_ID =
+
 
 async def adm_flt(_, __, m: Message):
 	return m.from_user.id in ADMINS
-- 
Muhammad Rizki

[PATCH v4 11/17] telegram: Add send_log_file() in the DaemonClient()

[Muhammad Rizki]

Add send_log_file() to the DaemonClient() class.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/telegram/packages/client.py | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/daemon/telegram/packages/client.py b/daemon/telegram/packages/client.py
index eb282eb..6c83a5a 100644
--- a/daemon/telegram/packages/client.py
+++ b/daemon/telegram/packages/client.py
@@ -11,6 +11,7 @@ from email.message import Message
 from atom import utils
 from enums import Platform
 from logger import BotLogger
+from telegram import config
 from telegram.database import DB
 from .decorator import handle_flood
 
@@ -26,6 +27,16 @@ class DaemonClient(Client):
 		self.logger = logger
 
 
+	@handle_flood
+	async def send_log_file(self, caption: str):
+		filename = self.logger.handlers[0].baseFilename
+		await self.send_document(
+			config.LOG_CHANNEL_ID,
+			filename,
+			caption=caption
+		)
+
+
 	@handle_flood
 	async def send_text_email(
 		self,
-- 
Muhammad Rizki

[PATCH v4 12/17] telegram: Implement the log message for catching errors

[Muhammad Rizki]

After created the BotLogger() class and initialized it to the
DaemonClient() attribute, implement it in the
telegram/mailer/listener.py to log it into the file if catching errors.

v2:
    - Remove unrelevant changelog

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/telegram/mailer/listener.py            | 44 ++++++++++++-------
 daemon/telegram/packages/client.py            |  4 +-
 daemon/telegram/packages/decorator.py         | 15 +++----
 .../packages/plugins/commands/scrape.py       | 10 ++++-
 4 files changed, 45 insertions(+), 28 deletions(-)

diff --git a/daemon/telegram/mailer/listener.py b/daemon/telegram/mailer/listener.py
index 46ccf93..6fb2e44 100644
--- a/daemon/telegram/mailer/listener.py
+++ b/daemon/telegram/mailer/listener.py
@@ -13,7 +13,6 @@ from atom import utils
 from enums import Platform
 import asyncio
 import re
-import traceback
 
 
 class BotMutexes():
@@ -29,6 +28,7 @@ class Bot():
 		self.scraper = scraper
 		self.mutexes = mutexes
 		self.db = client.db
+		self.logger = client.logger
 		self.isRunnerFixed = False
 
 
@@ -49,8 +49,8 @@ class Bot():
 		# TODO(ammarfaizi2):
 		# Ideally, we also want to log and report this situation.
 		#
-		print(f"Database error: {str(e)}")
-		print("Reconnecting to the database...")
+		self.logger.error(f"Database error: {str(e)}")
+		self.logger.info("Reconnecting to the database...")
 
 		#
 		# Don't do this too often to avoid reconnect burst.
@@ -63,14 +63,17 @@ class Bot():
 
 
 	async def __run(self):
-		print("[__run]: Running...")
+		self.logger.info("Running...")
 		try:
 			for url in self.db.get_atom_urls():
 				await self.__handle_atom_url(url)
 		except (OperationalError, DatabaseError) as e:
 			await self.handle_db_error(e)
 		except:
-			print(traceback.format_exc())
+			exc_str = utils.catch_err()
+			self.logger.warning(exc_str)
+			capt = "Unkown raw lore URL, see full details in the log file."
+			await self.client.send_log_file(capt)
 
 		if not self.isRunnerFixed:
 			self.isRunnerFixed = True
@@ -86,8 +89,13 @@ class Bot():
 	async def __handle_atom_url(self, url):
 		urls = await self.scraper.get_new_threads_urls(url)
 		for url in urls:
-			mail = await self.scraper.get_email_from_url(url)
-			await self.__handle_mail(url, mail)
+			try:
+				mail = await self.scraper.get_email_from_url(url)
+				await self.__handle_mail(url, mail)
+			except:
+				exc_str = utils.catch_err()
+				self.logger.warning(exc_str)
+				await self.client.send_log_file(url)
 
 
 	async def __handle_mail(self, url, mail):
@@ -104,23 +112,27 @@ class Bot():
 	# @__must_hold(self.mutexes.send_to_tg)
 	async def __send_mail(self, url, mail, tg_chat_id):
 		email_msg_id = utils.get_email_msg_id(mail)
+
 		if not email_msg_id:
-			#
-			# It doesn't have a Message-Id.
-			# A malformed email. Skip!
-			#
+			md = "email_msg_id not detected, skipping malformed email"
+			self.logger.debug(md)
 			return False
 
 		email_id = self.__mail_id_from_db(email_msg_id,
 							tg_chat_id)
 		if not email_id:
-			#
-			# Email has already been sent to Telegram.
-			# Skip!
-			#
+			md = f"Skipping {email_id} because has already been sent to Telegram"
+			self.logger.debug(md)
 			return False
 
-		text, files, is_patch = utils.create_template(mail, Platform.TELEGRAM)
+		try:
+			text, files, is_patch = utils.create_template(mail, Platform.TELEGRAM)
+		except:
+			exc_str = utils.catch_err()
+			self.logger.warning(exc_str)
+			await self.client.send_log_file(url)
+			return
+
 		reply_to = self.get_reply(mail, tg_chat_id)
 		url = str(re.sub(r"/raw$", "", url))
 
diff --git a/daemon/telegram/packages/client.py b/daemon/telegram/packages/client.py
index 6c83a5a..58195b9 100644
--- a/daemon/telegram/packages/client.py
+++ b/daemon/telegram/packages/client.py
@@ -46,7 +46,7 @@ class DaemonClient(Client):
 		url: str = None,
 		parse_mode: ParseMode = ParseMode.HTML
 	) -> Message:
-		print("[send_text_email]")
+		self.logger.debug("[send_text_email]")
 		return await self.send_message(
 			chat_id=chat_id,
 			text=text,
@@ -71,7 +71,7 @@ class DaemonClient(Client):
 		url: str = None,
 		parse_mode: ParseMode = ParseMode.HTML
 	) -> Message:
-		print("[send_patch_email]")
+		self.logger.debug("[send_patch_email]")
 		tmp, doc, caption, url = utils.prepare_patch(
 			mail, text, url, Platform.TELEGRAM
 		)
diff --git a/daemon/telegram/packages/decorator.py b/daemon/telegram/packages/decorator.py
index c7a5f02..153fa95 100644
--- a/daemon/telegram/packages/decorator.py
+++ b/daemon/telegram/packages/decorator.py
@@ -24,20 +24,19 @@ def handle_flood(func: Callable[[T], T]) -> Callable[[T], T]:
 			try:
 				return await func(*args)
 			except FloodWait as e:
-				#
-				# Aiee... we hit our limit.
-				# Let's slow down a bit.
-				#
-				_flood_exceptions(e)
-				print("[__handle_telegram_floodwait]: Woken up from flood wait...")
+				# Calling logger attr from the DaemonClient() class
+				logger = args[0].logger
+
+				_flood_exceptions(e, logger)
+				logger.info("Woken up from flood wait...")
 	return callback
 
 
-async def _flood_exceptions(e):
+async def _flood_exceptions(e, logger):
 	x = re.search(r"A wait of (\d+) seconds is required", str(e))
 	if not x:
 		raise e
 
 	n = int(x.group(1))
-	print(f"[____handle_telegram_floodwait]: Sleeping for {n} seconds due to Telegram limit")
+	logger.info(f"Sleeping for {n} seconds due to Telegram limit")
 	await asyncio.sleep(n)
diff --git a/daemon/telegram/packages/plugins/commands/scrape.py b/daemon/telegram/packages/plugins/commands/scrape.py
index 29cc8ad..f89727d 100644
--- a/daemon/telegram/packages/plugins/commands/scrape.py
+++ b/daemon/telegram/packages/plugins/commands/scrape.py
@@ -36,8 +36,14 @@ async def scrap_email(c: DaemonClient, m: Message):
 		return
 
 	s = Scraper()
-	mail = await s.get_email_from_url(url)
-	text, files, is_patch = utils.create_template(mail, Platform.TELEGRAM)
+
+	try:
+		mail = await s.get_email_from_url(url)
+		text, files, is_patch = utils.create_template(mail, Platform.TELEGRAM)
+	except:
+		exc_str = utils.catch_err()
+		c.logger.warning(exc_str)
+		await c.send_log_file(url)
 
 	if is_patch:
 		m = await c.send_patch_email(
-- 
Muhammad Rizki

[PATCH v4 13/17] discord: Add variable LOG_CHANNEL_ID declaration

[Muhammad Rizki]

Add LOG_CHANNEL_ID into the config.py.example to use it in the future
use to send the log file to its Discord channel logs using the
LOG_CHANNEL_ID.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/dscord/config.py.example | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/daemon/dscord/config.py.example b/daemon/dscord/config.py.example
index 6dd4313..6db826c 100644
--- a/daemon/dscord/config.py.example
+++ b/daemon/dscord/config.py.example
@@ -11,3 +11,7 @@ ADMIN_ROLE_ID =
 # The activity name like "Playing Genshin Impact"
 # you set the value as "Genshin Impact"
 ACTIVITY_NAME =
+
+# Insert Discord channel ID to send the log file
+# type (int)
+LOG_CHANNEL_ID =
-- 
Muhammad Rizki

[PATCH v4 14/17] discord: Use the BotLogger() to the GWClient()

[Muhammad Rizki]

Use the BotLogger() and initialized it to the GWClient() class attribute
`.logger` to use it in the future use for log a message into the file.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/dc.py                    | 10 ++++++++--
 daemon/dscord/gnuweeb/client.py |  6 ++++--
 2 files changed, 12 insertions(+), 4 deletions(-)

diff --git a/daemon/dc.py b/daemon/dc.py
index 5028e94..f9b57e5 100644
--- a/daemon/dc.py
+++ b/daemon/dc.py
@@ -11,6 +11,8 @@ from apscheduler.schedulers.asyncio import AsyncIOScheduler
 from dscord.gnuweeb import GWClient
 from dscord.mailer import Listener
 from dscord.mailer import Mutexes
+from enums.platform import Platform
+from logger import BotLogger
 from atom import Scraper
 
 
@@ -24,13 +26,17 @@ def main():
 		}
 	)
 
+	logger = BotLogger(Platform.DISCORD)
+	logger.init()
+
 	client = GWClient(
 		db_conn=connector.connect(
 			host=os.getenv("DB_HOST"),
 			user=os.getenv("DB_USER"),
 			password=os.getenv("DB_PASS"),
 			database=os.getenv("DB_NAME")
-		)
+		),
+		logger=logger
 	)
 
 	mailer = Listener(
@@ -39,8 +45,8 @@ def main():
 		scraper=Scraper(),
 		mutexes=Mutexes()
 	)
-	client.mailer = mailer
 
+	client.mailer = mailer
 	client.run(os.getenv("DISCORD_TOKEN"), log_handler=None)
 
 
diff --git a/daemon/dscord/gnuweeb/client.py b/daemon/dscord/gnuweeb/client.py
index 82858c5..42bd405 100644
--- a/daemon/dscord/gnuweeb/client.py
+++ b/daemon/dscord/gnuweeb/client.py
@@ -7,22 +7,24 @@ import discord
 from discord import Interaction
 from discord.ext import commands
 from discord import Intents
-from dscord.config import ACTIVITY_NAME
 from typing import Union
 
 from . import filters
 from . import models
 from atom import utils
 from enums import Platform
+from logger.log import BotLogger
+from dscord.config import ACTIVITY_NAME
 from dscord.database import DB
 
 
 class GWClient(commands.Bot):
-	def __init__(self, db_conn) -> None:
+	def __init__(self, db_conn, logger: BotLogger) -> None:
 		self.db = DB(db_conn)
 		intents = Intents.default()
 		intents.message_content = True
 		self.mailer = None
+		self.logger = logger
 		super().__init__(
 			command_prefix=["$", "."],
 			description="Just a bot for receiving lore emails.",
-- 
Muhammad Rizki

[PATCH v4 15/17] discord: Add send_log_file in the GWClient()

[Muhammad Rizki]

Add send_log_file() to send a log file into the Discord channel for logs
using the `config.LOG_CHANNEL_ID` to send it to the Discord channel.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/dscord/gnuweeb/client.py | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/daemon/dscord/gnuweeb/client.py b/daemon/dscord/gnuweeb/client.py
index 42bd405..1b260ea 100644
--- a/daemon/dscord/gnuweeb/client.py
+++ b/daemon/dscord/gnuweeb/client.py
@@ -14,7 +14,7 @@ from . import models
 from atom import utils
 from enums import Platform
 from logger.log import BotLogger
-from dscord.config import ACTIVITY_NAME
+from dscord.config import ACTIVITY_NAME, LOG_CHANNEL_ID
 from dscord.database import DB
 
 
@@ -40,6 +40,16 @@ class GWClient(commands.Bot):
 		)
 
 
+	@filters.wait_on_limit
+	async def send_log_file(self, caption: str):
+		filename = self.logger.handlers[0].baseFilename
+		channel = self.get_channel(LOG_CHANNEL_ID)
+		await channel.send(
+			content=caption,
+			file=discord.File(filename)
+		)
+
+
 	@filters.wait_on_limit
 	async def send_text_email(self, guild_id: int, chat_id: int, text: str,
 				reply_to: Union[int, None] = None, url: str = None):
-- 
Muhammad Rizki

[PATCH v4 16/17] discord: Implement the catch erros and logs

[Muhammad Rizki]

Implement the catch erros and log the message into the file then send
it into the Discord log channel.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/dscord/gnuweeb/client.py               |  4 +-
 daemon/dscord/gnuweeb/filters.py              | 10 +++--
 .../dscord/gnuweeb/plugins/events/on_ready.py |  2 +-
 .../plugins/slash_commands/get_lore_mail.py   | 10 ++++-
 daemon/dscord/mailer/listener.py              | 40 +++++++++++--------
 5 files changed, 41 insertions(+), 25 deletions(-)

diff --git a/daemon/dscord/gnuweeb/client.py b/daemon/dscord/gnuweeb/client.py
index 1b260ea..406c606 100644
--- a/daemon/dscord/gnuweeb/client.py
+++ b/daemon/dscord/gnuweeb/client.py
@@ -53,7 +53,7 @@ class GWClient(commands.Bot):
 	@filters.wait_on_limit
 	async def send_text_email(self, guild_id: int, chat_id: int, text: str,
 				reply_to: Union[int, None] = None, url: str = None):
-		print("[send_text_email]")
+		self.logger.debug("[send_text_email]")
 		channel = self.get_channel(chat_id)
 
 		return await channel.send(
@@ -70,7 +70,7 @@ class GWClient(commands.Bot):
 	@filters.wait_on_limit
 	async def send_patch_email(self, mail, guild_id: int, chat_id: int, text: str,
 				reply_to: Union[int, None] = None, url: str = None):
-		print("[send_patch_email]")
+		self.logger.debug("[send_patch_email]")
 		tmp, doc, caption, url = utils.prepare_patch(
 			mail, text, url, Platform.DISCORD
 		)
diff --git a/daemon/dscord/gnuweeb/filters.py b/daemon/dscord/gnuweeb/filters.py
index b994904..1fdb70d 100644
--- a/daemon/dscord/gnuweeb/filters.py
+++ b/daemon/dscord/gnuweeb/filters.py
@@ -15,6 +15,7 @@ from discord import Interaction
 
 # gnuweeb package import
 from dscord import config
+from logger import BotLogger
 
 
 def lore_admin(func):
@@ -42,12 +43,15 @@ def wait_on_limit(func):
 			try:
 				return await func(*args)
 			except discord.errors.RateLimited as e:
+				# Calling logger attr from the GWClient() class
+				logger = args[0].logger
+
 				_flood_exceptions(e)
-				print("[wait_on_limit]: Woken up from flood wait...")
+				logger.info("Woken up from flood wait...")
 	return callback
 
 
-async def _flood_exceptions(e: "discord.errors.RateLimited"):
+async def _flood_exceptions(e: "discord.errors.RateLimited", logger: BotLogger):
 	wait = e.retry_after
-	print(f"[wait_on_limit]: Sleeping for {wait} seconds due to Discord limit")
+	logger.info(f"Sleeping for {wait} seconds due to Discord limit")
 	await asyncio.sleep(wait)
diff --git a/daemon/dscord/gnuweeb/plugins/events/on_ready.py b/daemon/dscord/gnuweeb/plugins/events/on_ready.py
index 4f25764..e7f63cd 100644
--- a/daemon/dscord/gnuweeb/plugins/events/on_ready.py
+++ b/daemon/dscord/gnuweeb/plugins/events/on_ready.py
@@ -23,4 +23,4 @@ class OnReady(commands.Cog):
 		t += f"Send `{prefix}sync` message to the Discord channel "
 		t += "where the bot is running.\n"
 
-		print(t)
+		self.bot.logger.info(t)
diff --git a/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py b/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py
index 0c67b8c..e4c171c 100644
--- a/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py
+++ b/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py
@@ -26,8 +26,14 @@ class GetLoreSC(commands.Cog):
 	@app_commands.describe(url="Raw lore email URL")
 	async def get_lore(self, i: "Interaction", url: str):
 		s = Scraper()
-		mail = await s.get_email_from_url(url)
-		text, _, is_patch = utils.create_template(mail, Platform.DISCORD)
+
+		try:
+			mail = await s.get_email_from_url(url)
+			text, _, is_patch = utils.create_template(mail, Platform.DISCORD)
+		except:
+			exc_str = utils.catch_err()
+			self.bot.logger.warning(exc_str)
+			await self.bot.send_log_file(url)
 
 		if is_patch:
 			m = await self.bot.send_patch_mail_interaction(
diff --git a/daemon/dscord/mailer/listener.py b/daemon/dscord/mailer/listener.py
index d986fbd..1764ea2 100644
--- a/daemon/dscord/mailer/listener.py
+++ b/daemon/dscord/mailer/listener.py
@@ -35,6 +35,7 @@ class Listener:
 		self.scraper = scraper
 		self.mutexes = mutexes
 		self.db = client.db
+		self.logger = client.logger
 		self.isRunnerFixed = False
 		self.runner = None
 
@@ -44,18 +45,15 @@ class Listener:
 		# Execute __run() once to avoid high latency at
 		# initilization.
 		#
-		print("Initialize listener...\n")
+		self.logger.info("Initialize listener...\n")
 		self.sched.start()
 		self.runner = self.sched.add_job(func=self.__run)
 
 
 	async def __run(self):
-		print("[__run]: Running...")
+		self.logger.info("Running...")
 		for url in self.db.get_atom_urls():
-			try:
-				await self.__handle_atom_url(url)
-			except:
-				print(traceback.format_exc())
+			await self.__handle_atom_url(url)
 
 		if not self.isRunnerFixed:
 			self.isRunnerFixed = True
@@ -71,8 +69,13 @@ class Listener:
 	async def __handle_atom_url(self, url):
 		urls = await self.scraper.get_new_threads_urls(url)
 		for url in urls:
-			mail = await self.scraper.get_email_from_url(url)
-			await self.__handle_mail(url, mail)
+			try:
+				mail = await self.scraper.get_email_from_url(url)
+				await self.__handle_mail(url, mail)
+			except:
+				exc_str = utils.catch_err()
+				self.client.logger.warning(exc_str)
+				await self.client.send_log_file(url)
 
 
 	async def __handle_mail(self, url, mail):
@@ -91,10 +94,8 @@ class Listener:
 	async def __send_to_discord(self, url, mail, dc_guild_id, dc_chat_id):
 		email_msg_id = utils.get_email_msg_id(mail)
 		if not email_msg_id:
-			#
-			# It doesn't have a Message-Id.
-			# A malformed email. Skip!
-			#
+			md = "email_msg_id not detected, skipping malformed email"
+			self.logger.debug(md)
 			return False
 
 		email_id = self.__get_email_id_sent(
@@ -102,13 +103,18 @@ class Listener:
 			dc_chat_id=dc_chat_id
 		)
 		if not email_id:
-			#
-			# Email has already been sent to Discord.
-			# Skip!
-			#
+			md = f"Skipping {email_id} because has already been sent to Discord"
+			self.logger.debug(md)
 			return False
 
-		text, files, is_patch = utils.create_template(mail, Platform.DISCORD)
+		try:
+			text, files, is_patch = utils.create_template(mail, Platform.DISCORD)
+		except:
+			exc_str = utils.catch_err()
+			self.client.logger.warning(exc_str)
+			await self.client.send_log_file(url)
+			return
+
 		reply_to = self.get_discord_reply(mail, dc_chat_id)
 		url = str(re.sub(r"/raw$", "", url))
 
-- 
Muhammad Rizki

[PATCH v4 17/17] Remove some unused imports

[Muhammad Rizki]

Remove them while it is not used again, from the previous version, they
are not removed.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/atom/utils.py                                          | 3 +--
 daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py | 2 --
 daemon/dscord/mailer/listener.py                              | 1 -
 3 files changed, 1 insertion(+), 5 deletions(-)

diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py
index 7e41674..80c74b0 100644
--- a/daemon/atom/utils.py
+++ b/daemon/atom/utils.py
@@ -8,7 +8,7 @@ from enums import Platform
 
 from pyrogram.types import Chat, InlineKeyboardMarkup, InlineKeyboardButton
 from email.message import Message
-from typing import Dict, Union
+from typing import Union
 from slugify import slugify
 import traceback
 import hashlib
@@ -18,7 +18,6 @@ import re
 import shutil
 import httpx
 import html
-import quopri
 
 
 def get_email_msg_id(mail):
diff --git a/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py b/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py
index e4c171c..56330f3 100644
--- a/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py
+++ b/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py
@@ -3,8 +3,6 @@
 # Copyright (C) 2022  Muhammad Rizki <kiizuha@gnuweeb.org>
 #
 
-import asyncio
-import discord
 from discord.ext import commands
 from discord import Interaction
 from discord import app_commands
diff --git a/daemon/dscord/mailer/listener.py b/daemon/dscord/mailer/listener.py
index 1764ea2..24f0a6c 100644
--- a/daemon/dscord/mailer/listener.py
+++ b/daemon/dscord/mailer/listener.py
@@ -5,7 +5,6 @@
 #
 
 import asyncio
-import traceback
 import re
 from apscheduler.schedulers.asyncio import AsyncIOScheduler
 from discord import File
-- 
Muhammad Rizki

Re: [PATCH v4 00/17] Fix, improvement and implement a bot logger.

[Ammar Faizi]

From: Ammar Faizi <ammarfaizi2@gnuweeb.org>

On Wed, 21 Dec 2022 08:33:30 +0700, Muhammad Rizki wrote:
> This is a v4 revision of fix, improvement and implement a bot logger.
> this series is to improve create_template() caption substr/trim,
> prepare_patch() for the .patch file payload, remove unrelevant changelog
> commit in the commit description, use html.escape() once instead of
> twice, small change in the first commit to make it clearer.
> 
> This series contains fixes in the utility file and some improvement for
> it. I also implement a bot logger to log an error to the terminal and
> to the log file, if error may happen it will send the log file to the
> log channel either for Discord or Telegram, the LOG_CHANNEL_ID is
> declared in the config.py.example, copy the channel ID from the Discord
> or Telegram channel and then paste to the config.py.example.
> 
> [...]

Applied, thanks!

[01/17] discord: Fix typo on _flood_exception()
        commit: b58e0f7517a009cc212b467bf19156363c6e591e
[02/17] utils: Fix .patch file payload
        commit: ad154f246b31488fb4cd104c0c1464e215d08f72
[03/17] utils: add a substr for the patch media caption
        commit: 82edf7de491b18104c2f055388dcd989044cda0b
[04/17] utils: Fix the extract_list() utility function
        commit: 97e8352ad0b6163ffad33ff1eba5059c234dfa53
[05/17] utils: Back to use decode=True for the get_payload()
        commit: 087e1ac70426fff0855027408babec6cba06d2b8
[06/17] utils: Improve fix_utf8_char()
        commit: 97b217113d3802004a49dfde7e25a5c5aaafce74
[07/17] utils: Add catch_err() for the log message
        commit: f24336d484e17b04dafd0b0c9377482908e45fa2
[08/17] logger: Initial work for the bot logger for future use
        commit: 5a377dcb2ae2b0d71c373dcabf66b69d6d07dfdd
[09/17] telegram: Use the created BotLogger() class
        commit: bd20b32fab4f34112723a5ce348abbaeedbae40b
[10/17] telegram: Add variable LOG_CHANNEL_ID declaration
        commit: f1c4f54f7dbb460427538d5f491b9a87f15d80d0
[11/17] telegram: Add send_log_file() in the DaemonClient()
        commit: 5d67dc4b1085f472b8621c3b8bcda6f8eef907ce
[12/17] telegram: Implement the log message for catching errors
        commit: ce17f3e8016ca72a34a68bbe89edd0a3fa14d1b3
[13/17] discord: Add variable LOG_CHANNEL_ID declaration
        commit: 3749c653f7eb71a9e6783149e9c21244587446ae
[14/17] discord: Use the BotLogger() to the GWClient()
        commit: fe798886e189885d6e8f3d67e9c24d49cf1a0a88
[15/17] discord: Add send_log_file in the GWClient()
        commit: 9448095a77af095f36403c260f89dee0b694d172
[16/17] discord: Implement the catch erros and logs
        commit: f7d88b1a831c65e408e170d48a5b9dd5e2cca577
[17/17] Remove some unused imports
        commit: 16d7f9acc52518bf3d6410d736673d3759ade3dc

Best regards,
-- 
Ammar Faizi

Re: [PATCH v4 00/17] Fix, improvement and implement a bot logger.

[Ammar Faizi]

On 12/21/22 8:33 AM, Muhammad Rizki wrote:
> Hi sir, it's been a long time.
> 
> This is a v4 revision of fix, improvement and implement a bot logger.
> this series is to improve create_template() caption substr/trim,
> prepare_patch() for the .patch file payload, remove unrelevant changelog
> commit in the commit description, use html.escape() once instead of
> twice, small change in the first commit to make it clearer.

I applied this series. But I have an issue with the Discord daemon
when I restart the MySQL server. It can't recover from MySQL restart.

When you restart the MySQL server, it's always looping in the MySQL
error and keep sending the error log on Discord:

    "Lost connection to MySQL server during query"

Can you please fix it?

-- 
Ammar Faizi

Re: [PATCH v4 00/17] Fix, improvement and implement a bot logger.

[Ammar Faizi]

On 12/25/22 4:05 AM, Ammar Faizi wrote:
> On 12/21/22 8:33 AM, Muhammad Rizki wrote:
>> Hi sir, it's been a long time.
>>
>> This is a v4 revision of fix, improvement and implement a bot logger.
>> this series is to improve create_template() caption substr/trim,
>> prepare_patch() for the .patch file payload, remove unrelevant changelog
>> commit in the commit description, use html.escape() once instead of
>> twice, small change in the first commit to make it clearer.
> 
> I applied this series. But I have an issue with the Discord daemon
> when I restart the MySQL server. It can't recover from MySQL restart.
> 
> When you restart the MySQL server, it's always looping in the MySQL
> error and keep sending the error log on Discord:
> 
>     "Lost connection to MySQL server during query"
> 
> Can you please fix it?

This happens on the Telegram daemon too. I thought I fixed it in:

    529d804eb47518 ("daemon: telegram: Handle MySQL error")

but somehow my fix is no longer applicable on top of your series.

-- 
Ammar Faizi

Re: [PATCH v4 12/17] telegram: Implement the log message for catching errors

[Ammar Faizi]

On 12/21/22 8:33 AM, Muhammad Rizki wrote:
> @@ -63,14 +63,17 @@ class Bot():
>   
>   
>   	async def __run(self):
> -		print("[__run]: Running...")
> +		self.logger.info("Running...")
>   		try:
>   			for url in self.db.get_atom_urls():
>   				await self.__handle_atom_url(url)
>   		except (OperationalError, DatabaseError) as e:
>   			await self.handle_db_error(e)
>   		except:
> -			print(traceback.format_exc())
> +			exc_str = utils.catch_err()
> +			self.logger.warning(exc_str)
> +			capt = "Unkown raw lore URL, see full details in the log file."
> +			await self.client.send_log_file(capt)

At this point, @url may or may not be defined. If self.db.get_atom_urls()
succeeds and return more than 0 result(s), @url is defined.

>   		if not self.isRunnerFixed:
>   			self.isRunnerFixed = True
> @@ -86,8 +89,13 @@ class Bot():
>   	async def __handle_atom_url(self, url):
>   		urls = await self.scraper.get_new_threads_urls(url)
>   		for url in urls:
> -			mail = await self.scraper.get_email_from_url(url)
> -			await self.__handle_mail(url, mail)
> +			try:
> +				mail = await self.scraper.get_email_from_url(url)
> +				await self.__handle_mail(url, mail)
> +			except:
> +				exc_str = utils.catch_err()
> +				self.logger.warning(exc_str)
> +				await self.client.send_log_file(url)

This part is problematic, if you have MySQL error here, the previous
'try and except' statement won't catch it because it's already caught
by your error logging.

Your error logging kills the recovery function that the caller function
('__run') provides.

-- 
Ammar Faizi

Re: [PATCH v4 12/17] telegram: Implement the log message for catching errors

[Ammar Faizi]

On 12/25/22 4:22 AM, Ammar Faizi wrote:
> This part is problematic, if you have MySQL error here, the previous
> 'try and except' statement won't catch it because it's already caught
> by your error logging.
> 
> Your error logging kills the recovery function that the caller function
> ('__run') provides.

I just fixed this myself in commit:

    3eb6a95d6028 ("telegram: listener: Fix missing MySQL error recovery function")

It only fixes the Telegram daemon, I'll wait for your patch to fix
the Discord daemon.

-- 
Ammar Faizi

Re: [PATCH v4 00/17] Fix, improvement and implement a bot logger.

[Muhammad Rizki]

On 25/12/2022 04.09, Ammar Faizi wrote:
> On 12/25/22 4:05 AM, Ammar Faizi wrote:
>> On 12/21/22 8:33 AM, Muhammad Rizki wrote:
>>> Hi sir, it's been a long time.
>>>
>>> This is a v4 revision of fix, improvement and implement a bot logger.
>>> this series is to improve create_template() caption substr/trim,
>>> prepare_patch() for the .patch file payload, remove unrelevant changelog
>>> commit in the commit description, use html.escape() once instead of
>>> twice, small change in the first commit to make it clearer.
>>
>> I applied this series. But I have an issue with the Discord daemon
>> when I restart the MySQL server. It can't recover from MySQL restart.
>>
>> When you restart the MySQL server, it's always looping in the MySQL
>> error and keep sending the error log on Discord:
>>
>>     "Lost connection to MySQL server during query"
>>
>> Can you please fix it?
> 
> This happens on the Telegram daemon too. I thought I fixed it in:
> 
>     529d804eb47518 ("daemon: telegram: Handle MySQL error")
> 
> but somehow my fix is no longer applicable on top of your series.
> 

Can you give me the tips how to reproduce this error? Or I must wait or 
idle the bot? Or I just disconnect my internet and then reconnect and 
see what will happen on MySQL connection?

Re: [PATCH v4 00/17] Fix, improvement and implement a bot logger.

[Ammar Faizi]

On 12/25/22 6:31 AM, Muhammad Rizki wrote:
> Can you give me the tips how to reproduce this error? Or I must wait
> or idle the bot? Or I just disconnect my internet and then reconnect
> and see what will happen on MySQL connection?

Just restart your "MySQL server service" when the daemon is running.

    sudo service mysql restart

or whatever the MySQL restart procedure on your machine.

-- 
Ammar Faizi

Re: [PATCH v4 00/17] Fix, improvement and implement a bot logger.

[Muhammad Rizki]

On 25/12/2022 06.31, Muhammad Rizki wrote:
> On 25/12/2022 04.09, Ammar Faizi wrote:
>> On 12/25/22 4:05 AM, Ammar Faizi wrote:
>>> On 12/21/22 8:33 AM, Muhammad Rizki wrote:
>>>> Hi sir, it's been a long time.
>>>>
>>>> This is a v4 revision of fix, improvement and implement a bot logger.
>>>> this series is to improve create_template() caption substr/trim,
>>>> prepare_patch() for the .patch file payload, remove unrelevant 
>>>> changelog
>>>> commit in the commit description, use html.escape() once instead of
>>>> twice, small change in the first commit to make it clearer.
>>>
>>> I applied this series. But I have an issue with the Discord daemon
>>> when I restart the MySQL server. It can't recover from MySQL restart.
>>>
>>> When you restart the MySQL server, it's always looping in the MySQL
>>> error and keep sending the error log on Discord:
>>>
>>>     "Lost connection to MySQL server during query"
>>>
>>> Can you please fix it?
>>
>> This happens on the Telegram daemon too. I thought I fixed it in:
>>
>>     529d804eb47518 ("daemon: telegram: Handle MySQL error")
>>
>> but somehow my fix is no longer applicable on top of your series.
>>
> 
> Can you give me the tips how to reproduce this error? Or I must wait or 
> idle the bot? Or I just disconnect my internet and then reconnect and 
> see what will happen on MySQL connection?

https://sebhastian.com/lost-connection-mysql-server-during-query/

Or you can just set the connection timeout instead?

Re: [PATCH v4 00/17] Fix, improvement and implement a bot logger.

[Ammar Faizi]

On 12/25/22 6:34 AM, Muhammad Rizki wrote:
> https://sebhastian.com/lost-connection-mysql-server-during-query/
> 
> Or you can just set the connection timeout instead?

This is not about timeout. But when the server is restarted, the
TCP socket is also closed. It needs to reconnect.

-- 
Ammar Faizi

Re: [PATCH v4 00/17] Fix, improvement and implement a bot logger.

[Ammar Faizi]

On 12/25/22 6:35 AM, Ammar Faizi wrote:
> On 12/25/22 6:34 AM, Muhammad Rizki wrote:
>> https://sebhastian.com/lost-connection-mysql-server-during-query/
>>
>> Or you can just set the connection timeout instead?
> 
> This is not about timeout. But when the server is restarted, the
> TCP socket is also closed. It needs to reconnect.

The Telegram daemon issue has already been fixed by commit:

    3eb6a95d602839bb8ac84eacae7258f1de30ea1a ("telegram: listener: Fix missing MySQL error recovery function")

but the Discord daemon still has the same issue. Try it.
I'll wait for your patch.

-- 
Ammar Faizi

Re: [PATCH v4 00/17] Fix, improvement and implement a bot logger.

[Muhammad Rizki]

On 25/12/2022 06.33, Ammar Faizi wrote:
> On 12/25/22 6:31 AM, Muhammad Rizki wrote:
>> Can you give me the tips how to reproduce this error? Or I must wait
>> or idle the bot? Or I just disconnect my internet and then reconnect
>> and see what will happen on MySQL connection?
> 
> Just restart your "MySQL server service" when the daemon is running.
> 
>     sudo service mysql restart
> 
> or whatever the MySQL restart procedure on your machine.
> 

I see, I will test it later, thanks.

Re: [PATCH v4 00/17] Fix, improvement and implement a bot logger.

[Muhammad Rizki]

On 25/12/2022 06.36, Ammar Faizi wrote:
> On 12/25/22 6:35 AM, Ammar Faizi wrote:
>> On 12/25/22 6:34 AM, Muhammad Rizki wrote:
>>> https://sebhastian.com/lost-connection-mysql-server-during-query/
>>>
>>> Or you can just set the connection timeout instead?
>>
>> This is not about timeout. But when the server is restarted, the
>> TCP socket is also closed. It needs to reconnect.
> 
> The Telegram daemon issue has already been fixed by commit:
> 
>     3eb6a95d602839bb8ac84eacae7258f1de30ea1a ("telegram: listener: Fix 
> missing MySQL error recovery function")
> 
> but the Discord daemon still has the same issue. Try it.
> I'll wait for your patch.
> 

Ohhh, I thought the error is not fixed. I will patch for the Discord daemon.

Re: [PATCH v4 00/17] Fix, improvement and implement a bot logger.

[Ammar Faizi]

I don't like the directory naming here. This is confusing:

On 12/21/22 8:33 AM, Muhammad Rizki wrote:
>   daemon/atom/utils.py                          | 67 ++++++++++++++-----
>   daemon/dc.py                                  | 10 ++-
>   daemon/dscord/config.py.example               |  4 ++
>   daemon/dscord/gnuweeb/client.py               | 20 ++++--
>   daemon/dscord/gnuweeb/filters.py              | 10 ++-

For example, this one is named "gnuweeb". What does that exactly mean?
What is the directory supposed to contain?

>   .../dscord/gnuweeb/plugins/events/on_ready.py |  2 +-
>   .../plugins/slash_commands/get_lore_mail.py   | 12 ++--
>   daemon/dscord/mailer/listener.py              | 41 +++++++-----
>   daemon/logger/__init__.py                     |  1 +
>   daemon/logger/log.py                          | 52 ++++++++++++++
>   daemon/telegram/config.py.example             |  4 ++
>   daemon/telegram/mailer/listener.py            | 44 +++++++-----

Also we have "mailer" here. We don't have any mailer here.
I'm going to change the directory naming shortly.

-- 
Ammar Faizi

Re: [PATCH v4 00/17] Fix, improvement and implement a bot logger.

[Alviro Iskandar Setiawan]

On Sun, Dec 25, 2022 at 6:16 PM Ammar Faizi <ammarfaizi2@gnuweeb.org> wrote:
> I'm going to change the directory naming shortly.

Why didn't you ask for revision early?

Re: [PATCH v4 00/17] Fix, improvement and implement a bot logger.

[Ammar Faizi]

On 12/25/22 6:29 PM, Alviro Iskandar Setiawan wrote:
> On Sun, Dec 25, 2022 at 6:16 PM Ammar Faizi <ammarfaizi2@gnuweeb.org> wrote:
>> I'm going to change the directory naming shortly.
> 
> Why didn't you ask for revision early?

Because I'm dumb.

It's already pushed out. Whatever is in the master branch is already
set in stone. Sometimes shit happens.

-- 
Ammar Faizi