[PATCH v3 00/17] Code improvements

[PATCH v3 00/17] Code improvements

[Muhammad Rizki]

Morning sir,

In this series I just want to improve some codes like adding type in
@handle_flood and replace some codes that already created with a good
one.

There are 17 patches in this series:

- Patch 1 is to fix raw lore URL at the end of URL by regex replace
- Patch 2 is to fix the lack of __send_patch_msg function parameter
- Patch 3 is just add required imports and remove unused imports
- Patch 4 is to add default temporary directory name
- Patch 5 is to move Telegram bot session file
- Patch 6 is to use traceback to get the error detail while debugging
- Patch 7 is to create send email and patch msg in the DaemonClient
- Patch 8 is just remove the ____send_patch_msg
- Patch 9 is to move utility functions to the utils file
- Patch 10 is to create fix_utf8_chars function
- Patch 11 is to remove whitespace after newline
- Patch 12 is to remove unused functions because it's already moved
- Patch 13 is to create @handle_flood decorator to handle floodwait
- Patch 14 is to remove __send_patch_msg() and displace some codes
- Patch 15 is just remove unused imports
- Patch 16 is to replace some functions in packages/plugins/scrape.py
- Patch 17 is to Add some code typing in @handle_flood decorator

Already tested it and works fine too, please give it a test too.
Thanks!

## Changelog

v2 ->  v3
- Remove duplicated patch.
- Modify patch 10 to look more practice.
- Indentation patch issues were removed and patched in patch 7
- Replace send email and patch email with the custom one.
- Add code typing in @handle_flood decorator

v1 -> v2
- Create @handle_flood to handle the Telegram floodwait.
- Remove unnecessary codes to make it cleaner.
- Move some utility functions into the utility file.
- Remove whitespace after newline
- Fix indentation issues

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---

Ammar Faizi (1):
  daemon: Fix raw lore URL on the inline keyboard button

Muhammad Rizki (16):
  Fix __send_patch_msg function parameter
  Fix import problem
  Add default temporary directory
  Move the Telegram bot session into the storage directory
  daemon: Use traceback.format_exc() to get the error detail
  Re-design send email message to Telegram
  Move ____send_patch_msg
  Move prepare for patch and clean up patch functions
  Create fix_utf8_chars function
  Remove whitespace
  Remove prepare patch and clean up patch
  daemon: Add @handle_flood decorator and remove some functions
  daemon: Remove __send_patch_msg()
  daemon: Remove unused imports
  Replace send email functions
  Add typing in decorator

 daemon/packages/__init__.py       |   1 +
 daemon/packages/client.py         |  73 +++++++++++++++
 daemon/packages/decorator.py      |  43 +++++++++
 daemon/packages/plugins/scrape.py |  49 ++--------
 daemon/run.py                     |   8 +-
 daemon/scraper/bot.py             | 146 ++++--------------------------
 daemon/scraper/scraper.py         |   4 -
 daemon/scraper/utils.py           |  48 ++++++++--
 8 files changed, 187 insertions(+), 185 deletions(-)
 create mode 100644 daemon/packages/__init__.py
 create mode 100644 daemon/packages/client.py
 create mode 100644 daemon/packages/decorator.py


base-commit: 339d992bdd77f19e67d0d9a30e3ac6eef1e4035a
-- 
Muhammad Rizki

[PATCH v3 01/17] Fix __send_patch_msg function parameter

[Muhammad Rizki]

From: Muhammad Rizki <riskimuhammmad1@gmail.com>

Lack of reply_to parameter in __send_patch_msg function.

Signed-off-by: Muhammad Rizki <riskimuhammmad1@gmail.com>
---
 daemon/scraper/bot.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/daemon/scraper/bot.py b/daemon/scraper/bot.py
index 7746eb8..9b95d50 100644
--- a/daemon/scraper/bot.py
+++ b/daemon/scraper/bot.py
@@ -157,7 +157,7 @@ class Bot():
 		return self.db.get_tg_reply_to(reply_to, tg_chat_id)
 
 
-	async def __send_patch_msg(self, mail, tg_chat_id, text, url):
+	async def __send_patch_msg(self, mail, tg_chat_id, reply_to, text, url):
 		print("[__send_patch_msg]")
 		
 		tmp, fnm, caption, url = Bot.prepare_send_patch(mail, text, url)
-- 
Muhammad Rizki

[PATCH v3 02/17] Fix import problem

[Muhammad Rizki]

From: Muhammad Rizki <riskimuhammmad1@gmail.com>

In this commit, there are combined topic and it's my mistakes, allow me
to explain. There are unused imports and it has been removed, and the
other there are undefined modules and it has been imported.

Signed-off-by: Muhammad Rizki <riskimuhammmad1@gmail.com>
---
 daemon/packages/plugins/scrape.py | 1 +
 daemon/run.py                     | 2 --
 daemon/scraper/bot.py             | 1 -
 daemon/scraper/scraper.py         | 4 ----
 4 files changed, 1 insertion(+), 7 deletions(-)

diff --git a/daemon/packages/plugins/scrape.py b/daemon/packages/plugins/scrape.py
index c9da71a..1698c6d 100644
--- a/daemon/packages/plugins/scrape.py
+++ b/daemon/packages/plugins/scrape.py
@@ -15,6 +15,7 @@ from scraper import utils
 from scraper import Bot
 import shutil
 import re
+import asyncio
 
 
 #
diff --git a/daemon/run.py b/daemon/run.py
index ef1d8f4..0a650cd 100644
--- a/daemon/run.py
+++ b/daemon/run.py
@@ -5,14 +5,12 @@
 #
 
 from apscheduler.schedulers.asyncio import AsyncIOScheduler
-from email.message import Message
 from scraper import BotMutexes
 from dotenv import load_dotenv
 from mysql import connector
 from pyrogram import Client
 from scraper import Scraper
 from scraper import Bot
-import asyncio
 import os
 
 
diff --git a/daemon/scraper/bot.py b/daemon/scraper/bot.py
index 9b95d50..c62c554 100644
--- a/daemon/scraper/bot.py
+++ b/daemon/scraper/bot.py
@@ -13,7 +13,6 @@ from scraper import Scraper
 from pyrogram import enums
 from . import utils
 from .db import Db
-import xmltodict
 import pyrogram
 import asyncio
 import shutil
diff --git a/daemon/scraper/scraper.py b/daemon/scraper/scraper.py
index b70df31..2d5942b 100644
--- a/daemon/scraper/scraper.py
+++ b/daemon/scraper/scraper.py
@@ -5,14 +5,10 @@
 #
 
 from typing import Dict, List
-from .db import Db
 import email.policy
 import xmltodict
-import operator
-import asyncio
 import httpx
 import email
-import json
 
 
 class Scraper():
-- 
Muhammad Rizki

[PATCH v3 03/17] Add default temporary directory

[Muhammad Rizki]

From: Muhammad Rizki <riskimuhammmad1@gmail.com>

We want to avoid error when "STORAGE_DIR" env is not defined, so you put
a default value here.

Signed-off-by: Muhammad Rizki <riskimuhammmad1@gmail.com>
---
 daemon/scraper/utils.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/daemon/scraper/utils.py b/daemon/scraper/utils.py
index 9334b82..10fe956 100644
--- a/daemon/scraper/utils.py
+++ b/daemon/scraper/utils.py
@@ -110,7 +110,7 @@ def consruct_to_n_cc(to: list, cc: list):
 
 def gen_temp(name: str):
 	md5 = hashlib.md5(name.encode()).hexdigest()
-	ret = os.getenv("STORAGE_DIR") + "/" + md5
+	ret = os.getenv("STORAGE_DIR", "storage") + "/" + md5
 	try:
 		os.mkdir(ret)
 	except FileExistsError:
-- 
Muhammad Rizki

[PATCH v3 04/17] Move the Telegram bot session into the storage directory

[Muhammad Rizki]

From: Muhammad Rizki <riskimuhammmad1@gmail.com>

We want to move the Telegram bot session file into the storage file, so
it makes the work root directory much cleaner.

Signed-off-by: Muhammad Rizki <riskimuhammmad1@gmail.com>
---
 daemon/run.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/daemon/run.py b/daemon/run.py
index 0a650cd..83b2cdb 100644
--- a/daemon/run.py
+++ b/daemon/run.py
@@ -18,7 +18,7 @@ def main():
 	load_dotenv()
 
 	client = Client(
-		"EmailScraper",
+		"storage/EmailScraper",
 		api_id=int(os.getenv("API_ID")),
 		api_hash=os.getenv("API_HASH"),
 		bot_token=os.getenv("BOT_TOKEN"),
-- 
Muhammad Rizki

[PATCH v3 05/17] daemon: Fix raw lore URL on the inline keyboard button

[Muhammad Rizki]

From: Ammar Faizi <ammarfaizi2@gnuweeb.org>

The current url contains "/raw" at the end of the link. This will give
us the raw email. As this button is supposed to be read by users, we
shouldn't give the raw version. Remove "/raw" at the end of URL before
sending it to Telegram to make it more user friendly.

Signed-off-by: Ammar Faizi <ammarfaizi2@gnuweeb.org>
Signed-off-by: Muhammad Rizki <riskimuhammmad1@gmail.com>
---
 daemon/scraper/bot.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/daemon/scraper/bot.py b/daemon/scraper/bot.py
index c62c554..04caab6 100644
--- a/daemon/scraper/bot.py
+++ b/daemon/scraper/bot.py
@@ -115,6 +115,7 @@ class Bot():
 
 		text, files, is_patch = utils.create_template(mail)
 		reply_to = self.get_tg_reply_to(mail, tg_chat_id)
+		url = str(re.sub(r"/raw$", "", url))
 
 		if is_patch:
 			m = await self.__send_patch_msg(mail, tg_chat_id,
-- 
Muhammad Rizki

[PATCH v3 06/17] daemon: Use traceback.format_exc() to get the error detail

[Muhammad Rizki]

From: Muhammad Rizki <riskimuhammmad1@gmail.com>

`print(f"[__run]: Error: {e}")` doesn't give enough information for
debugging because it doesn't show the traceback, files, line numbers,
etc. It only shows the error message raised by an exception. This is
severely lacking and very bad for debugging experience.

Use `traceback.format_exc()` instead to get a better log because it
shows a complete traceback of the error.

Signed-off-by: Muhammad Rizki <riskimuhammmad1@gmail.com>
---
 daemon/scraper/bot.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/daemon/scraper/bot.py b/daemon/scraper/bot.py
index 04caab6..2392b61 100644
--- a/daemon/scraper/bot.py
+++ b/daemon/scraper/bot.py
@@ -17,6 +17,7 @@ import pyrogram
 import asyncio
 import shutil
 import re
+import traceback
 
 
 class BotMutexes():
@@ -63,8 +64,8 @@ class Bot():
 		for url in self.ATOM_URLS:
 			try:
 				await self.__handle_atom_url(url)
-			except Exception as e:
-				print(f"[__run]: Error: {e}")
+			except:
+				print(traceback.format_exc())
 
 		if not self.isRunnerFixed:
 			self.isRunnerFixed = True
-- 
Muhammad Rizki

[PATCH v3 07/17] Re-design send email message to Telegram

[Muhammad Rizki]

From: Muhammad Rizki <riskimuhammmad1@gmail.com>

We want to separate send message function and inherit with the Pyrogram
Client, so it should makes the code clean and clear.

Signed-off-by: Muhammad Rizki <riskimuhammmad1@gmail.com>
---
 daemon/packages/__init__.py |  1 +
 daemon/packages/client.py   | 64 +++++++++++++++++++++++++++++++++++++
 daemon/run.py               |  4 +--
 daemon/scraper/bot.py       | 35 ++++++--------------
 4 files changed, 76 insertions(+), 28 deletions(-)
 create mode 100644 daemon/packages/__init__.py
 create mode 100644 daemon/packages/client.py

diff --git a/daemon/packages/__init__.py b/daemon/packages/__init__.py
new file mode 100644
index 0000000..efef9ae
--- /dev/null
+++ b/daemon/packages/__init__.py
@@ -0,0 +1 @@
+from .client import DaemonClient
diff --git a/daemon/packages/client.py b/daemon/packages/client.py
new file mode 100644
index 0000000..f73b913
--- /dev/null
+++ b/daemon/packages/client.py
@@ -0,0 +1,64 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Copyright (C) 2022  Muhammad Rizki <riskimuhammmad1@gmail.com>
+#
+
+from pyrogram import Client
+from pyrogram.enums import ParseMode
+from pyrogram.types import Message, InlineKeyboardMarkup, InlineKeyboardButton
+from typing import Union, BinaryIO
+
+
+class DaemonClient(Client):
+	def __init__(self, name: str, api_id: int,
+			api_hash: str, **kwargs):
+		super().__init__(name, api_id,
+				api_hash, **kwargs)
+
+
+	async def send_text_email(
+		self,
+		chat_id: Union[int, str],
+		text: str,
+		reply_to: int,
+		url: str = None,
+		parse_mode: ParseMode = ParseMode.HTML
+	) -> Message:
+		print("[send_text_email]")
+		return await self.send_message(
+			chat_id=chat_id,
+			text=text,
+			reply_to_message_id=reply_to,
+			parse_mode=parse_mode,
+			reply_markup=InlineKeyboardMarkup([
+				[InlineKeyboardButton(
+					"See the full message",
+					url=url
+				)]
+			])
+		)
+
+
+	async def send_patch_email(
+		self,
+		chat_id: Union[int, str],
+		doc: Union[str, BinaryIO],
+		caption: str,
+		reply_to: int,
+		url: str = None,
+		parse_mode: ParseMode = ParseMode.HTML
+	) -> Message:
+		print("[send_patch_email]")
+		return await self.send_document(
+			chat_id=chat_id,
+			document=doc,
+			caption=caption,
+			reply_to_message_id=reply_to,
+			parse_mode=parse_mode,
+			reply_markup=InlineKeyboardMarkup([
+				[InlineKeyboardButton(
+					"See the full message",
+					url=url
+				)]
+			])
+		)
diff --git a/daemon/run.py b/daemon/run.py
index 83b2cdb..1151ccd 100644
--- a/daemon/run.py
+++ b/daemon/run.py
@@ -8,7 +8,7 @@ from apscheduler.schedulers.asyncio import AsyncIOScheduler
 from scraper import BotMutexes
 from dotenv import load_dotenv
 from mysql import connector
-from pyrogram import Client
+from packages import DaemonClient
 from scraper import Scraper
 from scraper import Bot
 import os
@@ -17,7 +17,7 @@ import os
 def main():
 	load_dotenv()
 
-	client = Client(
+	client = DaemonClient(
 		"storage/EmailScraper",
 		api_id=int(os.getenv("API_ID")),
 		api_hash=os.getenv("API_HASH"),
diff --git a/daemon/scraper/bot.py b/daemon/scraper/bot.py
index 2392b61..93e633a 100644
--- a/daemon/scraper/bot.py
+++ b/daemon/scraper/bot.py
@@ -8,7 +8,7 @@ from apscheduler.schedulers.asyncio import AsyncIOScheduler
 from pyrogram.types import InlineKeyboardMarkup
 from pyrogram.types import InlineKeyboardButton
 from slugify import slugify
-from pyrogram import Client
+from packages import DaemonClient
 from scraper import Scraper
 from pyrogram import enums
 from . import utils
@@ -32,12 +32,11 @@ class Bot():
 	]
 
 	TG_CHAT_IDS = [
-		-1001394203410,
-		-1001673279485,
+		"kiizuah"
 	]
 
 
-	def __init__(self, client: Client, sched: AsyncIOScheduler,
+	def __init__(self, client: DaemonClient, sched: AsyncIOScheduler,
 		     scraper: Scraper, mutexes: BotMutexes, conn):
 		self.client = client
 		self.sched = sched
@@ -123,8 +122,8 @@ class Bot():
 							reply_to, text, url)
 		else:
 			text = "#ml\n" + text
-			m = await self.__send_text_msg(tg_chat_id, reply_to,
-						       text, url)
+			m = await self.__send_text_msg(tg_chat_id, text,
+						       reply_to, url)
 
 		self.db.insert_telegram(email_id, m.chat.id, m.id)
 		for d, f in files:
@@ -161,10 +160,10 @@ class Bot():
 	async def __send_patch_msg(self, mail, tg_chat_id, reply_to, text, url):
 		print("[__send_patch_msg]")
 		
-		tmp, fnm, caption, url = Bot.prepare_send_patch(mail, text, url)
+		tmp, doc, caption, url = Bot.prepare_send_patch(mail, text, url)
 		ret = await self.__handle_telegram_floodwait(
-			self.____send_patch_msg,
-			*[tg_chat_id, reply_to, fnm, caption, url]
+			self.client.send_patch_email,
+			*[tg_chat_id, doc, caption, reply_to, url]
 		)
 		Bot.clean_up_after_send_patch(tmp)
 		return ret
@@ -207,7 +206,7 @@ class Bot():
 
 	async def __send_text_msg(self, *args):
 		return await self.__handle_telegram_floodwait(
-			self.____send_text_msg,
+			self.client.send_text_email,
 			*args
 		)
 
@@ -251,19 +250,3 @@ class Bot():
 				)]
 			])
 		)
-
-
-	async def ____send_text_msg(self, tg_chat_id, reply_to, text, url):
-		print("[__send_text_msg]")
-		return await self.client.send_message(
-			tg_chat_id,
-			text,
-			reply_to_message_id=reply_to,
-			parse_mode=enums.ParseMode.HTML,
-			reply_markup=InlineKeyboardMarkup([
-				[InlineKeyboardButton(
-					"See the full message",
-					url=url
-				)]
-			])
-		)
-- 
Muhammad Rizki

[PATCH v3 08/17] Move ____send_patch_msg

[Muhammad Rizki]

From: Muhammad Rizki <riskimuhammmad1@gmail.com>

I want to move the ____send_patch_msg into the DaemonClient, so it much
cleaner if we separate the send email message into the DaemonClient.

Signed-off-by: Muhammad Rizki <riskimuhammmad1@gmail.com>
---
 daemon/scraper/bot.py | 17 -----------------
 1 file changed, 17 deletions(-)

diff --git a/daemon/scraper/bot.py b/daemon/scraper/bot.py
index 93e633a..4903fef 100644
--- a/daemon/scraper/bot.py
+++ b/daemon/scraper/bot.py
@@ -233,20 +233,3 @@ class Bot():
 		n = int(x.group(1))
 		print(f"[____handle_telegram_floodwait]: Sleeping for {n} seconds due to Telegram limit")
 		await asyncio.sleep(n)
-
-
-	async def ____send_patch_msg(self, tg_chat_id, reply_to, fnm, caption,
-				     url):
-		return await self.client.send_document(
-			tg_chat_id,
-			fnm,
-			caption=caption,
-			reply_to_message_id=reply_to,
-			parse_mode=enums.ParseMode.HTML,
-			reply_markup=InlineKeyboardMarkup([
-				[InlineKeyboardButton(
-					"See the full message",
-					url=url
-				)]
-			])
-		)
-- 
Muhammad Rizki

[PATCH v3 09/17] Move prepare for patch and clean up patch functions

[Muhammad Rizki]

From: Muhammad Rizki <riskimuhammmad1@gmail.com>

I want these function is separate into the utils or utility file, I'm
the type of person who likes to separate the utility function into the
utility file.

Signed-off-by: Muhammad Rizki <riskimuhammmad1@gmail.com>
---
 daemon/scraper/utils.py | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)

diff --git a/daemon/scraper/utils.py b/daemon/scraper/utils.py
index 10fe956..5335fe2 100644
--- a/daemon/scraper/utils.py
+++ b/daemon/scraper/utils.py
@@ -6,10 +6,12 @@
 
 from email.message import Message
 from typing import Dict
+from slugify import slugify
 import hashlib
 import uuid
 import os
 import re
+import shutil
 
 
 def get_email_msg_id(mail):
@@ -190,6 +192,39 @@ def create_template(thread: Message, to=None, cc=None):
 	return ret, files, is_patch
 
 
+def prepare_send_patch(mail, text, url):
+	tmp = gen_temp(url)
+	fnm = str(mail.get("subject"))
+	sch = re.search(PATCH_PATTERN, fnm, re.IGNORECASE)
+
+	nr_patch = sch.group(1)
+	if not nr_patch:
+		nr_patch = 1
+	else:
+		nr_patch = int(nr_patch)
+
+	num = "%04d" % nr_patch
+	fnm = slugify(sch.group(3)).replace("_", "-")
+	file = f"{tmp}/{num}-{fnm}.patch"
+	cap = text.split("\n\n")[0]
+
+	with open(file, "wb") as f:
+		f.write(bytes(text, encoding="utf8"))
+
+	caption = (
+		"#patch #ml\n" +
+		cap.rstrip()
+			.replace("<", "&lt;")
+			.replace(">","&gt;")
+			.replace("�"," ")
+	)
+	return tmp, file, caption, url
+
+
+def clean_up_after_send_patch(tmp):
+	shutil.rmtree(tmp)
+
+
 EMAIL_MSG_ID_PATTERN = r"<([^\<\>]+)>"
 def extract_email_msg_id(msg_id):
 	ret = re.search(EMAIL_MSG_ID_PATTERN, msg_id)
-- 
Muhammad Rizki

[PATCH v3 10/17] Create fix_utf8_chars function

[Muhammad Rizki]

From: Muhammad Rizki <riskimuhammmad1@gmail.com>

Some of the codes are repeated, so I create this function to make it
cleaner.

Signed-off-by: Muhammad Rizki <riskimuhammmad1@gmail.com>
---
 daemon/scraper/utils.py | 25 ++++++++++++-------------
 1 file changed, 12 insertions(+), 13 deletions(-)

diff --git a/daemon/scraper/utils.py b/daemon/scraper/utils.py
index 5335fe2..30efb7b 100644
--- a/daemon/scraper/utils.py
+++ b/daemon/scraper/utils.py
@@ -182,12 +182,8 @@ def create_template(thread: Message, to=None, cc=None):
 		if len(ret) >= 4000:
 			ret = ret[:4000] + "..."
 
-		ret = (
-			ret.rstrip()
-			.replace("<", "&lt;")
-			.replace(">","&gt;")
-			.replace("�"," ")
-		) + "\n<code>------------------------------------------------------------------------</code>"
+		ret = fix_utf8_char(ret)
+		ret += "\n<code>------------------------------------------------------------------------</code>"
 
 	return ret, files, is_patch
 
@@ -211,13 +207,7 @@ def prepare_send_patch(mail, text, url):
 	with open(file, "wb") as f:
 		f.write(bytes(text, encoding="utf8"))
 
-	caption = (
-		"#patch #ml\n" +
-		cap.rstrip()
-			.replace("<", "&lt;")
-			.replace(">","&gt;")
-			.replace("�"," ")
-	)
+	caption = "#patch #ml\n" + fix_utf8_char(cap)
 	return tmp, file, caption, url
 
 
@@ -225,6 +215,15 @@ def clean_up_after_send_patch(tmp):
 	shutil.rmtree(tmp)
 
 
+def fix_utf8_char(text: str):
+	return (
+		text.rstrip()
+		.replace("<", "&lt;")
+		.replace(">","&gt;")
+		.replace("�"," ")
+	)
+
+
 EMAIL_MSG_ID_PATTERN = r"<([^\<\>]+)>"
 def extract_email_msg_id(msg_id):
 	ret = re.search(EMAIL_MSG_ID_PATTERN, msg_id)
-- 
Muhammad Rizki

[PATCH v3 11/17] Remove whitespace

[Muhammad Rizki]

From: Muhammad Rizki <riskimuhammmad1@gmail.com>

I found whitespace after newline, it's removed now

Signed-off-by: Muhammad Rizki <riskimuhammmad1@gmail.com>
---
 daemon/scraper/bot.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/daemon/scraper/bot.py b/daemon/scraper/bot.py
index 4903fef..33a9eaa 100644
--- a/daemon/scraper/bot.py
+++ b/daemon/scraper/bot.py
@@ -159,7 +159,7 @@ class Bot():
 
 	async def __send_patch_msg(self, mail, tg_chat_id, reply_to, text, url):
 		print("[__send_patch_msg]")
-		
+
 		tmp, doc, caption, url = Bot.prepare_send_patch(mail, text, url)
 		ret = await self.__handle_telegram_floodwait(
 			self.client.send_patch_email,
-- 
Muhammad Rizki

[PATCH v3 12/17] Remove prepare patch and clean up patch

[Muhammad Rizki]

From: Muhammad Rizki <riskimuhammmad1@gmail.com>

These functions already moved to scaper/utils.py, so I decide to remove
it in scaper/bot.py and change some of the code that required to change.

Signed-off-by: Muhammad Rizki <riskimuhammmad1@gmail.com>
---
 daemon/scraper/bot.py | 39 ++-------------------------------------
 1 file changed, 2 insertions(+), 37 deletions(-)

diff --git a/daemon/scraper/bot.py b/daemon/scraper/bot.py
index 33a9eaa..1abfa98 100644
--- a/daemon/scraper/bot.py
+++ b/daemon/scraper/bot.py
@@ -160,50 +160,15 @@ class Bot():
 	async def __send_patch_msg(self, mail, tg_chat_id, reply_to, text, url):
 		print("[__send_patch_msg]")
 
-		tmp, doc, caption, url = Bot.prepare_send_patch(mail, text, url)
+		tmp, doc, caption, url = utils.prepare_send_patch(mail, text, url)
 		ret = await self.__handle_telegram_floodwait(
 			self.client.send_patch_email,
 			*[tg_chat_id, doc, caption, reply_to, url]
 		)
-		Bot.clean_up_after_send_patch(tmp)
+		utils.clean_up_after_send_patch(tmp)
 		return ret
 
 
-	@staticmethod
-	def prepare_send_patch(mail, text, url):
-		tmp = utils.gen_temp(url)
-		fnm = str(mail.get("subject"))
-		sch = re.search(utils.PATCH_PATTERN, fnm, re.IGNORECASE)
-
-		nr_patch = sch.group(1)
-		if not nr_patch:
-			nr_patch = 1
-		else:
-			nr_patch = int(nr_patch)
-
-		num = "%04d" % nr_patch
-		fnm = slugify(sch.group(3)).replace("_", "-")
-		file = f"{tmp}/{num}-{fnm}.patch"
-		cap = text.split("\n\n")[0]
-
-		with open(file, "wb") as f:
-			f.write(bytes(text, encoding="utf8"))
-
-		caption = (
-			"#patch #ml\n" +
-			cap.rstrip()
-				.replace("<", "&lt;")
-				.replace(">","&gt;")
-				.replace("�"," ")
-		)
-		return tmp, file, caption, url
-
-
-	@staticmethod
-	def clean_up_after_send_patch(tmp):
-		shutil.rmtree(tmp)
-
-
 	async def __send_text_msg(self, *args):
 		return await self.__handle_telegram_floodwait(
 			self.client.send_text_email,
-- 
Muhammad Rizki

[PATCH v3 13/17] daemon: Add @handle_flood decorator and remove some functions

[Muhammad Rizki]

From: Muhammad Rizki <riskimuhammmad1@gmail.com>

I decided to create @handle_flood decorator to make it easier to handle
the Telegram floodwait. Remove __send_text_msg() and replace the
call of the __send_text_msg() with send_text_email() in
packages/client.py and it have handle_flood decorator support.

Signed-off-by: Muhammad Rizki <riskimuhammmad1@gmail.com>
---
 daemon/packages/client.py    |  3 +++
 daemon/packages/decorator.py | 38 +++++++++++++++++++++++++++++
 daemon/scraper/bot.py        | 47 ++++++------------------------------
 3 files changed, 49 insertions(+), 39 deletions(-)
 create mode 100644 daemon/packages/decorator.py

diff --git a/daemon/packages/client.py b/daemon/packages/client.py
index f73b913..45acac3 100644
--- a/daemon/packages/client.py
+++ b/daemon/packages/client.py
@@ -7,6 +7,7 @@ from pyrogram import Client
 from pyrogram.enums import ParseMode
 from pyrogram.types import Message, InlineKeyboardMarkup, InlineKeyboardButton
 from typing import Union, BinaryIO
+from .decorator import handle_flood
 
 
 class DaemonClient(Client):
@@ -16,6 +17,7 @@ class DaemonClient(Client):
 				api_hash, **kwargs)
 
 
+	@handle_flood
 	async def send_text_email(
 		self,
 		chat_id: Union[int, str],
@@ -39,6 +41,7 @@ class DaemonClient(Client):
 		)
 
 
+	@handle_flood
 	async def send_patch_email(
 		self,
 		chat_id: Union[int, str],
diff --git a/daemon/packages/decorator.py b/daemon/packages/decorator.py
new file mode 100644
index 0000000..7d7dc39
--- /dev/null
+++ b/daemon/packages/decorator.py
@@ -0,0 +1,38 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Copyright (C) 2022  Muhammad Rizki <riskimuhammmad1@gmail.com>
+#
+
+from pyrogram.errors.exceptions.flood_420 import FloodWait
+from typing import Any, Callable
+from functools import wraps
+import re
+import asyncio
+
+__all__ = ["handle_flood"]
+
+
+def handle_flood(func: Callable[[Any], Any]) -> Callable[[Any], Any]:
+	@wraps(func)
+	async def callback(*args: Any) -> Any:
+		while True:
+			try:
+				return await func(*args)
+			except FloodWait as e:
+				#
+				# Aiee... we hit our limit.
+				# Let's slow down a bit.
+				#
+				_flood_exceptions(e)
+				print("[__handle_telegram_floodwait]: Woken up from flood wait...")
+	return callback
+
+
+async def _flood_exceptions(e):
+	x = re.search(r"A wait of (\d+) seconds is required", str(e))
+	if not x:
+		raise e
+
+	n = int(x.group(1))
+	print(f"[____handle_telegram_floodwait]: Sleeping for {n} seconds due to Telegram limit")
+	await asyncio.sleep(n)
diff --git a/daemon/scraper/bot.py b/daemon/scraper/bot.py
index 1abfa98..38c6e94 100644
--- a/daemon/scraper/bot.py
+++ b/daemon/scraper/bot.py
@@ -37,7 +37,7 @@ class Bot():
 
 
 	def __init__(self, client: DaemonClient, sched: AsyncIOScheduler,
-		     scraper: Scraper, mutexes: BotMutexes, conn):
+			scraper: Scraper, mutexes: BotMutexes, conn):
 		self.client = client
 		self.sched = sched
 		self.scraper = scraper
@@ -88,7 +88,7 @@ class Bot():
 		for tg_chat_id in self.TG_CHAT_IDS:
 			async with self.mutexes.send_to_tg:
 				should_wait = await self.__send_to_tg(url, mail,
-								      tg_chat_id)
+									tg_chat_id)
 
 			if should_wait:
 				await asyncio.sleep(1)
@@ -105,7 +105,7 @@ class Bot():
 			return False
 
 		email_id = self.__need_to_send_to_telegram(email_msg_id,
-							   tg_chat_id)
+							tg_chat_id)
 		if not email_id:
 			#
 			# Email has already been sent to Telegram.
@@ -122,8 +122,9 @@ class Bot():
 							reply_to, text, url)
 		else:
 			text = "#ml\n" + text
-			m = await self.__send_text_msg(tg_chat_id, text,
-						       reply_to, url)
+			m = await self.client.send_text_email(
+				tg_chat_id, text,reply_to, url
+			)
 
 		self.db.insert_telegram(email_id, m.chat.id, m.id)
 		for d, f in files:
@@ -161,40 +162,8 @@ class Bot():
 		print("[__send_patch_msg]")
 
 		tmp, doc, caption, url = utils.prepare_send_patch(mail, text, url)
-		ret = await self.__handle_telegram_floodwait(
-			self.client.send_patch_email,
-			*[tg_chat_id, doc, caption, reply_to, url]
+		ret = await self.client.send_patch_email(
+			tg_chat_id, doc, caption, reply_to, url
 		)
 		utils.clean_up_after_send_patch(tmp)
 		return ret
-
-
-	async def __send_text_msg(self, *args):
-		return await self.__handle_telegram_floodwait(
-			self.client.send_text_email,
-			*args
-		)
-
-
-	async def __handle_telegram_floodwait(self, callback, *args):
-		while True:
-			try:
-				return await callback(*args)
-			except pyrogram.errors.exceptions.flood_420.FloodWait as e:
-				#
-				# Aiee... we hit our limit.
-				# Let's slow down a bit.
-				#
-				await self.____handle_telegram_floodwait(e)
-				print("[__handle_telegram_floodwait]: Woken up from flood wait...")
-
-
-	async def ____handle_telegram_floodwait(self, e):
-		x = str(e)
-		x = re.search(r"A wait of (\d+) seconds is required", x)
-		if not x:
-			raise e
-
-		n = int(x.group(1))
-		print(f"[____handle_telegram_floodwait]: Sleeping for {n} seconds due to Telegram limit")
-		await asyncio.sleep(n)
-- 
Muhammad Rizki

[PATCH v3 14/17] daemon: Remove __send_patch_msg()

[Muhammad Rizki]

From: Muhammad Rizki <riskimuhammmad1@gmail.com>

I want to remove the __send_patch_msg() and displace the prepare patch,
send patch, and clean up patch into the send_patch_email() in
packages/client.py.

Signed-off-by: Muhammad Rizki <riskimuhammmad1@gmail.com>
---
 daemon/packages/client.py | 12 +++++++++---
 daemon/scraper/bot.py     | 16 +++-------------
 2 files changed, 12 insertions(+), 16 deletions(-)

diff --git a/daemon/packages/client.py b/daemon/packages/client.py
index 45acac3..5be7427 100644
--- a/daemon/packages/client.py
+++ b/daemon/packages/client.py
@@ -7,6 +7,8 @@ from pyrogram import Client
 from pyrogram.enums import ParseMode
 from pyrogram.types import Message, InlineKeyboardMarkup, InlineKeyboardButton
 from typing import Union, BinaryIO
+from email.message import Message
+from scraper import utils
 from .decorator import handle_flood
 
 
@@ -44,15 +46,16 @@ class DaemonClient(Client):
 	@handle_flood
 	async def send_patch_email(
 		self,
+		mail: Message,
 		chat_id: Union[int, str],
-		doc: Union[str, BinaryIO],
-		caption: str,
+		text: str,
 		reply_to: int,
 		url: str = None,
 		parse_mode: ParseMode = ParseMode.HTML
 	) -> Message:
 		print("[send_patch_email]")
-		return await self.send_document(
+		tmp, doc, caption, url = utils.prepare_send_patch(mail, text, url)
+		m = await self.send_document(
 			chat_id=chat_id,
 			document=doc,
 			caption=caption,
@@ -65,3 +68,6 @@ class DaemonClient(Client):
 				)]
 			])
 		)
+
+		utils.clean_up_after_send_patch(tmp)
+		return m
diff --git a/daemon/scraper/bot.py b/daemon/scraper/bot.py
index 38c6e94..c89ed58 100644
--- a/daemon/scraper/bot.py
+++ b/daemon/scraper/bot.py
@@ -118,8 +118,9 @@ class Bot():
 		url = str(re.sub(r"/raw$", "", url))
 
 		if is_patch:
-			m = await self.__send_patch_msg(mail, tg_chat_id,
-							reply_to, text, url)
+			m = await self.client.send_patch_email(
+				mail, tg_chat_id, text, reply_to, url
+			)
 		else:
 			text = "#ml\n" + text
 			m = await self.client.send_text_email(
@@ -156,14 +157,3 @@ class Bot():
 			return None
 
 		return self.db.get_tg_reply_to(reply_to, tg_chat_id)
-
-
-	async def __send_patch_msg(self, mail, tg_chat_id, reply_to, text, url):
-		print("[__send_patch_msg]")
-
-		tmp, doc, caption, url = utils.prepare_send_patch(mail, text, url)
-		ret = await self.client.send_patch_email(
-			tg_chat_id, doc, caption, reply_to, url
-		)
-		utils.clean_up_after_send_patch(tmp)
-		return ret
-- 
Muhammad Rizki

[PATCH v3 15/17] daemon: Remove unused imports

[Muhammad Rizki]

Just remove some unused imports in packages/client.py and scraper/bot.py

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/packages/client.py | 2 +-
 daemon/scraper/bot.py     | 5 -----
 2 files changed, 1 insertion(+), 6 deletions(-)

diff --git a/daemon/packages/client.py b/daemon/packages/client.py
index 5be7427..d16fe46 100644
--- a/daemon/packages/client.py
+++ b/daemon/packages/client.py
@@ -6,7 +6,7 @@
 from pyrogram import Client
 from pyrogram.enums import ParseMode
 from pyrogram.types import Message, InlineKeyboardMarkup, InlineKeyboardButton
-from typing import Union, BinaryIO
+from typing import Union
 from email.message import Message
 from scraper import utils
 from .decorator import handle_flood
diff --git a/daemon/scraper/bot.py b/daemon/scraper/bot.py
index c89ed58..bc9d575 100644
--- a/daemon/scraper/bot.py
+++ b/daemon/scraper/bot.py
@@ -5,15 +5,10 @@
 #
 
 from apscheduler.schedulers.asyncio import AsyncIOScheduler
-from pyrogram.types import InlineKeyboardMarkup
-from pyrogram.types import InlineKeyboardButton
-from slugify import slugify
 from packages import DaemonClient
 from scraper import Scraper
-from pyrogram import enums
 from . import utils
 from .db import Db
-import pyrogram
 import asyncio
 import shutil
 import re
-- 
Muhammad Rizki

[PATCH v3 16/17] Replace send email functions

[Muhammad Rizki]

I want to replace called functions in packages/plugins/scrape.py because
I created custom functions for send email to Telegram using inheritance
method, so I replace it and remove related unused codes.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/packages/plugins/scrape.py | 48 ++++++-------------------------
 1 file changed, 9 insertions(+), 39 deletions(-)

diff --git a/daemon/packages/plugins/scrape.py b/daemon/packages/plugins/scrape.py
index 1698c6d..280c3f9 100644
--- a/daemon/packages/plugins/scrape.py
+++ b/daemon/packages/plugins/scrape.py
@@ -4,15 +4,11 @@
 # Copyright (C) 2022  Ammar Faizi <ammarfaizi2@gnuweeb.org>
 #
 
-from pyrogram.types import InlineKeyboardMarkup
-from pyrogram.types import InlineKeyboardButton
 from pyrogram.types import Message
 from pyrogram import filters
-from pyrogram import Client
+from packages import DaemonClient
 from scraper import Scraper
-from pyrogram import enums
 from scraper import utils
-from scraper import Bot
 import shutil
 import re
 import asyncio
@@ -25,11 +21,11 @@ import asyncio
 #    .lore https://lore.kernel.org/path/message_id/raw
 #
 LORE_CMD_URL_PATTERN = r"^(?:\/|\.|\!)lore\s+(https?:\/\/lore\.kernel\.org\/\S+)"
-@Client.on_message(
+@DaemonClient.on_message(
 	filters.regex(LORE_CMD_URL_PATTERN) &
 	filters.chat(["kiizuah", "nekoha", -1001673279485])
 )
-async def scrap_email(_, m: Message):
+async def scrap_email(c: DaemonClient, m: Message):
 	p = re.search(LORE_CMD_URL_PATTERN, m.text)
 	if not p:
 		return
@@ -43,10 +39,14 @@ async def scrap_email(_, m: Message):
 	text, files, is_patch = utils.create_template(mail)
 
 	if is_patch:
-		m = await __send_patch_msg(m, mail, text, url)
+		m = await c.send_patch_email(
+			mail, m.chat.id, text, m.id, url
+		)
 	else:
 		text = "#ml\n" + text
-		m = await __send_text_msg(m, text, url)
+		m = await c.send_text_email(
+			m.chat.id, text, m.id, url
+		)
 
 	for d, f in files:
 		await m.reply_document(f"{d}/{f}", file_name=f)
@@ -54,33 +54,3 @@ async def scrap_email(_, m: Message):
 
 	if files:
 		shutil.rmtree(str(files[0][0]))
-
-
-async def __send_patch_msg(m, mail, text, url):
-	tmp, fnm, caption, url = Bot.prepare_send_patch(mail, text, url)
-	ret = await m.reply_document(
-		fnm,
-		caption=caption,
-		parse_mode=enums.ParseMode.HTML,
-		reply_markup=InlineKeyboardMarkup([
-			[InlineKeyboardButton(
-				"See the full message",
-				url=url
-			)]
-		])
-	)
-	Bot.clean_up_after_send_patch(tmp)
-	return ret
-
-
-async def __send_text_msg(m, text, url):
-	return await m.reply(
-		text,
-		parse_mode=enums.ParseMode.HTML,
-		reply_markup=InlineKeyboardMarkup([
-			[InlineKeyboardButton(
-				"See the full message",
-				url=url.replace("/raw","")
-			)]
-		])
-	)
-- 
Muhammad Rizki

[PATCH v3 17/17] Add typing in decorator

[Muhammad Rizki]

I want to add code typing on @handle_flood, so when the function were
called, it display the hints.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/packages/decorator.py | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/daemon/packages/decorator.py b/daemon/packages/decorator.py
index 7d7dc39..c7a5f02 100644
--- a/daemon/packages/decorator.py
+++ b/daemon/packages/decorator.py
@@ -4,15 +4,20 @@
 #
 
 from pyrogram.errors.exceptions.flood_420 import FloodWait
-from typing import Any, Callable
+from pyrogram.types import Message
+from typing import Any, Callable, TypeVar
 from functools import wraps
 import re
 import asyncio
 
 __all__ = ["handle_flood"]
 
+T = TypeVar("T", bound=Message)
 
-def handle_flood(func: Callable[[Any], Any]) -> Callable[[Any], Any]:
+#
+# TODO(Muhammad Rizki): Add more typing for @handle_flood
+#
+def handle_flood(func: Callable[[T], T]) -> Callable[[T], T]:
 	@wraps(func)
 	async def callback(*args: Any) -> Any:
 		while True:
-- 
Muhammad Rizki

Re: [PATCH v3 00/17] Code improvements

[Ammar Faizi]

On Fri, 22 Jul 2022 06:29:21 +0700, Muhammad Rizki wrote:
> Morning sir,
> 
> In this series I just want to improve some codes like adding type in
> @handle_flood and replace some codes that already created with a good
> one.
> 
> There are 17 patches in this series:
> 
> [...]

Applied, thanks!

[01/17] daemon: Fix __send_patch_msg function parameter
        commit: 932f7bc8796758c0c219c59e571a32a3f7f08fd3
[02/17] daemon: Fix import problem
        commit: b15b476b912a20f1178adc859646e5c51febc25b
[03/17] daemon: Add default temporary directory
        commit: 80b999ed1ffd9c0974919e92c7a332fdf92e5450
[04/17] daemon: Move the Telegram bot session into the storage directory
        commit: bf230f9713587775a85fa4f96a197de48ff958b0
[05/17] daemon: Fix raw lore URL on the inline keyboard button
        commit: ac6ecf882c445d9a1a94f0c98b2f321283f68ccb
[06/17] daemon: Use traceback.format_exc() to get the error detail
        commit: af63fb0b198a96d538b60d8b30c1c84108a63a3e
[07/17] daemon: Re-design send email message to Telegram
        commit: 310c084fba8f4302a557fecea8c15f8c95f9058a
[08/17] daemon: Move ____send_patch_msg
        commit: 7bbe44635d86e70f0692e16dc72d0282ee66667c
[09/17] daemon: Move prepare for patch and clean up patch functions
        commit: 59d20af68a05a177fda6e3492b452f26052a5534
[10/17] daemon: Create fix_utf8_chars function
        commit: 6bae9f5e55e9f05529c78756bc194e71f8e6f7a5
[11/17] daemon: Remove whitespace
        commit: 08b423a82744a84b704a87c36ed524bfea9d64dc
[12/17] daemon: Remove prepare patch and clean up patch
        commit: c3acb8465f1dc8cc8e71bdc0c9f8a7fe5a12c561
[13/17] daemon: Add @handle_flood decorator and remove some functions
        commit: 9827fd22d17a80f7957ca82ecfc7509090a79987
[14/17] daemon: Remove __send_patch_msg()
        commit: 29262d303926f6ee138698f8eed009d7dc961e7d
[15/17] daemon: Remove unused imports
        commit: a98ca7e776a3a03eb929205c6b72783fdf99406f
[16/17] daemon: Replace send email functions
        commit: 91f2f491e4a3a0071973c28dfa304164739c462a
[17/17] daemon: Add typing in decorator
        commit: ba02aa3376a49afe142fe2c2b77b45ad358d1fe0

Best regards,
-- 
Ammar Faizi