[PATCH v2 00/18] Plan to redesign codes

[PATCH v2 00/18] Plan to redesign codes

[Muhammad Rizki]

Morning sir,

In this series I want to redesign codes to make it clear and clean.
I've inherit the DaemonClient with the Pyrogram's Client, so with
this method we can create our custom functions, such as
send email message and patch message in the DaemonClient. I've fixed
some bugs to and other improvements such as create a decorator.

There are 18 patches in this series:

- Patch 1 is to fix raw lore URL at the end of URL by regex replace
- Patch 2 is to fix the lack of __send_patch_msg function parameter
- Patch 3 is just add required imports and remove unused imports
- Patch 4 is to add default temporary directory name
- Patch 5 is to move Telegram bot session file
- Patch 6 is to use traceback to get the error detail while debugging
- Patch 7 is to create send email and patch msg in the DaemonClient
- Patch 8 is just remove the ____send_patch_msg
- Patch 9 is to move utility functions to the utils file
- Patch 10 is to create fix_utf8_chars function
- Patch 11 is just cleaning some codes to make it looks more practice
- Patch 12 is to change some indentation space to indentation tab
- Patch 13 is to remove whitespace after newline
- Patch 14 is to remove unused functions because it's already moved
- Patch 15 is just remove unused imports
- Patch 16 is to create @handle_flood decorator to handle floodwait
- Patch 17 is to remove __send_patch_msg() and displace some codes
- Patch 18 is just remove unused imports

I've already tested it last night and it fully works without errors,
so please give it a test too, thanks.

## Changelog

v1 -> v2
- Create @handle_flood to handle the Telegram floodwait.
- Remove unnecessary codes to make it cleaner.
- Move some utility functions into the utility file.
- Remove whitespace after newline
- Fix indentation issues

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---

Ammar Faizi (1):
  daemon: Fix raw lore URL on the inline keyboard button

Muhammad Rizki (17):
  Fix __send_patch_msg function parameter
  Fix import problem
  Add default temporary directory
  Move the Telegram bot session into the storage directory
  daemon: Use traceback.format_exc() to get the error detail
  Re-design send email message to Telegram
  Move ____send_patch_msg
  Move prepare for patch and clean up patch functions
  Create fix_utf8_chars function
  Recode some codes
  Fix indentation issue
  Remove whitespace
  Remove prepare patch and clean up patch
  daemon: Remove unused imports
  daemon: Add @handle_flood decorator and remove some functions
  daemon: Remove __send_patch_msg()
  daemon: Remove unused imports

 daemon/packages/__init__.py       |   1 +
 daemon/packages/client.py         |  68 +++++++++++++++
 daemon/packages/decorator.py      |  38 ++++++++
 daemon/packages/plugins/scrape.py |   1 +
 daemon/run.py                     |   8 +-
 daemon/scraper/bot.py             | 140 +++---------------------------
 daemon/scraper/scraper.py         |   4 -
 daemon/scraper/utils.py           |  50 +++++++++--
 8 files changed, 167 insertions(+), 143 deletions(-)
 create mode 100644 daemon/packages/__init__.py
 create mode 100644 daemon/packages/client.py
 create mode 100644 daemon/packages/decorator.py


base-commit: 339d992bdd77f19e67d0d9a30e3ac6eef1e4035a
-- 
Muhammad Rizki

[PATCH v2 01/18] Fix __send_patch_msg function parameter

[Muhammad Rizki]

From: Muhammad Rizki <riskimuhammmad1@gmail.com>

Lack of reply_to parameter in __send_patch_msg function.

Signed-off-by: Muhammad Rizki <riskimuhammmad1@gmail.com>
---
 daemon/scraper/bot.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/daemon/scraper/bot.py b/daemon/scraper/bot.py
index 7746eb8..9b95d50 100644
--- a/daemon/scraper/bot.py
+++ b/daemon/scraper/bot.py
@@ -157,7 +157,7 @@ class Bot():
 		return self.db.get_tg_reply_to(reply_to, tg_chat_id)
 
 
-	async def __send_patch_msg(self, mail, tg_chat_id, text, url):
+	async def __send_patch_msg(self, mail, tg_chat_id, reply_to, text, url):
 		print("[__send_patch_msg]")
 		
 		tmp, fnm, caption, url = Bot.prepare_send_patch(mail, text, url)
-- 
Muhammad Rizki

[PATCH v2 02/18] Fix import problem

[Muhammad Rizki]

From: Muhammad Rizki <riskimuhammmad1@gmail.com>

In this commit, there are combined topic and it's my mistakes, allow me
to explain. There are unused imports and it has been removed, and the
other there are undefined modules and it has been imported.

Signed-off-by: Muhammad Rizki <riskimuhammmad1@gmail.com>
---
 daemon/packages/plugins/scrape.py | 1 +
 daemon/run.py                     | 2 --
 daemon/scraper/bot.py             | 1 -
 daemon/scraper/scraper.py         | 4 ----
 4 files changed, 1 insertion(+), 7 deletions(-)

diff --git a/daemon/packages/plugins/scrape.py b/daemon/packages/plugins/scrape.py
index c9da71a..1698c6d 100644
--- a/daemon/packages/plugins/scrape.py
+++ b/daemon/packages/plugins/scrape.py
@@ -15,6 +15,7 @@ from scraper import utils
 from scraper import Bot
 import shutil
 import re
+import asyncio
 
 
 #
diff --git a/daemon/run.py b/daemon/run.py
index ef1d8f4..0a650cd 100644
--- a/daemon/run.py
+++ b/daemon/run.py
@@ -5,14 +5,12 @@
 #
 
 from apscheduler.schedulers.asyncio import AsyncIOScheduler
-from email.message import Message
 from scraper import BotMutexes
 from dotenv import load_dotenv
 from mysql import connector
 from pyrogram import Client
 from scraper import Scraper
 from scraper import Bot
-import asyncio
 import os
 
 
diff --git a/daemon/scraper/bot.py b/daemon/scraper/bot.py
index 9b95d50..c62c554 100644
--- a/daemon/scraper/bot.py
+++ b/daemon/scraper/bot.py
@@ -13,7 +13,6 @@ from scraper import Scraper
 from pyrogram import enums
 from . import utils
 from .db import Db
-import xmltodict
 import pyrogram
 import asyncio
 import shutil
diff --git a/daemon/scraper/scraper.py b/daemon/scraper/scraper.py
index b70df31..2d5942b 100644
--- a/daemon/scraper/scraper.py
+++ b/daemon/scraper/scraper.py
@@ -5,14 +5,10 @@
 #
 
 from typing import Dict, List
-from .db import Db
 import email.policy
 import xmltodict
-import operator
-import asyncio
 import httpx
 import email
-import json
 
 
 class Scraper():
-- 
Muhammad Rizki

[PATCH v2 03/18] Add default temporary directory

[Muhammad Rizki]

From: Muhammad Rizki <riskimuhammmad1@gmail.com>

We want to avoid error when "STORAGE_DIR" env is not defined, so you put
a default value here.

Signed-off-by: Muhammad Rizki <riskimuhammmad1@gmail.com>
---
 daemon/scraper/utils.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/daemon/scraper/utils.py b/daemon/scraper/utils.py
index 9334b82..10fe956 100644
--- a/daemon/scraper/utils.py
+++ b/daemon/scraper/utils.py
@@ -110,7 +110,7 @@ def consruct_to_n_cc(to: list, cc: list):
 
 def gen_temp(name: str):
 	md5 = hashlib.md5(name.encode()).hexdigest()
-	ret = os.getenv("STORAGE_DIR") + "/" + md5
+	ret = os.getenv("STORAGE_DIR", "storage") + "/" + md5
 	try:
 		os.mkdir(ret)
 	except FileExistsError:
-- 
Muhammad Rizki

[PATCH v2 04/18] Move the Telegram bot session into the storage directory

[Muhammad Rizki]

From: Muhammad Rizki <riskimuhammmad1@gmail.com>

We want to move the Telegram bot session file into the storage file, so
it makes the work root directory much cleaner.

Signed-off-by: Muhammad Rizki <riskimuhammmad1@gmail.com>
---
 daemon/run.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/daemon/run.py b/daemon/run.py
index 0a650cd..83b2cdb 100644
--- a/daemon/run.py
+++ b/daemon/run.py
@@ -18,7 +18,7 @@ def main():
 	load_dotenv()
 
 	client = Client(
-		"EmailScraper",
+		"storage/EmailScraper",
 		api_id=int(os.getenv("API_ID")),
 		api_hash=os.getenv("API_HASH"),
 		bot_token=os.getenv("BOT_TOKEN"),
-- 
Muhammad Rizki

[PATCH v2 05/18] daemon: Fix raw lore URL on the inline keyboard button

[Muhammad Rizki]

From: Ammar Faizi <ammarfaizi2@gnuweeb.org>

The current url contains "/raw" at the end of the link. This will give
us the raw email. As this button is supposed to be read by users, we
shouldn't give the raw version. Remove "/raw" at the end of URL before
sending it to Telegram to make it more user friendly.

Signed-off-by: Ammar Faizi <ammarfaizi2@gnuweeb.org>
Signed-off-by: Muhammad Rizki <riskimuhammmad1@gmail.com>
---
 daemon/scraper/bot.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/daemon/scraper/bot.py b/daemon/scraper/bot.py
index c62c554..04caab6 100644
--- a/daemon/scraper/bot.py
+++ b/daemon/scraper/bot.py
@@ -115,6 +115,7 @@ class Bot():
 
 		text, files, is_patch = utils.create_template(mail)
 		reply_to = self.get_tg_reply_to(mail, tg_chat_id)
+		url = str(re.sub(r"/raw$", "", url))
 
 		if is_patch:
 			m = await self.__send_patch_msg(mail, tg_chat_id,
-- 
Muhammad Rizki

[PATCH v2 06/18] daemon: Use traceback.format_exc() to get the error detail

[Muhammad Rizki]

From: Muhammad Rizki <riskimuhammmad1@gmail.com>

`print(f"[__run]: Error: {e}")` doesn't give enough information for
debugging because it doesn't show the traceback, files, line numbers,
etc. It only shows the error message raised by an exception. This is
severely lacking and very bad for debugging experience.

Use `traceback.format_exc()` instead to get a better log because it
shows a complete traceback of the error.

Signed-off-by: Muhammad Rizki <riskimuhammmad1@gmail.com>
---
 daemon/scraper/bot.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/daemon/scraper/bot.py b/daemon/scraper/bot.py
index 04caab6..2392b61 100644
--- a/daemon/scraper/bot.py
+++ b/daemon/scraper/bot.py
@@ -17,6 +17,7 @@ import pyrogram
 import asyncio
 import shutil
 import re
+import traceback
 
 
 class BotMutexes():
@@ -63,8 +64,8 @@ class Bot():
 		for url in self.ATOM_URLS:
 			try:
 				await self.__handle_atom_url(url)
-			except Exception as e:
-				print(f"[__run]: Error: {e}")
+			except:
+				print(traceback.format_exc())
 
 		if not self.isRunnerFixed:
 			self.isRunnerFixed = True
-- 
Muhammad Rizki

[PATCH v2 07/18] Re-design send email message to Telegram

[Muhammad Rizki]

From: Muhammad Rizki <riskimuhammmad1@gmail.com>

We want to separate send message function and inherit with the Pyrogram
Client, so it should makes the code clean and clear.

Signed-off-by: Muhammad Rizki <riskimuhammmad1@gmail.com>
---
 daemon/packages/__init__.py |  1 +
 daemon/packages/client.py   | 59 +++++++++++++++++++++++++++++++++++++
 daemon/run.py               |  4 +--
 daemon/scraper/bot.py       | 35 ++++++----------------
 4 files changed, 71 insertions(+), 28 deletions(-)
 create mode 100644 daemon/packages/__init__.py
 create mode 100644 daemon/packages/client.py

diff --git a/daemon/packages/__init__.py b/daemon/packages/__init__.py
new file mode 100644
index 0000000..efef9ae
--- /dev/null
+++ b/daemon/packages/__init__.py
@@ -0,0 +1 @@
+from .client import DaemonClient
diff --git a/daemon/packages/client.py b/daemon/packages/client.py
new file mode 100644
index 0000000..fd5b5ec
--- /dev/null
+++ b/daemon/packages/client.py
@@ -0,0 +1,59 @@
+from pyrogram import Client
+from pyrogram.enums import ParseMode
+from pyrogram.types import Message, InlineKeyboardMarkup, InlineKeyboardButton
+from typing import Union, BinaryIO
+
+
+class DaemonClient(Client):
+        def __init__(self, name: str, api_id: int,
+                     api_hash: str, **kwargs):
+                super().__init__(name, api_id,
+                                api_hash, **kwargs)
+
+
+        async def send_text_email(
+                self,
+                chat_id: Union[int, str],
+                text: str,
+                reply_to: int,
+                url: str = None,
+                parse_mode: ParseMode = ParseMode.HTML
+        ) -> Message:
+                print("[send_text_email]")
+                return await self.send_message(
+			chat_id=chat_id,
+			text=text,
+			reply_to_message_id=reply_to,
+			parse_mode=parse_mode,
+			reply_markup=InlineKeyboardMarkup([
+				[InlineKeyboardButton(
+					"See the full message",
+					url=url
+				)]
+			])
+		)
+
+
+        async def send_patch_email(
+                self,
+                chat_id: Union[int, str],
+                doc: Union[str, BinaryIO],
+                caption: str,
+                reply_to: int,
+                url: str = None,
+                parse_mode: ParseMode = ParseMode.HTML
+        ) -> Message:
+                print("[send_patch_email]")
+                return await self.send_document(
+			chat_id=chat_id,
+                        document=doc,
+			caption=caption,
+			reply_to_message_id=reply_to,
+			parse_mode=parse_mode,
+			reply_markup=InlineKeyboardMarkup([
+				[InlineKeyboardButton(
+					"See the full message",
+					url=url
+				)]
+			])
+		)
diff --git a/daemon/run.py b/daemon/run.py
index 83b2cdb..1151ccd 100644
--- a/daemon/run.py
+++ b/daemon/run.py
@@ -8,7 +8,7 @@ from apscheduler.schedulers.asyncio import AsyncIOScheduler
 from scraper import BotMutexes
 from dotenv import load_dotenv
 from mysql import connector
-from pyrogram import Client
+from packages import DaemonClient
 from scraper import Scraper
 from scraper import Bot
 import os
@@ -17,7 +17,7 @@ import os
 def main():
 	load_dotenv()
 
-	client = Client(
+	client = DaemonClient(
 		"storage/EmailScraper",
 		api_id=int(os.getenv("API_ID")),
 		api_hash=os.getenv("API_HASH"),
diff --git a/daemon/scraper/bot.py b/daemon/scraper/bot.py
index 2392b61..93e633a 100644
--- a/daemon/scraper/bot.py
+++ b/daemon/scraper/bot.py
@@ -8,7 +8,7 @@ from apscheduler.schedulers.asyncio import AsyncIOScheduler
 from pyrogram.types import InlineKeyboardMarkup
 from pyrogram.types import InlineKeyboardButton
 from slugify import slugify
-from pyrogram import Client
+from packages import DaemonClient
 from scraper import Scraper
 from pyrogram import enums
 from . import utils
@@ -32,12 +32,11 @@ class Bot():
 	]
 
 	TG_CHAT_IDS = [
-		-1001394203410,
-		-1001673279485,
+		"kiizuah"
 	]
 
 
-	def __init__(self, client: Client, sched: AsyncIOScheduler,
+	def __init__(self, client: DaemonClient, sched: AsyncIOScheduler,
 		     scraper: Scraper, mutexes: BotMutexes, conn):
 		self.client = client
 		self.sched = sched
@@ -123,8 +122,8 @@ class Bot():
 							reply_to, text, url)
 		else:
 			text = "#ml\n" + text
-			m = await self.__send_text_msg(tg_chat_id, reply_to,
-						       text, url)
+			m = await self.__send_text_msg(tg_chat_id, text,
+						       reply_to, url)
 
 		self.db.insert_telegram(email_id, m.chat.id, m.id)
 		for d, f in files:
@@ -161,10 +160,10 @@ class Bot():
 	async def __send_patch_msg(self, mail, tg_chat_id, reply_to, text, url):
 		print("[__send_patch_msg]")
 		
-		tmp, fnm, caption, url = Bot.prepare_send_patch(mail, text, url)
+		tmp, doc, caption, url = Bot.prepare_send_patch(mail, text, url)
 		ret = await self.__handle_telegram_floodwait(
-			self.____send_patch_msg,
-			*[tg_chat_id, reply_to, fnm, caption, url]
+			self.client.send_patch_email,
+			*[tg_chat_id, doc, caption, reply_to, url]
 		)
 		Bot.clean_up_after_send_patch(tmp)
 		return ret
@@ -207,7 +206,7 @@ class Bot():
 
 	async def __send_text_msg(self, *args):
 		return await self.__handle_telegram_floodwait(
-			self.____send_text_msg,
+			self.client.send_text_email,
 			*args
 		)
 
@@ -251,19 +250,3 @@ class Bot():
 				)]
 			])
 		)
-
-
-	async def ____send_text_msg(self, tg_chat_id, reply_to, text, url):
-		print("[__send_text_msg]")
-		return await self.client.send_message(
-			tg_chat_id,
-			text,
-			reply_to_message_id=reply_to,
-			parse_mode=enums.ParseMode.HTML,
-			reply_markup=InlineKeyboardMarkup([
-				[InlineKeyboardButton(
-					"See the full message",
-					url=url
-				)]
-			])
-		)
-- 
Muhammad Rizki

[PATCH v2 08/18] Move ____send_patch_msg

[Muhammad Rizki]

From: Muhammad Rizki <riskimuhammmad1@gmail.com>

I want to move the ____send_patch_msg into the DaemonClient, so it much
cleaner if we separate the send email message into the DaemonClient.

Signed-off-by: Muhammad Rizki <riskimuhammmad1@gmail.com>
---
 daemon/scraper/bot.py | 17 -----------------
 1 file changed, 17 deletions(-)

diff --git a/daemon/scraper/bot.py b/daemon/scraper/bot.py
index 93e633a..4903fef 100644
--- a/daemon/scraper/bot.py
+++ b/daemon/scraper/bot.py
@@ -233,20 +233,3 @@ class Bot():
 		n = int(x.group(1))
 		print(f"[____handle_telegram_floodwait]: Sleeping for {n} seconds due to Telegram limit")
 		await asyncio.sleep(n)
-
-
-	async def ____send_patch_msg(self, tg_chat_id, reply_to, fnm, caption,
-				     url):
-		return await self.client.send_document(
-			tg_chat_id,
-			fnm,
-			caption=caption,
-			reply_to_message_id=reply_to,
-			parse_mode=enums.ParseMode.HTML,
-			reply_markup=InlineKeyboardMarkup([
-				[InlineKeyboardButton(
-					"See the full message",
-					url=url
-				)]
-			])
-		)
-- 
Muhammad Rizki

[PATCH v2 09/18] Move prepare for patch and clean up patch functions

[Muhammad Rizki]

From: Muhammad Rizki <riskimuhammmad1@gmail.com>

I want these function is separate into the utils or utility file, I'm
the type of person who likes to separate the utility function into the
utility file.

Signed-off-by: Muhammad Rizki <riskimuhammmad1@gmail.com>
---
 daemon/scraper/utils.py | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)

diff --git a/daemon/scraper/utils.py b/daemon/scraper/utils.py
index 10fe956..5335fe2 100644
--- a/daemon/scraper/utils.py
+++ b/daemon/scraper/utils.py
@@ -6,10 +6,12 @@
 
 from email.message import Message
 from typing import Dict
+from slugify import slugify
 import hashlib
 import uuid
 import os
 import re
+import shutil
 
 
 def get_email_msg_id(mail):
@@ -190,6 +192,39 @@ def create_template(thread: Message, to=None, cc=None):
 	return ret, files, is_patch
 
 
+def prepare_send_patch(mail, text, url):
+	tmp = gen_temp(url)
+	fnm = str(mail.get("subject"))
+	sch = re.search(PATCH_PATTERN, fnm, re.IGNORECASE)
+
+	nr_patch = sch.group(1)
+	if not nr_patch:
+		nr_patch = 1
+	else:
+		nr_patch = int(nr_patch)
+
+	num = "%04d" % nr_patch
+	fnm = slugify(sch.group(3)).replace("_", "-")
+	file = f"{tmp}/{num}-{fnm}.patch"
+	cap = text.split("\n\n")[0]
+
+	with open(file, "wb") as f:
+		f.write(bytes(text, encoding="utf8"))
+
+	caption = (
+		"#patch #ml\n" +
+		cap.rstrip()
+			.replace("<", "&lt;")
+			.replace(">","&gt;")
+			.replace("�"," ")
+	)
+	return tmp, file, caption, url
+
+
+def clean_up_after_send_patch(tmp):
+	shutil.rmtree(tmp)
+
+
 EMAIL_MSG_ID_PATTERN = r"<([^\<\>]+)>"
 def extract_email_msg_id(msg_id):
 	ret = re.search(EMAIL_MSG_ID_PATTERN, msg_id)
-- 
Muhammad Rizki

[PATCH v2 10/18] Create fix_utf8_chars function

[Muhammad Rizki]

From: Muhammad Rizki <riskimuhammmad1@gmail.com>

Some of the codes are repeated, so I create this function to make it
cleaner.

Signed-off-by: Muhammad Rizki <riskimuhammmad1@gmail.com>
---
 daemon/scraper/utils.py | 19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)

diff --git a/daemon/scraper/utils.py b/daemon/scraper/utils.py
index 5335fe2..3f0d254 100644
--- a/daemon/scraper/utils.py
+++ b/daemon/scraper/utils.py
@@ -183,10 +183,7 @@ def create_template(thread: Message, to=None, cc=None):
 			ret = ret[:4000] + "..."
 
 		ret = (
-			ret.rstrip()
-			.replace("<", "&lt;")
-			.replace(">","&gt;")
-			.replace("�"," ")
+			fix_utf8_char(ret)
 		) + "\n<code>------------------------------------------------------------------------</code>"
 
 	return ret, files, is_patch
@@ -213,10 +210,7 @@ def prepare_send_patch(mail, text, url):
 
 	caption = (
 		"#patch #ml\n" +
-		cap.rstrip()
-			.replace("<", "&lt;")
-			.replace(">","&gt;")
-			.replace("�"," ")
+		fix_utf8_char(cap)
 	)
 	return tmp, file, caption, url
 
@@ -225,6 +219,15 @@ def clean_up_after_send_patch(tmp):
 	shutil.rmtree(tmp)
 
 
+def fix_utf8_char(text: str):
+	return (
+		text.rstrip()
+		.replace("<", "&lt;")
+		.replace(">","&gt;")
+		.replace("�"," ")
+	)
+
+
 EMAIL_MSG_ID_PATTERN = r"<([^\<\>]+)>"
 def extract_email_msg_id(msg_id):
 	ret = re.search(EMAIL_MSG_ID_PATTERN, msg_id)
-- 
Muhammad Rizki

[PATCH v2 11/18] Recode some codes

[Muhammad Rizki]

From: Muhammad Rizki <riskimuhammmad1@gmail.com>

Just recode some codes, I think it much practice and clean (?)

Signed-off-by: Muhammad Rizki <riskimuhammmad1@gmail.com>
---
 daemon/scraper/utils.py | 12 +++++-------
 1 file changed, 5 insertions(+), 7 deletions(-)

diff --git a/daemon/scraper/utils.py b/daemon/scraper/utils.py
index 3f0d254..5496483 100644
--- a/daemon/scraper/utils.py
+++ b/daemon/scraper/utils.py
@@ -182,9 +182,8 @@ def create_template(thread: Message, to=None, cc=None):
 		if len(ret) >= 4000:
 			ret = ret[:4000] + "..."
 
-		ret = (
-			fix_utf8_char(ret)
-		) + "\n<code>------------------------------------------------------------------------</code>"
+		ret = fix_utf8_char(ret)
+		ret += "\n<code>------------------------------------------------------------------------</code>"
 
 	return ret, files, is_patch
 
@@ -208,10 +207,9 @@ def prepare_send_patch(mail, text, url):
 	with open(file, "wb") as f:
 		f.write(bytes(text, encoding="utf8"))
 
-	caption = (
-		"#patch #ml\n" +
-		fix_utf8_char(cap)
-	)
+	caption = "#patch #ml\n"
+	caption += fix_utf8_char(cap)
+
 	return tmp, file, caption, url
 
 
-- 
Muhammad Rizki

[PATCH v2 12/18] Fix indentation issue

[Muhammad Rizki]

From: Muhammad Rizki <riskimuhammmad1@gmail.com>

I found indentation issue in packages/client.py where tab and spaces are
combined together so I decided to change just 8-tab only.

Signed-off-by: Muhammad Rizki <riskimuhammmad1@gmail.com>
---
 daemon/packages/client.py | 52 +++++++++++++++++++--------------------
 1 file changed, 26 insertions(+), 26 deletions(-)

diff --git a/daemon/packages/client.py b/daemon/packages/client.py
index fd5b5ec..bb62c86 100644
--- a/daemon/packages/client.py
+++ b/daemon/packages/client.py
@@ -5,22 +5,22 @@ from typing import Union, BinaryIO
 
 
 class DaemonClient(Client):
-        def __init__(self, name: str, api_id: int,
-                     api_hash: str, **kwargs):
-                super().__init__(name, api_id,
-                                api_hash, **kwargs)
+	def __init__(self, name: str, api_id: int,
+			api_hash: str, **kwargs):
+		super().__init__(name, api_id,
+				api_hash, **kwargs)
 
 
-        async def send_text_email(
-                self,
-                chat_id: Union[int, str],
-                text: str,
-                reply_to: int,
-                url: str = None,
-                parse_mode: ParseMode = ParseMode.HTML
-        ) -> Message:
-                print("[send_text_email]")
-                return await self.send_message(
+	async def send_text_email(
+		self,
+		chat_id: Union[int, str],
+		text: str,
+		reply_to: int,
+		url: str = None,
+		parse_mode: ParseMode = ParseMode.HTML
+	) -> Message:
+		print("[send_text_email]")
+		return await self.send_message(
 			chat_id=chat_id,
 			text=text,
 			reply_to_message_id=reply_to,
@@ -34,19 +34,19 @@ class DaemonClient(Client):
 		)
 
 
-        async def send_patch_email(
-                self,
-                chat_id: Union[int, str],
-                doc: Union[str, BinaryIO],
-                caption: str,
-                reply_to: int,
-                url: str = None,
-                parse_mode: ParseMode = ParseMode.HTML
-        ) -> Message:
-                print("[send_patch_email]")
-                return await self.send_document(
+	async def send_patch_email(
+		self,
+		chat_id: Union[int, str],
+		doc: Union[str, BinaryIO],
+		caption: str,
+		reply_to: int,
+		url: str = None,
+		parse_mode: ParseMode = ParseMode.HTML
+	) -> Message:
+		print("[send_patch_email]")
+		return await self.send_document(
 			chat_id=chat_id,
-                        document=doc,
+			document=doc,
 			caption=caption,
 			reply_to_message_id=reply_to,
 			parse_mode=parse_mode,
-- 
Muhammad Rizki

[PATCH v2 13/18] Remove whitespace

[Muhammad Rizki]

From: Muhammad Rizki <riskimuhammmad1@gmail.com>

I found whitespace after newline, it's removed now

Signed-off-by: Muhammad Rizki <riskimuhammmad1@gmail.com>
---
 daemon/scraper/bot.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/daemon/scraper/bot.py b/daemon/scraper/bot.py
index 4903fef..33a9eaa 100644
--- a/daemon/scraper/bot.py
+++ b/daemon/scraper/bot.py
@@ -159,7 +159,7 @@ class Bot():
 
 	async def __send_patch_msg(self, mail, tg_chat_id, reply_to, text, url):
 		print("[__send_patch_msg]")
-		
+
 		tmp, doc, caption, url = Bot.prepare_send_patch(mail, text, url)
 		ret = await self.__handle_telegram_floodwait(
 			self.client.send_patch_email,
-- 
Muhammad Rizki

[PATCH v2 14/18] Remove prepare patch and clean up patch

[Muhammad Rizki]

From: Muhammad Rizki <riskimuhammmad1@gmail.com>

These functions already moved to scaper/utils.py, so I decide to remove
it in scaper/bot.py and change some of the code that required to change.

Signed-off-by: Muhammad Rizki <riskimuhammmad1@gmail.com>
---
 daemon/scraper/bot.py | 39 ++-------------------------------------
 1 file changed, 2 insertions(+), 37 deletions(-)

diff --git a/daemon/scraper/bot.py b/daemon/scraper/bot.py
index 33a9eaa..1abfa98 100644
--- a/daemon/scraper/bot.py
+++ b/daemon/scraper/bot.py
@@ -160,50 +160,15 @@ class Bot():
 	async def __send_patch_msg(self, mail, tg_chat_id, reply_to, text, url):
 		print("[__send_patch_msg]")
 
-		tmp, doc, caption, url = Bot.prepare_send_patch(mail, text, url)
+		tmp, doc, caption, url = utils.prepare_send_patch(mail, text, url)
 		ret = await self.__handle_telegram_floodwait(
 			self.client.send_patch_email,
 			*[tg_chat_id, doc, caption, reply_to, url]
 		)
-		Bot.clean_up_after_send_patch(tmp)
+		utils.clean_up_after_send_patch(tmp)
 		return ret
 
 
-	@staticmethod
-	def prepare_send_patch(mail, text, url):
-		tmp = utils.gen_temp(url)
-		fnm = str(mail.get("subject"))
-		sch = re.search(utils.PATCH_PATTERN, fnm, re.IGNORECASE)
-
-		nr_patch = sch.group(1)
-		if not nr_patch:
-			nr_patch = 1
-		else:
-			nr_patch = int(nr_patch)
-
-		num = "%04d" % nr_patch
-		fnm = slugify(sch.group(3)).replace("_", "-")
-		file = f"{tmp}/{num}-{fnm}.patch"
-		cap = text.split("\n\n")[0]
-
-		with open(file, "wb") as f:
-			f.write(bytes(text, encoding="utf8"))
-
-		caption = (
-			"#patch #ml\n" +
-			cap.rstrip()
-				.replace("<", "&lt;")
-				.replace(">","&gt;")
-				.replace("�"," ")
-		)
-		return tmp, file, caption, url
-
-
-	@staticmethod
-	def clean_up_after_send_patch(tmp):
-		shutil.rmtree(tmp)
-
-
 	async def __send_text_msg(self, *args):
 		return await self.__handle_telegram_floodwait(
 			self.client.send_text_email,
-- 
Muhammad Rizki

[PATCH v2 15/18] daemon: Remove unused imports

[Muhammad Rizki]

From: Muhammad Rizki <riskimuhammmad1@gmail.com>

Some module imported are unused, will be better if remove them.

Signed-off-by: Muhammad Rizki <riskimuhammmad1@gmail.com>
---
 daemon/scraper/bot.py | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/daemon/scraper/bot.py b/daemon/scraper/bot.py
index 1abfa98..1acc783 100644
--- a/daemon/scraper/bot.py
+++ b/daemon/scraper/bot.py
@@ -5,12 +5,8 @@
 #
 
 from apscheduler.schedulers.asyncio import AsyncIOScheduler
-from pyrogram.types import InlineKeyboardMarkup
-from pyrogram.types import InlineKeyboardButton
-from slugify import slugify
 from packages import DaemonClient
 from scraper import Scraper
-from pyrogram import enums
 from . import utils
 from .db import Db
 import pyrogram
-- 
Muhammad Rizki

[PATCH v2 16/18] daemon: Add @handle_flood decorator and remove some functions

[Muhammad Rizki]

From: Muhammad Rizki <riskimuhammmad1@gmail.com>

I decided to create @handle_flood decorator to make it easier to handle
the Telegram floodwait. Remove __send_text_msg() and replace the
call of the __send_text_msg() with send_text_email() in
packages/client.py and it have handle_flood decorator support.

Signed-off-by: Muhammad Rizki <riskimuhammmad1@gmail.com>
---
 daemon/packages/client.py    |  3 +++
 daemon/packages/decorator.py | 38 +++++++++++++++++++++++++++++++++
 daemon/scraper/bot.py        | 41 +++++-------------------------------
 3 files changed, 46 insertions(+), 36 deletions(-)
 create mode 100644 daemon/packages/decorator.py

diff --git a/daemon/packages/client.py b/daemon/packages/client.py
index bb62c86..d4689d3 100644
--- a/daemon/packages/client.py
+++ b/daemon/packages/client.py
@@ -2,6 +2,7 @@ from pyrogram import Client
 from pyrogram.enums import ParseMode
 from pyrogram.types import Message, InlineKeyboardMarkup, InlineKeyboardButton
 from typing import Union, BinaryIO
+from .decorator import handle_flood
 
 
 class DaemonClient(Client):
@@ -11,6 +12,7 @@ class DaemonClient(Client):
 				api_hash, **kwargs)
 
 
+	@handle_flood
 	async def send_text_email(
 		self,
 		chat_id: Union[int, str],
@@ -34,6 +36,7 @@ class DaemonClient(Client):
 		)
 
 
+	@handle_flood
 	async def send_patch_email(
 		self,
 		chat_id: Union[int, str],
diff --git a/daemon/packages/decorator.py b/daemon/packages/decorator.py
new file mode 100644
index 0000000..7d7dc39
--- /dev/null
+++ b/daemon/packages/decorator.py
@@ -0,0 +1,38 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Copyright (C) 2022  Muhammad Rizki <riskimuhammmad1@gmail.com>
+#
+
+from pyrogram.errors.exceptions.flood_420 import FloodWait
+from typing import Any, Callable
+from functools import wraps
+import re
+import asyncio
+
+__all__ = ["handle_flood"]
+
+
+def handle_flood(func: Callable[[Any], Any]) -> Callable[[Any], Any]:
+	@wraps(func)
+	async def callback(*args: Any) -> Any:
+		while True:
+			try:
+				return await func(*args)
+			except FloodWait as e:
+				#
+				# Aiee... we hit our limit.
+				# Let's slow down a bit.
+				#
+				_flood_exceptions(e)
+				print("[__handle_telegram_floodwait]: Woken up from flood wait...")
+	return callback
+
+
+async def _flood_exceptions(e):
+	x = re.search(r"A wait of (\d+) seconds is required", str(e))
+	if not x:
+		raise e
+
+	n = int(x.group(1))
+	print(f"[____handle_telegram_floodwait]: Sleeping for {n} seconds due to Telegram limit")
+	await asyncio.sleep(n)
diff --git a/daemon/scraper/bot.py b/daemon/scraper/bot.py
index 1acc783..bc4025d 100644
--- a/daemon/scraper/bot.py
+++ b/daemon/scraper/bot.py
@@ -118,8 +118,9 @@ class Bot():
 							reply_to, text, url)
 		else:
 			text = "#ml\n" + text
-			m = await self.__send_text_msg(tg_chat_id, text,
-						       reply_to, url)
+			m = await self.client.send_text_email(
+				tg_chat_id, text,reply_to, url
+			)
 
 		self.db.insert_telegram(email_id, m.chat.id, m.id)
 		for d, f in files:
@@ -157,40 +158,8 @@ class Bot():
 		print("[__send_patch_msg]")
 
 		tmp, doc, caption, url = utils.prepare_send_patch(mail, text, url)
-		ret = await self.__handle_telegram_floodwait(
-			self.client.send_patch_email,
-			*[tg_chat_id, doc, caption, reply_to, url]
+		ret = await self.client.send_patch_email(
+			tg_chat_id, doc, caption, reply_to, url
 		)
 		utils.clean_up_after_send_patch(tmp)
 		return ret
-
-
-	async def __send_text_msg(self, *args):
-		return await self.__handle_telegram_floodwait(
-			self.client.send_text_email,
-			*args
-		)
-
-
-	async def __handle_telegram_floodwait(self, callback, *args):
-		while True:
-			try:
-				return await callback(*args)
-			except pyrogram.errors.exceptions.flood_420.FloodWait as e:
-				#
-				# Aiee... we hit our limit.
-				# Let's slow down a bit.
-				#
-				await self.____handle_telegram_floodwait(e)
-				print("[__handle_telegram_floodwait]: Woken up from flood wait...")
-
-
-	async def ____handle_telegram_floodwait(self, e):
-		x = str(e)
-		x = re.search(r"A wait of (\d+) seconds is required", x)
-		if not x:
-			raise e
-
-		n = int(x.group(1))
-		print(f"[____handle_telegram_floodwait]: Sleeping for {n} seconds due to Telegram limit")
-		await asyncio.sleep(n)
-- 
Muhammad Rizki

[PATCH v2 17/18] daemon: Remove __send_patch_msg()

[Muhammad Rizki]

From: Muhammad Rizki <riskimuhammmad1@gmail.com>

I want to remove the __send_patch_msg() and displace the prepare patch,
send patch, and clean up patch into the send_patch_email() in
packages/client.py.

Signed-off-by: Muhammad Rizki <riskimuhammmad1@gmail.com>
---
 daemon/packages/client.py | 12 +++++++++---
 daemon/scraper/bot.py     | 16 +++-------------
 2 files changed, 12 insertions(+), 16 deletions(-)

diff --git a/daemon/packages/client.py b/daemon/packages/client.py
index d4689d3..e125ed7 100644
--- a/daemon/packages/client.py
+++ b/daemon/packages/client.py
@@ -2,6 +2,8 @@ from pyrogram import Client
 from pyrogram.enums import ParseMode
 from pyrogram.types import Message, InlineKeyboardMarkup, InlineKeyboardButton
 from typing import Union, BinaryIO
+from email.message import Message
+from scraper import utils
 from .decorator import handle_flood
 
 
@@ -39,15 +41,16 @@ class DaemonClient(Client):
 	@handle_flood
 	async def send_patch_email(
 		self,
+		mail: Message,
 		chat_id: Union[int, str],
-		doc: Union[str, BinaryIO],
-		caption: str,
+		text: str,
 		reply_to: int,
 		url: str = None,
 		parse_mode: ParseMode = ParseMode.HTML
 	) -> Message:
 		print("[send_patch_email]")
-		return await self.send_document(
+		tmp, doc, caption, url = utils.prepare_send_patch(mail, text, url)
+		m = await self.send_document(
 			chat_id=chat_id,
 			document=doc,
 			caption=caption,
@@ -60,3 +63,6 @@ class DaemonClient(Client):
 				)]
 			])
 		)
+
+		utils.clean_up_after_send_patch(tmp)
+		return m
diff --git a/daemon/scraper/bot.py b/daemon/scraper/bot.py
index bc4025d..c193c9e 100644
--- a/daemon/scraper/bot.py
+++ b/daemon/scraper/bot.py
@@ -114,8 +114,9 @@ class Bot():
 		url = str(re.sub(r"/raw$", "", url))
 
 		if is_patch:
-			m = await self.__send_patch_msg(mail, tg_chat_id,
-							reply_to, text, url)
+			m = await self.client.send_patch_email(
+				mail, tg_chat_id, text, reply_to, url
+			)
 		else:
 			text = "#ml\n" + text
 			m = await self.client.send_text_email(
@@ -152,14 +153,3 @@ class Bot():
 			return None
 
 		return self.db.get_tg_reply_to(reply_to, tg_chat_id)
-
-
-	async def __send_patch_msg(self, mail, tg_chat_id, reply_to, text, url):
-		print("[__send_patch_msg]")
-
-		tmp, doc, caption, url = utils.prepare_send_patch(mail, text, url)
-		ret = await self.client.send_patch_email(
-			tg_chat_id, doc, caption, reply_to, url
-		)
-		utils.clean_up_after_send_patch(tmp)
-		return ret
-- 
Muhammad Rizki

[PATCH v2 18/18] daemon: Remove unused imports

[Muhammad Rizki]

Just remove some unused imports in packages/client.py and scraper/bot.py

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/packages/client.py | 2 +-
 daemon/scraper/bot.py     | 1 -
 2 files changed, 1 insertion(+), 2 deletions(-)

diff --git a/daemon/packages/client.py b/daemon/packages/client.py
index e125ed7..277bb99 100644
--- a/daemon/packages/client.py
+++ b/daemon/packages/client.py
@@ -1,7 +1,7 @@
 from pyrogram import Client
 from pyrogram.enums import ParseMode
 from pyrogram.types import Message, InlineKeyboardMarkup, InlineKeyboardButton
-from typing import Union, BinaryIO
+from typing import Union
 from email.message import Message
 from scraper import utils
 from .decorator import handle_flood
diff --git a/daemon/scraper/bot.py b/daemon/scraper/bot.py
index c193c9e..fbe3716 100644
--- a/daemon/scraper/bot.py
+++ b/daemon/scraper/bot.py
@@ -9,7 +9,6 @@ from packages import DaemonClient
 from scraper import Scraper
 from . import utils
 from .db import Db
-import pyrogram
 import asyncio
 import shutil
 import re
-- 
Muhammad Rizki

Re: [PATCH v2 12/18] Fix indentation issue

[Ammar Faizi]

On 7/19/22 7:17 AM, Muhammad Rizki wrote:
> From: Muhammad Rizki <riskimuhammmad1@gmail.com>
> 
> I found indentation issue in packages/client.py where tab and spaces are
> combined together so I decided to change just 8-tab only.
> 
> Signed-off-by: Muhammad Rizki <riskimuhammmad1@gmail.com>

This should've been folded into patch #7. Just rebase edit your patch #7
and fix the indentation there.

Don't add extra patch to fix your previous patch unless _the commit to be
fixed_ has already landed in master.

-- 
Ammar Faizi

Re: [PATCH v2 10/18] Create fix_utf8_chars function

[Ammar Faizi]

On 7/19/22 7:17 AM, Muhammad Rizki wrote:
>   	caption = (
>   		"#patch #ml\n" +
> -		cap.rstrip()
> -			.replace("<", "&lt;")
> -			.replace(">","&gt;")
> -			.replace("�"," ")
> +		fix_utf8_char(cap)
>   	)

Remove the parentheses, just do this:

	caption = fix_utf8_char(cap)

-- 
Ammar Faizi