public inbox for [email protected]
 help / color / mirror / Atom feed
* [PATCH v1 00/11] Plan to redesign code
@ 2022-07-18 11:19 Muhammad Rizki
  2022-07-18 11:19 ` [PATCH v1 01/11] Fix __send_patch_msg function parameter Muhammad Rizki
                   ` (12 more replies)
  0 siblings, 13 replies; 20+ messages in thread
From: Muhammad Rizki @ 2022-07-18 11:19 UTC (permalink / raw)
  To: Ammar Faizi; +Cc: Muhammad Rizki, GNU/Weeb Mailing List

Hi sir,

In this series I want to redesign codes to make it clear and clean.
I've inherit the DaemonClient with the Pyrogram's Client, so with
this method we can create our custom functions, such as
send email message and patch message in the DaemonClient.

There 10 patches in this series:

- Patch 1 is to fix the lack of __send_patch_msg function parameter
- Patch 2 is just add required imports and remove unused imports
- Patch 3 is to add default temporary directory name
- Patch 4 is to move Telegram bot session file
- Patch 5 is to add traceback to get the error detail while debugging
- Patch 6 is to create send email and patch msg in the DaemonClient
- Patch 7 is just remove the ____send_patch_msg
- Patch 8 is to move utility functions to the utils file
- Patch 9 is to create fix_utf8_chars function
- Patch 10 is just cleaning some codes to make it looks more practice

Please give it a test, thanks.

Signed-off-by: Muhammad Rizki <[email protected]>
---

Ammar Faizi (1):
  daemon: Fix raw lore URL on the inline keyboard button

Muhammad Rizki (10):
  Fix __send_patch_msg function parameter
  Fix import problem
  Add default temporary directory
  Move the Telegram bot session into the storage directory
  Add traceback to get the error detail
  Re-design send email message to Telegram
  Move ____send_patch_msg
  Move prepare for patch and clean up patch functions
  Create fix_utf8_chars function
  Recode some codes

 daemon/packages/__init__.py       |  1 +
 daemon/packages/client.py         | 59 ++++++++++++++++++++++++++++++
 daemon/packages/plugins/scrape.py |  1 +
 daemon/run.py                     |  8 ++--
 daemon/scraper/bot.py             | 61 +++++++------------------------
 daemon/scraper/scraper.py         |  4 --
 daemon/scraper/utils.py           | 50 +++++++++++++++++++++----
 7 files changed, 121 insertions(+), 63 deletions(-)
 create mode 100644 daemon/packages/__init__.py
 create mode 100644 daemon/packages/client.py


base-commit: 339d992bdd77f19e67d0d9a30e3ac6eef1e4035a
-- 
Muhammad Rizki


^ permalink raw reply	[flat|nested] 20+ messages in thread

* [PATCH v1 01/11] Fix __send_patch_msg function parameter
  2022-07-18 11:19 [PATCH v1 00/11] Plan to redesign code Muhammad Rizki
@ 2022-07-18 11:19 ` Muhammad Rizki
  2022-07-18 12:45   ` Ammar Faizi
  2022-07-18 11:19 ` [PATCH v1 02/11] Fix import problem Muhammad Rizki
                   ` (11 subsequent siblings)
  12 siblings, 1 reply; 20+ messages in thread
From: Muhammad Rizki @ 2022-07-18 11:19 UTC (permalink / raw)
  To: Ammar Faizi; +Cc: Muhammad Rizki, GNU/Weeb Mailing List

Lack of reply_to parameter in __send_patch_msg function.

Signed-off-by: Muhammad Rizki <[email protected]>
---
 daemon/scraper/bot.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/daemon/scraper/bot.py b/daemon/scraper/bot.py
index 7746eb8..9b95d50 100644
--- a/daemon/scraper/bot.py
+++ b/daemon/scraper/bot.py
@@ -157,7 +157,7 @@ class Bot():
 		return self.db.get_tg_reply_to(reply_to, tg_chat_id)
 
 
-	async def __send_patch_msg(self, mail, tg_chat_id, text, url):
+	async def __send_patch_msg(self, mail, tg_chat_id, reply_to, text, url):
 		print("[__send_patch_msg]")
 		
 		tmp, fnm, caption, url = Bot.prepare_send_patch(mail, text, url)
-- 
Muhammad Rizki


^ permalink raw reply related	[flat|nested] 20+ messages in thread

* [PATCH v1 02/11] Fix import problem
  2022-07-18 11:19 [PATCH v1 00/11] Plan to redesign code Muhammad Rizki
  2022-07-18 11:19 ` [PATCH v1 01/11] Fix __send_patch_msg function parameter Muhammad Rizki
@ 2022-07-18 11:19 ` Muhammad Rizki
  2022-07-18 11:19 ` [PATCH v1 03/11] Add default temporary directory Muhammad Rizki
                   ` (10 subsequent siblings)
  12 siblings, 0 replies; 20+ messages in thread
From: Muhammad Rizki @ 2022-07-18 11:19 UTC (permalink / raw)
  To: Ammar Faizi; +Cc: Muhammad Rizki, GNU/Weeb Mailing List

Signed-off-by: Muhammad Rizki <[email protected]>
---
 daemon/packages/plugins/scrape.py | 1 +
 daemon/run.py                     | 2 --
 daemon/scraper/bot.py             | 1 -
 daemon/scraper/scraper.py         | 4 ----
 4 files changed, 1 insertion(+), 7 deletions(-)

diff --git a/daemon/packages/plugins/scrape.py b/daemon/packages/plugins/scrape.py
index c9da71a..1698c6d 100644
--- a/daemon/packages/plugins/scrape.py
+++ b/daemon/packages/plugins/scrape.py
@@ -15,6 +15,7 @@ from scraper import utils
 from scraper import Bot
 import shutil
 import re
+import asyncio
 
 
 #
diff --git a/daemon/run.py b/daemon/run.py
index ef1d8f4..0a650cd 100644
--- a/daemon/run.py
+++ b/daemon/run.py
@@ -5,14 +5,12 @@
 #
 
 from apscheduler.schedulers.asyncio import AsyncIOScheduler
-from email.message import Message
 from scraper import BotMutexes
 from dotenv import load_dotenv
 from mysql import connector
 from pyrogram import Client
 from scraper import Scraper
 from scraper import Bot
-import asyncio
 import os
 
 
diff --git a/daemon/scraper/bot.py b/daemon/scraper/bot.py
index 9b95d50..c62c554 100644
--- a/daemon/scraper/bot.py
+++ b/daemon/scraper/bot.py
@@ -13,7 +13,6 @@ from scraper import Scraper
 from pyrogram import enums
 from . import utils
 from .db import Db
-import xmltodict
 import pyrogram
 import asyncio
 import shutil
diff --git a/daemon/scraper/scraper.py b/daemon/scraper/scraper.py
index b70df31..2d5942b 100644
--- a/daemon/scraper/scraper.py
+++ b/daemon/scraper/scraper.py
@@ -5,14 +5,10 @@
 #
 
 from typing import Dict, List
-from .db import Db
 import email.policy
 import xmltodict
-import operator
-import asyncio
 import httpx
 import email
-import json
 
 
 class Scraper():
-- 
Muhammad Rizki


^ permalink raw reply related	[flat|nested] 20+ messages in thread

* [PATCH v1 03/11] Add default temporary directory
  2022-07-18 11:19 [PATCH v1 00/11] Plan to redesign code Muhammad Rizki
  2022-07-18 11:19 ` [PATCH v1 01/11] Fix __send_patch_msg function parameter Muhammad Rizki
  2022-07-18 11:19 ` [PATCH v1 02/11] Fix import problem Muhammad Rizki
@ 2022-07-18 11:19 ` Muhammad Rizki
  2022-07-18 11:31   ` Ammar Faizi
  2022-07-18 11:20 ` [PATCH v1 04/11] Move the Telegram bot session into the storage directory Muhammad Rizki
                   ` (9 subsequent siblings)
  12 siblings, 1 reply; 20+ messages in thread
From: Muhammad Rizki @ 2022-07-18 11:19 UTC (permalink / raw)
  To: Ammar Faizi; +Cc: Muhammad Rizki, GNU/Weeb Mailing List

Signed-off-by: Muhammad Rizki <[email protected]>
---
 daemon/scraper/utils.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/daemon/scraper/utils.py b/daemon/scraper/utils.py
index 9334b82..10fe956 100644
--- a/daemon/scraper/utils.py
+++ b/daemon/scraper/utils.py
@@ -110,7 +110,7 @@ def consruct_to_n_cc(to: list, cc: list):
 
 def gen_temp(name: str):
 	md5 = hashlib.md5(name.encode()).hexdigest()
-	ret = os.getenv("STORAGE_DIR") + "/" + md5
+	ret = os.getenv("STORAGE_DIR", "storage") + "/" + md5
 	try:
 		os.mkdir(ret)
 	except FileExistsError:
-- 
Muhammad Rizki


^ permalink raw reply related	[flat|nested] 20+ messages in thread

* [PATCH v1 04/11] Move the Telegram bot session into the storage directory
  2022-07-18 11:19 [PATCH v1 00/11] Plan to redesign code Muhammad Rizki
                   ` (2 preceding siblings ...)
  2022-07-18 11:19 ` [PATCH v1 03/11] Add default temporary directory Muhammad Rizki
@ 2022-07-18 11:20 ` Muhammad Rizki
  2022-07-18 11:20 ` [PATCH v1 05/11] daemon: Fix raw lore URL on the inline keyboard button Muhammad Rizki
                   ` (8 subsequent siblings)
  12 siblings, 0 replies; 20+ messages in thread
From: Muhammad Rizki @ 2022-07-18 11:20 UTC (permalink / raw)
  To: Ammar Faizi; +Cc: Muhammad Rizki, GNU/Weeb Mailing List

Signed-off-by: Muhammad Rizki <[email protected]>
---
 daemon/run.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/daemon/run.py b/daemon/run.py
index 0a650cd..83b2cdb 100644
--- a/daemon/run.py
+++ b/daemon/run.py
@@ -18,7 +18,7 @@ def main():
 	load_dotenv()
 
 	client = Client(
-		"EmailScraper",
+		"storage/EmailScraper",
 		api_id=int(os.getenv("API_ID")),
 		api_hash=os.getenv("API_HASH"),
 		bot_token=os.getenv("BOT_TOKEN"),
-- 
Muhammad Rizki


^ permalink raw reply related	[flat|nested] 20+ messages in thread

* [PATCH v1 05/11] daemon: Fix raw lore URL on the inline keyboard button
  2022-07-18 11:19 [PATCH v1 00/11] Plan to redesign code Muhammad Rizki
                   ` (3 preceding siblings ...)
  2022-07-18 11:20 ` [PATCH v1 04/11] Move the Telegram bot session into the storage directory Muhammad Rizki
@ 2022-07-18 11:20 ` Muhammad Rizki
  2022-07-18 11:20 ` [PATCH v1 06/11] Add traceback to get the error detail Muhammad Rizki
                   ` (7 subsequent siblings)
  12 siblings, 0 replies; 20+ messages in thread
From: Muhammad Rizki @ 2022-07-18 11:20 UTC (permalink / raw)
  To: Ammar Faizi; +Cc: GNU/Weeb Mailing List, Muhammad Rizki

From: Ammar Faizi <[email protected]>

The current url contains "/raw" at the end of the link. This will give
us the raw email. As this button is supposed to be read by users, we
shouldn't give the raw version. Remove "/raw" at the end of URL before
sending it to Telegram to make it more user friendly.

Signed-off-by: Ammar Faizi <[email protected]>
Signed-off-by: Muhammad Rizki <[email protected]>
---
 daemon/scraper/bot.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/daemon/scraper/bot.py b/daemon/scraper/bot.py
index c62c554..04caab6 100644
--- a/daemon/scraper/bot.py
+++ b/daemon/scraper/bot.py
@@ -115,6 +115,7 @@ class Bot():
 
 		text, files, is_patch = utils.create_template(mail)
 		reply_to = self.get_tg_reply_to(mail, tg_chat_id)
+		url = str(re.sub(r"/raw$", "", url))
 
 		if is_patch:
 			m = await self.__send_patch_msg(mail, tg_chat_id,
-- 
Muhammad Rizki


^ permalink raw reply related	[flat|nested] 20+ messages in thread

* [PATCH v1 06/11] Add traceback to get the error detail
  2022-07-18 11:19 [PATCH v1 00/11] Plan to redesign code Muhammad Rizki
                   ` (4 preceding siblings ...)
  2022-07-18 11:20 ` [PATCH v1 05/11] daemon: Fix raw lore URL on the inline keyboard button Muhammad Rizki
@ 2022-07-18 11:20 ` Muhammad Rizki
  2022-07-18 11:27   ` Ammar Faizi
  2022-07-18 11:20 ` [PATCH v1 07/11] Re-design send email message to Telegram Muhammad Rizki
                   ` (6 subsequent siblings)
  12 siblings, 1 reply; 20+ messages in thread
From: Muhammad Rizki @ 2022-07-18 11:20 UTC (permalink / raw)
  To: Ammar Faizi; +Cc: Muhammad Rizki, GNU/Weeb Mailing List

Signed-off-by: Muhammad Rizki <[email protected]>
---
 daemon/scraper/bot.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/daemon/scraper/bot.py b/daemon/scraper/bot.py
index 04caab6..2392b61 100644
--- a/daemon/scraper/bot.py
+++ b/daemon/scraper/bot.py
@@ -17,6 +17,7 @@ import pyrogram
 import asyncio
 import shutil
 import re
+import traceback
 
 
 class BotMutexes():
@@ -63,8 +64,8 @@ class Bot():
 		for url in self.ATOM_URLS:
 			try:
 				await self.__handle_atom_url(url)
-			except Exception as e:
-				print(f"[__run]: Error: {e}")
+			except:
+				print(traceback.format_exc())
 
 		if not self.isRunnerFixed:
 			self.isRunnerFixed = True
-- 
Muhammad Rizki


^ permalink raw reply related	[flat|nested] 20+ messages in thread

* [PATCH v1 07/11] Re-design send email message to Telegram
  2022-07-18 11:19 [PATCH v1 00/11] Plan to redesign code Muhammad Rizki
                   ` (5 preceding siblings ...)
  2022-07-18 11:20 ` [PATCH v1 06/11] Add traceback to get the error detail Muhammad Rizki
@ 2022-07-18 11:20 ` Muhammad Rizki
  2022-07-18 12:49   ` Ammar Faizi
  2022-07-18 11:20 ` [PATCH v1 08/11] Move ____send_patch_msg Muhammad Rizki
                   ` (5 subsequent siblings)
  12 siblings, 1 reply; 20+ messages in thread
From: Muhammad Rizki @ 2022-07-18 11:20 UTC (permalink / raw)
  To: Ammar Faizi; +Cc: Muhammad Rizki, GNU/Weeb Mailing List

We want to separate send message function and inherit with the Pyrogram
Client, so it should makes the code clean and clear.

Signed-off-by: Muhammad Rizki <[email protected]>
---
 daemon/packages/__init__.py |  1 +
 daemon/packages/client.py   | 59 +++++++++++++++++++++++++++++++++++++
 daemon/run.py               |  4 +--
 daemon/scraper/bot.py       | 35 ++++++----------------
 4 files changed, 71 insertions(+), 28 deletions(-)
 create mode 100644 daemon/packages/__init__.py
 create mode 100644 daemon/packages/client.py

diff --git a/daemon/packages/__init__.py b/daemon/packages/__init__.py
new file mode 100644
index 0000000..efef9ae
--- /dev/null
+++ b/daemon/packages/__init__.py
@@ -0,0 +1 @@
+from .client import DaemonClient
diff --git a/daemon/packages/client.py b/daemon/packages/client.py
new file mode 100644
index 0000000..fd5b5ec
--- /dev/null
+++ b/daemon/packages/client.py
@@ -0,0 +1,59 @@
+from pyrogram import Client
+from pyrogram.enums import ParseMode
+from pyrogram.types import Message, InlineKeyboardMarkup, InlineKeyboardButton
+from typing import Union, BinaryIO
+
+
+class DaemonClient(Client):
+        def __init__(self, name: str, api_id: int,
+                     api_hash: str, **kwargs):
+                super().__init__(name, api_id,
+                                api_hash, **kwargs)
+
+
+        async def send_text_email(
+                self,
+                chat_id: Union[int, str],
+                text: str,
+                reply_to: int,
+                url: str = None,
+                parse_mode: ParseMode = ParseMode.HTML
+        ) -> Message:
+                print("[send_text_email]")
+                return await self.send_message(
+			chat_id=chat_id,
+			text=text,
+			reply_to_message_id=reply_to,
+			parse_mode=parse_mode,
+			reply_markup=InlineKeyboardMarkup([
+				[InlineKeyboardButton(
+					"See the full message",
+					url=url
+				)]
+			])
+		)
+
+
+        async def send_patch_email(
+                self,
+                chat_id: Union[int, str],
+                doc: Union[str, BinaryIO],
+                caption: str,
+                reply_to: int,
+                url: str = None,
+                parse_mode: ParseMode = ParseMode.HTML
+        ) -> Message:
+                print("[send_patch_email]")
+                return await self.send_document(
+			chat_id=chat_id,
+                        document=doc,
+			caption=caption,
+			reply_to_message_id=reply_to,
+			parse_mode=parse_mode,
+			reply_markup=InlineKeyboardMarkup([
+				[InlineKeyboardButton(
+					"See the full message",
+					url=url
+				)]
+			])
+		)
diff --git a/daemon/run.py b/daemon/run.py
index 83b2cdb..1151ccd 100644
--- a/daemon/run.py
+++ b/daemon/run.py
@@ -8,7 +8,7 @@ from apscheduler.schedulers.asyncio import AsyncIOScheduler
 from scraper import BotMutexes
 from dotenv import load_dotenv
 from mysql import connector
-from pyrogram import Client
+from packages import DaemonClient
 from scraper import Scraper
 from scraper import Bot
 import os
@@ -17,7 +17,7 @@ import os
 def main():
 	load_dotenv()
 
-	client = Client(
+	client = DaemonClient(
 		"storage/EmailScraper",
 		api_id=int(os.getenv("API_ID")),
 		api_hash=os.getenv("API_HASH"),
diff --git a/daemon/scraper/bot.py b/daemon/scraper/bot.py
index 2392b61..93e633a 100644
--- a/daemon/scraper/bot.py
+++ b/daemon/scraper/bot.py
@@ -8,7 +8,7 @@ from apscheduler.schedulers.asyncio import AsyncIOScheduler
 from pyrogram.types import InlineKeyboardMarkup
 from pyrogram.types import InlineKeyboardButton
 from slugify import slugify
-from pyrogram import Client
+from packages import DaemonClient
 from scraper import Scraper
 from pyrogram import enums
 from . import utils
@@ -32,12 +32,11 @@ class Bot():
 	]
 
 	TG_CHAT_IDS = [
-		-1001394203410,
-		-1001673279485,
+		"kiizuah"
 	]
 
 
-	def __init__(self, client: Client, sched: AsyncIOScheduler,
+	def __init__(self, client: DaemonClient, sched: AsyncIOScheduler,
 		     scraper: Scraper, mutexes: BotMutexes, conn):
 		self.client = client
 		self.sched = sched
@@ -123,8 +122,8 @@ class Bot():
 							reply_to, text, url)
 		else:
 			text = "#ml\n" + text
-			m = await self.__send_text_msg(tg_chat_id, reply_to,
-						       text, url)
+			m = await self.__send_text_msg(tg_chat_id, text,
+						       reply_to, url)
 
 		self.db.insert_telegram(email_id, m.chat.id, m.id)
 		for d, f in files:
@@ -161,10 +160,10 @@ class Bot():
 	async def __send_patch_msg(self, mail, tg_chat_id, reply_to, text, url):
 		print("[__send_patch_msg]")
 		
-		tmp, fnm, caption, url = Bot.prepare_send_patch(mail, text, url)
+		tmp, doc, caption, url = Bot.prepare_send_patch(mail, text, url)
 		ret = await self.__handle_telegram_floodwait(
-			self.____send_patch_msg,
-			*[tg_chat_id, reply_to, fnm, caption, url]
+			self.client.send_patch_email,
+			*[tg_chat_id, doc, caption, reply_to, url]
 		)
 		Bot.clean_up_after_send_patch(tmp)
 		return ret
@@ -207,7 +206,7 @@ class Bot():
 
 	async def __send_text_msg(self, *args):
 		return await self.__handle_telegram_floodwait(
-			self.____send_text_msg,
+			self.client.send_text_email,
 			*args
 		)
 
@@ -251,19 +250,3 @@ class Bot():
 				)]
 			])
 		)
-
-
-	async def ____send_text_msg(self, tg_chat_id, reply_to, text, url):
-		print("[__send_text_msg]")
-		return await self.client.send_message(
-			tg_chat_id,
-			text,
-			reply_to_message_id=reply_to,
-			parse_mode=enums.ParseMode.HTML,
-			reply_markup=InlineKeyboardMarkup([
-				[InlineKeyboardButton(
-					"See the full message",
-					url=url
-				)]
-			])
-		)
-- 
Muhammad Rizki


^ permalink raw reply related	[flat|nested] 20+ messages in thread

* [PATCH v1 08/11] Move ____send_patch_msg
  2022-07-18 11:19 [PATCH v1 00/11] Plan to redesign code Muhammad Rizki
                   ` (6 preceding siblings ...)
  2022-07-18 11:20 ` [PATCH v1 07/11] Re-design send email message to Telegram Muhammad Rizki
@ 2022-07-18 11:20 ` Muhammad Rizki
  2022-07-18 11:20 ` [PATCH v1 09/11] Move prepare for patch and clean up patch functions Muhammad Rizki
                   ` (4 subsequent siblings)
  12 siblings, 0 replies; 20+ messages in thread
From: Muhammad Rizki @ 2022-07-18 11:20 UTC (permalink / raw)
  To: Ammar Faizi; +Cc: Muhammad Rizki, GNU/Weeb Mailing List

I want to move the ____send_patch_msg into the DaemonClient, so it much
cleaner if we separate the send email message into the DaemonClient.

Signed-off-by: Muhammad Rizki <[email protected]>
---
 daemon/scraper/bot.py | 17 -----------------
 1 file changed, 17 deletions(-)

diff --git a/daemon/scraper/bot.py b/daemon/scraper/bot.py
index 93e633a..4903fef 100644
--- a/daemon/scraper/bot.py
+++ b/daemon/scraper/bot.py
@@ -233,20 +233,3 @@ class Bot():
 		n = int(x.group(1))
 		print(f"[____handle_telegram_floodwait]: Sleeping for {n} seconds due to Telegram limit")
 		await asyncio.sleep(n)
-
-
-	async def ____send_patch_msg(self, tg_chat_id, reply_to, fnm, caption,
-				     url):
-		return await self.client.send_document(
-			tg_chat_id,
-			fnm,
-			caption=caption,
-			reply_to_message_id=reply_to,
-			parse_mode=enums.ParseMode.HTML,
-			reply_markup=InlineKeyboardMarkup([
-				[InlineKeyboardButton(
-					"See the full message",
-					url=url
-				)]
-			])
-		)
-- 
Muhammad Rizki


^ permalink raw reply related	[flat|nested] 20+ messages in thread

* [PATCH v1 09/11] Move prepare for patch and clean up patch functions
  2022-07-18 11:19 [PATCH v1 00/11] Plan to redesign code Muhammad Rizki
                   ` (7 preceding siblings ...)
  2022-07-18 11:20 ` [PATCH v1 08/11] Move ____send_patch_msg Muhammad Rizki
@ 2022-07-18 11:20 ` Muhammad Rizki
  2022-07-18 11:20 ` [PATCH v1 10/11] Create fix_utf8_chars function Muhammad Rizki
                   ` (3 subsequent siblings)
  12 siblings, 0 replies; 20+ messages in thread
From: Muhammad Rizki @ 2022-07-18 11:20 UTC (permalink / raw)
  To: Ammar Faizi; +Cc: Muhammad Rizki, GNU/Weeb Mailing List

I want these function is separate into the utils or utility file, I'm
the type of person who likes to separate the utility function into the
utility file.

Signed-off-by: Muhammad Rizki <[email protected]>
---
 daemon/scraper/utils.py | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)

diff --git a/daemon/scraper/utils.py b/daemon/scraper/utils.py
index 10fe956..5335fe2 100644
--- a/daemon/scraper/utils.py
+++ b/daemon/scraper/utils.py
@@ -6,10 +6,12 @@
 
 from email.message import Message
 from typing import Dict
+from slugify import slugify
 import hashlib
 import uuid
 import os
 import re
+import shutil
 
 
 def get_email_msg_id(mail):
@@ -190,6 +192,39 @@ def create_template(thread: Message, to=None, cc=None):
 	return ret, files, is_patch
 
 
+def prepare_send_patch(mail, text, url):
+	tmp = gen_temp(url)
+	fnm = str(mail.get("subject"))
+	sch = re.search(PATCH_PATTERN, fnm, re.IGNORECASE)
+
+	nr_patch = sch.group(1)
+	if not nr_patch:
+		nr_patch = 1
+	else:
+		nr_patch = int(nr_patch)
+
+	num = "%04d" % nr_patch
+	fnm = slugify(sch.group(3)).replace("_", "-")
+	file = f"{tmp}/{num}-{fnm}.patch"
+	cap = text.split("\n\n")[0]
+
+	with open(file, "wb") as f:
+		f.write(bytes(text, encoding="utf8"))
+
+	caption = (
+		"#patch #ml\n" +
+		cap.rstrip()
+			.replace("<", "&lt;")
+			.replace(">","&gt;")
+			.replace("�"," ")
+	)
+	return tmp, file, caption, url
+
+
+def clean_up_after_send_patch(tmp):
+	shutil.rmtree(tmp)
+
+
 EMAIL_MSG_ID_PATTERN = r"<([^\<\>]+)>"
 def extract_email_msg_id(msg_id):
 	ret = re.search(EMAIL_MSG_ID_PATTERN, msg_id)
-- 
Muhammad Rizki


^ permalink raw reply related	[flat|nested] 20+ messages in thread

* [PATCH v1 10/11] Create fix_utf8_chars function
  2022-07-18 11:19 [PATCH v1 00/11] Plan to redesign code Muhammad Rizki
                   ` (8 preceding siblings ...)
  2022-07-18 11:20 ` [PATCH v1 09/11] Move prepare for patch and clean up patch functions Muhammad Rizki
@ 2022-07-18 11:20 ` Muhammad Rizki
  2022-07-18 11:20 ` [PATCH v1 11/11] Recode some codes Muhammad Rizki
                   ` (2 subsequent siblings)
  12 siblings, 0 replies; 20+ messages in thread
From: Muhammad Rizki @ 2022-07-18 11:20 UTC (permalink / raw)
  To: Ammar Faizi; +Cc: Muhammad Rizki, GNU/Weeb Mailing List

Some of the codes are repeated, so I create this function to make it
cleaner.

Signed-off-by: Muhammad Rizki <[email protected]>
---
 daemon/scraper/utils.py | 19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)

diff --git a/daemon/scraper/utils.py b/daemon/scraper/utils.py
index 5335fe2..3f0d254 100644
--- a/daemon/scraper/utils.py
+++ b/daemon/scraper/utils.py
@@ -183,10 +183,7 @@ def create_template(thread: Message, to=None, cc=None):
 			ret = ret[:4000] + "..."
 
 		ret = (
-			ret.rstrip()
-			.replace("<", "&lt;")
-			.replace(">","&gt;")
-			.replace("�"," ")
+			fix_utf8_char(ret)
 		) + "\n<code>------------------------------------------------------------------------</code>"
 
 	return ret, files, is_patch
@@ -213,10 +210,7 @@ def prepare_send_patch(mail, text, url):
 
 	caption = (
 		"#patch #ml\n" +
-		cap.rstrip()
-			.replace("<", "&lt;")
-			.replace(">","&gt;")
-			.replace("�"," ")
+		fix_utf8_char(cap)
 	)
 	return tmp, file, caption, url
 
@@ -225,6 +219,15 @@ def clean_up_after_send_patch(tmp):
 	shutil.rmtree(tmp)
 
 
+def fix_utf8_char(text: str):
+	return (
+		text.rstrip()
+		.replace("<", "&lt;")
+		.replace(">","&gt;")
+		.replace("�"," ")
+	)
+
+
 EMAIL_MSG_ID_PATTERN = r"<([^\<\>]+)>"
 def extract_email_msg_id(msg_id):
 	ret = re.search(EMAIL_MSG_ID_PATTERN, msg_id)
-- 
Muhammad Rizki


^ permalink raw reply related	[flat|nested] 20+ messages in thread

* [PATCH v1 11/11] Recode some codes
  2022-07-18 11:19 [PATCH v1 00/11] Plan to redesign code Muhammad Rizki
                   ` (9 preceding siblings ...)
  2022-07-18 11:20 ` [PATCH v1 10/11] Create fix_utf8_chars function Muhammad Rizki
@ 2022-07-18 11:20 ` Muhammad Rizki
  2022-07-18 11:33   ` Ammar Faizi
  2022-07-18 11:28 ` [PATCH v1 00/11] Plan to redesign code Ammar Faizi
  2022-07-18 11:34 ` Ammar Faizi
  12 siblings, 1 reply; 20+ messages in thread
From: Muhammad Rizki @ 2022-07-18 11:20 UTC (permalink / raw)
  To: Ammar Faizi; +Cc: Muhammad Rizki, GNU/Weeb Mailing List

Just recode some codes, I think it much practice and clean (?)

Signed-off-by: Muhammad Rizki <[email protected]>
---
 daemon/scraper/utils.py | 12 +++++-------
 1 file changed, 5 insertions(+), 7 deletions(-)

diff --git a/daemon/scraper/utils.py b/daemon/scraper/utils.py
index 3f0d254..5496483 100644
--- a/daemon/scraper/utils.py
+++ b/daemon/scraper/utils.py
@@ -182,9 +182,8 @@ def create_template(thread: Message, to=None, cc=None):
 		if len(ret) >= 4000:
 			ret = ret[:4000] + "..."
 
-		ret = (
-			fix_utf8_char(ret)
-		) + "\n<code>------------------------------------------------------------------------</code>"
+		ret = fix_utf8_char(ret)
+		ret += "\n<code>------------------------------------------------------------------------</code>"
 
 	return ret, files, is_patch
 
@@ -208,10 +207,9 @@ def prepare_send_patch(mail, text, url):
 	with open(file, "wb") as f:
 		f.write(bytes(text, encoding="utf8"))
 
-	caption = (
-		"#patch #ml\n" +
-		fix_utf8_char(cap)
-	)
+	caption = "#patch #ml\n"
+	caption += fix_utf8_char(cap)
+
 	return tmp, file, caption, url
 
 
-- 
Muhammad Rizki


^ permalink raw reply related	[flat|nested] 20+ messages in thread

* Re: [PATCH v1 06/11] Add traceback to get the error detail
  2022-07-18 11:20 ` [PATCH v1 06/11] Add traceback to get the error detail Muhammad Rizki
@ 2022-07-18 11:27   ` Ammar Faizi
  2022-07-18 12:24     ` Kiizuha
  0 siblings, 1 reply; 20+ messages in thread
From: Ammar Faizi @ 2022-07-18 11:27 UTC (permalink / raw)
  To: Muhammad Rizki; +Cc: GNU/Weeb Mailing List

On 7/18/22 6:20 PM, Muhammad Rizki wrote:
> Signed-off-by: Muhammad Rizki <[email protected]>
> ---
>   daemon/scraper/bot.py | 5 +++--
>   1 file changed, 3 insertions(+), 2 deletions(-)

Please give the reason of why do you make changes in this patch.

That patch is not trivial as it has a lengthy discussion background
story. It's something that needs an explanation. Not an empty body
like that.

A good commit message should explain "WHY" such a change is
necessary.

Maybe you can write it like this:

     daemon: Use traceback.format_exc() to get the error detail

     `print(f"[__run]: Error: {e}")` doesn't give enough information for
     debugging because it doesn't show the traceback, files, line numbers,
     etc. It only shows the error message raised by an exception. This is
     severely lacking and very bad for debugging experience.

     Use `traceback.format_exc()` instead to get a better log because it
     shows a complete traceback of the error.

     Signed-off-by: Your Name <[email protected]>

Improve it yourself if needed.

-- 
Ammar Faizi


^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [PATCH v1 00/11] Plan to redesign code
  2022-07-18 11:19 [PATCH v1 00/11] Plan to redesign code Muhammad Rizki
                   ` (10 preceding siblings ...)
  2022-07-18 11:20 ` [PATCH v1 11/11] Recode some codes Muhammad Rizki
@ 2022-07-18 11:28 ` Ammar Faizi
  2022-07-18 11:34 ` Ammar Faizi
  12 siblings, 0 replies; 20+ messages in thread
From: Ammar Faizi @ 2022-07-18 11:28 UTC (permalink / raw)
  To: Muhammad Rizki; +Cc: GNU/Weeb Mailing List

On 7/18/22 6:19 PM, Muhammad Rizki wrote:
> There 10 patches in this series:

That's 11 patches :-)

-- 
Ammar Faizi

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [PATCH v1 03/11] Add default temporary directory
  2022-07-18 11:19 ` [PATCH v1 03/11] Add default temporary directory Muhammad Rizki
@ 2022-07-18 11:31   ` Ammar Faizi
  0 siblings, 0 replies; 20+ messages in thread
From: Ammar Faizi @ 2022-07-18 11:31 UTC (permalink / raw)
  To: Muhammad Rizki; +Cc: GNU/Weeb Mailing List

On 7/18/22 6:19 PM, Muhammad Rizki wrote:
> Signed-off-by: Muhammad Rizki <[email protected]>
> ---
>   daemon/scraper/utils.py | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)


The same thing, missing a reason. The reason is simple, we want to
avoid error when "STORAGE_DIR" env is not defined, so you put a
default value here.

-- 
Ammar Faizi

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [PATCH v1 11/11] Recode some codes
  2022-07-18 11:20 ` [PATCH v1 11/11] Recode some codes Muhammad Rizki
@ 2022-07-18 11:33   ` Ammar Faizi
  0 siblings, 0 replies; 20+ messages in thread
From: Ammar Faizi @ 2022-07-18 11:33 UTC (permalink / raw)
  To: Muhammad Rizki; +Cc: GNU/Weeb Mailing List

On 7/18/22 6:20 PM, Muhammad Rizki wrote:
> Just recode some codes, I think it much practice and clean (?)

Yes.

-- 
Ammar Faizi

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [PATCH v1 00/11] Plan to redesign code
  2022-07-18 11:19 [PATCH v1 00/11] Plan to redesign code Muhammad Rizki
                   ` (11 preceding siblings ...)
  2022-07-18 11:28 ` [PATCH v1 00/11] Plan to redesign code Ammar Faizi
@ 2022-07-18 11:34 ` Ammar Faizi
  12 siblings, 0 replies; 20+ messages in thread
From: Ammar Faizi @ 2022-07-18 11:34 UTC (permalink / raw)
  To: Muhammad Rizki; +Cc: GNU/Weeb Mailing List

On 7/18/22 6:19 PM, Muhammad Rizki wrote:
> Hi sir,
> 
> In this series I want to redesign codes to make it clear and clean.
> I've inherit the DaemonClient with the Pyrogram's Client, so with
> this method we can create our custom functions, such as
> send email message and patch message in the DaemonClient.

Will take a look into this further, later...

-- 
Ammar Faizi

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [PATCH v1 06/11] Add traceback to get the error detail
  2022-07-18 11:27   ` Ammar Faizi
@ 2022-07-18 12:24     ` Kiizuha
  0 siblings, 0 replies; 20+ messages in thread
From: Kiizuha @ 2022-07-18 12:24 UTC (permalink / raw)
  To: Ammar Faizi; +Cc: GNU/Weeb Mailing List

>
> Please give the reason of why do you make changes in this patch.
>
> That patch is not trivial as it has a lengthy discussion background
> story. It's something that needs an explanation. Not an empty body
> like that.
>
> A good commit message should explain "WHY" such a change is
> necessary.
>
> Maybe you can write it like this:
>
>      daemon: Use traceback.format_exc() to get the error detail
>
>      `print(f"[__run]: Error: {e}")` doesn't give enough information for
>      debugging because it doesn't show the traceback, files, line numbers,
>      etc. It only shows the error message raised by an exception. This is
>      severely lacking and very bad for debugging experience.
>
>      Use `traceback.format_exc()` instead to get a better log because it
>      shows a complete traceback of the error.
>
>      Signed-off-by: Your Name <[email protected]>
>
> Improve it yourself if needed.
>

Ok, I will do it. I just forget to check the git log and it doesn't
have a description to explain WHY, I will improve any commit in the
future. Thanks for reminding me.

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [PATCH v1 01/11] Fix __send_patch_msg function parameter
  2022-07-18 11:19 ` [PATCH v1 01/11] Fix __send_patch_msg function parameter Muhammad Rizki
@ 2022-07-18 12:45   ` Ammar Faizi
  0 siblings, 0 replies; 20+ messages in thread
From: Ammar Faizi @ 2022-07-18 12:45 UTC (permalink / raw)
  To: Muhammad Rizki; +Cc: GNU/Weeb Mailing List

On 7/18/22 6:19 PM, Muhammad Rizki wrote:
> -	async def __send_patch_msg(self, mail, tg_chat_id, text, url):
> +	async def __send_patch_msg(self, mail, tg_chat_id, reply_to, text, url):
>   		print("[__send_patch_msg]")
>   		

^ There is a trailing whitespace here.

>   		tmp, fnm, caption, url = Bot.prepare_send_patch(mail, text, url)

-- 
Ammar Faizi

^ permalink raw reply	[flat|nested] 20+ messages in thread

* Re: [PATCH v1 07/11] Re-design send email message to Telegram
  2022-07-18 11:20 ` [PATCH v1 07/11] Re-design send email message to Telegram Muhammad Rizki
@ 2022-07-18 12:49   ` Ammar Faizi
  0 siblings, 0 replies; 20+ messages in thread
From: Ammar Faizi @ 2022-07-18 12:49 UTC (permalink / raw)
  To: Muhammad Rizki; +Cc: GNU/Weeb Mailing List

On 7/18/22 6:20 PM, Muhammad Rizki wrote:
> +class DaemonClient(Client):
> +        def __init__(self, name: str, api_id: int,
> +                     api_hash: str, **kwargs):
> +                super().__init__(name, api_id,
> +                                api_hash, **kwargs)
> +
> +
> +        async def send_text_email(
> +                self,
> +                chat_id: Union[int, str],
> +                text: str,
> +                reply_to: int,
> +                url: str = None,
> +                parse_mode: ParseMode = ParseMode.HTML
> +        ) -> Message:
> +                print("[send_text_email]")
> +                return await self.send_message(
> +			chat_id=chat_id,
> +			text=text,
> +			reply_to_message_id=reply_to,
> +			parse_mode=parse_mode,
> +			reply_markup=InlineKeyboardMarkup([
> +				[InlineKeyboardButton(
> +					"See the full message",
> +					url=url
> +				)]
> +			])
> +		)
[...]
> +                print("[send_patch_email]")
> +                return await self.send_document(
> +			chat_id=chat_id,
> +                        document=doc,
> +			caption=caption,
> +			reply_to_message_id=reply_to,
> +			parse_mode=parse_mode,
> +			reply_markup=InlineKeyboardMarkup([

Please always use tabs to start the indentation.

-- 
Ammar Faizi

^ permalink raw reply	[flat|nested] 20+ messages in thread

end of thread, other threads:[~2022-07-18 12:49 UTC | newest]

Thread overview: 20+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2022-07-18 11:19 [PATCH v1 00/11] Plan to redesign code Muhammad Rizki
2022-07-18 11:19 ` [PATCH v1 01/11] Fix __send_patch_msg function parameter Muhammad Rizki
2022-07-18 12:45   ` Ammar Faizi
2022-07-18 11:19 ` [PATCH v1 02/11] Fix import problem Muhammad Rizki
2022-07-18 11:19 ` [PATCH v1 03/11] Add default temporary directory Muhammad Rizki
2022-07-18 11:31   ` Ammar Faizi
2022-07-18 11:20 ` [PATCH v1 04/11] Move the Telegram bot session into the storage directory Muhammad Rizki
2022-07-18 11:20 ` [PATCH v1 05/11] daemon: Fix raw lore URL on the inline keyboard button Muhammad Rizki
2022-07-18 11:20 ` [PATCH v1 06/11] Add traceback to get the error detail Muhammad Rizki
2022-07-18 11:27   ` Ammar Faizi
2022-07-18 12:24     ` Kiizuha
2022-07-18 11:20 ` [PATCH v1 07/11] Re-design send email message to Telegram Muhammad Rizki
2022-07-18 12:49   ` Ammar Faizi
2022-07-18 11:20 ` [PATCH v1 08/11] Move ____send_patch_msg Muhammad Rizki
2022-07-18 11:20 ` [PATCH v1 09/11] Move prepare for patch and clean up patch functions Muhammad Rizki
2022-07-18 11:20 ` [PATCH v1 10/11] Create fix_utf8_chars function Muhammad Rizki
2022-07-18 11:20 ` [PATCH v1 11/11] Recode some codes Muhammad Rizki
2022-07-18 11:33   ` Ammar Faizi
2022-07-18 11:28 ` [PATCH v1 00/11] Plan to redesign code Ammar Faizi
2022-07-18 11:34 ` Ammar Faizi

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox