* [RFC PATCH v1 0/5] Refactor some Telegram bot source code
@ 2022-09-06 11:19 Muhammad Rizki
2022-09-06 11:19 ` [RFC PATCH v1 1/5] [telegram] Move the " Muhammad Rizki
` (4 more replies)
0 siblings, 5 replies; 11+ messages in thread
From: Muhammad Rizki @ 2022-09-06 11:19 UTC (permalink / raw)
To: Ammar Faizi
Cc: Muhammad Rizki, GNU/Weeb Mailing List, Alviro Iskandar Setiawan
Afternoon sir,
I just want to ask if this series okay to review or not?
These series is still undone, just to make sure that this series is
okay to review so that I can continue to the Discord bot.
There are 5 patches in this series:
- Patch 1 is to move Telegram bot SC, bcz there will a Discord bot
- Patch 2 is to refactor bot database to split between insert, del, get
- Patch 3 is to rename some functions to make it understandable
- Patch 4 is jsut remove some unecessary files
- Patch 5 is to refactor many files like atoms, scraper, etc
Patch 5 contains:
- Rename some functions in utils
- Rename file name in telegram such as scraper => mailer
- Move class Mutexes to utility file
- Rename the Mutexes attribute send_to_tg => lock
- Changes affected codes during this refactor
How to use:
1. Execute db.sql in daemon directory
2. Set value for environments for telegram bot, you can see the example
in the telegram.env.example, this env file name must telegram.env
3. Set config.py in the telegram/config.py
4. Install all requirements package by executing
`python3 -m pip install requirements.txt` in the telegram directory
5. Run the telegram bot by executing `python3 tg.py` in daemon directory
Tested and works fine, thanks.
Muhammad Rizki (5):
[telegram] Move the Telegram bot source code
[telegram] Refactor Telegram bot database method
[telegram] Renaming some functions in scraper/bot.py
[telegram] Remove unecessary files
Refactor many files
.gitignore | 4 +-
daemon/atom/__init__.py | 7 +
daemon/{scraper => atom}/scraper.py | 12 +-
daemon/{scraper => atom}/utils.py | 87 +++++--
daemon/db.sql | 18 +-
daemon/scraper/db.py | 217 ------------------
daemon/{.env.example => telegram.env.example} | 0
daemon/{ => telegram}/config.py.example | 0
daemon/telegram/database/__init__.py | 7 +
daemon/telegram/database/core.py | 20 ++
daemon/telegram/database/methods/__init__.py | 17 ++
.../database/methods/deletion/__init__.py | 14 ++
.../database/methods/deletion/delet_atom.py | 15 ++
.../methods/deletion/delete_broadcast.py | 15 ++
.../database/methods/getter/__init__.py | 18 ++
.../database/methods/getter/get_atom_urls.py | 21 ++
.../methods/getter/get_broadcast_chats.py | 21 ++
.../database/methods/getter/get_email_id.py | 62 +++++
.../methods/getter/get_telegram_reply.py | 33 +++
.../database/methods/insertion/__init__.py | 18 ++
.../database/methods/insertion/insert_atom.py | 27 +++
.../methods/insertion/insert_broadcast.py | 56 +++++
.../methods/insertion/insert_email.py | 27 +++
.../methods/insertion/insert_telegram.py | 21 ++
.../{scraper => telegram/mailer}/__init__.py | 4 +-
.../bot.py => telegram/mailer/listener.py} | 48 ++--
daemon/{ => telegram}/packages/__init__.py | 0
daemon/{ => telegram}/packages/client.py | 12 +-
daemon/{ => telegram}/packages/decorator.py | 0
.../packages/plugins/callbacks/del_atom.py | 6 +-
.../packages/plugins/callbacks/del_chat.py | 6 +-
.../packages/plugins/commands/debugger.py | 2 +-
.../packages/plugins/commands/manage_atom.py | 8 +-
.../plugins/commands/manage_broadcast.py | 8 +-
.../packages/plugins/commands/scrape.py | 10 +-
daemon/{ => telegram}/requirements.txt | 0
daemon/{ => telegram}/storage/.gitignore | 0
daemon/{run.py => tg.py} | 24 +-
38 files changed, 541 insertions(+), 324 deletions(-)
create mode 100644 daemon/atom/__init__.py
rename daemon/{scraper => atom}/scraper.py (79%)
rename daemon/{scraper => atom}/utils.py (72%)
delete mode 100644 daemon/scraper/db.py
rename daemon/{.env.example => telegram.env.example} (100%)
rename daemon/{ => telegram}/config.py.example (100%)
create mode 100644 daemon/telegram/database/__init__.py
create mode 100644 daemon/telegram/database/core.py
create mode 100644 daemon/telegram/database/methods/__init__.py
create mode 100644 daemon/telegram/database/methods/deletion/__init__.py
create mode 100644 daemon/telegram/database/methods/deletion/delet_atom.py
create mode 100644 daemon/telegram/database/methods/deletion/delete_broadcast.py
create mode 100644 daemon/telegram/database/methods/getter/__init__.py
create mode 100644 daemon/telegram/database/methods/getter/get_atom_urls.py
create mode 100644 daemon/telegram/database/methods/getter/get_broadcast_chats.py
create mode 100644 daemon/telegram/database/methods/getter/get_email_id.py
create mode 100644 daemon/telegram/database/methods/getter/get_telegram_reply.py
create mode 100644 daemon/telegram/database/methods/insertion/__init__.py
create mode 100644 daemon/telegram/database/methods/insertion/insert_atom.py
create mode 100644 daemon/telegram/database/methods/insertion/insert_broadcast.py
create mode 100644 daemon/telegram/database/methods/insertion/insert_email.py
create mode 100644 daemon/telegram/database/methods/insertion/insert_telegram.py
rename daemon/{scraper => telegram/mailer}/__init__.py (68%)
rename daemon/{scraper/bot.py => telegram/mailer/listener.py} (69%)
rename daemon/{ => telegram}/packages/__init__.py (100%)
rename daemon/{ => telegram}/packages/client.py (88%)
rename daemon/{ => telegram}/packages/decorator.py (100%)
rename daemon/{ => telegram}/packages/plugins/callbacks/del_atom.py (88%)
rename daemon/{ => telegram}/packages/plugins/callbacks/del_chat.py (90%)
rename daemon/{ => telegram}/packages/plugins/commands/debugger.py (97%)
rename daemon/{ => telegram}/packages/plugins/commands/manage_atom.py (91%)
rename daemon/{ => telegram}/packages/plugins/commands/manage_broadcast.py (92%)
rename daemon/{ => telegram}/packages/plugins/commands/scrape.py (86%)
rename daemon/{ => telegram}/requirements.txt (100%)
rename daemon/{ => telegram}/storage/.gitignore (100%)
rename daemon/{run.py => tg.py} (68%)
base-commit: 2582d7e5225d47a01f606808fc71e5e6aa7cb153
--
Muhammad Rizki
^ permalink raw reply [flat|nested] 11+ messages in thread
* [RFC PATCH v1 1/5] [telegram] Move the Telegram bot source code
2022-09-06 11:19 [RFC PATCH v1 0/5] Refactor some Telegram bot source code Muhammad Rizki
@ 2022-09-06 11:19 ` Muhammad Rizki
2022-09-06 11:19 ` [RFC PATCH v1 2/5] [telegram] Refactor Telegram bot database method Muhammad Rizki
` (3 subsequent siblings)
4 siblings, 0 replies; 11+ messages in thread
From: Muhammad Rizki @ 2022-09-06 11:19 UTC (permalink / raw)
To: Ammar Faizi
Cc: Muhammad Rizki, GNU/Weeb Mailing List, Alviro Iskandar Setiawan
Move them into their own directory because there is another platform bot
in the future, so it's better to split them.
Signed-off-by: Muhammad Rizki <[email protected]>
---
.gitignore | 3 +-
daemon/{ => telegram}/.env.example | 0
daemon/{ => telegram}/config.py.example | 0
daemon/{ => telegram}/db.sql | 0
daemon/{ => telegram}/packages/__init__.py | 0
daemon/{ => telegram}/packages/client.py | 0
daemon/{ => telegram}/packages/decorator.py | 0
daemon/telegram/packages/plugins/admin.py | 42 +++++++++
.../packages/plugins/callbacks/del_atom.py | 0
.../packages/plugins/callbacks/del_chat.py | 0
.../packages/plugins/commands/debugger.py | 0
.../packages/plugins/commands/manage_atom.py | 0
.../plugins/commands/manage_broadcast.py | 0
.../packages/plugins/commands/scrape.py | 0
daemon/telegram/packages/plugins/scrape.py | 86 +++++++++++++++++++
daemon/{ => telegram}/requirements.txt | 0
daemon/{ => telegram}/run.py | 0
daemon/{ => telegram}/scraper/__init__.py | 0
daemon/{ => telegram}/scraper/bot.py | 0
daemon/{ => telegram}/scraper/db.py | 0
daemon/{ => telegram}/scraper/scraper.py | 0
daemon/{ => telegram}/scraper/utils.py | 0
daemon/{ => telegram}/storage/.gitignore | 0
23 files changed, 130 insertions(+), 1 deletion(-)
rename daemon/{ => telegram}/.env.example (100%)
rename daemon/{ => telegram}/config.py.example (100%)
rename daemon/{ => telegram}/db.sql (100%)
rename daemon/{ => telegram}/packages/__init__.py (100%)
rename daemon/{ => telegram}/packages/client.py (100%)
rename daemon/{ => telegram}/packages/decorator.py (100%)
create mode 100644 daemon/telegram/packages/plugins/admin.py
rename daemon/{ => telegram}/packages/plugins/callbacks/del_atom.py (100%)
rename daemon/{ => telegram}/packages/plugins/callbacks/del_chat.py (100%)
rename daemon/{ => telegram}/packages/plugins/commands/debugger.py (100%)
rename daemon/{ => telegram}/packages/plugins/commands/manage_atom.py (100%)
rename daemon/{ => telegram}/packages/plugins/commands/manage_broadcast.py (100%)
rename daemon/{ => telegram}/packages/plugins/commands/scrape.py (100%)
create mode 100644 daemon/telegram/packages/plugins/scrape.py
rename daemon/{ => telegram}/requirements.txt (100%)
rename daemon/{ => telegram}/run.py (100%)
rename daemon/{ => telegram}/scraper/__init__.py (100%)
rename daemon/{ => telegram}/scraper/bot.py (100%)
rename daemon/{ => telegram}/scraper/db.py (100%)
rename daemon/{ => telegram}/scraper/scraper.py (100%)
rename daemon/{ => telegram}/scraper/utils.py (100%)
rename daemon/{ => telegram}/storage/.gitignore (100%)
diff --git a/.gitignore b/.gitignore
index 4de282d..4201a17 100644
--- a/.gitignore
+++ b/.gitignore
@@ -140,4 +140,5 @@ data.json
*.patch
# configuration file
-daemon/config.py
+daemon/telegram/config.py
+daemon/discord/config.py
diff --git a/daemon/.env.example b/daemon/telegram/.env.example
similarity index 100%
rename from daemon/.env.example
rename to daemon/telegram/.env.example
diff --git a/daemon/config.py.example b/daemon/telegram/config.py.example
similarity index 100%
rename from daemon/config.py.example
rename to daemon/telegram/config.py.example
diff --git a/daemon/db.sql b/daemon/telegram/db.sql
similarity index 100%
rename from daemon/db.sql
rename to daemon/telegram/db.sql
diff --git a/daemon/packages/__init__.py b/daemon/telegram/packages/__init__.py
similarity index 100%
rename from daemon/packages/__init__.py
rename to daemon/telegram/packages/__init__.py
diff --git a/daemon/packages/client.py b/daemon/telegram/packages/client.py
similarity index 100%
rename from daemon/packages/client.py
rename to daemon/telegram/packages/client.py
diff --git a/daemon/packages/decorator.py b/daemon/telegram/packages/decorator.py
similarity index 100%
rename from daemon/packages/decorator.py
rename to daemon/telegram/packages/decorator.py
diff --git a/daemon/telegram/packages/plugins/admin.py b/daemon/telegram/packages/plugins/admin.py
new file mode 100644
index 0000000..e0f145e
--- /dev/null
+++ b/daemon/telegram/packages/plugins/admin.py
@@ -0,0 +1,42 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Copyright (C) 2022 Muhammad Rizki <[email protected]>
+#
+
+from pyrogram import Client, filters, enums
+from pyrogram.types import Message
+from textwrap import indent
+import io, import_expression, contextlib, traceback
+
[email protected]_message(
+ filters.command(['d','debug']) &
+ filters.user(["nekoha", "kiizuah"])
+)
+async def execute_v2(c: Client, m: Message):
+ sep = m.text.split('\n')
+ body = m.text.replace(sep[0] + '\n','')
+
+ env = {"bot": c}
+ env.update(globals())
+
+ stdout = io.StringIO()
+ to_compile = f'async def func(_, m):\n{indent(body, " ")}'
+
+ try:
+ import_expression.exec(to_compile, env)
+ except Exception as e:
+ text = f"```{e.__class__.__name__}: {e}"[0:4096]+"```"
+
+ func = env["func"]
+
+ try:
+ with contextlib.redirect_stdout(stdout):
+ await func(c, m)
+ except Exception:
+ value = stdout.getvalue()
+ text = f"```{value}{traceback.format_exc()}"[0:4096]+"```"
+ else:
+ value = stdout.getvalue()
+ text = f"```{value}"[0:4096]+"```"
+
+ await c.send_message(m.chat.id, text, parse_mode=enums.ParseMode.MARKDOWN)
diff --git a/daemon/packages/plugins/callbacks/del_atom.py b/daemon/telegram/packages/plugins/callbacks/del_atom.py
similarity index 100%
rename from daemon/packages/plugins/callbacks/del_atom.py
rename to daemon/telegram/packages/plugins/callbacks/del_atom.py
diff --git a/daemon/packages/plugins/callbacks/del_chat.py b/daemon/telegram/packages/plugins/callbacks/del_chat.py
similarity index 100%
rename from daemon/packages/plugins/callbacks/del_chat.py
rename to daemon/telegram/packages/plugins/callbacks/del_chat.py
diff --git a/daemon/packages/plugins/commands/debugger.py b/daemon/telegram/packages/plugins/commands/debugger.py
similarity index 100%
rename from daemon/packages/plugins/commands/debugger.py
rename to daemon/telegram/packages/plugins/commands/debugger.py
diff --git a/daemon/packages/plugins/commands/manage_atom.py b/daemon/telegram/packages/plugins/commands/manage_atom.py
similarity index 100%
rename from daemon/packages/plugins/commands/manage_atom.py
rename to daemon/telegram/packages/plugins/commands/manage_atom.py
diff --git a/daemon/packages/plugins/commands/manage_broadcast.py b/daemon/telegram/packages/plugins/commands/manage_broadcast.py
similarity index 100%
rename from daemon/packages/plugins/commands/manage_broadcast.py
rename to daemon/telegram/packages/plugins/commands/manage_broadcast.py
diff --git a/daemon/packages/plugins/commands/scrape.py b/daemon/telegram/packages/plugins/commands/scrape.py
similarity index 100%
rename from daemon/packages/plugins/commands/scrape.py
rename to daemon/telegram/packages/plugins/commands/scrape.py
diff --git a/daemon/telegram/packages/plugins/scrape.py b/daemon/telegram/packages/plugins/scrape.py
new file mode 100644
index 0000000..1698c6d
--- /dev/null
+++ b/daemon/telegram/packages/plugins/scrape.py
@@ -0,0 +1,86 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Copyright (C) 2022 Muhammad Rizki <[email protected]>
+# Copyright (C) 2022 Ammar Faizi <[email protected]>
+#
+
+from pyrogram.types import InlineKeyboardMarkup
+from pyrogram.types import InlineKeyboardButton
+from pyrogram.types import Message
+from pyrogram import filters
+from pyrogram import Client
+from scraper import Scraper
+from pyrogram import enums
+from scraper import utils
+from scraper import Bot
+import shutil
+import re
+import asyncio
+
+
+#
+# This allows user to invoke the following commands:
+# /lore https://lore.kernel.org/path/message_id/raw
+# !lore https://lore.kernel.org/path/message_id/raw
+# .lore https://lore.kernel.org/path/message_id/raw
+#
+LORE_CMD_URL_PATTERN = r"^(?:\/|\.|\!)lore\s+(https?:\/\/lore\.kernel\.org\/\S+)"
[email protected]_message(
+ filters.regex(LORE_CMD_URL_PATTERN) &
+ filters.chat(["kiizuah", "nekoha", -1001673279485])
+)
+async def scrap_email(_, m: Message):
+ p = re.search(LORE_CMD_URL_PATTERN, m.text)
+ if not p:
+ return
+
+ url = p.group(1)
+ if not url:
+ return
+
+ s = Scraper()
+ mail = await s.get_email_from_url(url)
+ text, files, is_patch = utils.create_template(mail)
+
+ if is_patch:
+ m = await __send_patch_msg(m, mail, text, url)
+ else:
+ text = "#ml\n" + text
+ m = await __send_text_msg(m, text, url)
+
+ for d, f in files:
+ await m.reply_document(f"{d}/{f}", file_name=f)
+ await asyncio.sleep(1)
+
+ if files:
+ shutil.rmtree(str(files[0][0]))
+
+
+async def __send_patch_msg(m, mail, text, url):
+ tmp, fnm, caption, url = Bot.prepare_send_patch(mail, text, url)
+ ret = await m.reply_document(
+ fnm,
+ caption=caption,
+ parse_mode=enums.ParseMode.HTML,
+ reply_markup=InlineKeyboardMarkup([
+ [InlineKeyboardButton(
+ "See the full message",
+ url=url
+ )]
+ ])
+ )
+ Bot.clean_up_after_send_patch(tmp)
+ return ret
+
+
+async def __send_text_msg(m, text, url):
+ return await m.reply(
+ text,
+ parse_mode=enums.ParseMode.HTML,
+ reply_markup=InlineKeyboardMarkup([
+ [InlineKeyboardButton(
+ "See the full message",
+ url=url.replace("/raw","")
+ )]
+ ])
+ )
diff --git a/daemon/requirements.txt b/daemon/telegram/requirements.txt
similarity index 100%
rename from daemon/requirements.txt
rename to daemon/telegram/requirements.txt
diff --git a/daemon/run.py b/daemon/telegram/run.py
similarity index 100%
rename from daemon/run.py
rename to daemon/telegram/run.py
diff --git a/daemon/scraper/__init__.py b/daemon/telegram/scraper/__init__.py
similarity index 100%
rename from daemon/scraper/__init__.py
rename to daemon/telegram/scraper/__init__.py
diff --git a/daemon/scraper/bot.py b/daemon/telegram/scraper/bot.py
similarity index 100%
rename from daemon/scraper/bot.py
rename to daemon/telegram/scraper/bot.py
diff --git a/daemon/scraper/db.py b/daemon/telegram/scraper/db.py
similarity index 100%
rename from daemon/scraper/db.py
rename to daemon/telegram/scraper/db.py
diff --git a/daemon/scraper/scraper.py b/daemon/telegram/scraper/scraper.py
similarity index 100%
rename from daemon/scraper/scraper.py
rename to daemon/telegram/scraper/scraper.py
diff --git a/daemon/scraper/utils.py b/daemon/telegram/scraper/utils.py
similarity index 100%
rename from daemon/scraper/utils.py
rename to daemon/telegram/scraper/utils.py
diff --git a/daemon/storage/.gitignore b/daemon/telegram/storage/.gitignore
similarity index 100%
rename from daemon/storage/.gitignore
rename to daemon/telegram/storage/.gitignore
--
Muhammad Rizki
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [RFC PATCH v1 2/5] [telegram] Refactor Telegram bot database method
2022-09-06 11:19 [RFC PATCH v1 0/5] Refactor some Telegram bot source code Muhammad Rizki
2022-09-06 11:19 ` [RFC PATCH v1 1/5] [telegram] Move the " Muhammad Rizki
@ 2022-09-06 11:19 ` Muhammad Rizki
2022-09-06 11:19 ` [RFC PATCH v1 3/5] [telegram] Renaming some functions in scraper/bot.py Muhammad Rizki
` (2 subsequent siblings)
4 siblings, 0 replies; 11+ messages in thread
From: Muhammad Rizki @ 2022-09-06 11:19 UTC (permalink / raw)
To: Ammar Faizi
Cc: Muhammad Rizki, GNU/Weeb Mailing List, Alviro Iskandar Setiawan
I decide to refactor the Telegram bot database to make it more clean and
manageable to maintain.
Signed-off-by: Muhammad Rizki <[email protected]>
---
daemon/telegram/database/__init__.py | 7 +
daemon/telegram/database/core.py | 20 ++
daemon/telegram/database/methods/__init__.py | 17 ++
.../database/methods/deletion/__init__.py | 14 ++
.../database/methods/deletion/delet_atom.py | 15 ++
.../methods/deletion/delete_broadcast.py | 15 ++
.../database/methods/getter/__init__.py | 18 ++
.../database/methods/getter/get_atom_urls.py | 21 ++
.../methods/getter/get_broadcast_chats.py | 21 ++
.../database/methods/getter/get_email_id.py | 62 +++++
.../methods/getter/get_telegram_reply.py | 33 +++
.../database/methods/insertion/__init__.py | 18 ++
.../database/methods/insertion/insert_atom.py | 27 +++
.../methods/insertion/insert_broadcast.py | 56 +++++
.../methods/insertion/insert_email.py | 27 +++
.../methods/insertion/insert_telegram.py | 21 ++
daemon/telegram/db.sql | 18 +-
daemon/telegram/packages/client.py | 4 +-
.../packages/plugins/commands/manage_atom.py | 2 +-
.../plugins/commands/manage_broadcast.py | 2 +-
daemon/telegram/scraper/bot.py | 8 +-
daemon/telegram/scraper/db.py | 217 ------------------
22 files changed, 409 insertions(+), 234 deletions(-)
create mode 100644 daemon/telegram/database/__init__.py
create mode 100644 daemon/telegram/database/core.py
create mode 100644 daemon/telegram/database/methods/__init__.py
create mode 100644 daemon/telegram/database/methods/deletion/__init__.py
create mode 100644 daemon/telegram/database/methods/deletion/delet_atom.py
create mode 100644 daemon/telegram/database/methods/deletion/delete_broadcast.py
create mode 100644 daemon/telegram/database/methods/getter/__init__.py
create mode 100644 daemon/telegram/database/methods/getter/get_atom_urls.py
create mode 100644 daemon/telegram/database/methods/getter/get_broadcast_chats.py
create mode 100644 daemon/telegram/database/methods/getter/get_email_id.py
create mode 100644 daemon/telegram/database/methods/getter/get_telegram_reply.py
create mode 100644 daemon/telegram/database/methods/insertion/__init__.py
create mode 100644 daemon/telegram/database/methods/insertion/insert_atom.py
create mode 100644 daemon/telegram/database/methods/insertion/insert_broadcast.py
create mode 100644 daemon/telegram/database/methods/insertion/insert_email.py
create mode 100644 daemon/telegram/database/methods/insertion/insert_telegram.py
delete mode 100644 daemon/telegram/scraper/db.py
diff --git a/daemon/telegram/database/__init__.py b/daemon/telegram/database/__init__.py
new file mode 100644
index 0000000..930e3d9
--- /dev/null
+++ b/daemon/telegram/database/__init__.py
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Copyright (C) 2022 Muhammad Rizki <[email protected]>
+#
+
+
+from .core import DB
diff --git a/daemon/telegram/database/core.py b/daemon/telegram/database/core.py
new file mode 100644
index 0000000..c34d7a8
--- /dev/null
+++ b/daemon/telegram/database/core.py
@@ -0,0 +1,20 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Copyright (C) 2022 Muhammad Rizki <[email protected]>
+# Copyright (C) 2022 Ammar Faizi <[email protected]>
+#
+
+
+from .methods import DBMethods
+
+
+class DB(DBMethods):
+ def __init__(self, conn):
+ self.conn = conn
+ self.conn.autocommit = True
+ self.cur = self.conn.cursor(buffered=True)
+
+
+ def __del__(self):
+ self.cur.close()
+ self.conn.close()
diff --git a/daemon/telegram/database/methods/__init__.py b/daemon/telegram/database/methods/__init__.py
new file mode 100644
index 0000000..961b4e0
--- /dev/null
+++ b/daemon/telegram/database/methods/__init__.py
@@ -0,0 +1,17 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Copyright (C) 2022 Muhammad Rizki <[email protected]>
+# Copyright (C) 2022 Ammar Faizi <[email protected]>
+#
+
+
+from .deletion import Deletion
+from .getter import Getter
+from .insertion import Insertion
+
+
+class DBMethods(
+ Deletion,
+ Getter,
+ Insertion
+): pass
diff --git a/daemon/telegram/database/methods/deletion/__init__.py b/daemon/telegram/database/methods/deletion/__init__.py
new file mode 100644
index 0000000..b206929
--- /dev/null
+++ b/daemon/telegram/database/methods/deletion/__init__.py
@@ -0,0 +1,14 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Copyright (C) 2022 Muhammad Rizki <[email protected]>
+#
+
+
+from .delet_atom import DeleteAtom
+from .delete_broadcast import DeleteBroadcast
+
+
+class Deletion(
+ DeleteAtom,
+ DeleteBroadcast
+): pass
diff --git a/daemon/telegram/database/methods/deletion/delet_atom.py b/daemon/telegram/database/methods/deletion/delet_atom.py
new file mode 100644
index 0000000..d8ad4bf
--- /dev/null
+++ b/daemon/telegram/database/methods/deletion/delet_atom.py
@@ -0,0 +1,15 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Copyright (C) 2022 Muhammad Rizki <[email protected]>
+#
+
+
+class DeleteAtom:
+
+ def delete_atom(self, atom: str):
+ q = """
+ DELETE FROM tg_atoms
+ WHERE url = %(atom)s
+ """
+ self.cur.execute(q, {"atom": atom})
+ return self.cur.rowcount > 0
diff --git a/daemon/telegram/database/methods/deletion/delete_broadcast.py b/daemon/telegram/database/methods/deletion/delete_broadcast.py
new file mode 100644
index 0000000..d076dec
--- /dev/null
+++ b/daemon/telegram/database/methods/deletion/delete_broadcast.py
@@ -0,0 +1,15 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Copyright (C) 2022 Muhammad Rizki <[email protected]>
+#
+
+
+class DeleteBroadcast:
+
+ def delete_broadcast(self, chat_id: int):
+ q = """
+ DELETE FROM tg_broadcasts
+ WHERE chat_id = %(chat_id)s
+ """
+ self.cur.execute(q, {"chat_id": chat_id})
+ return self.cur.rowcount > 0
diff --git a/daemon/telegram/database/methods/getter/__init__.py b/daemon/telegram/database/methods/getter/__init__.py
new file mode 100644
index 0000000..e978c72
--- /dev/null
+++ b/daemon/telegram/database/methods/getter/__init__.py
@@ -0,0 +1,18 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Copyright (C) 2022 Muhammad Rizki <[email protected]>
+#
+
+
+from .get_atom_urls import GetAtomURL
+from .get_broadcast_chats import GetBroadcastChats
+from .get_email_id import GetEmailID
+from .get_telegram_reply import GetTelegramReply
+
+
+class Getter(
+ GetAtomURL,
+ GetBroadcastChats,
+ GetEmailID,
+ GetTelegramReply
+): pass
diff --git a/daemon/telegram/database/methods/getter/get_atom_urls.py b/daemon/telegram/database/methods/getter/get_atom_urls.py
new file mode 100644
index 0000000..04a9315
--- /dev/null
+++ b/daemon/telegram/database/methods/getter/get_atom_urls.py
@@ -0,0 +1,21 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Copyright (C) 2022 Muhammad Rizki <[email protected]>
+#
+
+
+class GetAtomURL:
+
+ def get_atom_urls(self):
+ '''
+ Get lore kernel raw email URLs.
+ - Return list of raw email URLs: `List[str]`
+ '''
+ q = """
+ SELECT tg_atoms.url
+ FROM tg_atoms
+ """
+ self.cur.execute(q)
+ urls = self.cur.fetchall()
+
+ return [u[0] for u in urls]
diff --git a/daemon/telegram/database/methods/getter/get_broadcast_chats.py b/daemon/telegram/database/methods/getter/get_broadcast_chats.py
new file mode 100644
index 0000000..d92e879
--- /dev/null
+++ b/daemon/telegram/database/methods/getter/get_broadcast_chats.py
@@ -0,0 +1,21 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Copyright (C) 2022 Muhammad Rizki <[email protected]>
+#
+
+
+class GetBroadcastChats:
+
+ def get_broadcast_chats(self):
+ '''
+ Get broadcast chats that are currently
+ listening for new email.
+ - Return list of chat object: `List[Object]`
+ '''
+ q = """
+ SELECT *
+ FROM tg_broadcasts
+ """
+ self.cur.execute(q)
+
+ return self.cur.fetchall()
diff --git a/daemon/telegram/database/methods/getter/get_email_id.py b/daemon/telegram/database/methods/getter/get_email_id.py
new file mode 100644
index 0000000..509aa99
--- /dev/null
+++ b/daemon/telegram/database/methods/getter/get_email_id.py
@@ -0,0 +1,62 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Copyright (C) 2022 Muhammad Rizki <[email protected]>
+# Copyright (C) 2022 Ammar Faizi <[email protected]>
+#
+
+
+class GetEmailID:
+
+ def get_email_id(self, email_id, chat_id):
+ '''
+ Determine whether the email needs to be sent to @tg_chat_id.
+ - Return an email id (PK) if it needs to be sent
+ - Return None if it doesn't need to be sent
+ '''
+ if self.__is_sent(email_id, chat_id):
+ return
+
+ res = self.__email_id(email_id)
+ if not bool(res):
+ return
+
+ return int(res[0])
+
+
+ def __is_sent(self, email_id, chat_id):
+ '''
+ Checking if this email has already been sent
+ or not.
+ - Return True if it's already been sent
+ '''
+
+ q = """
+ SELECT tg_emails.id, tg_mail_msg.id FROM tg_emails LEFT JOIN tg_mail_msg
+ ON tg_emails.id = tg_mail_msg.email_id
+ WHERE tg_emails.message_id = %(email_id)s
+ AND tg_mail_msg.chat_id = %(chat_id)s
+ LIMIT 1
+ """
+
+ self.cur.execute(q, {
+ "email_id": email_id,
+ "chat_id": chat_id
+ })
+
+ res = self.cur.fetchone()
+ return bool(res)
+
+
+ def __email_id(self, email_id):
+ '''
+ Get the email id if match with the email message_id.
+ - Return the result if it's match and exists
+ '''
+
+ q = """
+ SELECT id FROM tg_emails WHERE message_id = %(email_id)s
+ """
+
+ self.cur.execute(q, {"email_id": email_id})
+ res = self.cur.fetchone()
+ return res
\ No newline at end of file
diff --git a/daemon/telegram/database/methods/getter/get_telegram_reply.py b/daemon/telegram/database/methods/getter/get_telegram_reply.py
new file mode 100644
index 0000000..94e3138
--- /dev/null
+++ b/daemon/telegram/database/methods/getter/get_telegram_reply.py
@@ -0,0 +1,33 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Copyright (C) 2022 Muhammad Rizki <[email protected]>
+# Copyright (C) 2022 Ammar Faizi <[email protected]>
+#
+
+
+class GetTelegramReply:
+
+ def get_reply_id(self, email_msg_id, chat_id):
+ '''
+ Get Telegram message ID sent match with
+ email message ID and Telegram chat ID.
+ - Return Telegram message ID if exists: `int`
+ - Return None if not exists`
+ '''
+ q = """
+ SELECT tg_mail_msg.tg_msg_id
+ FROM tg_emails INNER JOIN tg_mail_msg
+ ON tg_emails.id = tg_mail_msg.email_id
+ WHERE tg_emails.message_id = %(email_msg_id)s
+ AND tg_mail_msg.chat_id = %(chat_id)s
+ """
+
+ self.cur.execute(q, {
+ "email_msg_id": email_msg_id,
+ "chat_id": chat_id
+ })
+ res = self.cur.fetchone()
+ if not bool(res):
+ return None
+
+ return res[0]
diff --git a/daemon/telegram/database/methods/insertion/__init__.py b/daemon/telegram/database/methods/insertion/__init__.py
new file mode 100644
index 0000000..3604f82
--- /dev/null
+++ b/daemon/telegram/database/methods/insertion/__init__.py
@@ -0,0 +1,18 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Copyright (C) 2022 Muhammad Rizki <[email protected]>
+# Copyright (C) 2022 Ammar Faizi <[email protected]>
+#
+
+from .insert_atom import InsertAtom
+from .insert_broadcast import InsertBroadcast
+from .insert_email import InsertEmail
+from .insert_telegram import InsertTelegram
+
+
+class Insertion(
+ InsertAtom,
+ InsertBroadcast,
+ InsertEmail,
+ InsertTelegram
+): pass
diff --git a/daemon/telegram/database/methods/insertion/insert_atom.py b/daemon/telegram/database/methods/insertion/insert_atom.py
new file mode 100644
index 0000000..ac068ae
--- /dev/null
+++ b/daemon/telegram/database/methods/insertion/insert_atom.py
@@ -0,0 +1,27 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Copyright (C) 2022 Muhammad Rizki <[email protected]>
+# Copyright (C) 2022 Ammar Faizi <[email protected]>
+#
+
+
+from mysql.connector import errors
+from datetime import datetime
+
+
+class InsertAtom:
+
+ def save_atom(self, atom: str):
+ try:
+ return self.__insert_atom(atom)
+ except errors.IntegrityError:
+ #
+ # Duplicate data, skip!
+ #
+ return None
+
+
+ def __insert_atom(self, atom: str):
+ q = "INSERT INTO tg_atoms (url, created_at) VALUES (%s, %s)"
+ self.cur.execute(q, (atom, datetime.utcnow()))
+ return self.cur.lastrowid
diff --git a/daemon/telegram/database/methods/insertion/insert_broadcast.py b/daemon/telegram/database/methods/insertion/insert_broadcast.py
new file mode 100644
index 0000000..11ebc6e
--- /dev/null
+++ b/daemon/telegram/database/methods/insertion/insert_broadcast.py
@@ -0,0 +1,56 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Copyright (C) 2022 Muhammad Rizki <[email protected]>
+# Copyright (C) 2022 Ammar Faizi <[email protected]>
+#
+
+
+from mysql.connector import errors
+from datetime import datetime
+
+
+class InsertBroadcast:
+
+ def save_broadcast(
+ self,
+ chat_id: int,
+ name: str,
+ type: str,
+ created_at: "datetime",
+ username: str = None,
+ link: str = None,
+ ):
+ try:
+ return self.__insert_broadcast(
+ chat_id=chat_id,
+ name=name,
+ type=type,
+ created_at=created_at,
+ username=username,
+ link=link
+ )
+ except errors.IntegrityError:
+ #
+ # Duplicate data, skip!
+ #
+ return None
+
+
+ def __insert_broadcast(
+ self,
+ chat_id: int,
+ name: str,
+ type: str,
+ created_at: "datetime",
+ username: str = None,
+ link: str = None,
+ ):
+ q = """
+ INSERT INTO tg_broadcasts
+ (chat_id, username, name, type, link, created_at)
+ VALUES
+ (%s, %s, %s, %s, %s, %s)
+ """
+ values = (chat_id, username, name, type, link, created_at)
+ self.cur.execute(q, values)
+ return self.cur.lastrowid
diff --git a/daemon/telegram/database/methods/insertion/insert_email.py b/daemon/telegram/database/methods/insertion/insert_email.py
new file mode 100644
index 0000000..adbe86b
--- /dev/null
+++ b/daemon/telegram/database/methods/insertion/insert_email.py
@@ -0,0 +1,27 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Copyright (C) 2022 Muhammad Rizki <[email protected]>
+# Copyright (C) 2022 Ammar Faizi <[email protected]>
+#
+
+
+from mysql.connector import errors
+from datetime import datetime
+
+
+class InsertEmail:
+
+ def save_email(self, email_msg_id):
+ try:
+ return self.__insert_email(email_msg_id)
+ except errors.IntegrityError:
+ #
+ # Duplicate data, skip!
+ #
+ return None
+
+
+ def __insert_email(self, email_msg_id):
+ q = "INSERT INTO tg_emails (message_id, created_at) VALUES (%s, %s)"
+ self.cur.execute(q, (email_msg_id, datetime.utcnow()))
+ return self.cur.lastrowid
diff --git a/daemon/telegram/database/methods/insertion/insert_telegram.py b/daemon/telegram/database/methods/insertion/insert_telegram.py
new file mode 100644
index 0000000..8e0615c
--- /dev/null
+++ b/daemon/telegram/database/methods/insertion/insert_telegram.py
@@ -0,0 +1,21 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Copyright (C) 2022 Muhammad Rizki <[email protected]>
+# Copyright (C) 2022 Ammar Faizi <[email protected]>
+#
+
+
+from datetime import datetime
+
+
+class InsertTelegram:
+
+ def save_telegram_mail(self, email_id, tg_chat_id, tg_msg_id):
+ q = """
+ INSERT INTO tg_mail_msg
+ (email_id, chat_id, tg_msg_id, created_at)
+ VALUES (%s, %s, %s, %s);
+ """
+ self.cur.execute(q, (email_id, tg_chat_id, tg_msg_id,
+ datetime.utcnow()))
+ return self.cur.lastrowid
diff --git a/daemon/telegram/db.sql b/daemon/telegram/db.sql
index ac6ed0d..c55c464 100644
--- a/daemon/telegram/db.sql
+++ b/daemon/telegram/db.sql
@@ -7,8 +7,8 @@ SET sql_mode = 'NO_AUTO_VALUE_ON_ZERO';
SET NAMES utf8mb4;
-DROP TABLE IF EXISTS `emails`;
-CREATE TABLE `emails` (
+DROP TABLE IF EXISTS `tg_emails`;
+CREATE TABLE `tg_emails` (
`id` bigint unsigned NOT NULL AUTO_INCREMENT,
`message_id` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_520_ci NOT NULL,
`created_at` datetime NOT NULL,
@@ -18,8 +18,8 @@ CREATE TABLE `emails` (
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_520_ci;
-DROP TABLE IF EXISTS `tg_emails`;
-CREATE TABLE `tg_emails` (
+DROP TABLE IF EXISTS `tg_mail_msg`;
+CREATE TABLE `tg_mail_msg` (
`id` bigint unsigned NOT NULL AUTO_INCREMENT,
`email_id` bigint unsigned NOT NULL,
`chat_id` bigint NOT NULL,
@@ -30,12 +30,12 @@ CREATE TABLE `tg_emails` (
KEY `chat_id` (`chat_id`),
KEY `tg_msg_id` (`tg_msg_id`),
KEY `created_at` (`created_at`),
- CONSTRAINT `tg_emails_ibfk_2` FOREIGN KEY (`email_id`) REFERENCES `emails` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
+ CONSTRAINT `tg_mail_msg_ibfk_2` FOREIGN KEY (`email_id`) REFERENCES `tg_emails` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_520_ci;
-DROP TABLE IF EXISTS `atom_urls`;
-CREATE TABLE `atom_urls` (
+DROP TABLE IF EXISTS `tg_atoms`;
+CREATE TABLE `tg_atoms` (
`id` bigint unsigned NOT NULL AUTO_INCREMENT,
`url` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_520_ci NOT NULL,
`created_at` datetime NOT NULL,
@@ -45,8 +45,8 @@ CREATE TABLE `atom_urls` (
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_520_ci;
-DROP TABLE IF EXISTS `broadcast_chats`;
-CREATE TABLE `broadcast_chats` (
+DROP TABLE IF EXISTS `tg_broadcasts`;
+CREATE TABLE `tg_broadcasts` (
`id` bigint unsigned NOT NULL AUTO_INCREMENT,
`chat_id` bigint NOT NULL,
`username` varchar(32),
diff --git a/daemon/telegram/packages/client.py b/daemon/telegram/packages/client.py
index 282daf6..820c3e2 100644
--- a/daemon/telegram/packages/client.py
+++ b/daemon/telegram/packages/client.py
@@ -9,7 +9,7 @@ from pyrogram.types import Message, InlineKeyboardMarkup, InlineKeyboardButton
from typing import Union
from email.message import Message
from scraper import utils
-from scraper.db import Db
+from database import DB
from .decorator import handle_flood
@@ -18,7 +18,7 @@ class DaemonClient(Client):
api_hash: str, conn, **kwargs):
super().__init__(name, api_id,
api_hash, **kwargs)
- self.db = Db(conn)
+ self.db = DB(conn)
@handle_flood
diff --git a/daemon/telegram/packages/plugins/commands/manage_atom.py b/daemon/telegram/packages/plugins/commands/manage_atom.py
index bcb2f35..4ba422a 100644
--- a/daemon/telegram/packages/plugins/commands/manage_atom.py
+++ b/daemon/telegram/packages/plugins/commands/manage_atom.py
@@ -25,7 +25,7 @@ async def add_atom_url(c: DaemonClient, m: Message):
if not is_atom:
return await m.reply("Invalid Atom URL")
- inserted = c.db.insert_atom(text)
+ inserted = c.db.save_atom(text)
if inserted is None:
return await m.reply(f"This URL already listened for new email.")
diff --git a/daemon/telegram/packages/plugins/commands/manage_broadcast.py b/daemon/telegram/packages/plugins/commands/manage_broadcast.py
index ffb5a6b..6d75c36 100644
--- a/daemon/telegram/packages/plugins/commands/manage_broadcast.py
+++ b/daemon/telegram/packages/plugins/commands/manage_broadcast.py
@@ -20,7 +20,7 @@ async def add_broadcast(c: DaemonClient, m: Message):
else:
chat_name = m.chat.title
- inserted = c.db.insert_broadcast(
+ inserted = c.db.save_broadcast(
chat_id=m.chat.id,
name=chat_name,
type=str(m.chat.type),
diff --git a/daemon/telegram/scraper/bot.py b/daemon/telegram/scraper/bot.py
index 7adfb12..9ef5508 100644
--- a/daemon/telegram/scraper/bot.py
+++ b/daemon/telegram/scraper/bot.py
@@ -112,7 +112,7 @@ class Bot():
tg_chat_id, text,reply_to, url
)
- self.db.insert_telegram(email_id, m.chat.id, m.id)
+ self.db.save_telegram_mail(email_id, m.chat.id, m.id)
for d, f in files:
await m.reply_document(f"{d}/{f}", file_name=f)
await asyncio.sleep(1)
@@ -124,11 +124,11 @@ class Bot():
def __need_to_send_to_telegram(self, email_msg_id, tg_chat_id):
- email_id = self.db.save_email_msg_id(email_msg_id)
+ email_id = self.db.save_email(email_msg_id)
if email_id:
return email_id
- email_id = self.db.need_to_send_to_tg(email_msg_id, tg_chat_id)
+ email_id = self.db.get_email_id(email_msg_id, tg_chat_id)
return email_id
@@ -141,4 +141,4 @@ class Bot():
if not reply_to:
return None
- return self.db.get_tg_reply_to(reply_to, tg_chat_id)
+ return self.db.get_reply_id(reply_to, tg_chat_id)
diff --git a/daemon/telegram/scraper/db.py b/daemon/telegram/scraper/db.py
deleted file mode 100644
index 58601d1..0000000
--- a/daemon/telegram/scraper/db.py
+++ /dev/null
@@ -1,217 +0,0 @@
-# SPDX-License-Identifier: GPL-2.0-only
-#
-# Copyright (C) 2022 Muhammad Rizki <[email protected]>
-# Copyright (C) 2022 Ammar Faizi <[email protected]>
-#
-
-from datetime import datetime
-import mysql
-
-
-class Db():
- def __init__(self, conn):
- self.conn = conn
- self.conn.autocommit = True
- self.cur = self.conn.cursor(buffered=True)
-
-
- def __del__(self):
- self.cur.close()
- self.conn.close()
-
-
- def save_email_msg_id(self, email_msg_id):
- try:
- return self.__save_email_msg_id(email_msg_id)
- except mysql.connector.errors.IntegrityError:
- #
- # Duplicate data, skip!
- #
- return None
-
-
- def __save_email_msg_id(self, email_msg_id):
- q = "INSERT INTO emails (message_id, created_at) VALUES (%s, %s)"
- self.cur.execute(q, (email_msg_id, datetime.utcnow()))
- return self.cur.lastrowid
-
-
- def insert_telegram(self, email_id, tg_chat_id, tg_msg_id):
- q = """
- INSERT INTO tg_emails
- (email_id, chat_id, tg_msg_id, created_at)
- VALUES (%s, %s, %s, %s);
- """
- self.cur.execute(q, (email_id, tg_chat_id, tg_msg_id,
- datetime.utcnow()))
- return self.cur.lastrowid
-
-
- #
- # Determine whether the email needs to be sent to @tg_chat_id.
- #
- # - Return an email id (PK) if it needs to be sent.
- # - Return None if it doesn't need to be sent.
- #
- def need_to_send_to_tg(self, email_msg_id, tg_chat_id):
- q = """
- SELECT emails.id, tg_emails.id FROM emails LEFT JOIN tg_emails
- ON emails.id = tg_emails.email_id
- WHERE emails.message_id = %(email_msg_id)s
- AND tg_emails.chat_id = %(tg_chat_id)s
- LIMIT 1
- """
-
- self.cur.execute(
- q,
- {
- "email_msg_id": email_msg_id,
- "tg_chat_id": tg_chat_id
- }
- )
- res = self.cur.fetchone()
- if bool(res):
- #
- # This email has already been sent to
- # @tg_chat_id.
- #
- return None
-
- q = """
- SELECT id FROM emails WHERE message_id = %(email_msg_id)s
- """
- self.cur.execute(q, {"email_msg_id": email_msg_id})
- res = self.cur.fetchone()
- if not bool(res):
- #
- # Something goes wrong, skip!
- #
- return None
-
- return int(res[0])
-
-
- def get_tg_reply_to(self, email_msg_id, tg_chat_id):
- q = """
- SELECT tg_emails.tg_msg_id
- FROM emails INNER JOIN tg_emails
- ON emails.id = tg_emails.email_id
- WHERE emails.message_id = %(email_msg_id)s
- AND tg_emails.chat_id = %(chat_id)s
- """
-
- self.cur.execute(
- q,
- {
- "email_msg_id": email_msg_id,
- "chat_id": tg_chat_id
- }
- )
- res = self.cur.fetchone()
- if not bool(res):
- return None
-
- return res[0]
-
-
- def insert_atom(self, atom: str):
- try:
- return self.__save_atom(atom)
- except mysql.connector.errors.IntegrityError:
- #
- # Duplicate data, skip!
- #
- return None
-
-
- def __save_atom(self, atom: str):
- q = "INSERT INTO atom_urls (url, created_at) VALUES (%s, %s)"
- self.cur.execute(q, (atom, datetime.utcnow()))
- return self.cur.lastrowid
-
-
- def delete_atom(self, atom: str):
- q = """
- DELETE FROM atom_urls
- WHERE url = %(atom)s
- """
- try:
- self.cur.execute(q, {"atom": atom})
- return True
- except:
- return False
-
-
- def get_atom_urls(self):
- q = """
- SELECT atom_urls.url
- FROM atom_urls
- """
- self.cur.execute(q)
- urls = self.cur.fetchall()
-
- return [u[0] for u in urls]
-
-
- def insert_broadcast(
- self,
- chat_id: int,
- name: str,
- type: str,
- created_at: "datetime",
- username: str = None,
- link: str = None,
- ):
- try:
- return self.__save_broadcast(
- chat_id=chat_id,
- name=name,
- type=type,
- created_at=created_at,
- username=username,
- link=link
- )
- except mysql.connector.errors.IntegrityError:
- #
- # Duplicate data, skip!
- #
- return None
-
-
- def __save_broadcast(
- self,
- chat_id: int,
- name: str,
- type: str,
- created_at: "datetime",
- username: str = None,
- link: str = None,
- ):
- q = """
- INSERT INTO broadcast_chats
- (chat_id, username, name, type, link, created_at)
- VALUES
- (%s, %s, %s, %s, %s, %s)
- """
- values = (chat_id, username, name, type, link, created_at)
- self.cur.execute(q, values)
- return self.cur.lastrowid
-
-
- def delete_broadcast(self, chat_id: int):
- q = """
- DELETE FROM broadcast_chats
- WHERE chat_id = %(chat_id)s
- """
- self.cur.execute(q, {"chat_id": chat_id})
- return self.cur.rowcount > 0
-
-
- def get_broadcast_chats(self):
- q = """
- SELECT *
- FROM broadcast_chats
- """
- self.cur.execute(q)
-
- return self.cur.fetchall()
--
Muhammad Rizki
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [RFC PATCH v1 3/5] [telegram] Renaming some functions in scraper/bot.py
2022-09-06 11:19 [RFC PATCH v1 0/5] Refactor some Telegram bot source code Muhammad Rizki
2022-09-06 11:19 ` [RFC PATCH v1 1/5] [telegram] Move the " Muhammad Rizki
2022-09-06 11:19 ` [RFC PATCH v1 2/5] [telegram] Refactor Telegram bot database method Muhammad Rizki
@ 2022-09-06 11:19 ` Muhammad Rizki
2022-09-06 16:11 ` Ammar Faizi
2022-09-06 11:19 ` [RFC PATCH v1 4/5] [telegram] Remove unecessary files Muhammad Rizki
2022-09-06 11:19 ` [RFC PATCH v1 5/5] Refactor many files Muhammad Rizki
4 siblings, 1 reply; 11+ messages in thread
From: Muhammad Rizki @ 2022-09-06 11:19 UTC (permalink / raw)
To: Ammar Faizi
Cc: Muhammad Rizki, GNU/Weeb Mailing List, Alviro Iskandar Setiawan
Just renameing some functions...
Signed-off-by: Muhammad Rizki <[email protected]>
---
daemon/telegram/scraper/bot.py | 17 +++++++++--------
1 file changed, 9 insertions(+), 8 deletions(-)
diff --git a/daemon/telegram/scraper/bot.py b/daemon/telegram/scraper/bot.py
index 9ef5508..a7087ad 100644
--- a/daemon/telegram/scraper/bot.py
+++ b/daemon/telegram/scraper/bot.py
@@ -4,6 +4,7 @@
# Copyright (C) 2022 Ammar Faizi <[email protected]>
#
+from pyrogram.types import Message
from apscheduler.schedulers.asyncio import AsyncIOScheduler
from packages import DaemonClient
from scraper import Scraper
@@ -72,7 +73,7 @@ class Bot():
chats = self.db.get_broadcast_chats()
for chat in chats:
async with self.mutexes.send_to_tg:
- should_wait = await self.__send_to_tg(url, mail,
+ should_wait = await self.__send_mail(url, mail,
chat[1])
if should_wait:
@@ -80,7 +81,7 @@ class Bot():
# @__must_hold(self.mutexes.send_to_tg)
- async def __send_to_tg(self, url, mail, tg_chat_id):
+ async def __send_mail(self, url, mail, tg_chat_id):
email_msg_id = utils.get_email_msg_id(mail)
if not email_msg_id:
#
@@ -89,7 +90,7 @@ class Bot():
#
return False
- email_id = self.__need_to_send_to_telegram(email_msg_id,
+ email_id = self.__mail_id_from_db(email_msg_id,
tg_chat_id)
if not email_id:
#
@@ -99,16 +100,16 @@ class Bot():
return False
text, files, is_patch = utils.create_template(mail)
- reply_to = self.get_tg_reply_to(mail, tg_chat_id)
+ reply_to = self.get_reply(mail, tg_chat_id)
url = str(re.sub(r"/raw$", "", url))
if is_patch:
- m = await self.client.send_patch_email(
+ m: "Message" = await self.client.send_patch_email(
mail, tg_chat_id, text, reply_to, url
)
else:
text = "#ml\n" + text
- m = await self.client.send_text_email(
+ m: "Message" = await self.client.send_text_email(
tg_chat_id, text,reply_to, url
)
@@ -123,7 +124,7 @@ class Bot():
return True
- def __need_to_send_to_telegram(self, email_msg_id, tg_chat_id):
+ def __mail_id_from_db(self, email_msg_id, tg_chat_id):
email_id = self.db.save_email(email_msg_id)
if email_id:
return email_id
@@ -132,7 +133,7 @@ class Bot():
return email_id
- def get_tg_reply_to(self, mail, tg_chat_id):
+ def get_reply(self, mail, tg_chat_id):
reply_to = mail.get("in-reply-to")
if not reply_to:
return None
--
Muhammad Rizki
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [RFC PATCH v1 4/5] [telegram] Remove unecessary files
2022-09-06 11:19 [RFC PATCH v1 0/5] Refactor some Telegram bot source code Muhammad Rizki
` (2 preceding siblings ...)
2022-09-06 11:19 ` [RFC PATCH v1 3/5] [telegram] Renaming some functions in scraper/bot.py Muhammad Rizki
@ 2022-09-06 11:19 ` Muhammad Rizki
2022-09-06 16:02 ` Ammar Faizi
2022-09-06 11:19 ` [RFC PATCH v1 5/5] Refactor many files Muhammad Rizki
4 siblings, 1 reply; 11+ messages in thread
From: Muhammad Rizki @ 2022-09-06 11:19 UTC (permalink / raw)
To: Ammar Faizi
Cc: Muhammad Rizki, GNU/Weeb Mailing List, Alviro Iskandar Setiawan
Just remove unecessary files because it's already moved to the
plugins/commands
Signed-off-by: Muhammad Rizki <[email protected]>
---
daemon/telegram/packages/plugins/admin.py | 42 -----------
daemon/telegram/packages/plugins/scrape.py | 86 ----------------------
2 files changed, 128 deletions(-)
delete mode 100644 daemon/telegram/packages/plugins/admin.py
delete mode 100644 daemon/telegram/packages/plugins/scrape.py
diff --git a/daemon/telegram/packages/plugins/admin.py b/daemon/telegram/packages/plugins/admin.py
deleted file mode 100644
index e0f145e..0000000
--- a/daemon/telegram/packages/plugins/admin.py
+++ /dev/null
@@ -1,42 +0,0 @@
-# SPDX-License-Identifier: GPL-2.0-only
-#
-# Copyright (C) 2022 Muhammad Rizki <[email protected]>
-#
-
-from pyrogram import Client, filters, enums
-from pyrogram.types import Message
-from textwrap import indent
-import io, import_expression, contextlib, traceback
-
[email protected]_message(
- filters.command(['d','debug']) &
- filters.user(["nekoha", "kiizuah"])
-)
-async def execute_v2(c: Client, m: Message):
- sep = m.text.split('\n')
- body = m.text.replace(sep[0] + '\n','')
-
- env = {"bot": c}
- env.update(globals())
-
- stdout = io.StringIO()
- to_compile = f'async def func(_, m):\n{indent(body, " ")}'
-
- try:
- import_expression.exec(to_compile, env)
- except Exception as e:
- text = f"```{e.__class__.__name__}: {e}"[0:4096]+"```"
-
- func = env["func"]
-
- try:
- with contextlib.redirect_stdout(stdout):
- await func(c, m)
- except Exception:
- value = stdout.getvalue()
- text = f"```{value}{traceback.format_exc()}"[0:4096]+"```"
- else:
- value = stdout.getvalue()
- text = f"```{value}"[0:4096]+"```"
-
- await c.send_message(m.chat.id, text, parse_mode=enums.ParseMode.MARKDOWN)
diff --git a/daemon/telegram/packages/plugins/scrape.py b/daemon/telegram/packages/plugins/scrape.py
deleted file mode 100644
index 1698c6d..0000000
--- a/daemon/telegram/packages/plugins/scrape.py
+++ /dev/null
@@ -1,86 +0,0 @@
-# SPDX-License-Identifier: GPL-2.0-only
-#
-# Copyright (C) 2022 Muhammad Rizki <[email protected]>
-# Copyright (C) 2022 Ammar Faizi <[email protected]>
-#
-
-from pyrogram.types import InlineKeyboardMarkup
-from pyrogram.types import InlineKeyboardButton
-from pyrogram.types import Message
-from pyrogram import filters
-from pyrogram import Client
-from scraper import Scraper
-from pyrogram import enums
-from scraper import utils
-from scraper import Bot
-import shutil
-import re
-import asyncio
-
-
-#
-# This allows user to invoke the following commands:
-# /lore https://lore.kernel.org/path/message_id/raw
-# !lore https://lore.kernel.org/path/message_id/raw
-# .lore https://lore.kernel.org/path/message_id/raw
-#
-LORE_CMD_URL_PATTERN = r"^(?:\/|\.|\!)lore\s+(https?:\/\/lore\.kernel\.org\/\S+)"
[email protected]_message(
- filters.regex(LORE_CMD_URL_PATTERN) &
- filters.chat(["kiizuah", "nekoha", -1001673279485])
-)
-async def scrap_email(_, m: Message):
- p = re.search(LORE_CMD_URL_PATTERN, m.text)
- if not p:
- return
-
- url = p.group(1)
- if not url:
- return
-
- s = Scraper()
- mail = await s.get_email_from_url(url)
- text, files, is_patch = utils.create_template(mail)
-
- if is_patch:
- m = await __send_patch_msg(m, mail, text, url)
- else:
- text = "#ml\n" + text
- m = await __send_text_msg(m, text, url)
-
- for d, f in files:
- await m.reply_document(f"{d}/{f}", file_name=f)
- await asyncio.sleep(1)
-
- if files:
- shutil.rmtree(str(files[0][0]))
-
-
-async def __send_patch_msg(m, mail, text, url):
- tmp, fnm, caption, url = Bot.prepare_send_patch(mail, text, url)
- ret = await m.reply_document(
- fnm,
- caption=caption,
- parse_mode=enums.ParseMode.HTML,
- reply_markup=InlineKeyboardMarkup([
- [InlineKeyboardButton(
- "See the full message",
- url=url
- )]
- ])
- )
- Bot.clean_up_after_send_patch(tmp)
- return ret
-
-
-async def __send_text_msg(m, text, url):
- return await m.reply(
- text,
- parse_mode=enums.ParseMode.HTML,
- reply_markup=InlineKeyboardMarkup([
- [InlineKeyboardButton(
- "See the full message",
- url=url.replace("/raw","")
- )]
- ])
- )
--
Muhammad Rizki
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [RFC PATCH v1 5/5] Refactor many files
2022-09-06 11:19 [RFC PATCH v1 0/5] Refactor some Telegram bot source code Muhammad Rizki
` (3 preceding siblings ...)
2022-09-06 11:19 ` [RFC PATCH v1 4/5] [telegram] Remove unecessary files Muhammad Rizki
@ 2022-09-06 11:19 ` Muhammad Rizki
2022-09-06 16:04 ` Ammar Faizi
4 siblings, 1 reply; 11+ messages in thread
From: Muhammad Rizki @ 2022-09-06 11:19 UTC (permalink / raw)
To: Ammar Faizi
Cc: Muhammad Rizki, GNU/Weeb Mailing List, Alviro Iskandar Setiawan
I want to refactor atom scraper file and utility file and create a
directory for both of it to make it reuseable in the future use. This
commit contains:
- Rename some functions in utils
- Rename file name in telegram such as scraper => mailer,
bot.py => listener
- Move class Mutexes to utility file
- Rename the Mutexes attribute send_to_tg => lock
- Changes affected codes during this refactor
Signed-off-by: Muhammad Rizki <[email protected]>
---
.gitignore | 1 +
daemon/atom/__init__.py | 7 ++
daemon/{telegram/scraper => atom}/scraper.py | 12 +--
daemon/{telegram/scraper => atom}/utils.py | 87 ++++++++++++++-----
daemon/{telegram => }/db.sql | 0
.../.env.example => telegram.env.example} | 0
.../telegram/{scraper => mailer}/__init__.py | 4 +-
.../{scraper/bot.py => mailer/listener.py} | 23 ++---
daemon/telegram/packages/client.py | 10 ++-
.../packages/plugins/callbacks/del_atom.py | 6 +-
.../packages/plugins/callbacks/del_chat.py | 6 +-
.../packages/plugins/commands/debugger.py | 2 +-
.../packages/plugins/commands/manage_atom.py | 6 +-
.../plugins/commands/manage_broadcast.py | 6 +-
.../packages/plugins/commands/scrape.py | 10 +--
daemon/{telegram/run.py => tg.py} | 24 +++--
16 files changed, 122 insertions(+), 82 deletions(-)
create mode 100644 daemon/atom/__init__.py
rename daemon/{telegram/scraper => atom}/scraper.py (79%)
rename daemon/{telegram/scraper => atom}/utils.py (72%)
rename daemon/{telegram => }/db.sql (100%)
rename daemon/{telegram/.env.example => telegram.env.example} (100%)
rename daemon/telegram/{scraper => mailer}/__init__.py (68%)
rename daemon/telegram/{scraper/bot.py => mailer/listener.py} (88%)
rename daemon/{telegram/run.py => tg.py} (68%)
diff --git a/.gitignore b/.gitignore
index 4201a17..53027d9 100644
--- a/.gitignore
+++ b/.gitignore
@@ -140,5 +140,6 @@ data.json
*.patch
# configuration file
+daemon/*.env
daemon/telegram/config.py
daemon/discord/config.py
diff --git a/daemon/atom/__init__.py b/daemon/atom/__init__.py
new file mode 100644
index 0000000..2fe4e31
--- /dev/null
+++ b/daemon/atom/__init__.py
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Copyright (C) 2022 Muhammad Rizki <[email protected]>
+# Copyright (C) 2022 Ammar Faizi <[email protected]>
+#
+
+from .scraper import Scraper
diff --git a/daemon/telegram/scraper/scraper.py b/daemon/atom/scraper.py
similarity index 79%
rename from daemon/telegram/scraper/scraper.py
rename to daemon/atom/scraper.py
index 2d5942b..8508ae9 100644
--- a/daemon/telegram/scraper/scraper.py
+++ b/daemon/atom/scraper.py
@@ -1,6 +1,6 @@
# SPDX-License-Identifier: GPL-2.0-only
#
-# Copyright (C) 2022 Muhammad Rizki <[email protected]>
+# Copyright (C) 2022 Muhammad Rizki <[email protected]>
# Copyright (C) 2022 Ammar Faizi <[email protected]>
#
@@ -11,7 +11,7 @@ import httpx
import email
-class Scraper():
+class Scraper:
async def get_new_threads_urls(self, atom_url):
ret = await self.__get_atom_content(atom_url)
return await self.__get_new_threads_from_atom(ret)
@@ -19,10 +19,10 @@ class Scraper():
async def __get_atom_content(self, atom_url):
async with httpx.AsyncClient() as client:
- res = await client.get(atom_url)
+ res = await client.get(atom_url, timeout=20)
if res.status_code == 200:
return res.text
- raise Exception(f"[get_atom_content]: Returned {res.status_code} HTTP code")
+ raise Exception(f"[__get_atom_content]: Returned {res.status_code} HTTP code")
async def __get_new_threads_from_atom(self, atom):
@@ -54,10 +54,10 @@ class Scraper():
async def get_email_from_url(self, url):
async with httpx.AsyncClient() as client:
- res = await client.get(url)
+ res = await client.get(url, timeout=20)
if res.status_code == 200:
return email.message_from_string(
res.text,
policy=email.policy.default
)
- raise Exception(f"[get_atom_content]: Returned {res.status_code} HTTP code")
+ raise Exception(f"[get_email_from_url]: Returned {res.status_code} HTTP code")
diff --git a/daemon/telegram/scraper/utils.py b/daemon/atom/utils.py
similarity index 72%
rename from daemon/telegram/scraper/utils.py
rename to daemon/atom/utils.py
index c428a33..d73d6bd 100644
--- a/daemon/telegram/scraper/utils.py
+++ b/daemon/atom/utils.py
@@ -1,6 +1,6 @@
# SPDX-License-Identifier: GPL-2.0-only
#
-# Copyright (C) 2022 Muhammad Rizki <[email protected]>
+# Copyright (C) 2022 Muhammad Rizki <[email protected]>
# Copyright (C) 2022 Ammar Faizi <[email protected]>
#
@@ -8,13 +8,19 @@ from pyrogram.types import Chat, InlineKeyboardMarkup, InlineKeyboardButton
from email.message import Message
from typing import Dict
from slugify import slugify
+import html
import hashlib
import uuid
import os
import re
import shutil
import httpx
-import html
+import asyncio
+
+
+class Mutexes:
+ def __init__(self):
+ self.lock = asyncio.Lock()
def get_email_msg_id(mail):
@@ -113,25 +119,37 @@ def consruct_to_n_cc(to: list, cc: list):
return ret
-def gen_temp(name: str):
+def gen_temp(name: str, platform: str):
+ platform = platform.lower()
+ plt_ls = ["telegram", "discord"]
+
+ if platform not in plt_ls:
+ t = f"Platform {platform} is not found, "
+ t += f"only {', '.join(plt_ls)} is available"
+ raise ValueError(f"Platform {platform} is not found")
+
md5 = hashlib.md5(name.encode()).hexdigest()
- ret = os.getenv("STORAGE_DIR", "storage") + "/" + md5
+ store_dir = os.getenv("STORAGE_DIR", "storage")
+ platform = platform.replace("discord", "dscord")
+ path = f"{platform}/{store_dir}/{md5}"
try:
- os.mkdir(ret)
+ os.mkdir(path)
except FileExistsError:
pass
- return ret
+ return path
-def extract_body(thread: Message):
+def extract_body(thread: Message, platform: str):
if not thread.is_multipart():
- p = thread.get_payload(decode=True)
- return f"{p.decode(errors='replace')}\n".lstrip(), []
+ p = thread.get_payload(decode=True).decode(errors='replace')
+ if platform == "discord":
+ p = quote_reply(p)
+ return f"{p}\n".lstrip(), []
ret = ""
files = []
- temp = gen_temp(str(uuid.uuid4()))
+ temp = gen_temp(str(uuid.uuid4()), platform)
for p in thread.get_payload():
fname = p.get_filename()
payload = p.get_payload(decode=True)
@@ -164,35 +182,42 @@ def __is_patch(subject, content):
return True
-def create_template(thread: Message, to=None, cc=None):
+def create_template(thread: Message, platform: str, to=None, cc=None):
if not to:
to = extract_list("to", thread)
if not cc:
cc = extract_list("cc", thread)
+ if platform == "telegram":
+ substr = 4000
+ border = f"\n<code>{'-'*72}</code>"
+ else:
+ substr = 1900
+ border = f"\n{'-'*80}"
subject = thread.get('subject')
ret = f"From: {thread.get('from')}\n"
ret += consruct_to_n_cc(to, cc)
ret += f"Date: {thread.get('date')}\n"
ret += f"Subject: {subject}\n\n"
- content, files = extract_body(thread)
+ content, files = extract_body(thread, platform)
is_patch = __is_patch(subject, content)
if is_patch:
ret += content
else:
ret += content.strip().replace("\t", " ")
- if len(ret) >= 4000:
- ret = ret[:4000] + "..."
- ret = fix_utf8_char(ret)
- ret += f"\n<code>{'-'*72}</code>"
+ if len(ret) >= substr:
+ ret = ret[:substr] + "..."
+
+ ret = fix_utf8_char(ret, platform == "telegram")
+ ret += border
return ret, files, is_patch
-def prepare_send_patch(mail, text, url):
- tmp = gen_temp(url)
+def prepare_patch(mail: "Message", text: str, url: str, platform: str):
+ tmp = gen_temp(url, platform)
fnm = str(mail.get("subject"))
sch = re.search(PATCH_PATTERN, fnm, re.IGNORECASE)
@@ -210,17 +235,31 @@ def prepare_send_patch(mail, text, url):
with open(file, "wb") as f:
f.write(bytes(text, encoding="utf8"))
- caption = "#patch #ml\n" + fix_utf8_char(cap)
+ caption = "#patch #ml"
+ if platform == "telegram":
+ caption += fix_utf8_char("\n" + cap, True)
return tmp, file, caption, url
-def clean_up_after_send_patch(tmp):
+def remove_patch(tmp):
shutil.rmtree(tmp)
-def fix_utf8_char(text: str):
- text = text.rstrip().replace("�"," ")
- return html.escape(html.escape(text))
+def fix_utf8_char(text: str, html_escape: bool = True):
+ t = text.rstrip().replace("�"," ")
+ if html_escape:
+ t = html.escape(html.escape(text))
+ return t
+
+
+def quote_reply(text: str):
+ a = ""
+ for b in text.split("\n"):
+ b = b.replace(">\n", "> ")
+ if b.startswith(">"):
+ a += "> "
+ a += f"{b}\n"
+ return a
EMAIL_MSG_ID_PATTERN = r"<([^\<\>]+)>"
@@ -240,6 +279,8 @@ async def is_atom_url(text: str):
return mime == "application/atom+xml"
except: return False
+
+
def remove_command(text: str):
txt = text.split(" ")
txt = text.replace(txt[0] + " ","")
diff --git a/daemon/telegram/db.sql b/daemon/db.sql
similarity index 100%
rename from daemon/telegram/db.sql
rename to daemon/db.sql
diff --git a/daemon/telegram/.env.example b/daemon/telegram.env.example
similarity index 100%
rename from daemon/telegram/.env.example
rename to daemon/telegram.env.example
diff --git a/daemon/telegram/scraper/__init__.py b/daemon/telegram/mailer/__init__.py
similarity index 68%
rename from daemon/telegram/scraper/__init__.py
rename to daemon/telegram/mailer/__init__.py
index 4294302..20f9034 100644
--- a/daemon/telegram/scraper/__init__.py
+++ b/daemon/telegram/mailer/__init__.py
@@ -4,6 +4,4 @@
# Copyright (C) 2022 Ammar Faizi <[email protected]>
#
-from .scraper import Scraper
-from .bot import BotMutexes
-from .bot import Bot
+from .listener import Listener
diff --git a/daemon/telegram/scraper/bot.py b/daemon/telegram/mailer/listener.py
similarity index 88%
rename from daemon/telegram/scraper/bot.py
rename to daemon/telegram/mailer/listener.py
index a7087ad..5e9acd2 100644
--- a/daemon/telegram/scraper/bot.py
+++ b/daemon/telegram/mailer/listener.py
@@ -6,26 +6,21 @@
from pyrogram.types import Message
from apscheduler.schedulers.asyncio import AsyncIOScheduler
-from packages import DaemonClient
-from scraper import Scraper
-from . import utils
+from telegram.packages import DaemonClient
+from atom import Scraper
+from atom import utils
import asyncio
import shutil
import re
import traceback
-class BotMutexes():
- def __init__(self):
- self.send_to_tg = asyncio.Lock()
-
-
-class Bot():
+class Listener:
def __init__(self, client: DaemonClient, sched: AsyncIOScheduler,
- scraper: Scraper, mutexes: BotMutexes):
+ mutexes: utils.Mutexes):
self.client = client
self.sched = sched
- self.scraper = scraper
+ self.scraper = Scraper()
self.mutexes = mutexes
self.db = client.db
self.isRunnerFixed = False
@@ -72,7 +67,7 @@ class Bot():
async def __handle_mail(self, url, mail):
chats = self.db.get_broadcast_chats()
for chat in chats:
- async with self.mutexes.send_to_tg:
+ async with self.mutexes.lock:
should_wait = await self.__send_mail(url, mail,
chat[1])
@@ -80,7 +75,7 @@ class Bot():
await asyncio.sleep(1)
- # @__must_hold(self.mutexes.send_to_tg)
+ # @__must_hold(self.mutexes.lock)
async def __send_mail(self, url, mail, tg_chat_id):
email_msg_id = utils.get_email_msg_id(mail)
if not email_msg_id:
@@ -99,7 +94,7 @@ class Bot():
#
return False
- text, files, is_patch = utils.create_template(mail)
+ text, files, is_patch = utils.create_template(mail, "telegram")
reply_to = self.get_reply(mail, tg_chat_id)
url = str(re.sub(r"/raw$", "", url))
diff --git a/daemon/telegram/packages/client.py b/daemon/telegram/packages/client.py
index 820c3e2..686e5ef 100644
--- a/daemon/telegram/packages/client.py
+++ b/daemon/telegram/packages/client.py
@@ -8,8 +8,8 @@ from pyrogram.enums import ParseMode
from pyrogram.types import Message, InlineKeyboardMarkup, InlineKeyboardButton
from typing import Union
from email.message import Message
-from scraper import utils
-from database import DB
+from atom import utils
+from telegram.database import DB
from .decorator import handle_flood
@@ -56,7 +56,9 @@ class DaemonClient(Client):
parse_mode: ParseMode = ParseMode.HTML
) -> Message:
print("[send_patch_email]")
- tmp, doc, caption, url = utils.prepare_send_patch(mail, text, url)
+ tmp, doc, caption, url = utils.prepare_patch(
+ mail, text, url, "telegram"
+ )
m = await self.send_document(
chat_id=chat_id,
document=doc,
@@ -71,5 +73,5 @@ class DaemonClient(Client):
])
)
- utils.clean_up_after_send_patch(tmp)
+ utils.remove_patch(tmp)
return m
diff --git a/daemon/telegram/packages/plugins/callbacks/del_atom.py b/daemon/telegram/packages/plugins/callbacks/del_atom.py
index 1510d60..b750e1c 100644
--- a/daemon/telegram/packages/plugins/callbacks/del_atom.py
+++ b/daemon/telegram/packages/plugins/callbacks/del_atom.py
@@ -3,10 +3,10 @@
# Copyright (C) 2022 Muhammad Rizki <[email protected]>
#
-from packages import DaemonClient
-from scraper import utils
+from telegram.packages import DaemonClient
+from atom import utils
from pyrogram.types import CallbackQuery
-import config
+from telegram import config
@DaemonClient.on_callback_query(config.admin_only, group=1)
diff --git a/daemon/telegram/packages/plugins/callbacks/del_chat.py b/daemon/telegram/packages/plugins/callbacks/del_chat.py
index 26c6dd8..90b557e 100644
--- a/daemon/telegram/packages/plugins/callbacks/del_chat.py
+++ b/daemon/telegram/packages/plugins/callbacks/del_chat.py
@@ -3,10 +3,10 @@
# Copyright (C) 2022 Muhammad Rizki <[email protected]>
#
-from packages import DaemonClient
-from scraper import utils
+from telegram.packages import DaemonClient
+from atom import utils
from pyrogram.types import CallbackQuery
-import config
+from telegram import config
@DaemonClient.on_callback_query(config.admin_only, group=2)
diff --git a/daemon/telegram/packages/plugins/commands/debugger.py b/daemon/telegram/packages/plugins/commands/debugger.py
index ae2d31d..7f6f367 100644
--- a/daemon/telegram/packages/plugins/commands/debugger.py
+++ b/daemon/telegram/packages/plugins/commands/debugger.py
@@ -7,7 +7,7 @@ from pyrogram import Client, filters, enums
from pyrogram.types import Message
from textwrap import indent
import io, import_expression, contextlib, traceback
-import config
+from telegram import config
@Client.on_message(
diff --git a/daemon/telegram/packages/plugins/commands/manage_atom.py b/daemon/telegram/packages/plugins/commands/manage_atom.py
index 4ba422a..99df7f7 100644
--- a/daemon/telegram/packages/plugins/commands/manage_atom.py
+++ b/daemon/telegram/packages/plugins/commands/manage_atom.py
@@ -5,9 +5,9 @@
from pyrogram.types import Message
from pyrogram import filters
-from packages import DaemonClient
-from scraper import utils
-import config
+from telegram.packages import DaemonClient
+from atom import utils
+from telegram import config
@DaemonClient.on_message(
diff --git a/daemon/telegram/packages/plugins/commands/manage_broadcast.py b/daemon/telegram/packages/plugins/commands/manage_broadcast.py
index 6d75c36..0aa70de 100644
--- a/daemon/telegram/packages/plugins/commands/manage_broadcast.py
+++ b/daemon/telegram/packages/plugins/commands/manage_broadcast.py
@@ -5,9 +5,9 @@
from pyrogram.types import Message
from pyrogram import filters, enums
-from packages import DaemonClient
-from scraper import utils
-import config
+from telegram.packages import DaemonClient
+from atom import utils
+from telegram import config
@DaemonClient.on_message(
diff --git a/daemon/telegram/packages/plugins/commands/scrape.py b/daemon/telegram/packages/plugins/commands/scrape.py
index 45b1581..4cdbf1c 100644
--- a/daemon/telegram/packages/plugins/commands/scrape.py
+++ b/daemon/telegram/packages/plugins/commands/scrape.py
@@ -6,10 +6,10 @@
from pyrogram.types import Message
from pyrogram import filters
-from packages import DaemonClient
-from scraper import Scraper
-from scraper import utils
-import config
+from telegram.packages import DaemonClient
+from atom import Scraper
+from atom import utils
+from telegram import config
import shutil
import re
import asyncio
@@ -37,7 +37,7 @@ async def scrap_email(c: DaemonClient, m: Message):
s = Scraper()
mail = await s.get_email_from_url(url)
- text, files, is_patch = utils.create_template(mail)
+ text, files, is_patch = utils.create_template(mail, "telegram")
if is_patch:
m = await c.send_patch_email(
diff --git a/daemon/telegram/run.py b/daemon/tg.py
similarity index 68%
rename from daemon/telegram/run.py
rename to daemon/tg.py
index 5360395..c3e85ab 100644
--- a/daemon/telegram/run.py
+++ b/daemon/tg.py
@@ -1,24 +1,23 @@
# SPDX-License-Identifier: GPL-2.0-only
#
-# Copyright (C) 2022 Muhammad Rizki <[email protected]>
+# Copyright (C) 2022 Muhammad Rizki <[email protected]>
# Copyright (C) 2022 Ammar Faizi <[email protected]>
#
from apscheduler.schedulers.asyncio import AsyncIOScheduler
-from scraper import BotMutexes
+from atom.utils import Mutexes
from dotenv import load_dotenv
from mysql import connector
-from packages import DaemonClient
-from scraper import Scraper
-from scraper import Bot
+from telegram.packages import DaemonClient
+from telegram.mailer import Listener
import os
def main():
- load_dotenv()
+ load_dotenv("telegram.env")
client = DaemonClient(
- "storage/EmailScraper",
+ "telegram/storage/EmailScraper",
api_id=int(os.getenv("API_ID")),
api_hash=os.getenv("API_HASH"),
bot_token=os.getenv("BOT_TOKEN"),
@@ -28,9 +27,7 @@ def main():
password=os.getenv("DB_PASS"),
database=os.getenv("DB_NAME")
),
- plugins=dict(
- root="packages.plugins"
- ),
+ plugins=dict(root="telegram.packages.plugins")
)
sched = AsyncIOScheduler(
@@ -40,14 +37,13 @@ def main():
}
)
- bot = Bot(
+ mailer = Listener(
client=client,
sched=sched,
- scraper=Scraper(),
- mutexes=BotMutexes()
+ mutexes=Mutexes()
)
sched.start()
- bot.run()
+ mailer.run()
client.run()
--
Muhammad Rizki
^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [RFC PATCH v1 4/5] [telegram] Remove unecessary files
2022-09-06 11:19 ` [RFC PATCH v1 4/5] [telegram] Remove unecessary files Muhammad Rizki
@ 2022-09-06 16:02 ` Ammar Faizi
2022-09-06 16:46 ` Muhammad Rizki
0 siblings, 1 reply; 11+ messages in thread
From: Ammar Faizi @ 2022-09-06 16:02 UTC (permalink / raw)
To: Muhammad Rizki; +Cc: GNU/Weeb Mailing List, Alviro Iskandar Setiawan
On 9/6/22 6:19 PM, Muhammad Rizki wrote:
> Just remove unecessary files because it's already moved to the
> plugins/commands
You moved them in patch #1, why not delete these files in the first place?
> Signed-off-by: Muhammad Rizki<[email protected]>
> ---
> daemon/telegram/packages/plugins/admin.py | 42 -----------
> daemon/telegram/packages/plugins/scrape.py | 86 ----------------------
> 2 files changed, 128 deletions(-)
> delete mode 100644 daemon/telegram/packages/plugins/admin.py
> delete mode 100644 daemon/telegram/packages/plugins/scrape.py
Can't we squash this into the first patch? Or I missed something?
--
Ammar Faizi
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [RFC PATCH v1 5/5] Refactor many files
2022-09-06 11:19 ` [RFC PATCH v1 5/5] Refactor many files Muhammad Rizki
@ 2022-09-06 16:04 ` Ammar Faizi
0 siblings, 0 replies; 11+ messages in thread
From: Ammar Faizi @ 2022-09-06 16:04 UTC (permalink / raw)
To: Muhammad Rizki; +Cc: GNU/Weeb Mailing List, Alviro Iskandar Setiawan
On 9/6/22 6:19 PM, Muhammad Rizki wrote:
> I want to refactor atom scraper file and utility file and create a
> directory for both of it to make it reuseable in the future use. This
> commit contains:
> - Rename some functions in utils
> - Rename file name in telegram such as scraper => mailer,
> bot.py => listener
> - Move class Mutexes to utility file
> - Rename the Mutexes attribute send_to_tg => lock
> - Changes affected codes during this refactor
Please split this one into smaller pieces.
--
Ammar Faizi
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [RFC PATCH v1 3/5] [telegram] Renaming some functions in scraper/bot.py
2022-09-06 11:19 ` [RFC PATCH v1 3/5] [telegram] Renaming some functions in scraper/bot.py Muhammad Rizki
@ 2022-09-06 16:11 ` Ammar Faizi
0 siblings, 0 replies; 11+ messages in thread
From: Ammar Faizi @ 2022-09-06 16:11 UTC (permalink / raw)
To: Muhammad Rizki; +Cc: GNU/Weeb Mailing List, Alviro Iskandar Setiawan
On 9/6/22 6:19 PM, Muhammad Rizki wrote:
> Just renameing some functions...
>
> Signed-off-by: Muhammad Rizki<[email protected]>
This one looks fine to me.
--
Ammar Faizi
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [RFC PATCH v1 4/5] [telegram] Remove unecessary files
2022-09-06 16:02 ` Ammar Faizi
@ 2022-09-06 16:46 ` Muhammad Rizki
2022-09-06 16:53 ` Ammar Faizi
0 siblings, 1 reply; 11+ messages in thread
From: Muhammad Rizki @ 2022-09-06 16:46 UTC (permalink / raw)
To: Ammar Faizi; +Cc: GNU/Weeb Mailing List, Alviro Iskandar Setiawan
On 06/09/2022 23.02, Ammar Faizi wrote:
> On 9/6/22 6:19 PM, Muhammad Rizki wrote:
>> Just remove unecessary files because it's already moved to the
>> plugins/commands
>
> You moved them in patch #1, why not delete these files in the first place?
Yeah, but I don't know yet how to place this commit to the first commit,
my knowledge only know to place this commit after the fist commit using
rebase then commit. Or you want to place this commit after the first commit?
>
>> Signed-off-by: Muhammad Rizki<[email protected]>
>> ---
>> daemon/telegram/packages/plugins/admin.py | 42 -----------
>> daemon/telegram/packages/plugins/scrape.py | 86 ----------------------
>> 2 files changed, 128 deletions(-)
>> delete mode 100644 daemon/telegram/packages/plugins/admin.py
>> delete mode 100644 daemon/telegram/packages/plugins/scrape.py
>
> Can't we squash this into the first patch? Or I missed something?
>
Yes, but I only know to place it after the first commit, I was wondering
why this file still exist while the master branch already removed that
files.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [RFC PATCH v1 4/5] [telegram] Remove unecessary files
2022-09-06 16:46 ` Muhammad Rizki
@ 2022-09-06 16:53 ` Ammar Faizi
0 siblings, 0 replies; 11+ messages in thread
From: Ammar Faizi @ 2022-09-06 16:53 UTC (permalink / raw)
To: Muhammad Rizki; +Cc: GNU/Weeb Mailing List, Alviro Iskandar Setiawan
On 9/6/22 11:46 PM, Muhammad Rizki wrote:
> On 06/09/2022 23.02, Ammar Faizi wrote:
>> On 9/6/22 6:19 PM, Muhammad Rizki wrote:
>>> Just remove unecessary files because it's already moved to the
>>> plugins/commands
>>
>> You moved them in patch #1, why not delete these files in the first place?
>
> Yeah, but I don't know yet how to place this commit to the first commit,
> my knowledge only know to place this commit after the fist commit using
> rebase then commit. Or you want to place this commit after the first commit?
Fold this one into the first commit.
# Download this patch
wget https://lore.gnuweeb.org/gwml/[email protected]/raw -O patch.diff;
# Rebase interactive
git rebase -i 2582d7e5225d47
# Open text editor, "edit" the first commit, save, close text editor
# Fold the diff
git apply patch.diff
# Amend it
git commit --amend --no-edit .
# Go...
git rebase --continue
>>
>>> Signed-off-by: Muhammad Rizki <[email protected]>
>>> ---
>>> daemon/telegram/packages/plugins/admin.py | 42 -----------
>>> daemon/telegram/packages/plugins/scrape.py | 86 ----------------------
>>> 2 files changed, 128 deletions(-)
>>> delete mode 100644 daemon/telegram/packages/plugins/admin.py
>>> delete mode 100644 daemon/telegram/packages/plugins/scrape.py
>>
>> Can't we squash this into the first patch? Or I missed something?
>>
>
> Yes, but I only know to place it after the first commit, I was wondering
> why this file still exist while the master branch already removed that
> files.
You didn't remove them in first patch, you copied them.
--
Ammar Faizi
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2022-09-06 16:53 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2022-09-06 11:19 [RFC PATCH v1 0/5] Refactor some Telegram bot source code Muhammad Rizki
2022-09-06 11:19 ` [RFC PATCH v1 1/5] [telegram] Move the " Muhammad Rizki
2022-09-06 11:19 ` [RFC PATCH v1 2/5] [telegram] Refactor Telegram bot database method Muhammad Rizki
2022-09-06 11:19 ` [RFC PATCH v1 3/5] [telegram] Renaming some functions in scraper/bot.py Muhammad Rizki
2022-09-06 16:11 ` Ammar Faizi
2022-09-06 11:19 ` [RFC PATCH v1 4/5] [telegram] Remove unecessary files Muhammad Rizki
2022-09-06 16:02 ` Ammar Faizi
2022-09-06 16:46 ` Muhammad Rizki
2022-09-06 16:53 ` Ammar Faizi
2022-09-06 11:19 ` [RFC PATCH v1 5/5] Refactor many files Muhammad Rizki
2022-09-06 16:04 ` Ammar Faizi
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox