[PATCH v3 00/18] New feature to handle atom URLs and broadcast

[PATCH v3 00/18] New feature to handle atom URLs and broadcast

[Muhammad Rizki]

Morning sir,

This is v3 revision of new feature to handle atom URLs and broadcast

In this series I combine the atom URLs and broadcast SQL tables to
db.sql and remove the atom_broadcast.sql. Also, I change the atom_urls
data type of `type` field to VARCHAR(255).

This series contains a new feature to handle the atom URLs and broadcast
chats, such as add atom, add chat, del_atom and del_chat, with this
feature, we do not manual manage them again, just using the bot commands
and it will update in database.

There are 18 patches in this series:
- Patch 1 is to Add new table for atom URLs and broadcast chats
- Patch 2 is to add get atom URLs and get broadcast chats
- Patch 3 is to fix MySQL InternalError when query the atom and chats
- Patch 4 is to create /add_atom bot command to atom URL to database
- Patch 5 is to add some utility functions to handle code in /add_atom
- Patch 6 is to refactor db class to easily access over DaemonClient
- Patch 7 is to add insert_atom() in scraper/db.py to insert atom URLs
- Patch 8 is to rename admin.py to debugger.
- Patch 9 is to add delete_atom() in scraper/db.py for /del_atom command
- Patch 10 is to add button_numbers() in utility file to create btn nums
- Patch 11 is to split files between bot commands and bot CallbackQuery
- Patch 12 is to add del_atom for CallbackQuery to handle delete atom
- Patch 13 is to add insert_broadcast() in scraper/db.py to insert chats
- Patch 14 is to add delete_broadcast() in scraper/db.py to delete chats
- Patch 15 is to create /add_bc bot command to add chat to the database
- Patch 16 is to add create_chat_link() in utility file for chat link
- Patch 17 is to add /del_bc bot command to remove chat in the database
- Patch 18 is to add del_chat in bot CallbackQuery to handle delete chat

Already tested and it works fine, please give it a test too if there is
a bug or unexpected errors, thanks.

## Changelog

v2 -> v3
- Combine the atom_urls and broadcast_chats tables in db.py
- Remove atom_broadcast.sql
- Change atom_urls data type of `type` field to VARCHAR(255)

v1 -> v2
- Improve delete_broadcast() source code in scraper/db.py
- Fix typehints to int for chat_id param in scraper/db.py

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---

Muhammad Rizki (18):
  Add new table for atom URLs and broadcast chats in db.sql
  Add ATOM_URLS and BROADCAST_CHAT methods
  Fix MySQL InternalError: Unread result found
  Create /add_atom command
  Add utility functions for /add_atom
  Refactor database methods
  Add insert_atom() in scraper/db.py
  Rename admin.py to debugger.py
  Add delete_atom() function in scraper/db.py
  Add utility function buton_numbers()
  Move bot commands related files to packages/plugins/commands/
  Add del_atom callback query
  Add insert_broadcast() in scraper/db.py
  Add delete_broadcast() in scraper/db.py
  Create /add_bc bot command
  Add create_chat_link() function
  Add /del_bc bot command
  Add del_chat callback query

 .gitignore                                    |   4 +
 daemon/db.sql                                 |  25 +++++
 daemon/packages/client.py                     |   4 +-
 daemon/packages/plugins/callbacks/del_atom.py |  31 +++++
 daemon/packages/plugins/callbacks/del_chat.py |  36 ++++++
 .../{admin.py => commands/debugger.py}        |   0
 .../packages/plugins/commands/manage_atom.py  |  50 +++++++++
 .../plugins/commands/manage_broadcast.py      |  66 +++++++++++
 .../packages/plugins/{ => commands}/scrape.py |   0
 daemon/run.py                                 |  16 +--
 daemon/scraper/bot.py                         |  22 +---
 daemon/scraper/db.py                          | 106 +++++++++++++++++-
 daemon/scraper/utils.py                       |  45 ++++++++
 13 files changed, 379 insertions(+), 26 deletions(-)
 create mode 100644 daemon/packages/plugins/callbacks/del_atom.py
 create mode 100644 daemon/packages/plugins/callbacks/del_chat.py
 rename daemon/packages/plugins/{admin.py => commands/debugger.py} (100%)
 create mode 100644 daemon/packages/plugins/commands/manage_atom.py
 create mode 100644 daemon/packages/plugins/commands/manage_broadcast.py
 rename daemon/packages/plugins/{ => commands}/scrape.py (100%)


base-commit: f8c5aef0aa85e1262c90f818638d255c3c38373f
-- 
Muhammad Rizki

[PATCH v3 01/18] Add new table for atom URLs and broadcast chats in db.sql

[Muhammad Rizki]

Just add new table for atom URLs and broadcast chats in db.sql to save
them in MySQL database.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 .gitignore    |  4 ++++
 daemon/db.sql | 25 +++++++++++++++++++++++++
 2 files changed, 29 insertions(+)

diff --git a/.gitignore b/.gitignore
index ca28962..511dbbe 100644
--- a/.gitignore
+++ b/.gitignore
@@ -136,4 +136,8 @@ data.json
 # VSCode editor
 .vscode/
 
+# PATCH
 *.patch
+
+# RFC directory
+RFC
\ No newline at end of file
diff --git a/daemon/db.sql b/daemon/db.sql
index 0e4841a..ac6ed0d 100644
--- a/daemon/db.sql
+++ b/daemon/db.sql
@@ -34,4 +34,29 @@ CREATE TABLE `tg_emails` (
 ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_520_ci;
 
 
+DROP TABLE IF EXISTS `atom_urls`;
+CREATE TABLE `atom_urls` (
+  `id` bigint unsigned NOT NULL AUTO_INCREMENT,
+  `url` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_520_ci NOT NULL,
+  `created_at` datetime NOT NULL,
+  PRIMARY KEY (`id`),
+  UNIQUE KEY `url` (`url`),
+  KEY `created_at` (`created_at`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_520_ci;
+
+
+DROP TABLE IF EXISTS `broadcast_chats`;
+CREATE TABLE `broadcast_chats` (
+  `id` bigint unsigned NOT NULL AUTO_INCREMENT,
+  `chat_id` bigint NOT NULL,
+  `username` varchar(32),
+  `name` varchar(64) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_520_ci NOT NULL,
+  `type` varchar(32) NOT NULL,
+  `link` varchar(64),
+  `created_at` datetime NOT NULL,
+  PRIMARY KEY (`id`),
+  UNIQUE KEY `chat_id` (`chat_id`),
+  KEY `created_at` (`created_at`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_520_ci;
+
 -- 2022-07-07 14:25:28
-- 
Muhammad Rizki

[PATCH v3 02/18] Add ATOM_URLS and BROADCAST_CHAT methods

[Muhammad Rizki]

I decided to add ATOM_URLS and BROADCAST_CHAT methods to get their
content in database which will be handle in scraper/bot.py

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/scraper/bot.py | 17 ++++-------------
 daemon/scraper/db.py  | 21 +++++++++++++++++++++
 2 files changed, 25 insertions(+), 13 deletions(-)

diff --git a/daemon/scraper/bot.py b/daemon/scraper/bot.py
index bc9d575..c475e34 100644
--- a/daemon/scraper/bot.py
+++ b/daemon/scraper/bot.py
@@ -21,16 +21,6 @@ class BotMutexes():
 
 
 class Bot():
-	ATOM_URLS = [
-		"https://lore.kernel.org/io-uring/new.atom",
-		"https://lore.kernel.org/linux-sgx/new.atom",
-	]
-
-	TG_CHAT_IDS = [
-		"kiizuah"
-	]
-
-
 	def __init__(self, client: DaemonClient, sched: AsyncIOScheduler,
 			scraper: Scraper, mutexes: BotMutexes, conn):
 		self.client = client
@@ -55,7 +45,7 @@ class Bot():
 
 	async def __run(self):
 		print("[__run]: Running...")
-		for url in self.ATOM_URLS:
+		for url in self.db.get_atom_urls():
 			try:
 				await self.__handle_atom_url(url)
 			except:
@@ -80,10 +70,11 @@ class Bot():
 
 
 	async def __handle_mail(self, url, mail):
-		for tg_chat_id in self.TG_CHAT_IDS:
+		chats = self.db.get_broadcast_chats()
+		for chat in chats:
 			async with self.mutexes.send_to_tg:
 				should_wait = await self.__send_to_tg(url, mail,
-									tg_chat_id)
+									chat[1])
 
 			if should_wait:
 				await asyncio.sleep(1)
diff --git a/daemon/scraper/db.py b/daemon/scraper/db.py
index a79f041..321661f 100644
--- a/daemon/scraper/db.py
+++ b/daemon/scraper/db.py
@@ -112,3 +112,24 @@ class Db():
 			return None
 
 		return res[0]
+
+
+	def get_atom_urls(self):
+		q = """
+			SELECT atom_urls.url
+			FROM atom_urls
+		"""
+		self.cur.execute(q)
+		urls = self.cur.fetchall()
+
+		return [u[0] for u in urls]
+
+
+	def get_broadcast_chats(self):
+		q = """
+			SELECT *
+			FROM broadcast_chats
+		"""
+		self.cur.execute(q)
+
+		return self.cur.fetchall()
-- 
Muhammad Rizki

[PATCH v3 03/18] Fix MySQL InternalError: Unread result found

[Muhammad Rizki]

Use con.cursor(buffered=True) to make it avoid from
mysql.connector.errors.InternalError: Unread result found.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/scraper/db.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/daemon/scraper/db.py b/daemon/scraper/db.py
index 321661f..296559e 100644
--- a/daemon/scraper/db.py
+++ b/daemon/scraper/db.py
@@ -12,7 +12,7 @@ class Db():
 	def __init__(self, conn):
 		self.conn = conn
 		self.conn.autocommit = True
-		self.cur = self.conn.cursor()
+		self.cur = self.conn.cursor(buffered=True)
 
 
 	def __del__(self):
-- 
Muhammad Rizki

[PATCH v3 04/18] Create /add_atom command

[Muhammad Rizki]

Create /add_atom command for adding the atom url to the database

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/packages/plugins/manage_atom.py | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
 create mode 100644 daemon/packages/plugins/manage_atom.py

diff --git a/daemon/packages/plugins/manage_atom.py b/daemon/packages/plugins/manage_atom.py
new file mode 100644
index 0000000..eca3a60
--- /dev/null
+++ b/daemon/packages/plugins/manage_atom.py
@@ -0,0 +1,22 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Copyright (C) 2022  Muhammad Rizki <riskimuhammmad1@gmail.com>
+#
+
+from pyrogram.types import Message
+from pyrogram import filters
+from packages import DaemonClient
+
+@DaemonClient.on_message(
+        filters.command("add_atom") &
+        filters.chat(["kiizuah", "nekoha", -1001673279485])
+)
+async def add_atom_url(c: DaemonClient, m: Message):
+        pass
+        # text = utils.remove_command(m.text)
+        # if not utils.is_atom_url(text):
+        #         return
+        
+        # ###
+        # ### TODO: Muhammad Rizki: Add atom url into the database
+        # ###
-- 
Muhammad Rizki

[PATCH v3 05/18] Add utility functions for /add_atom

[Muhammad Rizki]

Add some utility functions for /add_atom command in the scraper/utils.py
and call it in packages/plugins/add_atom.py

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/packages/plugins/manage_atom.py | 20 ++++++++++----------
 daemon/scraper/utils.py                | 15 +++++++++++++++
 2 files changed, 25 insertions(+), 10 deletions(-)

diff --git a/daemon/packages/plugins/manage_atom.py b/daemon/packages/plugins/manage_atom.py
index eca3a60..a10318f 100644
--- a/daemon/packages/plugins/manage_atom.py
+++ b/daemon/packages/plugins/manage_atom.py
@@ -6,17 +6,17 @@
 from pyrogram.types import Message
 from pyrogram import filters
 from packages import DaemonClient
+from scraper import utils
 
 @DaemonClient.on_message(
-        filters.command("add_atom") &
-        filters.chat(["kiizuah", "nekoha", -1001673279485])
+	filters.command("add_atom") &
+	filters.chat(["kiizuah", "nekoha", -1001673279485])
 )
 async def add_atom_url(c: DaemonClient, m: Message):
-        pass
-        # text = utils.remove_command(m.text)
-        # if not utils.is_atom_url(text):
-        #         return
-        
-        # ###
-        # ### TODO: Muhammad Rizki: Add atom url into the database
-        # ###
+	text = utils.remove_command(m.text)
+	if not utils.is_atom_url(text):
+		return
+
+	###
+	### TODO: Muhammad Rizki: Add atom url into the database
+	###
diff --git a/daemon/scraper/utils.py b/daemon/scraper/utils.py
index 30efb7b..f7de5ba 100644
--- a/daemon/scraper/utils.py
+++ b/daemon/scraper/utils.py
@@ -12,6 +12,7 @@ import uuid
 import os
 import re
 import shutil
+import httpx
 
 
 def get_email_msg_id(mail):
@@ -231,3 +232,17 @@ def extract_email_msg_id(msg_id):
 		return None
 	return ret.group(1)
 
+
+async def is_atom_url(text: str):
+	try:
+		async with httpx.AsyncClient() as ses:
+			res = await ses.get(text)
+			mime = res.headers.get("Content-Type")
+
+			return mime == "application/atom+xml"
+	except: return False
+
+def remove_command(text: str):
+	txt = text.split(" ")
+	txt = text.replace(txt[0] + " ","")
+	return txt
-- 
Muhammad Rizki

[PATCH v3 06/18] Refactor database methods

[Muhammad Rizki]

I decided to refactor the database methods to make it able to access
with DaemonClient, so in that client I just to call one of the database
function just like:

async def add_atom(c: DaemonClient, m: Message):
        c.db.insert_atom("atom_url_here")

The c here is the DaemonClient parameter in the add_atom MessageHandler
function

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/packages/client.py |  4 +++-
 daemon/run.py             | 16 ++++++++--------
 daemon/scraper/bot.py     |  5 ++---
 3 files changed, 13 insertions(+), 12 deletions(-)

diff --git a/daemon/packages/client.py b/daemon/packages/client.py
index d16fe46..282daf6 100644
--- a/daemon/packages/client.py
+++ b/daemon/packages/client.py
@@ -9,14 +9,16 @@ from pyrogram.types import Message, InlineKeyboardMarkup, InlineKeyboardButton
 from typing import Union
 from email.message import Message
 from scraper import utils
+from scraper.db import Db
 from .decorator import handle_flood
 
 
 class DaemonClient(Client):
 	def __init__(self, name: str, api_id: int,
-			api_hash: str, **kwargs):
+		api_hash: str, conn, **kwargs):
 		super().__init__(name, api_id,
 				api_hash, **kwargs)
+		self.db = Db(conn)
 
 
 	@handle_flood
diff --git a/daemon/run.py b/daemon/run.py
index 1151ccd..5360395 100644
--- a/daemon/run.py
+++ b/daemon/run.py
@@ -22,9 +22,15 @@ def main():
 		api_id=int(os.getenv("API_ID")),
 		api_hash=os.getenv("API_HASH"),
 		bot_token=os.getenv("BOT_TOKEN"),
+		conn=connector.connect(
+			host=os.getenv("DB_HOST"),
+			user=os.getenv("DB_USER"),
+			password=os.getenv("DB_PASS"),
+			database=os.getenv("DB_NAME")
+		),
 		plugins=dict(
 			root="packages.plugins"
-		)
+		),
 	)
 
 	sched = AsyncIOScheduler(
@@ -38,13 +44,7 @@ def main():
 		client=client,
 		sched=sched,
 		scraper=Scraper(),
-		mutexes=BotMutexes(),
-		conn=connector.connect(
-			host=os.getenv("DB_HOST"),
-			user=os.getenv("DB_USER"),
-			password=os.getenv("DB_PASS"),
-			database=os.getenv("DB_NAME")
-		)
+		mutexes=BotMutexes()
 	)
 	sched.start()
 	bot.run()
diff --git a/daemon/scraper/bot.py b/daemon/scraper/bot.py
index c475e34..7adfb12 100644
--- a/daemon/scraper/bot.py
+++ b/daemon/scraper/bot.py
@@ -8,7 +8,6 @@ from apscheduler.schedulers.asyncio import AsyncIOScheduler
 from packages import DaemonClient
 from scraper import Scraper
 from . import utils
-from .db import Db
 import asyncio
 import shutil
 import re
@@ -22,12 +21,12 @@ class BotMutexes():
 
 class Bot():
 	def __init__(self, client: DaemonClient, sched: AsyncIOScheduler,
-			scraper: Scraper, mutexes: BotMutexes, conn):
+			scraper: Scraper, mutexes: BotMutexes):
 		self.client = client
 		self.sched = sched
 		self.scraper = scraper
 		self.mutexes = mutexes
-		self.db = Db(conn)
+		self.db = client.db
 		self.isRunnerFixed = False
 
 
-- 
Muhammad Rizki

[PATCH v3 07/18] Add insert_atom() in scraper/db.py

[Muhammad Rizki]

Add insert_atom() in scraper/db.py and call it in
packages/plugins/add_atom.py to insert atom url into the database

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/packages/plugins/manage_atom.py | 18 +++++++++++++-----
 daemon/scraper/db.py                   | 16 ++++++++++++++++
 2 files changed, 29 insertions(+), 5 deletions(-)

diff --git a/daemon/packages/plugins/manage_atom.py b/daemon/packages/plugins/manage_atom.py
index a10318f..934aba9 100644
--- a/daemon/packages/plugins/manage_atom.py
+++ b/daemon/packages/plugins/manage_atom.py
@@ -13,10 +13,18 @@ from scraper import utils
 	filters.chat(["kiizuah", "nekoha", -1001673279485])
 )
 async def add_atom_url(c: DaemonClient, m: Message):
+	if len(m.command) <= 1:
+		tutor = "Please specify the URL\n"
+		tutor += "Example: `/add_atom https://lore.kernel.org/linux-sgx/new.atom`"
+		return await m.reply(tutor)
+
 	text = utils.remove_command(m.text)
-	if not utils.is_atom_url(text):
-		return
+	is_atom = await utils.is_atom_url(text)
+	if not is_atom:
+		return await m.reply("Invalid Atom URL")
+
+	inserted = c.db.insert_atom(text)
+	if inserted is None:
+		return await m.reply(f"This URL already listened for new email.")
 
-	###
-	### TODO: Muhammad Rizki: Add atom url into the database
-	###
+	await m.reply(f"Success add **{text}** for listening new email")
diff --git a/daemon/scraper/db.py b/daemon/scraper/db.py
index 296559e..30ed9a7 100644
--- a/daemon/scraper/db.py
+++ b/daemon/scraper/db.py
@@ -114,6 +114,22 @@ class Db():
 		return res[0]
 
 
+	def insert_atom(self, atom: str):
+		try:
+			return self.__save_atom(atom)
+		except mysql.connector.errors.IntegrityError:
+			#
+			# Duplicate data, skip!
+			#
+			return None
+
+
+	def __save_atom(self, atom: str):
+		q = "INSERT INTO atom_urls (url, created_at) VALUES (%s, %s)"
+		self.cur.execute(q, (atom, datetime.utcnow()))
+		return self.cur.lastrowid
+
+
 	def get_atom_urls(self):
 		q = """
 			SELECT atom_urls.url
-- 
Muhammad Rizki

[PATCH v3 08/18] Rename admin.py to debugger.py

[Muhammad Rizki]

I want to create a plugin file that the name is related to admin, so I
decided to rename packages/plugins/admin.py to
packages/plugins/debugger.py

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/packages/plugins/{admin.py => debugger.py} | 0
 1 file changed, 0 insertions(+), 0 deletions(-)
 rename daemon/packages/plugins/{admin.py => debugger.py} (100%)

diff --git a/daemon/packages/plugins/admin.py b/daemon/packages/plugins/debugger.py
similarity index 100%
rename from daemon/packages/plugins/admin.py
rename to daemon/packages/plugins/debugger.py
-- 
Muhammad Rizki

[PATCH v3 09/18] Add delete_atom() function in scraper/db.py

[Muhammad Rizki]

Add delete_atom() function to remove atom URL in database

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/packages/plugins/manage_atom.py | 19 ++++++++++++++++++-
 daemon/scraper/db.py                   | 12 ++++++++++++
 2 files changed, 30 insertions(+), 1 deletion(-)

diff --git a/daemon/packages/plugins/manage_atom.py b/daemon/packages/plugins/manage_atom.py
index 934aba9..0d2166a 100644
--- a/daemon/packages/plugins/manage_atom.py
+++ b/daemon/packages/plugins/manage_atom.py
@@ -3,7 +3,7 @@
 # Copyright (C) 2022  Muhammad Rizki <riskimuhammmad1@gmail.com>
 #
 
-from pyrogram.types import Message
+from pyrogram.types import Message, InlineKeyboardMarkup, InlineKeyboardButton
 from pyrogram import filters
 from packages import DaemonClient
 from scraper import utils
@@ -28,3 +28,20 @@ async def add_atom_url(c: DaemonClient, m: Message):
 		return await m.reply(f"This URL already listened for new email.")
 
 	await m.reply(f"Success add **{text}** for listening new email")
+
+
+@DaemonClient.on_message(
+	filters.command("del_atom") &
+	filters.chat(["kiizuah", "nekoha", -1001673279485])
+)
+async def del_atom_url(c: DaemonClient, m: Message):
+	atoms = c.db.get_atom_urls()
+	if len(atoms) == 0:
+		return await m.reply("Currently empty.")
+
+	text = "List of atom URL that currently listened:\n"
+	for u,i in zip(atoms, range(1, len(atoms)+1)):
+		text += f"{i}. {u}\n"
+	
+	text += "\nChoose one of the URL above to delete by index below."
+	await m.reply(text)
diff --git a/daemon/scraper/db.py b/daemon/scraper/db.py
index 30ed9a7..24fc2c6 100644
--- a/daemon/scraper/db.py
+++ b/daemon/scraper/db.py
@@ -130,6 +130,18 @@ class Db():
 		return self.cur.lastrowid
 
 
+	def delete_atom(self, atom: str):
+		q = """
+			DELETE FROM atom_urls
+			WHERE url = %(atom)s
+		"""
+		try:
+			self.cur.execute(q, {"atom": atom})
+			return True
+		except:
+			return False
+
+
 	def get_atom_urls(self):
 		q = """
 			SELECT atom_urls.url
-- 
Muhammad Rizki

[PATCH v3 10/18] Add utility function buton_numbers()

[Muhammad Rizki]

Add utility function buton_numbers() for easily create a number of
inline buttons to manage delete atom urls.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/packages/plugins/manage_atom.py |  5 ++++-
 daemon/scraper/utils.py                | 17 +++++++++++++++++
 2 files changed, 21 insertions(+), 1 deletion(-)

diff --git a/daemon/packages/plugins/manage_atom.py b/daemon/packages/plugins/manage_atom.py
index 0d2166a..0b916de 100644
--- a/daemon/packages/plugins/manage_atom.py
+++ b/daemon/packages/plugins/manage_atom.py
@@ -44,4 +44,7 @@ async def del_atom_url(c: DaemonClient, m: Message):
 		text += f"{i}. {u}\n"
 	
 	text += "\nChoose one of the URL above to delete by index below."
-	await m.reply(text)
+	await m.reply(text, reply_markup=utils.button_numbers(
+		data=atoms,
+		callback_prefix="del_atom"
+	))
diff --git a/daemon/scraper/utils.py b/daemon/scraper/utils.py
index f7de5ba..d894c32 100644
--- a/daemon/scraper/utils.py
+++ b/daemon/scraper/utils.py
@@ -4,6 +4,7 @@
 # Copyright (C) 2022  Ammar Faizi <ammarfaizi2@gnuweeb.org>
 #
 
+from pyrogram.types import InlineKeyboardMarkup, InlineKeyboardButton
 from email.message import Message
 from typing import Dict
 from slugify import slugify
@@ -246,3 +247,19 @@ def remove_command(text: str):
 	txt = text.split(" ")
 	txt = text.replace(txt[0] + " ","")
 	return txt
+
+
+def button_numbers(data: list, callback_prefix: str, limit: int = 8):
+	if limit > 8:
+		raise ValueError("Limit value cannot more than 8.")
+
+	lst = []
+	for i in range(1, len(data)+1):
+		button = InlineKeyboardButton(
+			str(i),
+			callback_data=f"{callback_prefix} {i}"
+		)
+		lst.append(button)
+
+	buttons = [lst[i:i + limit ] for i in range(0, len(lst), limit)]
+	return InlineKeyboardMarkup(buttons)
-- 
Muhammad Rizki

[PATCH v3 11/18] Move bot commands related files to packages/plugins/commands/

[Muhammad Rizki]

So, I want to split files between bot commands and bot callback queries
with their specific directory, it much cleaner than before.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/packages/plugins/{ => commands}/debugger.py    | 0
 daemon/packages/plugins/{ => commands}/manage_atom.py | 0
 daemon/packages/plugins/{ => commands}/scrape.py      | 0
 3 files changed, 0 insertions(+), 0 deletions(-)
 rename daemon/packages/plugins/{ => commands}/debugger.py (100%)
 rename daemon/packages/plugins/{ => commands}/manage_atom.py (100%)
 rename daemon/packages/plugins/{ => commands}/scrape.py (100%)

diff --git a/daemon/packages/plugins/debugger.py b/daemon/packages/plugins/commands/debugger.py
similarity index 100%
rename from daemon/packages/plugins/debugger.py
rename to daemon/packages/plugins/commands/debugger.py
diff --git a/daemon/packages/plugins/manage_atom.py b/daemon/packages/plugins/commands/manage_atom.py
similarity index 100%
rename from daemon/packages/plugins/manage_atom.py
rename to daemon/packages/plugins/commands/manage_atom.py
diff --git a/daemon/packages/plugins/scrape.py b/daemon/packages/plugins/commands/scrape.py
similarity index 100%
rename from daemon/packages/plugins/scrape.py
rename to daemon/packages/plugins/commands/scrape.py
-- 
Muhammad Rizki

[PATCH v3 12/18] Add del_atom callback query

[Muhammad Rizki]

I want to easily remove atom URL by index remotely using bot message
button, easily just press the button number in /del_atom bot command.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/packages/plugins/callbacks/del_atom.py | 31 +++++++++++++++++++
 1 file changed, 31 insertions(+)
 create mode 100644 daemon/packages/plugins/callbacks/del_atom.py

diff --git a/daemon/packages/plugins/callbacks/del_atom.py b/daemon/packages/plugins/callbacks/del_atom.py
new file mode 100644
index 0000000..0595e67
--- /dev/null
+++ b/daemon/packages/plugins/callbacks/del_atom.py
@@ -0,0 +1,31 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Copyright (C) 2022  Muhammad Rizki <riskimuhammmad1@gmail.com>
+#
+
+from packages import DaemonClient
+from scraper import utils
+from pyrogram.types import CallbackQuery
+
+
+@DaemonClient.on_callback_query(group=1)
+async def on_del_atom(c: DaemonClient, cb: CallbackQuery):
+	if not "del_atom" in cb.data:
+		return
+
+	atoms = c.db.get_atom_urls()
+	if len(atoms) == 0:
+		return await cb.answer("Currently empty.")
+
+	index = int(utils.remove_command(cb.data))
+	atom = atoms[index - 1]
+	c.db.delete_atom(atom)
+
+	await cb.answer(
+		f"Success delete {atom} from listening new email",
+		show_alert=True
+	)
+
+	await cb.message.delete()
+	if cb.message.reply_to_message:
+		await cb.message.reply_to_message.delete()
-- 
Muhammad Rizki

[PATCH v3 13/18] Add insert_broadcast() in scraper/db.py

[Muhammad Rizki]

Add insert_broadcast() in scraper/db.py to insert the chat room for
receiving new email into the database.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/scraper/db.py | 46 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)

diff --git a/daemon/scraper/db.py b/daemon/scraper/db.py
index 24fc2c6..a0f8d15 100644
--- a/daemon/scraper/db.py
+++ b/daemon/scraper/db.py
@@ -5,6 +5,7 @@
 #
 
 from datetime import datetime
+from typing import Union
 import mysql
 
 
@@ -153,6 +154,51 @@ class Db():
 		return [u[0] for u in urls]
 
 
+	def insert_broadcast(
+		self,
+		chat_id: int,
+		name: str,
+		type: str,
+		created_at: "datetime",
+		username: str = None,
+		link: str = None,
+	):
+		try:
+			return self.__save_broadcast(
+				chat_id=chat_id,
+				name=name,
+				type=type,
+				created_at=created_at,
+				username=username,
+				link=link
+			)
+		except mysql.connector.errors.IntegrityError:
+			#
+			# Duplicate data, skip!
+			#
+			return None
+
+
+	def __save_broadcast(
+		self,
+		chat_id: int,
+		name: str,
+		type: str,
+		created_at: "datetime",
+		username: str = None,
+		link: str = None,
+	):
+		q = """
+			INSERT INTO broadcast_chats
+			(chat_id, username, name, type, link, created_at)
+			VALUES
+			(%s, %s, %s, %s, %s, %s)
+		"""
+		values = (chat_id, username, name, type, link, created_at)
+		self.cur.execute(q, values)
+		return self.cur.lastrowid
+
+
 	def get_broadcast_chats(self):
 		q = """
 			SELECT *
-- 
Muhammad Rizki

[PATCH v3 14/18] Add delete_broadcast() in scraper/db.py

[Muhammad Rizki]

Add delete_broadcast() for delete a chat in the database from receiving
email messages.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/scraper/db.py | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/daemon/scraper/db.py b/daemon/scraper/db.py
index a0f8d15..15b0f99 100644
--- a/daemon/scraper/db.py
+++ b/daemon/scraper/db.py
@@ -199,6 +199,15 @@ class Db():
 		return self.cur.lastrowid
 
 
+	def delete_broadcast(self, chat_id: int):
+		q = """
+			DELETE FROM broadcast_chats
+			WHERE chat_id = %(chat_id)s
+		"""
+		self.cur.execute(q, {"chat_id": chat_id})
+		return self.cur.rowcount > 0
+
+
 	def get_broadcast_chats(self):
 		q = """
 			SELECT *
-- 
Muhammad Rizki

[PATCH v3 15/18] Create /add_bc bot command

[Muhammad Rizki]

Create /add_bc bot command for adding the current chat into the database

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 .../plugins/commands/manage_broadcast.py      | 35 +++++++++++++++++++
 1 file changed, 35 insertions(+)
 create mode 100644 daemon/packages/plugins/commands/manage_broadcast.py

diff --git a/daemon/packages/plugins/commands/manage_broadcast.py b/daemon/packages/plugins/commands/manage_broadcast.py
new file mode 100644
index 0000000..8397508
--- /dev/null
+++ b/daemon/packages/plugins/commands/manage_broadcast.py
@@ -0,0 +1,35 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Copyright (C) 2022  Muhammad Rizki <riskimuhammmad1@gmail.com>
+#
+
+from pyrogram.types import Message
+from pyrogram import filters, enums
+from packages import DaemonClient
+
+
+@DaemonClient.on_message(
+	filters.command("add_bc") &
+	(filters.private | filters.group)
+)
+async def add_broadcast(c: DaemonClient, m: Message):
+	if m.chat.type == enums.ChatType.PRIVATE:
+		chat_name = m.chat.first_name
+	else:
+		chat_name = m.chat.title
+
+	inserted = c.db.insert_broadcast(
+		chat_id=m.chat.id,
+		name=chat_name,
+		type=str(m.chat.type),
+		created_at=m.date,
+		username=m.chat.username,
+		link=m.chat.invite_link
+	)
+
+	if inserted is None:
+		msg = f"This chat already added for receiving email messages"
+	else:
+		msg = f"Success add this chat for receiving email messages"
+
+	await m.reply(msg)
-- 
Muhammad Rizki

[PATCH v3 16/18] Add create_chat_link() function

[Muhammad Rizki]

I want to easily create a chat link when the pyrogram's Chat object
sometimes doesn't have an invite_link value (None), so I create it
myself and call it in packages/plugins/commands/manage_broadcast.py

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 .../packages/plugins/commands/manage_broadcast.py |  3 ++-
 daemon/scraper/utils.py                           | 15 ++++++++++++++-
 2 files changed, 16 insertions(+), 2 deletions(-)

diff --git a/daemon/packages/plugins/commands/manage_broadcast.py b/daemon/packages/plugins/commands/manage_broadcast.py
index 8397508..e0cc0dc 100644
--- a/daemon/packages/plugins/commands/manage_broadcast.py
+++ b/daemon/packages/plugins/commands/manage_broadcast.py
@@ -6,6 +6,7 @@
 from pyrogram.types import Message
 from pyrogram import filters, enums
 from packages import DaemonClient
+from scraper import utils
 
 
 @DaemonClient.on_message(
@@ -24,7 +25,7 @@ async def add_broadcast(c: DaemonClient, m: Message):
 		type=str(m.chat.type),
 		created_at=m.date,
 		username=m.chat.username,
-		link=m.chat.invite_link
+		link=utils.create_chat_link(m.chat)
 	)
 
 	if inserted is None:
diff --git a/daemon/scraper/utils.py b/daemon/scraper/utils.py
index d894c32..e1c0c77 100644
--- a/daemon/scraper/utils.py
+++ b/daemon/scraper/utils.py
@@ -4,7 +4,7 @@
 # Copyright (C) 2022  Ammar Faizi <ammarfaizi2@gnuweeb.org>
 #
 
-from pyrogram.types import InlineKeyboardMarkup, InlineKeyboardButton
+from pyrogram.types import Chat, InlineKeyboardMarkup, InlineKeyboardButton
 from email.message import Message
 from typing import Dict
 from slugify import slugify
@@ -263,3 +263,16 @@ def button_numbers(data: list, callback_prefix: str, limit: int = 8):
 
 	buttons = [lst[i:i + limit ] for i in range(0, len(lst), limit)]
 	return InlineKeyboardMarkup(buttons)
+
+
+def create_chat_link(chat: Chat):
+	chat_id_str = str(chat.id).replace("-100","")
+	if chat.invite_link:
+		chat_link = chat.invite_link
+
+	elif chat.username:
+		chat_link = f"t.me/{chat.username}"
+	else:
+		chat_link = f"t.me/c/{chat_id_str}/1"
+
+	return chat_link
-- 
Muhammad Rizki

[PATCH v3 17/18] Add /del_bc bot command

[Muhammad Rizki]

Add /del_bc bot command for manage delete chat from receiving email
message.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 .../plugins/commands/manage_broadcast.py      | 30 +++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/daemon/packages/plugins/commands/manage_broadcast.py b/daemon/packages/plugins/commands/manage_broadcast.py
index e0cc0dc..b13d6ce 100644
--- a/daemon/packages/plugins/commands/manage_broadcast.py
+++ b/daemon/packages/plugins/commands/manage_broadcast.py
@@ -34,3 +34,33 @@ async def add_broadcast(c: DaemonClient, m: Message):
 		msg = f"Success add this chat for receiving email messages"
 
 	await m.reply(msg)
+
+
+@DaemonClient.on_message(
+	filters.command("del_bc") &
+	filters.chat(["kiizuah", "nekoha", -1001673279485])
+)
+async def del_broadcast(c: DaemonClient, m: Message):
+	if "--list" in m.text:
+		chats = c.db.get_broadcast_chats()
+		if len(chats) == 0:
+			return await m.reply("Currently empty.")
+
+		text = "List of chats that will receive email message:\n"
+		for u,i in zip(chats, range(1, len(chats)+1)):
+			text += f"{i}. **[{u[3]}]({u[5]})**\n"
+		
+		text += "\nChoose one of the chat above to delete by index below."
+		return await m.reply(text, reply_markup=utils.button_numbers(
+			data=chats,
+			callback_prefix="del_chat"
+		))
+	
+	success = c.db.delete_broadcast(m.chat.id)
+	if not success:
+		msg = "Failed to delete this chat from receiving email message\n"
+		msg += "Maybe it's already been deleted or not exists."
+	else:
+		msg = "Success delete this chat from receiving email message"
+
+	await m.reply(msg)
-- 
Muhammad Rizki

[PATCH v3 18/18] Add del_chat callback query

[Muhammad Rizki]

I want to easily remove chat by index remotely using bot message button,
easily just press the button number in /del_chat bot command.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/packages/plugins/callbacks/del_chat.py | 36 +++++++++++++++++++
 1 file changed, 36 insertions(+)
 create mode 100644 daemon/packages/plugins/callbacks/del_chat.py

diff --git a/daemon/packages/plugins/callbacks/del_chat.py b/daemon/packages/plugins/callbacks/del_chat.py
new file mode 100644
index 0000000..d8ba569
--- /dev/null
+++ b/daemon/packages/plugins/callbacks/del_chat.py
@@ -0,0 +1,36 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Copyright (C) 2022  Muhammad Rizki <riskimuhammmad1@gmail.com>
+#
+
+from packages import DaemonClient
+from scraper import utils
+from pyrogram.types import CallbackQuery
+
+
+@DaemonClient.on_callback_query(group=2)
+async def on_del_chat(c: DaemonClient, cb: CallbackQuery):
+	if not "del_chat" in cb.data:
+		return
+
+	chats = c.db.get_broadcast_chats()
+	if not bool(chats):
+		return await cb.answer("Currently empty.")
+
+	index = int(utils.remove_command(cb.data))
+	chat = chats[index - 1]
+	chat_id = chat[1]
+	chat_name = chat[3]
+	
+	success = c.db.delete_broadcast(chat_id)
+	if not success:
+		msg = f"Failed to delete {chat_name} from receiving email message\n"
+		msg += "Maybe it's already been deleted or not exists."
+	else:
+		msg = f"Success delete {chat_name} from receiving email message"
+
+	await cb.answer(msg, show_alert=True)
+
+	await cb.message.delete()
+	if cb.message.reply_to_message:
+		await cb.message.reply_to_message.delete()
-- 
Muhammad Rizki

Re: [PATCH v3 16/18] Add create_chat_link() function

[Ammar Faizi]

On 7/29/22 8:40 AM, Muhammad Rizki wrote:
> +def create_chat_link(chat: Chat):
> +	chat_id_str = str(chat.id).replace("-100","")
> +	if chat.invite_link:
> +		chat_link = chat.invite_link
> +
> +	elif chat.username:
> +		chat_link = f"t.me/{chat.username}"
> +	else:
> +		chat_link = f"t.me/c/{chat_id_str}/1"
> +
> +	return chat_link

This can be further simplified. And that replace() should only
be done when needed. Something like this:

def create_chat_link(chat: Chat):
	if chat.invite_link:
		return chat.invite_link

	if chat.username:
		return f"t.me/{chat.username}"

	chat_id_str = str(chat.id).replace("-100","")
	return f"t.me/c/{chat_id_str}/1"

-- 
Ammar Faizi

Re: [PATCH v3 14/18] Add delete_broadcast() in scraper/db.py

[Ammar Faizi]

On 7/29/22 8:40 AM, Muhammad Rizki wrote:
> +	def delete_broadcast(self, chat_id: int):
> +		q = """
> +			DELETE FROM broadcast_chats
> +			WHERE chat_id = %(chat_id)s
> +		"""
> +		self.cur.execute(q, {"chat_id": chat_id})
> +		return self.cur.rowcount > 0

OK, this one looks better. Did you manage to test this?

-- 
Ammar Faizi

Re: [PATCH v3 16/18] Add create_chat_link() function

[Muhammad Rizki]

On 30/07/2022 08.22, Ammar Faizi wrote:
> On 7/29/22 8:40 AM, Muhammad Rizki wrote:
>> +def create_chat_link(chat: Chat):
>> +    chat_id_str = str(chat.id).replace("-100","")
>> +    if chat.invite_link:
>> +        chat_link = chat.invite_link
>> +
>> +    elif chat.username:
>> +        chat_link = f"t.me/{chat.username}"
>> +    else:
>> +        chat_link = f"t.me/c/{chat_id_str}/1"
>> +
>> +    return chat_link
> 
> This can be further simplified. And that replace() should only
> be done when needed. Something like this:
> 
> def create_chat_link(chat: Chat):
>      if chat.invite_link:
>          return chat.invite_link
> 
>      if chat.username:
>          return f"t.me/{chat.username}"
> 
>      chat_id_str = str(chat.id).replace("-100","")
>      return f"t.me/c/{chat_id_str}/1"
> 

Thanks, I'll improve the code.

Re: [PATCH v3 01/18] Add new table for atom URLs and broadcast chats in db.sql

[Ammar Faizi]

On 7/29/22 8:40 AM, Muhammad Rizki wrote:
> diff --git a/.gitignore b/.gitignore
> index ca28962..511dbbe 100644
> --- a/.gitignore
> +++ b/.gitignore
> @@ -136,4 +136,8 @@ data.json
>   # VSCode editor
>   .vscode/
>   
> +# PATCH
>   *.patch
> +
> +# RFC directory
> +RFC
> \ No newline at end of file

This .gitignore part *should not* be in the same patch with this.

-- 
Ammar Faizi

Re: [PATCH v3 14/18] Add delete_broadcast() in scraper/db.py

[Muhammad Rizki]

On 30/07/2022 08.23, Ammar Faizi wrote:
> On 7/29/22 8:40 AM, Muhammad Rizki wrote:
>> +    def delete_broadcast(self, chat_id: int):
>> +        q = """
>> +            DELETE FROM broadcast_chats
>> +            WHERE chat_id = %(chat_id)s
>> +        """
>> +        self.cur.execute(q, {"chat_id": chat_id})
>> +        return self.cur.rowcount > 0
> 
> OK, this one looks better. Did you manage to test this?
> 

Yes, I did, and it works the same, thanks.

Re: [PATCH v3 01/18] Add new table for atom URLs and broadcast chats in db.sql

[Ammar Faizi]

On 7/30/22 8:26 AM, Ammar Faizi wrote:
> On 7/29/22 8:40 AM, Muhammad Rizki wrote:
>> diff --git a/.gitignore b/.gitignore
>> index ca28962..511dbbe 100644
>> --- a/.gitignore
>> +++ b/.gitignore
>> @@ -136,4 +136,8 @@ data.json
>>   # VSCode editor
>>   .vscode/
>> +# PATCH
>>   *.patch
>> +
>> +# RFC directory
>> +RFC
>> \ No newline at end of file
> 
> This .gitignore part *should not* be in the same patch with this.

Also, the .gitignore content should end with an LF char (enter/newline).

-- 
Ammar Faizi

Re: [PATCH v3 01/18] Add new table for atom URLs and broadcast chats in db.sql

[Muhammad Rizki]

On 30/07/2022 08.26, Ammar Faizi wrote:
> On 7/29/22 8:40 AM, Muhammad Rizki wrote:
>> diff --git a/.gitignore b/.gitignore
>> index ca28962..511dbbe 100644
>> --- a/.gitignore
>> +++ b/.gitignore
>> @@ -136,4 +136,8 @@ data.json
>>   # VSCode editor
>>   .vscode/
>> +# PATCH
>>   *.patch
>> +
>> +# RFC directory
>> +RFC
>> \ No newline at end of file
> 
> This .gitignore part *should not* be in the same patch with this.
> 

Oh yes, I will move my RFC out of work root directory to another directory.