[PATCH v1 00/18] New feature to handle atom URLs and broadcast

[PATCH v1 00/18] New feature to handle atom URLs and broadcast

[Muhammad Rizki]

Morning sir,

In this series I want to create a new feature to handle the atom URLs
and broadcast chats, such as add atom, add chat, del_atom and del_chat,
with this feature, we do not manual manage them again, just using
the bot commands and it will update in database.

There are 18 patches in this series:
- Patch 1 is just add atom_broadcast.sql to create tables
- Patch 2 is to add get atom URLs and get broadcast chats
- Patch 3 is to fix MySQL InternalError when query the atom and chats
- Patch 4 is to create /add_atom bot command to atom URL to database
- Patch 5 is to add some utility functions to handle code in /add_atom
- Patch 6 is to refactor db class to easily access over DaemonClient
- Patch 7 is to add insert_atom() in scraper/db.py to insert atom URLs
- Patch 8 is to rename admin.py to debugger.
- Patch 9 is to add delete_atom() in scraper/db.py for /del_atom command
- Patch 10 is to add button_numbers() in utility file to create btn nums
- Patch 11 is to split files between bot commands and bot CallbackQuery
- Patch 12 is to add del_atom for CallbackQuery to handle delete atom
- Patch 13 is to add insert_broadcast() in scraper/db.py to insert chats
- Patch 14 is to add delete_broadcast() in scraper/db.py to delete chats
- Patch 15 is to create /add_bc bot command to add chat to the database
- Patch 16 is to add create_chat_link() in utility file for chat link
- Patch 17 is to add /del_bc bot command to remove chat in the database
- Patch 18 is to add del_chat in bot CallbackQuery to handle delete chat

Already tested and it works fine, please give it a test too if there is
a bug or unexpected errors, thanks.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---

Muhammad Rizki (18):
  Add atom_broadcast.sql
  Add ATOM_URLS and BROADCAST_CHAT methods
  Fix MySQL InternalError: Unread result found
  Create /add_atom command
  Add utility functions for /add_atom
  Refactor database methods
  Add insert_atom() in scraper/db.py
  Rename admin.py to debugger.py
  Add delete_atom() function in scraper/db.py
  Add utility function buton_numbers()
  Move bot commands related files to packages/plugins/commands/
  Add del_atom callback query
  Add insert_broadcast() in scraper/db.py
  Add delete_broadcast() in scraper/db.py
  Create /add_bc bot command
  Add create_chat_link() function
  Add /del_bc bot command
  Add del_chat callback query

 .gitignore                                    |   4 +
 daemon/atom_broadcast.sql                     |  25 ++++
 daemon/packages/client.py                     |   4 +-
 daemon/packages/plugins/callbacks/del_atom.py |  31 +++++
 daemon/packages/plugins/callbacks/del_chat.py |  36 ++++++
 .../{admin.py => commands/debugger.py}        |   0
 .../packages/plugins/commands/manage_atom.py  |  50 ++++++++
 .../plugins/commands/manage_broadcast.py      |  66 ++++++++++
 .../packages/plugins/{ => commands}/scrape.py |   0
 daemon/run.py                                 |  16 +--
 daemon/scraper/bot.py                         |  22 +---
 daemon/scraper/db.py                          | 115 +++++++++++++++++-
 daemon/scraper/utils.py                       |  45 +++++++
 13 files changed, 388 insertions(+), 26 deletions(-)
 create mode 100644 daemon/atom_broadcast.sql
 create mode 100644 daemon/packages/plugins/callbacks/del_atom.py
 create mode 100644 daemon/packages/plugins/callbacks/del_chat.py
 rename daemon/packages/plugins/{admin.py => commands/debugger.py} (100%)
 create mode 100644 daemon/packages/plugins/commands/manage_atom.py
 create mode 100644 daemon/packages/plugins/commands/manage_broadcast.py
 rename daemon/packages/plugins/{ => commands}/scrape.py (100%)


base-commit: f8c5aef0aa85e1262c90f818638d255c3c38373f
-- 
Muhammad Rizki

[PATCH v1 01/18] Add atom_broadcast.sql

[Muhammad Rizki]

Just add an atom_broadcast.sql file to create new table.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 .gitignore                |  4 ++++
 daemon/atom_broadcast.sql | 25 +++++++++++++++++++++++++
 2 files changed, 29 insertions(+)
 create mode 100644 daemon/atom_broadcast.sql

diff --git a/.gitignore b/.gitignore
index ca28962..511dbbe 100644
--- a/.gitignore
+++ b/.gitignore
@@ -136,4 +136,8 @@ data.json
 # VSCode editor
 .vscode/
 
+# PATCH
 *.patch
+
+# RFC directory
+RFC
\ No newline at end of file
diff --git a/daemon/atom_broadcast.sql b/daemon/atom_broadcast.sql
new file mode 100644
index 0000000..580e46f
--- /dev/null
+++ b/daemon/atom_broadcast.sql
@@ -0,0 +1,25 @@
+DROP TABLE IF EXISTS `atom_urls`;
+CREATE TABLE `atom_urls` (
+  `id` bigint unsigned NOT NULL AUTO_INCREMENT,
+  `url` varchar(64) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_520_ci NOT NULL,
+  `created_at` datetime NOT NULL,
+  PRIMARY KEY (`id`),
+  UNIQUE KEY `url` (`url`),
+  KEY `created_at` (`created_at`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_520_ci;
+
+DROP TABLE IF EXISTS `broadcast_chats`;
+CREATE TABLE `broadcast_chats` (
+  `id` bigint unsigned NOT NULL AUTO_INCREMENT,
+  `chat_id` bigint NOT NULL,
+  `username` varchar(32),
+  `name` varchar(64) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_520_ci NOT NULL,
+  `type` varchar(32) NOT NULL,
+  `link` varchar(64),
+  `created_at` datetime NOT NULL,
+  PRIMARY KEY (`id`),
+  UNIQUE KEY `chat_id` (`chat_id`),
+  KEY `created_at` (`created_at`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_520_ci;
+
+-- 2022-07-25 09:26:43
-- 
Muhammad Rizki

[PATCH v1 02/18] Add ATOM_URLS and BROADCAST_CHAT methods

[Muhammad Rizki]

I decided to add ATOM_URLS and BROADCAST_CHAT methods to get their
content in database which will be handle in scraper/bot.py

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/scraper/bot.py | 17 ++++-------------
 daemon/scraper/db.py  | 21 +++++++++++++++++++++
 2 files changed, 25 insertions(+), 13 deletions(-)

diff --git a/daemon/scraper/bot.py b/daemon/scraper/bot.py
index bc9d575..c475e34 100644
--- a/daemon/scraper/bot.py
+++ b/daemon/scraper/bot.py
@@ -21,16 +21,6 @@ class BotMutexes():
 
 
 class Bot():
-	ATOM_URLS = [
-		"https://lore.kernel.org/io-uring/new.atom",
-		"https://lore.kernel.org/linux-sgx/new.atom",
-	]
-
-	TG_CHAT_IDS = [
-		"kiizuah"
-	]
-
-
 	def __init__(self, client: DaemonClient, sched: AsyncIOScheduler,
 			scraper: Scraper, mutexes: BotMutexes, conn):
 		self.client = client
@@ -55,7 +45,7 @@ class Bot():
 
 	async def __run(self):
 		print("[__run]: Running...")
-		for url in self.ATOM_URLS:
+		for url in self.db.get_atom_urls():
 			try:
 				await self.__handle_atom_url(url)
 			except:
@@ -80,10 +70,11 @@ class Bot():
 
 
 	async def __handle_mail(self, url, mail):
-		for tg_chat_id in self.TG_CHAT_IDS:
+		chats = self.db.get_broadcast_chats()
+		for chat in chats:
 			async with self.mutexes.send_to_tg:
 				should_wait = await self.__send_to_tg(url, mail,
-									tg_chat_id)
+									chat[1])
 
 			if should_wait:
 				await asyncio.sleep(1)
diff --git a/daemon/scraper/db.py b/daemon/scraper/db.py
index a79f041..321661f 100644
--- a/daemon/scraper/db.py
+++ b/daemon/scraper/db.py
@@ -112,3 +112,24 @@ class Db():
 			return None
 
 		return res[0]
+
+
+	def get_atom_urls(self):
+		q = """
+			SELECT atom_urls.url
+			FROM atom_urls
+		"""
+		self.cur.execute(q)
+		urls = self.cur.fetchall()
+
+		return [u[0] for u in urls]
+
+
+	def get_broadcast_chats(self):
+		q = """
+			SELECT *
+			FROM broadcast_chats
+		"""
+		self.cur.execute(q)
+
+		return self.cur.fetchall()
-- 
Muhammad Rizki

[PATCH v1 03/18] Fix MySQL InternalError: Unread result found

[Muhammad Rizki]

Use con.cursor(buffered=True) to make it avoid from
mysql.connector.errors.InternalError: Unread result found.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/scraper/db.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/daemon/scraper/db.py b/daemon/scraper/db.py
index 321661f..296559e 100644
--- a/daemon/scraper/db.py
+++ b/daemon/scraper/db.py
@@ -12,7 +12,7 @@ class Db():
 	def __init__(self, conn):
 		self.conn = conn
 		self.conn.autocommit = True
-		self.cur = self.conn.cursor()
+		self.cur = self.conn.cursor(buffered=True)
 
 
 	def __del__(self):
-- 
Muhammad Rizki

[PATCH v1 04/18] Create /add_atom command

[Muhammad Rizki]

Create /add_atom command for adding the atom url to the database

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/packages/plugins/manage_atom.py | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
 create mode 100644 daemon/packages/plugins/manage_atom.py

diff --git a/daemon/packages/plugins/manage_atom.py b/daemon/packages/plugins/manage_atom.py
new file mode 100644
index 0000000..eca3a60
--- /dev/null
+++ b/daemon/packages/plugins/manage_atom.py
@@ -0,0 +1,22 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Copyright (C) 2022  Muhammad Rizki <riskimuhammmad1@gmail.com>
+#
+
+from pyrogram.types import Message
+from pyrogram import filters
+from packages import DaemonClient
+
+@DaemonClient.on_message(
+        filters.command("add_atom") &
+        filters.chat(["kiizuah", "nekoha", -1001673279485])
+)
+async def add_atom_url(c: DaemonClient, m: Message):
+        pass
+        # text = utils.remove_command(m.text)
+        # if not utils.is_atom_url(text):
+        #         return
+        
+        # ###
+        # ### TODO: Muhammad Rizki: Add atom url into the database
+        # ###
-- 
Muhammad Rizki

[PATCH v1 05/18] Add utility functions for /add_atom

[Muhammad Rizki]

Add some utility functions for /add_atom command in the scraper/utils.py
and call it in packages/plugins/add_atom.py

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/packages/plugins/manage_atom.py | 20 ++++++++++----------
 daemon/scraper/utils.py                | 15 +++++++++++++++
 2 files changed, 25 insertions(+), 10 deletions(-)

diff --git a/daemon/packages/plugins/manage_atom.py b/daemon/packages/plugins/manage_atom.py
index eca3a60..a10318f 100644
--- a/daemon/packages/plugins/manage_atom.py
+++ b/daemon/packages/plugins/manage_atom.py
@@ -6,17 +6,17 @@
 from pyrogram.types import Message
 from pyrogram import filters
 from packages import DaemonClient
+from scraper import utils
 
 @DaemonClient.on_message(
-        filters.command("add_atom") &
-        filters.chat(["kiizuah", "nekoha", -1001673279485])
+	filters.command("add_atom") &
+	filters.chat(["kiizuah", "nekoha", -1001673279485])
 )
 async def add_atom_url(c: DaemonClient, m: Message):
-        pass
-        # text = utils.remove_command(m.text)
-        # if not utils.is_atom_url(text):
-        #         return
-        
-        # ###
-        # ### TODO: Muhammad Rizki: Add atom url into the database
-        # ###
+	text = utils.remove_command(m.text)
+	if not utils.is_atom_url(text):
+		return
+
+	###
+	### TODO: Muhammad Rizki: Add atom url into the database
+	###
diff --git a/daemon/scraper/utils.py b/daemon/scraper/utils.py
index 30efb7b..f7de5ba 100644
--- a/daemon/scraper/utils.py
+++ b/daemon/scraper/utils.py
@@ -12,6 +12,7 @@ import uuid
 import os
 import re
 import shutil
+import httpx
 
 
 def get_email_msg_id(mail):
@@ -231,3 +232,17 @@ def extract_email_msg_id(msg_id):
 		return None
 	return ret.group(1)
 
+
+async def is_atom_url(text: str):
+	try:
+		async with httpx.AsyncClient() as ses:
+			res = await ses.get(text)
+			mime = res.headers.get("Content-Type")
+
+			return mime == "application/atom+xml"
+	except: return False
+
+def remove_command(text: str):
+	txt = text.split(" ")
+	txt = text.replace(txt[0] + " ","")
+	return txt
-- 
Muhammad Rizki

[PATCH v1 06/18] Refactor database methods

[Muhammad Rizki]

I decided to refactor the database methods to make it able to access
with DaemonClient, so in that client I just to call one of the database
function just like:

async def add_atom(c: DaemonClient, m: Message):
        c.db.insert_atom("atom_url_here")

The c here is the DaemonClient parameter in the add_atom MessageHandler
function

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/packages/client.py |  4 +++-
 daemon/run.py             | 16 ++++++++--------
 daemon/scraper/bot.py     |  5 ++---
 3 files changed, 13 insertions(+), 12 deletions(-)

diff --git a/daemon/packages/client.py b/daemon/packages/client.py
index d16fe46..282daf6 100644
--- a/daemon/packages/client.py
+++ b/daemon/packages/client.py
@@ -9,14 +9,16 @@ from pyrogram.types import Message, InlineKeyboardMarkup, InlineKeyboardButton
 from typing import Union
 from email.message import Message
 from scraper import utils
+from scraper.db import Db
 from .decorator import handle_flood
 
 
 class DaemonClient(Client):
 	def __init__(self, name: str, api_id: int,
-			api_hash: str, **kwargs):
+		api_hash: str, conn, **kwargs):
 		super().__init__(name, api_id,
 				api_hash, **kwargs)
+		self.db = Db(conn)
 
 
 	@handle_flood
diff --git a/daemon/run.py b/daemon/run.py
index 1151ccd..5360395 100644
--- a/daemon/run.py
+++ b/daemon/run.py
@@ -22,9 +22,15 @@ def main():
 		api_id=int(os.getenv("API_ID")),
 		api_hash=os.getenv("API_HASH"),
 		bot_token=os.getenv("BOT_TOKEN"),
+		conn=connector.connect(
+			host=os.getenv("DB_HOST"),
+			user=os.getenv("DB_USER"),
+			password=os.getenv("DB_PASS"),
+			database=os.getenv("DB_NAME")
+		),
 		plugins=dict(
 			root="packages.plugins"
-		)
+		),
 	)
 
 	sched = AsyncIOScheduler(
@@ -38,13 +44,7 @@ def main():
 		client=client,
 		sched=sched,
 		scraper=Scraper(),
-		mutexes=BotMutexes(),
-		conn=connector.connect(
-			host=os.getenv("DB_HOST"),
-			user=os.getenv("DB_USER"),
-			password=os.getenv("DB_PASS"),
-			database=os.getenv("DB_NAME")
-		)
+		mutexes=BotMutexes()
 	)
 	sched.start()
 	bot.run()
diff --git a/daemon/scraper/bot.py b/daemon/scraper/bot.py
index c475e34..7adfb12 100644
--- a/daemon/scraper/bot.py
+++ b/daemon/scraper/bot.py
@@ -8,7 +8,6 @@ from apscheduler.schedulers.asyncio import AsyncIOScheduler
 from packages import DaemonClient
 from scraper import Scraper
 from . import utils
-from .db import Db
 import asyncio
 import shutil
 import re
@@ -22,12 +21,12 @@ class BotMutexes():
 
 class Bot():
 	def __init__(self, client: DaemonClient, sched: AsyncIOScheduler,
-			scraper: Scraper, mutexes: BotMutexes, conn):
+			scraper: Scraper, mutexes: BotMutexes):
 		self.client = client
 		self.sched = sched
 		self.scraper = scraper
 		self.mutexes = mutexes
-		self.db = Db(conn)
+		self.db = client.db
 		self.isRunnerFixed = False
 
 
-- 
Muhammad Rizki

[PATCH v1 07/18] Add insert_atom() in scraper/db.py

[Muhammad Rizki]

Add insert_atom() in scraper/db.py and call it in
packages/plugins/add_atom.py to insert atom url into the database

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/packages/plugins/manage_atom.py | 18 +++++++++++++-----
 daemon/scraper/db.py                   | 16 ++++++++++++++++
 2 files changed, 29 insertions(+), 5 deletions(-)

diff --git a/daemon/packages/plugins/manage_atom.py b/daemon/packages/plugins/manage_atom.py
index a10318f..934aba9 100644
--- a/daemon/packages/plugins/manage_atom.py
+++ b/daemon/packages/plugins/manage_atom.py
@@ -13,10 +13,18 @@ from scraper import utils
 	filters.chat(["kiizuah", "nekoha", -1001673279485])
 )
 async def add_atom_url(c: DaemonClient, m: Message):
+	if len(m.command) <= 1:
+		tutor = "Please specify the URL\n"
+		tutor += "Example: `/add_atom https://lore.kernel.org/linux-sgx/new.atom`"
+		return await m.reply(tutor)
+
 	text = utils.remove_command(m.text)
-	if not utils.is_atom_url(text):
-		return
+	is_atom = await utils.is_atom_url(text)
+	if not is_atom:
+		return await m.reply("Invalid Atom URL")
+
+	inserted = c.db.insert_atom(text)
+	if inserted is None:
+		return await m.reply(f"This URL already listened for new email.")
 
-	###
-	### TODO: Muhammad Rizki: Add atom url into the database
-	###
+	await m.reply(f"Success add **{text}** for listening new email")
diff --git a/daemon/scraper/db.py b/daemon/scraper/db.py
index 296559e..30ed9a7 100644
--- a/daemon/scraper/db.py
+++ b/daemon/scraper/db.py
@@ -114,6 +114,22 @@ class Db():
 		return res[0]
 
 
+	def insert_atom(self, atom: str):
+		try:
+			return self.__save_atom(atom)
+		except mysql.connector.errors.IntegrityError:
+			#
+			# Duplicate data, skip!
+			#
+			return None
+
+
+	def __save_atom(self, atom: str):
+		q = "INSERT INTO atom_urls (url, created_at) VALUES (%s, %s)"
+		self.cur.execute(q, (atom, datetime.utcnow()))
+		return self.cur.lastrowid
+
+
 	def get_atom_urls(self):
 		q = """
 			SELECT atom_urls.url
-- 
Muhammad Rizki

[PATCH v1 08/18] Rename admin.py to debugger.py

[Muhammad Rizki]

I want to create a plugin file that the name is related to admin, so I
decided to rename packages/plugins/admin.py to
packages/plugins/debugger.py

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/packages/plugins/{admin.py => debugger.py} | 0
 1 file changed, 0 insertions(+), 0 deletions(-)
 rename daemon/packages/plugins/{admin.py => debugger.py} (100%)

diff --git a/daemon/packages/plugins/admin.py b/daemon/packages/plugins/debugger.py
similarity index 100%
rename from daemon/packages/plugins/admin.py
rename to daemon/packages/plugins/debugger.py
-- 
Muhammad Rizki

[PATCH v1 09/18] Add delete_atom() function in scraper/db.py

[Muhammad Rizki]

Add delete_atom() function to remove atom URL in database

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/packages/plugins/manage_atom.py | 19 ++++++++++++++++++-
 daemon/scraper/db.py                   | 12 ++++++++++++
 2 files changed, 30 insertions(+), 1 deletion(-)

diff --git a/daemon/packages/plugins/manage_atom.py b/daemon/packages/plugins/manage_atom.py
index 934aba9..0d2166a 100644
--- a/daemon/packages/plugins/manage_atom.py
+++ b/daemon/packages/plugins/manage_atom.py
@@ -3,7 +3,7 @@
 # Copyright (C) 2022  Muhammad Rizki <riskimuhammmad1@gmail.com>
 #
 
-from pyrogram.types import Message
+from pyrogram.types import Message, InlineKeyboardMarkup, InlineKeyboardButton
 from pyrogram import filters
 from packages import DaemonClient
 from scraper import utils
@@ -28,3 +28,20 @@ async def add_atom_url(c: DaemonClient, m: Message):
 		return await m.reply(f"This URL already listened for new email.")
 
 	await m.reply(f"Success add **{text}** for listening new email")
+
+
+@DaemonClient.on_message(
+	filters.command("del_atom") &
+	filters.chat(["kiizuah", "nekoha", -1001673279485])
+)
+async def del_atom_url(c: DaemonClient, m: Message):
+	atoms = c.db.get_atom_urls()
+	if len(atoms) == 0:
+		return await m.reply("Currently empty.")
+
+	text = "List of atom URL that currently listened:\n"
+	for u,i in zip(atoms, range(1, len(atoms)+1)):
+		text += f"{i}. {u}\n"
+	
+	text += "\nChoose one of the URL above to delete by index below."
+	await m.reply(text)
diff --git a/daemon/scraper/db.py b/daemon/scraper/db.py
index 30ed9a7..24fc2c6 100644
--- a/daemon/scraper/db.py
+++ b/daemon/scraper/db.py
@@ -130,6 +130,18 @@ class Db():
 		return self.cur.lastrowid
 
 
+	def delete_atom(self, atom: str):
+		q = """
+			DELETE FROM atom_urls
+			WHERE url = %(atom)s
+		"""
+		try:
+			self.cur.execute(q, {"atom": atom})
+			return True
+		except:
+			return False
+
+
 	def get_atom_urls(self):
 		q = """
 			SELECT atom_urls.url
-- 
Muhammad Rizki

[PATCH v1 10/18] Add utility function buton_numbers()

[Muhammad Rizki]

Add utility function buton_numbers() for easily create a number of
inline buttons to manage delete atom urls.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/packages/plugins/manage_atom.py |  5 ++++-
 daemon/scraper/utils.py                | 17 +++++++++++++++++
 2 files changed, 21 insertions(+), 1 deletion(-)

diff --git a/daemon/packages/plugins/manage_atom.py b/daemon/packages/plugins/manage_atom.py
index 0d2166a..0b916de 100644
--- a/daemon/packages/plugins/manage_atom.py
+++ b/daemon/packages/plugins/manage_atom.py
@@ -44,4 +44,7 @@ async def del_atom_url(c: DaemonClient, m: Message):
 		text += f"{i}. {u}\n"
 	
 	text += "\nChoose one of the URL above to delete by index below."
-	await m.reply(text)
+	await m.reply(text, reply_markup=utils.button_numbers(
+		data=atoms,
+		callback_prefix="del_atom"
+	))
diff --git a/daemon/scraper/utils.py b/daemon/scraper/utils.py
index f7de5ba..d894c32 100644
--- a/daemon/scraper/utils.py
+++ b/daemon/scraper/utils.py
@@ -4,6 +4,7 @@
 # Copyright (C) 2022  Ammar Faizi <ammarfaizi2@gnuweeb.org>
 #
 
+from pyrogram.types import InlineKeyboardMarkup, InlineKeyboardButton
 from email.message import Message
 from typing import Dict
 from slugify import slugify
@@ -246,3 +247,19 @@ def remove_command(text: str):
 	txt = text.split(" ")
 	txt = text.replace(txt[0] + " ","")
 	return txt
+
+
+def button_numbers(data: list, callback_prefix: str, limit: int = 8):
+	if limit > 8:
+		raise ValueError("Limit value cannot more than 8.")
+
+	lst = []
+	for i in range(1, len(data)+1):
+		button = InlineKeyboardButton(
+			str(i),
+			callback_data=f"{callback_prefix} {i}"
+		)
+		lst.append(button)
+
+	buttons = [lst[i:i + limit ] for i in range(0, len(lst), limit)]
+	return InlineKeyboardMarkup(buttons)
-- 
Muhammad Rizki

[PATCH v1 11/18] Move bot commands related files to packages/plugins/commands/

[Muhammad Rizki]

So, I want to split files between bot commands and bot callback queries
with their specific directory, it much cleaner than before.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/packages/plugins/{ => commands}/debugger.py    | 0
 daemon/packages/plugins/{ => commands}/manage_atom.py | 0
 daemon/packages/plugins/{ => commands}/scrape.py      | 0
 3 files changed, 0 insertions(+), 0 deletions(-)
 rename daemon/packages/plugins/{ => commands}/debugger.py (100%)
 rename daemon/packages/plugins/{ => commands}/manage_atom.py (100%)
 rename daemon/packages/plugins/{ => commands}/scrape.py (100%)

diff --git a/daemon/packages/plugins/debugger.py b/daemon/packages/plugins/commands/debugger.py
similarity index 100%
rename from daemon/packages/plugins/debugger.py
rename to daemon/packages/plugins/commands/debugger.py
diff --git a/daemon/packages/plugins/manage_atom.py b/daemon/packages/plugins/commands/manage_atom.py
similarity index 100%
rename from daemon/packages/plugins/manage_atom.py
rename to daemon/packages/plugins/commands/manage_atom.py
diff --git a/daemon/packages/plugins/scrape.py b/daemon/packages/plugins/commands/scrape.py
similarity index 100%
rename from daemon/packages/plugins/scrape.py
rename to daemon/packages/plugins/commands/scrape.py
-- 
Muhammad Rizki

[PATCH v1 12/18] Add del_atom callback query

[Muhammad Rizki]

I want to easily remove atom URL by index remotely using bot message
button, easily just press the button number in /del_atom bot command.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/packages/plugins/callbacks/del_atom.py | 31 +++++++++++++++++++
 1 file changed, 31 insertions(+)
 create mode 100644 daemon/packages/plugins/callbacks/del_atom.py

diff --git a/daemon/packages/plugins/callbacks/del_atom.py b/daemon/packages/plugins/callbacks/del_atom.py
new file mode 100644
index 0000000..0595e67
--- /dev/null
+++ b/daemon/packages/plugins/callbacks/del_atom.py
@@ -0,0 +1,31 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Copyright (C) 2022  Muhammad Rizki <riskimuhammmad1@gmail.com>
+#
+
+from packages import DaemonClient
+from scraper import utils
+from pyrogram.types import CallbackQuery
+
+
+@DaemonClient.on_callback_query(group=1)
+async def on_del_atom(c: DaemonClient, cb: CallbackQuery):
+	if not "del_atom" in cb.data:
+		return
+
+	atoms = c.db.get_atom_urls()
+	if len(atoms) == 0:
+		return await cb.answer("Currently empty.")
+
+	index = int(utils.remove_command(cb.data))
+	atom = atoms[index - 1]
+	c.db.delete_atom(atom)
+
+	await cb.answer(
+		f"Success delete {atom} from listening new email",
+		show_alert=True
+	)
+
+	await cb.message.delete()
+	if cb.message.reply_to_message:
+		await cb.message.reply_to_message.delete()
-- 
Muhammad Rizki

[PATCH v1 13/18] Add insert_broadcast() in scraper/db.py

[Muhammad Rizki]

Add insert_broadcast() in scraper/db.py to insert the chat room for
receiving new email into the database.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/scraper/db.py | 46 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)

diff --git a/daemon/scraper/db.py b/daemon/scraper/db.py
index 24fc2c6..805b591 100644
--- a/daemon/scraper/db.py
+++ b/daemon/scraper/db.py
@@ -5,6 +5,7 @@
 #
 
 from datetime import datetime
+from typing import Union
 import mysql
 
 
@@ -153,6 +154,51 @@ class Db():
 		return [u[0] for u in urls]
 
 
+	def insert_broadcast(
+		self,
+		chat_id: Union[int, str],
+		name: str,
+		type: str,
+		created_at: "datetime",
+		username: str = None,
+		link: str = None,
+	):
+		try:
+			return self.__save_broadcast(
+				chat_id=chat_id,
+				name=name,
+				type=type,
+				created_at=created_at,
+				username=username,
+				link=link
+			)
+		except mysql.connector.errors.IntegrityError:
+			#
+			# Duplicate data, skip!
+			#
+			return None
+
+
+	def __save_broadcast(
+		self,
+		chat_id: Union[int, str],
+		name: str,
+		type: str,
+		created_at: "datetime",
+		username: str = None,
+		link: str = None,
+	):
+		q = """
+			INSERT INTO broadcast_chats
+			(chat_id, username, name, type, link, created_at)
+			VALUES
+			(%s, %s, %s, %s, %s, %s)
+		"""
+		values = (chat_id, username, name, type, link, created_at)
+		self.cur.execute(q, values)
+		return self.cur.lastrowid
+
+
 	def get_broadcast_chats(self):
 		q = """
 			SELECT *
-- 
Muhammad Rizki

[PATCH v1 14/18] Add delete_broadcast() in scraper/db.py

[Muhammad Rizki]

Add delete_broadcast() for delete a chat in the database from receiving
email messages.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/scraper/db.py | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/daemon/scraper/db.py b/daemon/scraper/db.py
index 805b591..6faedb6 100644
--- a/daemon/scraper/db.py
+++ b/daemon/scraper/db.py
@@ -199,6 +199,24 @@ class Db():
 		return self.cur.lastrowid
 
 
+	def delete_broadcast(self, chat_id: Union[int, str]):
+		get_query = """
+			SELECT id FROM broadcast_chats
+			WHERE chat_id = %(chat_id)s
+		"""
+		delete_query = """
+			DELETE FROM broadcast_chats
+			WHERE chat_id = %(chat_id)s
+		"""
+		self.cur.execute(get_query, {"chat_id": chat_id})
+		exist = self.cur.fetchone()
+		if bool(exist):
+			self.cur.execute(delete_query, {"chat_id": chat_id})
+			return True
+
+		return False
+
+
 	def get_broadcast_chats(self):
 		q = """
 			SELECT *
-- 
Muhammad Rizki

[PATCH v1 15/18] Create /add_bc bot command

[Muhammad Rizki]

Create /add_bc bot command for adding the current chat into the database

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 .../plugins/commands/manage_broadcast.py      | 35 +++++++++++++++++++
 1 file changed, 35 insertions(+)
 create mode 100644 daemon/packages/plugins/commands/manage_broadcast.py

diff --git a/daemon/packages/plugins/commands/manage_broadcast.py b/daemon/packages/plugins/commands/manage_broadcast.py
new file mode 100644
index 0000000..8397508
--- /dev/null
+++ b/daemon/packages/plugins/commands/manage_broadcast.py
@@ -0,0 +1,35 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Copyright (C) 2022  Muhammad Rizki <riskimuhammmad1@gmail.com>
+#
+
+from pyrogram.types import Message
+from pyrogram import filters, enums
+from packages import DaemonClient
+
+
+@DaemonClient.on_message(
+	filters.command("add_bc") &
+	(filters.private | filters.group)
+)
+async def add_broadcast(c: DaemonClient, m: Message):
+	if m.chat.type == enums.ChatType.PRIVATE:
+		chat_name = m.chat.first_name
+	else:
+		chat_name = m.chat.title
+
+	inserted = c.db.insert_broadcast(
+		chat_id=m.chat.id,
+		name=chat_name,
+		type=str(m.chat.type),
+		created_at=m.date,
+		username=m.chat.username,
+		link=m.chat.invite_link
+	)
+
+	if inserted is None:
+		msg = f"This chat already added for receiving email messages"
+	else:
+		msg = f"Success add this chat for receiving email messages"
+
+	await m.reply(msg)
-- 
Muhammad Rizki

[PATCH v1 16/18] Add create_chat_link() function

[Muhammad Rizki]

I want to easily create a chat link when the pyrogram's Chat object
sometimes doesn't have an invite_link value (None), so I create it
myself and call it in packages/plugins/commands/manage_broadcast.py

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 .../packages/plugins/commands/manage_broadcast.py |  3 ++-
 daemon/scraper/utils.py                           | 15 ++++++++++++++-
 2 files changed, 16 insertions(+), 2 deletions(-)

diff --git a/daemon/packages/plugins/commands/manage_broadcast.py b/daemon/packages/plugins/commands/manage_broadcast.py
index 8397508..e0cc0dc 100644
--- a/daemon/packages/plugins/commands/manage_broadcast.py
+++ b/daemon/packages/plugins/commands/manage_broadcast.py
@@ -6,6 +6,7 @@
 from pyrogram.types import Message
 from pyrogram import filters, enums
 from packages import DaemonClient
+from scraper import utils
 
 
 @DaemonClient.on_message(
@@ -24,7 +25,7 @@ async def add_broadcast(c: DaemonClient, m: Message):
 		type=str(m.chat.type),
 		created_at=m.date,
 		username=m.chat.username,
-		link=m.chat.invite_link
+		link=utils.create_chat_link(m.chat)
 	)
 
 	if inserted is None:
diff --git a/daemon/scraper/utils.py b/daemon/scraper/utils.py
index d894c32..e1c0c77 100644
--- a/daemon/scraper/utils.py
+++ b/daemon/scraper/utils.py
@@ -4,7 +4,7 @@
 # Copyright (C) 2022  Ammar Faizi <ammarfaizi2@gnuweeb.org>
 #
 
-from pyrogram.types import InlineKeyboardMarkup, InlineKeyboardButton
+from pyrogram.types import Chat, InlineKeyboardMarkup, InlineKeyboardButton
 from email.message import Message
 from typing import Dict
 from slugify import slugify
@@ -263,3 +263,16 @@ def button_numbers(data: list, callback_prefix: str, limit: int = 8):
 
 	buttons = [lst[i:i + limit ] for i in range(0, len(lst), limit)]
 	return InlineKeyboardMarkup(buttons)
+
+
+def create_chat_link(chat: Chat):
+	chat_id_str = str(chat.id).replace("-100","")
+	if chat.invite_link:
+		chat_link = chat.invite_link
+
+	elif chat.username:
+		chat_link = f"t.me/{chat.username}"
+	else:
+		chat_link = f"t.me/c/{chat_id_str}/1"
+
+	return chat_link
-- 
Muhammad Rizki

[PATCH v1 17/18] Add /del_bc bot command

[Muhammad Rizki]

Add /del_bc bot command for manage delete chat from receiving email
message.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 .../plugins/commands/manage_broadcast.py      | 30 +++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/daemon/packages/plugins/commands/manage_broadcast.py b/daemon/packages/plugins/commands/manage_broadcast.py
index e0cc0dc..b13d6ce 100644
--- a/daemon/packages/plugins/commands/manage_broadcast.py
+++ b/daemon/packages/plugins/commands/manage_broadcast.py
@@ -34,3 +34,33 @@ async def add_broadcast(c: DaemonClient, m: Message):
 		msg = f"Success add this chat for receiving email messages"
 
 	await m.reply(msg)
+
+
+@DaemonClient.on_message(
+	filters.command("del_bc") &
+	filters.chat(["kiizuah", "nekoha", -1001673279485])
+)
+async def del_broadcast(c: DaemonClient, m: Message):
+	if "--list" in m.text:
+		chats = c.db.get_broadcast_chats()
+		if len(chats) == 0:
+			return await m.reply("Currently empty.")
+
+		text = "List of chats that will receive email message:\n"
+		for u,i in zip(chats, range(1, len(chats)+1)):
+			text += f"{i}. **[{u[3]}]({u[5]})**\n"
+		
+		text += "\nChoose one of the chat above to delete by index below."
+		return await m.reply(text, reply_markup=utils.button_numbers(
+			data=chats,
+			callback_prefix="del_chat"
+		))
+	
+	success = c.db.delete_broadcast(m.chat.id)
+	if not success:
+		msg = "Failed to delete this chat from receiving email message\n"
+		msg += "Maybe it's already been deleted or not exists."
+	else:
+		msg = "Success delete this chat from receiving email message"
+
+	await m.reply(msg)
-- 
Muhammad Rizki

[PATCH v1 18/18] Add del_chat callback query

[Muhammad Rizki]

I want to easily remove chat by index remotely using bot message button,
easily just press the button number in /del_chat bot command.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/packages/plugins/callbacks/del_chat.py | 36 +++++++++++++++++++
 1 file changed, 36 insertions(+)
 create mode 100644 daemon/packages/plugins/callbacks/del_chat.py

diff --git a/daemon/packages/plugins/callbacks/del_chat.py b/daemon/packages/plugins/callbacks/del_chat.py
new file mode 100644
index 0000000..d8ba569
--- /dev/null
+++ b/daemon/packages/plugins/callbacks/del_chat.py
@@ -0,0 +1,36 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Copyright (C) 2022  Muhammad Rizki <riskimuhammmad1@gmail.com>
+#
+
+from packages import DaemonClient
+from scraper import utils
+from pyrogram.types import CallbackQuery
+
+
+@DaemonClient.on_callback_query(group=2)
+async def on_del_chat(c: DaemonClient, cb: CallbackQuery):
+	if not "del_chat" in cb.data:
+		return
+
+	chats = c.db.get_broadcast_chats()
+	if not bool(chats):
+		return await cb.answer("Currently empty.")
+
+	index = int(utils.remove_command(cb.data))
+	chat = chats[index - 1]
+	chat_id = chat[1]
+	chat_name = chat[3]
+	
+	success = c.db.delete_broadcast(chat_id)
+	if not success:
+		msg = f"Failed to delete {chat_name} from receiving email message\n"
+		msg += "Maybe it's already been deleted or not exists."
+	else:
+		msg = f"Success delete {chat_name} from receiving email message"
+
+	await cb.answer(msg, show_alert=True)
+
+	await cb.message.delete()
+	if cb.message.reply_to_message:
+		await cb.message.reply_to_message.delete()
-- 
Muhammad Rizki

Re: [PATCH v1 01/18] Add atom_broadcast.sql

[Ammar Faizi]

On 7/27/22 9:46 AM, Muhammad Rizki wrote:
> Just add an atom_broadcast.sql file to create new table.
> 
> Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
> ---
>   .gitignore                |  4 ++++
>   daemon/atom_broadcast.sql | 25 +++++++++++++++++++++++++
>   2 files changed, 29 insertions(+)
>   create mode 100644 daemon/atom_broadcast.sql

Please don't create another SQL file, just use the same database with
different table. I really don't see the need to use 2 databases. Hmmm?

> +CREATE TABLE `atom_urls` (
> +  `id` bigint unsigned NOT NULL AUTO_INCREMENT,
> +  `url` varchar(64) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_520_ci NOT NULL,

Just max this one out to varchar(255).

> +  `created_at` datetime NOT NULL,
> +  PRIMARY KEY (`id`),
> +  UNIQUE KEY `url` (`url`),
> +  KEY `created_at` (`created_at`)
> +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_520_ci;
> +
> +DROP TABLE IF EXISTS `broadcast_chats`;
> +CREATE TABLE `broadcast_chats` (
> +  `id` bigint unsigned NOT NULL AUTO_INCREMENT,
> +  `chat_id` bigint NOT NULL,
> +  `username` varchar(32),
> +  `name` varchar(64) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_520_ci NOT NULL,
> +  `type` varchar(32) NOT NULL,

What is the purpose of the `type` field here?

> +  `link` varchar(64),
> +  `created_at` datetime NOT NULL,
> +  PRIMARY KEY (`id`),
> +  UNIQUE KEY `chat_id` (`chat_id`),
> +  KEY `created_at` (`created_at`)
> +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_520_ci;
> +
> +-- 2022-07-25 09:26:43

-- 
Ammar Faizi

Re: [PATCH v1 14/18] Add delete_broadcast() in scraper/db.py

[Ammar Faizi]

On 7/27/22 9:46 AM, Muhammad Rizki wrote:
> +	def delete_broadcast(self, chat_id: Union[int, str]):
> +		get_query = """
> +			SELECT id FROM broadcast_chats
> +			WHERE chat_id = %(chat_id)s
> +		"""
> +		delete_query = """
> +			DELETE FROM broadcast_chats
> +			WHERE chat_id = %(chat_id)s
> +		"""
> +		self.cur.execute(get_query, {"chat_id": chat_id})
> +		exist = self.cur.fetchone()
> +		if bool(exist):
> +			self.cur.execute(delete_query, {"chat_id": chat_id})
> +			return True
> +
> +		return False

What?! What is the purpose of the SELECT query here?

    - You check whether there is at least a row that matches with the chat_id.
    - If there exists, you delete all rows that match with the chat_id.

Why not just run the DELETE query directly? The SELECT query here is useless!

You may be wondering: "How can I return False if there is no matching row?".
Simple: Check the affected rows of the DELETE query.

IOW, something like this:

	def delete_broadcast(self, chat_id: Union[int, str]):
		delete_query = """
			DELETE FROM broadcast_chats WHERE chat_id = %(chat_id)s
		"""
		self.cur.execute(delete_query, {"chat_id": chat_id})
		return self.cur.rowcount > 0

Totally untested, but you get the idea. No?

-- 
Ammar Faizi

Re: [PATCH v1 05/18] Add utility functions for /add_atom

[Ammar Faizi]

On 7/27/22 9:46 AM, Muhammad Rizki wrote:
>   @DaemonClient.on_message(
> -        filters.command("add_atom") &
> -        filters.chat(["kiizuah", "nekoha", -1001673279485])
> +	filters.command("add_atom") &
> +	filters.chat(["kiizuah", "nekoha", -1001673279485])
>   )

I think we should kill this and its friends. Move it to a config.py or
something git-ignored.

-- 
Ammar Faizi

Re: [PATCH v1 16/18] Add create_chat_link() function

[Ammar Faizi]

On 7/27/22 9:46 AM, Muhammad Rizki wrote:
> +def create_chat_link(chat: Chat):
> +	chat_id_str = str(chat.id).replace("-100","")
> +	if chat.invite_link:
> +		chat_link = chat.invite_link
> +
> +	elif chat.username:
> +		chat_link = f"t.me/{chat.username}"
> +	else:
> +		chat_link = f"t.me/c/{chat_id_str}/1"
> +
> +	return chat_link

I don't get it. Can you explain a bit why do we get rid of "-100" here?

-- 
Ammar Faizi

Re: [PATCH v1 05/18] Add utility functions for /add_atom

[Ammar Faizi]

On 7/28/22 7:43 AM, Muhammad Rizki wrote:
> On 27/07/2022 22.09, Ammar Faizi wrote:
>> On 7/27/22 9:46 AM, Muhammad Rizki wrote:
>>>   @DaemonClient.on_message(
>>> -        filters.command("add_atom") &
>>> -        filters.chat(["kiizuah", "nekoha", -1001673279485])
>>> +    filters.command("add_atom") &
>>> +    filters.chat(["kiizuah", "nekoha", -1001673279485])
>>>   )
>>
>> I think we should kill this and its friends. Move it to a config.py or
>> something git-ignored.
>>
> Which code? only `filters.chat()`?

Yes, the filter.chat().

> I was about to make an RFC about to make `add admin` feature and
> create a custom filter like `filters.command() & custom_filter.admin`.
OK, keep it like this until you do your RFC later.

-- 
Ammar Faizi

Re: [PATCH v1 16/18] Add create_chat_link() function

[Ammar Faizi]

On 7/28/22 7:47 AM, Muhammad Rizki wrote:
> To make a chat link, you must remove the -100. You can't make a chat link like this, for example:
> - This link is works http://t.me/c/1673279485/1
> - This link doesn't works http://t.me/c/-1001673279485/1

Understand, thanks!

-- 
Ammar Faizi

Re: [PATCH v1 01/18] Add atom_broadcast.sql

[Muhammad Rizki]

On 27/07/2022 21.45, Ammar Faizi wrote:
> On 7/27/22 9:46 AM, Muhammad Rizki wrote:
>> Just add an atom_broadcast.sql file to create new table.
>>
>> Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
>> ---
>>   .gitignore                |  4 ++++
>>   daemon/atom_broadcast.sql | 25 +++++++++++++++++++++++++
>>   2 files changed, 29 insertions(+)
>>   create mode 100644 daemon/atom_broadcast.sql
> 
> Please don't create another SQL file, just use the same database with
> different table. I really don't see the need to use 2 databases. Hmmm?
> 

Oh yes I forget to combine the SQL with the old one, I will amend 
thiscommit and merge atom_broadcast.sql with the old one.

>> +CREATE TABLE `atom_urls` (
>> +  `id` bigint unsigned NOT NULL AUTO_INCREMENT,
>> +  `url` varchar(64) CHARACTER SET utf8mb4 COLLATE 
>> utf8mb4_unicode_520_ci NOT NULL,
> 
> Just max this one out to varchar(255).
> 

Noted, thanks!

>> +  `created_at` datetime NOT NULL,
>> +  PRIMARY KEY (`id`),
>> +  UNIQUE KEY `url` (`url`),
>> +  KEY `created_at` (`created_at`)
>> +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_520_ci;
>> +
>> +DROP TABLE IF EXISTS `broadcast_chats`;
>> +CREATE TABLE `broadcast_chats` (
>> +  `id` bigint unsigned NOT NULL AUTO_INCREMENT,
>> +  `chat_id` bigint NOT NULL,
>> +  `username` varchar(32),
>> +  `name` varchar(64) CHARACTER SET utf8mb4 COLLATE 
>> utf8mb4_unicode_520_ci NOT NULL,
>> +  `type` varchar(32) NOT NULL,
> 
> What is the purpose of the `type` field here?
> 

It's a chat type to define is it a group chat or private chat, so what 
I'm thinking is, if someone want to receive via private chat I just
make the `type` column. If this is not necessary I can just amend
this commit and edit all related code.

>> +  `link` varchar(64),
>> +  `created_at` datetime NOT NULL,
>> +  PRIMARY KEY (`id`),
>> +  UNIQUE KEY `chat_id` (`chat_id`),
>> +  KEY `created_at` (`created_at`)
>> +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_520_ci;
>> +
>> +-- 2022-07-25 09:26:43
> 

Re: [PATCH v1 01/18] Add atom_broadcast.sql

[Ammar Faizi]

On 7/28/22 12:00 PM, Muhammad Rizki wrote:
> It's a chat type to define is it a group chat or private chat, so what I'm thinking is, if someone want to receive via private chat I just
> make the `type` column. If this is not necessary I can just amend
> this commit and edit all related code.

Sounds fine by me. Go on.

-- 
Ammar Faizi

Re: [PATCH v1 05/18] Add utility functions for /add_atom

[Ammar Nofan Faizi]

On 7/28/22 7:43 AM, Muhammad Rizki wrote:
>> I think we should kill this and its friends. Move it to a config.py or
>> something git-ignored.
>>
> Which code? only `filters.chat()`? I was about to make an RFC about
> to make `add admin` feature and create a custom filter like
> `filters.command() & custom_filter.admin`.

BTW, just a reminder on this. I suggest you to finish it and get the rest
of your remuneration. You won't run out of things to work on anytime soon.
I still have a lot of things which I want you to do it.

Just finish it, please...

-- 
Ammar Faizi

Re: [PATCH v1 05/18] Add utility functions for /add_atom

[Muhammad Rizki]

On 11/08/2022 05.05, Ammar Nofan Faizi wrote:
> On 7/28/22 7:43 AM, Muhammad Rizki wrote:
>>> I think we should kill this and its friends. Move it to a config.py or
>>> something git-ignored.
>>>
>> Which code? only `filters.chat()`? I was about to make an RFC about
>> to make `add admin` feature and create a custom filter like
>> `filters.command() & custom_filter.admin`.
> 
> BTW, just a reminder on this. I suggest you to finish it and get the rest
> of your remuneration. You won't run out of things to work on anytime soon.
> I still have a lot of things which I want you to do it.
> 
> Just finish it, please...
> 

I was about to make that RFC, but I'm thinking again that this feature 
will not be use, because it is just an admin management feature, we know 
that bot only maintained by 3 people: You, Alviro, and Me. But, if you 
really want this feature I will continue this, thanks for the reminder.

Re: [PATCH v1 05/18] Add utility functions for /add_atom

[Ammar Nofan Faizi]

On 8/11/22 9:41 AM, Muhammad Rizki wrote:
> I was about to make that RFC, but I'm thinking again that this feature 
> will not be use, because it is just an admin management feature, we know 
> that bot only maintained by 3 people: You, Alviro, and Me. But, if you 
> really want this feature I will continue this, thanks for the reminder.

I understand. Please just make it git-ignored. I do not ever want to see
this admin filter in the core code. You have to make it not committed!

Please just create a config.py file and put it in the .gitignore, make it
our responsibility to create a config.py every time we setup the environment.

-- 
Ammar Faizi

Re: [PATCH v1 05/18] Add utility functions for /add_atom

[Ammar Nofan Faizi]

On 8/11/22 9:47 AM, Ammar Nofan Faizi wrote:
> On 8/11/22 9:41 AM, Muhammad Rizki wrote:
>> I was about to make that RFC, but I'm thinking again that this feature 
>> will not be use, because it is just an admin management feature, we know 
>> that bot only maintained by 3 people: You, Alviro, and Me. But, if you 
>> really want this feature I will continue this, thanks for the reminder.
> 
> I understand. Please just make it git-ignored. I do not ever want to see
> this admin filter in the core code. You have to make it not committed!
> 
> Please just create a config.py file and put it in the .gitignore, make it
> our responsibility to create a config.py every time we setup the environment.

Remember, "config.py" should not be committed, but you have to create a template
"config.py.example" so we can just copy it whenever we setup it. Understand?

I'll wait for your patch.

-- 
Ammar Faizi

[PATCH v1 05/18] Add utility functions for /add_atom

[Muhammad Rizki]

Add some utility functions for /add_atom command in the scraper/utils.py
and call it in packages/plugins/add_atom.py

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/packages/plugins/manage_atom.py | 20 ++++++++++----------
 daemon/scraper/utils.py                | 15 +++++++++++++++
 2 files changed, 25 insertions(+), 10 deletions(-)

diff --git a/daemon/packages/plugins/manage_atom.py b/daemon/packages/plugins/manage_atom.py
index eca3a60..a10318f 100644
--- a/daemon/packages/plugins/manage_atom.py
+++ b/daemon/packages/plugins/manage_atom.py
@@ -6,17 +6,17 @@
 from pyrogram.types import Message
 from pyrogram import filters
 from packages import DaemonClient
+from scraper import utils
 
 @DaemonClient.on_message(
-        filters.command("add_atom") &
-        filters.chat(["kiizuah", "nekoha", -1001673279485])
+	filters.command("add_atom") &
+	filters.chat(["kiizuah", "nekoha", -1001673279485])
 )
 async def add_atom_url(c: DaemonClient, m: Message):
-        pass
-        # text = utils.remove_command(m.text)
-        # if not utils.is_atom_url(text):
-        #         return
-        
-        # ###
-        # ### TODO: Muhammad Rizki: Add atom url into the database
-        # ###
+	text = utils.remove_command(m.text)
+	if not utils.is_atom_url(text):
+		return
+
+	###
+	### TODO: Muhammad Rizki: Add atom url into the database
+	###
diff --git a/daemon/scraper/utils.py b/daemon/scraper/utils.py
index 30efb7b..f7de5ba 100644
--- a/daemon/scraper/utils.py
+++ b/daemon/scraper/utils.py
@@ -12,6 +12,7 @@ import uuid
 import os
 import re
 import shutil
+import httpx
 
 
 def get_email_msg_id(mail):
@@ -231,3 +232,17 @@ def extract_email_msg_id(msg_id):
 		return None
 	return ret.group(1)
 
+
+async def is_atom_url(text: str):
+	try:
+		async with httpx.AsyncClient() as ses:
+			res = await ses.get(text)
+			mime = res.headers.get("Content-Type")
+
+			return mime == "application/atom+xml"
+	except: return False
+
+def remove_command(text: str):
+	txt = text.split(" ")
+	txt = text.replace(txt[0] + " ","")
+	return txt
-- 
Muhammad Rizki