* [PATCH v3 00/11] Improvements, refactors, and features
@ 2023-01-08 8:07 Muhammad Rizki
2023-01-08 8:07 ` [PATCH v3 01/11] discord: Remove unnecessary try/except block Muhammad Rizki
` (11 more replies)
0 siblings, 12 replies; 13+ messages in thread
From: Muhammad Rizki @ 2023-01-08 8:07 UTC (permalink / raw)
To: Ammar Faizi
Cc: Muhammad Rizki, Alviro Iskandar Setiawan, GNU/Weeb Mailing List
Hir sir,
This is v3 of improvements, refactors, and features. This series is to
add more `except` block for both Discord and Telegram, and fix some
grammatical errors in the commit messages.
In the last series, I didn't care so much with the Discord
implementation. So, I forget to add a reconnect MySQL method for the
Discord bot. In this series, I've added it and implement it in the
listener.py exactly in the handle_db_error().
In the last commit of the series, Telegram report_err() is implemented
in the listener.py. In this series, I've been move it to the Telegram
bot custom client instead. I've added a report_err() for the Discord bot
too in the client.
I've added DaemonException() error class too to easily save a thread URL
if errors may happen, this error class is to easily reproducing errors.
Some unnecessary try/except block is already removed and add the
try/except block again in the __handle_atom_url(), we need to catch the
thread URL to easily reproducing errors if errors may happen in the
future.
I've implemented handle_db_error() for the Discord bot too.
What's new?
- Added a report_err() for the Discord bot.
- Added a DaemonException() error class for both bots.
- Added a ping() for reconnecting MySQL conn for the Discord bot.
- Added a handle_db_error() for the Discord bot mail listener.
Code Refactors:
v3: No code refactoring is needed.
v2:
1) Move Telegram bot connection to the listener's run() function.
We use start(), idle() and stop() rather than the run() that
are the same with 3 of them. The different is, we need to connect
to the Telegram server first, then we add a scheduler to listen a
new email message, and then idle() the Telegram bot to be able to
manage the atom URLs and broadcast chats. Why ordering from
start() -> idle() -> stop()? As I said, we need to connect to the
Telegram server first, since the code only is an initialization,
we shouldn't worry about them, they are the same as pryogram's
run().
2) Change the report_err() codes for both Telegram and Discord, we
use DaemonException as it's argument type.
v1:
1) Move report_err() to the DaemonTelegram() for the Telegram bot.
2) Remove try/except for both bots in their listener.py.
3) Ensure the Telegram bot has been started, so start the client
earlier.
4) Rename the Telegram bot custom client to "DaemonTelegram".
There are 11 patches in this series:
- Patch 1 is to remove unnecessary try/except in the listener for DC bot.
- Patch 2 is to change the TG runner method to perform graceful exit.
- Patch 3 is to add a DaemonException() error class to handle error.
- Patch 4 is to refactor report_err() and move it to the DaemonClient().
- Patch 5 is to implement the DaemonException() and change report_err().
- Patch 6 is to remove unnecessary try/except in the listener for TG bot.
- Patch 7 is to add a report_err() for the Discord bot.
- Patch 8 is to implement report_err() and DaemonException() for DC bot.
- Patch 9 is to add a ping() for reconnecting the MySQL connection.
- Patch 10 is to add a handle_db_error() to handle the MySQL errors.
- Patch 11 is to rename the Telegram bot custom client.
How to set log channel ID for the Telegram bot:
1. Make a channel for log in the Telegram.
2. Copy the channel ID, if you unable to see the ID you can use API or
bot who has the feature, or use a 64gram desktop.
3. Paste the copied ID into the config.py in the telegram directory.
How to set log channel ID for the Discord bot:
1. Make a channel for log in your Discord server.
2. Copy the channel ID by right click and copy, if you unable to get the
ID you can go to your account settings, then enable the
`Developer mode` or use bot who has the feature.
3. Paste the copied ID into the config.py in the dscord directory.
How to use:
1. Execute the db.sql file in the daemon directory.
2. Setup .env file, the example is there with suffix .example, this
file name must remove the suffix name .example.
3. Set up the config.py in each bot directory, such as dscord and
telegram. The example is there with suffix .example & the file name
must remove suffix name .example.
4. Run `pip3 install -r requirements.txt` in each bot directory.
5. STORAGE_DIR env value must `storage` to make it work fine.
6. Run the bot by `python3 dc.py` or `python3 tg.py`.
Please give it a test and give me more comments or suggestion, thanks!
## Changelog
v2 -> v3:
- Add `except` block for both Discord and Telegram in the listener.py
- Fix grammatical errors for some commit messages
- Fix the ack certification prefix for Viro
v1 -> v2:
- Move Telegram bot runner
- Change the report_err() codes for both Telegram and Discord
- Change all commits title prefixes
- Change some commits descriptions
Signed-off-by: Muhammad Rizki <[email protected]>
---
Muhammad Rizki (11):
discord: Remove unnecessary try/except block
telegram: Perform graceful exit when interrupted by a signal
Add DaemonException() class
telegram: Refactor report_err() and move to the DaemonClient() class
telegram: Implement the DaemonException() and refactor report_err()
telegram: listener: Remove unnecessary try/except block
discord: add report_err() for the Discord bot
discord: Implement the report_err() and DaemonException() error class
discord: database: Add ping() method for reconnect MySQL
discord: listener: Add handle_db_error to handle the MySQL errors
telegram: Rename the Telegram bot custom client
daemon/dscord/database/core.py | 6 +++
daemon/dscord/gnuweeb/client.py | 8 +++
daemon/dscord/mailer/listener.py | 53 ++++++++++++++-----
daemon/exceptions.py | 18 +++++++
daemon/telegram/mailer/listener.py | 45 +++++++++-------
daemon/telegram/packages/__init__.py | 2 +-
daemon/telegram/packages/client.py | 10 +++-
daemon/telegram/packages/decorator.py | 2 +-
.../packages/plugins/callbacks/del_atom.py | 6 +--
.../packages/plugins/callbacks/del_chat.py | 6 +--
.../packages/plugins/commands/manage_atom.py | 10 ++--
.../plugins/commands/manage_broadcast.py | 10 ++--
.../packages/plugins/commands/scrape.py | 6 +--
daemon/tg.py | 8 ++-
14 files changed, 131 insertions(+), 59 deletions(-)
create mode 100644 daemon/exceptions.py
base-commit: d72337c346686b7faa48ee654c967dba1dfebea0
--
Muhammad Rizki
^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH v3 01/11] discord: Remove unnecessary try/except block
2023-01-08 8:07 [PATCH v3 00/11] Improvements, refactors, and features Muhammad Rizki
@ 2023-01-08 8:07 ` Muhammad Rizki
2023-01-08 8:07 ` [PATCH v3 02/11] telegram: Perform graceful exit when interrupted by a signal Muhammad Rizki
` (10 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Muhammad Rizki @ 2023-01-08 8:07 UTC (permalink / raw)
To: Ammar Faizi
Cc: Muhammad Rizki, Alviro Iskandar Setiawan, GNU/Weeb Mailing List
We don't need this anymore since it already has a try/except earlier.
Signed-off-by: Muhammad Rizki <[email protected]>
---
daemon/dscord/mailer/listener.py | 9 +--------
1 file changed, 1 insertion(+), 8 deletions(-)
diff --git a/daemon/dscord/mailer/listener.py b/daemon/dscord/mailer/listener.py
index 24f0a6c..a225f46 100644
--- a/daemon/dscord/mailer/listener.py
+++ b/daemon/dscord/mailer/listener.py
@@ -106,14 +106,7 @@ class Listener:
self.logger.debug(md)
return False
- try:
- text, files, is_patch = utils.create_template(mail, Platform.DISCORD)
- except:
- exc_str = utils.catch_err()
- self.client.logger.warning(exc_str)
- await self.client.send_log_file(url)
- return
-
+ text, files, is_patch = utils.create_template(mail, Platform.DISCORD)
reply_to = self.get_discord_reply(mail, dc_chat_id)
url = str(re.sub(r"/raw$", "", url))
--
Muhammad Rizki
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH v3 02/11] telegram: Perform graceful exit when interrupted by a signal
2023-01-08 8:07 [PATCH v3 00/11] Improvements, refactors, and features Muhammad Rizki
2023-01-08 8:07 ` [PATCH v3 01/11] discord: Remove unnecessary try/except block Muhammad Rizki
@ 2023-01-08 8:07 ` Muhammad Rizki
2023-01-08 8:07 ` [PATCH v3 03/11] Add DaemonException() class Muhammad Rizki
` (9 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Muhammad Rizki @ 2023-01-08 8:07 UTC (permalink / raw)
To: Ammar Faizi
Cc: Muhammad Rizki, Alviro Iskandar Setiawan, GNU/Weeb Mailing List
We need to start the Telegram bot before starting the listener. This
way, if an error occurs, especially while parsing the email message, the
listener will be able to send a report message to the channel, since the
Telegram bot will already be running.
Also, move them to the run() function in the listener.
Acked-by: Alviro Iskandar Setiawan <[email protected]>
Signed-off-by: Muhammad Rizki <[email protected]>
---
daemon/telegram/mailer/listener.py | 7 +++++++
daemon/tg.py | 4 +---
2 files changed, 8 insertions(+), 3 deletions(-)
diff --git a/daemon/telegram/mailer/listener.py b/daemon/telegram/mailer/listener.py
index 044de6a..6482a66 100644
--- a/daemon/telegram/mailer/listener.py
+++ b/daemon/telegram/mailer/listener.py
@@ -4,6 +4,7 @@
# Copyright (C) 2022 Ammar Faizi <[email protected]>
#
+from pyrogram import idle
from pyrogram.types import Message
from mysql.connector.errors import OperationalError, DatabaseError
from apscheduler.schedulers.asyncio import AsyncIOScheduler
@@ -37,12 +38,18 @@ class Bot():
# Execute __run() once to avoid high latency at
# initilization.
#
+ self.sched.start()
+ self.client.start()
+
self.runner = self.sched.add_job(
func=self.__run,
misfire_grace_time=None,
max_instances=1
)
+ idle()
+ self.client.stop()
+
async def handle_db_error(self, e):
#
diff --git a/daemon/tg.py b/daemon/tg.py
index 73ff2b9..a676cf5 100644
--- a/daemon/tg.py
+++ b/daemon/tg.py
@@ -7,6 +7,7 @@
from apscheduler.schedulers.asyncio import AsyncIOScheduler
from dotenv import load_dotenv
from mysql import connector
+from pyrogram import idle
from atom import Scraper
from telegram.packages import DaemonClient
from telegram.mailer import BotMutexes
@@ -58,10 +59,7 @@ def main():
scraper=Scraper(),
mutexes=BotMutexes()
)
- sched.start()
bot.run()
- client.run()
-
if __name__ == '__main__':
main()
--
Muhammad Rizki
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH v3 03/11] Add DaemonException() class
2023-01-08 8:07 [PATCH v3 00/11] Improvements, refactors, and features Muhammad Rizki
2023-01-08 8:07 ` [PATCH v3 01/11] discord: Remove unnecessary try/except block Muhammad Rizki
2023-01-08 8:07 ` [PATCH v3 02/11] telegram: Perform graceful exit when interrupted by a signal Muhammad Rizki
@ 2023-01-08 8:07 ` Muhammad Rizki
2023-01-08 8:07 ` [PATCH v3 04/11] telegram: Refactor report_err() and move to the DaemonClient() class Muhammad Rizki
` (8 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Muhammad Rizki @ 2023-01-08 8:07 UTC (permalink / raw)
To: Ammar Faizi
Cc: Muhammad Rizki, Alviro Iskandar Setiawan, GNU/Weeb Mailing List
Add DaemonException() class in the daemon/exceptions.py to be able to
throw custom error, the use case is to easily catch the email thread
URL.
This error class is reusable for both Discord and Telegram bot.
Signed-off-by: Muhammad Rizki <[email protected]>
---
daemon/exceptions.py | 18 ++++++++++++++++++
1 file changed, 18 insertions(+)
create mode 100644 daemon/exceptions.py
diff --git a/daemon/exceptions.py b/daemon/exceptions.py
new file mode 100644
index 0000000..34f6823
--- /dev/null
+++ b/daemon/exceptions.py
@@ -0,0 +1,18 @@
+from typing import Optional
+
+class DaemonException(Exception):
+ thread_url: Optional[str] = None
+ atom_url: Optional[str] = None
+ original_exception: Optional[str] = None
+
+ def __str__(self) -> str:
+ return self.original_exception
+
+ def set_atom_url(self, url: str):
+ self.atom_url = url
+
+ def set_thread_url(self, url: str):
+ self.thread_url = url
+
+ def set_message(self, msg: str):
+ self.original_exception = msg
--
Muhammad Rizki
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH v3 04/11] telegram: Refactor report_err() and move to the DaemonClient() class
2023-01-08 8:07 [PATCH v3 00/11] Improvements, refactors, and features Muhammad Rizki
` (2 preceding siblings ...)
2023-01-08 8:07 ` [PATCH v3 03/11] Add DaemonException() class Muhammad Rizki
@ 2023-01-08 8:07 ` Muhammad Rizki
2023-01-08 8:07 ` [PATCH v3 05/11] telegram: Implement the DaemonException() and refactor report_err() Muhammad Rizki
` (7 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Muhammad Rizki @ 2023-01-08 8:07 UTC (permalink / raw)
To: Ammar Faizi
Cc: Muhammad Rizki, Alviro Iskandar Setiawan, GNU/Weeb Mailing List
Instead of adding the report_err() function in the Bot() class, move it
to the DaemonClient() class. Also, change the report_err() function as
follows:
def report_err(self, e: DaemonException):
# code for reporting error goes here
Signed-off-by: Muhammad Rizki <[email protected]>
---
daemon/telegram/mailer/listener.py | 8 +-------
daemon/telegram/packages/client.py | 8 ++++++++
2 files changed, 9 insertions(+), 7 deletions(-)
diff --git a/daemon/telegram/mailer/listener.py b/daemon/telegram/mailer/listener.py
index 6482a66..5ac618f 100644
--- a/daemon/telegram/mailer/listener.py
+++ b/daemon/telegram/mailer/listener.py
@@ -68,12 +68,6 @@ class Bot():
self.db.ping(reconnect=True, attempts=reconnect_attempts,
delay=delay_in_secs)
- async def report_err(caption):
- if not caption:
- caption = "No lore URL"
- exc_str = utils.catch_err()
- self.logger.warning(exc_str)
- await self.client.send_log_file(caption)
async def __run(self):
self.logger.info("Running...")
@@ -84,7 +78,7 @@ class Bot():
except (OperationalError, DatabaseError) as e:
await self.handle_db_error(e)
except:
- await self.report_err(url)
+ await self.client.report_err(url)
if not self.isRunnerFixed:
self.isRunnerFixed = True
diff --git a/daemon/telegram/packages/client.py b/daemon/telegram/packages/client.py
index 58195b9..c5b0b33 100644
--- a/daemon/telegram/packages/client.py
+++ b/daemon/telegram/packages/client.py
@@ -14,6 +14,7 @@ from logger import BotLogger
from telegram import config
from telegram.database import DB
from .decorator import handle_flood
+from exceptions import DaemonException
class DaemonClient(Client):
@@ -27,6 +28,13 @@ class DaemonClient(Client):
self.logger = logger
+ async def report_err(self, e: DaemonException):
+ capt = f"Atom URL: {e.atom_url}\n"
+ capt += f"Thread URL: {e.thread_url}"
+ self.logger.warning(e.original_exception)
+ await self.send_log_file(capt)
+
+
@handle_flood
async def send_log_file(self, caption: str):
filename = self.logger.handlers[0].baseFilename
--
Muhammad Rizki
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH v3 05/11] telegram: Implement the DaemonException() and refactor report_err()
2023-01-08 8:07 [PATCH v3 00/11] Improvements, refactors, and features Muhammad Rizki
` (3 preceding siblings ...)
2023-01-08 8:07 ` [PATCH v3 04/11] telegram: Refactor report_err() and move to the DaemonClient() class Muhammad Rizki
@ 2023-01-08 8:07 ` Muhammad Rizki
2023-01-08 8:07 ` [PATCH v3 06/11] telegram: listener: Remove unnecessary try/except block Muhammad Rizki
` (6 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Muhammad Rizki @ 2023-01-08 8:07 UTC (permalink / raw)
To: Ammar Faizi
Cc: Muhammad Rizki, Alviro Iskandar Setiawan, GNU/Weeb Mailing List
This exception class is used to handle errors that are easy to catch,
such as the email thread URL. This class has been customized and
inherited from the built-in Exception() class. Also, change the
report_err() value from the DaemonException.
Signed-off-by: Muhammad Rizki <[email protected]>
---
daemon/telegram/mailer/listener.py | 19 ++++++++++++++++---
1 file changed, 16 insertions(+), 3 deletions(-)
diff --git a/daemon/telegram/mailer/listener.py b/daemon/telegram/mailer/listener.py
index 5ac618f..7ec70b0 100644
--- a/daemon/telegram/mailer/listener.py
+++ b/daemon/telegram/mailer/listener.py
@@ -9,6 +9,7 @@ from pyrogram.types import Message
from mysql.connector.errors import OperationalError, DatabaseError
from apscheduler.schedulers.asyncio import AsyncIOScheduler
from telegram.packages import DaemonClient
+from exceptions import DaemonException
from atom import Scraper
from atom import utils
from enums import Platform
@@ -77,8 +78,14 @@ class Bot():
await self.__handle_atom_url(url)
except (OperationalError, DatabaseError) as e:
await self.handle_db_error(e)
+ except DaemonException as e:
+ e.set_atom_url(url)
+ await self.client.report_err(e)
except:
- await self.client.report_err(url)
+ e = DaemonException()
+ e.set_atom_url(url)
+ e.set_message(utils.catch_err())
+ await self.client.report_err(e)
if not self.isRunnerFixed:
self.isRunnerFixed = True
@@ -94,8 +101,14 @@ class Bot():
async def __handle_atom_url(self, url):
urls = await self.scraper.get_new_threads_urls(url)
for url in urls:
- mail = await self.scraper.get_email_from_url(url)
- await self.__handle_mail(url, mail)
+ try:
+ mail = await self.scraper.get_email_from_url(url)
+ await self.__handle_mail(url, mail)
+ except:
+ e = DaemonException()
+ e.set_thread_url(url)
+ e.set_message(utils.catch_err())
+ raise e
async def __handle_mail(self, url, mail):
--
Muhammad Rizki
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH v3 06/11] telegram: listener: Remove unnecessary try/except block
2023-01-08 8:07 [PATCH v3 00/11] Improvements, refactors, and features Muhammad Rizki
` (4 preceding siblings ...)
2023-01-08 8:07 ` [PATCH v3 05/11] telegram: Implement the DaemonException() and refactor report_err() Muhammad Rizki
@ 2023-01-08 8:07 ` Muhammad Rizki
2023-01-08 8:07 ` [PATCH v3 07/11] discord: add report_err() for the Discord bot Muhammad Rizki
` (5 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Muhammad Rizki @ 2023-01-08 8:07 UTC (permalink / raw)
To: Ammar Faizi
Cc: Muhammad Rizki, Alviro Iskandar Setiawan, GNU/Weeb Mailing List
The error handling is already been moved to the __handle_atom_url()
so it's not necessary to use try/except block again.
Signed-off-by: Muhammad Rizki <[email protected]>
---
daemon/telegram/mailer/listener.py | 9 +--------
1 file changed, 1 insertion(+), 8 deletions(-)
diff --git a/daemon/telegram/mailer/listener.py b/daemon/telegram/mailer/listener.py
index 7ec70b0..505cf44 100644
--- a/daemon/telegram/mailer/listener.py
+++ b/daemon/telegram/mailer/listener.py
@@ -138,14 +138,7 @@ class Bot():
self.logger.debug(md)
return False
- try:
- text, files, is_patch = utils.create_template(mail, Platform.TELEGRAM)
- except:
- exc_str = utils.catch_err()
- self.logger.warning(exc_str)
- await self.client.send_log_file(url)
- return
-
+ text, files, is_patch = utils.create_template(mail, Platform.TELEGRAM)
reply_to = self.get_reply(mail, tg_chat_id)
url = str(re.sub(r"/raw$", "", url))
--
Muhammad Rizki
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH v3 07/11] discord: add report_err() for the Discord bot
2023-01-08 8:07 [PATCH v3 00/11] Improvements, refactors, and features Muhammad Rizki
` (5 preceding siblings ...)
2023-01-08 8:07 ` [PATCH v3 06/11] telegram: listener: Remove unnecessary try/except block Muhammad Rizki
@ 2023-01-08 8:07 ` Muhammad Rizki
2023-01-08 8:07 ` [PATCH v3 08/11] discord: Implement the report_err() and DaemonException() error class Muhammad Rizki
` (4 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Muhammad Rizki @ 2023-01-08 8:07 UTC (permalink / raw)
To: Ammar Faizi
Cc: Muhammad Rizki, Alviro Iskandar Setiawan, GNU/Weeb Mailing List
Add report_err() in the Discord bot GWClient() class for reporting
error to the Discord channel.
Signed-off-by: Muhammad Rizki <[email protected]>
---
daemon/dscord/gnuweeb/client.py | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/daemon/dscord/gnuweeb/client.py b/daemon/dscord/gnuweeb/client.py
index 406c606..78ec3f9 100644
--- a/daemon/dscord/gnuweeb/client.py
+++ b/daemon/dscord/gnuweeb/client.py
@@ -13,6 +13,7 @@ from . import filters
from . import models
from atom import utils
from enums import Platform
+from exceptions import DaemonException
from logger.log import BotLogger
from dscord.config import ACTIVITY_NAME, LOG_CHANNEL_ID
from dscord.database import DB
@@ -40,6 +41,13 @@ class GWClient(commands.Bot):
)
+ async def report_err(self, e: DaemonException):
+ self.logger.warning(e.original_exception)
+ capt = f"Atom URL: {e.atom_url}\n"
+ capt += f"Thread URL: {e.thread_url}"
+ await self.send_log_file(capt)
+
+
@filters.wait_on_limit
async def send_log_file(self, caption: str):
filename = self.logger.handlers[0].baseFilename
--
Muhammad Rizki
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH v3 08/11] discord: Implement the report_err() and DaemonException() error class
2023-01-08 8:07 [PATCH v3 00/11] Improvements, refactors, and features Muhammad Rizki
` (6 preceding siblings ...)
2023-01-08 8:07 ` [PATCH v3 07/11] discord: add report_err() for the Discord bot Muhammad Rizki
@ 2023-01-08 8:07 ` Muhammad Rizki
2023-01-08 8:07 ` [PATCH v3 09/11] discord: database: Add ping() method for reconnect MySQL Muhammad Rizki
` (3 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Muhammad Rizki @ 2023-01-08 8:07 UTC (permalink / raw)
To: Ammar Faizi
Cc: Muhammad Rizki, Alviro Iskandar Setiawan, GNU/Weeb Mailing List
After adding a report_err() function within the GWClient() class,
implement it in dscord/mailer/listener.py to report error logs to the
Discord channel. Additionally, also implement the DaemonException()
function to handle errors.
Signed-off-by: Muhammad Rizki <[email protected]>
---
daemon/dscord/mailer/listener.py | 23 ++++++++++++++++++-----
1 file changed, 18 insertions(+), 5 deletions(-)
diff --git a/daemon/dscord/mailer/listener.py b/daemon/dscord/mailer/listener.py
index a225f46..aa44603 100644
--- a/daemon/dscord/mailer/listener.py
+++ b/daemon/dscord/mailer/listener.py
@@ -14,6 +14,7 @@ from dscord.gnuweeb import GWClient
from atom.scraper import Scraper
from atom import utils
from enums import Platform
+from exceptions import DaemonException
class Mutexes:
@@ -51,8 +52,19 @@ class Listener:
async def __run(self):
self.logger.info("Running...")
- for url in self.db.get_atom_urls():
- await self.__handle_atom_url(url)
+ url = None
+
+ try:
+ for url in self.db.get_atom_urls():
+ await self.__handle_atom_url(url)
+ except DaemonException as e:
+ e.set_atom_url(url)
+ await self.client.report_err(e)
+ except:
+ e = DaemonException()
+ e.set_atom_url(url)
+ e.set_message(utils.catch_err())
+ await self.client.report_err(e)
if not self.isRunnerFixed:
self.isRunnerFixed = True
@@ -72,9 +84,10 @@ class Listener:
mail = await self.scraper.get_email_from_url(url)
await self.__handle_mail(url, mail)
except:
- exc_str = utils.catch_err()
- self.client.logger.warning(exc_str)
- await self.client.send_log_file(url)
+ e = DaemonException()
+ e.set_thread_url(url)
+ e.set_message(utils.catch_err())
+ raise e
async def __handle_mail(self, url, mail):
--
Muhammad Rizki
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH v3 09/11] discord: database: Add ping() method for reconnect MySQL
2023-01-08 8:07 [PATCH v3 00/11] Improvements, refactors, and features Muhammad Rizki
` (7 preceding siblings ...)
2023-01-08 8:07 ` [PATCH v3 08/11] discord: Implement the report_err() and DaemonException() error class Muhammad Rizki
@ 2023-01-08 8:07 ` Muhammad Rizki
2023-01-08 8:07 ` [PATCH v3 10/11] discord: listener: Add handle_db_error to handle the MySQL errors Muhammad Rizki
` (2 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Muhammad Rizki @ 2023-01-08 8:07 UTC (permalink / raw)
To: Ammar Faizi
Cc: Muhammad Rizki, Alviro Iskandar Setiawan, GNU/Weeb Mailing List
Add ping() in the Discord database core.py for reconnecting the MySQL
database. This function is to reconnect the MySQL connection.
Signed-off-by: Muhammad Rizki <[email protected]>
---
daemon/dscord/database/core.py | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/daemon/dscord/database/core.py b/daemon/dscord/database/core.py
index c34d7a8..c8826bb 100644
--- a/daemon/dscord/database/core.py
+++ b/daemon/dscord/database/core.py
@@ -15,6 +15,12 @@ class DB(DBMethods):
self.cur = self.conn.cursor(buffered=True)
+ def ping(self, reconnect=True, attempts=3, delay=3):
+ self.conn.ping(reconnect=reconnect, attempts=attempts,
+ delay=delay)
+ self.cur = self.conn.cursor(buffered=True)
+
+
def __del__(self):
self.cur.close()
self.conn.close()
--
Muhammad Rizki
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH v3 10/11] discord: listener: Add handle_db_error to handle the MySQL errors
2023-01-08 8:07 [PATCH v3 00/11] Improvements, refactors, and features Muhammad Rizki
` (8 preceding siblings ...)
2023-01-08 8:07 ` [PATCH v3 09/11] discord: database: Add ping() method for reconnect MySQL Muhammad Rizki
@ 2023-01-08 8:07 ` Muhammad Rizki
2023-01-08 8:07 ` [PATCH v3 11/11] telegram: Rename the Telegram bot custom client Muhammad Rizki
2023-01-08 8:34 ` [PATCH v3 00/11] Improvements, refactors, and features Ammar Faizi
11 siblings, 0 replies; 13+ messages in thread
From: Muhammad Rizki @ 2023-01-08 8:07 UTC (permalink / raw)
To: Ammar Faizi
Cc: Muhammad Rizki, Alviro Iskandar Setiawan, GNU/Weeb Mailing List
After adding a ping() method to the database, you should handle database
errors such as connection errors. This function will be used to handle
connection errors that occur when the MySQL service is restarted.
Signed-off-by: Muhammad Rizki <[email protected]>
---
daemon/dscord/mailer/listener.py | 21 +++++++++++++++++++++
1 file changed, 21 insertions(+)
diff --git a/daemon/dscord/mailer/listener.py b/daemon/dscord/mailer/listener.py
index aa44603..fc066b7 100644
--- a/daemon/dscord/mailer/listener.py
+++ b/daemon/dscord/mailer/listener.py
@@ -6,6 +6,7 @@
import asyncio
import re
+from mysql.connector.errors import OperationalError, DatabaseError
from apscheduler.schedulers.asyncio import AsyncIOScheduler
from discord import File
from discord import Message
@@ -50,6 +51,24 @@ class Listener:
self.runner = self.sched.add_job(func=self.__run)
+ async def handle_db_error(self, e):
+ #
+ # TODO(ammarfaizi2):
+ # Ideally, we also want to log and report this situation.
+ #
+ self.logger.error(f"Database error: {str(e)}")
+ self.logger.info("Reconnecting to the database...")
+
+ #
+ # Don't do this too often to avoid reconnect burst.
+ #
+ delay_in_secs = 3
+ reconnect_attempts = 3
+
+ self.db.ping(reconnect=True, attempts=reconnect_attempts,
+ delay=delay_in_secs)
+
+
async def __run(self):
self.logger.info("Running...")
url = None
@@ -57,6 +76,8 @@ class Listener:
try:
for url in self.db.get_atom_urls():
await self.__handle_atom_url(url)
+ except (OperationalError, DatabaseError) as e:
+ await self.handle_db_error(e)
except DaemonException as e:
e.set_atom_url(url)
await self.client.report_err(e)
--
Muhammad Rizki
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH v3 11/11] telegram: Rename the Telegram bot custom client
2023-01-08 8:07 [PATCH v3 00/11] Improvements, refactors, and features Muhammad Rizki
` (9 preceding siblings ...)
2023-01-08 8:07 ` [PATCH v3 10/11] discord: listener: Add handle_db_error to handle the MySQL errors Muhammad Rizki
@ 2023-01-08 8:07 ` Muhammad Rizki
2023-01-08 8:34 ` [PATCH v3 00/11] Improvements, refactors, and features Ammar Faizi
11 siblings, 0 replies; 13+ messages in thread
From: Muhammad Rizki @ 2023-01-08 8:07 UTC (permalink / raw)
To: Ammar Faizi
Cc: Muhammad Rizki, Alviro Iskandar Setiawan, GNU/Weeb Mailing List
Rename the custom client for the Telegram bot and update all affected
code in the other files to reflect the change.
Signed-off-by: Muhammad Rizki <[email protected]>
---
daemon/telegram/mailer/listener.py | 4 ++--
daemon/telegram/packages/__init__.py | 2 +-
daemon/telegram/packages/client.py | 2 +-
daemon/telegram/packages/decorator.py | 2 +-
daemon/telegram/packages/plugins/callbacks/del_atom.py | 6 +++---
daemon/telegram/packages/plugins/callbacks/del_chat.py | 6 +++---
.../telegram/packages/plugins/commands/manage_atom.py | 10 +++++-----
.../packages/plugins/commands/manage_broadcast.py | 10 +++++-----
daemon/telegram/packages/plugins/commands/scrape.py | 6 +++---
daemon/tg.py | 4 ++--
10 files changed, 26 insertions(+), 26 deletions(-)
diff --git a/daemon/telegram/mailer/listener.py b/daemon/telegram/mailer/listener.py
index 505cf44..2e8eeda 100644
--- a/daemon/telegram/mailer/listener.py
+++ b/daemon/telegram/mailer/listener.py
@@ -8,7 +8,7 @@ from pyrogram import idle
from pyrogram.types import Message
from mysql.connector.errors import OperationalError, DatabaseError
from apscheduler.schedulers.asyncio import AsyncIOScheduler
-from telegram.packages import DaemonClient
+from telegram.packages import DaemonTelegram
from exceptions import DaemonException
from atom import Scraper
from atom import utils
@@ -23,7 +23,7 @@ class BotMutexes():
class Bot():
- def __init__(self, client: DaemonClient, sched: AsyncIOScheduler,
+ def __init__(self, client: DaemonTelegram, sched: AsyncIOScheduler,
scraper: Scraper, mutexes: BotMutexes):
self.client = client
self.sched = sched
diff --git a/daemon/telegram/packages/__init__.py b/daemon/telegram/packages/__init__.py
index efef9ae..15a270f 100644
--- a/daemon/telegram/packages/__init__.py
+++ b/daemon/telegram/packages/__init__.py
@@ -1 +1 @@
-from .client import DaemonClient
+from .client import DaemonTelegram
diff --git a/daemon/telegram/packages/client.py b/daemon/telegram/packages/client.py
index c5b0b33..a58aaf8 100644
--- a/daemon/telegram/packages/client.py
+++ b/daemon/telegram/packages/client.py
@@ -17,7 +17,7 @@ from .decorator import handle_flood
from exceptions import DaemonException
-class DaemonClient(Client):
+class DaemonTelegram(Client):
def __init__(self, name: str, api_id: int,
api_hash: str, conn, logger: BotLogger,
**kwargs
diff --git a/daemon/telegram/packages/decorator.py b/daemon/telegram/packages/decorator.py
index 153fa95..efcfc38 100644
--- a/daemon/telegram/packages/decorator.py
+++ b/daemon/telegram/packages/decorator.py
@@ -24,7 +24,7 @@ def handle_flood(func: Callable[[T], T]) -> Callable[[T], T]:
try:
return await func(*args)
except FloodWait as e:
- # Calling logger attr from the DaemonClient() class
+ # Calling logger attr from the DaemonTelegram() class
logger = args[0].logger
_flood_exceptions(e, logger)
diff --git a/daemon/telegram/packages/plugins/callbacks/del_atom.py b/daemon/telegram/packages/plugins/callbacks/del_atom.py
index b750e1c..84524c2 100644
--- a/daemon/telegram/packages/plugins/callbacks/del_atom.py
+++ b/daemon/telegram/packages/plugins/callbacks/del_atom.py
@@ -3,14 +3,14 @@
# Copyright (C) 2022 Muhammad Rizki <[email protected]>
#
-from telegram.packages import DaemonClient
+from telegram.packages import DaemonTelegram
from atom import utils
from pyrogram.types import CallbackQuery
from telegram import config
[email protected]_callback_query(config.admin_only, group=1)
-async def on_del_atom(c: DaemonClient, cb: CallbackQuery):
[email protected]_callback_query(config.admin_only, group=1)
+async def on_del_atom(c: DaemonTelegram, cb: CallbackQuery):
if not "del_atom" in cb.data:
return
diff --git a/daemon/telegram/packages/plugins/callbacks/del_chat.py b/daemon/telegram/packages/plugins/callbacks/del_chat.py
index 90b557e..c7135e0 100644
--- a/daemon/telegram/packages/plugins/callbacks/del_chat.py
+++ b/daemon/telegram/packages/plugins/callbacks/del_chat.py
@@ -3,14 +3,14 @@
# Copyright (C) 2022 Muhammad Rizki <[email protected]>
#
-from telegram.packages import DaemonClient
+from telegram.packages import DaemonTelegram
from atom import utils
from pyrogram.types import CallbackQuery
from telegram import config
[email protected]_callback_query(config.admin_only, group=2)
-async def on_del_chat(c: DaemonClient, cb: CallbackQuery):
[email protected]_callback_query(config.admin_only, group=2)
+async def on_del_chat(c: DaemonTelegram, cb: CallbackQuery):
if not "del_chat" in cb.data:
return
diff --git a/daemon/telegram/packages/plugins/commands/manage_atom.py b/daemon/telegram/packages/plugins/commands/manage_atom.py
index 99df7f7..f686adf 100644
--- a/daemon/telegram/packages/plugins/commands/manage_atom.py
+++ b/daemon/telegram/packages/plugins/commands/manage_atom.py
@@ -5,16 +5,16 @@
from pyrogram.types import Message
from pyrogram import filters
-from telegram.packages import DaemonClient
+from telegram.packages import DaemonTelegram
from atom import utils
from telegram import config
[email protected]_message(
[email protected]_message(
filters.command("add_atom") &
config.admin_only
)
-async def add_atom_url(c: DaemonClient, m: Message):
+async def add_atom_url(c: DaemonTelegram, m: Message):
if len(m.command) <= 1:
tutor = "Please specify the URL\n"
tutor += "Example: `/add_atom https://lore.kernel.org/linux-sgx/new.atom`"
@@ -32,11 +32,11 @@ async def add_atom_url(c: DaemonClient, m: Message):
await m.reply(f"Success add **{text}** for listening new email")
[email protected]_message(
[email protected]_message(
filters.command("del_atom") &
config.admin_only
)
-async def del_atom_url(c: DaemonClient, m: Message):
+async def del_atom_url(c: DaemonTelegram, m: Message):
atoms = c.db.get_atom_urls()
if len(atoms) == 0:
return await m.reply("Currently empty.")
diff --git a/daemon/telegram/packages/plugins/commands/manage_broadcast.py b/daemon/telegram/packages/plugins/commands/manage_broadcast.py
index 0aa70de..63f9547 100644
--- a/daemon/telegram/packages/plugins/commands/manage_broadcast.py
+++ b/daemon/telegram/packages/plugins/commands/manage_broadcast.py
@@ -5,16 +5,16 @@
from pyrogram.types import Message
from pyrogram import filters, enums
-from telegram.packages import DaemonClient
+from telegram.packages import DaemonTelegram
from atom import utils
from telegram import config
[email protected]_message(
[email protected]_message(
filters.command("add_bc") &
config.admin_only
)
-async def add_broadcast(c: DaemonClient, m: Message):
+async def add_broadcast(c: DaemonTelegram, m: Message):
if m.chat.type == enums.ChatType.PRIVATE:
chat_name = m.chat.first_name
else:
@@ -37,11 +37,11 @@ async def add_broadcast(c: DaemonClient, m: Message):
await m.reply(msg)
[email protected]_message(
[email protected]_message(
filters.command("del_bc") &
config.admin_only
)
-async def del_broadcast(c: DaemonClient, m: Message):
+async def del_broadcast(c: DaemonTelegram, m: Message):
if "--list" in m.text:
chats = c.db.get_broadcast_chats()
if len(chats) == 0:
diff --git a/daemon/telegram/packages/plugins/commands/scrape.py b/daemon/telegram/packages/plugins/commands/scrape.py
index f89727d..89581b6 100644
--- a/daemon/telegram/packages/plugins/commands/scrape.py
+++ b/daemon/telegram/packages/plugins/commands/scrape.py
@@ -6,7 +6,7 @@
from pyrogram.types import Message
from pyrogram import filters
-from telegram.packages import DaemonClient
+from telegram.packages import DaemonTelegram
from atom import Scraper
from atom import utils
from enums import Platform
@@ -22,11 +22,11 @@ import asyncio
# .lore https://lore.kernel.org/path/message_id/raw
#
LORE_CMD_URL_PATTERN = r"^(?:\/|\.|\!)lore\s+(https?:\/\/lore\.kernel\.org\/\S+)"
[email protected]_message(
[email protected]_message(
filters.regex(LORE_CMD_URL_PATTERN) &
config.admin_only
)
-async def scrap_email(c: DaemonClient, m: Message):
+async def scrap_email(c: DaemonTelegram, m: Message):
p = re.search(LORE_CMD_URL_PATTERN, m.text)
if not p:
return
diff --git a/daemon/tg.py b/daemon/tg.py
index a676cf5..9f36e54 100644
--- a/daemon/tg.py
+++ b/daemon/tg.py
@@ -9,7 +9,7 @@ from dotenv import load_dotenv
from mysql import connector
from pyrogram import idle
from atom import Scraper
-from telegram.packages import DaemonClient
+from telegram.packages import DaemonTelegram
from telegram.mailer import BotMutexes
from telegram.mailer import Bot
from logger import BotLogger
@@ -28,7 +28,7 @@ def main():
else:
port = int(port)
- client = DaemonClient(
+ client = DaemonTelegram(
"telegram/storage/EmailScraper",
api_id=int(os.getenv("API_ID")),
api_hash=os.getenv("API_HASH"),
--
Muhammad Rizki
^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: [PATCH v3 00/11] Improvements, refactors, and features
2023-01-08 8:07 [PATCH v3 00/11] Improvements, refactors, and features Muhammad Rizki
` (10 preceding siblings ...)
2023-01-08 8:07 ` [PATCH v3 11/11] telegram: Rename the Telegram bot custom client Muhammad Rizki
@ 2023-01-08 8:34 ` Ammar Faizi
11 siblings, 0 replies; 13+ messages in thread
From: Ammar Faizi @ 2023-01-08 8:34 UTC (permalink / raw)
To: Muhammad Rizki
Cc: Ammar Faizi, GNU/Weeb Mailing List, Alviro Iskandar Setiawan
From: Ammar Faizi <[email protected]>
On Sun, 8 Jan 2023 15:07:30 +0700, Muhammad Rizki wrote:
> Hir sir,
> This is v3 of improvements, refactors, and features. This series is to
> add more `except` block for both Discord and Telegram, and fix some
> grammatical errors in the commit messages.
>
> In the last series, I didn't care so much with the Discord
> implementation. So, I forget to add a reconnect MySQL method for the
> Discord bot. In this series, I've added it and implement it in the
> listener.py exactly in the handle_db_error().
>
> [...]
Applied, thanks!
[01/11] discord: Remove unnecessary try/except block
commit: e6ee5d8bc329e5f38d240f0677e9fa2e4b09245f
[02/11] telegram: Perform graceful exit when interrupted by a signal
commit: 41109f4b8a1d3a79a1baadcd39a64e373de40acb
[03/11] Add DaemonException() class
commit: 22c27fe3390dad7dbba1e883d7c2cf658ca38316
[04/11] telegram: Refactor report_err() and move to the DaemonClient() class
commit: 2e291fdeac69a5f262d5871290c3aaff6f655e41
[05/11] telegram: Implement the DaemonException() and refactor report_err()
commit: fcc88f5df7cfa887a03458ab79baa6a218426ffc
[06/11] telegram: listener: Remove unnecessary try/except block
commit: 3610759d92213436ff2c089afc5992ec1c33c29f
[07/11] discord: add report_err() for the Discord bot
commit: deb93bc9c701f0bdaf5764f5955d0734a3772982
[08/11] discord: Implement the report_err() and DaemonException() error class
commit: 4e260bcd7fd9046b3fa390b15c4fc692ec20d6f6
[09/11] discord: database: Add ping() method for reconnect MySQL
commit: 61f26a6a15e28627466fd38d754fdc655ec2b511
[10/11] discord: listener: Add handle_db_error to handle the MySQL errors
commit: 9302239c8e0c7c1b19b1a2aacc69df496801ba6c
[11/11] telegram: Rename the Telegram bot custom client
commit: 2a538e96d90102cdee075a8c31e48316b3659b9e
Best regards,
--
Ammar Faizi
^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2023-01-08 8:34 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-01-08 8:07 [PATCH v3 00/11] Improvements, refactors, and features Muhammad Rizki
2023-01-08 8:07 ` [PATCH v3 01/11] discord: Remove unnecessary try/except block Muhammad Rizki
2023-01-08 8:07 ` [PATCH v3 02/11] telegram: Perform graceful exit when interrupted by a signal Muhammad Rizki
2023-01-08 8:07 ` [PATCH v3 03/11] Add DaemonException() class Muhammad Rizki
2023-01-08 8:07 ` [PATCH v3 04/11] telegram: Refactor report_err() and move to the DaemonClient() class Muhammad Rizki
2023-01-08 8:07 ` [PATCH v3 05/11] telegram: Implement the DaemonException() and refactor report_err() Muhammad Rizki
2023-01-08 8:07 ` [PATCH v3 06/11] telegram: listener: Remove unnecessary try/except block Muhammad Rizki
2023-01-08 8:07 ` [PATCH v3 07/11] discord: add report_err() for the Discord bot Muhammad Rizki
2023-01-08 8:07 ` [PATCH v3 08/11] discord: Implement the report_err() and DaemonException() error class Muhammad Rizki
2023-01-08 8:07 ` [PATCH v3 09/11] discord: database: Add ping() method for reconnect MySQL Muhammad Rizki
2023-01-08 8:07 ` [PATCH v3 10/11] discord: listener: Add handle_db_error to handle the MySQL errors Muhammad Rizki
2023-01-08 8:07 ` [PATCH v3 11/11] telegram: Rename the Telegram bot custom client Muhammad Rizki
2023-01-08 8:34 ` [PATCH v3 00/11] Improvements, refactors, and features Ammar Faizi
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox