* [PATCH v3 0/9] Fix some bugs and add some features @ 2022-10-21 13:45 Muhammad Rizki 2022-10-21 13:45 ` [PATCH v3 1/9] discord: Add send_text_mail_interaction() Muhammad Rizki ` (8 more replies) 0 siblings, 9 replies; 40+ messages in thread From: Muhammad Rizki @ 2022-10-21 13:45 UTC (permalink / raw) To: Ammar Faizi Cc: Muhammad Rizki, Alviro Iskandar Setiawan, GNU/Weeb Mailing List Hi sir, This is v3 revision of fix some bugs and add some features. This series contain add Platform class enumeration and improve manage_payload() utils function to handle the transfer encodings. This series is to fix some bugs, improve some codes and add some new features. These bugs should have been fixed now, and the email file attachments should have been removed after all attachments have been sent. Known bugs: 1. Email payload extraction result become unicode if the email payload contain non-UTF8 characters like chinese, japanese, and similar like that. 2. remove_patch() doesn't remove all file attachments properly. Improvements: 1. Improve remove_patch() to make it all file attachments removed after sending them. 2. Fix `/lore {raw atom url}` to add a "telegram" onto the create_template() platform parameter. 3. Improvement on extract_body() to ignore whenever the email contain text/html content-type. New features: 1. Add send_text_mail_interaction() for the `/lore {raw atom url}` slash command. 2. Add send_patch_mail_interaction() for the `/lore {raw atom url}` slash command. 3. Add `/lore {raw atom url}` slash command. 4. Platform class enumeration There are 9 patches in this series: - Patch 1 is to add send_text_mail_interaction() - Patch 2 is to add send_patch_mail_interaction() - Patch 3 is to add `/lore` slash command - Patch 4 is to improve remove_patch() code to make it more stable - Patch 5 is to add manage_payload() for manage email payload extraction - Patch 6 is to fix the Telegram `/lore` command - Patch 7 is to improve extract_body() to ignore text/html content-type - Patch 8 is to add initial Platform class enumeration - Patch 9 is to use the created Platform class enumeration How to use: 1. Execute the db.sql file in the daemon directory, 2. Setup .env file, the example is there with suffix .example, this file name must remove the suffix name .example, 3. Set up the config.py in each bot directory, such as dscord and telegram. The example is there with suffix .example & the file name must remove suffix name .example, 4. Run `pip3 install -r requirements.txt` in each bot directory, 5. STORAGE_DIR env value must `storage` to make it work fine, 6. Run the bot by `python3 dc.py` or `python3 tg.py`. Tested and works, what do you think with this patch? ## Changelog v2 -> v3: - Add Platform class enumeration - Use the old one of fix_utf8_char() instead - Add Content-Transfer-Encoding handler in manage_payload() v1 -> v2: - Remove re.sub() in the fix_utf8_char() when set unescape to True - Add if statement logic to ignore text/html in the extract_body() Signed-off-by: Muhammad Rizki <[email protected]> --- Muhammad Rizki (9): discord: Add send_text_mail_interaction() discord: Add send_patch_mail_interaction() discord: Add get lore mail slash command atom: Improve remove_patch() atom: add manage_payload() telegram: Fix get lore command atom: Improve extract_body() enum: Add Platform enumeration enum: Use the created Platform class enumeration daemon/atom/utils.py | 57 ++++++++++++------- daemon/dscord/gnuweeb/client.py | 26 ++++++++- .../plugins/slash_commands/__init__.py | 2 + .../plugins/slash_commands/get_lore_mail.py | 40 +++++++++++++ daemon/dscord/mailer/listener.py | 7 ++- daemon/enums/__init__.py | 1 + daemon/enums/base.py | 9 +++ daemon/enums/platform.py | 7 +++ daemon/telegram/mailer/listener.py | 7 +-- daemon/telegram/packages/client.py | 3 +- .../packages/plugins/commands/scrape.py | 7 +-- 11 files changed, 133 insertions(+), 33 deletions(-) create mode 100644 daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py create mode 100644 daemon/enums/__init__.py create mode 100644 daemon/enums/base.py create mode 100644 daemon/enums/platform.py base-commit: d9b20dab81202b93f48d5365ad680796e5839d80 -- Muhammad Rizki ^ permalink raw reply [flat|nested] 40+ messages in thread
* [PATCH v3 1/9] discord: Add send_text_mail_interaction() 2022-10-21 13:45 [PATCH v3 0/9] Fix some bugs and add some features Muhammad Rizki @ 2022-10-21 13:45 ` Muhammad Rizki 2022-10-21 13:45 ` [PATCH v3 2/9] discord: Add send_patch_mail_interaction() Muhammad Rizki ` (7 subsequent siblings) 8 siblings, 0 replies; 40+ messages in thread From: Muhammad Rizki @ 2022-10-21 13:45 UTC (permalink / raw) To: Ammar Faizi Cc: Muhammad Rizki, Alviro Iskandar Setiawan, GNU/Weeb Mailing List Add send_text_mail_interaction() for the `/atom get {url}` to get the specific lore email message from an URL in future use. Signed-off-by: Muhammad Rizki <[email protected]> --- daemon/dscord/gnuweeb/client.py | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/daemon/dscord/gnuweeb/client.py b/daemon/dscord/gnuweeb/client.py index 03a1b8c..15a6eee 100644 --- a/daemon/dscord/gnuweeb/client.py +++ b/daemon/dscord/gnuweeb/client.py @@ -4,6 +4,7 @@ # import discord +from discord import Interaction from discord.ext import commands from discord import Intents from dscord.config import ACTIVITY_NAME @@ -76,3 +77,11 @@ class GWClient(commands.Bot): utils.remove_patch(tmp) return m + + + async def send_text_mail_interaction(self, i: "Interaction", + text: str, url: str = None): + return await i.response.send_message( + content=text, + view=models.FullMessageBtn(url) + ) -- Muhammad Rizki ^ permalink raw reply related [flat|nested] 40+ messages in thread
* [PATCH v3 2/9] discord: Add send_patch_mail_interaction() 2022-10-21 13:45 [PATCH v3 0/9] Fix some bugs and add some features Muhammad Rizki 2022-10-21 13:45 ` [PATCH v3 1/9] discord: Add send_text_mail_interaction() Muhammad Rizki @ 2022-10-21 13:45 ` Muhammad Rizki 2022-10-21 13:45 ` [PATCH v3 3/9] discord: Add get lore mail slash command Muhammad Rizki ` (6 subsequent siblings) 8 siblings, 0 replies; 40+ messages in thread From: Muhammad Rizki @ 2022-10-21 13:45 UTC (permalink / raw) To: Ammar Faizi Cc: Muhammad Rizki, Alviro Iskandar Setiawan, GNU/Weeb Mailing List This function is just the same as send_text_mail_interaction(), the different between them is send_patch_mail_interaction() for sending a lore email message with patch file attachment. Signed-off-by: Muhammad Rizki <[email protected]> --- daemon/dscord/gnuweeb/client.py | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/daemon/dscord/gnuweeb/client.py b/daemon/dscord/gnuweeb/client.py index 15a6eee..b921f7d 100644 --- a/daemon/dscord/gnuweeb/client.py +++ b/daemon/dscord/gnuweeb/client.py @@ -85,3 +85,17 @@ class GWClient(commands.Bot): content=text, view=models.FullMessageBtn(url) ) + + + async def send_patch_mail_interaction(self, mail, i: "Interaction", + text: str, url: str = None): + tmp, doc, caption, url = utils.prepare_patch( + mail, text, url, "discord" + ) + m = await i.response.send_message( + content=caption, + file=discord.File(doc), + view=models.FullMessageBtn(url) + ) + utils.remove_patch(tmp) + return m -- Muhammad Rizki ^ permalink raw reply related [flat|nested] 40+ messages in thread
* [PATCH v3 3/9] discord: Add get lore mail slash command 2022-10-21 13:45 [PATCH v3 0/9] Fix some bugs and add some features Muhammad Rizki 2022-10-21 13:45 ` [PATCH v3 1/9] discord: Add send_text_mail_interaction() Muhammad Rizki 2022-10-21 13:45 ` [PATCH v3 2/9] discord: Add send_patch_mail_interaction() Muhammad Rizki @ 2022-10-21 13:45 ` Muhammad Rizki 2022-10-21 13:45 ` [PATCH v3 4/9] atom: Improve remove_patch() Muhammad Rizki ` (5 subsequent siblings) 8 siblings, 0 replies; 40+ messages in thread From: Muhammad Rizki @ 2022-10-21 13:45 UTC (permalink / raw) To: Ammar Faizi Cc: Muhammad Rizki, Alviro Iskandar Setiawan, GNU/Weeb Mailing List Add a `/lore {url}` slash command to get the specific lore email message from the raw atom URL. Signed-off-by: Muhammad Rizki <[email protected]> --- .../plugins/slash_commands/__init__.py | 2 + .../plugins/slash_commands/get_lore_mail.py | 39 +++++++++++++++++++ 2 files changed, 41 insertions(+) create mode 100644 daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py diff --git a/daemon/dscord/gnuweeb/plugins/slash_commands/__init__.py b/daemon/dscord/gnuweeb/plugins/slash_commands/__init__.py index a6d913c..126af45 100644 --- a/daemon/dscord/gnuweeb/plugins/slash_commands/__init__.py +++ b/daemon/dscord/gnuweeb/plugins/slash_commands/__init__.py @@ -5,9 +5,11 @@ from .manage_atom import ManageAtomSC from .manage_broadcast import ManageBroadcastSC +from .get_lore_mail import GetLoreSC class SlashCommands( + GetLoreSC, ManageAtomSC, ManageBroadcastSC ): pass diff --git a/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py b/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py new file mode 100644 index 0000000..2d55d16 --- /dev/null +++ b/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py @@ -0,0 +1,39 @@ +# SPDX-License-Identifier: GPL-2.0-only +# +# Copyright (C) 2022 Muhammad Rizki <[email protected]> +# + +import asyncio +import discord +from discord.ext import commands +from discord import Interaction +from discord import app_commands + +from atom import utils +from atom import Scraper + + +class GetLoreSC(commands.Cog): + def __init__(self, bot) -> None: + self.bot = bot + + + @app_commands.command( + name="lore", + description="Get lore email from raw email URL." + ) + @app_commands.describe(url="Raw lore email URL") + async def get_lore(self, i: "Interaction", url: str): + s = Scraper() + mail = await s.get_email_from_url(url) + text, _, is_patch = utils.create_template(mail, "discord") + + if is_patch: + m = await self.bot.send_patch_mail_interaction( + mail=mail, i=i, text=text, url=url + ) + else: + text = "#ml\n" + text + m = await self.bot.send_text_mail_interaction( + i=i, text=text, url=url + ) -- Muhammad Rizki ^ permalink raw reply related [flat|nested] 40+ messages in thread
* [PATCH v3 4/9] atom: Improve remove_patch() 2022-10-21 13:45 [PATCH v3 0/9] Fix some bugs and add some features Muhammad Rizki ` (2 preceding siblings ...) 2022-10-21 13:45 ` [PATCH v3 3/9] discord: Add get lore mail slash command Muhammad Rizki @ 2022-10-21 13:45 ` Muhammad Rizki 2022-10-21 13:45 ` [PATCH v3 5/9] atom: add manage_payload() Muhammad Rizki ` (4 subsequent siblings) 8 siblings, 0 replies; 40+ messages in thread From: Muhammad Rizki @ 2022-10-21 13:45 UTC (permalink / raw) To: Ammar Faizi Cc: Muhammad Rizki, Alviro Iskandar Setiawan, GNU/Weeb Mailing List Improvement on remove_patch(). So, after send all the email attachments then remove them. On the previous patch, this function can't remove them properly. Signed-off-by: Muhammad Rizki <[email protected]> --- daemon/atom/utils.py | 10 +++++++--- daemon/dscord/mailer/listener.py | 4 ++-- daemon/telegram/mailer/listener.py | 3 +-- daemon/telegram/packages/plugins/commands/scrape.py | 3 +-- 4 files changed, 11 insertions(+), 9 deletions(-) diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py index a30d5cb..f554f6f 100644 --- a/daemon/atom/utils.py +++ b/daemon/atom/utils.py @@ -6,7 +6,7 @@ from pyrogram.types import Chat, InlineKeyboardMarkup, InlineKeyboardButton from email.message import Message -from typing import Dict +from typing import Dict, Union from slugify import slugify import hashlib import uuid @@ -238,8 +238,12 @@ def prepare_patch(mail, text, url, platform: str): return tmp, file, caption, url -def remove_patch(tmp): - shutil.rmtree(tmp) +def remove_patch(tmp: Union[str, list]): + if isinstance(tmp, str): + return shutil.rmtree(tmp) + + for d,_ in tmp: + shutil.rmtree(d) def fix_utf8_char(text: str, html_escape: bool = True): diff --git a/daemon/dscord/mailer/listener.py b/daemon/dscord/mailer/listener.py index a280a58..cc0a9f7 100644 --- a/daemon/dscord/mailer/listener.py +++ b/daemon/dscord/mailer/listener.py @@ -125,10 +125,10 @@ class Listener: for d, f in files: await m.reply(file=File(f"{d}/{f}")) - if files.index((d,f)) == len(files)-1: - utils.remove_patch(d) await asyncio.sleep(1) + utils.remove_patch(files) + return True diff --git a/daemon/telegram/mailer/listener.py b/daemon/telegram/mailer/listener.py index 208aed0..08feddd 100644 --- a/daemon/telegram/mailer/listener.py +++ b/daemon/telegram/mailer/listener.py @@ -118,8 +118,7 @@ class Bot(): await m.reply_document(f"{d}/{f}", file_name=f) await asyncio.sleep(1) - if files: - shutil.rmtree(str(files[0][0])) + utils.remove_patch(files) return True diff --git a/daemon/telegram/packages/plugins/commands/scrape.py b/daemon/telegram/packages/plugins/commands/scrape.py index d4d10a9..52ddb0b 100644 --- a/daemon/telegram/packages/plugins/commands/scrape.py +++ b/daemon/telegram/packages/plugins/commands/scrape.py @@ -53,5 +53,4 @@ async def scrap_email(c: DaemonClient, m: Message): await m.reply_document(f"{d}/{f}", file_name=f) await asyncio.sleep(1) - if files: - shutil.rmtree(str(files[0][0])) + utils.remove_patch(files) -- Muhammad Rizki ^ permalink raw reply related [flat|nested] 40+ messages in thread
* [PATCH v3 5/9] atom: add manage_payload() 2022-10-21 13:45 [PATCH v3 0/9] Fix some bugs and add some features Muhammad Rizki ` (3 preceding siblings ...) 2022-10-21 13:45 ` [PATCH v3 4/9] atom: Improve remove_patch() Muhammad Rizki @ 2022-10-21 13:45 ` Muhammad Rizki 2022-10-21 14:01 ` Ammar Faizi 2022-10-21 13:45 ` [PATCH v3 6/9] telegram: Fix get lore command Muhammad Rizki ` (3 subsequent siblings) 8 siblings, 1 reply; 40+ messages in thread From: Muhammad Rizki @ 2022-10-21 13:45 UTC (permalink / raw) To: Ammar Faizi Cc: Muhammad Rizki, Alviro Iskandar Setiawan, GNU/Weeb Mailing List Add manage_payload() to handle the email decoding to utf-8. This include a non-UTF8 character and base64 decoding. Signed-off-by: Muhammad Rizki <[email protected]> --- daemon/atom/utils.py | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py index f554f6f..ed5ca03 100644 --- a/daemon/atom/utils.py +++ b/daemon/atom/utils.py @@ -8,6 +8,7 @@ from pyrogram.types import Chat, InlineKeyboardMarkup, InlineKeyboardButton from email.message import Message from typing import Dict, Union from slugify import slugify +from base64 import b64decode import hashlib import uuid import os @@ -15,6 +16,7 @@ import re import shutil import httpx import html +import quopri def get_email_msg_id(mail): @@ -136,7 +138,7 @@ def gen_temp(name: str, platform: str): def extract_body(thread: Message, platform: str): if not thread.is_multipart(): - p = thread.get_payload(decode=True).decode(errors='replace') + p = manage_payload(thread) if platform == "discord": p = quote_reply(p) @@ -253,6 +255,18 @@ def fix_utf8_char(text: str, html_escape: bool = True): return t +def manage_payload(payload: Message): + p = str(payload.get_payload()) + tf_encode = payload.get("Content-Transfer-Encoding") + + if tf_encode == "base64": + return b64decode(p).decode("utf-8") + if tf_encode == "quoted-printable": + return quopri.decodestring(p.encode()).decode() + + return p.encode().decode("utf-8", errors="replace") + + EMAIL_MSG_ID_PATTERN = r"<([^\<\>]+)>" def extract_email_msg_id(msg_id): ret = re.search(EMAIL_MSG_ID_PATTERN, msg_id) -- Muhammad Rizki ^ permalink raw reply related [flat|nested] 40+ messages in thread
* Re: [PATCH v3 5/9] atom: add manage_payload() 2022-10-21 13:45 ` [PATCH v3 5/9] atom: add manage_payload() Muhammad Rizki @ 2022-10-21 14:01 ` Ammar Faizi 2022-10-21 14:02 ` Muhammad Rizki 0 siblings, 1 reply; 40+ messages in thread From: Ammar Faizi @ 2022-10-21 14:01 UTC (permalink / raw) To: Muhammad Rizki; +Cc: Alviro Iskandar Setiawan, GNU/Weeb Mailing List On 10/21/22 8:45 PM, Muhammad Rizki wrote: > +def manage_payload(payload: Message): > + p = str(payload.get_payload()) > + tf_encode = payload.get("Content-Transfer-Encoding") > + > + if tf_encode == "base64": > + return b64decode(p).decode("utf-8") > + if tf_encode == "quoted-printable": > + return quopri.decodestring(p.encode()).decode() > + > + return p.encode().decode("utf-8", errors="replace") I don't feel strongly about this helper name. What about name it with get_decoded_paylaod()? -- Ammar Faizi ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v3 5/9] atom: add manage_payload() 2022-10-21 14:01 ` Ammar Faizi @ 2022-10-21 14:02 ` Muhammad Rizki 2022-10-21 14:06 ` Ammar Faizi 0 siblings, 1 reply; 40+ messages in thread From: Muhammad Rizki @ 2022-10-21 14:02 UTC (permalink / raw) To: Ammar Faizi; +Cc: Alviro Iskandar Setiawan, GNU/Weeb Mailing List On 21/10/2022 21.01, Ammar Faizi wrote: > On 10/21/22 8:45 PM, Muhammad Rizki wrote: >> +def manage_payload(payload: Message): >> + p = str(payload.get_payload()) >> + tf_encode = payload.get("Content-Transfer-Encoding") >> + >> + if tf_encode == "base64": >> + return b64decode(p).decode("utf-8") >> + if tf_encode == "quoted-printable": >> + return quopri.decodestring(p.encode()).decode() >> + >> + return p.encode().decode("utf-8", errors="replace") > > I don't feel strongly about this helper name. What about name it > with get_decoded_paylaod()? > Sure, or just decode_payload()? ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v3 5/9] atom: add manage_payload() 2022-10-21 14:02 ` Muhammad Rizki @ 2022-10-21 14:06 ` Ammar Faizi 2022-10-21 14:07 ` Muhammad Rizki 0 siblings, 1 reply; 40+ messages in thread From: Ammar Faizi @ 2022-10-21 14:06 UTC (permalink / raw) To: Muhammad Rizki; +Cc: Alviro Iskandar Setiawan, GNU/Weeb Mailing List On 10/21/22 9:02 PM, Muhammad Rizki wrote: > On 21/10/2022 21.01, Ammar Faizi wrote: >> On 10/21/22 8:45 PM, Muhammad Rizki wrote: >>> +def manage_payload(payload: Message): >>> + p = str(payload.get_payload()) >>> + tf_encode = payload.get("Content-Transfer-Encoding") >>> + >>> + if tf_encode == "base64": >>> + return b64decode(p).decode("utf-8") >>> + if tf_encode == "quoted-printable": >>> + return quopri.decodestring(p.encode()).decode() >>> + >>> + return p.encode().decode("utf-8", errors="replace") >> >> I don't feel strongly about this helper name. What about name it >> with get_decoded_paylaod()? >> > > Sure, or just decode_payload()? Since you pass a Message object and it contains headers and body, I prefer to emphasize the "get". So I still prefer the name get_decoded_paylaod() for this function. What we call payload is the body anyway. -- Ammar Faizi ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v3 5/9] atom: add manage_payload() 2022-10-21 14:06 ` Ammar Faizi @ 2022-10-21 14:07 ` Muhammad Rizki 0 siblings, 0 replies; 40+ messages in thread From: Muhammad Rizki @ 2022-10-21 14:07 UTC (permalink / raw) To: Ammar Faizi; +Cc: Alviro Iskandar Setiawan, GNU/Weeb Mailing List On 21/10/2022 21.06, Ammar Faizi wrote: > On 10/21/22 9:02 PM, Muhammad Rizki wrote: >> On 21/10/2022 21.01, Ammar Faizi wrote: >>> On 10/21/22 8:45 PM, Muhammad Rizki wrote: >>>> +def manage_payload(payload: Message): >>>> + p = str(payload.get_payload()) >>>> + tf_encode = payload.get("Content-Transfer-Encoding") >>>> + >>>> + if tf_encode == "base64": >>>> + return b64decode(p).decode("utf-8") >>>> + if tf_encode == "quoted-printable": >>>> + return quopri.decodestring(p.encode()).decode() >>>> + >>>> + return p.encode().decode("utf-8", errors="replace") >>> >>> I don't feel strongly about this helper name. What about name it >>> with get_decoded_paylaod()? >>> >> >> Sure, or just decode_payload()? > > Since you pass a Message object and it contains headers and body, > I prefer to emphasize the "get". So I still prefer the name > get_decoded_paylaod() for this function. What we call payload is > the body anyway. > OK. ^ permalink raw reply [flat|nested] 40+ messages in thread
* [PATCH v3 6/9] telegram: Fix get lore command 2022-10-21 13:45 [PATCH v3 0/9] Fix some bugs and add some features Muhammad Rizki ` (4 preceding siblings ...) 2022-10-21 13:45 ` [PATCH v3 5/9] atom: add manage_payload() Muhammad Rizki @ 2022-10-21 13:45 ` Muhammad Rizki 2022-10-21 13:45 ` [PATCH v3 7/9] atom: Improve extract_body() Muhammad Rizki ` (2 subsequent siblings) 8 siblings, 0 replies; 40+ messages in thread From: Muhammad Rizki @ 2022-10-21 13:45 UTC (permalink / raw) To: Ammar Faizi Cc: Muhammad Rizki, Alviro Iskandar Setiawan, GNU/Weeb Mailing List Fix the `/lore {raw lore url}` to add "telegram" on the create_template() platform parameter. Signed-off-by: Muhammad Rizki <[email protected]> --- daemon/telegram/packages/plugins/commands/scrape.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/daemon/telegram/packages/plugins/commands/scrape.py b/daemon/telegram/packages/plugins/commands/scrape.py index 52ddb0b..860e993 100644 --- a/daemon/telegram/packages/plugins/commands/scrape.py +++ b/daemon/telegram/packages/plugins/commands/scrape.py @@ -37,7 +37,7 @@ async def scrap_email(c: DaemonClient, m: Message): s = Scraper() mail = await s.get_email_from_url(url) - text, files, is_patch = utils.create_template(mail) + text, files, is_patch = utils.create_template(mail, "telegram") if is_patch: m = await c.send_patch_email( -- Muhammad Rizki ^ permalink raw reply related [flat|nested] 40+ messages in thread
* [PATCH v3 7/9] atom: Improve extract_body() 2022-10-21 13:45 [PATCH v3 0/9] Fix some bugs and add some features Muhammad Rizki ` (5 preceding siblings ...) 2022-10-21 13:45 ` [PATCH v3 6/9] telegram: Fix get lore command Muhammad Rizki @ 2022-10-21 13:45 ` Muhammad Rizki 2022-10-21 14:34 ` Alviro Iskandar Setiawan 2022-10-21 13:45 ` [PATCH v3 8/9] enum: Add Platform enumeration Muhammad Rizki 2022-10-21 13:45 ` [PATCH v3 9/9] enum: Use the created Platform class enumeration Muhammad Rizki 8 siblings, 1 reply; 40+ messages in thread From: Muhammad Rizki @ 2022-10-21 13:45 UTC (permalink / raw) To: Ammar Faizi Cc: Muhammad Rizki, Alviro Iskandar Setiawan, GNU/Weeb Mailing List Add an if statement to ignore whenever the email payload containing a text/html content-type header. Signed-off-by: Muhammad Rizki <[email protected]> --- daemon/atom/utils.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py index ed5ca03..6a9e5c8 100644 --- a/daemon/atom/utils.py +++ b/daemon/atom/utils.py @@ -151,6 +151,9 @@ def extract_body(thread: Message, platform: str): fname = p.get_filename() payload = p.get_payload(decode=True) + if p.get("content-type").split(";")[0] == "text/html": + continue + if not payload: continue -- Muhammad Rizki ^ permalink raw reply related [flat|nested] 40+ messages in thread
* Re: [PATCH v3 7/9] atom: Improve extract_body() 2022-10-21 13:45 ` [PATCH v3 7/9] atom: Improve extract_body() Muhammad Rizki @ 2022-10-21 14:34 ` Alviro Iskandar Setiawan 0 siblings, 0 replies; 40+ messages in thread From: Alviro Iskandar Setiawan @ 2022-10-21 14:34 UTC (permalink / raw) To: Muhammad Rizki; +Cc: Ammar Faizi, GNU/Weeb Mailing List On Fri, Oct 21, 2022 at 8:45 PM Muhammad Rizki wrote: > Add an if statement to ignore whenever the email payload containing a > text/html content-type header. > > Signed-off-by: Muhammad Rizki <[email protected]> Reviewed-by: Alviro Iskandar Setiawan <[email protected]> tq -- Viro ^ permalink raw reply [flat|nested] 40+ messages in thread
* [PATCH v3 8/9] enum: Add Platform enumeration 2022-10-21 13:45 [PATCH v3 0/9] Fix some bugs and add some features Muhammad Rizki ` (6 preceding siblings ...) 2022-10-21 13:45 ` [PATCH v3 7/9] atom: Improve extract_body() Muhammad Rizki @ 2022-10-21 13:45 ` Muhammad Rizki 2022-10-21 14:02 ` Ammar Faizi 2022-10-21 13:45 ` [PATCH v3 9/9] enum: Use the created Platform class enumeration Muhammad Rizki 8 siblings, 1 reply; 40+ messages in thread From: Muhammad Rizki @ 2022-10-21 13:45 UTC (permalink / raw) To: Ammar Faizi Cc: Muhammad Rizki, Alviro Iskandar Setiawan, GNU/Weeb Mailing List Add a Platform enum class to use it for enumeration instead of a string. Signed-off-by: Muhammad Rizki <[email protected]> --- daemon/enums/__init__.py | 1 + daemon/enums/base.py | 9 +++++++++ daemon/enums/platform.py | 7 +++++++ 3 files changed, 17 insertions(+) create mode 100644 daemon/enums/__init__.py create mode 100644 daemon/enums/base.py create mode 100644 daemon/enums/platform.py diff --git a/daemon/enums/__init__.py b/daemon/enums/__init__.py new file mode 100644 index 0000000..efe9c79 --- /dev/null +++ b/daemon/enums/__init__.py @@ -0,0 +1 @@ +from .platform import Platform diff --git a/daemon/enums/base.py b/daemon/enums/base.py new file mode 100644 index 0000000..526aba5 --- /dev/null +++ b/daemon/enums/base.py @@ -0,0 +1,9 @@ +import enum + + +class EnumBase(enum.Enum): + def _generate_next_value_(self, *args): + return self.lower() + + def __repr__(self): + return f"enums.{self}" diff --git a/daemon/enums/platform.py b/daemon/enums/platform.py new file mode 100644 index 0000000..638d8dd --- /dev/null +++ b/daemon/enums/platform.py @@ -0,0 +1,7 @@ +import enum +from .base import EnumBase + + +class Platform(EnumBase): + DISCORD = enum.auto() + TELEGRAM = enum.auto() -- Muhammad Rizki ^ permalink raw reply related [flat|nested] 40+ messages in thread
* Re: [PATCH v3 8/9] enum: Add Platform enumeration 2022-10-21 13:45 ` [PATCH v3 8/9] enum: Add Platform enumeration Muhammad Rizki @ 2022-10-21 14:02 ` Ammar Faizi 2022-10-21 14:04 ` Muhammad Rizki 0 siblings, 1 reply; 40+ messages in thread From: Ammar Faizi @ 2022-10-21 14:02 UTC (permalink / raw) To: Muhammad Rizki; +Cc: Alviro Iskandar Setiawan, GNU/Weeb Mailing List On 10/21/22 8:45 PM, Muhammad Rizki wrote: > --- /dev/null > +++ b/daemon/enums/platform.py > @@ -0,0 +1,7 @@ > +import enum > +from .base import EnumBase > + > + > +class Platform(EnumBase): > + DISCORD = enum.auto() > + TELEGRAM = enum.auto() Why is this necessary to be auto generated enum? I don't see any reason of why we should use an extra library here. I would just define them manually like: DISCORD = 1 TELEGRAM = 2 Thoughts? -- Ammar Faizi ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v3 8/9] enum: Add Platform enumeration 2022-10-21 14:02 ` Ammar Faizi @ 2022-10-21 14:04 ` Muhammad Rizki 2022-10-21 14:21 ` Ammar Faizi 0 siblings, 1 reply; 40+ messages in thread From: Muhammad Rizki @ 2022-10-21 14:04 UTC (permalink / raw) To: Ammar Faizi; +Cc: Alviro Iskandar Setiawan, GNU/Weeb Mailing List On 21/10/2022 21.02, Ammar Faizi wrote: > On 10/21/22 8:45 PM, Muhammad Rizki wrote: >> --- /dev/null >> +++ b/daemon/enums/platform.py >> @@ -0,0 +1,7 @@ >> +import enum >> +from .base import EnumBase >> + >> + >> +class Platform(EnumBase): >> + DISCORD = enum.auto() >> + TELEGRAM = enum.auto() > > Why is this necessary to be auto generated enum? > > I don't see any reason of why we should use an extra library > here. I would just define them manually like: > > DISCORD = 1 > TELEGRAM = 2 > > Thoughts? > You want so? But, I really like to use enum library here like everybody does. ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v3 8/9] enum: Add Platform enumeration 2022-10-21 14:04 ` Muhammad Rizki @ 2022-10-21 14:21 ` Ammar Faizi 2022-10-21 14:34 ` Muhammad Rizki 0 siblings, 1 reply; 40+ messages in thread From: Ammar Faizi @ 2022-10-21 14:21 UTC (permalink / raw) To: Muhammad Rizki; +Cc: Alviro Iskandar Setiawan, GNU/Weeb Mailing List On 10/21/22 9:04 PM, Muhammad Rizki wrote: > You want so? But, I really like to use enum library here like everybody does. "Like everybody does" is not a real reason. Please convince me with the advantages of doing your way. For example, see how I argue below: I prefer to use a constant enum because: 1) It reduces the function call and you don't have to import an extra library. It's simpler to use and to write, it doesn't have to call a function to define the value. 2) Better for debugging, you can see the value of each enum member by reading the source code. This is good for debugging because at some point you can have a @var_x where it was assigned with a value from that enum. When you try to print(var_x), it's just a number. How do you know what number it refers to if you don't know the value of each enum from reading the source code? Say, if you have: TELEGRAM = 1 DISCORD = 2 And then somewhere else you have: var_x = TELEGRAM And somewhere else you use that var_x, you know that var_x value must be within the enum range, and you want to see the value, so you do: print(var_x) It shows 1, you know it's TELEGRAM because you write 1 in the source code. Auto enum throws away that information. If using auto generated enum can give us something better at all, what is that? Tell me. -- Ammar Faizi ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v3 8/9] enum: Add Platform enumeration 2022-10-21 14:21 ` Ammar Faizi @ 2022-10-21 14:34 ` Muhammad Rizki 2022-10-21 14:45 ` Ammar Faizi 0 siblings, 1 reply; 40+ messages in thread From: Muhammad Rizki @ 2022-10-21 14:34 UTC (permalink / raw) To: Ammar Faizi; +Cc: Alviro Iskandar Setiawan, GNU/Weeb Mailing List On 21/10/2022 21.21, Ammar Faizi wrote: > 1) It reduces the function call and you don't have to import an > extra library. It's simpler to use and to write, it doesn't > have to call a function to define the value. > So, you don't want to use extra library or worry about the enum.auto()? If you worry about the enum.auto(), then I just simply declare it as a number. If you don't want to use the extra library then I follow what you want here. > > If using auto generated enum can give us something better at all, > what is that? Tell me. > It just auto generated name to make it readable when printing out or use it as a string value. You can see at gen_temp() in utils.py, I use it as its string name value to choose where the temporary folder to create is. It's simpler, if you want to declare it as an integer, then it's more way complicated because it will use an if statement like the old one does. What do you think? ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v3 8/9] enum: Add Platform enumeration 2022-10-21 14:34 ` Muhammad Rizki @ 2022-10-21 14:45 ` Ammar Faizi 2022-10-21 14:54 ` Muhammad Rizki 0 siblings, 1 reply; 40+ messages in thread From: Ammar Faizi @ 2022-10-21 14:45 UTC (permalink / raw) To: Muhammad Rizki; +Cc: Alviro Iskandar Setiawan, GNU/Weeb Mailing List On 10/21/22 9:34 PM, Muhammad Rizki wrote: > So, you don't want to use extra library or worry about the enum.auto()? > If you worry about the enum.auto(), then I just simply declare it as a > number. If you don't want to use the extra library then I follow what > you want here. I'm not against using an extra library, I just don't see the need of using a library to define enum. That's why I'm bothered. > It just auto generated name to make it readable when printing out or use > it as a string value. You can see at gen_temp() in utils.py, I use it as > its string name value to choose where the temporary folder to create is. > It's simpler, if you want to declare it as an integer, then it's more > way complicated because it will use an if statement like the old one does. > > What do you think? Oh, I didn't know. I thought auto() just returns a simple unique number. I have a question about that. x = Platform.DISCORD print(x) What does the print() statement show? And if it shows something about DISCORD. How does it save the information about DISCORD by only defining it as: DISCORD = enum.auto() Care to explain? -- Ammar Faizi ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v3 8/9] enum: Add Platform enumeration 2022-10-21 14:45 ` Ammar Faizi @ 2022-10-21 14:54 ` Muhammad Rizki 2022-10-21 15:07 ` Ammar Faizi 2022-10-21 15:08 ` Ammar Faizi 0 siblings, 2 replies; 40+ messages in thread From: Muhammad Rizki @ 2022-10-21 14:54 UTC (permalink / raw) To: Ammar Faizi; +Cc: Alviro Iskandar Setiawan, GNU/Weeb Mailing List On 21/10/2022 21.45, Ammar Faizi wrote: > > Oh, I didn't know. I thought auto() just returns a simple unique > number. I have a question about that. If returns a simple unique number it should use `enum.IntEnum` > > x = Platform.DISCORD > print(x) > > What does the print() statement show? > > And if it shows something about DISCORD. How does it save the > information about DISCORD by only defining it as: > > DISCORD = enum.auto() > > Care to explain? > If use it like that it will return `<enum.auto object at 0x7fc59766c8d0>` and if use like `DISCORD.value` it will return `<object object at 0x7fc5c8dc5d30>` What do you think? ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v3 8/9] enum: Add Platform enumeration 2022-10-21 14:54 ` Muhammad Rizki @ 2022-10-21 15:07 ` Ammar Faizi 2022-10-21 15:10 ` Muhammad Rizki 2022-10-21 15:08 ` Ammar Faizi 1 sibling, 1 reply; 40+ messages in thread From: Ammar Faizi @ 2022-10-21 15:07 UTC (permalink / raw) To: Muhammad Rizki; +Cc: Alviro Iskandar Setiawan, GNU/Weeb Mailing List On 10/21/22 9:54 PM, Muhammad Rizki wrote: > If use it like that it will return `<enum.auto object at > 0x7fc59766c8d0>` and if use like `DISCORD.value` it will return `<object > object at 0x7fc5c8dc5d30>` Maybe it's a language feature, but I'm not a Python guy, I really don't know. How does DISCORD.value save the string information about "discord"? Where did you put that information? Does Python reflects a variable name to something? -- Ammar Faizi ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v3 8/9] enum: Add Platform enumeration 2022-10-21 15:07 ` Ammar Faizi @ 2022-10-21 15:10 ` Muhammad Rizki 0 siblings, 0 replies; 40+ messages in thread From: Muhammad Rizki @ 2022-10-21 15:10 UTC (permalink / raw) To: Ammar Faizi; +Cc: Alviro Iskandar Setiawan, GNU/Weeb Mailing List On 21/10/2022 22.07, Ammar Faizi wrote: > On 10/21/22 9:54 PM, Muhammad Rizki wrote: >> If use it like that it will return `<enum.auto object at >> 0x7fc59766c8d0>` and if use like `DISCORD.value` it will return `<object >> object at 0x7fc5c8dc5d30>` > > Maybe it's a language feature, but I'm not a Python guy, I really > don't know. How does DISCORD.value save the string information about > "discord"? > > Where did you put that information? > > Does Python reflects a variable name to something? > It's from the enum library itself. You can see at https://docs.python.org/3/library/enum.html#using-a-descriptive-string ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v3 8/9] enum: Add Platform enumeration 2022-10-21 14:54 ` Muhammad Rizki 2022-10-21 15:07 ` Ammar Faizi @ 2022-10-21 15:08 ` Ammar Faizi 2022-10-21 15:12 ` Muhammad Rizki 1 sibling, 1 reply; 40+ messages in thread From: Ammar Faizi @ 2022-10-21 15:08 UTC (permalink / raw) To: Muhammad Rizki; +Cc: Alviro Iskandar Setiawan, GNU/Weeb Mailing List On 10/21/22 9:54 PM, Muhammad Rizki wrote: > If use it like that it will return `<enum.auto object at > 0x7fc59766c8d0>` and if use like `DISCORD.value` it will return `<object > object at 0x7fc5c8dc5d30>` Maybe it's a language feature, but I'm not a Python guy, I really don't know. How does DISCORD.value save the string information about "discord"? Where did you put that information? Does Python reflect a variable name to something? -- Ammar Faizi ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v3 8/9] enum: Add Platform enumeration 2022-10-21 15:08 ` Ammar Faizi @ 2022-10-21 15:12 ` Muhammad Rizki 2022-10-21 15:21 ` Ammar Faizi 0 siblings, 1 reply; 40+ messages in thread From: Muhammad Rizki @ 2022-10-21 15:12 UTC (permalink / raw) To: Ammar Faizi; +Cc: Alviro Iskandar Setiawan, GNU/Weeb Mailing List On 21/10/2022 22.08, Ammar Faizi wrote: > On 10/21/22 9:54 PM, Muhammad Rizki wrote: >> If use it like that it will return `<enum.auto object at >> 0x7fc59766c8d0>` and if use like `DISCORD.value` it will return `<object >> object at 0x7fc5c8dc5d30>` > > Maybe it's a language feature, but I'm not a Python guy, I really > don't know. How does DISCORD.value save the string information about > "discord"? > > Where did you put that information? > > Does Python reflect a variable name to something? > It's from the enum library itself. You can see at https://docs.python.org/3/library/enum.html#using-a-descriptive-string ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v3 8/9] enum: Add Platform enumeration 2022-10-21 15:12 ` Muhammad Rizki @ 2022-10-21 15:21 ` Ammar Faizi 2022-10-21 15:35 ` Muhammad Rizki 2022-10-21 17:03 ` Alviro Iskandar Setiawan 0 siblings, 2 replies; 40+ messages in thread From: Ammar Faizi @ 2022-10-21 15:21 UTC (permalink / raw) To: Muhammad Rizki; +Cc: Alviro Iskandar Setiawan, GNU/Weeb Mailing List On 10/21/22 10:12 PM, Muhammad Rizki wrote: > It's from the enum library itself. You can see at https://docs.python.org/3/library/enum.html#using-a-descriptive-string OK, I leave the decision to you, feel free to use this fancy enum library. With the function name change and review tag from Al picked up, let's get the v4 applied. Thanks. -- Ammar Faizi ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v3 8/9] enum: Add Platform enumeration 2022-10-21 15:21 ` Ammar Faizi @ 2022-10-21 15:35 ` Muhammad Rizki 2022-10-21 15:37 ` Ammar Faizi 2022-10-21 15:39 ` Ammar Faizi 2022-10-21 17:03 ` Alviro Iskandar Setiawan 1 sibling, 2 replies; 40+ messages in thread From: Muhammad Rizki @ 2022-10-21 15:35 UTC (permalink / raw) To: Ammar Faizi; +Cc: Alviro Iskandar Setiawan, GNU/Weeb Mailing List On 21/10/2022 22.21, Ammar Faizi wrote: > On 10/21/22 10:12 PM, Muhammad Rizki wrote: >> It's from the enum library itself. You can see at >> https://docs.python.org/3/library/enum.html#using-a-descriptive-string > > OK, I leave the decision to you, feel free to use this fancy enum > library. With the function name change and review tag from Al picked > up, let's get the v4 applied. > > Thanks. > OK. But, I catched error just now. Traceback (most recent call last): File "D:\Riski\Coding\Github\lore-daemon\daemon\telegram\mailer\listener.py", line 50, in __run await self.__handle_atom_url(url) File "D:\Riski\Coding\Github\lore-daemon\daemon\telegram\mailer\listener.py", line 69, in __handle_atom_url await self.__handle_mail(url, mail) File "D:\Riski\Coding\Github\lore-daemon\daemon\telegram\mailer\listener.py", line 76, in __handle_mail should_wait = await self.__send_mail(url, mail, File "D:\Riski\Coding\Github\lore-daemon\daemon\telegram\mailer\listener.py", line 102, in __send_mail text, files, is_patch = utils.create_template(mail, Platform.TELEGRAM) File "D:\Riski\Coding\Github\lore-daemon\daemon\atom\utils.py", line 199, in create_template content, files = extract_body(thread, platform) File "D:\Riski\Coding\Github\lore-daemon\daemon\atom\utils.py", line 137, in extract_body p = get_decoded_payload(thread) File "D:\Riski\Coding\Github\lore-daemon\daemon\atom\utils.py", line 264, in get_decoded_payload return quopri.decodestring(p.encode()).decode() UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf6 in position 75: invalid start byte It's from the quoted-printable transfer encoding. See: https://bytetool.web.app/en/ascii/code/0xf6/ Can we just ignore it temporary or solve it now? It's hard to catch them again, I didn't catch the raw URL. ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v3 8/9] enum: Add Platform enumeration 2022-10-21 15:35 ` Muhammad Rizki @ 2022-10-21 15:37 ` Ammar Faizi 2022-10-21 15:54 ` Muhammad Rizki 2022-10-21 15:39 ` Ammar Faizi 1 sibling, 1 reply; 40+ messages in thread From: Ammar Faizi @ 2022-10-21 15:37 UTC (permalink / raw) To: Muhammad Rizki; +Cc: Alviro Iskandar Setiawan, GNU/Weeb Mailing List On 10/21/22 10:35 PM, Muhammad Rizki wrote: > Can we just ignore it temporary or solve it now? It's hard to catch them again, I didn't catch the raw URL. OK, let's defer the fix for this one. -- Ammar Faizi ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v3 8/9] enum: Add Platform enumeration 2022-10-21 15:37 ` Ammar Faizi @ 2022-10-21 15:54 ` Muhammad Rizki 2022-10-21 15:59 ` Ammar Faizi 0 siblings, 1 reply; 40+ messages in thread From: Muhammad Rizki @ 2022-10-21 15:54 UTC (permalink / raw) To: Ammar Faizi; +Cc: Alviro Iskandar Setiawan, GNU/Weeb Mailing List On 21/10/2022 22.37, Ammar Faizi wrote: > On 10/21/22 10:35 PM, Muhammad Rizki wrote: >> Can we just ignore it temporary or solve it now? It's hard to catch >> them again, I didn't catch the raw URL. > > OK, let's defer the fix for this one. > I think we should use decode() <- and put the Content-Type charset of the email? like quopri.decodestring(p.encode()).decode("iso-8859-1") I was catch the same error with different byte issue and trying to do decode with it's specific charset. Sample: https://lore.kernel.org/all/PH1P110MB1068C08C780AD75F20DC9BF0A42D9@PH1P110MB1068.NAMP110.PROD.OUTLOOK.COM/ ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v3 8/9] enum: Add Platform enumeration 2022-10-21 15:54 ` Muhammad Rizki @ 2022-10-21 15:59 ` Ammar Faizi 0 siblings, 0 replies; 40+ messages in thread From: Ammar Faizi @ 2022-10-21 15:59 UTC (permalink / raw) To: Muhammad Rizki; +Cc: Alviro Iskandar Setiawan, GNU/Weeb Mailing List On 10/21/22 10:54 PM, Muhammad Rizki wrote: > I think we should use decode() <- and put the Content-Type charset of > the email? like quopri.decodestring(p.encode()).decode("iso-8859-1") I don't have any comment on that. Let's do that if it fixes the issue. I hope it doesn't introduce a new content corruption bug. > I was catch the same error with different byte issue and trying to do > decode with it's specific charset. > > Sample: > https://lore.kernel.org/all/PH1P110MB1068C08C780AD75F20DC9BF0A42D9@PH1P110MB1068.NAMP110.PROD.OUTLOOK.COM/ Bah, that's a disgusting email. It's top posting and it wraps all URLs with totally unreadable safelink. -- Ammar Faizi ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v3 8/9] enum: Add Platform enumeration 2022-10-21 15:35 ` Muhammad Rizki 2022-10-21 15:37 ` Ammar Faizi @ 2022-10-21 15:39 ` Ammar Faizi 2022-10-21 15:45 ` Muhammad Rizki 1 sibling, 1 reply; 40+ messages in thread From: Ammar Faizi @ 2022-10-21 15:39 UTC (permalink / raw) To: Muhammad Rizki; +Cc: Alviro Iskandar Setiawan, GNU/Weeb Mailing List On 10/21/22 10:35 PM, Muhammad Rizki wrote: > I didn't catch the raw URL. But you know the atom? If it's a recent email, there is a big chance you'll hit the same email again if you reset the table. -- Ammar Faizi ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v3 8/9] enum: Add Platform enumeration 2022-10-21 15:39 ` Ammar Faizi @ 2022-10-21 15:45 ` Muhammad Rizki 2022-10-21 15:48 ` Ammar Faizi 0 siblings, 1 reply; 40+ messages in thread From: Muhammad Rizki @ 2022-10-21 15:45 UTC (permalink / raw) To: Ammar Faizi; +Cc: Alviro Iskandar Setiawan, GNU/Weeb Mailing List On 21/10/2022 22.39, Ammar Faizi wrote: > On 10/21/22 10:35 PM, Muhammad Rizki wrote: >> I didn't catch the raw URL. > > But you know the atom? If it's a recent email, there is a big > chance you'll hit the same email again if you reset the table. > It's from the `all` topic and I don't know the raw URL. ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v3 8/9] enum: Add Platform enumeration 2022-10-21 15:45 ` Muhammad Rizki @ 2022-10-21 15:48 ` Ammar Faizi 2022-10-21 18:29 ` Muhammad Rizki 0 siblings, 1 reply; 40+ messages in thread From: Ammar Faizi @ 2022-10-21 15:48 UTC (permalink / raw) To: Muhammad Rizki; +Cc: Alviro Iskandar Setiawan, GNU/Weeb Mailing List On 10/21/22 10:45 PM, Muhammad Rizki wrote: > It's from the `all` topic and I don't know the raw URL. Ah, that's the most crowded lore path. Ignore it for now. We probably should log any error and what email it's processing when the error occurred. But that's for another series... -- Ammar Faizi ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v3 8/9] enum: Add Platform enumeration 2022-10-21 15:48 ` Ammar Faizi @ 2022-10-21 18:29 ` Muhammad Rizki 2022-10-21 18:53 ` Ammar Faizi 0 siblings, 1 reply; 40+ messages in thread From: Muhammad Rizki @ 2022-10-21 18:29 UTC (permalink / raw) To: Ammar Faizi; +Cc: Alviro Iskandar Setiawan, GNU/Weeb Mailing List On 21/10/2022 22.48, Ammar Faizi wrote: > On 10/21/22 10:45 PM, Muhammad Rizki wrote: >> It's from the `all` topic and I don't know the raw URL. > > Ah, that's the most crowded lore path. Ignore it for now. > > We probably should log any error and what email it's processing > when the error occurred. But that's for another series... > OK, I will create a log for it but maybe I'll do it kinda slow as long as its not a fatal error, because I'm working on my portofolio for finding a permanent job with enough salary for my future. ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v3 8/9] enum: Add Platform enumeration 2022-10-21 18:29 ` Muhammad Rizki @ 2022-10-21 18:53 ` Ammar Faizi 2022-10-21 19:05 ` Alviro Iskandar Setiawan 2022-10-21 19:07 ` Muhammad Rizki 0 siblings, 2 replies; 40+ messages in thread From: Ammar Faizi @ 2022-10-21 18:53 UTC (permalink / raw) To: Muhammad Rizki; +Cc: Alviro Iskandar Setiawan, GNU/Weeb Mailing List On 10/22/22 1:29 AM, Muhammad Rizki wrote: > OK, I will create a log for it but maybe I'll do it kinda slow as long > as its not a fatal error, because I'm working on my portofolio for > finding a permanent job with enough salary for my future. Good luck. I'm looking forward to the day when you've become an expert Software Engineer. I don't know whether you'll take a path to become a kernel dev. But there was a hope that you'll do so. You showed an interest in that area as well, but you didn't take that seriously. I won't care anyway. Ignore. Anyway, we won't run out of things to work on anytime soon. But it won't be stressful work concerning the deadline. So have fun. And don't bother reporting your condition if you're busy. I'll definitely pay you more when the number of tasks increases, and also give you a bonus sometimes. Best regards, -- Ammar Faizi ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v3 8/9] enum: Add Platform enumeration 2022-10-21 18:53 ` Ammar Faizi @ 2022-10-21 19:05 ` Alviro Iskandar Setiawan 2022-10-21 19:15 ` Ammar Faizi 2022-10-21 19:07 ` Muhammad Rizki 1 sibling, 1 reply; 40+ messages in thread From: Alviro Iskandar Setiawan @ 2022-10-21 19:05 UTC (permalink / raw) To: Ammar Faizi; +Cc: Muhammad Rizki, GNU/Weeb Mailing List On Sat, Oct 22, 2022 at 1:53 AM Ammar Faizi wrote: > Good luck. I'm looking forward to the day when you've become an expert > Software Engineer. I don't know whether you'll take a path to become a > kernel dev. But there was a hope that you'll do so. You showed an > interest in that area as well, but you didn't take that seriously. I > won't care anyway. Ignore. Be careful with your charismatic words, most people who want to be a kernel dev will end up like Fernanda. Other examples are not bad enough as long as they're not a parasite like that hypocrite person. -- Viro ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v3 8/9] enum: Add Platform enumeration 2022-10-21 19:05 ` Alviro Iskandar Setiawan @ 2022-10-21 19:15 ` Ammar Faizi 2022-10-21 19:24 ` Alviro Iskandar Setiawan 0 siblings, 1 reply; 40+ messages in thread From: Ammar Faizi @ 2022-10-21 19:15 UTC (permalink / raw) To: Alviro Iskandar Setiawan; +Cc: Muhammad Rizki, GNU/Weeb Mailing List On 10/22/22 2:05 AM, Alviro Iskandar Setiawan wrote: > On Sat, Oct 22, 2022 at 1:53 AM Ammar Faizi wrote: >> Good luck. I'm looking forward to the day when you've become an expert >> Software Engineer. I don't know whether you'll take a path to become a >> kernel dev. But there was a hope that you'll do so. You showed an >> interest in that area as well, but you didn't take that seriously. I >> won't care anyway. Ignore. > > Be careful with your charismatic words, most people who want to be a > kernel dev will end up like Fernanda. Other examples are not bad > enough as long as they're not a parasite like that hypocrite person. I'm not stupid enough to fall in the same hole twice. Not even a donkey falls in the same hole twice. -- Ammar Faizi ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v3 8/9] enum: Add Platform enumeration 2022-10-21 19:15 ` Ammar Faizi @ 2022-10-21 19:24 ` Alviro Iskandar Setiawan 0 siblings, 0 replies; 40+ messages in thread From: Alviro Iskandar Setiawan @ 2022-10-21 19:24 UTC (permalink / raw) To: Ammar Faizi; +Cc: Muhammad Rizki, GNU/Weeb Mailing List On Sat, Oct 22, 2022 at 2:15 AM Ammar Faizi wrote: > On 10/22/22 2:05 AM, Alviro Iskandar Setiawan wrote: >> On Sat, Oct 22, 2022 at 1:53 AM Ammar Faizi wrote: >>> Good luck. I'm looking forward to the day when you've become an expert >>> Software Engineer. I don't know whether you'll take a path to become a >>> kernel dev. But there was a hope that you'll do so. You showed an >>> interest in that area as well, but you didn't take that seriously. I >>> won't care anyway. Ignore. >> >> Be careful with your charismatic words, most people who want to be a >> kernel dev will end up like Fernanda. Other examples are not bad >> enough as long as they're not a parasite like that hypocrite person. > > I'm not stupid enough to fall in the same hole twice. Not even a donkey > falls in the same hole twice. I'm looking forward to the day when I punch your face after you fall in the same hole twice. Keep that in mind! -- Viro ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v3 8/9] enum: Add Platform enumeration 2022-10-21 18:53 ` Ammar Faizi 2022-10-21 19:05 ` Alviro Iskandar Setiawan @ 2022-10-21 19:07 ` Muhammad Rizki 1 sibling, 0 replies; 40+ messages in thread From: Muhammad Rizki @ 2022-10-21 19:07 UTC (permalink / raw) To: Ammar Faizi; +Cc: Alviro Iskandar Setiawan, GNU/Weeb Mailing List On 22/10/2022 01.53, Ammar Faizi wrote: > > Good luck. I'm looking forward to the day when you've become an expert > Software Engineer. I don't know whether you'll take a path to become a > kernel dev. But there was a hope that you'll do so. You showed an > interest in that area as well, but you didn't take that seriously. I > won't care anyway. Ignore. Actually I would, but it will take years for me just to learn the C language to intermediate/pro level. Because my first language is Python so its kinda hard to learn compiler language especially C/C++. > > Anyway, we won't run out of things to work on anytime soon. But it won't > be stressful work concerning the deadline. So have fun. And don't bother > reporting your condition if you're busy. I'll definitely pay you more > when the number of tasks increases, and also give you a bonus sometimes. > > Best regards, Thanks, its nice to work with you on this project. ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [PATCH v3 8/9] enum: Add Platform enumeration 2022-10-21 15:21 ` Ammar Faizi 2022-10-21 15:35 ` Muhammad Rizki @ 2022-10-21 17:03 ` Alviro Iskandar Setiawan 1 sibling, 0 replies; 40+ messages in thread From: Alviro Iskandar Setiawan @ 2022-10-21 17:03 UTC (permalink / raw) To: Ammar Faizi; +Cc: Muhammad Rizki, GNU/Weeb Mailing List On Fri, Oct 21, 2022 at 10:21 PM Ammar Faizi wrote: > On 10/21/22 10:12 PM, Muhammad Rizki wrote: > > It's from the enum library itself. You can see at https://docs.python.org/3/library/enum.html#using-a-descriptive-string > > OK, I leave the decision to you, feel free to use this fancy enum > library. With the function name change and review tag from Al picked > up, let's get the v4 applied. Acked-by: Alviro Iskandar Setiawan <[email protected]> tq -- Viro ^ permalink raw reply [flat|nested] 40+ messages in thread
* [PATCH v3 9/9] enum: Use the created Platform class enumeration 2022-10-21 13:45 [PATCH v3 0/9] Fix some bugs and add some features Muhammad Rizki ` (7 preceding siblings ...) 2022-10-21 13:45 ` [PATCH v3 8/9] enum: Add Platform enumeration Muhammad Rizki @ 2022-10-21 13:45 ` Muhammad Rizki 8 siblings, 0 replies; 40+ messages in thread From: Muhammad Rizki @ 2022-10-21 13:45 UTC (permalink / raw) To: Ammar Faizi Cc: Muhammad Rizki, Alviro Iskandar Setiawan, GNU/Weeb Mailing List Use the Platform class enumeration to make it more practice, avoid getting typo, type-hinted suggestion to easily manageable-maintainable. Signed-off-by: Muhammad Rizki <[email protected]> --- daemon/atom/utils.py | 28 ++++++++----------- daemon/dscord/gnuweeb/client.py | 5 ++-- .../plugins/slash_commands/get_lore_mail.py | 3 +- daemon/dscord/mailer/listener.py | 3 +- daemon/telegram/mailer/listener.py | 4 +-- daemon/telegram/packages/client.py | 3 +- .../packages/plugins/commands/scrape.py | 4 +-- 7 files changed, 25 insertions(+), 25 deletions(-) diff --git a/daemon/atom/utils.py b/daemon/atom/utils.py index 6a9e5c8..b7a7aaf 100644 --- a/daemon/atom/utils.py +++ b/daemon/atom/utils.py @@ -4,6 +4,8 @@ # Copyright (C) 2022 Ammar Faizi <[email protected]> # +from enums import Platform + from pyrogram.types import Chat, InlineKeyboardMarkup, InlineKeyboardButton from email.message import Message from typing import Dict, Union @@ -115,19 +117,13 @@ def consruct_to_n_cc(to: list, cc: list): return ret -def gen_temp(name: str, platform: str): - platform = platform.lower() - plt_ls = ["telegram", "discord"] - - if platform not in plt_ls: - t = f"Platform {platform} is not found, " - t += f"only {', '.join(plt_ls)} is available" - raise ValueError(f"Platform {platform} is not found") - +def gen_temp(name: str, platform: Platform): + platform: str = platform.value md5 = hashlib.md5(name.encode()).hexdigest() store_dir = os.getenv("STORAGE_DIR", "storage") platform = platform.replace("discord", "dscord") path = f"{platform}/{store_dir}/{md5}" + try: os.mkdir(path) except FileExistsError: @@ -136,11 +132,11 @@ def gen_temp(name: str, platform: str): return path -def extract_body(thread: Message, platform: str): +def extract_body(thread: Message, platform: Platform): if not thread.is_multipart(): p = manage_payload(thread) - if platform == "discord": + if platform is Platform.DISCORD: p = quote_reply(p) return f"{p}\n".lstrip(), [] @@ -183,12 +179,12 @@ def __is_patch(subject, content): return True -def create_template(thread: Message, platform: str, to=None, cc=None): +def create_template(thread: Message, platform: Platform, to=None, cc=None): if not to: to = extract_list("to", thread) if not cc: cc = extract_list("cc", thread) - if platform == "telegram": + if platform is Platform.TELEGRAM: substr = 4000 border = f"\n<code>{'-'*72}</code>" else: @@ -211,13 +207,13 @@ def create_template(thread: Message, platform: str, to=None, cc=None): if len(ret) >= substr: ret = ret[:substr] + "..." - ret = fix_utf8_char(ret, platform == "telegram") + ret = fix_utf8_char(ret, platform is Platform.TELEGRAM) ret += border return ret, files, is_patch -def prepare_patch(mail, text, url, platform: str): +def prepare_patch(mail, text, url, platform: Platform): tmp = gen_temp(url, platform) fnm = str(mail.get("subject")) sch = re.search(PATCH_PATTERN, fnm, re.IGNORECASE) @@ -237,7 +233,7 @@ def prepare_patch(mail, text, url, platform: str): f.write(bytes(text, encoding="utf8")) caption = "#patch #ml" - if platform == "telegram": + if platform is Platform.TELEGRAM: caption += fix_utf8_char("\n" + cap, True) return tmp, file, caption, url diff --git a/daemon/dscord/gnuweeb/client.py b/daemon/dscord/gnuweeb/client.py index b921f7d..82858c5 100644 --- a/daemon/dscord/gnuweeb/client.py +++ b/daemon/dscord/gnuweeb/client.py @@ -13,6 +13,7 @@ from typing import Union from . import filters from . import models from atom import utils +from enums import Platform from dscord.database import DB @@ -59,7 +60,7 @@ class GWClient(commands.Bot): reply_to: Union[int, None] = None, url: str = None): print("[send_patch_email]") tmp, doc, caption, url = utils.prepare_patch( - mail, text, url, "discord" + mail, text, url, Platform.DISCORD ) channel = self.get_channel(chat_id) @@ -90,7 +91,7 @@ class GWClient(commands.Bot): async def send_patch_mail_interaction(self, mail, i: "Interaction", text: str, url: str = None): tmp, doc, caption, url = utils.prepare_patch( - mail, text, url, "discord" + mail, text, url, Platform.DISCORD ) m = await i.response.send_message( content=caption, diff --git a/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py b/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py index 2d55d16..0c67b8c 100644 --- a/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py +++ b/daemon/dscord/gnuweeb/plugins/slash_commands/get_lore_mail.py @@ -11,6 +11,7 @@ from discord import app_commands from atom import utils from atom import Scraper +from enums import Platform class GetLoreSC(commands.Cog): @@ -26,7 +27,7 @@ class GetLoreSC(commands.Cog): async def get_lore(self, i: "Interaction", url: str): s = Scraper() mail = await s.get_email_from_url(url) - text, _, is_patch = utils.create_template(mail, "discord") + text, _, is_patch = utils.create_template(mail, Platform.DISCORD) if is_patch: m = await self.bot.send_patch_mail_interaction( diff --git a/daemon/dscord/mailer/listener.py b/daemon/dscord/mailer/listener.py index cc0a9f7..d986fbd 100644 --- a/daemon/dscord/mailer/listener.py +++ b/daemon/dscord/mailer/listener.py @@ -14,6 +14,7 @@ from discord import Message from dscord.gnuweeb import GWClient from atom.scraper import Scraper from atom import utils +from enums import Platform class Mutexes: @@ -107,7 +108,7 @@ class Listener: # return False - text, files, is_patch = utils.create_template(mail, "discord") + text, files, is_patch = utils.create_template(mail, Platform.DISCORD) reply_to = self.get_discord_reply(mail, dc_chat_id) url = str(re.sub(r"/raw$", "", url)) diff --git a/daemon/telegram/mailer/listener.py b/daemon/telegram/mailer/listener.py index 08feddd..1c92f23 100644 --- a/daemon/telegram/mailer/listener.py +++ b/daemon/telegram/mailer/listener.py @@ -9,8 +9,8 @@ from apscheduler.schedulers.asyncio import AsyncIOScheduler from telegram.packages import DaemonClient from atom import Scraper from atom import utils +from enums import Platform import asyncio -import shutil import re import traceback @@ -99,7 +99,7 @@ class Bot(): # return False - text, files, is_patch = utils.create_template(mail, "telegram") + text, files, is_patch = utils.create_template(mail, Platform.TELEGRAM) reply_to = self.get_reply(mail, tg_chat_id) url = str(re.sub(r"/raw$", "", url)) diff --git a/daemon/telegram/packages/client.py b/daemon/telegram/packages/client.py index 686e5ef..c971ea1 100644 --- a/daemon/telegram/packages/client.py +++ b/daemon/telegram/packages/client.py @@ -9,6 +9,7 @@ from pyrogram.types import Message, InlineKeyboardMarkup, InlineKeyboardButton from typing import Union from email.message import Message from atom import utils +from enums import Platform from telegram.database import DB from .decorator import handle_flood @@ -57,7 +58,7 @@ class DaemonClient(Client): ) -> Message: print("[send_patch_email]") tmp, doc, caption, url = utils.prepare_patch( - mail, text, url, "telegram" + mail, text, url, Platform.TELEGRAM ) m = await self.send_document( chat_id=chat_id, diff --git a/daemon/telegram/packages/plugins/commands/scrape.py b/daemon/telegram/packages/plugins/commands/scrape.py index 860e993..29cc8ad 100644 --- a/daemon/telegram/packages/plugins/commands/scrape.py +++ b/daemon/telegram/packages/plugins/commands/scrape.py @@ -9,8 +9,8 @@ from pyrogram import filters from telegram.packages import DaemonClient from atom import Scraper from atom import utils +from enums import Platform from telegram import config -import shutil import re import asyncio @@ -37,7 +37,7 @@ async def scrap_email(c: DaemonClient, m: Message): s = Scraper() mail = await s.get_email_from_url(url) - text, files, is_patch = utils.create_template(mail, "telegram") + text, files, is_patch = utils.create_template(mail, Platform.TELEGRAM) if is_patch: m = await c.send_patch_email( -- Muhammad Rizki ^ permalink raw reply related [flat|nested] 40+ messages in thread
end of thread, other threads:[~2022-10-21 19:25 UTC | newest] Thread overview: 40+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2022-10-21 13:45 [PATCH v3 0/9] Fix some bugs and add some features Muhammad Rizki 2022-10-21 13:45 ` [PATCH v3 1/9] discord: Add send_text_mail_interaction() Muhammad Rizki 2022-10-21 13:45 ` [PATCH v3 2/9] discord: Add send_patch_mail_interaction() Muhammad Rizki 2022-10-21 13:45 ` [PATCH v3 3/9] discord: Add get lore mail slash command Muhammad Rizki 2022-10-21 13:45 ` [PATCH v3 4/9] atom: Improve remove_patch() Muhammad Rizki 2022-10-21 13:45 ` [PATCH v3 5/9] atom: add manage_payload() Muhammad Rizki 2022-10-21 14:01 ` Ammar Faizi 2022-10-21 14:02 ` Muhammad Rizki 2022-10-21 14:06 ` Ammar Faizi 2022-10-21 14:07 ` Muhammad Rizki 2022-10-21 13:45 ` [PATCH v3 6/9] telegram: Fix get lore command Muhammad Rizki 2022-10-21 13:45 ` [PATCH v3 7/9] atom: Improve extract_body() Muhammad Rizki 2022-10-21 14:34 ` Alviro Iskandar Setiawan 2022-10-21 13:45 ` [PATCH v3 8/9] enum: Add Platform enumeration Muhammad Rizki 2022-10-21 14:02 ` Ammar Faizi 2022-10-21 14:04 ` Muhammad Rizki 2022-10-21 14:21 ` Ammar Faizi 2022-10-21 14:34 ` Muhammad Rizki 2022-10-21 14:45 ` Ammar Faizi 2022-10-21 14:54 ` Muhammad Rizki 2022-10-21 15:07 ` Ammar Faizi 2022-10-21 15:10 ` Muhammad Rizki 2022-10-21 15:08 ` Ammar Faizi 2022-10-21 15:12 ` Muhammad Rizki 2022-10-21 15:21 ` Ammar Faizi 2022-10-21 15:35 ` Muhammad Rizki 2022-10-21 15:37 ` Ammar Faizi 2022-10-21 15:54 ` Muhammad Rizki 2022-10-21 15:59 ` Ammar Faizi 2022-10-21 15:39 ` Ammar Faizi 2022-10-21 15:45 ` Muhammad Rizki 2022-10-21 15:48 ` Ammar Faizi 2022-10-21 18:29 ` Muhammad Rizki 2022-10-21 18:53 ` Ammar Faizi 2022-10-21 19:05 ` Alviro Iskandar Setiawan 2022-10-21 19:15 ` Ammar Faizi 2022-10-21 19:24 ` Alviro Iskandar Setiawan 2022-10-21 19:07 ` Muhammad Rizki 2022-10-21 17:03 ` Alviro Iskandar Setiawan 2022-10-21 13:45 ` [PATCH v3 9/9] enum: Use the created Platform class enumeration Muhammad Rizki
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox