[PATCH v1 0/2] Improve some utility codes

[PATCH v1 0/2] Improve some utility codes

[Muhammad Rizki]

Afternoon guys,

In this series, I'm improving some utility codes in scraper/utils.py,
like multiple dash `-` just like "-"*72, I think it's more practice than
before. I fix a bug too while receiving email payload like `<text>`,
`&reg`, and more using double html.escape().

There are 2 patches in this series:
- Patch 1 is to improve bottom border string template.
- Patch 2 is to use double html.escape() for fix_utf8_chars()

This should be works fine and tested too, thanks.

Muhammad Rizki (2):
  Improve code for bottom border
  Use html.escape() for fix_utf8_chars()

 daemon/scraper/utils.py | 11 ++++-------
 1 file changed, 4 insertions(+), 7 deletions(-)


base-commit: f42828bf0aa6fe23f1f5257c4a2483cb8798e89a
-- 
Muhammad Rizki

[PATCH v1 1/2] Improve code for bottom border

[Muhammad Rizki]

Just improve code for creating bottom border for create_template() in
scraper/utils.py

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---
 daemon/scraper/utils.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/daemon/scraper/utils.py b/daemon/scraper/utils.py
index e89255b..765468c 100644
--- a/daemon/scraper/utils.py
+++ b/daemon/scraper/utils.py
@@ -185,7 +185,7 @@ def create_template(thread: Message, to=None, cc=None):
 			ret = ret[:4000] + "..."
 
 		ret = fix_utf8_char(ret)
-		ret += "\n<code>------------------------------------------------------------------------</code>"
+		ret += f"\n<code>{'-'*72}</code>"
 
 	return ret, files, is_patch
 
-- 
Muhammad Rizki

[PATCH v1 2/2] Use html.escape() for fix_utf8_chars()

[Muhammad Rizki]

We found a bug when receiving email payload like `<text>`, `&reg`, and
more. Using double html.escape() will fix the bug for now.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
Reported-by: Alviro Iskandar Setiawan <alviro.iskandar@gnuweeb.org>
---
 daemon/scraper/utils.py | 9 +++------
 1 file changed, 3 insertions(+), 6 deletions(-)

diff --git a/daemon/scraper/utils.py b/daemon/scraper/utils.py
index 765468c..c428a33 100644
--- a/daemon/scraper/utils.py
+++ b/daemon/scraper/utils.py
@@ -14,6 +14,7 @@ import os
 import re
 import shutil
 import httpx
+import html
 
 
 def get_email_msg_id(mail):
@@ -218,12 +219,8 @@ def clean_up_after_send_patch(tmp):
 
 
 def fix_utf8_char(text: str):
-	return (
-		text.rstrip()
-		.replace("<", "&lt;")
-		.replace(">","&gt;")
-		.replace("�"," ")
-	)
+	text = text.rstrip().replace("�"," ")
+	return html.escape(html.escape(text))
 
 
 EMAIL_MSG_ID_PATTERN = r"<([^\<\>]+)>"
-- 
Muhammad Rizki

Re: [PATCH v1 2/2] Use html.escape() for fix_utf8_chars()

[Ammar Faizi]

On 8/3/22 5:27 PM, Muhammad Rizki wrote:
> return html.escape(html.escape(text))

I doubt about this double escape. Just to clear my doubt. Did you test
this?

I didn't, but you said previously that "<" is shown as "&gt;" which is
totally garbage if that's the actual output. Rizki?

-- 
Ammar Faizi

Re: (subset) [PATCH v1 0/2] Improve some utility codes

[Ammar Faizi]

On Wed, 3 Aug 2022 17:27:16 +0700, Muhammad Rizki wrote:
> Afternoon guys,
> 
> In this series, I'm improving some utility codes in scraper/utils.py,
> like multiple dash `-` just like "-"*72, I think it's more practice than
> before. I fix a bug too while receiving email payload like `<text>`,
> `&reg`, and more using double html.escape().
> 
> [...]

Applied, thanks!

[1/2] Improve code for bottom border
      commit: 09960d39aefedd07678bdadb6df26f79c9854c69

Best regards,
-- 
Ammar Faizi

Re: [PATCH v1 2/2] Use html.escape() for fix_utf8_chars()

[Ammar Faizi]

On 8/3/22 7:49 PM, Ammar Faizi wrote:
> On 8/3/22 5:27 PM, Muhammad Rizki wrote:
>> return html.escape(html.escape(text))
> 
> I doubt about this double escape. Just to clear my doubt. Did you test
> this?
> 
> I didn't, but you said previously that "<" is shown as "&gt;" which is
> totally garbage if that's the actual output. Rizki?

I mean ">" is shown as "&gt;".

-- 
Ammar Faizi

Re: [PATCH v1 2/2] Use html.escape() for fix_utf8_chars()

[Muhammad Rizki]

On 03/08/2022 19.49, Ammar Faizi wrote:
> On 8/3/22 5:27 PM, Muhammad Rizki wrote:
>> return html.escape(html.escape(text))
> 
> I doubt about this double escape. Just to clear my doubt. Did you test
> this?

Tested and work as my expected.

> 
> I didn't, but you said previously that "<" is shown as "&gt;" which is
> totally garbage if that's the actual output. Rizki?
> 

Yes, with double html.escape() it can replace the previous code (with 
replace())

Re: [PATCH v1 0/2] Improve some utility codes

[Ammar Faizi]

On Wed, 3 Aug 2022 17:27:16 +0700, Muhammad Rizki wrote:
> Afternoon guys,
> 
> In this series, I'm improving some utility codes in scraper/utils.py,
> like multiple dash `-` just like "-"*72, I think it's more practice than
> before. I fix a bug too while receiving email payload like `<text>`,
> `&reg`, and more using double html.escape().
> 
> [...]

Applied, thanks!

[1/2] Improve code for bottom border
      commit: 09960d39aefedd07678bdadb6df26f79c9854c69
[2/2] Use html.escape() for fix_utf8_chars()
      commit: a6280937ce9623e961bb66a76fad2c2f5cd10244

Best regards,
-- 
Ammar Faizi