[PATCH v4 00/17] Fix, improvement and implement a bot logger.

[Muhammad Rizki <kiizuha@gnuweeb.org>]

Hi sir, it's been a long time.

This is a v4 revision of fix, improvement and implement a bot logger.
this series is to improve create_template() caption substr/trim,
prepare_patch() for the .patch file payload, remove unrelevant changelog
commit in the commit description, use html.escape() once instead of
twice, small change in the first commit to make it clearer.

This series contains fixes in the utility file and some improvement for
it. I also implement a bot logger to log an error to the terminal and
to the log file, if error may happen it will send the log file to the
log channel either for Discord or Telegram, the LOG_CHANNEL_ID is
declared in the config.py.example, copy the channel ID from the Discord
or Telegram channel and then paste to the config.py.example.

What's new?
- Implementation of the BotLogger to log error message.
- Add a catch_err() in the atom/utils.py.
- Add a send_log_file() in each bot client.py.
- Add a variable LOG_CHANNEL_ID to configure the log channel.

Fixes:
- Fix media caption too long error for media/attachment message
- Fix .patch file payload
- Fix TO/CC list when an email doesn't contain TO/CC headers
- Fix typo on _flood_exception()

There are 17 patches in this series:
- Patch 1 is to fix a typo on _flood_exception() for the Discord bot.
- Patch 2 is to Fix .patch file payload to have a full email payload.
- Patch 3 is to add a substr/trim caption on the create_template().
- Patch 4 is to fix an extract_list() when there is no receipients.
- Patch 5 is to use the old one method, use decode=True in get_payload()
- Patch 6 is to improve fix_utf8_char(), make sure to unescape the HTML.
- Patch 7 is to add a utility catch_err() function to get error message.
- Patch 8 is to just add an initial BotLogger() to use it in next commit
- Patch 9 is to use the created BotLogger() for the Telegram bot.
- Patch 10 is to add a Telegram LOG_CHANNEL_ID for the send_log_file().
- Patch 11 is to add a send_log_file() for the Telegram bot DaemonClient
- Patch 12 is to implement the log message and send log file for TG bot.
- Patch 13 is to add a Discord LOG_CHANNEL_ID for the send_log_file().
- Patch 14 is to use the created BotLogger() for the Discord bot.
- Patch 15 is to add a send_log_file() for the Discord bot GWClient().
- Patch 16 is to implement the log message and send log file for DC bot.
- Patch 17 is to remove unused imports.

How to set log channel ID for the Telegram bot:
1. Make a channel for log in the Telegram.
2. Copy the channel ID, if you unable to see the ID you can use API or
   bot who has the feature, or use a 64gram desktop.
3. Paste the copied ID into the config.py in the telegram directory.

How to set log channel ID for the Discord bot:
1. Make a channel for log in your Discord server.
2. Copy the channel ID by right click and copy, if you unable to get the
   ID you can go to your account settings, then enable the
   `Developer mode` or use bot who has the feature.
3. Paste the copied ID into the config.py in the dscord directory.

How to use:
1. Execute the db.sql file in the daemon directory.
2. Setup .env file, the example is there with suffix .example, this
   file name must remove the suffix name .example.
3. Set up the config.py in each bot directory, such as dscord and
   telegram. The example is there with suffix .example & the file name
   must remove suffix name .example.
4. Run `pip3 install -r requirements.txt` in each bot directory.
5. STORAGE_DIR env value must `storage` to make it work fine.
6. Run the bot by `python3 dc.py` or `python3 tg.py`.

** Tested and works, please give it a test, thanks! **

Oh, about the keeping both .env and config.py we can discuss in this
series. I was asking a question for that, but you let it ignore.

## Changelog

v3 -> v4:
  - Change commit description

v2 -> v3:
  - Fix prepare_patch() from media caption too long error.
  - Fix prepare_patch() for the .patch file.
  - Improve fix_utf8_char() function, don't use escape() twice and
    unecessary unescape().

v1 -> v2:
  - Improvement on the create_template() for the substr/trim caption.
  - Fix the prepare_patch() to use the full email payload instead of the
    trimmed caption taken from the `text` parameter.
  - Remove unrelevant changelog in the commit description.
  - Small change in the first commit description to make it clearer to
    read.

Signed-off-by: Muhammad Rizki <kiizuha@gnuweeb.org>
---

Muhammad Rizki (17):
  discord: Fix typo on _flood_exception()
  fix: utils: Fix .patch file payload
  fix: utils: add a substr for the patch media caption
  fix: utils: Fix the extract_list() utility function
  utils: Back to use decode=True for the get_payload()
  utils: Improve fix_utf8_char()
  utils: Add catch_err() for the log message
  logger: Initial work for the bot logger for future use
  telegram: Use the created BotLogger() class
  telegram: Add variable LOG_CHANNEL_ID declaration
  telegram: Add send_log_file() in the DaemonClient()
  telegram: Implement the log message for catching errors
  discord: Add variable LOG_CHANNEL_ID declaration
  discord: Use the BotLogger() to the GWClient()
  discord: Add send_log_file in the GWClient()
  discord: Implement the catch erros and logs
  Remove some unused imports

 daemon/atom/utils.py                          | 67 ++++++++++++++-----
 daemon/dc.py                                  | 10 ++-
 daemon/dscord/config.py.example               |  4 ++
 daemon/dscord/gnuweeb/client.py               | 20 ++++--
 daemon/dscord/gnuweeb/filters.py              | 10 ++-
 .../dscord/gnuweeb/plugins/events/on_ready.py |  2 +-
 .../plugins/slash_commands/get_lore_mail.py   | 12 ++--
 daemon/dscord/mailer/listener.py              | 41 +++++++-----
 daemon/logger/__init__.py                     |  1 +
 daemon/logger/log.py                          | 52 ++++++++++++++
 daemon/telegram/config.py.example             |  4 ++
 daemon/telegram/mailer/listener.py            | 44 +++++++-----
 daemon/telegram/packages/client.py            | 21 +++++-
 daemon/telegram/packages/decorator.py         | 15 ++---
 .../packages/plugins/commands/scrape.py       | 10 ++-
 daemon/tg.py                                  |  5 ++
 16 files changed, 240 insertions(+), 78 deletions(-)
 create mode 100644 daemon/logger/__init__.py
 create mode 100644 daemon/logger/log.py


base-commit: 91e86e6d1004da4526ca5514fa009bc253d5c4a3
--
Muhammad Rizki