[PATCH server-haj002 v1 0/6] Server update and new service elk.gnuweeb.org

[PATCH server-haj002 v1 0/6] Server update and new service elk.gnuweeb.org

[Alviro Iskandar Setiawan]

Greetings,

This series contains server-haj002 fixes and a new service called
elk.gnuweeb.org as requested by chief in https://t.me/GNUWeeb/807595

elk is a nimble Mastodon web client. While it's currently not a
widely used app, it's already quite usable. More detail here:
https://github.com/elk-zone/elk

The elk service for GNU/Weeb is now up and ready to use at:

  https://elk.gnuweeb.org

There are 6 pathces in this series, summary:

# 1. Allow trusted traffic from/to local addresses.
Simplify iptables rules by allowing INPUT traffic from these CIDR
ranges:

  ip:
    10.0.0.0/8
    192.168.0.0/16
    172.16.0.0/12

  ip6:
    fc00::/7

Also, allow FORWARD from/to them.

# 2. Properly enable IPv6 for the masto-net network.
The IPv6 was not properly enabled. Enable it by adding
"enable_ipv6: true" in the network section in the docker-compose
file.

# 3. Fix DNS resolution error.
The default DNS system is not properly configured in this environment
and it leads to DNS resolution error. Provide custom resolv.conf file
and hosts file to satisfy the need of mastodon container.

The custom resolv.conf script will use the host IP address as the DNS
server. Therefore, the host must provide a working DNS server. It can
simply be done by adding these two lines to /etc/systemd/resolved.conf

   DNSStubListenerExtra=0.0.0.0:53
   DNSStubListenerExtra=[::]:53

# 4. Move data directory inside the git project
Move data directory inside the git project to allow the docker-comose.yml
file to use relative path. Then sync the docker-compose file with the
prod file (add hostname entry and DNS related changes).

# 5. Add elk server using docker.
Bash script to build and run the elk service using docker.

# 6. Add elk setup instructions.
Note crontab setup and nginx vhost config.

Signed-off-by: Alviro Iskandar Setiawan <alviro.iskandar@gnuweeb.org>
---

Alviro Iskandar Setiawan (6):
  init_net: Allow trusted traffic from/to local addresses
  mastodon: Enable IPv6 for the masto-net network
  mastodon: Fix DNS resolution error
  mastodon: Move data directory inside the git project
  Add elk server using docker
  setup.txt: Add elk setup instructions

 .gitmodules                         |  3 +++
 elk/.gitignore                      |  2 ++
 elk/docker-compose.yml              | 30 +++++++++++++++++++++++++++++
 elk/start.sh                        | 23 ++++++++++++++++++++++
 init_net.sh                         | 26 ++++++++++++++++++++++++-
 mastodon/.gitignore                 |  3 ++-
 mastodon/docker-compose.example.yml | 19 ++++++++++++------
 mastodon/docker-mastodon            |  1 +
 mastodon/hosts                      |  8 ++++++++
 mastodon/resolv.conf                |  3 +++
 setup.txt                           | 19 ++++++++++++++++++
 11 files changed, 129 insertions(+), 8 deletions(-)
 create mode 100644 .gitmodules
 create mode 100644 elk/.gitignore
 create mode 100644 elk/docker-compose.yml
 create mode 100644 elk/start.sh
 create mode 160000 mastodon/docker-mastodon
 create mode 100644 mastodon/hosts
 create mode 100644 mastodon/resolv.conf


base-commit: 9fb75192df6b55699b8716199b16b42d0b82e860
-- 
Alviro Iskandar Setiawan

[PATCH server-haj002 v1 1/6] init_net: Allow trusted traffic from/to local addresses

[Alviro Iskandar Setiawan]

Simplify iptables rules by allowing INPUT traffic from these CIDR
ranges:

ip:
  10.0.0.0/8
  192.168.0.0/16
  172.16.0.0/12

ip6:
  fc00::/7

Also, allow FORWARD from/to them.

Signed-off-by: Alviro Iskandar Setiawan <alviro.iskandar@gnuweeb.org>
---
 init_net.sh | 26 +++++++++++++++++++++++++-
 1 file changed, 25 insertions(+), 1 deletion(-)

diff --git a/init_net.sh b/init_net.sh
index 404e79b..7d21069 100755
--- a/init_net.sh
+++ b/init_net.sh
@@ -41,6 +41,8 @@ $EM sysctl -w net.ipv6.conf.eth0.proxy_ndp=1;
 $EM sysctl -w net.ipv6.conf.eth1p.proxy_ndp=1;
 $EM sysctl -w net.ipv4.ip_forward=1;
 
+$EM iptables -t nat -F;
+$EM iptables -t nat -X;
 $EM iptables -t nat -I PREROUTING -d $IPV4_PUB -j DNAT --to-destination 10.3.3.2;
 $EM iptables -t nat -I OUTPUT -d $IPV4_PUB -j DNAT --to-destination 10.3.3.2;
 $EM iptables -t nat -I POSTROUTING -s 10.3.3.0/24 ! -d 10.3.3.0/24 -j SNAT --to-source $IPV4_PUB;
@@ -51,13 +53,28 @@ iptables -t filter -P FORWARD ACCEPT;
 iptables -t filter -P OUTPUT ACCEPT;
 iptables -t filter -F;
 iptables -t filter -X;
+
+iptables -t filter -N ACCEPT_LOCAL_SRC;
+iptables -t filter -A ACCEPT_LOCAL_SRC -s 10.0.0.0/8 -j ACCEPT;
+iptables -t filter -A ACCEPT_LOCAL_SRC -s 192.168.0.0/16 -j ACCEPT;
+iptables -t filter -A ACCEPT_LOCAL_SRC -s 172.16.0.0/12 -j ACCEPT;
+iptables -t filter -A ACCEPT_LOCAL_SRC -j RETURN;
+
+iptables -t filter -N ACCEPT_LOCAL_DST;
+iptables -t filter -A ACCEPT_LOCAL_DST -d 10.0.0.0/8 -j ACCEPT;
+iptables -t filter -A ACCEPT_LOCAL_DST -d 192.168.0.0/16 -j ACCEPT;
+iptables -t filter -A ACCEPT_LOCAL_DST -d 172.16.0.0/12 -j ACCEPT;
+iptables -t filter -A ACCEPT_LOCAL_DST -j RETURN;
+
 iptables -t filter -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT;
 iptables -t filter -A INPUT -p tcp -m multiport --dports 80,443,48588 -j ACCEPT;
 iptables -t filter -A INPUT -p icmp -j ACCEPT;
-iptables -t filter -A INPUT -s 10.3.3.0/24 -j ACCEPT;
+iptables -t filter -A INPUT -j ACCEPT_LOCAL_SRC;
 iptables -t filter -A INPUT -i lo -j ACCEPT;
 iptables -t filter -P INPUT DROP;
 
+iptables -t filter -A FORWARD -j ACCEPT_LOCAL_SRC;
+iptables -t filter -A FORWARD -j ACCEPT_LOCAL_DST;
 iptables -t filter -P FORWARD DROP;
 iptables -t filter -P OUTPUT ACCEPT;
 
@@ -65,13 +82,20 @@ iptables -t filter -P OUTPUT ACCEPT;
 ip6tables -t filter -P INPUT ACCEPT;
 ip6tables -t filter -P FORWARD ACCEPT;
 ip6tables -t filter -P OUTPUT ACCEPT;
+
 ip6tables -t filter -F;
 ip6tables -t filter -X;
+
 ip6tables -t filter -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT;
 ip6tables -t filter -A INPUT -p tcp -m multiport --dports 80,443,48588 -j ACCEPT;
 ip6tables -t filter -A INPUT -p icmpv6 -j ACCEPT;
+ip6tables -t filter -A INPUT -s fc00::/7 -j ACCEPT;
 ip6tables -t filter -A INPUT -i lo -j ACCEPT;
 ip6tables -t filter -P INPUT DROP;
 
+ip6tables -t filter -A FORWARD -s fc00::/7 -j ACCEPT;
+ip6tables -t filter -A FORWARD -d fc00::/7 -j ACCEPT;
 ip6tables -t filter -P FORWARD DROP;
 ip6tables -t filter -P OUTPUT ACCEPT;
+
+service docker restart;
-- 
Alviro Iskandar Setiawan

[PATCH server-haj002 v1 2/6] mastodon: Enable IPv6 for the masto-net network

[Alviro Iskandar Setiawan]

The IPv6 was not properly enabled. Enable it.

Signed-off-by: Alviro Iskandar Setiawan <alviro.iskandar@gnuweeb.org>
---
 mastodon/docker-compose.example.yml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/mastodon/docker-compose.example.yml b/mastodon/docker-compose.example.yml
index 5e2cd56..e695b0a 100644
--- a/mastodon/docker-compose.example.yml
+++ b/mastodon/docker-compose.example.yml
@@ -2,6 +2,7 @@ version: "2.1"
 
 networks:
   masto-net:
+    enable_ipv6: true
     driver: bridge
     ipam:
       driver: default
-- 
Alviro Iskandar Setiawan

[PATCH server-haj002 v1 3/6] mastodon: Fix DNS resolution error

[Alviro Iskandar Setiawan]

The default DNS system is not properly configured in this environment
and it leads to DNS resolution error. Provide custom resolv.conf file
and hosts file to satisfy the need of mastodon container.

The custom resolv.conf script will use the host IP address as the DNS
server. Therefore, the host must provide a working DNS server. It can
simply be done by adding these two lines to /etc/systemd/resolved.conf

   DNSStubListenerExtra=0.0.0.0:53
   DNSStubListenerExtra=[::]:53

Signed-off-by: Alviro Iskandar Setiawan <alviro.iskandar@gnuweeb.org>
---
 mastodon/docker-compose.example.yml | 2 ++
 mastodon/hosts                      | 8 ++++++++
 mastodon/resolv.conf                | 3 +++
 3 files changed, 13 insertions(+)
 create mode 100644 mastodon/hosts
 create mode 100644 mastodon/resolv.conf

diff --git a/mastodon/docker-compose.example.yml b/mastodon/docker-compose.example.yml
index e695b0a..769800f 100644
--- a/mastodon/docker-compose.example.yml
+++ b/mastodon/docker-compose.example.yml
@@ -94,3 +94,5 @@ services:
       - DB_POOL=5 #optional
     volumes:
       - /opt/mastodon/mastodon/config:/config
+      - ./resolv.conf:/etc/resolv.conf
+      - ./hosts:/etc/hosts
diff --git a/mastodon/hosts b/mastodon/hosts
new file mode 100644
index 0000000..23c9d56
--- /dev/null
+++ b/mastodon/hosts
@@ -0,0 +1,8 @@
+127.0.0.1	localhost
+::1		localhost ip6-localhost ip6-loopback
+fe00::0		ip6-localnet
+ff00::0		ip6-mcastprefix
+ff02::1		ip6-allnodes
+ff02::2		ip6-allrouters
+10.4.4.2	social social.gnuweeb.org
+51.81.211.47	mail1.gnuweeb.org
diff --git a/mastodon/resolv.conf b/mastodon/resolv.conf
new file mode 100644
index 0000000..2adb825
--- /dev/null
+++ b/mastodon/resolv.conf
@@ -0,0 +1,3 @@
+nameserver 10.4.4.1
+nameserver 8.8.8.8
+nameserver 8.8.4.4
-- 
Alviro Iskandar Setiawan

[PATCH server-haj002 v1 4/6] mastodon: Move data directory inside the git project

[Alviro Iskandar Setiawan]

Move data directory inside the git project to allow the docker-comose.yml
file to use relative path. Then sync the docker-compose file with the
prod file (add hostname entry and DNS related changes).

Signed-off-by: Alviro Iskandar Setiawan <alviro.iskandar@gnuweeb.org>
---
 .gitmodules                         |  3 +++
 mastodon/.gitignore                 |  3 ++-
 mastodon/docker-compose.example.yml | 16 ++++++++++------
 mastodon/docker-mastodon            |  1 +
 4 files changed, 16 insertions(+), 7 deletions(-)
 create mode 100644 .gitmodules
 create mode 160000 mastodon/docker-mastodon

diff --git a/.gitmodules b/.gitmodules
new file mode 100644
index 0000000..aeb163f
--- /dev/null
+++ b/.gitmodules
@@ -0,0 +1,3 @@
+[submodule "mastodon/docker-mastodon"]
+	path = mastodon/docker-mastodon
+	url = https://github.com/alviroiskandar/docker-mastodon.git
diff --git a/mastodon/.gitignore b/mastodon/.gitignore
index 2f0cdc6..a38d3b2 100644
--- a/mastodon/.gitignore
+++ b/mastodon/.gitignore
@@ -1 +1,2 @@
-/docker-compose.yml
\ No newline at end of file
+/docker-compose.yml
+/data/mastodon/
diff --git a/mastodon/docker-compose.example.yml b/mastodon/docker-compose.example.yml
index 769800f..b1fc36d 100644
--- a/mastodon/docker-compose.example.yml
+++ b/mastodon/docker-compose.example.yml
@@ -26,7 +26,7 @@ services:
     environment:
       - ALLOW_EMPTY_PASSWORD=yes
     volumes:
-      - /opt/mastodon/redis:/bitnami/redis/data
+      - ./data/mastodon/redis:/bitnami/redis/data
 
   postgres:
     networks:
@@ -42,7 +42,7 @@ services:
       - POSTGRES_DB=mastodon
       - PGDATA=/var/lib/postgresql/data
     volumes:
-      - /opt/mastodon/postgres:/var/lib/postgresql/data
+      - ./data/mastodon/postgres:/var/lib/postgresql/data
 
   mastodon:
     networks:
@@ -52,9 +52,11 @@ services:
     depends_on:
       - postgres
       - redis
-    image: lscr.io/linuxserver/mastodon:latest
+    image: mastodon-gw
+    build: ./docker-mastodon
     restart: unless-stopped
     container_name: mastodon
+    hostname: social.gnuweeb.org
     environment:
       - PUID=1000
       - PGID=1000
@@ -90,9 +92,11 @@ services:
       - SIDEKIQ_ONLY=false #optional
       - SIDEKIQ_QUEUE= #optional
       - SIDEKIQ_DEFAULT=false #optional
-      - SIDEKIQ_THREADS=5 #optional
-      - DB_POOL=5 #optional
+      - SIDEKIQ_THREADS=32 #optional
+      - DB_POOL=32 #optional
+      - HCAPTCHA_SECRET_KEY= #optional
+      - HCAPTCHA_SITE_KEY=
     volumes:
-      - /opt/mastodon/mastodon/config:/config
+      - ./data/mastodon/mastodon/config:/config
       - ./resolv.conf:/etc/resolv.conf
       - ./hosts:/etc/hosts
diff --git a/mastodon/docker-mastodon b/mastodon/docker-mastodon
new file mode 160000
index 0000000..962c135
--- /dev/null
+++ b/mastodon/docker-mastodon
@@ -0,0 +1 @@
+Subproject commit 962c135c5f31d9051057c92594449369522c674f
-- 
Alviro Iskandar Setiawan

[PATCH server-haj002 v1 5/6] Add elk server using docker

[Alviro Iskandar Setiawan]

elk is a nimble Mastodon web client.

Link: https://t.me/GNUWeeb/807595
Cc: Michael William Jonathan <moe@gnuweeb.org>
Signed-off-by: Alviro Iskandar Setiawan <alviro.iskandar@gnuweeb.org>
---
 elk/.gitignore         |  2 ++
 elk/docker-compose.yml | 30 ++++++++++++++++++++++++++++++
 elk/start.sh           | 23 +++++++++++++++++++++++
 3 files changed, 55 insertions(+)
 create mode 100644 elk/.gitignore
 create mode 100644 elk/docker-compose.yml
 create mode 100644 elk/start.sh

diff --git a/elk/.gitignore b/elk/.gitignore
new file mode 100644
index 0000000..d7fae55
--- /dev/null
+++ b/elk/.gitignore
@@ -0,0 +1,2 @@
+/elk_git/
+/elk-storage/
diff --git a/elk/docker-compose.yml b/elk/docker-compose.yml
new file mode 100644
index 0000000..0ec1c6f
--- /dev/null
+++ b/elk/docker-compose.yml
@@ -0,0 +1,30 @@
+version: "2.1"
+
+networks:
+  elk-net:
+    enable_ipv6: true
+    driver: bridge
+    ipam:
+      driver: default
+      config:
+      - subnet: 10.4.5.0/24
+        gateway: 10.4.5.1
+      - subnet: fcdd:af::/64
+        gateway: fcdd:af::1
+    driver_opts:
+      com.docker.network.bridge.name: elk-net
+
+services:
+  elk:
+    networks:
+      elk-net:
+        ipv4_address: 10.4.5.2
+        ipv6_address: fcdd:af::2
+    build:
+      context: ./elk_git
+      dockerfile: Dockerfile
+    volumes:
+      # make sure this directory has the same ownership as the elk user from the Dockerfile
+      # otherwise Elk will not be able to store configs for accounts
+      # e.q. mkdir ./elk-storage; sudo chown 911:911 ./elk-storage
+      - './elk-storage:/elk/data'
diff --git a/elk/start.sh b/elk/start.sh
new file mode 100644
index 0000000..6662d92
--- /dev/null
+++ b/elk/start.sh
@@ -0,0 +1,23 @@
+#!/bin/bash
+
+#
+# For an unknown reason, a git submodule of elk cannot be built
+# using docker-compose up --build. This script will clone
+# the elk repository and build the docker image if there is
+# no available elk instance on the machine.
+#
+
+parent_path=$( cd "$(dirname "${BASH_SOURCE[0]}")" ; pwd -P );
+
+cd "${parent_path}";
+
+if [ ! -f "${parent_path}/elk_git" ]; then
+    mkdir -pv ./elk-storage;
+    chown -R 911:911 ./elk-storage;
+fi;
+
+if [ ! -f "${parent_path}/elk_git/Dockerfile" ]; then
+    git clone https://github.com/elk-zone/elk.git "${parent_path}/elk_git";
+fi;
+
+docker-compose up --build -d;
-- 
Alviro Iskandar Setiawan

[PATCH server-haj002 v1 6/6] setup.txt: Add elk setup instructions

[Alviro Iskandar Setiawan]

Signed-off-by: Alviro Iskandar Setiawan <alviro.iskandar@gnuweeb.org>
---
 setup.txt | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/setup.txt b/setup.txt
index ea10b8d..36240ad 100644
--- a/setup.txt
+++ b/setup.txt
@@ -30,6 +30,7 @@ mkswap swap0.swo;
 ### Set crontab
 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
 @reboot /usr/bin/bash /opt/server/init_net.sh
+@reboot /usr/bin/bash /opt/server/elk/start.sh
 @reboot /usr/sbin/swapon /opt/server/swap0.swo
 
 
@@ -38,6 +39,8 @@ sudo make-ssl-cert generate-default-snakeoil --force-overwrite
 
 
 ### Create a nginx vhost config
+# For /etc/nginx/sites-enabled/social.gnuweeb.org:
+
 map $http_upgrade $connection_upgrade {
     default upgrade;
     '' close;
@@ -67,6 +70,22 @@ server {
     }
 }
 
+# For /etc/nginx/sites-enabled/elk.gnuweeb.org:
+server {
+    listen 80;
+    listen [::]:80;
+    listen 443 ssl http2;
+    listen [::]:443 ssl http2;
+
+    server_name elk.gnuweeb.org;
+    ssl_certificate /etc/ssl/certs/ssl-cert-snakeoil.pem;
+    ssl_certificate_key /etc/ssl/private/ssl-cert-snakeoil.key;
+
+    location / {
+        proxy_set_header "Host" "elk.gnuweeb.org";
+        proxy_pass http://127.0.0.1:5314;
+    }
+}
 
 
 ### Run mastodon
-- 
Alviro Iskandar Setiawan

Re: [PATCH server-haj002 v1 0/6] Server update and new service elk.gnuweeb.org

[Ammar Faizi]

On Thu, Jul 13, 2023 at 05:28:44PM +0200, Alviro Iskandar Setiawan wrote:
> The elk service for GNU/Weeb is now up and ready to use at:
> 
>   https://elk.gnuweeb.org

I can login using my GNUWeeb-Mastodon account on that web. But I see
many network errors in the browser console:

"""
Access to fetch at 'https://social.gnuweeb.org/nodeinfo/2.0' from origin
'https://elk.gnuweeb.org' has been blocked by CORS policy: No
'Access-Control-Allow-Origin' header is present on the requested
resource. If an opaque response serves your needs, set the request's
mode to 'no-cors' to fetch the resource with CORS disabled.
"""

It seems you need to add 'Access-Control-Allow-Origin' header in the
nginx host config to fix it.

-- 
Ammar Faizi

Re: [PATCH server-haj002 v1 0/6] Server update and new service elk.gnuweeb.org

[Alviro Iskandar Setiawan]

On Thu, Jul 13, 2023 at 10:42 PM Ammar Faizi wrote:
> It seems you need to add 'Access-Control-Allow-Origin' header in the
> nginx host config to fix it.

OK, I fixed that. But we still have one error from the elk:

GET https://social.gnuweeb.org/api/v1/push/subscription net::ERR_FAILED 404

I don't know what the problem is. I guess it's a compatibility problem
with the Mastodon version. But I'm not sure.

-- Viro

Re: [PATCH server-haj002 v1 0/6] Server update and new service elk.gnuweeb.org

[Ammar Faizi]

On Thu, Jul 13, 2023 at 11:47:51PM +0700, Alviro Iskandar Setiawan wrote: 
> OK, I fixed that. But we still have one error from the elk:
> 
> GET https://social.gnuweeb.org/api/v1/push/subscription net::ERR_FAILED 404
> 
> I don't know what the problem is. I guess it's a compatibility problem
> with the Mastodon version. But I'm not sure.

Neither am I. Let's wait for @moepoi's comments on this. He knows
about Mastodon better than me.

-- 
Ammar Faizi

Re: [PATCH server-haj002 v1 0/6] Server update and new service elk.gnuweeb.org

[Michael William Jonathan]

[-- Attachment #1: Type: text/plain, Size: 733 bytes --]

On Thu, Jul 13, 2023 at 11:55 PM Ammar Faizi <ammarfaizi2@gnuweeb.org>
wrote:

> On Thu, Jul 13, 2023 at 11:47:51PM +0700, Alviro Iskandar Setiawan wrote:
> > OK, I fixed that. But we still have one error from the elk:
> >
> > GET https://social.gnuweeb.org/api/v1/push/subscription net::ERR_FAILED
> 404
> >
> > I don't know what the problem is. I guess it's a compatibility problem
> > with the Mastodon version. But I'm not sure.
>
> Neither am I. Let's wait for @moepoi's comments on this. He knows
> about Mastodon better than me.
>
> --
> Ammar Faizi
>
> Hmm, it seems that the mastodon backend will return 404 when there is no
push subscription.

https://github.com/elk-zone/elk/issues/1737

--
Moe

[-- Attachment #2: Type: text/html, Size: 1296 bytes --]

Re: [PATCH server-haj002 v1 0/6] Server update and new service elk.gnuweeb.org

[Alviro Iskandar Setiawan]

On Fri, Jul 14, 2023 at 9:25 AM Michael William Jonathan wrote:
> Hmm, it seems that the mastodon backend will return 404 when there is no push subscription.
>
> https://github.com/elk-zone/elk/issues/1737

Alright, it's normal then. So this 404 error isn't a problem, that
means we only had issues with the CORS (but already fixed).

I'll send a v2 revision with the CORS issues fixed.

tq

-- Viro