Disk I/O di Neo Lite kembali bermasalah

Disk I/O di Neo Lite kembali bermasalah

[Ammar Faizi]

Melanjutkan masalah IOPS di Neo Lite.

IOPS di Neo Lite kembali bermasalah. Sudah sejak bulan Juni kemarin
saya mengalami masalah disk I/O di Neo Lite.

Gejala: process stuck di "D" state cukup lama ketika melakukan disk
operation.

Masalah ini kadang muncul, kadang hilang. Dengan kata lain, disk I/O
yang stuck ini *tidak selalu* terjadi, tapi ketika terjadi dampaknya
bisa membuat VM menjadi freeze seperti yang dikeluhkan beberapa user
Biznet Gio lain juga (di Facebook).

Kemudian sejak tanggal 8 Juli, saya melakukan stress test menggunakan
fio 24 jam non stop untuk pembuktian masalah ini. Command yang saya
jalankan:

  while true; do
     fio --name=io_uring --ioengine=io_uring --iodepth=1024 --filename=./test --rw=rw --bs=16K --numjobs=2 --size=10G --direct=1;
  done;

IP VPS yang digunakan untuk testing: 103.175.218.159

Selama 4 hari berjalan normal, namun hari ini sekitar jam 17:26
terkendala process stuck di "D" state lebih dari 30 detik.

  [Jul12 17:26] INFO: task jbd2/sda1-8:377 blocked for more than 30 seconds.
  [  +0.001509]       Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8
  [  +0.001374] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  [  +0.001361] task:jbd2/sda1-8     state:D stack:0     pid:377   tgid:377   ppid:2      flags:0x00004000
  [  +0.000021] Call Trace:
  [  +0.000021]  <TASK>
  [  +0.000028]  __schedule+0x5e7/0x14a0
  [  +0.000083]  ? submit_bio_noacct_nocheck+0x2cc/0x360
  [  +0.000022]  schedule+0x8e/0x160
  [  +0.000004]  bit_wait_io+0x45/0xa0
  [  +0.000006]  __wait_on_bit+0xe1/0x170
  [  +0.000003]  ? bit_wait_io_timeout+0xc0/0xc0
  [  +0.000004]  jbd2_journal_commit_transaction+0x191b/0x2320
  [  +0.000021]  ? wake_up_bit+0xd0/0xd0
  [  +0.000018]  kjournald2+0x120/0x460
  [  +0.000013]  ? init_wait_entry+0x30/0x30
  [  +0.000004]  kthread+0x1f9/0x230
  [  +0.000014]  ? jbd2_journal_update_sb_log_tail+0x180/0x180
  [  +0.000005]  ? kthreadd+0x380/0x380
  [  +0.000002]  ret_from_fork+0x30/0x40
  [  +0.000031]  ? kthreadd+0x380/0x380
  [  +0.000002]  ret_from_fork_asm+0x11/0x20
  [  +0.000015]  </TASK>

jbd2/sda1 sendiri adalah sebuah kernel thread dari module ext4.

Mohon solusi dari pihak Biznet Gio terkait masalah ini. Masalah ini
kadang muncul, kadang hilang.

Terima kasih.

-- 
Ammar Faizi

Re:[## 385523 ##] Disk I/O di Neo Lite kembali bermasalah

[Ammar Rafli]

[-- Attachment #1: Type: text/plain, Size: 3741 bytes --]

Hi Bapak Ammar
 
 Selamat Malam
 
 Menanggapi email yang disampaikan, Saat Bapak melakukan pengetesan FIO secara konstan selama beberapa hari, apakah Bapak dapat men-capture load average server secara berkala Pak? dan apakah Bapak mempunyai capture load average server sebelum kendala terjadi?
 
 Mohon informasinya pada ticket ini agar dapat kami koordinasikan dahulu.

Terima kasih atas perhatian dan kerjasamanya.

Penilaian Anda sangat membantu perbaikan dan evaluasi pelayanan kami. Ketika tiket pengaduan Anda kami tutup akan ada otomatis link survei kepuasan pelanggan di mailbox Anda. Mohon untuk memberikan penilaian Anda.

Regards
 
 Ammar Rafli Arkan
Product Operation Support
-----------------------------------------------------------
PT. Biznet Gio Nusantara
MidPlaza 1, 7th Floor. Jl. Jend. Sudirman Kav 10 - 11. Jakarta 10220 - Indonesia. 
-----------------------------------------------------------
Click here to submit your feature request.

The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. 
 
 
---- on Fri, 12 Jul 2024 18:12:17 +0700 "Ammar Faizi"<ammarfaizi2@gnuweeb.org> wrote ---- 
 Melanjutkan masalah IOPS di Neo Lite. 
 
IOPS di Neo Lite kembali bermasalah. Sudah sejak bulan Juni kemarin 
saya mengalami masalah disk I/O di Neo Lite. 
 
Gejala: process stuck di "D" state cukup lama ketika melakukan disk 
operation. 
 
Masalah ini kadang muncul, kadang hilang. Dengan kata lain, disk I/O 
yang stuck ini *tidak selalu* terjadi, tapi ketika terjadi dampaknya 
bisa membuat VM menjadi freeze seperti yang dikeluhkan beberapa user 
Biznet Gio lain juga (di Facebook). 
 
Kemudian sejak tanggal 8 Juli, saya melakukan stress test menggunakan 
fio 24 jam non stop untuk pembuktian masalah ini. Command yang saya 
jalankan: 
 
 while true; do 
 fio --name=io_uring --ioengine=io_uring --iodepth=1024 --filename=./test --rw=rw --bs=16K --numjobs=2 --size=10G --direct=1; 
 done; 
 
IP VPS yang digunakan untuk testing: 103.175.218.159 
 
Selama 4 hari berjalan normal, namun hari ini sekitar jam 17:26 
terkendala process stuck di "D" state lebih dari 30 detik. 
 
 [Jul12 17:26] INFO: task jbd2/sda1-8:377 blocked for more than 30 seconds. 
 [ +0.001509] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8 
 [ +0.001374] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 
 [ +0.001361] task:jbd2/sda1-8 state:D stack:0 pid:377 tgid:377 ppid:2 flags:0x00004000 
 [ +0.000021] Call Trace: 
 [ +0.000021] <TASK> 
 [ +0.000028] __schedule+0x5e7/0x14a0 
 [ +0.000083] ? submit_bio_noacct_nocheck+0x2cc/0x360 
 [ +0.000022] schedule+0x8e/0x160 
 [ +0.000004] bit_wait_io+0x45/0xa0 
 [ +0.000006] __wait_on_bit+0xe1/0x170 
 [ +0.000003] ? bit_wait_io_timeout+0xc0/0xc0 
 [ +0.000004] jbd2_journal_commit_transaction+0x191b/0x2320 
 [ +0.000021] ? wake_up_bit+0xd0/0xd0 
 [ +0.000018] kjournald2+0x120/0x460 
 [ +0.000013] ? init_wait_entry+0x30/0x30 
 [ +0.000004] kthread+0x1f9/0x230 
 [ +0.000014] ? jbd2_journal_update_sb_log_tail+0x180/0x180 
 [ +0.000005] ? kthreadd+0x380/0x380 
 [ +0.000002] ret_from_fork+0x30/0x40 
 [ +0.000031] ? kthreadd+0x380/0x380 
 [ +0.000002] ret_from_fork_asm+0x11/0x20 
 [ +0.000015] </TASK> 
 
jbd2/sda1 sendiri adalah sebuah kernel thread dari module ext4. 
 
Mohon solusi dari pihak Biznet Gio terkait masalah ini. Masalah ini 
kadang muncul, kadang hilang. 
 
Terima kasih. 
 
-- 
Ammar Faizi 
 
 

[-- Attachment #2: Type: text/html, Size: 5938 bytes --]

Re: [## 385523 ##] Disk I/O di Neo Lite kembali bermasalah

[Ammar Faizi]

[-- Attachment #1: Type: text/plain, Size: 748 bytes --]

On Fri, Jul 12, 2024 at 6:28 PM Ammar Rafli wrote:
> Hi Bapak Ammar
>
> Selamat Malam
>
> Menanggapi email yang disampaikan, Saat Bapak melakukan pengetesan FIO secara konstan selama beberapa hari, apakah Bapak dapat men-capture load average server secara berkala Pak? dan apakah Bapak mempunyai capture load average server sebelum kendala terjadi?

Saya tidak mempunyai capture load average sebelum kendala terjadi,
tapi load average saya pastikan di atas 16 (di bagian 1 min, 5 min, 15
min hampir semua rata full load), karena memang saya lakukan stress
test 24 jam non stop. Berikut saya lampirkan screenshot load average.

Side note: Saya sempat reboot sekali, jadi uptime di screenshot tidak
sampai 4 hari.

-- 
Ammar Faizi

[-- Attachment #2: Screenshot from 2024-07-12 18-29-26.png --]
[-- Type: image/png, Size: 34737 bytes --]

Re:[## 385523 ##] Disk I/O di Neo Lite kembali bermasalah

[Ammar Rafli]

[-- Attachment #1: Type: text/plain, Size: 2169 bytes --]

Hi Bapak Ammar Faizi
 
 Selamat Malam
 
Menanggapi email yang disampaikan, Apakah saat ini Bapak dapat men-capture full command top -c Pak? agar kami dapat melihat semua parameter proses yang ada di server nya ya Pak untuk review lebih lanjut. 
 Sekian informasi yang disampaiakn.

Terima kasih atas perhatian dan kerjasamanya.

Penilaian Anda sangat membantu perbaikan dan evaluasi pelayanan kami. Ketika tiket pengaduan Anda kami tutup akan ada otomatis link survei kepuasan pelanggan di mailbox Anda. Mohon untuk memberikan penilaian Anda.

Regards
 
 Ammar Rafli Arkan
Product Operation Support
-----------------------------------------------------------
PT. Biznet Gio Nusantara
MidPlaza 1, 7th Floor. Jl. Jend. Sudirman Kav 10 - 11. Jakarta 10220 - Indonesia. 
-----------------------------------------------------------
Click here to submit your feature request.

The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. 
 
 
---- on Fri, 12 Jul 2024 18:34:43 +0700 "Ammar Faizi"<ammarfaizi2@gnuweeb.org> wrote ---- 
 On Fri, Jul 12, 2024 at 6:28 PM Ammar Rafli wrote: 
> Hi Bapak Ammar 
> 
> Selamat Malam 
> 
> Menanggapi email yang disampaikan, Saat Bapak melakukan pengetesan FIO secara konstan selama beberapa hari, apakah Bapak dapat men-capture load average server secara berkala Pak? dan apakah Bapak mempunyai capture load average server sebelum kendala terjadi? 
 
Saya tidak mempunyai capture load average sebelum kendala terjadi, 
tapi load average saya pastikan di atas 16 (di bagian 1 min, 5 min, 15 
min hampir semua rata full load), karena memang saya lakukan stress 
test 24 jam non stop. Berikut saya lampirkan screenshot load average. 
 
Side note: Saya sempat reboot sekali, jadi uptime di screenshot tidak 
sampai 4 hari. 
 
-- 
Ammar Faizi 
 

[-- Attachment #2: Type: text/html, Size: 4016 bytes --]

Re: [## 385523 ##] Disk I/O di Neo Lite kembali bermasalah

[Ammar Faizi]

[-- Attachment #1: Type: text/plain, Size: 275 bytes --]

On Fri, Jul 12, 2024 at 6:40 PM Ammar Rafli wrote:
> Hi Bapak Ammar Faizi
>
> Selamat Malam
>
> Menanggapi email yang disampaikan, Apakah saat ini Bapak dapat men-capture full command top -c Pak?

Berikut saya lampirkan output command `top -c`.

-- 
Ammar Faizi

[-- Attachment #2: image.png --]
[-- Type: image/png, Size: 313221 bytes --]

Re:[## 385523 ##] Disk I/O di Neo Lite kembali bermasalah

[Ammar Rafli]

[-- Attachment #1: Type: text/plain, Size: 1744 bytes --]

Hi Bapak Ammar 
 
 Selamat Malam
 
Menanggapi email yang disampaikan, Jika kami check load average server nya tinggi dan terdapat script yang menggunakan 15 core cpu, apakah script tersebut adlaah script fio Pak? 
 Dan bisa dicapture hasil pengetesan terbaru yang dilakukan pada script tersebut?
 
 Mohon informasi yang disampaikan.

Terima kasih atas perhatian dan kerjasamanya.

Penilaian Anda sangat membantu perbaikan dan evaluasi pelayanan kami. Ketika tiket pengaduan Anda kami tutup akan ada otomatis link survei kepuasan pelanggan di mailbox Anda. Mohon untuk memberikan penilaian Anda.

Regards
 
 Ammar Rafli Arkan
Product Operation Support
-----------------------------------------------------------
PT. Biznet Gio Nusantara
MidPlaza 1, 7th Floor. Jl. Jend. Sudirman Kav 10 - 11. Jakarta 10220 - Indonesia. 
-----------------------------------------------------------
Click here to submit your feature request.

The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. 
 
 
---- on Fri, 12 Jul 2024 18:47:05 +0700 "Ammar Faizi"<ammarfaizi2@gnuweeb.org> wrote ---- 
 On Fri, Jul 12, 2024 at 6:40 PM Ammar Rafli wrote: 
> Hi Bapak Ammar Faizi 
> 
> Selamat Malam 
> 
> Menanggapi email yang disampaikan, Apakah saat ini Bapak dapat men-capture full command top -c Pak? 
 
Berikut saya lampirkan output command `top -c`. 
 
-- 
Ammar Faizi 
 

[-- Attachment #2: Type: text/html, Size: 3597 bytes --]

Re: [## 385523 ##] Disk I/O di Neo Lite kembali bermasalah

[Ammar Faizi]

On Fri, Jul 12, 2024 at 6:57 PM Ammar Rafli wrote:
> Menanggapi email yang disampaikan, Jika kami check load average server nya tinggi dan terdapat script yang menggunakan 15 core cpu, apakah script tersebut adlaah script fio Pak?

Itu bukan fio, tapi itu adalah program stress test CPU.

https://gist.githubusercontent.com/ammarfaizi2/f55dbb7c918a660776c68c50319c7882/raw/1dd37e1fb96054b31f345381273e8a1702215002/ct.c

Saya jalankan stress test CPU, Disk, dan RAM.

-- 
Ammar Faizi

Re:[## 385523 ##] Disk I/O di Neo Lite kembali bermasalah

[Ammar Rafli]

[-- Attachment #1: Type: text/plain, Size: 1867 bytes --]

Hi Bapak Ammar 
 
 Selamat Malam
 
Menanggapi email yang disampaikan, Baik, terkait hal tersebut akan kami koordinasikan dahulu. 
 Mohon kesediannya untuk menunggu informasi update dari sisi kami.
 
 Sekian informasi yang disampaikan.

Terima kasih atas perhatian dan kerjasamanya.

Penilaian Anda sangat membantu perbaikan dan evaluasi pelayanan kami. Ketika tiket pengaduan Anda kami tutup akan ada otomatis link survei kepuasan pelanggan di mailbox Anda. Mohon untuk memberikan penilaian Anda.

Regards
 
 Ammar Rafli Arkan
Product Operation Support
-----------------------------------------------------------
PT. Biznet Gio Nusantara
MidPlaza 1, 7th Floor. Jl. Jend. Sudirman Kav 10 - 11. Jakarta 10220 - Indonesia. 
-----------------------------------------------------------
Click here to submit your feature request.

The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. 
 
 
---- on Fri, 12 Jul 2024 19:09:19 +0700 "Ammar Faizi"<ammarfaizi2@gnuweeb.org> wrote ---- 
 On Fri, Jul 12, 2024 at 6:57 PM Ammar Rafli wrote: 
> Menanggapi email yang disampaikan, Jika kami check load average server nya tinggi dan terdapat script yang menggunakan 15 core cpu, apakah script tersebut adlaah script fio Pak? 
 
Itu bukan fio, tapi itu adalah program stress test CPU. 
 
https://gist.githubusercontent.com/ammarfaizi2/f55dbb7c918a660776c68c50319c7882/raw/1dd37e1fb96054b31f345381273e8a1702215002/ct.c 
 
Saya jalankan stress test CPU, Disk, dan RAM. 
 
-- 
Ammar Faizi 
 

[-- Attachment #2: Type: text/html, Size: 3708 bytes --]

Re:[## 385523 ##] Disk I/O di Neo Lite kembali bermasalah

[Ammar Rafli]

[-- Attachment #1: Type: text/plain, Size: 4782 bytes --]

Hi Bapak Ammar
 
 Selamat Malam
 
Menanggapi email yang disampaikan, Terkait kendala IOPS pada layanan Neo Lite, apakah Bapak berkenan melakukan beberapa percobaan berikut ini Pak? 
 =============================================================================================================
 # fio --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --rw=read --filename=/root/fio --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=75 --group_reporting
 
 # fio --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --rw=write --filename=/root/fio --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=75 --group_reporting
 
 # fio --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --rw=rw --filename=/root/fio --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=50 --group_reporting
 
 # fio --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --rw=randread --filename=/root/fio --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=100 --group_reporting
 
 # fio --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --rw=randwrite --filename=/root/fio --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=75 --group_reporting
 
 # fio --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --rw=randrw --filename=/root/fio --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=0 --group_reporting 
 =========================================================================================================
 
 Silakan untuk melampirkan capture hasil fio nya pada ticket ini agar kami dapat melihat me-review detail IOPS nya ya Pak.
 
 Sekian informasi yang disampaikan.
 
Terima kasih atas perhatian dan kerjasamanya.

Penilaian Anda sangat membantu perbaikan dan evaluasi pelayanan kami. Ketika tiket pengaduan Anda kami tutup akan ada otomatis link survei kepuasan pelanggan di mailbox Anda. Mohon untuk memberikan penilaian Anda.

Regards
 
 Ammar Rafli Arkan
Product Operation Support
-----------------------------------------------------------
PT. Biznet Gio Nusantara
MidPlaza 1, 7th Floor. Jl. Jend. Sudirman Kav 10 - 11. Jakarta 10220 - Indonesia. 
-----------------------------------------------------------
Click here to submit your feature request.

The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. 
 
 
---- on Fri, 12 Jul 2024 19:20:09 +0700 Ammar Rafli<support@biznetgio.com> wrote ---- 
 Hi Bapak Ammar 
 
 Selamat Malam
 
Menanggapi email yang disampaikan, Baik, terkait hal tersebut akan kami koordinasikan dahulu. 
 Mohon kesediannya untuk menunggu informasi update dari sisi kami.
 
 Sekian informasi yang disampaikan.

Terima kasih atas perhatian dan kerjasamanya.

Penilaian Anda sangat membantu perbaikan dan evaluasi pelayanan kami. Ketika tiket pengaduan Anda kami tutup akan ada otomatis link survei kepuasan pelanggan di mailbox Anda. Mohon untuk memberikan penilaian Anda.

Regards
 
 Ammar Rafli Arkan
Product Operation Support
-----------------------------------------------------------
PT. Biznet Gio Nusantara
MidPlaza 1, 7th Floor. Jl. Jend. Sudirman Kav 10 - 11. Jakarta 10220 - Indonesia. 
-----------------------------------------------------------
Click here to submit your feature request.

The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. 
 
 
---- on Fri, 12 Jul 2024 19:09:19 +0700 "Ammar Faizi"<ammarfaizi2@gnuweeb.org> wrote ---- 
 On Fri, Jul 12, 2024 at 6:57 PM Ammar Rafli wrote: 
> Menanggapi email yang disampaikan, Jika kami check load average server nya tinggi dan terdapat script yang menggunakan 15 core cpu, apakah script tersebut adlaah script fio Pak? 
 
Itu bukan fio, tapi itu adalah program stress test CPU. 
 
https://gist.githubusercontent.com/ammarfaizi2/f55dbb7c918a660776c68c50319c7882/raw/1dd37e1fb96054b31f345381273e8a1702215002/ct.c 
 
Saya jalankan stress test CPU, Disk, dan RAM. 
 
-- 
Ammar Faizi 
 
 

[-- Attachment #2: Type: text/html, Size: 8098 bytes --]

Re: [## 385523 ##] Disk I/O di Neo Lite kembali bermasalah

[Ammar Faizi]

On Fri, Jul 12, 2024 at 05:23:09AM -0700, Ammar Rafli wrote:
>  Silakan untuk melampirkan capture hasil fio nya pada ticket ini agar kami dapat melihat me-review detail IOPS nya ya Pak.

Saya lampirkan output fio berikut:

root@wjv001:~/fio# ./fio --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --rw=read --filename=/root/fio --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=75 --group_reporting
fio: engine libaio not loadable
fio: failed to load engine

# Note: libaio tidak available, saya ubah ke io_uring.

root@wjv001:~/fio# ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=read --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=75 --group_reporting
test: (g=0): rw=read, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256
...
fio-3.37-60-g6eaa64
Starting 4 processes
Jobs: 2 (f=2): [_(2),R(2)][34.3%][r=5853KiB/s][r=731 IOPS][eta 01m:59s]
test: (groupid=0, jobs=4): err= 0: pid=18651: Fri Jul 12 19:25:53 2024
  read: IOPS=1001, BW=8013KiB/s (8205kB/s)(477MiB/60951msec)
   bw (  KiB/s): min=   96, max=20864, per=100.00%, avg=8536.02, stdev=1493.19, samples=385
   iops        : min=   12, max= 2608, avg=1066.99, stdev=186.65, samples=385
  cpu          : usr=0.11%, sys=0.25%, ctx=70031, majf=0, minf=2066
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=61048,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
   READ: bw=8013KiB/s (8205kB/s), 8013KiB/s-8013KiB/s (8205kB/s-8205kB/s), io=477MiB (500MB), run=60951-60951msec

Disk stats (read/write):
  sda: ios=60990/53, sectors=976152/528, merge=0/10, ticks=55711072/42055, in_queue=55753611, util=99.96%

=============================================================================================================

root@wjv001:~/fio# ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=rw --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=50 --group_reporting
test: (g=0): rw=rw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256
...
fio-3.37-60-g6eaa64
Starting 4 processes
Jobs: 4 (f=4): [M(4)][100.0%][r=8032KiB/s,w=8104KiB/s][r=1004,w=1013 IOPS][eta 00m:00s]
test: (groupid=0, jobs=4): err= 0: pid=18664: Fri Jul 12 19:31:16 2024
  read: IOPS=994, BW=7959KiB/s (8150kB/s)(471MiB/60624msec)
   bw (  KiB/s): min=  448, max=21488, per=99.84%, avg=7947.47, stdev=1237.33, samples=481
   iops        : min=   56, max= 2686, avg=993.41, stdev=154.67, samples=481
  write: IOPS=1001, BW=8010KiB/s (8202kB/s)(474MiB/60624msec); 0 zone resets
   bw (  KiB/s): min=  416, max=21520, per=99.71%, avg=7987.95, stdev=1227.20, samples=481
   iops        : min=   52, max= 2690, avg=998.47, stdev=153.40, samples=481
  cpu          : usr=0.20%, sys=0.46%, ctx=96868, majf=0, minf=26
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=60316,60700,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
   READ: bw=7959KiB/s (8150kB/s), 7959KiB/s-7959KiB/s (8150kB/s-8150kB/s), io=471MiB (494MB), run=60624-60624msec
  WRITE: bw=8010KiB/s (8202kB/s), 8010KiB/s-8010KiB/s (8202kB/s-8202kB/s), io=474MiB (497MB), run=60624-60624msec

Disk stats (read/write):
  sda: ios=60146/60471, sectors=962336/967448, merge=0/12, ticks=27376373/30599469, in_queue=57977062, util=99.97%


=============================================================================================================

root@wjv001:~/fio# ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=randread --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=100 --group_reporting
test: (g=0): rw=randread, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256
...
fio-3.37-60-g6eaa64
Starting 4 processes
Jobs: 2 (f=2): [r(1),_(1),r(1),_(1)][34.3%][r=5904KiB/s][r=738 IOPS][eta 01m:59s]
test: (groupid=0, jobs=4): err= 0: pid=18673: Fri Jul 12 19:35:08 2024
  read: IOPS=1001, BW=8013KiB/s (8206kB/s)(478MiB/61139msec)
   bw (  KiB/s): min= 1824, max=18288, per=100.00%, avg=8101.39, stdev=1076.94, samples=445
   iops        : min=  228, max= 2286, avg=1012.67, stdev=134.62, samples=445
  cpu          : usr=0.12%, sys=0.26%, ctx=71964, majf=0, minf=2076
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=61240,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
   READ: bw=8013KiB/s (8206kB/s), 8013KiB/s-8013KiB/s (8206kB/s-8206kB/s), io=478MiB (502MB), run=61139-61139msec

Disk stats (read/write):
  sda: ios=61240/7, sectors=980080/112, merge=0/7, ticks=61595911/6429, in_queue=61602360, util=99.95%

=============================================================================================================

root@wjv001:~/fio# ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=randwrite --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=75 --group_reporting
test: (g=0): rw=randwrite, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256
...
fio-3.37-60-g6eaa64
Starting 4 processes
Jobs: 4 (f=4): [w(4)][3.5%][w=1936KiB/s][w=242 IOPS][eta 28m:31s]  
test: (groupid=0, jobs=4): err= 0: pid=18683: Fri Jul 12 19:37:33 2024
  write: IOPS=1001, BW=8010KiB/s (8202kB/s)(478MiB/61102msec); 0 zone resets
   bw (  KiB/s): min=   96, max=27984, per=100.00%, avg=8245.91, stdev=1833.66, samples=465
   iops        : min=   12, max= 3498, avg=1030.70, stdev=229.20, samples=465
  cpu          : usr=0.13%, sys=0.27%, ctx=44021, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=0,61176,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
  WRITE: bw=8010KiB/s (8202kB/s), 8010KiB/s-8010KiB/s (8202kB/s-8202kB/s), io=478MiB (501MB), run=61102-61102msec

Disk stats (read/write):
  sda: ios=0/60981, sectors=0/975608, merge=0/11, ticks=0/52000246, in_queue=52001504, util=99.95%

=============================================================================================================

root@wjv001:~/fio# ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=randrw --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=0 --group_reporting
test: (g=0): rw=randrw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256
...
fio-3.37-60-g6eaa64
Starting 4 processes
Jobs: 4 (f=4): [w(4)][100.0%][w=6904KiB/s][w=863 IOPS][eta 00m:00s]
test: (groupid=0, jobs=4): err= 0: pid=18693: Fri Jul 12 19:39:00 2024
  write: IOPS=1001, BW=8009KiB/s (8201kB/s)(476MiB/60853msec); 0 zone resets
   bw (  KiB/s): min=  256, max=24624, per=100.00%, avg=8209.34, stdev=1657.90, samples=470
   iops        : min=   32, max= 3078, avg=1026.15, stdev=207.24, samples=470
  cpu          : usr=0.15%, sys=0.31%, ctx=49963, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=0,60920,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
  WRITE: bw=8009KiB/s (8201kB/s), 8009KiB/s-8009KiB/s (8201kB/s-8201kB/s), io=476MiB (499MB), run=60853-60853msec

Disk stats (read/write):
  sda: ios=0/60744, sectors=0/971808, merge=0/12, ticks=0/56919694, in_queue=56920300, util=99.97%

=============================================================================================================

-- 
Ammar Faizi

Re:[## 385523 ##] Disk I/O di Neo Lite kembali bermasalah

[Ammar Rafli]

[-- Attachment #1: Type: text/plain, Size: 10098 bytes --]

Hi Bapak Ammar
 
 Selamat Malam
 
Menanggapi email yang disampaikan, Mohon informasinya apakah Bapak melakukan fio nya saat script stresstest berjalan? 
 Jika iya, apakah Bapak bisa men-kill script stresstest nya dahulu lalu coba jalankan command fio nya kembali Pak?
 
 Mohon informasinya pada ticket ini.

Terima kasih atas perhatian dan kerjasamanya.

Penilaian Anda sangat membantu perbaikan dan evaluasi pelayanan kami. Ketika tiket pengaduan Anda kami tutup akan ada otomatis link survei kepuasan pelanggan di mailbox Anda. Mohon untuk memberikan penilaian Anda.

Regards
 
 Ammar Rafli Arkan
Product Operation Support
-----------------------------------------------------------
PT. Biznet Gio Nusantara
MidPlaza 1, 7th Floor. Jl. Jend. Sudirman Kav 10 - 11. Jakarta 10220 - Indonesia. 
-----------------------------------------------------------
Click here to submit your feature request.

The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. 
 
 
---- on Fri, 12 Jul 2024 19:42:03 +0700 "Ammar Faizi"<ammarfaizi2@gnuweeb.org> wrote ---- 
 On Fri, Jul 12, 2024 at 05:23:09AM -0700, Ammar Rafli wrote: 
> Silakan untuk melampirkan capture hasil fio nya pada ticket ini agar kami dapat melihat me-review detail IOPS nya ya Pak. 
 
Saya lampirkan output fio berikut: 
 
root@wjv001:~/fio# ./fio --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --rw=read --filename=/root/fio --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=75 --group_reporting 
fio: engine libaio not loadable 
fio: failed to load engine 
 
# Note: libaio tidak available, saya ubah ke io_uring. 
 
root@wjv001:~/fio# ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=read --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=75 --group_reporting 
test: (g=0): rw=read, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256 
... 
fio-3.37-60-g6eaa64 
Starting 4 processes 
Jobs: 2 (f=2): [_(2),R(2)][34.3%][r=5853KiB/s][r=731 IOPS][eta 01m:59s] 
test: (groupid=0, jobs=4): err= 0: pid=18651: Fri Jul 12 19:25:53 2024 
 read: IOPS=1001, BW=8013KiB/s (8205kB/s)(477MiB/60951msec) 
 bw ( KiB/s): min= 96, max=20864, per=100.00%, avg=8536.02, stdev=1493.19, samples=385 
 iops : min= 12, max= 2608, avg=1066.99, stdev=186.65, samples=385 
 cpu : usr=0.11%, sys=0.25%, ctx=70031, majf=0, minf=2066 
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6% 
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% 
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% 
 issued rwts: total=61048,0,0,0 short=0,0,0,0 dropped=0,0,0,0 
 latency : target=0, window=0, percentile=100.00%, depth=256 
 
Run status group 0 (all jobs): 
 READ: bw=8013KiB/s (8205kB/s), 8013KiB/s-8013KiB/s (8205kB/s-8205kB/s), io=477MiB (500MB), run=60951-60951msec 
 
Disk stats (read/write): 
 sda: ios=60990/53, sectors=976152/528, merge=0/10, ticks=55711072/42055, in_queue=55753611, util=99.96% 
 
============================================================================================================= 
 
root@wjv001:~/fio# ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=rw --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=50 --group_reporting 
test: (g=0): rw=rw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256 
... 
fio-3.37-60-g6eaa64 
Starting 4 processes 
Jobs: 4 (f=4): [M(4)][100.0%][r=8032KiB/s,w=8104KiB/s][r=1004,w=1013 IOPS][eta 00m:00s] 
test: (groupid=0, jobs=4): err= 0: pid=18664: Fri Jul 12 19:31:16 2024 
 read: IOPS=994, BW=7959KiB/s (8150kB/s)(471MiB/60624msec) 
 bw ( KiB/s): min= 448, max=21488, per=99.84%, avg=7947.47, stdev=1237.33, samples=481 
 iops : min= 56, max= 2686, avg=993.41, stdev=154.67, samples=481 
 write: IOPS=1001, BW=8010KiB/s (8202kB/s)(474MiB/60624msec); 0 zone resets 
 bw ( KiB/s): min= 416, max=21520, per=99.71%, avg=7987.95, stdev=1227.20, samples=481 
 iops : min= 52, max= 2690, avg=998.47, stdev=153.40, samples=481 
 cpu : usr=0.20%, sys=0.46%, ctx=96868, majf=0, minf=26 
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8% 
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% 
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% 
 issued rwts: total=60316,60700,0,0 short=0,0,0,0 dropped=0,0,0,0 
 latency : target=0, window=0, percentile=100.00%, depth=256 
 
Run status group 0 (all jobs): 
 READ: bw=7959KiB/s (8150kB/s), 7959KiB/s-7959KiB/s (8150kB/s-8150kB/s), io=471MiB (494MB), run=60624-60624msec 
 WRITE: bw=8010KiB/s (8202kB/s), 8010KiB/s-8010KiB/s (8202kB/s-8202kB/s), io=474MiB (497MB), run=60624-60624msec 
 
Disk stats (read/write): 
 sda: ios=60146/60471, sectors=962336/967448, merge=0/12, ticks=27376373/30599469, in_queue=57977062, util=99.97% 
 
 
============================================================================================================= 
 
root@wjv001:~/fio# ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=randread --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=100 --group_reporting 
test: (g=0): rw=randread, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256 
... 
fio-3.37-60-g6eaa64 
Starting 4 processes 
Jobs: 2 (f=2): [r(1),_(1),r(1),_(1)][34.3%][r=5904KiB/s][r=738 IOPS][eta 01m:59s] 
test: (groupid=0, jobs=4): err= 0: pid=18673: Fri Jul 12 19:35:08 2024 
 read: IOPS=1001, BW=8013KiB/s (8206kB/s)(478MiB/61139msec) 
 bw ( KiB/s): min= 1824, max=18288, per=100.00%, avg=8101.39, stdev=1076.94, samples=445 
 iops : min= 228, max= 2286, avg=1012.67, stdev=134.62, samples=445 
 cpu : usr=0.12%, sys=0.26%, ctx=71964, majf=0, minf=2076 
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6% 
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% 
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% 
 issued rwts: total=61240,0,0,0 short=0,0,0,0 dropped=0,0,0,0 
 latency : target=0, window=0, percentile=100.00%, depth=256 
 
Run status group 0 (all jobs): 
 READ: bw=8013KiB/s (8206kB/s), 8013KiB/s-8013KiB/s (8206kB/s-8206kB/s), io=478MiB (502MB), run=61139-61139msec 
 
Disk stats (read/write): 
 sda: ios=61240/7, sectors=980080/112, merge=0/7, ticks=61595911/6429, in_queue=61602360, util=99.95% 
 
============================================================================================================= 
 
root@wjv001:~/fio# ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=randwrite --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=75 --group_reporting 
test: (g=0): rw=randwrite, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256 
... 
fio-3.37-60-g6eaa64 
Starting 4 processes 
Jobs: 4 (f=4): [w(4)][3.5%][w=1936KiB/s][w=242 IOPS][eta 28m:31s] 
test: (groupid=0, jobs=4): err= 0: pid=18683: Fri Jul 12 19:37:33 2024 
 write: IOPS=1001, BW=8010KiB/s (8202kB/s)(478MiB/61102msec); 0 zone resets 
 bw ( KiB/s): min= 96, max=27984, per=100.00%, avg=8245.91, stdev=1833.66, samples=465 
 iops : min= 12, max= 3498, avg=1030.70, stdev=229.20, samples=465 
 cpu : usr=0.13%, sys=0.27%, ctx=44021, majf=0, minf=28 
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6% 
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% 
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% 
 issued rwts: total=0,61176,0,0 short=0,0,0,0 dropped=0,0,0,0 
 latency : target=0, window=0, percentile=100.00%, depth=256 
 
Run status group 0 (all jobs): 
 WRITE: bw=8010KiB/s (8202kB/s), 8010KiB/s-8010KiB/s (8202kB/s-8202kB/s), io=478MiB (501MB), run=61102-61102msec 
 
Disk stats (read/write): 
 sda: ios=0/60981, sectors=0/975608, merge=0/11, ticks=0/52000246, in_queue=52001504, util=99.95% 
 
============================================================================================================= 
 
root@wjv001:~/fio# ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=randrw --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=0 --group_reporting 
test: (g=0): rw=randrw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256 
... 
fio-3.37-60-g6eaa64 
Starting 4 processes 
Jobs: 4 (f=4): [w(4)][100.0%][w=6904KiB/s][w=863 IOPS][eta 00m:00s] 
test: (groupid=0, jobs=4): err= 0: pid=18693: Fri Jul 12 19:39:00 2024 
 write: IOPS=1001, BW=8009KiB/s (8201kB/s)(476MiB/60853msec); 0 zone resets 
 bw ( KiB/s): min= 256, max=24624, per=100.00%, avg=8209.34, stdev=1657.90, samples=470 
 iops : min= 32, max= 3078, avg=1026.15, stdev=207.24, samples=470 
 cpu : usr=0.15%, sys=0.31%, ctx=49963, majf=0, minf=28 
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6% 
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% 
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% 
 issued rwts: total=0,60920,0,0 short=0,0,0,0 dropped=0,0,0,0 
 latency : target=0, window=0, percentile=100.00%, depth=256 
 
Run status group 0 (all jobs): 
 WRITE: bw=8009KiB/s (8201kB/s), 8009KiB/s-8009KiB/s (8201kB/s-8201kB/s), io=476MiB (499MB), run=60853-60853msec 
 
Disk stats (read/write): 
 sda: ios=0/60744, sectors=0/971808, merge=0/12, ticks=0/56919694, in_queue=56920300, util=99.97% 
 
============================================================================================================= 
 
-- 
Ammar Faizi 
 
 

[-- Attachment #2: Type: text/html, Size: 12848 bytes --]

Re: [## 385523 ##] Disk I/O di Neo Lite kembali bermasalah

[Yusuf Hadiwinata]

[-- Attachment #1: Type: text/plain, Size: 10136 bytes --]

Dear Ammar dan Team


Kita mau analisa beberapa suspect di Host yang terdapat instance yang di laporkan, boleh kita move instance nya ? Akan ada downtime estimas < 5 menit jika berkenan
Kalo ada preferensi waktu boleh di info juga kira2 enak nya jam berapaan


Thanks






________________________________
From: Ammar Faizi <ammarfaizi2@gnuweeb.org>
Sent: Friday, 12 July 2024 19:42
To: Support <support@biznetgio.com>
Cc: Ammar Rafli Arkan <ammar@biznetgio.com>; Yusuf Hadiwinata <yusuf@biznetgio.com>; Alviro Iskandar Setiawan <alviro.iskandar@gnuweeb.org>; GNU/Weeb Mailing List <gwml@vger.gnuweeb.org>; Paber Panjaitan <faberpanjaitan1@gmail.com>
Subject: Re: [## 385523 ##] Disk I/O di Neo Lite kembali bermasalah

On Fri, Jul 12, 2024 at 05:23:09AM -0700, Ammar Rafli wrote:
>  Silakan untuk melampirkan capture hasil fio nya pada ticket ini agar kami dapat melihat me-review detail IOPS nya ya Pak.

Saya lampirkan output fio berikut:

root@wjv001:~/fio# ./fio --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --rw=read --filename=/root/fio --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=75 --group_reporting
fio: engine libaio not loadable
fio: failed to load engine

# Note: libaio tidak available, saya ubah ke io_uring.

root@wjv001:~/fio# ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=read --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=75 --group_reporting
test: (g=0): rw=read, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256
...
fio-3.37-60-g6eaa64
Starting 4 processes
Jobs: 2 (f=2): [_(2),R(2)][34.3%][r=5853KiB/s][r=731 IOPS][eta 01m:59s]
test: (groupid=0, jobs=4): err= 0: pid=18651: Fri Jul 12 19:25:53 2024
  read: IOPS=1001, BW=8013KiB/s (8205kB/s)(477MiB/60951msec)
   bw (  KiB/s): min=   96, max=20864, per=100.00%, avg=8536.02, stdev=1493.19, samples=385
   iops        : min=   12, max= 2608, avg=1066.99, stdev=186.65, samples=385
  cpu          : usr=0.11%, sys=0.25%, ctx=70031, majf=0, minf=2066
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=61048,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
   READ: bw=8013KiB/s (8205kB/s), 8013KiB/s-8013KiB/s (8205kB/s-8205kB/s), io=477MiB (500MB), run=60951-60951msec

Disk stats (read/write):
  sda: ios=60990/53, sectors=976152/528, merge=0/10, ticks=55711072/42055, in_queue=55753611, util=99.96%

=============================================================================================================

root@wjv001:~/fio# ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=rw --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=50 --group_reporting
test: (g=0): rw=rw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256
...
fio-3.37-60-g6eaa64
Starting 4 processes
Jobs: 4 (f=4): [M(4)][100.0%][r=8032KiB/s,w=8104KiB/s][r=1004,w=1013 IOPS][eta 00m:00s]
test: (groupid=0, jobs=4): err= 0: pid=18664: Fri Jul 12 19:31:16 2024
  read: IOPS=994, BW=7959KiB/s (8150kB/s)(471MiB/60624msec)
   bw (  KiB/s): min=  448, max=21488, per=99.84%, avg=7947.47, stdev=1237.33, samples=481
   iops        : min=   56, max= 2686, avg=993.41, stdev=154.67, samples=481
  write: IOPS=1001, BW=8010KiB/s (8202kB/s)(474MiB/60624msec); 0 zone resets
   bw (  KiB/s): min=  416, max=21520, per=99.71%, avg=7987.95, stdev=1227.20, samples=481
   iops        : min=   52, max= 2690, avg=998.47, stdev=153.40, samples=481
  cpu          : usr=0.20%, sys=0.46%, ctx=96868, majf=0, minf=26
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=60316,60700,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
   READ: bw=7959KiB/s (8150kB/s), 7959KiB/s-7959KiB/s (8150kB/s-8150kB/s), io=471MiB (494MB), run=60624-60624msec
  WRITE: bw=8010KiB/s (8202kB/s), 8010KiB/s-8010KiB/s (8202kB/s-8202kB/s), io=474MiB (497MB), run=60624-60624msec

Disk stats (read/write):
  sda: ios=60146/60471, sectors=962336/967448, merge=0/12, ticks=27376373/30599469, in_queue=57977062, util=99.97%


=============================================================================================================

root@wjv001:~/fio# ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=randread --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=100 --group_reporting
test: (g=0): rw=randread, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256
...
fio-3.37-60-g6eaa64
Starting 4 processes
Jobs: 2 (f=2): [r(1),_(1),r(1),_(1)][34.3%][r=5904KiB/s][r=738 IOPS][eta 01m:59s]
test: (groupid=0, jobs=4): err= 0: pid=18673: Fri Jul 12 19:35:08 2024
  read: IOPS=1001, BW=8013KiB/s (8206kB/s)(478MiB/61139msec)
   bw (  KiB/s): min= 1824, max=18288, per=100.00%, avg=8101.39, stdev=1076.94, samples=445
   iops        : min=  228, max= 2286, avg=1012.67, stdev=134.62, samples=445
  cpu          : usr=0.12%, sys=0.26%, ctx=71964, majf=0, minf=2076
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=61240,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
   READ: bw=8013KiB/s (8206kB/s), 8013KiB/s-8013KiB/s (8206kB/s-8206kB/s), io=478MiB (502MB), run=61139-61139msec

Disk stats (read/write):
  sda: ios=61240/7, sectors=980080/112, merge=0/7, ticks=61595911/6429, in_queue=61602360, util=99.95%

=============================================================================================================

root@wjv001:~/fio# ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=randwrite --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=75 --group_reporting
test: (g=0): rw=randwrite, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256
...
fio-3.37-60-g6eaa64
Starting 4 processes
Jobs: 4 (f=4): [w(4)][3.5%][w=1936KiB/s][w=242 IOPS][eta 28m:31s]
test: (groupid=0, jobs=4): err= 0: pid=18683: Fri Jul 12 19:37:33 2024
  write: IOPS=1001, BW=8010KiB/s (8202kB/s)(478MiB/61102msec); 0 zone resets
   bw (  KiB/s): min=   96, max=27984, per=100.00%, avg=8245.91, stdev=1833.66, samples=465
   iops        : min=   12, max= 3498, avg=1030.70, stdev=229.20, samples=465
  cpu          : usr=0.13%, sys=0.27%, ctx=44021, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=0,61176,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
  WRITE: bw=8010KiB/s (8202kB/s), 8010KiB/s-8010KiB/s (8202kB/s-8202kB/s), io=478MiB (501MB), run=61102-61102msec

Disk stats (read/write):
  sda: ios=0/60981, sectors=0/975608, merge=0/11, ticks=0/52000246, in_queue=52001504, util=99.95%

=============================================================================================================

root@wjv001:~/fio# ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=randrw --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=0 --group_reporting
test: (g=0): rw=randrw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256
...
fio-3.37-60-g6eaa64
Starting 4 processes
Jobs: 4 (f=4): [w(4)][100.0%][w=6904KiB/s][w=863 IOPS][eta 00m:00s]
test: (groupid=0, jobs=4): err= 0: pid=18693: Fri Jul 12 19:39:00 2024
  write: IOPS=1001, BW=8009KiB/s (8201kB/s)(476MiB/60853msec); 0 zone resets
   bw (  KiB/s): min=  256, max=24624, per=100.00%, avg=8209.34, stdev=1657.90, samples=470
   iops        : min=   32, max= 3078, avg=1026.15, stdev=207.24, samples=470
  cpu          : usr=0.15%, sys=0.31%, ctx=49963, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=0,60920,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
  WRITE: bw=8009KiB/s (8201kB/s), 8009KiB/s-8009KiB/s (8201kB/s-8201kB/s), io=476MiB (499MB), run=60853-60853msec

Disk stats (read/write):
  sda: ios=0/60744, sectors=0/971808, merge=0/12, ticks=0/56919694, in_queue=56920300, util=99.97%

=============================================================================================================

--
Ammar Faizi

The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments.

[-- Attachment #2: Type: text/html, Size: 15312 bytes --]

Re: [## 385523 ##] Disk I/O di Neo Lite kembali bermasalah

[Yusuf Hadiwinata]

[-- Attachment #1: Type: text/plain, Size: 11167 bytes --]

Dear Ammar dan Team

Sorry setelah kita cek IP 103.175.218.159 tidak di host yang kita suspect, mohon ignore email sebelumnya

Kita coba analisa lagi ya

thnks
________________________________
From: Yusuf Hadiwinata <yusuf@biznetgio.com>
Sent: Friday, 12 July 2024 20:06
To: Ammar Faizi <ammarfaizi2@gnuweeb.org>; Support <support@biznetgio.com>
Cc: Ammar Rafli Arkan <ammar@biznetgio.com>; Alviro Iskandar Setiawan <alviro.iskandar@gnuweeb.org>; GNU/Weeb Mailing List <gwml@vger.gnuweeb.org>; Paber Panjaitan <faberpanjaitan1@gmail.com>
Subject: Re: [## 385523 ##] Disk I/O di Neo Lite kembali bermasalah

Dear Ammar dan Team


Kita mau analisa beberapa suspect di Host yang terdapat instance yang di laporkan, boleh kita move instance nya ? Akan ada downtime estimas < 5 menit jika berkenan
Kalo ada preferensi waktu boleh di info juga kira2 enak nya jam berapaan


Thanks






________________________________
From: Ammar Faizi <ammarfaizi2@gnuweeb.org>
Sent: Friday, 12 July 2024 19:42
To: Support <support@biznetgio.com>
Cc: Ammar Rafli Arkan <ammar@biznetgio.com>; Yusuf Hadiwinata <yusuf@biznetgio.com>; Alviro Iskandar Setiawan <alviro.iskandar@gnuweeb.org>; GNU/Weeb Mailing List <gwml@vger.gnuweeb.org>; Paber Panjaitan <faberpanjaitan1@gmail.com>
Subject: Re: [## 385523 ##] Disk I/O di Neo Lite kembali bermasalah

On Fri, Jul 12, 2024 at 05:23:09AM -0700, Ammar Rafli wrote:
>  Silakan untuk melampirkan capture hasil fio nya pada ticket ini agar kami dapat melihat me-review detail IOPS nya ya Pak.

Saya lampirkan output fio berikut:

root@wjv001:~/fio# ./fio --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --rw=read --filename=/root/fio --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=75 --group_reporting
fio: engine libaio not loadable
fio: failed to load engine

# Note: libaio tidak available, saya ubah ke io_uring.

root@wjv001:~/fio# ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=read --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=75 --group_reporting
test: (g=0): rw=read, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256
...
fio-3.37-60-g6eaa64
Starting 4 processes
Jobs: 2 (f=2): [_(2),R(2)][34.3%][r=5853KiB/s][r=731 IOPS][eta 01m:59s]
test: (groupid=0, jobs=4): err= 0: pid=18651: Fri Jul 12 19:25:53 2024
  read: IOPS=1001, BW=8013KiB/s (8205kB/s)(477MiB/60951msec)
   bw (  KiB/s): min=   96, max=20864, per=100.00%, avg=8536.02, stdev=1493.19, samples=385
   iops        : min=   12, max= 2608, avg=1066.99, stdev=186.65, samples=385
  cpu          : usr=0.11%, sys=0.25%, ctx=70031, majf=0, minf=2066
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=61048,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
   READ: bw=8013KiB/s (8205kB/s), 8013KiB/s-8013KiB/s (8205kB/s-8205kB/s), io=477MiB (500MB), run=60951-60951msec

Disk stats (read/write):
  sda: ios=60990/53, sectors=976152/528, merge=0/10, ticks=55711072/42055, in_queue=55753611, util=99.96%

=============================================================================================================

root@wjv001:~/fio# ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=rw --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=50 --group_reporting
test: (g=0): rw=rw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256
...
fio-3.37-60-g6eaa64
Starting 4 processes
Jobs: 4 (f=4): [M(4)][100.0%][r=8032KiB/s,w=8104KiB/s][r=1004,w=1013 IOPS][eta 00m:00s]
test: (groupid=0, jobs=4): err= 0: pid=18664: Fri Jul 12 19:31:16 2024
  read: IOPS=994, BW=7959KiB/s (8150kB/s)(471MiB/60624msec)
   bw (  KiB/s): min=  448, max=21488, per=99.84%, avg=7947.47, stdev=1237.33, samples=481
   iops        : min=   56, max= 2686, avg=993.41, stdev=154.67, samples=481
  write: IOPS=1001, BW=8010KiB/s (8202kB/s)(474MiB/60624msec); 0 zone resets
   bw (  KiB/s): min=  416, max=21520, per=99.71%, avg=7987.95, stdev=1227.20, samples=481
   iops        : min=   52, max= 2690, avg=998.47, stdev=153.40, samples=481
  cpu          : usr=0.20%, sys=0.46%, ctx=96868, majf=0, minf=26
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=60316,60700,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
   READ: bw=7959KiB/s (8150kB/s), 7959KiB/s-7959KiB/s (8150kB/s-8150kB/s), io=471MiB (494MB), run=60624-60624msec
  WRITE: bw=8010KiB/s (8202kB/s), 8010KiB/s-8010KiB/s (8202kB/s-8202kB/s), io=474MiB (497MB), run=60624-60624msec

Disk stats (read/write):
  sda: ios=60146/60471, sectors=962336/967448, merge=0/12, ticks=27376373/30599469, in_queue=57977062, util=99.97%


=============================================================================================================

root@wjv001:~/fio# ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=randread --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=100 --group_reporting
test: (g=0): rw=randread, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256
...
fio-3.37-60-g6eaa64
Starting 4 processes
Jobs: 2 (f=2): [r(1),_(1),r(1),_(1)][34.3%][r=5904KiB/s][r=738 IOPS][eta 01m:59s]
test: (groupid=0, jobs=4): err= 0: pid=18673: Fri Jul 12 19:35:08 2024
  read: IOPS=1001, BW=8013KiB/s (8206kB/s)(478MiB/61139msec)
   bw (  KiB/s): min= 1824, max=18288, per=100.00%, avg=8101.39, stdev=1076.94, samples=445
   iops        : min=  228, max= 2286, avg=1012.67, stdev=134.62, samples=445
  cpu          : usr=0.12%, sys=0.26%, ctx=71964, majf=0, minf=2076
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=61240,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
   READ: bw=8013KiB/s (8206kB/s), 8013KiB/s-8013KiB/s (8206kB/s-8206kB/s), io=478MiB (502MB), run=61139-61139msec

Disk stats (read/write):
  sda: ios=61240/7, sectors=980080/112, merge=0/7, ticks=61595911/6429, in_queue=61602360, util=99.95%

=============================================================================================================

root@wjv001:~/fio# ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=randwrite --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=75 --group_reporting
test: (g=0): rw=randwrite, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256
...
fio-3.37-60-g6eaa64
Starting 4 processes
Jobs: 4 (f=4): [w(4)][3.5%][w=1936KiB/s][w=242 IOPS][eta 28m:31s]
test: (groupid=0, jobs=4): err= 0: pid=18683: Fri Jul 12 19:37:33 2024
  write: IOPS=1001, BW=8010KiB/s (8202kB/s)(478MiB/61102msec); 0 zone resets
   bw (  KiB/s): min=   96, max=27984, per=100.00%, avg=8245.91, stdev=1833.66, samples=465
   iops        : min=   12, max= 3498, avg=1030.70, stdev=229.20, samples=465
  cpu          : usr=0.13%, sys=0.27%, ctx=44021, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=0,61176,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
  WRITE: bw=8010KiB/s (8202kB/s), 8010KiB/s-8010KiB/s (8202kB/s-8202kB/s), io=478MiB (501MB), run=61102-61102msec

Disk stats (read/write):
  sda: ios=0/60981, sectors=0/975608, merge=0/11, ticks=0/52000246, in_queue=52001504, util=99.95%

=============================================================================================================

root@wjv001:~/fio# ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=randrw --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=0 --group_reporting
test: (g=0): rw=randrw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256
...
fio-3.37-60-g6eaa64
Starting 4 processes
Jobs: 4 (f=4): [w(4)][100.0%][w=6904KiB/s][w=863 IOPS][eta 00m:00s]
test: (groupid=0, jobs=4): err= 0: pid=18693: Fri Jul 12 19:39:00 2024
  write: IOPS=1001, BW=8009KiB/s (8201kB/s)(476MiB/60853msec); 0 zone resets
   bw (  KiB/s): min=  256, max=24624, per=100.00%, avg=8209.34, stdev=1657.90, samples=470
   iops        : min=   32, max= 3078, avg=1026.15, stdev=207.24, samples=470
  cpu          : usr=0.15%, sys=0.31%, ctx=49963, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=0,60920,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
  WRITE: bw=8009KiB/s (8201kB/s), 8009KiB/s-8009KiB/s (8201kB/s-8201kB/s), io=476MiB (499MB), run=60853-60853msec

Disk stats (read/write):
  sda: ios=0/60744, sectors=0/971808, merge=0/12, ticks=0/56919694, in_queue=56920300, util=99.97%

=============================================================================================================

--
Ammar Faizi

The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments.
The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments.

[-- Attachment #2: Type: text/html, Size: 18057 bytes --]

Re: [## 385523 ##] Disk I/O di Neo Lite kembali bermasalah

[Ammar Faizi]

On Fri, Jul 12, 2024 at 8:06 PM Yusuf Hadiwinata wrote:
> Kita mau analisa beberapa suspect di Host yang terdapat instance
> yang di laporkan, boleh kita move instance nya ? Akan ada downtime
> estimas < 5 menit jika berkenan

Boleh.

> Kalo ada preferensi waktu boleh di info juga kira2 enak nya jam berapaan

Untuk VM dengan IP 103.175.218.159 bisa sekarang. Karena yang itu
saya beli memang khusus untuk testing Neo Lite dan pembuktian masalah
kemarin. Jadi tidak saya pakai untuk production.

-- 
Ammar Faizi

Re: [## 385523 ##] Disk I/O di Neo Lite kembali bermasalah

[Ammar Faizi]

On Fri, Jul 12, 2024 at 8:12 PM Yusuf Hadiwinata <yusuf@biznetgio.com> wrote:
> Dear Ammar dan Team
>
> Sorry setelah kita cek IP 103.175.218.159 tidak di host yang kita suspect, mohon ignore email sebelumnya
>
> Kita coba analisa lagi ya

Oke.

-- 
Ammar Faizi

Re: [## 385523 ##] Disk I/O di Neo Lite kembali bermasalah

[Ammar Faizi]

On Fri, Jul 12, 2024 at 06:00:04AM -0700, Ammar Rafli wrote:
> Hi Bapak Ammar
>  
>  Selamat Malam
>  
> Menanggapi email yang disampaikan, Mohon informasinya apakah Bapak melakukan fio nya saat script stresstest berjalan? 
>  Jika iya, apakah Bapak bisa men-kill script stresstest nya dahulu lalu coba jalankan command fio nya kembali Pak?
>  
>  Mohon informasinya pada ticket ini.

Iya, tadi dalam keadaan stress test berjalan, berikut hasilnya setelah
saya test fio lagi tanpa dibarengi stress test:

root@wjv001:~/fio# ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=write --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=75 --group_reporting && \
./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=rw --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=50 --group_reporting && \
./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=randread --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=100 --group_reporting && \
./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=randwrite --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=75 --group_reporting && \
./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=randrw --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=0 --group_reporting
test: (g=0): rw=write, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256
...
fio-3.37-60-g6eaa64
Starting 4 processes
Jobs: 4 (f=4): [W(4)][5.4%][w=1840KiB/s][w=230 IOPS][eta 18m:15s]  
test: (groupid=0, jobs=4): err= 0: pid=18772: Fri Jul 12 20:10:11 2024
  write: IOPS=1000, BW=8003KiB/s (8195kB/s)(477MiB/61088msec); 0 zone resets
   bw (  KiB/s): min=  416, max=21296, per=100.00%, avg=8027.61, stdev=1013.58, samples=479
   iops        : min=   52, max= 2662, avg=1003.43, stdev=126.70, samples=479
  cpu          : usr=0.09%, sys=0.30%, ctx=26174, majf=0, minf=19
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=0,61112,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
  WRITE: bw=8003KiB/s (8195kB/s), 8003KiB/s-8003KiB/s (8195kB/s-8195kB/s), io=477MiB (501MB), run=61088-61088msec

Disk stats (read/write):
  sda: ios=1/60973, sectors=8/975392, merge=0/23, ticks=1/54371694, in_queue=54372476, util=99.97%
test: (g=0): rw=rw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256
...
fio-3.37-60-g6eaa64
Starting 4 processes
Jobs: 4 (f=4): [M(4)][100.0%][r=8368KiB/s,w=8312KiB/s][r=1046,w=1039 IOPS][eta 00m:00s]
test: (groupid=0, jobs=4): err= 0: pid=18830: Fri Jul 12 20:11:12 2024
  read: IOPS=993, BW=7949KiB/s (8139kB/s)(469MiB/60477msec)
   bw (  KiB/s): min=  928, max=15264, per=100.00%, avg=7953.51, stdev=572.87, samples=479
   iops        : min=  116, max= 1908, avg=994.16, stdev=71.61, samples=479
  write: IOPS=998, BW=7992KiB/s (8184kB/s)(472MiB/60477msec); 0 zone resets
   bw (  KiB/s): min= 2768, max=15456, per=100.00%, avg=8061.87, stdev=546.67, samples=476
   iops        : min=  346, max= 1932, avg=1007.71, stdev=68.33, samples=476
  cpu          : usr=0.18%, sys=0.61%, ctx=57583, majf=0, minf=27
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=60089,60415,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
   READ: bw=7949KiB/s (8139kB/s), 7949KiB/s-7949KiB/s (8139kB/s-8139kB/s), io=469MiB (492MB), run=60477-60477msec
  WRITE: bw=7992KiB/s (8184kB/s), 7992KiB/s-7992KiB/s (8184kB/s-8184kB/s), io=472MiB (495MB), run=60477-60477msec

Disk stats (read/write):
  sda: ios=60000/60435, sectors=960000/966856, merge=0/7, ticks=29529611/28322494, in_queue=57852707, util=100.00%
test: (g=0): rw=randread, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256
...
fio-3.37-60-g6eaa64
Starting 4 processes
Jobs: 4 (f=4): [r(4)][5.1%][r=1873KiB/s][r=234 IOPS][eta 18m:54s]   
test: (groupid=0, jobs=4): err= 0: pid=18836: Fri Jul 12 20:12:13 2024
  read: IOPS=1001, BW=8012KiB/s (8205kB/s)(478MiB/61145msec)
   bw (  KiB/s): min=  224, max=18592, per=100.00%, avg=8096.67, stdev=934.34, samples=476
   iops        : min=   28, max= 2326, avg=1012.10, stdev=116.80, samples=476
  cpu          : usr=0.15%, sys=0.33%, ctx=41342, majf=0, minf=2075
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=61240,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
   READ: bw=8012KiB/s (8205kB/s), 8012KiB/s-8012KiB/s (8205kB/s-8205kB/s), io=478MiB (502MB), run=61145-61145msec

Disk stats (read/write):
  sda: ios=61234/3, sectors=979744/32, merge=0/1, ticks=56379750/13301, in_queue=56393126, util=99.98%
test: (g=0): rw=randwrite, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256
...
fio-3.37-60-g6eaa64
Starting 4 processes
Jobs: 4 (f=4): [w(4)][5.5%][w=1912KiB/s][w=239 IOPS][eta 17m:29s]  
test: (groupid=0, jobs=4): err= 0: pid=18842: Fri Jul 12 20:13:14 2024
  write: IOPS=1001, BW=8010KiB/s (8202kB/s)(477MiB/60974msec); 0 zone resets
   bw (  KiB/s): min=   64, max=20992, per=100.00%, avg=8018.16, stdev=1004.72, samples=479
   iops        : min=    8, max= 2624, avg=1002.12, stdev=125.60, samples=479
  cpu          : usr=0.14%, sys=0.32%, ctx=27728, majf=0, minf=26
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=0,61048,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
  WRITE: bw=8010KiB/s (8202kB/s), 8010KiB/s-8010KiB/s (8202kB/s-8202kB/s), io=477MiB (500MB), run=60974-60974msec

Disk stats (read/write):
  sda: ios=0/60980, sectors=0/975616, merge=0/8, ticks=0/53681693, in_queue=53682164, util=100.00%
test: (g=0): rw=randrw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256
...
fio-3.37-60-g6eaa64
Starting 4 processes
Jobs: 4 (f=4): [w(4)][4.9%][w=1896KiB/s][w=237 IOPS][eta 19m:54s]  
test: (groupid=0, jobs=4): err= 0: pid=18848: Fri Jul 12 20:14:16 2024
  write: IOPS=1001, BW=8012KiB/s (8204kB/s)(477MiB/61021msec); 0 zone resets
   bw (  KiB/s): min=   80, max=18352, per=100.00%, avg=8012.34, stdev=973.54, samples=480
   iops        : min=   10, max= 2294, avg=1001.54, stdev=121.69, samples=480
  cpu          : usr=0.15%, sys=0.29%, ctx=28583, majf=0, minf=18
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=0,61112,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
  WRITE: bw=8012KiB/s (8204kB/s), 8012KiB/s-8012KiB/s (8204kB/s-8204kB/s), io=477MiB (501MB), run=61021-61021msec

Disk stats (read/write):
  sda: ios=0/60986, sectors=0/975768, merge=0/2, ticks=0/53705577, in_queue=53706042, util=99.98%
root@wjv001:~/fio# 

-- 
Ammar Faizi

Re: [## 385523 ##] Disk I/O di Neo Lite kembali bermasalah

[Yusuf Hadiwinata]

[-- Attachment #1: Type: text/plain, Size: 1230 bytes --]

Untuk IP 103.175.218.159 boleh di infokan OS dan Kernel Version yang di gunakan

thnks

________________________________
From: Ammar Faizi <ammarfaizi2@gnuweeb.org>
Sent: Friday, 12 July 2024 20:13
To: Yusuf Hadiwinata <yusuf@biznetgio.com>
Cc: Support <support@biznetgio.com>; Ammar Rafli Arkan <ammar@biznetgio.com>; Alviro Iskandar Setiawan <alviro.iskandar@gnuweeb.org>; GNU/Weeb Mailing List <gwml@vger.gnuweeb.org>; Paber Panjaitan <faberpanjaitan1@gmail.com>
Subject: Re: [## 385523 ##] Disk I/O di Neo Lite kembali bermasalah

On Fri, Jul 12, 2024 at 8:12 PM Yusuf Hadiwinata <yusuf@biznetgio.com> wrote:
> Dear Ammar dan Team
>
> Sorry setelah kita cek IP 103.175.218.159 tidak di host yang kita suspect, mohon ignore email sebelumnya
>
> Kita coba analisa lagi ya

Oke.

--
Ammar Faizi
The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments.

[-- Attachment #2: Type: text/html, Size: 2661 bytes --]

Re: [## 385523 ##] Disk I/O di Neo Lite kembali bermasalah

[Ammar Faizi]

On Fri, Jul 12, 2024 at 01:20:29PM +0000, Yusuf Hadiwinata wrote:
> Untuk IP 103.175.218.159 boleh di infokan OS dan Kernel Version yang di gunakan

root@wjv001:~/fio# curl ip.me
103.175.218.159
root@wjv001:~/fio# uname -r
6.8.1-af-server-2024-03-19-g760d1c206de8
root@wjv001:~/fio# cat /etc/*release*
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.4 LTS"
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
root@wjv001:~/fio# 

-- 
Ammar Faizi

Re: [## 385523 ##] Disk I/O di Neo Lite kembali bermasalah

[Yusuf Hadiwinata]

[-- Attachment #1: Type: text/plain, Size: 1809 bytes --]

Thnks you

Mau memastikan, apakah sebelum terjadi Blocking Thread ada OOM

thnks

________________________________
From: Ammar Faizi <ammarfaizi2@gnuweeb.org>
Sent: Friday, 12 July 2024 20:21
To: Yusuf Hadiwinata <yusuf@biznetgio.com>
Cc: Support <support@biznetgio.com>; Ammar Rafli Arkan <ammar@biznetgio.com>; Alviro Iskandar Setiawan <alviro.iskandar@gnuweeb.org>; GNU/Weeb Mailing List <gwml@vger.gnuweeb.org>; Paber Panjaitan <faberpanjaitan1@gmail.com>
Subject: Re: [## 385523 ##] Disk I/O di Neo Lite kembali bermasalah

On Fri, Jul 12, 2024 at 01:20:29PM +0000, Yusuf Hadiwinata wrote:
> Untuk IP 103.175.218.159 boleh di infokan OS dan Kernel Version yang di gunakan

root@wjv001:~/fio# curl ip.me
103.175.218.159
root@wjv001:~/fio# uname -r
6.8.1-af-server-2024-03-19-g760d1c206de8
root@wjv001:~/fio# cat /etc/*release*
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.4 LTS"
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
root@wjv001:~/fio#

--
Ammar Faizi

The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments.

[-- Attachment #2: Type: text/html, Size: 3953 bytes --]

Re: [## 385523 ##] Disk I/O di Neo Lite kembali bermasalah

[Ammar Faizi]

On Fri, Jul 12, 2024 at 01:31:30PM +0000, Yusuf Hadiwinata wrote:
> Thnks you
> 
> Mau memastikan, apakah sebelum terjadi Blocking Thread ada OOM

Iya ada, saya juga stress test memory, sempat hit OOM killer juga.

Untuk testing memory saya pakai simple PHP script yang memakan memory
sekitar 11 GB seperti ini:

<?php
$nr = 1024*1024*1024*5;
$a = str_repeat("a", $nr);
echo "ar\n";
$b = str_repeat("b", $nr);
echo "br\n";

while (1) {

	$tmp = "";
	for ($i = 0; $i < $nr; $i++) {
		$tmp .= $a[$i];
	}
	echo "A done!\n";

	$tmp .= "";
	for ($i = 0; $i < $nr; $i++) {
		$tmp .= $b[$i];
	}
	echo "B done!\n";

	sleep(5);
}

----
dmesg:

[15832.976830] Out of memory: Killed process 7142 (php) total-vm:12649176kB, anon-rss:4339492kB, file-rss:3200kB, shmem-rss:0kB, UID:0 pgtables:24720kB oom_score_adj:0
[15836.351730] oom_reaper: reaped process 7142 (php), now anon-rss:636kB, file-rss:796kB, shmem-rss:0kB
[22471.722388] ct invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
[22471.722420] CPU: 7 PID: 8149 Comm: ct Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8 5aa4ec4a5a8034d921df04c799edf17a90d1ec90
[22471.722434] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.1-0-g3208b098f51a-prebuilt.qemu.org 04/01/2014
[22471.722446] Call Trace:
[22471.722508]  <TASK>
[22471.722543]  dump_header+0x50/0x170
[22471.722612]  oom_kill_process+0x1c0/0x5b0
[22471.722624]  out_of_memory+0x7af/0xdf0
[22471.722628]  __alloc_pages_slowpath+0xba1/0x1210
[22471.722648]  __alloc_pages+0x23d/0x330
[22471.722657]  alloc_pages_mpol+0xf4/0x1a0
[22471.722670]  filemap_alloc_folio+0xa3/0x1d0
[22471.722690]  __filemap_get_folio+0x195/0x480
[22471.722694]  filemap_fault+0x212/0x830
[22471.722700]  ? __slab_free+0x5a/0x1a0
[22471.722704]  __do_fault+0x3b/0x120
[22471.722714]  do_pte_missing+0x16b/0x1060
[22471.722716]  ? skb_release_data+0x1c6/0x2b0
[22471.722740]  handle_mm_fault+0x93c/0x1380
[22471.722743]  do_user_addr_fault+0x202/0xa10
[22471.722761]  exc_page_fault+0x53/0xc0
[22471.722788]  asm_exc_page_fault+0x22/0x30
[22471.722802] RIP: 0033:0x557bbad655bb
[22471.722814] Code: Unable to access opcode bytes at 0x557bbad65591.
[22471.722815] RSP: 002b:00007fbd6e9ffc48 EFLAGS: 00010206
[22471.722818] RAX: 0000000000000012 RBX: 00007fbd6ea00640 RCX: 0000000000000001
[22471.722820] RDX: 0000000000000000 RSI: 00007fbd70000b70 RDI: 00007fbd6e9ff8f0
[22471.722821] RBP: 00007fbd6e9ffe48 R08: 0000000000000000 R09: 00007fbd6e9ffd27
[22471.722823] R10: 0000000000000000 R11: 0000000000000293 R12: 00007fbd6ea00640
[22471.722824] R13: 0000000000000000 R14: 00007fbd774947d0 R15: 00007ffdf6585af0
[22471.722826]  </TASK>
[22471.722827] Mem-Info:
[22471.722835] active_anon:1046703 inactive_anon:469261 isolated_anon:96
                active_file:0 inactive_file:41 isolated_file:0
                unevictable:1579777 dirty:3 writeback:0
                slab_reclaimable:6224 slab_unreclaimable:70607
                mapped:3882 shmem:18 pagetables:11208
                sec_pagetables:0 bounce:0
                kernel_misc_reclaimable:0
                free:66954 free_pcp:0 free_cma:0
[22471.722840] Node 0 active_anon:4186812kB inactive_anon:1877044kB active_file:0kB inactive_file:164kB unevictable:6319108kB isolated(anon):384kB isolated(file):0kB mapped:15528kB dirty:12kB writeback:0kB shmem:72kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:0kB writeback_tmp:0kB kernel_stack:5328kB pagetables:44832kB sec_pagetables:0kB all_unreclaimable? no
[22471.722867] Node 0 DMA free:15360kB boost:0kB min:60kB low:72kB high:84kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[22471.722878] lowmem_reserve[]: 0 2856 15884 15884 15884
[22471.722882] Node 0 DMA32 free:64128kB boost:0kB min:12144kB low:15180kB high:18216kB reserved_highatomic:0KB active_anon:472036kB inactive_anon:161520kB active_file:48kB inactive_file:0kB unevictable:2115584kB writepending:0kB present:3129192kB managed:2989864kB mlocked:2115584kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[22471.722889] lowmem_reserve[]: 0 0 13027 13027 13027
[22471.722892] Node 0 Normal free:188328kB boost:0kB min:55372kB low:69212kB high:83052kB reserved_highatomic:133120KB active_anon:3714332kB inactive_anon:1716504kB active_file:300kB inactive_file:592kB unevictable:4203524kB writepending:12kB present:13631488kB managed:13339936kB mlocked:4203524kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[22471.722899] lowmem_reserve[]: 0 0 0 0 0
[22471.722903] Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15360kB
[22471.722964] Node 0 DMA32: 123*4kB (UME) 46*8kB (UME) 48*16kB (UE) 20*32kB (UE) 27*64kB (UE) 29*128kB (UME) 41*256kB (UME) 18*512kB (UME) 6*1024kB (UME) 1*2048kB (M) 7*4096kB (UM) = 64284kB
[22471.722977] Node 0 Normal: 118*4kB (UME) 111*8kB (UME) 135*16kB (UM) 25*32kB (UM) 2330*64kB (UM) 272*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 188256kB
[22471.722994] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[22471.722996] 22547 total pagecache pages
[22471.722997] 20249 pages in swap cache
[22471.722998] Free swap  = 0kB
[22471.722998] Total swap = 8388604kB
[22471.723002] 4194168 pages RAM
[22471.723003] 0 pages HighMem/MovableOnly
[22471.723004] 107878 pages reserved
[22471.723007] 0 pages hwpoisoned
[22471.723008] Tasks state (memory values in pages):
[22471.723012] [  pid  ]   uid  tgid total_vm      rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
[22471.723021] [    451]     0   451     9889      992      128      864         0   102400      192          -250 systemd-journal
[22471.723026] [    485]     0   485    72338     6880     4640     2240         0   110592        0         -1000 multipathd
[22471.723034] [    489]     0   489     6527      858       96      762         0    77824      416         -1000 systemd-udevd
[22471.723037] [    651]   103   651    22341     1088       96      992         0    73728       96             0 systemd-timesyn
[22471.723040] [    694]   100   694     4032      928      128      800         0    65536      128             0 systemd-network
[22471.723043] [    697]   101   697     6385     1082      218      864         0    81920      864             0 systemd-resolve
[22471.723045] [    734]     0   734     1822      768       64      704         0    45056        0             0 cron
[22471.723048] [    736]   102   736     2189     1184      224      960         0    49152       32          -900 dbus-daemon
[22471.723050] [    749]     0   749    20705      832        0      832         0    61440       96             0 irqbalance
[22471.723055] [    751]     0   751     8296      862       30      832         0    98304     2144             0 networkd-dispat
[22471.723058] [    752]     0   752    58626      992       96      896         0    94208       64             0 polkitd
[22471.723060] [    755]     0   755    20013      768       32      736         0    57344        0             0 qemu-ga
[22471.723063] [    761]     0   761   588565     1194      650      544         0   450560     2176          -900 snapd
[22471.723065] [    764]     0   764     3883      992      128      864         0    65536      128             0 systemd-logind
[22471.723067] [    766]     0   766    98156     1216      192     1024         0   139264      416             0 udisksd
[22471.723083] [    772]     0   772     1555      512       32      480         0    45056        0             0 agetty
[22471.723085] [    782]     0   782     1544      480       32      448         0    45056        0             0 agetty
[22471.723090] [    806]     0   806    27526      896       32      864         0   122880     1984             0 unattended-upgr
[22471.723093] [    826]     0   826    61061     1024       96      928         0   106496      352             0 ModemManager
[22471.723095] [    827]     0   827     3859     1056       96      960         0    61440      288         -1000 sshd
[22471.723098] [   1098]     0  1098     4278      768       64      704         0    73728      352             0 systemd
[22471.723103] [   1099]     0  1099    42339      430       78      352         0    90112      864             0 (sd-pam)
[22471.723106] [   1680]     0  1680    74007     1056      128      928         0   155648      672             0 packagekitd
[22471.723108] [   2138]   104  2138    55601     1184      160     1024         0    90112      192             0 rsyslogd
[22471.723116] [   2212]     0  2212     2033      576       32      544         0    40960      256             0 screen
[22471.723119] [   2213]     0  2213     2040      608        0      608         0    45056      160             0 bash
[22471.723123] [   3567]     0  3567     1996      608       64      544         0    40960      160             0 screen
[22471.723132] [   3568]     0  3568     2041      896      192      704         0    53248        0             0 bash
[22471.723139] [   4804]     0  4804     1961      512        0      512         0    40960      224             0 screen
[22471.723144] [   4805]     0  4805     2041      576       32      544         0    49152      160             0 bash
[22471.723147] [   5361]     0  5361     1961      480        0      480         0    49152      192             0 screen
[22471.723149] [   5362]     0  5362     2041      576       64      512         0    49152      160             0 bash
[22471.723151] [   5385]     0  5385     2241     1280       64     1216         0    57344        0             0 arpflood
[22471.723166] [   5389]     0  5389     2241     1248       64     1184         0    53248        0             0 arpflood
[22471.723168] [   5921]     0  5921     1928      512       32      480         0    45056      160             0 screen
[22471.723174] [   5922]     0  5922     2041      544       32      512         0    53248      160             0 bash
[22471.723182] [   5928]     0  5928     2241     1216       64     1152         0    57344        0             0 arpflood
[22471.723187] [   5951]     0  5951     1961      480        0      480         0    49152      224             0 screen
[22471.723201] [   5952]     0  5952     2041      640       96      544         0    49152      128             0 bash
[22471.723204] [   5963]     0  5963     1993      544       96      448         0    45056      160             0 screen
[22471.723206] [   5964]     0  5964     2041      608       32      576         0    53248      160             0 bash
[22471.723208] [   6064]     0  6064     1961      448        0      448         0    49152      224             0 screen
[22471.723214] [   6065]     0  6065     2121      832      128      704         0    53248      160             0 bash
[22471.723219] [   6900]     0  6900     1961      512       32      480         0    45056      160             0 screen
[22471.723221] [   6901]     0  6901     2087      896      160      736         0    57344       32             0 bash
[22471.723223] [   6916]     0  6916     1961      512       32      480         0    40960      160             0 screen
[22471.723225] [   6917]     0  6917     2084      864      192      672         0    57344       96             0 bash
[22471.723228] [   8141]     0  8141    49862      320        0      320         0   172032        0             0 ct
[22471.723231] [   9210]     0  9210  1573558  1573088  1572768      320         0 12648448        0             0 memt
[22471.723248] [   9246]     0  9246  1589430   250395   249659      736         0 11841536  1214465             0 php
[22471.723252] [   9631]     0  9631    24532      864       96      768         0    94208      640             0 curl
[22471.723255] [   9638]     0  9638    24532      832       96      736         0    94208      640             0 curl
[22471.723257] [   9663]     0  9663    24532      832      128      704         0    90112      640             0 curl
[22471.723259] [   9665]     0  9665    53998      640       64      576         0   106496      160             0 fio
[22471.723261] [   9667]     0  9667  1091336   756214   755926      288         0  8511488   297120             0 fio
[22471.723264] [   9668]     0  9668  1091337   490038   489782      256         0  8511488   563264             0 fio
[22471.723269] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/user.slice/user-0.slice/session-3.scope,task=memt,pid=9210,uid=0
[22471.723310] Out of memory: Killed process 9210 (memt) total-vm:6294232kB, anon-rss:6291072kB, file-rss:1280kB, shmem-rss:0kB, UID:0 pgtables:12352kB oom_score_adj:0
[61796.908278] INFO: task jbd2/sda1-8:377 blocked for more than 30 seconds.
[61796.909531]       Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8
[61796.910820] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[61796.912369] task:jbd2/sda1-8     state:D stack:0     pid:377   tgid:377   ppid:2      flags:0x00004000
[61796.912389] Call Trace:
[61796.912399]  <TASK>
[61796.912426]  __schedule+0x5e7/0x14a0
[61796.912487]  ? submit_bio_noacct_nocheck+0x2cc/0x360
[61796.912508]  schedule+0x8e/0x160
[61796.912512]  bit_wait_io+0x45/0xa0
[61796.912518]  __wait_on_bit+0xe1/0x170
[61796.912521]  ? bit_wait_io_timeout+0xc0/0xc0
[61796.912524]  jbd2_journal_commit_transaction+0x191b/0x2320
[61796.912547]  ? wake_up_bit+0xd0/0xd0
[61796.912564]  kjournald2+0x120/0x460
[61796.912577]  ? init_wait_entry+0x30/0x30
[61796.912580]  kthread+0x1f9/0x230
[61796.912594]  ? jbd2_journal_update_sb_log_tail+0x180/0x180
[61796.912599]  ? kthreadd+0x380/0x380
[61796.912601]  ret_from_fork+0x30/0x40
[61796.912620]  ? kthreadd+0x380/0x380
[61796.912622]  ret_from_fork_asm+0x11/0x20
[61796.912665]  </TASK>
[61875.585039] show_signal_msg: 8 callbacks suppressed
[61875.585050] cq[14120]: segfault at 7f611add1000 ip 000055f05765c4e8 sp 00007f61193ffdb0 error 4 in cq[55f05765c000+1000] likely on CPU 2 (core 2, socket 0)
[61875.585106] Code: 89 d1 48 d3 e6 49 09 34 c4 e8 64 fc ff ff 4c 89 e2 be 80 00 00 00 48 c1 eb 02 48 89 c7 e8 10 fc ff ff 89 d9 48 89 ef 0f 57 c9 <0f> 28 07 0f 58 c8 0f 29 0f 48 83 c7 10 e2 f1 eb ef 48 8b 84 24 88
[61887.147744] INFO: task jbd2/sda1-8:377 blocked for more than 30 seconds.
[61887.148999]       Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8
[61887.150213] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[61887.151554] task:jbd2/sda1-8     state:D stack:0     pid:377   tgid:377   ppid:2      flags:0x00004000
[61887.151561] Call Trace:
[61887.151563]  <TASK>
[61887.151576]  __schedule+0x5e7/0x14a0
[61887.151586]  ? submit_bio_noacct_nocheck+0x2cc/0x360
[61887.151591]  schedule+0x8e/0x160
[61887.151594]  bit_wait_io+0x45/0xa0
[61887.151597]  __wait_on_bit+0xe1/0x170
[61887.151600]  ? bit_wait_io_timeout+0xc0/0xc0
[61887.151604]  jbd2_journal_commit_transaction+0x191b/0x2320
[61887.151609]  ? wake_up_bit+0xd0/0xd0
[61887.151613]  kjournald2+0x120/0x460
[61887.151618]  ? init_wait_entry+0x30/0x30
[61887.151621]  kthread+0x1f9/0x230
[61887.151624]  ? jbd2_journal_update_sb_log_tail+0x180/0x180
[61887.151628]  ? kthreadd+0x380/0x380
[61887.151630]  ret_from_fork+0x30/0x40
[61887.151634]  ? kthreadd+0x380/0x380
[61887.151636]  ret_from_fork_asm+0x11/0x20
[61887.151641]  </TASK>
[61887.151644] INFO: task journal-offline:14113 blocked for more than 30 seconds.
[61887.153004]       Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8
[61887.154263] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[61887.157148] task:journal-offline state:D stack:0     pid:14113 tgid:451   ppid:1      flags:0x00004002
[61887.157158] Call Trace:
[61887.157162]  <TASK>
[61887.157170]  __schedule+0x5e7/0x14a0
[61887.157185]  ? wait_for_common_io+0x171/0x190
[61887.157191]  ? wbc_detach_inode+0x103/0x230
[61887.157213]  ? prepare_to_wait_event+0x125/0x140
[61887.157218]  schedule+0x8e/0x160
[61887.157221]  jbd2_log_wait_commit+0x1b7/0x200
[61887.157230]  ? init_wait_entry+0x30/0x30
[61887.157232]  ext4_sync_file+0x47c/0x590
[61887.157254]  __x64_sys_fsync+0xb0/0x100
[61887.157268]  do_syscall_64+0x50/0x110
[61887.157293]  entry_SYSCALL_64_after_hwframe+0x24/0x2c
[61887.157311] RIP: 0033:0x7f923d11b8ab
[61887.157319] RSP: 002b:00007f923b7ffc50 EFLAGS: 00000293 ORIG_RAX: 000000000000004a
[61887.157342] RAX: ffffffffffffffda RBX: 000055c68b01c370 RCX: 00007f923d11b8ab
[61887.157344] RDX: 0000000000000000 RSI: 00007f923d64ea34 RDI: 0000000000000019
[61887.157346] RBP: 00007f923d6502c0 R08: 0000000000000000 R09: 00007ffc269604cf
[61887.157348] R10: 0000000000000a34 R11: 0000000000000293 R12: 0000000000000002
[61887.157349] R13: 0000000000000002 R14: 00007f923d0947d0 R15: 00007ffc26960510
[61887.157360]  </TASK>
[63099.559295] traps: cq[14344] trap invalid opcode ip:564f9853d3aa sp:7f779bbffc20 error:0
[63099.559294] traps: cq[14343] trap invalid opcode ip:564f9853d3aa sp:7f779c5ffc20 error:0
[63099.559348]  in cq[564f9853d000+1000]
[63099.559348]  in cq[564f9853d000+1000]


[63421.219463] INFO: task jbd2/sda1-8:377 blocked for more than 30 seconds.
[63421.220972]       Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8
[63421.222346] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[63421.223707] task:jbd2/sda1-8     state:D stack:0     pid:377   tgid:377   ppid:2      flags:0x00004000
[63421.223728] Call Trace:
[63421.223749]  <TASK>
[63421.223777]  __schedule+0x5e7/0x14a0
[63421.223860]  ? submit_bio_noacct_nocheck+0x2cc/0x360
[63421.223882]  schedule+0x8e/0x160
[63421.223886]  bit_wait_io+0x45/0xa0
[63421.223892]  __wait_on_bit+0xe1/0x170
[63421.223895]  ? bit_wait_io_timeout+0xc0/0xc0
[63421.223899]  jbd2_journal_commit_transaction+0x191b/0x2320
[63421.223920]  ? wake_up_bit+0xd0/0xd0
[63421.223938]  kjournald2+0x120/0x460
[63421.223951]  ? init_wait_entry+0x30/0x30
[63421.223955]  kthread+0x1f9/0x230
[63421.223969]  ? jbd2_journal_update_sb_log_tail+0x180/0x180
[63421.223974]  ? kthreadd+0x380/0x380
[63421.223976]  ret_from_fork+0x30/0x40
[63421.224007]  ? kthreadd+0x380/0x380
[63421.224009]  ret_from_fork_asm+0x11/0x20
[63421.224024]  </TASK>
[68384.390473] INFO: task jbd2/sda1-8:377 blocked for more than 30 seconds.
[68384.392485]       Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8
[68384.394706] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[68384.396367] task:jbd2/sda1-8     state:D stack:0     pid:377   tgid:377   ppid:2      flags:0x00004000
[68384.396381] Call Trace:
[68384.396393]  <TASK>
[68384.396416]  __schedule+0x5e7/0x14a0
[68384.396476]  ? submit_bio_noacct_nocheck+0x2cc/0x360
[68384.396497]  schedule+0x8e/0x160
[68384.396501]  bit_wait_io+0x45/0xa0
[68384.396507]  __wait_on_bit+0xe1/0x170
[68384.396510]  ? bit_wait_io_timeout+0xc0/0xc0
[68384.396529]  jbd2_journal_commit_transaction+0x191b/0x2320
[68384.396551]  ? wake_up_bit+0xd0/0xd0
[68384.396568]  kjournald2+0x120/0x460
[68384.396581]  ? init_wait_entry+0x30/0x30
[68384.396584]  kthread+0x1f9/0x230
[68384.396600]  ? jbd2_journal_update_sb_log_tail+0x180/0x180
[68384.396604]  ? kthreadd+0x380/0x380
[68384.396606]  ret_from_fork+0x30/0x40
[68384.396626]  ? kthreadd+0x380/0x380
[68384.396628]  ret_from_fork_asm+0x11/0x20
[68384.396644]  </TASK>
[68655.118907] INFO: task jbd2/sda1-8:377 blocked for more than 30 seconds.
[68655.124237]       Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8
[68655.125785] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[68655.128624] task:jbd2/sda1-8     state:D stack:0     pid:377   tgid:377   ppid:2      flags:0x00004000
[68655.128669] Call Trace:
[68655.128683]  <TASK>
[68655.128718]  __schedule+0x5e7/0x14a0
[68655.128801]  ? submit_bio_noacct_nocheck+0x2cc/0x360
[68655.128829]  schedule+0x8e/0x160
[68655.128833]  bit_wait_io+0x45/0xa0
[68655.128908]  __wait_on_bit+0xe1/0x170
[68655.128913]  ? bit_wait_io_timeout+0xc0/0xc0
[68655.128918]  jbd2_journal_commit_transaction+0x191b/0x2320
[68655.128997]  ? wake_up_bit+0xd0/0xd0
[68655.129025]  kjournald2+0x120/0x460
[68655.129048]  ? init_wait_entry+0x30/0x30
[68655.129053]  kthread+0x1f9/0x230
[68655.129080]  ? jbd2_journal_update_sb_log_tail+0x180/0x180
[68655.129086]  ? kthreadd+0x380/0x380
[68655.129089]  ret_from_fork+0x30/0x40
[68655.129124]  ? kthreadd+0x380/0x380
[68655.129127]  ret_from_fork_asm+0x11/0x20
[68655.129156]  </TASK>
[68895.747345] INFO: task jbd2/sda1-8:377 blocked for more than 30 seconds.
[68895.748831]       Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8
[68895.750303] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[68895.752488] task:jbd2/sda1-8     state:D stack:0     pid:377   tgid:377   ppid:2      flags:0x00004000
[68895.752513] Call Trace:
[68895.752527]  <TASK>
[68895.752590]  __schedule+0x5e7/0x14a0
[68895.752671]  ? submit_bio_noacct_nocheck+0x2cc/0x360
[68895.752699]  schedule+0x8e/0x160
[68895.752703]  bit_wait_io+0x45/0xa0
[68895.752712]  __wait_on_bit+0xe1/0x170
[68895.752716]  ? bit_wait_io_timeout+0xc0/0xc0
[68895.752739]  jbd2_journal_commit_transaction+0x191b/0x2320
[68895.752768]  ? wake_up_bit+0xd0/0xd0
[68895.752792]  kjournald2+0x120/0x460
[68895.752809]  ? init_wait_entry+0x30/0x30
[68895.752813]  kthread+0x1f9/0x230
[68895.752833]  ? jbd2_journal_update_sb_log_tail+0x180/0x180
[68895.752839]  ? kthreadd+0x380/0x380
[68895.752842]  ret_from_fork+0x30/0x40
[68895.752869]  ? kthreadd+0x380/0x380
[68895.752872]  ret_from_fork_asm+0x11/0x20
[68895.752892]  </TASK>
[69677.822593] INFO: task jbd2/sda1-8:377 blocked for more than 30 seconds.
[69677.824222]       Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8
[69677.826159] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[69677.828000] task:jbd2/sda1-8     state:D stack:0     pid:377   tgid:377   ppid:2      flags:0x00004000
[69677.828026] Call Trace:
[69677.828040]  <TASK>
[69677.828076]  __schedule+0x5e7/0x14a0
[69677.828159]  ? submit_bio_noacct_nocheck+0x2cc/0x360
[69677.828187]  schedule+0x8e/0x160
[69677.828191]  bit_wait_io+0x45/0xa0
[69677.828199]  __wait_on_bit+0xe1/0x170
[69677.828204]  ? bit_wait_io_timeout+0xc0/0xc0
[69677.828208]  jbd2_journal_commit_transaction+0x191b/0x2320
[69677.828239]  ? wake_up_bit+0xd0/0xd0
[69677.828263]  kjournald2+0x120/0x460
[69677.828280]  ? init_wait_entry+0x30/0x30
[69677.828299]  kthread+0x1f9/0x230
[69677.828321]  ? jbd2_journal_update_sb_log_tail+0x180/0x180
[69677.828327]  ? kthreadd+0x380/0x380
[69677.828330]  ret_from_fork+0x30/0x40
[69677.828357]  ? kthreadd+0x380/0x380
[69677.828360]  ret_from_fork_asm+0x11/0x20
[69677.828381]  </TASK>
[69677.828385] INFO: task journal-offline:18592 blocked for more than 30 seconds.
[69677.831042]       Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8
[69677.833149] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[69677.835422] task:journal-offline state:D stack:0     pid:18592 tgid:451   ppid:1      flags:0x00004002
[69677.835430] Call Trace:
[69677.835432]  <TASK>
[69677.835436]  __schedule+0x5e7/0x14a0
[69677.835444]  ? wait_for_common_io+0x171/0x190
[69677.835449]  ? prepare_to_wait_event+0x125/0x140
[69677.835454]  schedule+0x8e/0x160
[69677.835457]  jbd2_log_wait_commit+0x1b7/0x200
[69677.835464]  ? init_wait_entry+0x30/0x30
[69677.835468]  ext4_sync_file+0x47c/0x590
[69677.835513]  __x64_sys_fsync+0xb0/0x100
[69677.835542]  do_syscall_64+0x50/0x110
[69677.835569]  entry_SYSCALL_64_after_hwframe+0x24/0x2c
[69677.835593] RIP: 0033:0x7f923d11b8ab
[69677.835602] RSP: 002b:00007f923b7ffc50 EFLAGS: 00000293 ORIG_RAX: 000000000000004a
[69677.835606] RAX: ffffffffffffffda RBX: 000055c68b01c370 RCX: 00007f923d11b8ab
[69677.835609] RDX: 0000000000000000 RSI: 00007f923d64ea34 RDI: 0000000000000019
[69677.835632] RBP: 00007f923d6502c0 R08: 0000000000000000 R09: 00007ffc269604cf
[69677.835634] R10: 0000000000000a34 R11: 0000000000000293 R12: 0000000000000002
[69677.835640] R13: 0000000000000002 R14: 00007f923d0947d0 R15: 00007ffc26960510
[69677.835651]  </TASK>
[69707.922952] INFO: task jbd2/sda1-8:377 blocked for more than 60 seconds.
[69707.926000]       Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8
[69707.928249] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[69707.930623] task:jbd2/sda1-8     state:D stack:0     pid:377   tgid:377   ppid:2      flags:0x00004000
[69707.930632] Call Trace:
[69707.930640]  <TASK>
[69707.930655]  __schedule+0x5e7/0x14a0
[69707.930666]  ? submit_bio_noacct_nocheck+0x2cc/0x360
[69707.930673]  schedule+0x8e/0x160
[69707.930677]  bit_wait_io+0x45/0xa0
[69707.930682]  __wait_on_bit+0xe1/0x170
[69707.930686]  ? bit_wait_io_timeout+0xc0/0xc0
[69707.930691]  jbd2_journal_commit_transaction+0x191b/0x2320
[69707.930698]  ? wake_up_bit+0xd0/0xd0
[69707.930705]  kjournald2+0x120/0x460
[69707.930711]  ? init_wait_entry+0x30/0x30
[69707.930715]  kthread+0x1f9/0x230
[69707.930719]  ? jbd2_journal_update_sb_log_tail+0x180/0x180
[69707.930725]  ? kthreadd+0x380/0x380
[69707.930728]  ret_from_fork+0x30/0x40
[69707.930733]  ? kthreadd+0x380/0x380
[69707.930736]  ret_from_fork_asm+0x11/0x20
[69707.930743]  </TASK>
[69707.930744] Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings
[69794.689696] systemd[1]: systemd-journald.service: Main process exited, code=dumped, status=6/ABRT
[69794.689877] systemd[1]: systemd-journald.service: Failed with result 'watchdog'.
[69794.691811] systemd[1]: systemd-journald.service: Consumed 5.443s CPU time.
[69794.693515] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1.
[69794.693939] systemd[1]: Stopped Journal Service.
[69794.693993] systemd[1]: systemd-journald.service: Consumed 5.443s CPU time.
[69794.813640] systemd[1]: Starting Journal Service...
[69846.623844] systemd[1]: Started Journal Service.

-- 
Ammar Faizi

Re:[## 385523 ##] Disk I/O di Neo Lite kembali bermasalah

[Ammar Rafli]

[-- Attachment #1: Type: text/plain, Size: 24490 bytes --]

Hi Bapak Ammar
 
 Selamat Malam
 
Menanggapi email yang disampaikan, Baik, terkait hal tersebut akan kami koordinasikan dahulu ya Pak. 
 Mohon kesediannya untuk menunggu informasi update dari sisi kami.
 
 Sekian informasi yang disampaikan.

Terima kasih atas perhatian dan kerjasamanya.

Penilaian Anda sangat membantu perbaikan dan evaluasi pelayanan kami. Ketika tiket pengaduan Anda kami tutup akan ada otomatis link survei kepuasan pelanggan di mailbox Anda. Mohon untuk memberikan penilaian Anda.

Regards
 
 Ammar Rafli Arkan
Product Operation Support
-----------------------------------------------------------
PT. Biznet Gio Nusantara
MidPlaza 1, 7th Floor. Jl. Jend. Sudirman Kav 10 - 11. Jakarta 10220 - Indonesia. 
-----------------------------------------------------------
Click here to submit your feature request.

The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. 
 
 
---- on Fri, 12 Jul 2024 20:40:15 +0700 "Ammar Faizi"<ammarfaizi2@gnuweeb.org> wrote ---- 
 On Fri, Jul 12, 2024 at 01:31:30PM +0000, Yusuf Hadiwinata wrote: 
> Thnks you 
> 
> Mau memastikan, apakah sebelum terjadi Blocking Thread ada OOM 
 
Iya ada, saya juga stress test memory, sempat hit OOM killer juga. 
 
Untuk testing memory saya pakai simple PHP script yang memakan memory 
sekitar 11 GB seperti ini: 
 
<?php 
$nr = 1024*1024*1024*5; 
$a = str_repeat("a", $nr); 
echo "ar\n"; 
$b = str_repeat("b", $nr); 
echo "br\n"; 
 
while (1) { 
 
 $tmp = ""; 
 for ($i = 0; $i < $nr; $i++) { 
 $tmp .= $a[$i]; 
 } 
 echo "A done!\n"; 
 
 $tmp .= ""; 
 for ($i = 0; $i < $nr; $i++) { 
 $tmp .= $b[$i]; 
 } 
 echo "B done!\n"; 
 
 sleep(5); 
} 
 
---- 
dmesg: 
 
[15832.976830] Out of memory: Killed process 7142 (php) total-vm:12649176kB, anon-rss:4339492kB, file-rss:3200kB, shmem-rss:0kB, UID:0 pgtables:24720kB oom_score_adj:0 
[15836.351730] oom_reaper: reaped process 7142 (php), now anon-rss:636kB, file-rss:796kB, shmem-rss:0kB 
[22471.722388] ct invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0 
[22471.722420] CPU: 7 PID: 8149 Comm: ct Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8 5aa4ec4a5a8034d921df04c799edf17a90d1ec90 
[22471.722434] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.1-0-g3208b098f51a-prebuilt.qemu.org 04/01/2014 
[22471.722446] Call Trace: 
[22471.722508] <TASK> 
[22471.722543] dump_header+0x50/0x170 
[22471.722612] oom_kill_process+0x1c0/0x5b0 
[22471.722624] out_of_memory+0x7af/0xdf0 
[22471.722628] __alloc_pages_slowpath+0xba1/0x1210 
[22471.722648] __alloc_pages+0x23d/0x330 
[22471.722657] alloc_pages_mpol+0xf4/0x1a0 
[22471.722670] filemap_alloc_folio+0xa3/0x1d0 
[22471.722690] __filemap_get_folio+0x195/0x480 
[22471.722694] filemap_fault+0x212/0x830 
[22471.722700] ? __slab_free+0x5a/0x1a0 
[22471.722704] __do_fault+0x3b/0x120 
[22471.722714] do_pte_missing+0x16b/0x1060 
[22471.722716] ? skb_release_data+0x1c6/0x2b0 
[22471.722740] handle_mm_fault+0x93c/0x1380 
[22471.722743] do_user_addr_fault+0x202/0xa10 
[22471.722761] exc_page_fault+0x53/0xc0 
[22471.722788] asm_exc_page_fault+0x22/0x30 
[22471.722802] RIP: 0033:0x557bbad655bb 
[22471.722814] Code: Unable to access opcode bytes at 0x557bbad65591. 
[22471.722815] RSP: 002b:00007fbd6e9ffc48 EFLAGS: 00010206 
[22471.722818] RAX: 0000000000000012 RBX: 00007fbd6ea00640 RCX: 0000000000000001 
[22471.722820] RDX: 0000000000000000 RSI: 00007fbd70000b70 RDI: 00007fbd6e9ff8f0 
[22471.722821] RBP: 00007fbd6e9ffe48 R08: 0000000000000000 R09: 00007fbd6e9ffd27 
[22471.722823] R10: 0000000000000000 R11: 0000000000000293 R12: 00007fbd6ea00640 
[22471.722824] R13: 0000000000000000 R14: 00007fbd774947d0 R15: 00007ffdf6585af0 
[22471.722826] </TASK> 
[22471.722827] Mem-Info: 
[22471.722835] active_anon:1046703 inactive_anon:469261 isolated_anon:96 
 active_file:0 inactive_file:41 isolated_file:0 
 unevictable:1579777 dirty:3 writeback:0 
 slab_reclaimable:6224 slab_unreclaimable:70607 
 mapped:3882 shmem:18 pagetables:11208 
 sec_pagetables:0 bounce:0 
 kernel_misc_reclaimable:0 
 free:66954 free_pcp:0 free_cma:0 
[22471.722840] Node 0 active_anon:4186812kB inactive_anon:1877044kB active_file:0kB inactive_file:164kB unevictable:6319108kB isolated(anon):384kB isolated(file):0kB mapped:15528kB dirty:12kB writeback:0kB shmem:72kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:0kB writeback_tmp:0kB kernel_stack:5328kB pagetables:44832kB sec_pagetables:0kB all_unreclaimable? no 
[22471.722867] Node 0 DMA free:15360kB boost:0kB min:60kB low:72kB high:84kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB 
[22471.722878] lowmem_reserve[]: 0 2856 15884 15884 15884 
[22471.722882] Node 0 DMA32 free:64128kB boost:0kB min:12144kB low:15180kB high:18216kB reserved_highatomic:0KB active_anon:472036kB inactive_anon:161520kB active_file:48kB inactive_file:0kB unevictable:2115584kB writepending:0kB present:3129192kB managed:2989864kB mlocked:2115584kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB 
[22471.722889] lowmem_reserve[]: 0 0 13027 13027 13027 
[22471.722892] Node 0 Normal free:188328kB boost:0kB min:55372kB low:69212kB high:83052kB reserved_highatomic:133120KB active_anon:3714332kB inactive_anon:1716504kB active_file:300kB inactive_file:592kB unevictable:4203524kB writepending:12kB present:13631488kB managed:13339936kB mlocked:4203524kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB 
[22471.722899] lowmem_reserve[]: 0 0 0 0 0 
[22471.722903] Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15360kB 
[22471.722964] Node 0 DMA32: 123*4kB (UME) 46*8kB (UME) 48*16kB (UE) 20*32kB (UE) 27*64kB (UE) 29*128kB (UME) 41*256kB (UME) 18*512kB (UME) 6*1024kB (UME) 1*2048kB (M) 7*4096kB (UM) = 64284kB 
[22471.722977] Node 0 Normal: 118*4kB (UME) 111*8kB (UME) 135*16kB (UM) 25*32kB (UM) 2330*64kB (UM) 272*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 188256kB 
[22471.722994] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB 
[22471.722996] 22547 total pagecache pages 
[22471.722997] 20249 pages in swap cache 
[22471.722998] Free swap = 0kB 
[22471.722998] Total swap = 8388604kB 
[22471.723002] 4194168 pages RAM 
[22471.723003] 0 pages HighMem/MovableOnly 
[22471.723004] 107878 pages reserved 
[22471.723007] 0 pages hwpoisoned 
[22471.723008] Tasks state (memory values in pages): 
[22471.723012] [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name 
[22471.723021] [ 451] 0 451 9889 992 128 864 0 102400 192 -250 systemd-journal 
[22471.723026] [ 485] 0 485 72338 6880 4640 2240 0 110592 0 -1000 multipathd 
[22471.723034] [ 489] 0 489 6527 858 96 762 0 77824 416 -1000 systemd-udevd 
[22471.723037] [ 651] 103 651 22341 1088 96 992 0 73728 96 0 systemd-timesyn 
[22471.723040] [ 694] 100 694 4032 928 128 800 0 65536 128 0 systemd-network 
[22471.723043] [ 697] 101 697 6385 1082 218 864 0 81920 864 0 systemd-resolve 
[22471.723045] [ 734] 0 734 1822 768 64 704 0 45056 0 0 cron 
[22471.723048] [ 736] 102 736 2189 1184 224 960 0 49152 32 -900 dbus-daemon 
[22471.723050] [ 749] 0 749 20705 832 0 832 0 61440 96 0 irqbalance 
[22471.723055] [ 751] 0 751 8296 862 30 832 0 98304 2144 0 networkd-dispat 
[22471.723058] [ 752] 0 752 58626 992 96 896 0 94208 64 0 polkitd 
[22471.723060] [ 755] 0 755 20013 768 32 736 0 57344 0 0 qemu-ga 
[22471.723063] [ 761] 0 761 588565 1194 650 544 0 450560 2176 -900 snapd 
[22471.723065] [ 764] 0 764 3883 992 128 864 0 65536 128 0 systemd-logind 
[22471.723067] [ 766] 0 766 98156 1216 192 1024 0 139264 416 0 udisksd 
[22471.723083] [ 772] 0 772 1555 512 32 480 0 45056 0 0 agetty 
[22471.723085] [ 782] 0 782 1544 480 32 448 0 45056 0 0 agetty 
[22471.723090] [ 806] 0 806 27526 896 32 864 0 122880 1984 0 unattended-upgr 
[22471.723093] [ 826] 0 826 61061 1024 96 928 0 106496 352 0 ModemManager 
[22471.723095] [ 827] 0 827 3859 1056 96 960 0 61440 288 -1000 sshd 
[22471.723098] [ 1098] 0 1098 4278 768 64 704 0 73728 352 0 systemd 
[22471.723103] [ 1099] 0 1099 42339 430 78 352 0 90112 864 0 (sd-pam) 
[22471.723106] [ 1680] 0 1680 74007 1056 128 928 0 155648 672 0 packagekitd 
[22471.723108] [ 2138] 104 2138 55601 1184 160 1024 0 90112 192 0 rsyslogd 
[22471.723116] [ 2212] 0 2212 2033 576 32 544 0 40960 256 0 screen 
[22471.723119] [ 2213] 0 2213 2040 608 0 608 0 45056 160 0 bash 
[22471.723123] [ 3567] 0 3567 1996 608 64 544 0 40960 160 0 screen 
[22471.723132] [ 3568] 0 3568 2041 896 192 704 0 53248 0 0 bash 
[22471.723139] [ 4804] 0 4804 1961 512 0 512 0 40960 224 0 screen 
[22471.723144] [ 4805] 0 4805 2041 576 32 544 0 49152 160 0 bash 
[22471.723147] [ 5361] 0 5361 1961 480 0 480 0 49152 192 0 screen 
[22471.723149] [ 5362] 0 5362 2041 576 64 512 0 49152 160 0 bash 
[22471.723151] [ 5385] 0 5385 2241 1280 64 1216 0 57344 0 0 arpflood 
[22471.723166] [ 5389] 0 5389 2241 1248 64 1184 0 53248 0 0 arpflood 
[22471.723168] [ 5921] 0 5921 1928 512 32 480 0 45056 160 0 screen 
[22471.723174] [ 5922] 0 5922 2041 544 32 512 0 53248 160 0 bash 
[22471.723182] [ 5928] 0 5928 2241 1216 64 1152 0 57344 0 0 arpflood 
[22471.723187] [ 5951] 0 5951 1961 480 0 480 0 49152 224 0 screen 
[22471.723201] [ 5952] 0 5952 2041 640 96 544 0 49152 128 0 bash 
[22471.723204] [ 5963] 0 5963 1993 544 96 448 0 45056 160 0 screen 
[22471.723206] [ 5964] 0 5964 2041 608 32 576 0 53248 160 0 bash 
[22471.723208] [ 6064] 0 6064 1961 448 0 448 0 49152 224 0 screen 
[22471.723214] [ 6065] 0 6065 2121 832 128 704 0 53248 160 0 bash 
[22471.723219] [ 6900] 0 6900 1961 512 32 480 0 45056 160 0 screen 
[22471.723221] [ 6901] 0 6901 2087 896 160 736 0 57344 32 0 bash 
[22471.723223] [ 6916] 0 6916 1961 512 32 480 0 40960 160 0 screen 
[22471.723225] [ 6917] 0 6917 2084 864 192 672 0 57344 96 0 bash 
[22471.723228] [ 8141] 0 8141 49862 320 0 320 0 172032 0 0 ct 
[22471.723231] [ 9210] 0 9210 1573558 1573088 1572768 320 0 12648448 0 0 memt 
[22471.723248] [ 9246] 0 9246 1589430 250395 249659 736 0 11841536 1214465 0 php 
[22471.723252] [ 9631] 0 9631 24532 864 96 768 0 94208 640 0 curl 
[22471.723255] [ 9638] 0 9638 24532 832 96 736 0 94208 640 0 curl 
[22471.723257] [ 9663] 0 9663 24532 832 128 704 0 90112 640 0 curl 
[22471.723259] [ 9665] 0 9665 53998 640 64 576 0 106496 160 0 fio 
[22471.723261] [ 9667] 0 9667 1091336 756214 755926 288 0 8511488 297120 0 fio 
[22471.723264] [ 9668] 0 9668 1091337 490038 489782 256 0 8511488 563264 0 fio 
[22471.723269] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/user.slice/user-0.slice/session-3.scope,task=memt,pid=9210,uid=0 
[22471.723310] Out of memory: Killed process 9210 (memt) total-vm:6294232kB, anon-rss:6291072kB, file-rss:1280kB, shmem-rss:0kB, UID:0 pgtables:12352kB oom_score_adj:0 
[61796.908278] INFO: task jbd2/sda1-8:377 blocked for more than 30 seconds. 
[61796.909531] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8 
[61796.910820] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 
[61796.912369] task:jbd2/sda1-8 state:D stack:0 pid:377 tgid:377 ppid:2 flags:0x00004000 
[61796.912389] Call Trace: 
[61796.912399] <TASK> 
[61796.912426] __schedule+0x5e7/0x14a0 
[61796.912487] ? submit_bio_noacct_nocheck+0x2cc/0x360 
[61796.912508] schedule+0x8e/0x160 
[61796.912512] bit_wait_io+0x45/0xa0 
[61796.912518] __wait_on_bit+0xe1/0x170 
[61796.912521] ? bit_wait_io_timeout+0xc0/0xc0 
[61796.912524] jbd2_journal_commit_transaction+0x191b/0x2320 
[61796.912547] ? wake_up_bit+0xd0/0xd0 
[61796.912564] kjournald2+0x120/0x460 
[61796.912577] ? init_wait_entry+0x30/0x30 
[61796.912580] kthread+0x1f9/0x230 
[61796.912594] ? jbd2_journal_update_sb_log_tail+0x180/0x180 
[61796.912599] ? kthreadd+0x380/0x380 
[61796.912601] ret_from_fork+0x30/0x40 
[61796.912620] ? kthreadd+0x380/0x380 
[61796.912622] ret_from_fork_asm+0x11/0x20 
[61796.912665] </TASK> 
[61875.585039] show_signal_msg: 8 callbacks suppressed 
[61875.585050] cq[14120]: segfault at 7f611add1000 ip 000055f05765c4e8 sp 00007f61193ffdb0 error 4 in cq[55f05765c000+1000] likely on CPU 2 (core 2, socket 0) 
[61875.585106] Code: 89 d1 48 d3 e6 49 09 34 c4 e8 64 fc ff ff 4c 89 e2 be 80 00 00 00 48 c1 eb 02 48 89 c7 e8 10 fc ff ff 89 d9 48 89 ef 0f 57 c9 <0f> 28 07 0f 58 c8 0f 29 0f 48 83 c7 10 e2 f1 eb ef 48 8b 84 24 88 
[61887.147744] INFO: task jbd2/sda1-8:377 blocked for more than 30 seconds. 
[61887.148999] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8 
[61887.150213] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 
[61887.151554] task:jbd2/sda1-8 state:D stack:0 pid:377 tgid:377 ppid:2 flags:0x00004000 
[61887.151561] Call Trace: 
[61887.151563] <TASK> 
[61887.151576] __schedule+0x5e7/0x14a0 
[61887.151586] ? submit_bio_noacct_nocheck+0x2cc/0x360 
[61887.151591] schedule+0x8e/0x160 
[61887.151594] bit_wait_io+0x45/0xa0 
[61887.151597] __wait_on_bit+0xe1/0x170 
[61887.151600] ? bit_wait_io_timeout+0xc0/0xc0 
[61887.151604] jbd2_journal_commit_transaction+0x191b/0x2320 
[61887.151609] ? wake_up_bit+0xd0/0xd0 
[61887.151613] kjournald2+0x120/0x460 
[61887.151618] ? init_wait_entry+0x30/0x30 
[61887.151621] kthread+0x1f9/0x230 
[61887.151624] ? jbd2_journal_update_sb_log_tail+0x180/0x180 
[61887.151628] ? kthreadd+0x380/0x380 
[61887.151630] ret_from_fork+0x30/0x40 
[61887.151634] ? kthreadd+0x380/0x380 
[61887.151636] ret_from_fork_asm+0x11/0x20 
[61887.151641] </TASK> 
[61887.151644] INFO: task journal-offline:14113 blocked for more than 30 seconds. 
[61887.153004] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8 
[61887.154263] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 
[61887.157148] task:journal-offline state:D stack:0 pid:14113 tgid:451 ppid:1 flags:0x00004002 
[61887.157158] Call Trace: 
[61887.157162] <TASK> 
[61887.157170] __schedule+0x5e7/0x14a0 
[61887.157185] ? wait_for_common_io+0x171/0x190 
[61887.157191] ? wbc_detach_inode+0x103/0x230 
[61887.157213] ? prepare_to_wait_event+0x125/0x140 
[61887.157218] schedule+0x8e/0x160 
[61887.157221] jbd2_log_wait_commit+0x1b7/0x200 
[61887.157230] ? init_wait_entry+0x30/0x30 
[61887.157232] ext4_sync_file+0x47c/0x590 
[61887.157254] __x64_sys_fsync+0xb0/0x100 
[61887.157268] do_syscall_64+0x50/0x110 
[61887.157293] entry_SYSCALL_64_after_hwframe+0x24/0x2c 
[61887.157311] RIP: 0033:0x7f923d11b8ab 
[61887.157319] RSP: 002b:00007f923b7ffc50 EFLAGS: 00000293 ORIG_RAX: 000000000000004a 
[61887.157342] RAX: ffffffffffffffda RBX: 000055c68b01c370 RCX: 00007f923d11b8ab 
[61887.157344] RDX: 0000000000000000 RSI: 00007f923d64ea34 RDI: 0000000000000019 
[61887.157346] RBP: 00007f923d6502c0 R08: 0000000000000000 R09: 00007ffc269604cf 
[61887.157348] R10: 0000000000000a34 R11: 0000000000000293 R12: 0000000000000002 
[61887.157349] R13: 0000000000000002 R14: 00007f923d0947d0 R15: 00007ffc26960510 
[61887.157360] </TASK> 
[63099.559295] traps: cq[14344] trap invalid opcode ip:564f9853d3aa sp:7f779bbffc20 error:0 
[63099.559294] traps: cq[14343] trap invalid opcode ip:564f9853d3aa sp:7f779c5ffc20 error:0 
[63099.559348] in cq[564f9853d000+1000] 
[63099.559348] in cq[564f9853d000+1000] 
 
 
[63421.219463] INFO: task jbd2/sda1-8:377 blocked for more than 30 seconds. 
[63421.220972] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8 
[63421.222346] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 
[63421.223707] task:jbd2/sda1-8 state:D stack:0 pid:377 tgid:377 ppid:2 flags:0x00004000 
[63421.223728] Call Trace: 
[63421.223749] <TASK> 
[63421.223777] __schedule+0x5e7/0x14a0 
[63421.223860] ? submit_bio_noacct_nocheck+0x2cc/0x360 
[63421.223882] schedule+0x8e/0x160 
[63421.223886] bit_wait_io+0x45/0xa0 
[63421.223892] __wait_on_bit+0xe1/0x170 
[63421.223895] ? bit_wait_io_timeout+0xc0/0xc0 
[63421.223899] jbd2_journal_commit_transaction+0x191b/0x2320 
[63421.223920] ? wake_up_bit+0xd0/0xd0 
[63421.223938] kjournald2+0x120/0x460 
[63421.223951] ? init_wait_entry+0x30/0x30 
[63421.223955] kthread+0x1f9/0x230 
[63421.223969] ? jbd2_journal_update_sb_log_tail+0x180/0x180 
[63421.223974] ? kthreadd+0x380/0x380 
[63421.223976] ret_from_fork+0x30/0x40 
[63421.224007] ? kthreadd+0x380/0x380 
[63421.224009] ret_from_fork_asm+0x11/0x20 
[63421.224024] </TASK> 
[68384.390473] INFO: task jbd2/sda1-8:377 blocked for more than 30 seconds. 
[68384.392485] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8 
[68384.394706] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 
[68384.396367] task:jbd2/sda1-8 state:D stack:0 pid:377 tgid:377 ppid:2 flags:0x00004000 
[68384.396381] Call Trace: 
[68384.396393] <TASK> 
[68384.396416] __schedule+0x5e7/0x14a0 
[68384.396476] ? submit_bio_noacct_nocheck+0x2cc/0x360 
[68384.396497] schedule+0x8e/0x160 
[68384.396501] bit_wait_io+0x45/0xa0 
[68384.396507] __wait_on_bit+0xe1/0x170 
[68384.396510] ? bit_wait_io_timeout+0xc0/0xc0 
[68384.396529] jbd2_journal_commit_transaction+0x191b/0x2320 
[68384.396551] ? wake_up_bit+0xd0/0xd0 
[68384.396568] kjournald2+0x120/0x460 
[68384.396581] ? init_wait_entry+0x30/0x30 
[68384.396584] kthread+0x1f9/0x230 
[68384.396600] ? jbd2_journal_update_sb_log_tail+0x180/0x180 
[68384.396604] ? kthreadd+0x380/0x380 
[68384.396606] ret_from_fork+0x30/0x40 
[68384.396626] ? kthreadd+0x380/0x380 
[68384.396628] ret_from_fork_asm+0x11/0x20 
[68384.396644] </TASK> 
[68655.118907] INFO: task jbd2/sda1-8:377 blocked for more than 30 seconds. 
[68655.124237] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8 
[68655.125785] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 
[68655.128624] task:jbd2/sda1-8 state:D stack:0 pid:377 tgid:377 ppid:2 flags:0x00004000 
[68655.128669] Call Trace: 
[68655.128683] <TASK> 
[68655.128718] __schedule+0x5e7/0x14a0 
[68655.128801] ? submit_bio_noacct_nocheck+0x2cc/0x360 
[68655.128829] schedule+0x8e/0x160 
[68655.128833] bit_wait_io+0x45/0xa0 
[68655.128908] __wait_on_bit+0xe1/0x170 
[68655.128913] ? bit_wait_io_timeout+0xc0/0xc0 
[68655.128918] jbd2_journal_commit_transaction+0x191b/0x2320 
[68655.128997] ? wake_up_bit+0xd0/0xd0 
[68655.129025] kjournald2+0x120/0x460 
[68655.129048] ? init_wait_entry+0x30/0x30 
[68655.129053] kthread+0x1f9/0x230 
[68655.129080] ? jbd2_journal_update_sb_log_tail+0x180/0x180 
[68655.129086] ? kthreadd+0x380/0x380 
[68655.129089] ret_from_fork+0x30/0x40 
[68655.129124] ? kthreadd+0x380/0x380 
[68655.129127] ret_from_fork_asm+0x11/0x20 
[68655.129156] </TASK> 
[68895.747345] INFO: task jbd2/sda1-8:377 blocked for more than 30 seconds. 
[68895.748831] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8 
[68895.750303] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 
[68895.752488] task:jbd2/sda1-8 state:D stack:0 pid:377 tgid:377 ppid:2 flags:0x00004000 
[68895.752513] Call Trace: 
[68895.752527] <TASK> 
[68895.752590] __schedule+0x5e7/0x14a0 
[68895.752671] ? submit_bio_noacct_nocheck+0x2cc/0x360 
[68895.752699] schedule+0x8e/0x160 
[68895.752703] bit_wait_io+0x45/0xa0 
[68895.752712] __wait_on_bit+0xe1/0x170 
[68895.752716] ? bit_wait_io_timeout+0xc0/0xc0 
[68895.752739] jbd2_journal_commit_transaction+0x191b/0x2320 
[68895.752768] ? wake_up_bit+0xd0/0xd0 
[68895.752792] kjournald2+0x120/0x460 
[68895.752809] ? init_wait_entry+0x30/0x30 
[68895.752813] kthread+0x1f9/0x230 
[68895.752833] ? jbd2_journal_update_sb_log_tail+0x180/0x180 
[68895.752839] ? kthreadd+0x380/0x380 
[68895.752842] ret_from_fork+0x30/0x40 
[68895.752869] ? kthreadd+0x380/0x380 
[68895.752872] ret_from_fork_asm+0x11/0x20 
[68895.752892] </TASK> 
[69677.822593] INFO: task jbd2/sda1-8:377 blocked for more than 30 seconds. 
[69677.824222] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8 
[69677.826159] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 
[69677.828000] task:jbd2/sda1-8 state:D stack:0 pid:377 tgid:377 ppid:2 flags:0x00004000 
[69677.828026] Call Trace: 
[69677.828040] <TASK> 
[69677.828076] __schedule+0x5e7/0x14a0 
[69677.828159] ? submit_bio_noacct_nocheck+0x2cc/0x360 
[69677.828187] schedule+0x8e/0x160 
[69677.828191] bit_wait_io+0x45/0xa0 
[69677.828199] __wait_on_bit+0xe1/0x170 
[69677.828204] ? bit_wait_io_timeout+0xc0/0xc0 
[69677.828208] jbd2_journal_commit_transaction+0x191b/0x2320 
[69677.828239] ? wake_up_bit+0xd0/0xd0 
[69677.828263] kjournald2+0x120/0x460 
[69677.828280] ? init_wait_entry+0x30/0x30 
[69677.828299] kthread+0x1f9/0x230 
[69677.828321] ? jbd2_journal_update_sb_log_tail+0x180/0x180 
[69677.828327] ? kthreadd+0x380/0x380 
[69677.828330] ret_from_fork+0x30/0x40 
[69677.828357] ? kthreadd+0x380/0x380 
[69677.828360] ret_from_fork_asm+0x11/0x20 
[69677.828381] </TASK> 
[69677.828385] INFO: task journal-offline:18592 blocked for more than 30 seconds. 
[69677.831042] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8 
[69677.833149] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 
[69677.835422] task:journal-offline state:D stack:0 pid:18592 tgid:451 ppid:1 flags:0x00004002 
[69677.835430] Call Trace: 
[69677.835432] <TASK> 
[69677.835436] __schedule+0x5e7/0x14a0 
[69677.835444] ? wait_for_common_io+0x171/0x190 
[69677.835449] ? prepare_to_wait_event+0x125/0x140 
[69677.835454] schedule+0x8e/0x160 
[69677.835457] jbd2_log_wait_commit+0x1b7/0x200 
[69677.835464] ? init_wait_entry+0x30/0x30 
[69677.835468] ext4_sync_file+0x47c/0x590 
[69677.835513] __x64_sys_fsync+0xb0/0x100 
[69677.835542] do_syscall_64+0x50/0x110 
[69677.835569] entry_SYSCALL_64_after_hwframe+0x24/0x2c 
[69677.835593] RIP: 0033:0x7f923d11b8ab 
[69677.835602] RSP: 002b:00007f923b7ffc50 EFLAGS: 00000293 ORIG_RAX: 000000000000004a 
[69677.835606] RAX: ffffffffffffffda RBX: 000055c68b01c370 RCX: 00007f923d11b8ab 
[69677.835609] RDX: 0000000000000000 RSI: 00007f923d64ea34 RDI: 0000000000000019 
[69677.835632] RBP: 00007f923d6502c0 R08: 0000000000000000 R09: 00007ffc269604cf 
[69677.835634] R10: 0000000000000a34 R11: 0000000000000293 R12: 0000000000000002 
[69677.835640] R13: 0000000000000002 R14: 00007f923d0947d0 R15: 00007ffc26960510 
[69677.835651] </TASK> 
[69707.922952] INFO: task jbd2/sda1-8:377 blocked for more than 60 seconds. 
[69707.926000] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8 
[69707.928249] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 
[69707.930623] task:jbd2/sda1-8 state:D stack:0 pid:377 tgid:377 ppid:2 flags:0x00004000 
[69707.930632] Call Trace: 
[69707.930640] <TASK> 
[69707.930655] __schedule+0x5e7/0x14a0 
[69707.930666] ? submit_bio_noacct_nocheck+0x2cc/0x360 
[69707.930673] schedule+0x8e/0x160 
[69707.930677] bit_wait_io+0x45/0xa0 
[69707.930682] __wait_on_bit+0xe1/0x170 
[69707.930686] ? bit_wait_io_timeout+0xc0/0xc0 
[69707.930691] jbd2_journal_commit_transaction+0x191b/0x2320 
[69707.930698] ? wake_up_bit+0xd0/0xd0 
[69707.930705] kjournald2+0x120/0x460 
[69707.930711] ? init_wait_entry+0x30/0x30 
[69707.930715] kthread+0x1f9/0x230 
[69707.930719] ? jbd2_journal_update_sb_log_tail+0x180/0x180 
[69707.930725] ? kthreadd+0x380/0x380 
[69707.930728] ret_from_fork+0x30/0x40 
[69707.930733] ? kthreadd+0x380/0x380 
[69707.930736] ret_from_fork_asm+0x11/0x20 
[69707.930743] </TASK> 
[69707.930744] Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings 
[69794.689696] systemd[1]: systemd-journald.service: Main process exited, code=dumped, status=6/ABRT 
[69794.689877] systemd[1]: systemd-journald.service: Failed with result 'watchdog'. 
[69794.691811] systemd[1]: systemd-journald.service: Consumed 5.443s CPU time. 
[69794.693515] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. 
[69794.693939] systemd[1]: Stopped Journal Service. 
[69794.693993] systemd[1]: systemd-journald.service: Consumed 5.443s CPU time. 
[69794.813640] systemd[1]: Starting Journal Service... 
[69846.623844] systemd[1]: Started Journal Service. 
 
-- 
Ammar Faizi 
 
 

[-- Attachment #2: Type: text/html, Size: 32164 bytes --]

Re: [## 385523 ##] Disk I/O di Neo Lite kembali bermasalah

[Yusuf Hadiwinata]

[-- Attachment #1: Type: text/plain, Size: 3922 bytes --]

Dear Pak Ammar dan Team

Setelah berkodinasi dengan Principal dan melalukan Analisa kami dapat memberikan kesimpulan sebagai berikut:


  1.
Untuk concern mengenai "task:jbd2/sda1-8 state:D" dan  "jbd2_journal_commit_transaction" kami bisa sampaikan tidak terkait dengan issue sebelumnya pada VM yang sama (jika ada).
  2.
Masih untuk concern mengenai "task:jbd2/sda1-8 state:D" dan case tersebut hari ini, kami cukup confident ini tidak related dengan issue customer lain.
  3.
Kami juga sempat mengecek beberap VM Service Level Indicator kami baik yang running di host yang sama ataupun berbeda dan di cluster yang sama, tidak di temukan Issue sejenis.
  4.
Setelah menghubungi Principal dan Berkordinasi, tidak di temukan issue di Cluster/ Host tempat VM dengan  IP 103.175.218.159, berikut beberapa area yang kami cek:
     *
Network: CRC Error, SignalDrop, PacketDrop, bufferOverrun, DiscardPacket, KernelQueue, RecieveRingBuffer, TransmitRingBuffer, TXQueue, Interface statistic pada host dan Switch
     *
Host: Log pada system, kernel dan services yang related serta CPU/RAM/etc Utilization
     *
Storage: Semua hal yang berkaitan dengan Status Placement Group, status Disk, Status Cluster dll
  5.
Analisa dari kami dan Principal adalah sebagai berikut mengenai "task jbd2/sda1-8:377 blocked for more than 30 seconds." pada jam 17:26
     *
Terjadi process hanging/deadlocking/blocked di VM tersebut karena berbarengan dengan process Burn-Test / Stress-Test CPU dan RAM
     *
Suspect kami ada pada CFQ scheduler dan RT scheduler class di kondisi CPU dan RAM yang utilisasi nya sangat tinggi, sehingga "jbd2_journal_commit_transaction" gagal dilakukan, dari Call trace nya kami jg belum menemukan warn_slowpath_*
     *
Menurut Principal kami hal ini sering terjadi pada sistem dengan kondisi Heavy Load, mohon untuk menstop Stress-Test CPU dan RAM pada saat melakukan FIO
  6.
Saran dari kami:
     *
Mohon mengaktifkan Kernel Crash Dump dan sharing vmcore nya kepada kami, sehingga pada saat terjadi lagi kami akan berusaha melakukan analisai berdasarkan vmcore tersebut.
     *
Mohon sertakan juga output sosreport agar dapat digunakan sebagai cross reference issue yang ada
     *
Jika berkenan, mohon untuk melakukan Update kernel ke latest version just in case concern pada email ini related dengan issue pada email ini.
  7.
Sebagai informasi tambahan, pada kondisi normal load sebagai mestinya (Running apps dll), seharusnya hal ini jarang terjadi, jika memang terjadi maka kemungkinan besar ada di capacity Planning, terutama pada IOPS. Untuk pilihan IOPS 5k - 15k ada pada product Neo High Performance Storage di layanan Neo Virtual Compute
  8.
Kami masih terus menggali kemungkinan issue lain yang berhubungan dengan issue yang dihadapi hari ini, dan jika ada temuan lebih lanjut kami akan update.
  9.
Di satu sisi sebagaimana yang sempat saya sampaikan, kami menemukan beberapa point suspect pada cluster lain yang akan kami investigasi lebih jauh, namun dapat dipastikan suspect issue pada cluster tersebut tidak berhugungan dengan concern yang di sampaikan  pada email ini.
  10.
Kami juga berusaha mempelajari lebih line pada Call Trace satu persatu untuk mencari kemungkinan ada nya  bugs namun sejauh ini belum menemukan yang benar2 relate.
  11.
Kami terbuka jika ada masukan/mengetahui sesuatu yang mungkin Pak Ammar dan Team ingin sampaikan, jika ada clue/masukan jangan sungkan untuk disampaikan di email ini ya.


Terim Kasih


The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments.

[-- Attachment #2: Type: text/html, Size: 9737 bytes --]