* Disk I/O di Neo Lite kembali bermasalah @ 2024-07-12 11:12 Ammar Faizi 2024-07-12 11:27 ` Re:[## 385523 ##] " Ammar Rafli 0 siblings, 1 reply; 22+ messages in thread From: Ammar Faizi @ 2024-07-12 11:12 UTC (permalink / raw) To: Biznet Gio Support Cc: Yusuf Hadiwinata, GNU/Weeb Mailing List, Alviro Iskandar Setiawan, Paber Panjaitan Melanjutkan masalah IOPS di Neo Lite. IOPS di Neo Lite kembali bermasalah. Sudah sejak bulan Juni kemarin saya mengalami masalah disk I/O di Neo Lite. Gejala: process stuck di "D" state cukup lama ketika melakukan disk operation. Masalah ini kadang muncul, kadang hilang. Dengan kata lain, disk I/O yang stuck ini *tidak selalu* terjadi, tapi ketika terjadi dampaknya bisa membuat VM menjadi freeze seperti yang dikeluhkan beberapa user Biznet Gio lain juga (di Facebook). Kemudian sejak tanggal 8 Juli, saya melakukan stress test menggunakan fio 24 jam non stop untuk pembuktian masalah ini. Command yang saya jalankan: while true; do fio --name=io_uring --ioengine=io_uring --iodepth=1024 --filename=./test --rw=rw --bs=16K --numjobs=2 --size=10G --direct=1; done; IP VPS yang digunakan untuk testing: 103.175.218.159 Selama 4 hari berjalan normal, namun hari ini sekitar jam 17:26 terkendala process stuck di "D" state lebih dari 30 detik. [Jul12 17:26] INFO: task jbd2/sda1-8:377 blocked for more than 30 seconds. [ +0.001509] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8 [ +0.001374] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ +0.001361] task:jbd2/sda1-8 state:D stack:0 pid:377 tgid:377 ppid:2 flags:0x00004000 [ +0.000021] Call Trace: [ +0.000021] <TASK> [ +0.000028] __schedule+0x5e7/0x14a0 [ +0.000083] ? submit_bio_noacct_nocheck+0x2cc/0x360 [ +0.000022] schedule+0x8e/0x160 [ +0.000004] bit_wait_io+0x45/0xa0 [ +0.000006] __wait_on_bit+0xe1/0x170 [ +0.000003] ? bit_wait_io_timeout+0xc0/0xc0 [ +0.000004] jbd2_journal_commit_transaction+0x191b/0x2320 [ +0.000021] ? wake_up_bit+0xd0/0xd0 [ +0.000018] kjournald2+0x120/0x460 [ +0.000013] ? init_wait_entry+0x30/0x30 [ +0.000004] kthread+0x1f9/0x230 [ +0.000014] ? jbd2_journal_update_sb_log_tail+0x180/0x180 [ +0.000005] ? kthreadd+0x380/0x380 [ +0.000002] ret_from_fork+0x30/0x40 [ +0.000031] ? kthreadd+0x380/0x380 [ +0.000002] ret_from_fork_asm+0x11/0x20 [ +0.000015] </TASK> jbd2/sda1 sendiri adalah sebuah kernel thread dari module ext4. Mohon solusi dari pihak Biznet Gio terkait masalah ini. Masalah ini kadang muncul, kadang hilang. Terima kasih. -- Ammar Faizi ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re:[## 385523 ##] Disk I/O di Neo Lite kembali bermasalah 2024-07-12 11:12 Disk I/O di Neo Lite kembali bermasalah Ammar Faizi @ 2024-07-12 11:27 ` Ammar Rafli 2024-07-12 11:34 ` [## " Ammar Faizi 0 siblings, 1 reply; 22+ messages in thread From: Ammar Rafli @ 2024-07-12 11:27 UTC (permalink / raw) To: Ammar Faizi Cc: support, yusuf hadiwinata, gnu/weeb mailing list, alviro iskandar setiawan, paber panjaitan [-- Attachment #1: Type: text/plain, Size: 3741 bytes --] Hi Bapak Ammar Selamat Malam Menanggapi email yang disampaikan, Saat Bapak melakukan pengetesan FIO secara konstan selama beberapa hari, apakah Bapak dapat men-capture load average server secara berkala Pak? dan apakah Bapak mempunyai capture load average server sebelum kendala terjadi? Mohon informasinya pada ticket ini agar dapat kami koordinasikan dahulu. Terima kasih atas perhatian dan kerjasamanya. Penilaian Anda sangat membantu perbaikan dan evaluasi pelayanan kami. Ketika tiket pengaduan Anda kami tutup akan ada otomatis link survei kepuasan pelanggan di mailbox Anda. Mohon untuk memberikan penilaian Anda. Regards Ammar Rafli Arkan Product Operation Support ----------------------------------------------------------- PT. Biznet Gio Nusantara MidPlaza 1, 7th Floor. Jl. Jend. Sudirman Kav 10 - 11. Jakarta 10220 - Indonesia. ----------------------------------------------------------- Click here to submit your feature request. The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. ---- on Fri, 12 Jul 2024 18:12:17 +0700 "Ammar Faizi"<[email protected]> wrote ---- Melanjutkan masalah IOPS di Neo Lite. IOPS di Neo Lite kembali bermasalah. Sudah sejak bulan Juni kemarin saya mengalami masalah disk I/O di Neo Lite. Gejala: process stuck di "D" state cukup lama ketika melakukan disk operation. Masalah ini kadang muncul, kadang hilang. Dengan kata lain, disk I/O yang stuck ini *tidak selalu* terjadi, tapi ketika terjadi dampaknya bisa membuat VM menjadi freeze seperti yang dikeluhkan beberapa user Biznet Gio lain juga (di Facebook). Kemudian sejak tanggal 8 Juli, saya melakukan stress test menggunakan fio 24 jam non stop untuk pembuktian masalah ini. Command yang saya jalankan: while true; do fio --name=io_uring --ioengine=io_uring --iodepth=1024 --filename=./test --rw=rw --bs=16K --numjobs=2 --size=10G --direct=1; done; IP VPS yang digunakan untuk testing: 103.175.218.159 Selama 4 hari berjalan normal, namun hari ini sekitar jam 17:26 terkendala process stuck di "D" state lebih dari 30 detik. [Jul12 17:26] INFO: task jbd2/sda1-8:377 blocked for more than 30 seconds. [ +0.001509] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8 [ +0.001374] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ +0.001361] task:jbd2/sda1-8 state:D stack:0 pid:377 tgid:377 ppid:2 flags:0x00004000 [ +0.000021] Call Trace: [ +0.000021] <TASK> [ +0.000028] __schedule+0x5e7/0x14a0 [ +0.000083] ? submit_bio_noacct_nocheck+0x2cc/0x360 [ +0.000022] schedule+0x8e/0x160 [ +0.000004] bit_wait_io+0x45/0xa0 [ +0.000006] __wait_on_bit+0xe1/0x170 [ +0.000003] ? bit_wait_io_timeout+0xc0/0xc0 [ +0.000004] jbd2_journal_commit_transaction+0x191b/0x2320 [ +0.000021] ? wake_up_bit+0xd0/0xd0 [ +0.000018] kjournald2+0x120/0x460 [ +0.000013] ? init_wait_entry+0x30/0x30 [ +0.000004] kthread+0x1f9/0x230 [ +0.000014] ? jbd2_journal_update_sb_log_tail+0x180/0x180 [ +0.000005] ? kthreadd+0x380/0x380 [ +0.000002] ret_from_fork+0x30/0x40 [ +0.000031] ? kthreadd+0x380/0x380 [ +0.000002] ret_from_fork_asm+0x11/0x20 [ +0.000015] </TASK> jbd2/sda1 sendiri adalah sebuah kernel thread dari module ext4. Mohon solusi dari pihak Biznet Gio terkait masalah ini. Masalah ini kadang muncul, kadang hilang. Terima kasih. -- Ammar Faizi [-- Attachment #2: Type: text/html, Size: 5938 bytes --] ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [## 385523 ##] Disk I/O di Neo Lite kembali bermasalah 2024-07-12 11:27 ` Re:[## 385523 ##] " Ammar Rafli @ 2024-07-12 11:34 ` Ammar Faizi 2024-07-12 11:40 ` Ammar Rafli 0 siblings, 1 reply; 22+ messages in thread From: Ammar Faizi @ 2024-07-12 11:34 UTC (permalink / raw) To: Biznet Gio Support Cc: Yusuf Hadiwinata, Alviro Iskandar Setiawan, GNU/Weeb Mailing List, Paber Panjaitan [-- Attachment #1: Type: text/plain, Size: 748 bytes --] On Fri, Jul 12, 2024 at 6:28 PM Ammar Rafli wrote: > Hi Bapak Ammar > > Selamat Malam > > Menanggapi email yang disampaikan, Saat Bapak melakukan pengetesan FIO secara konstan selama beberapa hari, apakah Bapak dapat men-capture load average server secara berkala Pak? dan apakah Bapak mempunyai capture load average server sebelum kendala terjadi? Saya tidak mempunyai capture load average sebelum kendala terjadi, tapi load average saya pastikan di atas 16 (di bagian 1 min, 5 min, 15 min hampir semua rata full load), karena memang saya lakukan stress test 24 jam non stop. Berikut saya lampirkan screenshot load average. Side note: Saya sempat reboot sekali, jadi uptime di screenshot tidak sampai 4 hari. -- Ammar Faizi [-- Attachment #2: Screenshot from 2024-07-12 18-29-26.png --] [-- Type: image/png, Size: 34737 bytes --] ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re:[## 385523 ##] Disk I/O di Neo Lite kembali bermasalah 2024-07-12 11:34 ` [## " Ammar Faizi @ 2024-07-12 11:40 ` Ammar Rafli 2024-07-12 11:47 ` [## " Ammar Faizi 0 siblings, 1 reply; 22+ messages in thread From: Ammar Rafli @ 2024-07-12 11:40 UTC (permalink / raw) To: Ammar Faizi Cc: support, yusuf hadiwinata, alviro iskandar setiawan, gnu/weeb mailing list, paber panjaitan [-- Attachment #1: Type: text/plain, Size: 2169 bytes --] Hi Bapak Ammar Faizi Selamat Malam Menanggapi email yang disampaikan, Apakah saat ini Bapak dapat men-capture full command top -c Pak? agar kami dapat melihat semua parameter proses yang ada di server nya ya Pak untuk review lebih lanjut. Sekian informasi yang disampaiakn. Terima kasih atas perhatian dan kerjasamanya. Penilaian Anda sangat membantu perbaikan dan evaluasi pelayanan kami. Ketika tiket pengaduan Anda kami tutup akan ada otomatis link survei kepuasan pelanggan di mailbox Anda. Mohon untuk memberikan penilaian Anda. Regards Ammar Rafli Arkan Product Operation Support ----------------------------------------------------------- PT. Biznet Gio Nusantara MidPlaza 1, 7th Floor. Jl. Jend. Sudirman Kav 10 - 11. Jakarta 10220 - Indonesia. ----------------------------------------------------------- Click here to submit your feature request. The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. ---- on Fri, 12 Jul 2024 18:34:43 +0700 "Ammar Faizi"<[email protected]> wrote ---- On Fri, Jul 12, 2024 at 6:28 PM Ammar Rafli wrote: > Hi Bapak Ammar > > Selamat Malam > > Menanggapi email yang disampaikan, Saat Bapak melakukan pengetesan FIO secara konstan selama beberapa hari, apakah Bapak dapat men-capture load average server secara berkala Pak? dan apakah Bapak mempunyai capture load average server sebelum kendala terjadi? Saya tidak mempunyai capture load average sebelum kendala terjadi, tapi load average saya pastikan di atas 16 (di bagian 1 min, 5 min, 15 min hampir semua rata full load), karena memang saya lakukan stress test 24 jam non stop. Berikut saya lampirkan screenshot load average. Side note: Saya sempat reboot sekali, jadi uptime di screenshot tidak sampai 4 hari. -- Ammar Faizi [-- Attachment #2: Type: text/html, Size: 4016 bytes --] ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [## 385523 ##] Disk I/O di Neo Lite kembali bermasalah 2024-07-12 11:40 ` Ammar Rafli @ 2024-07-12 11:47 ` Ammar Faizi 2024-07-12 11:57 ` Ammar Rafli 0 siblings, 1 reply; 22+ messages in thread From: Ammar Faizi @ 2024-07-12 11:47 UTC (permalink / raw) To: support Cc: Ammar Rafli Arkan, Yusuf Hadiwinata, Alviro Iskandar Setiawan, GNU/Weeb Mailing List, Paber Panjaitan [-- Attachment #1: Type: text/plain, Size: 275 bytes --] On Fri, Jul 12, 2024 at 6:40 PM Ammar Rafli wrote: > Hi Bapak Ammar Faizi > > Selamat Malam > > Menanggapi email yang disampaikan, Apakah saat ini Bapak dapat men-capture full command top -c Pak? Berikut saya lampirkan output command `top -c`. -- Ammar Faizi [-- Attachment #2: image.png --] [-- Type: image/png, Size: 313221 bytes --] ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re:[## 385523 ##] Disk I/O di Neo Lite kembali bermasalah 2024-07-12 11:47 ` [## " Ammar Faizi @ 2024-07-12 11:57 ` Ammar Rafli 2024-07-12 12:09 ` [## " Ammar Faizi 0 siblings, 1 reply; 22+ messages in thread From: Ammar Rafli @ 2024-07-12 11:57 UTC (permalink / raw) To: Ammar Faizi Cc: support, ammar rafli arkan, yusuf hadiwinata, alviro iskandar setiawan, gnu/weeb mailing list, paber panjaitan [-- Attachment #1: Type: text/plain, Size: 1744 bytes --] Hi Bapak Ammar Selamat Malam Menanggapi email yang disampaikan, Jika kami check load average server nya tinggi dan terdapat script yang menggunakan 15 core cpu, apakah script tersebut adlaah script fio Pak? Dan bisa dicapture hasil pengetesan terbaru yang dilakukan pada script tersebut? Mohon informasi yang disampaikan. Terima kasih atas perhatian dan kerjasamanya. Penilaian Anda sangat membantu perbaikan dan evaluasi pelayanan kami. Ketika tiket pengaduan Anda kami tutup akan ada otomatis link survei kepuasan pelanggan di mailbox Anda. Mohon untuk memberikan penilaian Anda. Regards Ammar Rafli Arkan Product Operation Support ----------------------------------------------------------- PT. Biznet Gio Nusantara MidPlaza 1, 7th Floor. Jl. Jend. Sudirman Kav 10 - 11. Jakarta 10220 - Indonesia. ----------------------------------------------------------- Click here to submit your feature request. The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. ---- on Fri, 12 Jul 2024 18:47:05 +0700 "Ammar Faizi"<[email protected]> wrote ---- On Fri, Jul 12, 2024 at 6:40 PM Ammar Rafli wrote: > Hi Bapak Ammar Faizi > > Selamat Malam > > Menanggapi email yang disampaikan, Apakah saat ini Bapak dapat men-capture full command top -c Pak? Berikut saya lampirkan output command `top -c`. -- Ammar Faizi [-- Attachment #2: Type: text/html, Size: 3597 bytes --] ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [## 385523 ##] Disk I/O di Neo Lite kembali bermasalah 2024-07-12 11:57 ` Ammar Rafli @ 2024-07-12 12:09 ` Ammar Faizi 2024-07-12 12:20 ` Ammar Rafli 0 siblings, 1 reply; 22+ messages in thread From: Ammar Faizi @ 2024-07-12 12:09 UTC (permalink / raw) To: Biznet Gio Support Cc: Ammar Rafli Arkan, Yusuf Hadiwinata, Alviro Iskandar Setiawan, GNU/Weeb Mailing List, Paber Panjaitan On Fri, Jul 12, 2024 at 6:57 PM Ammar Rafli wrote: > Menanggapi email yang disampaikan, Jika kami check load average server nya tinggi dan terdapat script yang menggunakan 15 core cpu, apakah script tersebut adlaah script fio Pak? Itu bukan fio, tapi itu adalah program stress test CPU. https://gist.githubusercontent.com/ammarfaizi2/f55dbb7c918a660776c68c50319c7882/raw/1dd37e1fb96054b31f345381273e8a1702215002/ct.c Saya jalankan stress test CPU, Disk, dan RAM. -- Ammar Faizi ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re:[## 385523 ##] Disk I/O di Neo Lite kembali bermasalah 2024-07-12 12:09 ` [## " Ammar Faizi @ 2024-07-12 12:20 ` Ammar Rafli 2024-07-12 12:23 ` Ammar Rafli 0 siblings, 1 reply; 22+ messages in thread From: Ammar Rafli @ 2024-07-12 12:20 UTC (permalink / raw) To: Ammar Faizi Cc: support, ammar rafli arkan, yusuf hadiwinata, alviro iskandar setiawan, gnu/weeb mailing list, paber panjaitan [-- Attachment #1: Type: text/plain, Size: 1867 bytes --] Hi Bapak Ammar Selamat Malam Menanggapi email yang disampaikan, Baik, terkait hal tersebut akan kami koordinasikan dahulu. Mohon kesediannya untuk menunggu informasi update dari sisi kami. Sekian informasi yang disampaikan. Terima kasih atas perhatian dan kerjasamanya. Penilaian Anda sangat membantu perbaikan dan evaluasi pelayanan kami. Ketika tiket pengaduan Anda kami tutup akan ada otomatis link survei kepuasan pelanggan di mailbox Anda. Mohon untuk memberikan penilaian Anda. Regards Ammar Rafli Arkan Product Operation Support ----------------------------------------------------------- PT. Biznet Gio Nusantara MidPlaza 1, 7th Floor. Jl. Jend. Sudirman Kav 10 - 11. Jakarta 10220 - Indonesia. ----------------------------------------------------------- Click here to submit your feature request. The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. ---- on Fri, 12 Jul 2024 19:09:19 +0700 "Ammar Faizi"<[email protected]> wrote ---- On Fri, Jul 12, 2024 at 6:57 PM Ammar Rafli wrote: > Menanggapi email yang disampaikan, Jika kami check load average server nya tinggi dan terdapat script yang menggunakan 15 core cpu, apakah script tersebut adlaah script fio Pak? Itu bukan fio, tapi itu adalah program stress test CPU. https://gist.githubusercontent.com/ammarfaizi2/f55dbb7c918a660776c68c50319c7882/raw/1dd37e1fb96054b31f345381273e8a1702215002/ct.c Saya jalankan stress test CPU, Disk, dan RAM. -- Ammar Faizi [-- Attachment #2: Type: text/html, Size: 3708 bytes --] ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re:[## 385523 ##] Disk I/O di Neo Lite kembali bermasalah 2024-07-12 12:20 ` Ammar Rafli @ 2024-07-12 12:23 ` Ammar Rafli 2024-07-12 12:42 ` [## " Ammar Faizi 0 siblings, 1 reply; 22+ messages in thread From: Ammar Rafli @ 2024-07-12 12:23 UTC (permalink / raw) To: Ammar Faizi Cc: support, ammar rafli arkan, yusuf hadiwinata, alviro iskandar setiawan, gnu/weeb mailing list, paber panjaitan [-- Attachment #1: Type: text/plain, Size: 4782 bytes --] Hi Bapak Ammar Selamat Malam Menanggapi email yang disampaikan, Terkait kendala IOPS pada layanan Neo Lite, apakah Bapak berkenan melakukan beberapa percobaan berikut ini Pak? ============================================================================================================= # fio --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --rw=read --filename=/root/fio --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=75 --group_reporting # fio --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --rw=write --filename=/root/fio --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=75 --group_reporting # fio --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --rw=rw --filename=/root/fio --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=50 --group_reporting # fio --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --rw=randread --filename=/root/fio --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=100 --group_reporting # fio --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --rw=randwrite --filename=/root/fio --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=75 --group_reporting # fio --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --rw=randrw --filename=/root/fio --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=0 --group_reporting ========================================================================================================= Silakan untuk melampirkan capture hasil fio nya pada ticket ini agar kami dapat melihat me-review detail IOPS nya ya Pak. Sekian informasi yang disampaikan. Terima kasih atas perhatian dan kerjasamanya. Penilaian Anda sangat membantu perbaikan dan evaluasi pelayanan kami. Ketika tiket pengaduan Anda kami tutup akan ada otomatis link survei kepuasan pelanggan di mailbox Anda. Mohon untuk memberikan penilaian Anda. Regards Ammar Rafli Arkan Product Operation Support ----------------------------------------------------------- PT. Biznet Gio Nusantara MidPlaza 1, 7th Floor. Jl. Jend. Sudirman Kav 10 - 11. Jakarta 10220 - Indonesia. ----------------------------------------------------------- Click here to submit your feature request. The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. ---- on Fri, 12 Jul 2024 19:20:09 +0700 Ammar Rafli<[email protected]> wrote ---- Hi Bapak Ammar Selamat Malam Menanggapi email yang disampaikan, Baik, terkait hal tersebut akan kami koordinasikan dahulu. Mohon kesediannya untuk menunggu informasi update dari sisi kami. Sekian informasi yang disampaikan. Terima kasih atas perhatian dan kerjasamanya. Penilaian Anda sangat membantu perbaikan dan evaluasi pelayanan kami. Ketika tiket pengaduan Anda kami tutup akan ada otomatis link survei kepuasan pelanggan di mailbox Anda. Mohon untuk memberikan penilaian Anda. Regards Ammar Rafli Arkan Product Operation Support ----------------------------------------------------------- PT. Biznet Gio Nusantara MidPlaza 1, 7th Floor. Jl. Jend. Sudirman Kav 10 - 11. Jakarta 10220 - Indonesia. ----------------------------------------------------------- Click here to submit your feature request. The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. ---- on Fri, 12 Jul 2024 19:09:19 +0700 "Ammar Faizi"<[email protected]> wrote ---- On Fri, Jul 12, 2024 at 6:57 PM Ammar Rafli wrote: > Menanggapi email yang disampaikan, Jika kami check load average server nya tinggi dan terdapat script yang menggunakan 15 core cpu, apakah script tersebut adlaah script fio Pak? Itu bukan fio, tapi itu adalah program stress test CPU. https://gist.githubusercontent.com/ammarfaizi2/f55dbb7c918a660776c68c50319c7882/raw/1dd37e1fb96054b31f345381273e8a1702215002/ct.c Saya jalankan stress test CPU, Disk, dan RAM. -- Ammar Faizi [-- Attachment #2: Type: text/html, Size: 8098 bytes --] ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [## 385523 ##] Disk I/O di Neo Lite kembali bermasalah 2024-07-12 12:23 ` Ammar Rafli @ 2024-07-12 12:42 ` Ammar Faizi 2024-07-12 13:00 ` Ammar Rafli 2024-07-12 13:06 ` Yusuf Hadiwinata 0 siblings, 2 replies; 22+ messages in thread From: Ammar Faizi @ 2024-07-12 12:42 UTC (permalink / raw) To: Biznet Gio Support Cc: Ammar Rafli Arkan, Yusuf Hadiwinata, Alviro Iskandar Setiawan, GNU/Weeb Mailing List, Paber Panjaitan On Fri, Jul 12, 2024 at 05:23:09AM -0700, Ammar Rafli wrote: > Silakan untuk melampirkan capture hasil fio nya pada ticket ini agar kami dapat melihat me-review detail IOPS nya ya Pak. Saya lampirkan output fio berikut: root@wjv001:~/fio# ./fio --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --rw=read --filename=/root/fio --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=75 --group_reporting fio: engine libaio not loadable fio: failed to load engine # Note: libaio tidak available, saya ubah ke io_uring. root@wjv001:~/fio# ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=read --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=75 --group_reporting test: (g=0): rw=read, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256 ... fio-3.37-60-g6eaa64 Starting 4 processes Jobs: 2 (f=2): [_(2),R(2)][34.3%][r=5853KiB/s][r=731 IOPS][eta 01m:59s] test: (groupid=0, jobs=4): err= 0: pid=18651: Fri Jul 12 19:25:53 2024 read: IOPS=1001, BW=8013KiB/s (8205kB/s)(477MiB/60951msec) bw ( KiB/s): min= 96, max=20864, per=100.00%, avg=8536.02, stdev=1493.19, samples=385 iops : min= 12, max= 2608, avg=1066.99, stdev=186.65, samples=385 cpu : usr=0.11%, sys=0.25%, ctx=70031, majf=0, minf=2066 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=61048,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=256 Run status group 0 (all jobs): READ: bw=8013KiB/s (8205kB/s), 8013KiB/s-8013KiB/s (8205kB/s-8205kB/s), io=477MiB (500MB), run=60951-60951msec Disk stats (read/write): sda: ios=60990/53, sectors=976152/528, merge=0/10, ticks=55711072/42055, in_queue=55753611, util=99.96% ============================================================================================================= root@wjv001:~/fio# ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=rw --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=50 --group_reporting test: (g=0): rw=rw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256 ... fio-3.37-60-g6eaa64 Starting 4 processes Jobs: 4 (f=4): [M(4)][100.0%][r=8032KiB/s,w=8104KiB/s][r=1004,w=1013 IOPS][eta 00m:00s] test: (groupid=0, jobs=4): err= 0: pid=18664: Fri Jul 12 19:31:16 2024 read: IOPS=994, BW=7959KiB/s (8150kB/s)(471MiB/60624msec) bw ( KiB/s): min= 448, max=21488, per=99.84%, avg=7947.47, stdev=1237.33, samples=481 iops : min= 56, max= 2686, avg=993.41, stdev=154.67, samples=481 write: IOPS=1001, BW=8010KiB/s (8202kB/s)(474MiB/60624msec); 0 zone resets bw ( KiB/s): min= 416, max=21520, per=99.71%, avg=7987.95, stdev=1227.20, samples=481 iops : min= 52, max= 2690, avg=998.47, stdev=153.40, samples=481 cpu : usr=0.20%, sys=0.46%, ctx=96868, majf=0, minf=26 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=60316,60700,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=256 Run status group 0 (all jobs): READ: bw=7959KiB/s (8150kB/s), 7959KiB/s-7959KiB/s (8150kB/s-8150kB/s), io=471MiB (494MB), run=60624-60624msec WRITE: bw=8010KiB/s (8202kB/s), 8010KiB/s-8010KiB/s (8202kB/s-8202kB/s), io=474MiB (497MB), run=60624-60624msec Disk stats (read/write): sda: ios=60146/60471, sectors=962336/967448, merge=0/12, ticks=27376373/30599469, in_queue=57977062, util=99.97% ============================================================================================================= root@wjv001:~/fio# ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=randread --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=100 --group_reporting test: (g=0): rw=randread, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256 ... fio-3.37-60-g6eaa64 Starting 4 processes Jobs: 2 (f=2): [r(1),_(1),r(1),_(1)][34.3%][r=5904KiB/s][r=738 IOPS][eta 01m:59s] test: (groupid=0, jobs=4): err= 0: pid=18673: Fri Jul 12 19:35:08 2024 read: IOPS=1001, BW=8013KiB/s (8206kB/s)(478MiB/61139msec) bw ( KiB/s): min= 1824, max=18288, per=100.00%, avg=8101.39, stdev=1076.94, samples=445 iops : min= 228, max= 2286, avg=1012.67, stdev=134.62, samples=445 cpu : usr=0.12%, sys=0.26%, ctx=71964, majf=0, minf=2076 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=61240,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=256 Run status group 0 (all jobs): READ: bw=8013KiB/s (8206kB/s), 8013KiB/s-8013KiB/s (8206kB/s-8206kB/s), io=478MiB (502MB), run=61139-61139msec Disk stats (read/write): sda: ios=61240/7, sectors=980080/112, merge=0/7, ticks=61595911/6429, in_queue=61602360, util=99.95% ============================================================================================================= root@wjv001:~/fio# ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=randwrite --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=75 --group_reporting test: (g=0): rw=randwrite, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256 ... fio-3.37-60-g6eaa64 Starting 4 processes Jobs: 4 (f=4): [w(4)][3.5%][w=1936KiB/s][w=242 IOPS][eta 28m:31s] test: (groupid=0, jobs=4): err= 0: pid=18683: Fri Jul 12 19:37:33 2024 write: IOPS=1001, BW=8010KiB/s (8202kB/s)(478MiB/61102msec); 0 zone resets bw ( KiB/s): min= 96, max=27984, per=100.00%, avg=8245.91, stdev=1833.66, samples=465 iops : min= 12, max= 3498, avg=1030.70, stdev=229.20, samples=465 cpu : usr=0.13%, sys=0.27%, ctx=44021, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=0,61176,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=256 Run status group 0 (all jobs): WRITE: bw=8010KiB/s (8202kB/s), 8010KiB/s-8010KiB/s (8202kB/s-8202kB/s), io=478MiB (501MB), run=61102-61102msec Disk stats (read/write): sda: ios=0/60981, sectors=0/975608, merge=0/11, ticks=0/52000246, in_queue=52001504, util=99.95% ============================================================================================================= root@wjv001:~/fio# ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=randrw --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=0 --group_reporting test: (g=0): rw=randrw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256 ... fio-3.37-60-g6eaa64 Starting 4 processes Jobs: 4 (f=4): [w(4)][100.0%][w=6904KiB/s][w=863 IOPS][eta 00m:00s] test: (groupid=0, jobs=4): err= 0: pid=18693: Fri Jul 12 19:39:00 2024 write: IOPS=1001, BW=8009KiB/s (8201kB/s)(476MiB/60853msec); 0 zone resets bw ( KiB/s): min= 256, max=24624, per=100.00%, avg=8209.34, stdev=1657.90, samples=470 iops : min= 32, max= 3078, avg=1026.15, stdev=207.24, samples=470 cpu : usr=0.15%, sys=0.31%, ctx=49963, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=0,60920,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=256 Run status group 0 (all jobs): WRITE: bw=8009KiB/s (8201kB/s), 8009KiB/s-8009KiB/s (8201kB/s-8201kB/s), io=476MiB (499MB), run=60853-60853msec Disk stats (read/write): sda: ios=0/60744, sectors=0/971808, merge=0/12, ticks=0/56919694, in_queue=56920300, util=99.97% ============================================================================================================= -- Ammar Faizi ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re:[## 385523 ##] Disk I/O di Neo Lite kembali bermasalah 2024-07-12 12:42 ` [## " Ammar Faizi @ 2024-07-12 13:00 ` Ammar Rafli 2024-07-12 13:19 ` [## " Ammar Faizi 2024-07-12 13:06 ` Yusuf Hadiwinata 1 sibling, 1 reply; 22+ messages in thread From: Ammar Rafli @ 2024-07-12 13:00 UTC (permalink / raw) To: Ammar Faizi Cc: support, ammar rafli arkan, yusuf hadiwinata, alviro iskandar setiawan, gnu/weeb mailing list, paber panjaitan [-- Attachment #1: Type: text/plain, Size: 10098 bytes --] Hi Bapak Ammar Selamat Malam Menanggapi email yang disampaikan, Mohon informasinya apakah Bapak melakukan fio nya saat script stresstest berjalan? Jika iya, apakah Bapak bisa men-kill script stresstest nya dahulu lalu coba jalankan command fio nya kembali Pak? Mohon informasinya pada ticket ini. Terima kasih atas perhatian dan kerjasamanya. Penilaian Anda sangat membantu perbaikan dan evaluasi pelayanan kami. Ketika tiket pengaduan Anda kami tutup akan ada otomatis link survei kepuasan pelanggan di mailbox Anda. Mohon untuk memberikan penilaian Anda. Regards Ammar Rafli Arkan Product Operation Support ----------------------------------------------------------- PT. Biznet Gio Nusantara MidPlaza 1, 7th Floor. Jl. Jend. Sudirman Kav 10 - 11. Jakarta 10220 - Indonesia. ----------------------------------------------------------- Click here to submit your feature request. The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. ---- on Fri, 12 Jul 2024 19:42:03 +0700 "Ammar Faizi"<[email protected]> wrote ---- On Fri, Jul 12, 2024 at 05:23:09AM -0700, Ammar Rafli wrote: > Silakan untuk melampirkan capture hasil fio nya pada ticket ini agar kami dapat melihat me-review detail IOPS nya ya Pak. Saya lampirkan output fio berikut: root@wjv001:~/fio# ./fio --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --rw=read --filename=/root/fio --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=75 --group_reporting fio: engine libaio not loadable fio: failed to load engine # Note: libaio tidak available, saya ubah ke io_uring. root@wjv001:~/fio# ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=read --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=75 --group_reporting test: (g=0): rw=read, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256 ... fio-3.37-60-g6eaa64 Starting 4 processes Jobs: 2 (f=2): [_(2),R(2)][34.3%][r=5853KiB/s][r=731 IOPS][eta 01m:59s] test: (groupid=0, jobs=4): err= 0: pid=18651: Fri Jul 12 19:25:53 2024 read: IOPS=1001, BW=8013KiB/s (8205kB/s)(477MiB/60951msec) bw ( KiB/s): min= 96, max=20864, per=100.00%, avg=8536.02, stdev=1493.19, samples=385 iops : min= 12, max= 2608, avg=1066.99, stdev=186.65, samples=385 cpu : usr=0.11%, sys=0.25%, ctx=70031, majf=0, minf=2066 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=61048,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=256 Run status group 0 (all jobs): READ: bw=8013KiB/s (8205kB/s), 8013KiB/s-8013KiB/s (8205kB/s-8205kB/s), io=477MiB (500MB), run=60951-60951msec Disk stats (read/write): sda: ios=60990/53, sectors=976152/528, merge=0/10, ticks=55711072/42055, in_queue=55753611, util=99.96% ============================================================================================================= root@wjv001:~/fio# ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=rw --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=50 --group_reporting test: (g=0): rw=rw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256 ... fio-3.37-60-g6eaa64 Starting 4 processes Jobs: 4 (f=4): [M(4)][100.0%][r=8032KiB/s,w=8104KiB/s][r=1004,w=1013 IOPS][eta 00m:00s] test: (groupid=0, jobs=4): err= 0: pid=18664: Fri Jul 12 19:31:16 2024 read: IOPS=994, BW=7959KiB/s (8150kB/s)(471MiB/60624msec) bw ( KiB/s): min= 448, max=21488, per=99.84%, avg=7947.47, stdev=1237.33, samples=481 iops : min= 56, max= 2686, avg=993.41, stdev=154.67, samples=481 write: IOPS=1001, BW=8010KiB/s (8202kB/s)(474MiB/60624msec); 0 zone resets bw ( KiB/s): min= 416, max=21520, per=99.71%, avg=7987.95, stdev=1227.20, samples=481 iops : min= 52, max= 2690, avg=998.47, stdev=153.40, samples=481 cpu : usr=0.20%, sys=0.46%, ctx=96868, majf=0, minf=26 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=60316,60700,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=256 Run status group 0 (all jobs): READ: bw=7959KiB/s (8150kB/s), 7959KiB/s-7959KiB/s (8150kB/s-8150kB/s), io=471MiB (494MB), run=60624-60624msec WRITE: bw=8010KiB/s (8202kB/s), 8010KiB/s-8010KiB/s (8202kB/s-8202kB/s), io=474MiB (497MB), run=60624-60624msec Disk stats (read/write): sda: ios=60146/60471, sectors=962336/967448, merge=0/12, ticks=27376373/30599469, in_queue=57977062, util=99.97% ============================================================================================================= root@wjv001:~/fio# ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=randread --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=100 --group_reporting test: (g=0): rw=randread, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256 ... fio-3.37-60-g6eaa64 Starting 4 processes Jobs: 2 (f=2): [r(1),_(1),r(1),_(1)][34.3%][r=5904KiB/s][r=738 IOPS][eta 01m:59s] test: (groupid=0, jobs=4): err= 0: pid=18673: Fri Jul 12 19:35:08 2024 read: IOPS=1001, BW=8013KiB/s (8206kB/s)(478MiB/61139msec) bw ( KiB/s): min= 1824, max=18288, per=100.00%, avg=8101.39, stdev=1076.94, samples=445 iops : min= 228, max= 2286, avg=1012.67, stdev=134.62, samples=445 cpu : usr=0.12%, sys=0.26%, ctx=71964, majf=0, minf=2076 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=61240,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=256 Run status group 0 (all jobs): READ: bw=8013KiB/s (8206kB/s), 8013KiB/s-8013KiB/s (8206kB/s-8206kB/s), io=478MiB (502MB), run=61139-61139msec Disk stats (read/write): sda: ios=61240/7, sectors=980080/112, merge=0/7, ticks=61595911/6429, in_queue=61602360, util=99.95% ============================================================================================================= root@wjv001:~/fio# ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=randwrite --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=75 --group_reporting test: (g=0): rw=randwrite, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256 ... fio-3.37-60-g6eaa64 Starting 4 processes Jobs: 4 (f=4): [w(4)][3.5%][w=1936KiB/s][w=242 IOPS][eta 28m:31s] test: (groupid=0, jobs=4): err= 0: pid=18683: Fri Jul 12 19:37:33 2024 write: IOPS=1001, BW=8010KiB/s (8202kB/s)(478MiB/61102msec); 0 zone resets bw ( KiB/s): min= 96, max=27984, per=100.00%, avg=8245.91, stdev=1833.66, samples=465 iops : min= 12, max= 3498, avg=1030.70, stdev=229.20, samples=465 cpu : usr=0.13%, sys=0.27%, ctx=44021, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=0,61176,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=256 Run status group 0 (all jobs): WRITE: bw=8010KiB/s (8202kB/s), 8010KiB/s-8010KiB/s (8202kB/s-8202kB/s), io=478MiB (501MB), run=61102-61102msec Disk stats (read/write): sda: ios=0/60981, sectors=0/975608, merge=0/11, ticks=0/52000246, in_queue=52001504, util=99.95% ============================================================================================================= root@wjv001:~/fio# ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=randrw --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=0 --group_reporting test: (g=0): rw=randrw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256 ... fio-3.37-60-g6eaa64 Starting 4 processes Jobs: 4 (f=4): [w(4)][100.0%][w=6904KiB/s][w=863 IOPS][eta 00m:00s] test: (groupid=0, jobs=4): err= 0: pid=18693: Fri Jul 12 19:39:00 2024 write: IOPS=1001, BW=8009KiB/s (8201kB/s)(476MiB/60853msec); 0 zone resets bw ( KiB/s): min= 256, max=24624, per=100.00%, avg=8209.34, stdev=1657.90, samples=470 iops : min= 32, max= 3078, avg=1026.15, stdev=207.24, samples=470 cpu : usr=0.15%, sys=0.31%, ctx=49963, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=0,60920,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=256 Run status group 0 (all jobs): WRITE: bw=8009KiB/s (8201kB/s), 8009KiB/s-8009KiB/s (8201kB/s-8201kB/s), io=476MiB (499MB), run=60853-60853msec Disk stats (read/write): sda: ios=0/60744, sectors=0/971808, merge=0/12, ticks=0/56919694, in_queue=56920300, util=99.97% ============================================================================================================= -- Ammar Faizi [-- Attachment #2: Type: text/html, Size: 12848 bytes --] ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [## 385523 ##] Disk I/O di Neo Lite kembali bermasalah 2024-07-12 13:00 ` Ammar Rafli @ 2024-07-12 13:19 ` Ammar Faizi 0 siblings, 0 replies; 22+ messages in thread From: Ammar Faizi @ 2024-07-12 13:19 UTC (permalink / raw) To: Ammar Rafli Cc: ammar rafli arkan, yusuf hadiwinata, alviro iskandar setiawan, gnu/weeb mailing list, paber panjaitan On Fri, Jul 12, 2024 at 06:00:04AM -0700, Ammar Rafli wrote: > Hi Bapak Ammar > > Selamat Malam > > Menanggapi email yang disampaikan, Mohon informasinya apakah Bapak melakukan fio nya saat script stresstest berjalan? > Jika iya, apakah Bapak bisa men-kill script stresstest nya dahulu lalu coba jalankan command fio nya kembali Pak? > > Mohon informasinya pada ticket ini. Iya, tadi dalam keadaan stress test berjalan, berikut hasilnya setelah saya test fio lagi tanpa dibarengi stress test: root@wjv001:~/fio# ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=write --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=75 --group_reporting && \ ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=rw --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=50 --group_reporting && \ ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=randread --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=100 --group_reporting && \ ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=randwrite --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=75 --group_reporting && \ ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=randrw --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=0 --group_reporting test: (g=0): rw=write, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256 ... fio-3.37-60-g6eaa64 Starting 4 processes Jobs: 4 (f=4): [W(4)][5.4%][w=1840KiB/s][w=230 IOPS][eta 18m:15s] test: (groupid=0, jobs=4): err= 0: pid=18772: Fri Jul 12 20:10:11 2024 write: IOPS=1000, BW=8003KiB/s (8195kB/s)(477MiB/61088msec); 0 zone resets bw ( KiB/s): min= 416, max=21296, per=100.00%, avg=8027.61, stdev=1013.58, samples=479 iops : min= 52, max= 2662, avg=1003.43, stdev=126.70, samples=479 cpu : usr=0.09%, sys=0.30%, ctx=26174, majf=0, minf=19 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=0,61112,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=256 Run status group 0 (all jobs): WRITE: bw=8003KiB/s (8195kB/s), 8003KiB/s-8003KiB/s (8195kB/s-8195kB/s), io=477MiB (501MB), run=61088-61088msec Disk stats (read/write): sda: ios=1/60973, sectors=8/975392, merge=0/23, ticks=1/54371694, in_queue=54372476, util=99.97% test: (g=0): rw=rw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256 ... fio-3.37-60-g6eaa64 Starting 4 processes Jobs: 4 (f=4): [M(4)][100.0%][r=8368KiB/s,w=8312KiB/s][r=1046,w=1039 IOPS][eta 00m:00s] test: (groupid=0, jobs=4): err= 0: pid=18830: Fri Jul 12 20:11:12 2024 read: IOPS=993, BW=7949KiB/s (8139kB/s)(469MiB/60477msec) bw ( KiB/s): min= 928, max=15264, per=100.00%, avg=7953.51, stdev=572.87, samples=479 iops : min= 116, max= 1908, avg=994.16, stdev=71.61, samples=479 write: IOPS=998, BW=7992KiB/s (8184kB/s)(472MiB/60477msec); 0 zone resets bw ( KiB/s): min= 2768, max=15456, per=100.00%, avg=8061.87, stdev=546.67, samples=476 iops : min= 346, max= 1932, avg=1007.71, stdev=68.33, samples=476 cpu : usr=0.18%, sys=0.61%, ctx=57583, majf=0, minf=27 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=60089,60415,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=256 Run status group 0 (all jobs): READ: bw=7949KiB/s (8139kB/s), 7949KiB/s-7949KiB/s (8139kB/s-8139kB/s), io=469MiB (492MB), run=60477-60477msec WRITE: bw=7992KiB/s (8184kB/s), 7992KiB/s-7992KiB/s (8184kB/s-8184kB/s), io=472MiB (495MB), run=60477-60477msec Disk stats (read/write): sda: ios=60000/60435, sectors=960000/966856, merge=0/7, ticks=29529611/28322494, in_queue=57852707, util=100.00% test: (g=0): rw=randread, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256 ... fio-3.37-60-g6eaa64 Starting 4 processes Jobs: 4 (f=4): [r(4)][5.1%][r=1873KiB/s][r=234 IOPS][eta 18m:54s] test: (groupid=0, jobs=4): err= 0: pid=18836: Fri Jul 12 20:12:13 2024 read: IOPS=1001, BW=8012KiB/s (8205kB/s)(478MiB/61145msec) bw ( KiB/s): min= 224, max=18592, per=100.00%, avg=8096.67, stdev=934.34, samples=476 iops : min= 28, max= 2326, avg=1012.10, stdev=116.80, samples=476 cpu : usr=0.15%, sys=0.33%, ctx=41342, majf=0, minf=2075 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=61240,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=256 Run status group 0 (all jobs): READ: bw=8012KiB/s (8205kB/s), 8012KiB/s-8012KiB/s (8205kB/s-8205kB/s), io=478MiB (502MB), run=61145-61145msec Disk stats (read/write): sda: ios=61234/3, sectors=979744/32, merge=0/1, ticks=56379750/13301, in_queue=56393126, util=99.98% test: (g=0): rw=randwrite, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256 ... fio-3.37-60-g6eaa64 Starting 4 processes Jobs: 4 (f=4): [w(4)][5.5%][w=1912KiB/s][w=239 IOPS][eta 17m:29s] test: (groupid=0, jobs=4): err= 0: pid=18842: Fri Jul 12 20:13:14 2024 write: IOPS=1001, BW=8010KiB/s (8202kB/s)(477MiB/60974msec); 0 zone resets bw ( KiB/s): min= 64, max=20992, per=100.00%, avg=8018.16, stdev=1004.72, samples=479 iops : min= 8, max= 2624, avg=1002.12, stdev=125.60, samples=479 cpu : usr=0.14%, sys=0.32%, ctx=27728, majf=0, minf=26 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=0,61048,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=256 Run status group 0 (all jobs): WRITE: bw=8010KiB/s (8202kB/s), 8010KiB/s-8010KiB/s (8202kB/s-8202kB/s), io=477MiB (500MB), run=60974-60974msec Disk stats (read/write): sda: ios=0/60980, sectors=0/975616, merge=0/8, ticks=0/53681693, in_queue=53682164, util=100.00% test: (g=0): rw=randrw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256 ... fio-3.37-60-g6eaa64 Starting 4 processes Jobs: 4 (f=4): [w(4)][4.9%][w=1896KiB/s][w=237 IOPS][eta 19m:54s] test: (groupid=0, jobs=4): err= 0: pid=18848: Fri Jul 12 20:14:16 2024 write: IOPS=1001, BW=8012KiB/s (8204kB/s)(477MiB/61021msec); 0 zone resets bw ( KiB/s): min= 80, max=18352, per=100.00%, avg=8012.34, stdev=973.54, samples=480 iops : min= 10, max= 2294, avg=1001.54, stdev=121.69, samples=480 cpu : usr=0.15%, sys=0.29%, ctx=28583, majf=0, minf=18 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=0,61112,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=256 Run status group 0 (all jobs): WRITE: bw=8012KiB/s (8204kB/s), 8012KiB/s-8012KiB/s (8204kB/s-8204kB/s), io=477MiB (501MB), run=61021-61021msec Disk stats (read/write): sda: ios=0/60986, sectors=0/975768, merge=0/2, ticks=0/53705577, in_queue=53706042, util=99.98% root@wjv001:~/fio# -- Ammar Faizi ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [## 385523 ##] Disk I/O di Neo Lite kembali bermasalah 2024-07-12 12:42 ` [## " Ammar Faizi 2024-07-12 13:00 ` Ammar Rafli @ 2024-07-12 13:06 ` Yusuf Hadiwinata 2024-07-12 13:12 ` Yusuf Hadiwinata 2024-07-12 13:12 ` Ammar Faizi 1 sibling, 2 replies; 22+ messages in thread From: Yusuf Hadiwinata @ 2024-07-12 13:06 UTC (permalink / raw) To: Ammar Faizi, Support Cc: Ammar Rafli Arkan, Alviro Iskandar Setiawan, GNU/Weeb Mailing List, Paber Panjaitan [-- Attachment #1: Type: text/plain, Size: 10136 bytes --] Dear Ammar dan Team Kita mau analisa beberapa suspect di Host yang terdapat instance yang di laporkan, boleh kita move instance nya ? Akan ada downtime estimas < 5 menit jika berkenan Kalo ada preferensi waktu boleh di info juga kira2 enak nya jam berapaan Thanks ________________________________ From: Ammar Faizi <[email protected]> Sent: Friday, 12 July 2024 19:42 To: Support <[email protected]> Cc: Ammar Rafli Arkan <[email protected]>; Yusuf Hadiwinata <[email protected]>; Alviro Iskandar Setiawan <[email protected]>; GNU/Weeb Mailing List <[email protected]>; Paber Panjaitan <[email protected]> Subject: Re: [## 385523 ##] Disk I/O di Neo Lite kembali bermasalah On Fri, Jul 12, 2024 at 05:23:09AM -0700, Ammar Rafli wrote: > Silakan untuk melampirkan capture hasil fio nya pada ticket ini agar kami dapat melihat me-review detail IOPS nya ya Pak. Saya lampirkan output fio berikut: root@wjv001:~/fio# ./fio --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --rw=read --filename=/root/fio --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=75 --group_reporting fio: engine libaio not loadable fio: failed to load engine # Note: libaio tidak available, saya ubah ke io_uring. root@wjv001:~/fio# ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=read --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=75 --group_reporting test: (g=0): rw=read, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256 ... fio-3.37-60-g6eaa64 Starting 4 processes Jobs: 2 (f=2): [_(2),R(2)][34.3%][r=5853KiB/s][r=731 IOPS][eta 01m:59s] test: (groupid=0, jobs=4): err= 0: pid=18651: Fri Jul 12 19:25:53 2024 read: IOPS=1001, BW=8013KiB/s (8205kB/s)(477MiB/60951msec) bw ( KiB/s): min= 96, max=20864, per=100.00%, avg=8536.02, stdev=1493.19, samples=385 iops : min= 12, max= 2608, avg=1066.99, stdev=186.65, samples=385 cpu : usr=0.11%, sys=0.25%, ctx=70031, majf=0, minf=2066 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=61048,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=256 Run status group 0 (all jobs): READ: bw=8013KiB/s (8205kB/s), 8013KiB/s-8013KiB/s (8205kB/s-8205kB/s), io=477MiB (500MB), run=60951-60951msec Disk stats (read/write): sda: ios=60990/53, sectors=976152/528, merge=0/10, ticks=55711072/42055, in_queue=55753611, util=99.96% ============================================================================================================= root@wjv001:~/fio# ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=rw --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=50 --group_reporting test: (g=0): rw=rw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256 ... fio-3.37-60-g6eaa64 Starting 4 processes Jobs: 4 (f=4): [M(4)][100.0%][r=8032KiB/s,w=8104KiB/s][r=1004,w=1013 IOPS][eta 00m:00s] test: (groupid=0, jobs=4): err= 0: pid=18664: Fri Jul 12 19:31:16 2024 read: IOPS=994, BW=7959KiB/s (8150kB/s)(471MiB/60624msec) bw ( KiB/s): min= 448, max=21488, per=99.84%, avg=7947.47, stdev=1237.33, samples=481 iops : min= 56, max= 2686, avg=993.41, stdev=154.67, samples=481 write: IOPS=1001, BW=8010KiB/s (8202kB/s)(474MiB/60624msec); 0 zone resets bw ( KiB/s): min= 416, max=21520, per=99.71%, avg=7987.95, stdev=1227.20, samples=481 iops : min= 52, max= 2690, avg=998.47, stdev=153.40, samples=481 cpu : usr=0.20%, sys=0.46%, ctx=96868, majf=0, minf=26 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=60316,60700,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=256 Run status group 0 (all jobs): READ: bw=7959KiB/s (8150kB/s), 7959KiB/s-7959KiB/s (8150kB/s-8150kB/s), io=471MiB (494MB), run=60624-60624msec WRITE: bw=8010KiB/s (8202kB/s), 8010KiB/s-8010KiB/s (8202kB/s-8202kB/s), io=474MiB (497MB), run=60624-60624msec Disk stats (read/write): sda: ios=60146/60471, sectors=962336/967448, merge=0/12, ticks=27376373/30599469, in_queue=57977062, util=99.97% ============================================================================================================= root@wjv001:~/fio# ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=randread --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=100 --group_reporting test: (g=0): rw=randread, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256 ... fio-3.37-60-g6eaa64 Starting 4 processes Jobs: 2 (f=2): [r(1),_(1),r(1),_(1)][34.3%][r=5904KiB/s][r=738 IOPS][eta 01m:59s] test: (groupid=0, jobs=4): err= 0: pid=18673: Fri Jul 12 19:35:08 2024 read: IOPS=1001, BW=8013KiB/s (8206kB/s)(478MiB/61139msec) bw ( KiB/s): min= 1824, max=18288, per=100.00%, avg=8101.39, stdev=1076.94, samples=445 iops : min= 228, max= 2286, avg=1012.67, stdev=134.62, samples=445 cpu : usr=0.12%, sys=0.26%, ctx=71964, majf=0, minf=2076 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=61240,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=256 Run status group 0 (all jobs): READ: bw=8013KiB/s (8206kB/s), 8013KiB/s-8013KiB/s (8206kB/s-8206kB/s), io=478MiB (502MB), run=61139-61139msec Disk stats (read/write): sda: ios=61240/7, sectors=980080/112, merge=0/7, ticks=61595911/6429, in_queue=61602360, util=99.95% ============================================================================================================= root@wjv001:~/fio# ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=randwrite --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=75 --group_reporting test: (g=0): rw=randwrite, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256 ... fio-3.37-60-g6eaa64 Starting 4 processes Jobs: 4 (f=4): [w(4)][3.5%][w=1936KiB/s][w=242 IOPS][eta 28m:31s] test: (groupid=0, jobs=4): err= 0: pid=18683: Fri Jul 12 19:37:33 2024 write: IOPS=1001, BW=8010KiB/s (8202kB/s)(478MiB/61102msec); 0 zone resets bw ( KiB/s): min= 96, max=27984, per=100.00%, avg=8245.91, stdev=1833.66, samples=465 iops : min= 12, max= 3498, avg=1030.70, stdev=229.20, samples=465 cpu : usr=0.13%, sys=0.27%, ctx=44021, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=0,61176,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=256 Run status group 0 (all jobs): WRITE: bw=8010KiB/s (8202kB/s), 8010KiB/s-8010KiB/s (8202kB/s-8202kB/s), io=478MiB (501MB), run=61102-61102msec Disk stats (read/write): sda: ios=0/60981, sectors=0/975608, merge=0/11, ticks=0/52000246, in_queue=52001504, util=99.95% ============================================================================================================= root@wjv001:~/fio# ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=randrw --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=0 --group_reporting test: (g=0): rw=randrw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256 ... fio-3.37-60-g6eaa64 Starting 4 processes Jobs: 4 (f=4): [w(4)][100.0%][w=6904KiB/s][w=863 IOPS][eta 00m:00s] test: (groupid=0, jobs=4): err= 0: pid=18693: Fri Jul 12 19:39:00 2024 write: IOPS=1001, BW=8009KiB/s (8201kB/s)(476MiB/60853msec); 0 zone resets bw ( KiB/s): min= 256, max=24624, per=100.00%, avg=8209.34, stdev=1657.90, samples=470 iops : min= 32, max= 3078, avg=1026.15, stdev=207.24, samples=470 cpu : usr=0.15%, sys=0.31%, ctx=49963, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=0,60920,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=256 Run status group 0 (all jobs): WRITE: bw=8009KiB/s (8201kB/s), 8009KiB/s-8009KiB/s (8201kB/s-8201kB/s), io=476MiB (499MB), run=60853-60853msec Disk stats (read/write): sda: ios=0/60744, sectors=0/971808, merge=0/12, ticks=0/56919694, in_queue=56920300, util=99.97% ============================================================================================================= -- Ammar Faizi The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. [-- Attachment #2: Type: text/html, Size: 15312 bytes --] ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [## 385523 ##] Disk I/O di Neo Lite kembali bermasalah 2024-07-12 13:06 ` Yusuf Hadiwinata @ 2024-07-12 13:12 ` Yusuf Hadiwinata 2024-07-12 13:13 ` Ammar Faizi 2024-07-12 13:12 ` Ammar Faizi 1 sibling, 1 reply; 22+ messages in thread From: Yusuf Hadiwinata @ 2024-07-12 13:12 UTC (permalink / raw) To: Ammar Faizi, Support Cc: Ammar Rafli Arkan, Alviro Iskandar Setiawan, GNU/Weeb Mailing List, Paber Panjaitan [-- Attachment #1: Type: text/plain, Size: 11167 bytes --] Dear Ammar dan Team Sorry setelah kita cek IP 103.175.218.159 tidak di host yang kita suspect, mohon ignore email sebelumnya Kita coba analisa lagi ya thnks ________________________________ From: Yusuf Hadiwinata <[email protected]> Sent: Friday, 12 July 2024 20:06 To: Ammar Faizi <[email protected]>; Support <[email protected]> Cc: Ammar Rafli Arkan <[email protected]>; Alviro Iskandar Setiawan <[email protected]>; GNU/Weeb Mailing List <[email protected]>; Paber Panjaitan <[email protected]> Subject: Re: [## 385523 ##] Disk I/O di Neo Lite kembali bermasalah Dear Ammar dan Team Kita mau analisa beberapa suspect di Host yang terdapat instance yang di laporkan, boleh kita move instance nya ? Akan ada downtime estimas < 5 menit jika berkenan Kalo ada preferensi waktu boleh di info juga kira2 enak nya jam berapaan Thanks ________________________________ From: Ammar Faizi <[email protected]> Sent: Friday, 12 July 2024 19:42 To: Support <[email protected]> Cc: Ammar Rafli Arkan <[email protected]>; Yusuf Hadiwinata <[email protected]>; Alviro Iskandar Setiawan <[email protected]>; GNU/Weeb Mailing List <[email protected]>; Paber Panjaitan <[email protected]> Subject: Re: [## 385523 ##] Disk I/O di Neo Lite kembali bermasalah On Fri, Jul 12, 2024 at 05:23:09AM -0700, Ammar Rafli wrote: > Silakan untuk melampirkan capture hasil fio nya pada ticket ini agar kami dapat melihat me-review detail IOPS nya ya Pak. Saya lampirkan output fio berikut: root@wjv001:~/fio# ./fio --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --rw=read --filename=/root/fio --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=75 --group_reporting fio: engine libaio not loadable fio: failed to load engine # Note: libaio tidak available, saya ubah ke io_uring. root@wjv001:~/fio# ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=read --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=75 --group_reporting test: (g=0): rw=read, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256 ... fio-3.37-60-g6eaa64 Starting 4 processes Jobs: 2 (f=2): [_(2),R(2)][34.3%][r=5853KiB/s][r=731 IOPS][eta 01m:59s] test: (groupid=0, jobs=4): err= 0: pid=18651: Fri Jul 12 19:25:53 2024 read: IOPS=1001, BW=8013KiB/s (8205kB/s)(477MiB/60951msec) bw ( KiB/s): min= 96, max=20864, per=100.00%, avg=8536.02, stdev=1493.19, samples=385 iops : min= 12, max= 2608, avg=1066.99, stdev=186.65, samples=385 cpu : usr=0.11%, sys=0.25%, ctx=70031, majf=0, minf=2066 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=61048,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=256 Run status group 0 (all jobs): READ: bw=8013KiB/s (8205kB/s), 8013KiB/s-8013KiB/s (8205kB/s-8205kB/s), io=477MiB (500MB), run=60951-60951msec Disk stats (read/write): sda: ios=60990/53, sectors=976152/528, merge=0/10, ticks=55711072/42055, in_queue=55753611, util=99.96% ============================================================================================================= root@wjv001:~/fio# ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=rw --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=50 --group_reporting test: (g=0): rw=rw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256 ... fio-3.37-60-g6eaa64 Starting 4 processes Jobs: 4 (f=4): [M(4)][100.0%][r=8032KiB/s,w=8104KiB/s][r=1004,w=1013 IOPS][eta 00m:00s] test: (groupid=0, jobs=4): err= 0: pid=18664: Fri Jul 12 19:31:16 2024 read: IOPS=994, BW=7959KiB/s (8150kB/s)(471MiB/60624msec) bw ( KiB/s): min= 448, max=21488, per=99.84%, avg=7947.47, stdev=1237.33, samples=481 iops : min= 56, max= 2686, avg=993.41, stdev=154.67, samples=481 write: IOPS=1001, BW=8010KiB/s (8202kB/s)(474MiB/60624msec); 0 zone resets bw ( KiB/s): min= 416, max=21520, per=99.71%, avg=7987.95, stdev=1227.20, samples=481 iops : min= 52, max= 2690, avg=998.47, stdev=153.40, samples=481 cpu : usr=0.20%, sys=0.46%, ctx=96868, majf=0, minf=26 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=60316,60700,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=256 Run status group 0 (all jobs): READ: bw=7959KiB/s (8150kB/s), 7959KiB/s-7959KiB/s (8150kB/s-8150kB/s), io=471MiB (494MB), run=60624-60624msec WRITE: bw=8010KiB/s (8202kB/s), 8010KiB/s-8010KiB/s (8202kB/s-8202kB/s), io=474MiB (497MB), run=60624-60624msec Disk stats (read/write): sda: ios=60146/60471, sectors=962336/967448, merge=0/12, ticks=27376373/30599469, in_queue=57977062, util=99.97% ============================================================================================================= root@wjv001:~/fio# ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=randread --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=100 --group_reporting test: (g=0): rw=randread, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256 ... fio-3.37-60-g6eaa64 Starting 4 processes Jobs: 2 (f=2): [r(1),_(1),r(1),_(1)][34.3%][r=5904KiB/s][r=738 IOPS][eta 01m:59s] test: (groupid=0, jobs=4): err= 0: pid=18673: Fri Jul 12 19:35:08 2024 read: IOPS=1001, BW=8013KiB/s (8206kB/s)(478MiB/61139msec) bw ( KiB/s): min= 1824, max=18288, per=100.00%, avg=8101.39, stdev=1076.94, samples=445 iops : min= 228, max= 2286, avg=1012.67, stdev=134.62, samples=445 cpu : usr=0.12%, sys=0.26%, ctx=71964, majf=0, minf=2076 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=61240,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=256 Run status group 0 (all jobs): READ: bw=8013KiB/s (8206kB/s), 8013KiB/s-8013KiB/s (8206kB/s-8206kB/s), io=478MiB (502MB), run=61139-61139msec Disk stats (read/write): sda: ios=61240/7, sectors=980080/112, merge=0/7, ticks=61595911/6429, in_queue=61602360, util=99.95% ============================================================================================================= root@wjv001:~/fio# ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=randwrite --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=75 --group_reporting test: (g=0): rw=randwrite, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256 ... fio-3.37-60-g6eaa64 Starting 4 processes Jobs: 4 (f=4): [w(4)][3.5%][w=1936KiB/s][w=242 IOPS][eta 28m:31s] test: (groupid=0, jobs=4): err= 0: pid=18683: Fri Jul 12 19:37:33 2024 write: IOPS=1001, BW=8010KiB/s (8202kB/s)(478MiB/61102msec); 0 zone resets bw ( KiB/s): min= 96, max=27984, per=100.00%, avg=8245.91, stdev=1833.66, samples=465 iops : min= 12, max= 3498, avg=1030.70, stdev=229.20, samples=465 cpu : usr=0.13%, sys=0.27%, ctx=44021, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=0,61176,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=256 Run status group 0 (all jobs): WRITE: bw=8010KiB/s (8202kB/s), 8010KiB/s-8010KiB/s (8202kB/s-8202kB/s), io=478MiB (501MB), run=61102-61102msec Disk stats (read/write): sda: ios=0/60981, sectors=0/975608, merge=0/11, ticks=0/52000246, in_queue=52001504, util=99.95% ============================================================================================================= root@wjv001:~/fio# ./fio --ioengine=io_uring --direct=1 --gtod_reduce=1 --name=test --rw=randrw --filename=./test --bs=8k --iodepth=256 --size=2Gb --runtime=60 --numjobs=4 --time_based --rwmixread=0 --group_reporting test: (g=0): rw=randrw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=io_uring, iodepth=256 ... fio-3.37-60-g6eaa64 Starting 4 processes Jobs: 4 (f=4): [w(4)][100.0%][w=6904KiB/s][w=863 IOPS][eta 00m:00s] test: (groupid=0, jobs=4): err= 0: pid=18693: Fri Jul 12 19:39:00 2024 write: IOPS=1001, BW=8009KiB/s (8201kB/s)(476MiB/60853msec); 0 zone resets bw ( KiB/s): min= 256, max=24624, per=100.00%, avg=8209.34, stdev=1657.90, samples=470 iops : min= 32, max= 3078, avg=1026.15, stdev=207.24, samples=470 cpu : usr=0.15%, sys=0.31%, ctx=49963, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=0,60920,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=256 Run status group 0 (all jobs): WRITE: bw=8009KiB/s (8201kB/s), 8009KiB/s-8009KiB/s (8201kB/s-8201kB/s), io=476MiB (499MB), run=60853-60853msec Disk stats (read/write): sda: ios=0/60744, sectors=0/971808, merge=0/12, ticks=0/56919694, in_queue=56920300, util=99.97% ============================================================================================================= -- Ammar Faizi The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. [-- Attachment #2: Type: text/html, Size: 18057 bytes --] ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [## 385523 ##] Disk I/O di Neo Lite kembali bermasalah 2024-07-12 13:12 ` Yusuf Hadiwinata @ 2024-07-12 13:13 ` Ammar Faizi 2024-07-12 13:20 ` Yusuf Hadiwinata 0 siblings, 1 reply; 22+ messages in thread From: Ammar Faizi @ 2024-07-12 13:13 UTC (permalink / raw) To: Yusuf Hadiwinata Cc: Support, Ammar Rafli Arkan, Alviro Iskandar Setiawan, GNU/Weeb Mailing List, Paber Panjaitan On Fri, Jul 12, 2024 at 8:12 PM Yusuf Hadiwinata <[email protected]> wrote: > Dear Ammar dan Team > > Sorry setelah kita cek IP 103.175.218.159 tidak di host yang kita suspect, mohon ignore email sebelumnya > > Kita coba analisa lagi ya Oke. -- Ammar Faizi ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [## 385523 ##] Disk I/O di Neo Lite kembali bermasalah 2024-07-12 13:13 ` Ammar Faizi @ 2024-07-12 13:20 ` Yusuf Hadiwinata 2024-07-12 13:21 ` Ammar Faizi 0 siblings, 1 reply; 22+ messages in thread From: Yusuf Hadiwinata @ 2024-07-12 13:20 UTC (permalink / raw) To: Ammar Faizi Cc: Support, Ammar Rafli Arkan, Alviro Iskandar Setiawan, GNU/Weeb Mailing List, Paber Panjaitan [-- Attachment #1: Type: text/plain, Size: 1230 bytes --] Untuk IP 103.175.218.159 boleh di infokan OS dan Kernel Version yang di gunakan thnks ________________________________ From: Ammar Faizi <[email protected]> Sent: Friday, 12 July 2024 20:13 To: Yusuf Hadiwinata <[email protected]> Cc: Support <[email protected]>; Ammar Rafli Arkan <[email protected]>; Alviro Iskandar Setiawan <[email protected]>; GNU/Weeb Mailing List <[email protected]>; Paber Panjaitan <[email protected]> Subject: Re: [## 385523 ##] Disk I/O di Neo Lite kembali bermasalah On Fri, Jul 12, 2024 at 8:12 PM Yusuf Hadiwinata <[email protected]> wrote: > Dear Ammar dan Team > > Sorry setelah kita cek IP 103.175.218.159 tidak di host yang kita suspect, mohon ignore email sebelumnya > > Kita coba analisa lagi ya Oke. -- Ammar Faizi The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. [-- Attachment #2: Type: text/html, Size: 2661 bytes --] ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [## 385523 ##] Disk I/O di Neo Lite kembali bermasalah 2024-07-12 13:20 ` Yusuf Hadiwinata @ 2024-07-12 13:21 ` Ammar Faizi 2024-07-12 13:31 ` Yusuf Hadiwinata 0 siblings, 1 reply; 22+ messages in thread From: Ammar Faizi @ 2024-07-12 13:21 UTC (permalink / raw) To: Yusuf Hadiwinata Cc: Support, Ammar Rafli Arkan, Alviro Iskandar Setiawan, GNU/Weeb Mailing List, Paber Panjaitan On Fri, Jul 12, 2024 at 01:20:29PM +0000, Yusuf Hadiwinata wrote: > Untuk IP 103.175.218.159 boleh di infokan OS dan Kernel Version yang di gunakan root@wjv001:~/fio# curl ip.me 103.175.218.159 root@wjv001:~/fio# uname -r 6.8.1-af-server-2024-03-19-g760d1c206de8 root@wjv001:~/fio# cat /etc/*release* DISTRIB_ID=Ubuntu DISTRIB_RELEASE=22.04 DISTRIB_CODENAME=jammy DISTRIB_DESCRIPTION="Ubuntu 22.04.4 LTS" PRETTY_NAME="Ubuntu 22.04.4 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.4 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=jammy root@wjv001:~/fio# -- Ammar Faizi ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [## 385523 ##] Disk I/O di Neo Lite kembali bermasalah 2024-07-12 13:21 ` Ammar Faizi @ 2024-07-12 13:31 ` Yusuf Hadiwinata 2024-07-12 13:40 ` Ammar Faizi 2024-07-12 14:45 ` [## " Yusuf Hadiwinata 0 siblings, 2 replies; 22+ messages in thread From: Yusuf Hadiwinata @ 2024-07-12 13:31 UTC (permalink / raw) To: Ammar Faizi Cc: Support, Ammar Rafli Arkan, Alviro Iskandar Setiawan, GNU/Weeb Mailing List, Paber Panjaitan [-- Attachment #1: Type: text/plain, Size: 1809 bytes --] Thnks you Mau memastikan, apakah sebelum terjadi Blocking Thread ada OOM thnks ________________________________ From: Ammar Faizi <[email protected]> Sent: Friday, 12 July 2024 20:21 To: Yusuf Hadiwinata <[email protected]> Cc: Support <[email protected]>; Ammar Rafli Arkan <[email protected]>; Alviro Iskandar Setiawan <[email protected]>; GNU/Weeb Mailing List <[email protected]>; Paber Panjaitan <[email protected]> Subject: Re: [## 385523 ##] Disk I/O di Neo Lite kembali bermasalah On Fri, Jul 12, 2024 at 01:20:29PM +0000, Yusuf Hadiwinata wrote: > Untuk IP 103.175.218.159 boleh di infokan OS dan Kernel Version yang di gunakan root@wjv001:~/fio# curl ip.me 103.175.218.159 root@wjv001:~/fio# uname -r 6.8.1-af-server-2024-03-19-g760d1c206de8 root@wjv001:~/fio# cat /etc/*release* DISTRIB_ID=Ubuntu DISTRIB_RELEASE=22.04 DISTRIB_CODENAME=jammy DISTRIB_DESCRIPTION="Ubuntu 22.04.4 LTS" PRETTY_NAME="Ubuntu 22.04.4 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.4 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=jammy root@wjv001:~/fio# -- Ammar Faizi The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. [-- Attachment #2: Type: text/html, Size: 3953 bytes --] ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [## 385523 ##] Disk I/O di Neo Lite kembali bermasalah 2024-07-12 13:31 ` Yusuf Hadiwinata @ 2024-07-12 13:40 ` Ammar Faizi 2024-07-12 14:12 ` Ammar Rafli 2024-07-12 14:45 ` [## " Yusuf Hadiwinata 1 sibling, 1 reply; 22+ messages in thread From: Ammar Faizi @ 2024-07-12 13:40 UTC (permalink / raw) To: Yusuf Hadiwinata Cc: Support, Ammar Rafli Arkan, Alviro Iskandar Setiawan, GNU/Weeb Mailing List, Paber Panjaitan On Fri, Jul 12, 2024 at 01:31:30PM +0000, Yusuf Hadiwinata wrote: > Thnks you > > Mau memastikan, apakah sebelum terjadi Blocking Thread ada OOM Iya ada, saya juga stress test memory, sempat hit OOM killer juga. Untuk testing memory saya pakai simple PHP script yang memakan memory sekitar 11 GB seperti ini: <?php $nr = 1024*1024*1024*5; $a = str_repeat("a", $nr); echo "ar\n"; $b = str_repeat("b", $nr); echo "br\n"; while (1) { $tmp = ""; for ($i = 0; $i < $nr; $i++) { $tmp .= $a[$i]; } echo "A done!\n"; $tmp .= ""; for ($i = 0; $i < $nr; $i++) { $tmp .= $b[$i]; } echo "B done!\n"; sleep(5); } ---- dmesg: [15832.976830] Out of memory: Killed process 7142 (php) total-vm:12649176kB, anon-rss:4339492kB, file-rss:3200kB, shmem-rss:0kB, UID:0 pgtables:24720kB oom_score_adj:0 [15836.351730] oom_reaper: reaped process 7142 (php), now anon-rss:636kB, file-rss:796kB, shmem-rss:0kB [22471.722388] ct invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0 [22471.722420] CPU: 7 PID: 8149 Comm: ct Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8 5aa4ec4a5a8034d921df04c799edf17a90d1ec90 [22471.722434] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.1-0-g3208b098f51a-prebuilt.qemu.org 04/01/2014 [22471.722446] Call Trace: [22471.722508] <TASK> [22471.722543] dump_header+0x50/0x170 [22471.722612] oom_kill_process+0x1c0/0x5b0 [22471.722624] out_of_memory+0x7af/0xdf0 [22471.722628] __alloc_pages_slowpath+0xba1/0x1210 [22471.722648] __alloc_pages+0x23d/0x330 [22471.722657] alloc_pages_mpol+0xf4/0x1a0 [22471.722670] filemap_alloc_folio+0xa3/0x1d0 [22471.722690] __filemap_get_folio+0x195/0x480 [22471.722694] filemap_fault+0x212/0x830 [22471.722700] ? __slab_free+0x5a/0x1a0 [22471.722704] __do_fault+0x3b/0x120 [22471.722714] do_pte_missing+0x16b/0x1060 [22471.722716] ? skb_release_data+0x1c6/0x2b0 [22471.722740] handle_mm_fault+0x93c/0x1380 [22471.722743] do_user_addr_fault+0x202/0xa10 [22471.722761] exc_page_fault+0x53/0xc0 [22471.722788] asm_exc_page_fault+0x22/0x30 [22471.722802] RIP: 0033:0x557bbad655bb [22471.722814] Code: Unable to access opcode bytes at 0x557bbad65591. [22471.722815] RSP: 002b:00007fbd6e9ffc48 EFLAGS: 00010206 [22471.722818] RAX: 0000000000000012 RBX: 00007fbd6ea00640 RCX: 0000000000000001 [22471.722820] RDX: 0000000000000000 RSI: 00007fbd70000b70 RDI: 00007fbd6e9ff8f0 [22471.722821] RBP: 00007fbd6e9ffe48 R08: 0000000000000000 R09: 00007fbd6e9ffd27 [22471.722823] R10: 0000000000000000 R11: 0000000000000293 R12: 00007fbd6ea00640 [22471.722824] R13: 0000000000000000 R14: 00007fbd774947d0 R15: 00007ffdf6585af0 [22471.722826] </TASK> [22471.722827] Mem-Info: [22471.722835] active_anon:1046703 inactive_anon:469261 isolated_anon:96 active_file:0 inactive_file:41 isolated_file:0 unevictable:1579777 dirty:3 writeback:0 slab_reclaimable:6224 slab_unreclaimable:70607 mapped:3882 shmem:18 pagetables:11208 sec_pagetables:0 bounce:0 kernel_misc_reclaimable:0 free:66954 free_pcp:0 free_cma:0 [22471.722840] Node 0 active_anon:4186812kB inactive_anon:1877044kB active_file:0kB inactive_file:164kB unevictable:6319108kB isolated(anon):384kB isolated(file):0kB mapped:15528kB dirty:12kB writeback:0kB shmem:72kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:0kB writeback_tmp:0kB kernel_stack:5328kB pagetables:44832kB sec_pagetables:0kB all_unreclaimable? no [22471.722867] Node 0 DMA free:15360kB boost:0kB min:60kB low:72kB high:84kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [22471.722878] lowmem_reserve[]: 0 2856 15884 15884 15884 [22471.722882] Node 0 DMA32 free:64128kB boost:0kB min:12144kB low:15180kB high:18216kB reserved_highatomic:0KB active_anon:472036kB inactive_anon:161520kB active_file:48kB inactive_file:0kB unevictable:2115584kB writepending:0kB present:3129192kB managed:2989864kB mlocked:2115584kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [22471.722889] lowmem_reserve[]: 0 0 13027 13027 13027 [22471.722892] Node 0 Normal free:188328kB boost:0kB min:55372kB low:69212kB high:83052kB reserved_highatomic:133120KB active_anon:3714332kB inactive_anon:1716504kB active_file:300kB inactive_file:592kB unevictable:4203524kB writepending:12kB present:13631488kB managed:13339936kB mlocked:4203524kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [22471.722899] lowmem_reserve[]: 0 0 0 0 0 [22471.722903] Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15360kB [22471.722964] Node 0 DMA32: 123*4kB (UME) 46*8kB (UME) 48*16kB (UE) 20*32kB (UE) 27*64kB (UE) 29*128kB (UME) 41*256kB (UME) 18*512kB (UME) 6*1024kB (UME) 1*2048kB (M) 7*4096kB (UM) = 64284kB [22471.722977] Node 0 Normal: 118*4kB (UME) 111*8kB (UME) 135*16kB (UM) 25*32kB (UM) 2330*64kB (UM) 272*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 188256kB [22471.722994] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [22471.722996] 22547 total pagecache pages [22471.722997] 20249 pages in swap cache [22471.722998] Free swap = 0kB [22471.722998] Total swap = 8388604kB [22471.723002] 4194168 pages RAM [22471.723003] 0 pages HighMem/MovableOnly [22471.723004] 107878 pages reserved [22471.723007] 0 pages hwpoisoned [22471.723008] Tasks state (memory values in pages): [22471.723012] [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name [22471.723021] [ 451] 0 451 9889 992 128 864 0 102400 192 -250 systemd-journal [22471.723026] [ 485] 0 485 72338 6880 4640 2240 0 110592 0 -1000 multipathd [22471.723034] [ 489] 0 489 6527 858 96 762 0 77824 416 -1000 systemd-udevd [22471.723037] [ 651] 103 651 22341 1088 96 992 0 73728 96 0 systemd-timesyn [22471.723040] [ 694] 100 694 4032 928 128 800 0 65536 128 0 systemd-network [22471.723043] [ 697] 101 697 6385 1082 218 864 0 81920 864 0 systemd-resolve [22471.723045] [ 734] 0 734 1822 768 64 704 0 45056 0 0 cron [22471.723048] [ 736] 102 736 2189 1184 224 960 0 49152 32 -900 dbus-daemon [22471.723050] [ 749] 0 749 20705 832 0 832 0 61440 96 0 irqbalance [22471.723055] [ 751] 0 751 8296 862 30 832 0 98304 2144 0 networkd-dispat [22471.723058] [ 752] 0 752 58626 992 96 896 0 94208 64 0 polkitd [22471.723060] [ 755] 0 755 20013 768 32 736 0 57344 0 0 qemu-ga [22471.723063] [ 761] 0 761 588565 1194 650 544 0 450560 2176 -900 snapd [22471.723065] [ 764] 0 764 3883 992 128 864 0 65536 128 0 systemd-logind [22471.723067] [ 766] 0 766 98156 1216 192 1024 0 139264 416 0 udisksd [22471.723083] [ 772] 0 772 1555 512 32 480 0 45056 0 0 agetty [22471.723085] [ 782] 0 782 1544 480 32 448 0 45056 0 0 agetty [22471.723090] [ 806] 0 806 27526 896 32 864 0 122880 1984 0 unattended-upgr [22471.723093] [ 826] 0 826 61061 1024 96 928 0 106496 352 0 ModemManager [22471.723095] [ 827] 0 827 3859 1056 96 960 0 61440 288 -1000 sshd [22471.723098] [ 1098] 0 1098 4278 768 64 704 0 73728 352 0 systemd [22471.723103] [ 1099] 0 1099 42339 430 78 352 0 90112 864 0 (sd-pam) [22471.723106] [ 1680] 0 1680 74007 1056 128 928 0 155648 672 0 packagekitd [22471.723108] [ 2138] 104 2138 55601 1184 160 1024 0 90112 192 0 rsyslogd [22471.723116] [ 2212] 0 2212 2033 576 32 544 0 40960 256 0 screen [22471.723119] [ 2213] 0 2213 2040 608 0 608 0 45056 160 0 bash [22471.723123] [ 3567] 0 3567 1996 608 64 544 0 40960 160 0 screen [22471.723132] [ 3568] 0 3568 2041 896 192 704 0 53248 0 0 bash [22471.723139] [ 4804] 0 4804 1961 512 0 512 0 40960 224 0 screen [22471.723144] [ 4805] 0 4805 2041 576 32 544 0 49152 160 0 bash [22471.723147] [ 5361] 0 5361 1961 480 0 480 0 49152 192 0 screen [22471.723149] [ 5362] 0 5362 2041 576 64 512 0 49152 160 0 bash [22471.723151] [ 5385] 0 5385 2241 1280 64 1216 0 57344 0 0 arpflood [22471.723166] [ 5389] 0 5389 2241 1248 64 1184 0 53248 0 0 arpflood [22471.723168] [ 5921] 0 5921 1928 512 32 480 0 45056 160 0 screen [22471.723174] [ 5922] 0 5922 2041 544 32 512 0 53248 160 0 bash [22471.723182] [ 5928] 0 5928 2241 1216 64 1152 0 57344 0 0 arpflood [22471.723187] [ 5951] 0 5951 1961 480 0 480 0 49152 224 0 screen [22471.723201] [ 5952] 0 5952 2041 640 96 544 0 49152 128 0 bash [22471.723204] [ 5963] 0 5963 1993 544 96 448 0 45056 160 0 screen [22471.723206] [ 5964] 0 5964 2041 608 32 576 0 53248 160 0 bash [22471.723208] [ 6064] 0 6064 1961 448 0 448 0 49152 224 0 screen [22471.723214] [ 6065] 0 6065 2121 832 128 704 0 53248 160 0 bash [22471.723219] [ 6900] 0 6900 1961 512 32 480 0 45056 160 0 screen [22471.723221] [ 6901] 0 6901 2087 896 160 736 0 57344 32 0 bash [22471.723223] [ 6916] 0 6916 1961 512 32 480 0 40960 160 0 screen [22471.723225] [ 6917] 0 6917 2084 864 192 672 0 57344 96 0 bash [22471.723228] [ 8141] 0 8141 49862 320 0 320 0 172032 0 0 ct [22471.723231] [ 9210] 0 9210 1573558 1573088 1572768 320 0 12648448 0 0 memt [22471.723248] [ 9246] 0 9246 1589430 250395 249659 736 0 11841536 1214465 0 php [22471.723252] [ 9631] 0 9631 24532 864 96 768 0 94208 640 0 curl [22471.723255] [ 9638] 0 9638 24532 832 96 736 0 94208 640 0 curl [22471.723257] [ 9663] 0 9663 24532 832 128 704 0 90112 640 0 curl [22471.723259] [ 9665] 0 9665 53998 640 64 576 0 106496 160 0 fio [22471.723261] [ 9667] 0 9667 1091336 756214 755926 288 0 8511488 297120 0 fio [22471.723264] [ 9668] 0 9668 1091337 490038 489782 256 0 8511488 563264 0 fio [22471.723269] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/user.slice/user-0.slice/session-3.scope,task=memt,pid=9210,uid=0 [22471.723310] Out of memory: Killed process 9210 (memt) total-vm:6294232kB, anon-rss:6291072kB, file-rss:1280kB, shmem-rss:0kB, UID:0 pgtables:12352kB oom_score_adj:0 [61796.908278] INFO: task jbd2/sda1-8:377 blocked for more than 30 seconds. [61796.909531] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8 [61796.910820] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [61796.912369] task:jbd2/sda1-8 state:D stack:0 pid:377 tgid:377 ppid:2 flags:0x00004000 [61796.912389] Call Trace: [61796.912399] <TASK> [61796.912426] __schedule+0x5e7/0x14a0 [61796.912487] ? submit_bio_noacct_nocheck+0x2cc/0x360 [61796.912508] schedule+0x8e/0x160 [61796.912512] bit_wait_io+0x45/0xa0 [61796.912518] __wait_on_bit+0xe1/0x170 [61796.912521] ? bit_wait_io_timeout+0xc0/0xc0 [61796.912524] jbd2_journal_commit_transaction+0x191b/0x2320 [61796.912547] ? wake_up_bit+0xd0/0xd0 [61796.912564] kjournald2+0x120/0x460 [61796.912577] ? init_wait_entry+0x30/0x30 [61796.912580] kthread+0x1f9/0x230 [61796.912594] ? jbd2_journal_update_sb_log_tail+0x180/0x180 [61796.912599] ? kthreadd+0x380/0x380 [61796.912601] ret_from_fork+0x30/0x40 [61796.912620] ? kthreadd+0x380/0x380 [61796.912622] ret_from_fork_asm+0x11/0x20 [61796.912665] </TASK> [61875.585039] show_signal_msg: 8 callbacks suppressed [61875.585050] cq[14120]: segfault at 7f611add1000 ip 000055f05765c4e8 sp 00007f61193ffdb0 error 4 in cq[55f05765c000+1000] likely on CPU 2 (core 2, socket 0) [61875.585106] Code: 89 d1 48 d3 e6 49 09 34 c4 e8 64 fc ff ff 4c 89 e2 be 80 00 00 00 48 c1 eb 02 48 89 c7 e8 10 fc ff ff 89 d9 48 89 ef 0f 57 c9 <0f> 28 07 0f 58 c8 0f 29 0f 48 83 c7 10 e2 f1 eb ef 48 8b 84 24 88 [61887.147744] INFO: task jbd2/sda1-8:377 blocked for more than 30 seconds. [61887.148999] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8 [61887.150213] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [61887.151554] task:jbd2/sda1-8 state:D stack:0 pid:377 tgid:377 ppid:2 flags:0x00004000 [61887.151561] Call Trace: [61887.151563] <TASK> [61887.151576] __schedule+0x5e7/0x14a0 [61887.151586] ? submit_bio_noacct_nocheck+0x2cc/0x360 [61887.151591] schedule+0x8e/0x160 [61887.151594] bit_wait_io+0x45/0xa0 [61887.151597] __wait_on_bit+0xe1/0x170 [61887.151600] ? bit_wait_io_timeout+0xc0/0xc0 [61887.151604] jbd2_journal_commit_transaction+0x191b/0x2320 [61887.151609] ? wake_up_bit+0xd0/0xd0 [61887.151613] kjournald2+0x120/0x460 [61887.151618] ? init_wait_entry+0x30/0x30 [61887.151621] kthread+0x1f9/0x230 [61887.151624] ? jbd2_journal_update_sb_log_tail+0x180/0x180 [61887.151628] ? kthreadd+0x380/0x380 [61887.151630] ret_from_fork+0x30/0x40 [61887.151634] ? kthreadd+0x380/0x380 [61887.151636] ret_from_fork_asm+0x11/0x20 [61887.151641] </TASK> [61887.151644] INFO: task journal-offline:14113 blocked for more than 30 seconds. [61887.153004] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8 [61887.154263] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [61887.157148] task:journal-offline state:D stack:0 pid:14113 tgid:451 ppid:1 flags:0x00004002 [61887.157158] Call Trace: [61887.157162] <TASK> [61887.157170] __schedule+0x5e7/0x14a0 [61887.157185] ? wait_for_common_io+0x171/0x190 [61887.157191] ? wbc_detach_inode+0x103/0x230 [61887.157213] ? prepare_to_wait_event+0x125/0x140 [61887.157218] schedule+0x8e/0x160 [61887.157221] jbd2_log_wait_commit+0x1b7/0x200 [61887.157230] ? init_wait_entry+0x30/0x30 [61887.157232] ext4_sync_file+0x47c/0x590 [61887.157254] __x64_sys_fsync+0xb0/0x100 [61887.157268] do_syscall_64+0x50/0x110 [61887.157293] entry_SYSCALL_64_after_hwframe+0x24/0x2c [61887.157311] RIP: 0033:0x7f923d11b8ab [61887.157319] RSP: 002b:00007f923b7ffc50 EFLAGS: 00000293 ORIG_RAX: 000000000000004a [61887.157342] RAX: ffffffffffffffda RBX: 000055c68b01c370 RCX: 00007f923d11b8ab [61887.157344] RDX: 0000000000000000 RSI: 00007f923d64ea34 RDI: 0000000000000019 [61887.157346] RBP: 00007f923d6502c0 R08: 0000000000000000 R09: 00007ffc269604cf [61887.157348] R10: 0000000000000a34 R11: 0000000000000293 R12: 0000000000000002 [61887.157349] R13: 0000000000000002 R14: 00007f923d0947d0 R15: 00007ffc26960510 [61887.157360] </TASK> [63099.559295] traps: cq[14344] trap invalid opcode ip:564f9853d3aa sp:7f779bbffc20 error:0 [63099.559294] traps: cq[14343] trap invalid opcode ip:564f9853d3aa sp:7f779c5ffc20 error:0 [63099.559348] in cq[564f9853d000+1000] [63099.559348] in cq[564f9853d000+1000] [63421.219463] INFO: task jbd2/sda1-8:377 blocked for more than 30 seconds. [63421.220972] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8 [63421.222346] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [63421.223707] task:jbd2/sda1-8 state:D stack:0 pid:377 tgid:377 ppid:2 flags:0x00004000 [63421.223728] Call Trace: [63421.223749] <TASK> [63421.223777] __schedule+0x5e7/0x14a0 [63421.223860] ? submit_bio_noacct_nocheck+0x2cc/0x360 [63421.223882] schedule+0x8e/0x160 [63421.223886] bit_wait_io+0x45/0xa0 [63421.223892] __wait_on_bit+0xe1/0x170 [63421.223895] ? bit_wait_io_timeout+0xc0/0xc0 [63421.223899] jbd2_journal_commit_transaction+0x191b/0x2320 [63421.223920] ? wake_up_bit+0xd0/0xd0 [63421.223938] kjournald2+0x120/0x460 [63421.223951] ? init_wait_entry+0x30/0x30 [63421.223955] kthread+0x1f9/0x230 [63421.223969] ? jbd2_journal_update_sb_log_tail+0x180/0x180 [63421.223974] ? kthreadd+0x380/0x380 [63421.223976] ret_from_fork+0x30/0x40 [63421.224007] ? kthreadd+0x380/0x380 [63421.224009] ret_from_fork_asm+0x11/0x20 [63421.224024] </TASK> [68384.390473] INFO: task jbd2/sda1-8:377 blocked for more than 30 seconds. [68384.392485] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8 [68384.394706] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [68384.396367] task:jbd2/sda1-8 state:D stack:0 pid:377 tgid:377 ppid:2 flags:0x00004000 [68384.396381] Call Trace: [68384.396393] <TASK> [68384.396416] __schedule+0x5e7/0x14a0 [68384.396476] ? submit_bio_noacct_nocheck+0x2cc/0x360 [68384.396497] schedule+0x8e/0x160 [68384.396501] bit_wait_io+0x45/0xa0 [68384.396507] __wait_on_bit+0xe1/0x170 [68384.396510] ? bit_wait_io_timeout+0xc0/0xc0 [68384.396529] jbd2_journal_commit_transaction+0x191b/0x2320 [68384.396551] ? wake_up_bit+0xd0/0xd0 [68384.396568] kjournald2+0x120/0x460 [68384.396581] ? init_wait_entry+0x30/0x30 [68384.396584] kthread+0x1f9/0x230 [68384.396600] ? jbd2_journal_update_sb_log_tail+0x180/0x180 [68384.396604] ? kthreadd+0x380/0x380 [68384.396606] ret_from_fork+0x30/0x40 [68384.396626] ? kthreadd+0x380/0x380 [68384.396628] ret_from_fork_asm+0x11/0x20 [68384.396644] </TASK> [68655.118907] INFO: task jbd2/sda1-8:377 blocked for more than 30 seconds. [68655.124237] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8 [68655.125785] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [68655.128624] task:jbd2/sda1-8 state:D stack:0 pid:377 tgid:377 ppid:2 flags:0x00004000 [68655.128669] Call Trace: [68655.128683] <TASK> [68655.128718] __schedule+0x5e7/0x14a0 [68655.128801] ? submit_bio_noacct_nocheck+0x2cc/0x360 [68655.128829] schedule+0x8e/0x160 [68655.128833] bit_wait_io+0x45/0xa0 [68655.128908] __wait_on_bit+0xe1/0x170 [68655.128913] ? bit_wait_io_timeout+0xc0/0xc0 [68655.128918] jbd2_journal_commit_transaction+0x191b/0x2320 [68655.128997] ? wake_up_bit+0xd0/0xd0 [68655.129025] kjournald2+0x120/0x460 [68655.129048] ? init_wait_entry+0x30/0x30 [68655.129053] kthread+0x1f9/0x230 [68655.129080] ? jbd2_journal_update_sb_log_tail+0x180/0x180 [68655.129086] ? kthreadd+0x380/0x380 [68655.129089] ret_from_fork+0x30/0x40 [68655.129124] ? kthreadd+0x380/0x380 [68655.129127] ret_from_fork_asm+0x11/0x20 [68655.129156] </TASK> [68895.747345] INFO: task jbd2/sda1-8:377 blocked for more than 30 seconds. [68895.748831] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8 [68895.750303] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [68895.752488] task:jbd2/sda1-8 state:D stack:0 pid:377 tgid:377 ppid:2 flags:0x00004000 [68895.752513] Call Trace: [68895.752527] <TASK> [68895.752590] __schedule+0x5e7/0x14a0 [68895.752671] ? submit_bio_noacct_nocheck+0x2cc/0x360 [68895.752699] schedule+0x8e/0x160 [68895.752703] bit_wait_io+0x45/0xa0 [68895.752712] __wait_on_bit+0xe1/0x170 [68895.752716] ? bit_wait_io_timeout+0xc0/0xc0 [68895.752739] jbd2_journal_commit_transaction+0x191b/0x2320 [68895.752768] ? wake_up_bit+0xd0/0xd0 [68895.752792] kjournald2+0x120/0x460 [68895.752809] ? init_wait_entry+0x30/0x30 [68895.752813] kthread+0x1f9/0x230 [68895.752833] ? jbd2_journal_update_sb_log_tail+0x180/0x180 [68895.752839] ? kthreadd+0x380/0x380 [68895.752842] ret_from_fork+0x30/0x40 [68895.752869] ? kthreadd+0x380/0x380 [68895.752872] ret_from_fork_asm+0x11/0x20 [68895.752892] </TASK> [69677.822593] INFO: task jbd2/sda1-8:377 blocked for more than 30 seconds. [69677.824222] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8 [69677.826159] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [69677.828000] task:jbd2/sda1-8 state:D stack:0 pid:377 tgid:377 ppid:2 flags:0x00004000 [69677.828026] Call Trace: [69677.828040] <TASK> [69677.828076] __schedule+0x5e7/0x14a0 [69677.828159] ? submit_bio_noacct_nocheck+0x2cc/0x360 [69677.828187] schedule+0x8e/0x160 [69677.828191] bit_wait_io+0x45/0xa0 [69677.828199] __wait_on_bit+0xe1/0x170 [69677.828204] ? bit_wait_io_timeout+0xc0/0xc0 [69677.828208] jbd2_journal_commit_transaction+0x191b/0x2320 [69677.828239] ? wake_up_bit+0xd0/0xd0 [69677.828263] kjournald2+0x120/0x460 [69677.828280] ? init_wait_entry+0x30/0x30 [69677.828299] kthread+0x1f9/0x230 [69677.828321] ? jbd2_journal_update_sb_log_tail+0x180/0x180 [69677.828327] ? kthreadd+0x380/0x380 [69677.828330] ret_from_fork+0x30/0x40 [69677.828357] ? kthreadd+0x380/0x380 [69677.828360] ret_from_fork_asm+0x11/0x20 [69677.828381] </TASK> [69677.828385] INFO: task journal-offline:18592 blocked for more than 30 seconds. [69677.831042] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8 [69677.833149] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [69677.835422] task:journal-offline state:D stack:0 pid:18592 tgid:451 ppid:1 flags:0x00004002 [69677.835430] Call Trace: [69677.835432] <TASK> [69677.835436] __schedule+0x5e7/0x14a0 [69677.835444] ? wait_for_common_io+0x171/0x190 [69677.835449] ? prepare_to_wait_event+0x125/0x140 [69677.835454] schedule+0x8e/0x160 [69677.835457] jbd2_log_wait_commit+0x1b7/0x200 [69677.835464] ? init_wait_entry+0x30/0x30 [69677.835468] ext4_sync_file+0x47c/0x590 [69677.835513] __x64_sys_fsync+0xb0/0x100 [69677.835542] do_syscall_64+0x50/0x110 [69677.835569] entry_SYSCALL_64_after_hwframe+0x24/0x2c [69677.835593] RIP: 0033:0x7f923d11b8ab [69677.835602] RSP: 002b:00007f923b7ffc50 EFLAGS: 00000293 ORIG_RAX: 000000000000004a [69677.835606] RAX: ffffffffffffffda RBX: 000055c68b01c370 RCX: 00007f923d11b8ab [69677.835609] RDX: 0000000000000000 RSI: 00007f923d64ea34 RDI: 0000000000000019 [69677.835632] RBP: 00007f923d6502c0 R08: 0000000000000000 R09: 00007ffc269604cf [69677.835634] R10: 0000000000000a34 R11: 0000000000000293 R12: 0000000000000002 [69677.835640] R13: 0000000000000002 R14: 00007f923d0947d0 R15: 00007ffc26960510 [69677.835651] </TASK> [69707.922952] INFO: task jbd2/sda1-8:377 blocked for more than 60 seconds. [69707.926000] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8 [69707.928249] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [69707.930623] task:jbd2/sda1-8 state:D stack:0 pid:377 tgid:377 ppid:2 flags:0x00004000 [69707.930632] Call Trace: [69707.930640] <TASK> [69707.930655] __schedule+0x5e7/0x14a0 [69707.930666] ? submit_bio_noacct_nocheck+0x2cc/0x360 [69707.930673] schedule+0x8e/0x160 [69707.930677] bit_wait_io+0x45/0xa0 [69707.930682] __wait_on_bit+0xe1/0x170 [69707.930686] ? bit_wait_io_timeout+0xc0/0xc0 [69707.930691] jbd2_journal_commit_transaction+0x191b/0x2320 [69707.930698] ? wake_up_bit+0xd0/0xd0 [69707.930705] kjournald2+0x120/0x460 [69707.930711] ? init_wait_entry+0x30/0x30 [69707.930715] kthread+0x1f9/0x230 [69707.930719] ? jbd2_journal_update_sb_log_tail+0x180/0x180 [69707.930725] ? kthreadd+0x380/0x380 [69707.930728] ret_from_fork+0x30/0x40 [69707.930733] ? kthreadd+0x380/0x380 [69707.930736] ret_from_fork_asm+0x11/0x20 [69707.930743] </TASK> [69707.930744] Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings [69794.689696] systemd[1]: systemd-journald.service: Main process exited, code=dumped, status=6/ABRT [69794.689877] systemd[1]: systemd-journald.service: Failed with result 'watchdog'. [69794.691811] systemd[1]: systemd-journald.service: Consumed 5.443s CPU time. [69794.693515] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [69794.693939] systemd[1]: Stopped Journal Service. [69794.693993] systemd[1]: systemd-journald.service: Consumed 5.443s CPU time. [69794.813640] systemd[1]: Starting Journal Service... [69846.623844] systemd[1]: Started Journal Service. -- Ammar Faizi ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re:[## 385523 ##] Disk I/O di Neo Lite kembali bermasalah 2024-07-12 13:40 ` Ammar Faizi @ 2024-07-12 14:12 ` Ammar Rafli 0 siblings, 0 replies; 22+ messages in thread From: Ammar Rafli @ 2024-07-12 14:12 UTC (permalink / raw) To: Ammar Faizi Cc: support, ammar rafli arkan, alviro iskandar setiawan, gnu/weeb mailing list, paber panjaitan, yusuf hadiwinata [-- Attachment #1: Type: text/plain, Size: 24490 bytes --] Hi Bapak Ammar Selamat Malam Menanggapi email yang disampaikan, Baik, terkait hal tersebut akan kami koordinasikan dahulu ya Pak. Mohon kesediannya untuk menunggu informasi update dari sisi kami. Sekian informasi yang disampaikan. Terima kasih atas perhatian dan kerjasamanya. Penilaian Anda sangat membantu perbaikan dan evaluasi pelayanan kami. Ketika tiket pengaduan Anda kami tutup akan ada otomatis link survei kepuasan pelanggan di mailbox Anda. Mohon untuk memberikan penilaian Anda. Regards Ammar Rafli Arkan Product Operation Support ----------------------------------------------------------- PT. Biznet Gio Nusantara MidPlaza 1, 7th Floor. Jl. Jend. Sudirman Kav 10 - 11. Jakarta 10220 - Indonesia. ----------------------------------------------------------- Click here to submit your feature request. The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. ---- on Fri, 12 Jul 2024 20:40:15 +0700 "Ammar Faizi"<[email protected]> wrote ---- On Fri, Jul 12, 2024 at 01:31:30PM +0000, Yusuf Hadiwinata wrote: > Thnks you > > Mau memastikan, apakah sebelum terjadi Blocking Thread ada OOM Iya ada, saya juga stress test memory, sempat hit OOM killer juga. Untuk testing memory saya pakai simple PHP script yang memakan memory sekitar 11 GB seperti ini: <?php $nr = 1024*1024*1024*5; $a = str_repeat("a", $nr); echo "ar\n"; $b = str_repeat("b", $nr); echo "br\n"; while (1) { $tmp = ""; for ($i = 0; $i < $nr; $i++) { $tmp .= $a[$i]; } echo "A done!\n"; $tmp .= ""; for ($i = 0; $i < $nr; $i++) { $tmp .= $b[$i]; } echo "B done!\n"; sleep(5); } ---- dmesg: [15832.976830] Out of memory: Killed process 7142 (php) total-vm:12649176kB, anon-rss:4339492kB, file-rss:3200kB, shmem-rss:0kB, UID:0 pgtables:24720kB oom_score_adj:0 [15836.351730] oom_reaper: reaped process 7142 (php), now anon-rss:636kB, file-rss:796kB, shmem-rss:0kB [22471.722388] ct invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0 [22471.722420] CPU: 7 PID: 8149 Comm: ct Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8 5aa4ec4a5a8034d921df04c799edf17a90d1ec90 [22471.722434] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.1-0-g3208b098f51a-prebuilt.qemu.org 04/01/2014 [22471.722446] Call Trace: [22471.722508] <TASK> [22471.722543] dump_header+0x50/0x170 [22471.722612] oom_kill_process+0x1c0/0x5b0 [22471.722624] out_of_memory+0x7af/0xdf0 [22471.722628] __alloc_pages_slowpath+0xba1/0x1210 [22471.722648] __alloc_pages+0x23d/0x330 [22471.722657] alloc_pages_mpol+0xf4/0x1a0 [22471.722670] filemap_alloc_folio+0xa3/0x1d0 [22471.722690] __filemap_get_folio+0x195/0x480 [22471.722694] filemap_fault+0x212/0x830 [22471.722700] ? __slab_free+0x5a/0x1a0 [22471.722704] __do_fault+0x3b/0x120 [22471.722714] do_pte_missing+0x16b/0x1060 [22471.722716] ? skb_release_data+0x1c6/0x2b0 [22471.722740] handle_mm_fault+0x93c/0x1380 [22471.722743] do_user_addr_fault+0x202/0xa10 [22471.722761] exc_page_fault+0x53/0xc0 [22471.722788] asm_exc_page_fault+0x22/0x30 [22471.722802] RIP: 0033:0x557bbad655bb [22471.722814] Code: Unable to access opcode bytes at 0x557bbad65591. [22471.722815] RSP: 002b:00007fbd6e9ffc48 EFLAGS: 00010206 [22471.722818] RAX: 0000000000000012 RBX: 00007fbd6ea00640 RCX: 0000000000000001 [22471.722820] RDX: 0000000000000000 RSI: 00007fbd70000b70 RDI: 00007fbd6e9ff8f0 [22471.722821] RBP: 00007fbd6e9ffe48 R08: 0000000000000000 R09: 00007fbd6e9ffd27 [22471.722823] R10: 0000000000000000 R11: 0000000000000293 R12: 00007fbd6ea00640 [22471.722824] R13: 0000000000000000 R14: 00007fbd774947d0 R15: 00007ffdf6585af0 [22471.722826] </TASK> [22471.722827] Mem-Info: [22471.722835] active_anon:1046703 inactive_anon:469261 isolated_anon:96 active_file:0 inactive_file:41 isolated_file:0 unevictable:1579777 dirty:3 writeback:0 slab_reclaimable:6224 slab_unreclaimable:70607 mapped:3882 shmem:18 pagetables:11208 sec_pagetables:0 bounce:0 kernel_misc_reclaimable:0 free:66954 free_pcp:0 free_cma:0 [22471.722840] Node 0 active_anon:4186812kB inactive_anon:1877044kB active_file:0kB inactive_file:164kB unevictable:6319108kB isolated(anon):384kB isolated(file):0kB mapped:15528kB dirty:12kB writeback:0kB shmem:72kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:0kB writeback_tmp:0kB kernel_stack:5328kB pagetables:44832kB sec_pagetables:0kB all_unreclaimable? no [22471.722867] Node 0 DMA free:15360kB boost:0kB min:60kB low:72kB high:84kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [22471.722878] lowmem_reserve[]: 0 2856 15884 15884 15884 [22471.722882] Node 0 DMA32 free:64128kB boost:0kB min:12144kB low:15180kB high:18216kB reserved_highatomic:0KB active_anon:472036kB inactive_anon:161520kB active_file:48kB inactive_file:0kB unevictable:2115584kB writepending:0kB present:3129192kB managed:2989864kB mlocked:2115584kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [22471.722889] lowmem_reserve[]: 0 0 13027 13027 13027 [22471.722892] Node 0 Normal free:188328kB boost:0kB min:55372kB low:69212kB high:83052kB reserved_highatomic:133120KB active_anon:3714332kB inactive_anon:1716504kB active_file:300kB inactive_file:592kB unevictable:4203524kB writepending:12kB present:13631488kB managed:13339936kB mlocked:4203524kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [22471.722899] lowmem_reserve[]: 0 0 0 0 0 [22471.722903] Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15360kB [22471.722964] Node 0 DMA32: 123*4kB (UME) 46*8kB (UME) 48*16kB (UE) 20*32kB (UE) 27*64kB (UE) 29*128kB (UME) 41*256kB (UME) 18*512kB (UME) 6*1024kB (UME) 1*2048kB (M) 7*4096kB (UM) = 64284kB [22471.722977] Node 0 Normal: 118*4kB (UME) 111*8kB (UME) 135*16kB (UM) 25*32kB (UM) 2330*64kB (UM) 272*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 188256kB [22471.722994] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [22471.722996] 22547 total pagecache pages [22471.722997] 20249 pages in swap cache [22471.722998] Free swap = 0kB [22471.722998] Total swap = 8388604kB [22471.723002] 4194168 pages RAM [22471.723003] 0 pages HighMem/MovableOnly [22471.723004] 107878 pages reserved [22471.723007] 0 pages hwpoisoned [22471.723008] Tasks state (memory values in pages): [22471.723012] [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name [22471.723021] [ 451] 0 451 9889 992 128 864 0 102400 192 -250 systemd-journal [22471.723026] [ 485] 0 485 72338 6880 4640 2240 0 110592 0 -1000 multipathd [22471.723034] [ 489] 0 489 6527 858 96 762 0 77824 416 -1000 systemd-udevd [22471.723037] [ 651] 103 651 22341 1088 96 992 0 73728 96 0 systemd-timesyn [22471.723040] [ 694] 100 694 4032 928 128 800 0 65536 128 0 systemd-network [22471.723043] [ 697] 101 697 6385 1082 218 864 0 81920 864 0 systemd-resolve [22471.723045] [ 734] 0 734 1822 768 64 704 0 45056 0 0 cron [22471.723048] [ 736] 102 736 2189 1184 224 960 0 49152 32 -900 dbus-daemon [22471.723050] [ 749] 0 749 20705 832 0 832 0 61440 96 0 irqbalance [22471.723055] [ 751] 0 751 8296 862 30 832 0 98304 2144 0 networkd-dispat [22471.723058] [ 752] 0 752 58626 992 96 896 0 94208 64 0 polkitd [22471.723060] [ 755] 0 755 20013 768 32 736 0 57344 0 0 qemu-ga [22471.723063] [ 761] 0 761 588565 1194 650 544 0 450560 2176 -900 snapd [22471.723065] [ 764] 0 764 3883 992 128 864 0 65536 128 0 systemd-logind [22471.723067] [ 766] 0 766 98156 1216 192 1024 0 139264 416 0 udisksd [22471.723083] [ 772] 0 772 1555 512 32 480 0 45056 0 0 agetty [22471.723085] [ 782] 0 782 1544 480 32 448 0 45056 0 0 agetty [22471.723090] [ 806] 0 806 27526 896 32 864 0 122880 1984 0 unattended-upgr [22471.723093] [ 826] 0 826 61061 1024 96 928 0 106496 352 0 ModemManager [22471.723095] [ 827] 0 827 3859 1056 96 960 0 61440 288 -1000 sshd [22471.723098] [ 1098] 0 1098 4278 768 64 704 0 73728 352 0 systemd [22471.723103] [ 1099] 0 1099 42339 430 78 352 0 90112 864 0 (sd-pam) [22471.723106] [ 1680] 0 1680 74007 1056 128 928 0 155648 672 0 packagekitd [22471.723108] [ 2138] 104 2138 55601 1184 160 1024 0 90112 192 0 rsyslogd [22471.723116] [ 2212] 0 2212 2033 576 32 544 0 40960 256 0 screen [22471.723119] [ 2213] 0 2213 2040 608 0 608 0 45056 160 0 bash [22471.723123] [ 3567] 0 3567 1996 608 64 544 0 40960 160 0 screen [22471.723132] [ 3568] 0 3568 2041 896 192 704 0 53248 0 0 bash [22471.723139] [ 4804] 0 4804 1961 512 0 512 0 40960 224 0 screen [22471.723144] [ 4805] 0 4805 2041 576 32 544 0 49152 160 0 bash [22471.723147] [ 5361] 0 5361 1961 480 0 480 0 49152 192 0 screen [22471.723149] [ 5362] 0 5362 2041 576 64 512 0 49152 160 0 bash [22471.723151] [ 5385] 0 5385 2241 1280 64 1216 0 57344 0 0 arpflood [22471.723166] [ 5389] 0 5389 2241 1248 64 1184 0 53248 0 0 arpflood [22471.723168] [ 5921] 0 5921 1928 512 32 480 0 45056 160 0 screen [22471.723174] [ 5922] 0 5922 2041 544 32 512 0 53248 160 0 bash [22471.723182] [ 5928] 0 5928 2241 1216 64 1152 0 57344 0 0 arpflood [22471.723187] [ 5951] 0 5951 1961 480 0 480 0 49152 224 0 screen [22471.723201] [ 5952] 0 5952 2041 640 96 544 0 49152 128 0 bash [22471.723204] [ 5963] 0 5963 1993 544 96 448 0 45056 160 0 screen [22471.723206] [ 5964] 0 5964 2041 608 32 576 0 53248 160 0 bash [22471.723208] [ 6064] 0 6064 1961 448 0 448 0 49152 224 0 screen [22471.723214] [ 6065] 0 6065 2121 832 128 704 0 53248 160 0 bash [22471.723219] [ 6900] 0 6900 1961 512 32 480 0 45056 160 0 screen [22471.723221] [ 6901] 0 6901 2087 896 160 736 0 57344 32 0 bash [22471.723223] [ 6916] 0 6916 1961 512 32 480 0 40960 160 0 screen [22471.723225] [ 6917] 0 6917 2084 864 192 672 0 57344 96 0 bash [22471.723228] [ 8141] 0 8141 49862 320 0 320 0 172032 0 0 ct [22471.723231] [ 9210] 0 9210 1573558 1573088 1572768 320 0 12648448 0 0 memt [22471.723248] [ 9246] 0 9246 1589430 250395 249659 736 0 11841536 1214465 0 php [22471.723252] [ 9631] 0 9631 24532 864 96 768 0 94208 640 0 curl [22471.723255] [ 9638] 0 9638 24532 832 96 736 0 94208 640 0 curl [22471.723257] [ 9663] 0 9663 24532 832 128 704 0 90112 640 0 curl [22471.723259] [ 9665] 0 9665 53998 640 64 576 0 106496 160 0 fio [22471.723261] [ 9667] 0 9667 1091336 756214 755926 288 0 8511488 297120 0 fio [22471.723264] [ 9668] 0 9668 1091337 490038 489782 256 0 8511488 563264 0 fio [22471.723269] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/user.slice/user-0.slice/session-3.scope,task=memt,pid=9210,uid=0 [22471.723310] Out of memory: Killed process 9210 (memt) total-vm:6294232kB, anon-rss:6291072kB, file-rss:1280kB, shmem-rss:0kB, UID:0 pgtables:12352kB oom_score_adj:0 [61796.908278] INFO: task jbd2/sda1-8:377 blocked for more than 30 seconds. [61796.909531] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8 [61796.910820] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [61796.912369] task:jbd2/sda1-8 state:D stack:0 pid:377 tgid:377 ppid:2 flags:0x00004000 [61796.912389] Call Trace: [61796.912399] <TASK> [61796.912426] __schedule+0x5e7/0x14a0 [61796.912487] ? submit_bio_noacct_nocheck+0x2cc/0x360 [61796.912508] schedule+0x8e/0x160 [61796.912512] bit_wait_io+0x45/0xa0 [61796.912518] __wait_on_bit+0xe1/0x170 [61796.912521] ? bit_wait_io_timeout+0xc0/0xc0 [61796.912524] jbd2_journal_commit_transaction+0x191b/0x2320 [61796.912547] ? wake_up_bit+0xd0/0xd0 [61796.912564] kjournald2+0x120/0x460 [61796.912577] ? init_wait_entry+0x30/0x30 [61796.912580] kthread+0x1f9/0x230 [61796.912594] ? jbd2_journal_update_sb_log_tail+0x180/0x180 [61796.912599] ? kthreadd+0x380/0x380 [61796.912601] ret_from_fork+0x30/0x40 [61796.912620] ? kthreadd+0x380/0x380 [61796.912622] ret_from_fork_asm+0x11/0x20 [61796.912665] </TASK> [61875.585039] show_signal_msg: 8 callbacks suppressed [61875.585050] cq[14120]: segfault at 7f611add1000 ip 000055f05765c4e8 sp 00007f61193ffdb0 error 4 in cq[55f05765c000+1000] likely on CPU 2 (core 2, socket 0) [61875.585106] Code: 89 d1 48 d3 e6 49 09 34 c4 e8 64 fc ff ff 4c 89 e2 be 80 00 00 00 48 c1 eb 02 48 89 c7 e8 10 fc ff ff 89 d9 48 89 ef 0f 57 c9 <0f> 28 07 0f 58 c8 0f 29 0f 48 83 c7 10 e2 f1 eb ef 48 8b 84 24 88 [61887.147744] INFO: task jbd2/sda1-8:377 blocked for more than 30 seconds. [61887.148999] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8 [61887.150213] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [61887.151554] task:jbd2/sda1-8 state:D stack:0 pid:377 tgid:377 ppid:2 flags:0x00004000 [61887.151561] Call Trace: [61887.151563] <TASK> [61887.151576] __schedule+0x5e7/0x14a0 [61887.151586] ? submit_bio_noacct_nocheck+0x2cc/0x360 [61887.151591] schedule+0x8e/0x160 [61887.151594] bit_wait_io+0x45/0xa0 [61887.151597] __wait_on_bit+0xe1/0x170 [61887.151600] ? bit_wait_io_timeout+0xc0/0xc0 [61887.151604] jbd2_journal_commit_transaction+0x191b/0x2320 [61887.151609] ? wake_up_bit+0xd0/0xd0 [61887.151613] kjournald2+0x120/0x460 [61887.151618] ? init_wait_entry+0x30/0x30 [61887.151621] kthread+0x1f9/0x230 [61887.151624] ? jbd2_journal_update_sb_log_tail+0x180/0x180 [61887.151628] ? kthreadd+0x380/0x380 [61887.151630] ret_from_fork+0x30/0x40 [61887.151634] ? kthreadd+0x380/0x380 [61887.151636] ret_from_fork_asm+0x11/0x20 [61887.151641] </TASK> [61887.151644] INFO: task journal-offline:14113 blocked for more than 30 seconds. [61887.153004] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8 [61887.154263] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [61887.157148] task:journal-offline state:D stack:0 pid:14113 tgid:451 ppid:1 flags:0x00004002 [61887.157158] Call Trace: [61887.157162] <TASK> [61887.157170] __schedule+0x5e7/0x14a0 [61887.157185] ? wait_for_common_io+0x171/0x190 [61887.157191] ? wbc_detach_inode+0x103/0x230 [61887.157213] ? prepare_to_wait_event+0x125/0x140 [61887.157218] schedule+0x8e/0x160 [61887.157221] jbd2_log_wait_commit+0x1b7/0x200 [61887.157230] ? init_wait_entry+0x30/0x30 [61887.157232] ext4_sync_file+0x47c/0x590 [61887.157254] __x64_sys_fsync+0xb0/0x100 [61887.157268] do_syscall_64+0x50/0x110 [61887.157293] entry_SYSCALL_64_after_hwframe+0x24/0x2c [61887.157311] RIP: 0033:0x7f923d11b8ab [61887.157319] RSP: 002b:00007f923b7ffc50 EFLAGS: 00000293 ORIG_RAX: 000000000000004a [61887.157342] RAX: ffffffffffffffda RBX: 000055c68b01c370 RCX: 00007f923d11b8ab [61887.157344] RDX: 0000000000000000 RSI: 00007f923d64ea34 RDI: 0000000000000019 [61887.157346] RBP: 00007f923d6502c0 R08: 0000000000000000 R09: 00007ffc269604cf [61887.157348] R10: 0000000000000a34 R11: 0000000000000293 R12: 0000000000000002 [61887.157349] R13: 0000000000000002 R14: 00007f923d0947d0 R15: 00007ffc26960510 [61887.157360] </TASK> [63099.559295] traps: cq[14344] trap invalid opcode ip:564f9853d3aa sp:7f779bbffc20 error:0 [63099.559294] traps: cq[14343] trap invalid opcode ip:564f9853d3aa sp:7f779c5ffc20 error:0 [63099.559348] in cq[564f9853d000+1000] [63099.559348] in cq[564f9853d000+1000] [63421.219463] INFO: task jbd2/sda1-8:377 blocked for more than 30 seconds. [63421.220972] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8 [63421.222346] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [63421.223707] task:jbd2/sda1-8 state:D stack:0 pid:377 tgid:377 ppid:2 flags:0x00004000 [63421.223728] Call Trace: [63421.223749] <TASK> [63421.223777] __schedule+0x5e7/0x14a0 [63421.223860] ? submit_bio_noacct_nocheck+0x2cc/0x360 [63421.223882] schedule+0x8e/0x160 [63421.223886] bit_wait_io+0x45/0xa0 [63421.223892] __wait_on_bit+0xe1/0x170 [63421.223895] ? bit_wait_io_timeout+0xc0/0xc0 [63421.223899] jbd2_journal_commit_transaction+0x191b/0x2320 [63421.223920] ? wake_up_bit+0xd0/0xd0 [63421.223938] kjournald2+0x120/0x460 [63421.223951] ? init_wait_entry+0x30/0x30 [63421.223955] kthread+0x1f9/0x230 [63421.223969] ? jbd2_journal_update_sb_log_tail+0x180/0x180 [63421.223974] ? kthreadd+0x380/0x380 [63421.223976] ret_from_fork+0x30/0x40 [63421.224007] ? kthreadd+0x380/0x380 [63421.224009] ret_from_fork_asm+0x11/0x20 [63421.224024] </TASK> [68384.390473] INFO: task jbd2/sda1-8:377 blocked for more than 30 seconds. [68384.392485] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8 [68384.394706] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [68384.396367] task:jbd2/sda1-8 state:D stack:0 pid:377 tgid:377 ppid:2 flags:0x00004000 [68384.396381] Call Trace: [68384.396393] <TASK> [68384.396416] __schedule+0x5e7/0x14a0 [68384.396476] ? submit_bio_noacct_nocheck+0x2cc/0x360 [68384.396497] schedule+0x8e/0x160 [68384.396501] bit_wait_io+0x45/0xa0 [68384.396507] __wait_on_bit+0xe1/0x170 [68384.396510] ? bit_wait_io_timeout+0xc0/0xc0 [68384.396529] jbd2_journal_commit_transaction+0x191b/0x2320 [68384.396551] ? wake_up_bit+0xd0/0xd0 [68384.396568] kjournald2+0x120/0x460 [68384.396581] ? init_wait_entry+0x30/0x30 [68384.396584] kthread+0x1f9/0x230 [68384.396600] ? jbd2_journal_update_sb_log_tail+0x180/0x180 [68384.396604] ? kthreadd+0x380/0x380 [68384.396606] ret_from_fork+0x30/0x40 [68384.396626] ? kthreadd+0x380/0x380 [68384.396628] ret_from_fork_asm+0x11/0x20 [68384.396644] </TASK> [68655.118907] INFO: task jbd2/sda1-8:377 blocked for more than 30 seconds. [68655.124237] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8 [68655.125785] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [68655.128624] task:jbd2/sda1-8 state:D stack:0 pid:377 tgid:377 ppid:2 flags:0x00004000 [68655.128669] Call Trace: [68655.128683] <TASK> [68655.128718] __schedule+0x5e7/0x14a0 [68655.128801] ? submit_bio_noacct_nocheck+0x2cc/0x360 [68655.128829] schedule+0x8e/0x160 [68655.128833] bit_wait_io+0x45/0xa0 [68655.128908] __wait_on_bit+0xe1/0x170 [68655.128913] ? bit_wait_io_timeout+0xc0/0xc0 [68655.128918] jbd2_journal_commit_transaction+0x191b/0x2320 [68655.128997] ? wake_up_bit+0xd0/0xd0 [68655.129025] kjournald2+0x120/0x460 [68655.129048] ? init_wait_entry+0x30/0x30 [68655.129053] kthread+0x1f9/0x230 [68655.129080] ? jbd2_journal_update_sb_log_tail+0x180/0x180 [68655.129086] ? kthreadd+0x380/0x380 [68655.129089] ret_from_fork+0x30/0x40 [68655.129124] ? kthreadd+0x380/0x380 [68655.129127] ret_from_fork_asm+0x11/0x20 [68655.129156] </TASK> [68895.747345] INFO: task jbd2/sda1-8:377 blocked for more than 30 seconds. [68895.748831] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8 [68895.750303] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [68895.752488] task:jbd2/sda1-8 state:D stack:0 pid:377 tgid:377 ppid:2 flags:0x00004000 [68895.752513] Call Trace: [68895.752527] <TASK> [68895.752590] __schedule+0x5e7/0x14a0 [68895.752671] ? submit_bio_noacct_nocheck+0x2cc/0x360 [68895.752699] schedule+0x8e/0x160 [68895.752703] bit_wait_io+0x45/0xa0 [68895.752712] __wait_on_bit+0xe1/0x170 [68895.752716] ? bit_wait_io_timeout+0xc0/0xc0 [68895.752739] jbd2_journal_commit_transaction+0x191b/0x2320 [68895.752768] ? wake_up_bit+0xd0/0xd0 [68895.752792] kjournald2+0x120/0x460 [68895.752809] ? init_wait_entry+0x30/0x30 [68895.752813] kthread+0x1f9/0x230 [68895.752833] ? jbd2_journal_update_sb_log_tail+0x180/0x180 [68895.752839] ? kthreadd+0x380/0x380 [68895.752842] ret_from_fork+0x30/0x40 [68895.752869] ? kthreadd+0x380/0x380 [68895.752872] ret_from_fork_asm+0x11/0x20 [68895.752892] </TASK> [69677.822593] INFO: task jbd2/sda1-8:377 blocked for more than 30 seconds. [69677.824222] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8 [69677.826159] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [69677.828000] task:jbd2/sda1-8 state:D stack:0 pid:377 tgid:377 ppid:2 flags:0x00004000 [69677.828026] Call Trace: [69677.828040] <TASK> [69677.828076] __schedule+0x5e7/0x14a0 [69677.828159] ? submit_bio_noacct_nocheck+0x2cc/0x360 [69677.828187] schedule+0x8e/0x160 [69677.828191] bit_wait_io+0x45/0xa0 [69677.828199] __wait_on_bit+0xe1/0x170 [69677.828204] ? bit_wait_io_timeout+0xc0/0xc0 [69677.828208] jbd2_journal_commit_transaction+0x191b/0x2320 [69677.828239] ? wake_up_bit+0xd0/0xd0 [69677.828263] kjournald2+0x120/0x460 [69677.828280] ? init_wait_entry+0x30/0x30 [69677.828299] kthread+0x1f9/0x230 [69677.828321] ? jbd2_journal_update_sb_log_tail+0x180/0x180 [69677.828327] ? kthreadd+0x380/0x380 [69677.828330] ret_from_fork+0x30/0x40 [69677.828357] ? kthreadd+0x380/0x380 [69677.828360] ret_from_fork_asm+0x11/0x20 [69677.828381] </TASK> [69677.828385] INFO: task journal-offline:18592 blocked for more than 30 seconds. [69677.831042] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8 [69677.833149] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [69677.835422] task:journal-offline state:D stack:0 pid:18592 tgid:451 ppid:1 flags:0x00004002 [69677.835430] Call Trace: [69677.835432] <TASK> [69677.835436] __schedule+0x5e7/0x14a0 [69677.835444] ? wait_for_common_io+0x171/0x190 [69677.835449] ? prepare_to_wait_event+0x125/0x140 [69677.835454] schedule+0x8e/0x160 [69677.835457] jbd2_log_wait_commit+0x1b7/0x200 [69677.835464] ? init_wait_entry+0x30/0x30 [69677.835468] ext4_sync_file+0x47c/0x590 [69677.835513] __x64_sys_fsync+0xb0/0x100 [69677.835542] do_syscall_64+0x50/0x110 [69677.835569] entry_SYSCALL_64_after_hwframe+0x24/0x2c [69677.835593] RIP: 0033:0x7f923d11b8ab [69677.835602] RSP: 002b:00007f923b7ffc50 EFLAGS: 00000293 ORIG_RAX: 000000000000004a [69677.835606] RAX: ffffffffffffffda RBX: 000055c68b01c370 RCX: 00007f923d11b8ab [69677.835609] RDX: 0000000000000000 RSI: 00007f923d64ea34 RDI: 0000000000000019 [69677.835632] RBP: 00007f923d6502c0 R08: 0000000000000000 R09: 00007ffc269604cf [69677.835634] R10: 0000000000000a34 R11: 0000000000000293 R12: 0000000000000002 [69677.835640] R13: 0000000000000002 R14: 00007f923d0947d0 R15: 00007ffc26960510 [69677.835651] </TASK> [69707.922952] INFO: task jbd2/sda1-8:377 blocked for more than 60 seconds. [69707.926000] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8 [69707.928249] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [69707.930623] task:jbd2/sda1-8 state:D stack:0 pid:377 tgid:377 ppid:2 flags:0x00004000 [69707.930632] Call Trace: [69707.930640] <TASK> [69707.930655] __schedule+0x5e7/0x14a0 [69707.930666] ? submit_bio_noacct_nocheck+0x2cc/0x360 [69707.930673] schedule+0x8e/0x160 [69707.930677] bit_wait_io+0x45/0xa0 [69707.930682] __wait_on_bit+0xe1/0x170 [69707.930686] ? bit_wait_io_timeout+0xc0/0xc0 [69707.930691] jbd2_journal_commit_transaction+0x191b/0x2320 [69707.930698] ? wake_up_bit+0xd0/0xd0 [69707.930705] kjournald2+0x120/0x460 [69707.930711] ? init_wait_entry+0x30/0x30 [69707.930715] kthread+0x1f9/0x230 [69707.930719] ? jbd2_journal_update_sb_log_tail+0x180/0x180 [69707.930725] ? kthreadd+0x380/0x380 [69707.930728] ret_from_fork+0x30/0x40 [69707.930733] ? kthreadd+0x380/0x380 [69707.930736] ret_from_fork_asm+0x11/0x20 [69707.930743] </TASK> [69707.930744] Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings [69794.689696] systemd[1]: systemd-journald.service: Main process exited, code=dumped, status=6/ABRT [69794.689877] systemd[1]: systemd-journald.service: Failed with result 'watchdog'. [69794.691811] systemd[1]: systemd-journald.service: Consumed 5.443s CPU time. [69794.693515] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [69794.693939] systemd[1]: Stopped Journal Service. [69794.693993] systemd[1]: systemd-journald.service: Consumed 5.443s CPU time. [69794.813640] systemd[1]: Starting Journal Service... [69846.623844] systemd[1]: Started Journal Service. -- Ammar Faizi [-- Attachment #2: Type: text/html, Size: 32164 bytes --] ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [## 385523 ##] Disk I/O di Neo Lite kembali bermasalah 2024-07-12 13:31 ` Yusuf Hadiwinata 2024-07-12 13:40 ` Ammar Faizi @ 2024-07-12 14:45 ` Yusuf Hadiwinata 1 sibling, 0 replies; 22+ messages in thread From: Yusuf Hadiwinata @ 2024-07-12 14:45 UTC (permalink / raw) To: Ammar Faizi Cc: Support, Ammar Rafli Arkan, Alviro Iskandar Setiawan, GNU/Weeb Mailing List, Paber Panjaitan [-- Attachment #1: Type: text/plain, Size: 3922 bytes --] Dear Pak Ammar dan Team Setelah berkodinasi dengan Principal dan melalukan Analisa kami dapat memberikan kesimpulan sebagai berikut: 1. Untuk concern mengenai "task:jbd2/sda1-8 state:D" dan "jbd2_journal_commit_transaction" kami bisa sampaikan tidak terkait dengan issue sebelumnya pada VM yang sama (jika ada). 2. Masih untuk concern mengenai "task:jbd2/sda1-8 state:D" dan case tersebut hari ini, kami cukup confident ini tidak related dengan issue customer lain. 3. Kami juga sempat mengecek beberap VM Service Level Indicator kami baik yang running di host yang sama ataupun berbeda dan di cluster yang sama, tidak di temukan Issue sejenis. 4. Setelah menghubungi Principal dan Berkordinasi, tidak di temukan issue di Cluster/ Host tempat VM dengan IP 103.175.218.159, berikut beberapa area yang kami cek: * Network: CRC Error, SignalDrop, PacketDrop, bufferOverrun, DiscardPacket, KernelQueue, RecieveRingBuffer, TransmitRingBuffer, TXQueue, Interface statistic pada host dan Switch * Host: Log pada system, kernel dan services yang related serta CPU/RAM/etc Utilization * Storage: Semua hal yang berkaitan dengan Status Placement Group, status Disk, Status Cluster dll 5. Analisa dari kami dan Principal adalah sebagai berikut mengenai "task jbd2/sda1-8:377 blocked for more than 30 seconds." pada jam 17:26 * Terjadi process hanging/deadlocking/blocked di VM tersebut karena berbarengan dengan process Burn-Test / Stress-Test CPU dan RAM * Suspect kami ada pada CFQ scheduler dan RT scheduler class di kondisi CPU dan RAM yang utilisasi nya sangat tinggi, sehingga "jbd2_journal_commit_transaction" gagal dilakukan, dari Call trace nya kami jg belum menemukan warn_slowpath_* * Menurut Principal kami hal ini sering terjadi pada sistem dengan kondisi Heavy Load, mohon untuk menstop Stress-Test CPU dan RAM pada saat melakukan FIO 6. Saran dari kami: * Mohon mengaktifkan Kernel Crash Dump dan sharing vmcore nya kepada kami, sehingga pada saat terjadi lagi kami akan berusaha melakukan analisai berdasarkan vmcore tersebut. * Mohon sertakan juga output sosreport agar dapat digunakan sebagai cross reference issue yang ada * Jika berkenan, mohon untuk melakukan Update kernel ke latest version just in case concern pada email ini related dengan issue pada email ini. 7. Sebagai informasi tambahan, pada kondisi normal load sebagai mestinya (Running apps dll), seharusnya hal ini jarang terjadi, jika memang terjadi maka kemungkinan besar ada di capacity Planning, terutama pada IOPS. Untuk pilihan IOPS 5k - 15k ada pada product Neo High Performance Storage di layanan Neo Virtual Compute 8. Kami masih terus menggali kemungkinan issue lain yang berhubungan dengan issue yang dihadapi hari ini, dan jika ada temuan lebih lanjut kami akan update. 9. Di satu sisi sebagaimana yang sempat saya sampaikan, kami menemukan beberapa point suspect pada cluster lain yang akan kami investigasi lebih jauh, namun dapat dipastikan suspect issue pada cluster tersebut tidak berhugungan dengan concern yang di sampaikan pada email ini. 10. Kami juga berusaha mempelajari lebih line pada Call Trace satu persatu untuk mencari kemungkinan ada nya bugs namun sejauh ini belum menemukan yang benar2 relate. 11. Kami terbuka jika ada masukan/mengetahui sesuatu yang mungkin Pak Ammar dan Team ingin sampaikan, jika ada clue/masukan jangan sungkan untuk disampaikan di email ini ya. Terim Kasih The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. [-- Attachment #2: Type: text/html, Size: 9737 bytes --] ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [## 385523 ##] Disk I/O di Neo Lite kembali bermasalah 2024-07-12 13:06 ` Yusuf Hadiwinata 2024-07-12 13:12 ` Yusuf Hadiwinata @ 2024-07-12 13:12 ` Ammar Faizi 1 sibling, 0 replies; 22+ messages in thread From: Ammar Faizi @ 2024-07-12 13:12 UTC (permalink / raw) To: Yusuf Hadiwinata Cc: Support, Ammar Rafli Arkan, Alviro Iskandar Setiawan, GNU/Weeb Mailing List, Paber Panjaitan On Fri, Jul 12, 2024 at 8:06 PM Yusuf Hadiwinata wrote: > Kita mau analisa beberapa suspect di Host yang terdapat instance > yang di laporkan, boleh kita move instance nya ? Akan ada downtime > estimas < 5 menit jika berkenan Boleh. > Kalo ada preferensi waktu boleh di info juga kira2 enak nya jam berapaan Untuk VM dengan IP 103.175.218.159 bisa sekarang. Karena yang itu saya beli memang khusus untuk testing Neo Lite dan pembuktian masalah kemarin. Jadi tidak saya pakai untuk production. -- Ammar Faizi ^ permalink raw reply [flat|nested] 22+ messages in thread
end of thread, other threads:[~2024-07-12 14:45 UTC | newest] Thread overview: 22+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2024-07-12 11:12 Disk I/O di Neo Lite kembali bermasalah Ammar Faizi 2024-07-12 11:27 ` Re:[## 385523 ##] " Ammar Rafli 2024-07-12 11:34 ` [## " Ammar Faizi 2024-07-12 11:40 ` Ammar Rafli 2024-07-12 11:47 ` [## " Ammar Faizi 2024-07-12 11:57 ` Ammar Rafli 2024-07-12 12:09 ` [## " Ammar Faizi 2024-07-12 12:20 ` Ammar Rafli 2024-07-12 12:23 ` Ammar Rafli 2024-07-12 12:42 ` [## " Ammar Faizi 2024-07-12 13:00 ` Ammar Rafli 2024-07-12 13:19 ` [## " Ammar Faizi 2024-07-12 13:06 ` Yusuf Hadiwinata 2024-07-12 13:12 ` Yusuf Hadiwinata 2024-07-12 13:13 ` Ammar Faizi 2024-07-12 13:20 ` Yusuf Hadiwinata 2024-07-12 13:21 ` Ammar Faizi 2024-07-12 13:31 ` Yusuf Hadiwinata 2024-07-12 13:40 ` Ammar Faizi 2024-07-12 14:12 ` Ammar Rafli 2024-07-12 14:45 ` [## " Yusuf Hadiwinata 2024-07-12 13:12 ` Ammar Faizi
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox