On Fri, Jul 12, 2024 at 01:31:30PM +0000, Yusuf Hadiwinata wrote:
> Thnks you
>
> Mau memastikan, apakah sebelum terjadi Blocking Thread ada OOM

Iya ada, saya juga stress test memory, sempat hit OOM killer juga.

Untuk testing memory saya pakai simple PHP script yang memakan memory
sekitar 11 GB seperti ini:

<?php
$nr = 1024*1024*1024*5;
$a = str_repeat("a", $nr);
echo "ar\n";
$b = str_repeat("b", $nr);
echo "br\n";

while (1) {

    $tmp = "";
    for ($i = 0; $i < $nr; $i++) {
        $tmp .= $a[$i];
    }
    echo "A done!\n";

    $tmp .= "";
    for ($i = 0; $i < $nr; $i++) {
        $tmp .= $b[$i];
    }
    echo "B done!\n";

    sleep(5);
}

----
dmesg:

[15832.976830] Out of memory: Killed process 7142 (php) total-vm:12649176kB, anon-rss:4339492kB, file-rss:3200kB, shmem-rss:0kB, UID:0 pgtables:24720kB oom_score_adj:0
[15836.351730] oom_reaper: reaped process 7142 (php), now anon-rss:636kB, file-rss:796kB, shmem-rss:0kB
[22471.722388] ct invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
[22471.722420] CPU: 7 PID: 8149 Comm: ct Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8 5aa4ec4a5a8034d921df04c799edf17a90d1ec90
[22471.722434] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.1-0-g3208b098f51a-prebuilt.qemu.org 04/01/2014
[22471.722446] Call Trace:
[22471.722508] <TASK>
[22471.722543] dump_header+0x50/0x170
[22471.722612] oom_kill_process+0x1c0/0x5b0
[22471.722624] out_of_memory+0x7af/0xdf0
[22471.722628] __alloc_pages_slowpath+0xba1/0x1210
[22471.722648] __alloc_pages+0x23d/0x330
[22471.722657] alloc_pages_mpol+0xf4/0x1a0
[22471.722670] filemap_alloc_folio+0xa3/0x1d0
[22471.722690] __filemap_get_folio+0x195/0x480
[22471.722694] filemap_fault+0x212/0x830
[22471.722700] ? __slab_free+0x5a/0x1a0
[22471.722704] __do_fault+0x3b/0x120
[22471.722714] do_pte_missing+0x16b/0x1060
[22471.722716] ? skb_release_data+0x1c6/0x2b0
[22471.722740] handle_mm_fault+0x93c/0x1380
[22471.722743] do_user_addr_fault+0x202/0xa10
[22471.722761] exc_page_fault+0x53/0xc0
[22471.722788] asm_exc_page_fault+0x22/0x30
[22471.722802] RIP: 0033:0x557bbad655bb
[22471.722814] Code: Unable to access opcode bytes at 0x557bbad65591.
[22471.722815] RSP: 002b:00007fbd6e9ffc48 EFLAGS: 00010206
[22471.722818] RAX: 0000000000000012 RBX: 00007fbd6ea00640 RCX: 0000000000000001
[22471.722820] RDX: 0000000000000000 RSI: 00007fbd70000b70 RDI: 00007fbd6e9ff8f0
[22471.722821] RBP: 00007fbd6e9ffe48 R08: 0000000000000000 R09: 00007fbd6e9ffd27
[22471.722823] R10: 0000000000000000 R11: 0000000000000293 R12: 00007fbd6ea00640
[22471.722824] R13: 0000000000000000 R14: 00007fbd774947d0 R15: 00007ffdf6585af0
[22471.722826] </TASK>
[22471.722827] Mem-Info:
[22471.722835] active_anon:1046703 inactive_anon:469261 isolated_anon:96
active_file:0 inactive_file:41 isolated_file:0
unevictable:1579777 dirty:3 writeback:0
slab_reclaimable:6224 slab_unreclaimable:70607
mapped:3882 shmem:18 pagetables:11208
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:66954 free_pcp:0 free_cma:0
[22471.722840] Node 0 active_anon:4186812kB inactive_anon:1877044kB active_file:0kB inactive_file:164kB unevictable:6319108kB isolated(anon):384kB isolated(file):0kB mapped:15528kB dirty:12kB writeback:0kB shmem:72kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:0kB writeback_tmp:0kB kernel_stack:5328kB pagetables:44832kB sec_pagetables:0kB all_unreclaimable? no
[22471.722867] Node 0 DMA free:15360kB boost:0kB min:60kB low:72kB high:84kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[22471.722878] lowmem_reserve[]: 0 2856 15884 15884 15884
[22471.722882] Node 0 DMA32 free:64128kB boost:0kB min:12144kB low:15180kB high:18216kB reserved_highatomic:0KB active_anon:472036kB inactive_anon:161520kB active_file:48kB inactive_file:0kB unevictable:2115584kB writepending:0kB present:3129192kB managed:2989864kB mlocked:2115584kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[22471.722889] lowmem_reserve[]: 0 0 13027 13027 13027
[22471.722892] Node 0 Normal free:188328kB boost:0kB min:55372kB low:69212kB high:83052kB reserved_highatomic:133120KB active_anon:3714332kB inactive_anon:1716504kB active_file:300kB inactive_file:592kB unevictable:4203524kB writepending:12kB present:13631488kB managed:13339936kB mlocked:4203524kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[22471.722899] lowmem_reserve[]: 0 0 0 0 0
[22471.722903] Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15360kB
[22471.722964] Node 0 DMA32: 123*4kB (UME) 46*8kB (UME) 48*16kB (UE) 20*32kB (UE) 27*64kB (UE) 29*128kB (UME) 41*256kB (UME) 18*512kB (UME) 6*1024kB (UME) 1*2048kB (M) 7*4096kB (UM) = 64284kB
[22471.722977] Node 0 Normal: 118*4kB (UME) 111*8kB (UME) 135*16kB (UM) 25*32kB (UM) 2330*64kB (UM) 272*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 188256kB
[22471.722994] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[22471.722996] 22547 total pagecache pages
[22471.722997] 20249 pages in swap cache
[22471.722998] Free swap = 0kB
[22471.722998] Total swap = 8388604kB
[22471.723002] 4194168 pages RAM
[22471.723003] 0 pages HighMem/MovableOnly
[22471.723004] 107878 pages reserved
[22471.723007] 0 pages hwpoisoned
[22471.723008] Tasks state (memory values in pages):
[22471.723012] [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
[22471.723021] [ 451] 0 451 9889 992 128 864 0 102400 192 -250 systemd-journal
[22471.723026] [ 485] 0 485 72338 6880 4640 2240 0 110592 0 -1000 multipathd
[22471.723034] [ 489] 0 489 6527 858 96 762 0 77824 416 -1000 systemd-udevd
[22471.723037] [ 651] 103 651 22341 1088 96 992 0 73728 96 0 systemd-timesyn
[22471.723040] [ 694] 100 694 4032 928 128 800 0 65536 128 0 systemd-network
[22471.723043] [ 697] 101 697 6385 1082 218 864 0 81920 864 0 systemd-resolve
[22471.723045] [ 734] 0 734 1822 768 64 704 0 45056 0 0 cron
[22471.723048] [ 736] 102 736 2189 1184 224 960 0 49152 32 -900 dbus-daemon
[22471.723050] [ 749] 0 749 20705 832 0 832 0 61440 96 0 irqbalance
[22471.723055] [ 751] 0 751 8296 862 30 832 0 98304 2144 0 networkd-dispat
[22471.723058] [ 752] 0 752 58626 992 96 896 0 94208 64 0 polkitd
[22471.723060] [ 755] 0 755 20013 768 32 736 0 57344 0 0 qemu-ga
[22471.723063] [ 761] 0 761 588565 1194 650 544 0 450560 2176 -900 snapd
[22471.723065] [ 764] 0 764 3883 992 128 864 0 65536 128 0 systemd-logind
[22471.723067] [ 766] 0 766 98156 1216 192 1024 0 139264 416 0 udisksd
[22471.723083] [ 772] 0 772 1555 512 32 480 0 45056 0 0 agetty
[22471.723085] [ 782] 0 782 1544 480 32 448 0 45056 0 0 agetty
[22471.723090] [ 806] 0 806 27526 896 32 864 0 122880 1984 0 unattended-upgr
[22471.723093] [ 826] 0 826 61061 1024 96 928 0 106496 352 0 ModemManager
[22471.723095] [ 827] 0 827 3859 1056 96 960 0 61440 288 -1000 sshd
[22471.723098] [ 1098] 0 1098 4278 768 64 704 0 73728 352 0 systemd
[22471.723103] [ 1099] 0 1099 42339 430 78 352 0 90112 864 0 (sd-pam)
[22471.723106] [ 1680] 0 1680 74007 1056 128 928 0 155648 672 0 packagekitd
[22471.723108] [ 2138] 104 2138 55601 1184 160 1024 0 90112 192 0 rsyslogd
[22471.723116] [ 2212] 0 2212 2033 576 32 544 0 40960 256 0 screen
[22471.723119] [ 2213] 0 2213 2040 608 0 608 0 45056 160 0 bash
[22471.723123] [ 3567] 0 3567 1996 608 64 544 0 40960 160 0 screen
[22471.723132] [ 3568] 0 3568 2041 896 192 704 0 53248 0 0 bash
[22471.723139] [ 4804] 0 4804 1961 512 0 512 0 40960 224 0 screen
[22471.723144] [ 4805] 0 4805 2041 576 32 544 0 49152 160 0 bash
[22471.723147] [ 5361] 0 5361 1961 480 0 480 0 49152 192 0 screen
[22471.723149] [ 5362] 0 5362 2041 576 64 512 0 49152 160 0 bash
[22471.723151] [ 5385] 0 5385 2241 1280 64 1216 0 57344 0 0 arpflood
[22471.723166] [ 5389] 0 5389 2241 1248 64 1184 0 53248 0 0 arpflood
[22471.723168] [ 5921] 0 5921 1928 512 32 480 0 45056 160 0 screen
[22471.723174] [ 5922] 0 5922 2041 544 32 512 0 53248 160 0 bash
[22471.723182] [ 5928] 0 5928 2241 1216 64 1152 0 57344 0 0 arpflood
[22471.723187] [ 5951] 0 5951 1961 480 0 480 0 49152 224 0 screen
[22471.723201] [ 5952] 0 5952 2041 640 96 544 0 49152 128 0 bash
[22471.723204] [ 5963] 0 5963 1993 544 96 448 0 45056 160 0 screen
[22471.723206] [ 5964] 0 5964 2041 608 32 576 0 53248 160 0 bash
[22471.723208] [ 6064] 0 6064 1961 448 0 448 0 49152 224 0 screen
[22471.723214] [ 6065] 0 6065 2121 832 128 704 0 53248 160 0 bash
[22471.723219] [ 6900] 0 6900 1961 512 32 480 0 45056 160 0 screen
[22471.723221] [ 6901] 0 6901 2087 896 160 736 0 57344 32 0 bash
[22471.723223] [ 6916] 0 6916 1961 512 32 480 0 40960 160 0 screen
[22471.723225] [ 6917] 0 6917 2084 864 192 672 0 57344 96 0 bash
[22471.723228] [ 8141] 0 8141 49862 320 0 320 0 172032 0 0 ct
[22471.723231] [ 9210] 0 9210 1573558 1573088 1572768 320 0 12648448 0 0 memt
[22471.723248] [ 9246] 0 9246 1589430 250395 249659 736 0 11841536 1214465 0 php
[22471.723252] [ 9631] 0 9631 24532 864 96 768 0 94208 640 0 curl
[22471.723255] [ 9638] 0 9638 24532 832 96 736 0 94208 640 0 curl
[22471.723257] [ 9663] 0 9663 24532 832 128 704 0 90112 640 0 curl
[22471.723259] [ 9665] 0 9665 53998 640 64 576 0 106496 160 0 fio
[22471.723261] [ 9667] 0 9667 1091336 756214 755926 288 0 8511488 297120 0 fio
[22471.723264] [ 9668] 0 9668 1091337 490038 489782 256 0 8511488 563264 0 fio
[22471.723269] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/user.slice/user-0.slice/session-3.scope,task=memt,pid=9210,uid=0
[22471.723310] Out of memory: Killed process 9210 (memt) total-vm:6294232kB, anon-rss:6291072kB, file-rss:1280kB, shmem-rss:0kB, UID:0 pgtables:12352kB oom_score_adj:0
[61796.908278] INFO: task jbd2/sda1-8:377 blocked for more than 30 seconds.
[61796.909531] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8
[61796.910820] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[61796.912369] task:jbd2/sda1-8 state:D stack:0 pid:377 tgid:377 ppid:2 flags:0x00004000
[61796.912389] Call Trace:
[61796.912399] <TASK>
[61796.912426] __schedule+0x5e7/0x14a0
[61796.912487] ? submit_bio_noacct_nocheck+0x2cc/0x360
[61796.912508] schedule+0x8e/0x160
[61796.912512] bit_wait_io+0x45/0xa0
[61796.912518] __wait_on_bit+0xe1/0x170
[61796.912521] ? bit_wait_io_timeout+0xc0/0xc0
[61796.912524] jbd2_journal_commit_transaction+0x191b/0x2320
[61796.912547] ? wake_up_bit+0xd0/0xd0
[61796.912564] kjournald2+0x120/0x460
[61796.912577] ? init_wait_entry+0x30/0x30
[61796.912580] kthread+0x1f9/0x230
[61796.912594] ? jbd2_journal_update_sb_log_tail+0x180/0x180
[61796.912599] ? kthreadd+0x380/0x380
[61796.912601] ret_from_fork+0x30/0x40
[61796.912620] ? kthreadd+0x380/0x380
[61796.912622] ret_from_fork_asm+0x11/0x20
[61796.912665] </TASK>
[61875.585039] show_signal_msg: 8 callbacks suppressed
[61875.585050] cq[14120]: segfault at 7f611add1000 ip 000055f05765c4e8 sp 00007f61193ffdb0 error 4 in cq[55f05765c000+1000] likely on CPU 2 (core 2, socket 0)
[61875.585106] Code: 89 d1 48 d3 e6 49 09 34 c4 e8 64 fc ff ff 4c 89 e2 be 80 00 00 00 48 c1 eb 02 48 89 c7 e8 10 fc ff ff 89 d9 48 89 ef 0f 57 c9 <0f> 28 07 0f 58 c8 0f 29 0f 48 83 c7 10 e2 f1 eb ef 48 8b 84 24 88
[61887.147744] INFO: task jbd2/sda1-8:377 blocked for more than 30 seconds.
[61887.148999] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8
[61887.150213] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[61887.151554] task:jbd2/sda1-8 state:D stack:0 pid:377 tgid:377 ppid:2 flags:0x00004000
[61887.151561] Call Trace:
[61887.151563] <TASK>
[61887.151576] __schedule+0x5e7/0x14a0
[61887.151586] ? submit_bio_noacct_nocheck+0x2cc/0x360
[61887.151591] schedule+0x8e/0x160
[61887.151594] bit_wait_io+0x45/0xa0
[61887.151597] __wait_on_bit+0xe1/0x170
[61887.151600] ? bit_wait_io_timeout+0xc0/0xc0
[61887.151604] jbd2_journal_commit_transaction+0x191b/0x2320
[61887.151609] ? wake_up_bit+0xd0/0xd0
[61887.151613] kjournald2+0x120/0x460
[61887.151618] ? init_wait_entry+0x30/0x30
[61887.151621] kthread+0x1f9/0x230
[61887.151624] ? jbd2_journal_update_sb_log_tail+0x180/0x180
[61887.151628] ? kthreadd+0x380/0x380
[61887.151630] ret_from_fork+0x30/0x40
[61887.151634] ? kthreadd+0x380/0x380
[61887.151636] ret_from_fork_asm+0x11/0x20
[61887.151641] </TASK>
[61887.151644] INFO: task journal-offline:14113 blocked for more than 30 seconds.
[61887.153004] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8
[61887.154263] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[61887.157148] task:journal-offline state:D stack:0 pid:14113 tgid:451 ppid:1 flags:0x00004002
[61887.157158] Call Trace:
[61887.157162] <TASK>
[61887.157170] __schedule+0x5e7/0x14a0
[61887.157185] ? wait_for_common_io+0x171/0x190
[61887.157191] ? wbc_detach_inode+0x103/0x230
[61887.157213] ? prepare_to_wait_event+0x125/0x140
[61887.157218] schedule+0x8e/0x160
[61887.157221] jbd2_log_wait_commit+0x1b7/0x200
[61887.157230] ? init_wait_entry+0x30/0x30
[61887.157232] ext4_sync_file+0x47c/0x590
[61887.157254] __x64_sys_fsync+0xb0/0x100
[61887.157268] do_syscall_64+0x50/0x110
[61887.157293] entry_SYSCALL_64_after_hwframe+0x24/0x2c
[61887.157311] RIP: 0033:0x7f923d11b8ab
[61887.157319] RSP: 002b:00007f923b7ffc50 EFLAGS: 00000293 ORIG_RAX: 000000000000004a
[61887.157342] RAX: ffffffffffffffda RBX: 000055c68b01c370 RCX: 00007f923d11b8ab
[61887.157344] RDX: 0000000000000000 RSI: 00007f923d64ea34 RDI: 0000000000000019
[61887.157346] RBP: 00007f923d6502c0 R08: 0000000000000000 R09: 00007ffc269604cf
[61887.157348] R10: 0000000000000a34 R11: 0000000000000293 R12: 0000000000000002
[61887.157349] R13: 0000000000000002 R14: 00007f923d0947d0 R15: 00007ffc26960510
[61887.157360] </TASK>
[63099.559295] traps: cq[14344] trap invalid opcode ip:564f9853d3aa sp:7f779bbffc20 error:0
[63099.559294] traps: cq[14343] trap invalid opcode ip:564f9853d3aa sp:7f779c5ffc20 error:0
[63099.559348] in cq[564f9853d000+1000]
[63099.559348] in cq[564f9853d000+1000]


[63421.219463] INFO: task jbd2/sda1-8:377 blocked for more than 30 seconds.
[63421.220972] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8
[63421.222346] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[63421.223707] task:jbd2/sda1-8 state:D stack:0 pid:377 tgid:377 ppid:2 flags:0x00004000
[63421.223728] Call Trace:
[63421.223749] <TASK>
[63421.223777] __schedule+0x5e7/0x14a0
[63421.223860] ? submit_bio_noacct_nocheck+0x2cc/0x360
[63421.223882] schedule+0x8e/0x160
[63421.223886] bit_wait_io+0x45/0xa0
[63421.223892] __wait_on_bit+0xe1/0x170
[63421.223895] ? bit_wait_io_timeout+0xc0/0xc0
[63421.223899] jbd2_journal_commit_transaction+0x191b/0x2320
[63421.223920] ? wake_up_bit+0xd0/0xd0
[63421.223938] kjournald2+0x120/0x460
[63421.223951] ? init_wait_entry+0x30/0x30
[63421.223955] kthread+0x1f9/0x230
[63421.223969] ? jbd2_journal_update_sb_log_tail+0x180/0x180
[63421.223974] ? kthreadd+0x380/0x380
[63421.223976] ret_from_fork+0x30/0x40
[63421.224007] ? kthreadd+0x380/0x380
[63421.224009] ret_from_fork_asm+0x11/0x20
[63421.224024] </TASK>
[68384.390473] INFO: task jbd2/sda1-8:377 blocked for more than 30 seconds.
[68384.392485] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8
[68384.394706] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[68384.396367] task:jbd2/sda1-8 state:D stack:0 pid:377 tgid:377 ppid:2 flags:0x00004000
[68384.396381] Call Trace:
[68384.396393] <TASK>
[68384.396416] __schedule+0x5e7/0x14a0
[68384.396476] ? submit_bio_noacct_nocheck+0x2cc/0x360
[68384.396497] schedule+0x8e/0x160
[68384.396501] bit_wait_io+0x45/0xa0
[68384.396507] __wait_on_bit+0xe1/0x170
[68384.396510] ? bit_wait_io_timeout+0xc0/0xc0
[68384.396529] jbd2_journal_commit_transaction+0x191b/0x2320
[68384.396551] ? wake_up_bit+0xd0/0xd0
[68384.396568] kjournald2+0x120/0x460
[68384.396581] ? init_wait_entry+0x30/0x30
[68384.396584] kthread+0x1f9/0x230
[68384.396600] ? jbd2_journal_update_sb_log_tail+0x180/0x180
[68384.396604] ? kthreadd+0x380/0x380
[68384.396606] ret_from_fork+0x30/0x40
[68384.396626] ? kthreadd+0x380/0x380
[68384.396628] ret_from_fork_asm+0x11/0x20
[68384.396644] </TASK>
[68655.118907] INFO: task jbd2/sda1-8:377 blocked for more than 30 seconds.
[68655.124237] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8
[68655.125785] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[68655.128624] task:jbd2/sda1-8 state:D stack:0 pid:377 tgid:377 ppid:2 flags:0x00004000
[68655.128669] Call Trace:
[68655.128683] <TASK>
[68655.128718] __schedule+0x5e7/0x14a0
[68655.128801] ? submit_bio_noacct_nocheck+0x2cc/0x360
[68655.128829] schedule+0x8e/0x160
[68655.128833] bit_wait_io+0x45/0xa0
[68655.128908] __wait_on_bit+0xe1/0x170
[68655.128913] ? bit_wait_io_timeout+0xc0/0xc0
[68655.128918] jbd2_journal_commit_transaction+0x191b/0x2320
[68655.128997] ? wake_up_bit+0xd0/0xd0
[68655.129025] kjournald2+0x120/0x460
[68655.129048] ? init_wait_entry+0x30/0x30
[68655.129053] kthread+0x1f9/0x230
[68655.129080] ? jbd2_journal_update_sb_log_tail+0x180/0x180
[68655.129086] ? kthreadd+0x380/0x380
[68655.129089] ret_from_fork+0x30/0x40
[68655.129124] ? kthreadd+0x380/0x380
[68655.129127] ret_from_fork_asm+0x11/0x20
[68655.129156] </TASK>
[68895.747345] INFO: task jbd2/sda1-8:377 blocked for more than 30 seconds.
[68895.748831] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8
[68895.750303] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[68895.752488] task:jbd2/sda1-8 state:D stack:0 pid:377 tgid:377 ppid:2 flags:0x00004000
[68895.752513] Call Trace:
[68895.752527] <TASK>
[68895.752590] __schedule+0x5e7/0x14a0
[68895.752671] ? submit_bio_noacct_nocheck+0x2cc/0x360
[68895.752699] schedule+0x8e/0x160
[68895.752703] bit_wait_io+0x45/0xa0
[68895.752712] __wait_on_bit+0xe1/0x170
[68895.752716] ? bit_wait_io_timeout+0xc0/0xc0
[68895.752739] jbd2_journal_commit_transaction+0x191b/0x2320
[68895.752768] ? wake_up_bit+0xd0/0xd0
[68895.752792] kjournald2+0x120/0x460
[68895.752809] ? init_wait_entry+0x30/0x30
[68895.752813] kthread+0x1f9/0x230
[68895.752833] ? jbd2_journal_update_sb_log_tail+0x180/0x180
[68895.752839] ? kthreadd+0x380/0x380
[68895.752842] ret_from_fork+0x30/0x40
[68895.752869] ? kthreadd+0x380/0x380
[68895.752872] ret_from_fork_asm+0x11/0x20
[68895.752892] </TASK>
[69677.822593] INFO: task jbd2/sda1-8:377 blocked for more than 30 seconds.
[69677.824222] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8
[69677.826159] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[69677.828000] task:jbd2/sda1-8 state:D stack:0 pid:377 tgid:377 ppid:2 flags:0x00004000
[69677.828026] Call Trace:
[69677.828040] <TASK>
[69677.828076] __schedule+0x5e7/0x14a0
[69677.828159] ? submit_bio_noacct_nocheck+0x2cc/0x360
[69677.828187] schedule+0x8e/0x160
[69677.828191] bit_wait_io+0x45/0xa0
[69677.828199] __wait_on_bit+0xe1/0x170
[69677.828204] ? bit_wait_io_timeout+0xc0/0xc0
[69677.828208] jbd2_journal_commit_transaction+0x191b/0x2320
[69677.828239] ? wake_up_bit+0xd0/0xd0
[69677.828263] kjournald2+0x120/0x460
[69677.828280] ? init_wait_entry+0x30/0x30
[69677.828299] kthread+0x1f9/0x230
[69677.828321] ? jbd2_journal_update_sb_log_tail+0x180/0x180
[69677.828327] ? kthreadd+0x380/0x380
[69677.828330] ret_from_fork+0x30/0x40
[69677.828357] ? kthreadd+0x380/0x380
[69677.828360] ret_from_fork_asm+0x11/0x20
[69677.828381] </TASK>
[69677.828385] INFO: task journal-offline:18592 blocked for more than 30 seconds.
[69677.831042] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8
[69677.833149] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[69677.835422] task:journal-offline state:D stack:0 pid:18592 tgid:451 ppid:1 flags:0x00004002
[69677.835430] Call Trace:
[69677.835432] <TASK>
[69677.835436] __schedule+0x5e7/0x14a0
[69677.835444] ? wait_for_common_io+0x171/0x190
[69677.835449] ? prepare_to_wait_event+0x125/0x140
[69677.835454] schedule+0x8e/0x160
[69677.835457] jbd2_log_wait_commit+0x1b7/0x200
[69677.835464] ? init_wait_entry+0x30/0x30
[69677.835468] ext4_sync_file+0x47c/0x590
[69677.835513] __x64_sys_fsync+0xb0/0x100
[69677.835542] do_syscall_64+0x50/0x110
[69677.835569] entry_SYSCALL_64_after_hwframe+0x24/0x2c
[69677.835593] RIP: 0033:0x7f923d11b8ab
[69677.835602] RSP: 002b:00007f923b7ffc50 EFLAGS: 00000293 ORIG_RAX: 000000000000004a
[69677.835606] RAX: ffffffffffffffda RBX: 000055c68b01c370 RCX: 00007f923d11b8ab
[69677.835609] RDX: 0000000000000000 RSI: 00007f923d64ea34 RDI: 0000000000000019
[69677.835632] RBP: 00007f923d6502c0 R08: 0000000000000000 R09: 00007ffc269604cf
[69677.835634] R10: 0000000000000a34 R11: 0000000000000293 R12: 0000000000000002
[69677.835640] R13: 0000000000000002 R14: 00007f923d0947d0 R15: 00007ffc26960510
[69677.835651] </TASK>
[69707.922952] INFO: task jbd2/sda1-8:377 blocked for more than 60 seconds.
[69707.926000] Not tainted 6.8.1-af-server-2024-03-19-g760d1c206de8 #8
[69707.928249] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[69707.930623] task:jbd2/sda1-8 state:D stack:0 pid:377 tgid:377 ppid:2 flags:0x00004000
[69707.930632] Call Trace:
[69707.930640] <TASK>
[69707.930655] __schedule+0x5e7/0x14a0
[69707.930666] ? submit_bio_noacct_nocheck+0x2cc/0x360
[69707.930673] schedule+0x8e/0x160
[69707.930677] bit_wait_io+0x45/0xa0
[69707.930682] __wait_on_bit+0xe1/0x170
[69707.930686] ? bit_wait_io_timeout+0xc0/0xc0
[69707.930691] jbd2_journal_commit_transaction+0x191b/0x2320
[69707.930698] ? wake_up_bit+0xd0/0xd0
[69707.930705] kjournald2+0x120/0x460
[69707.930711] ? init_wait_entry+0x30/0x30
[69707.930715] kthread+0x1f9/0x230
[69707.930719] ? jbd2_journal_update_sb_log_tail+0x180/0x180
[69707.930725] ? kthreadd+0x380/0x380
[69707.930728] ret_from_fork+0x30/0x40
[69707.930733] ? kthreadd+0x380/0x380
[69707.930736] ret_from_fork_asm+0x11/0x20
[69707.930743] </TASK>
[69707.930744] Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings
[69794.689696] systemd[1]: systemd-journald.service: Main process exited, code=dumped, status=6/ABRT
[69794.689877] systemd[1]: systemd-journald.service: Failed with result 'watchdog'.
[69794.691811] systemd[1]: systemd-journald.service: Consumed 5.443s CPU time.
[69794.693515] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1.
[69794.693939] systemd[1]: Stopped Journal Service.
[69794.693993] systemd[1]: systemd-journald.service: Consumed 5.443s CPU time.
[69794.813640] systemd[1]: Starting Journal Service...
[69846.623844] systemd[1]: Started Journal Service.

--
Ammar Faizi