* [BUG] Layerscape CAAM+kTLS+io_uring
@ 2022-04-27 8:20 Steffen Trumtrar
2022-05-05 19:20 ` Marc Kleine-Budde
0 siblings, 1 reply; 5+ messages in thread
From: Steffen Trumtrar @ 2022-04-27 8:20 UTC (permalink / raw)
To: linux-crypto; +Cc: io-uring
Hi all,
I have a Layerscape-1046a based board where I'm trying to use a
combination of liburing (v2.0) with splice, kTLS and CAAM (kernel
v5.17). The problem I see is that on shutdown the last bytes are
missing. It looks like io_uring is not waiting for all completions
from the CAAM driver.
With ARM-ASM instead of the CAAM, the setup works fine.
I tried to debug with ftrace and see where it goes wrong. Here is what
seems to be (at least to me) some of the last bytes:
webserver-612 [002] ..... 135.300350: io_uring_file_get: ring 0000000078f4a859, fd 6 00078f4a859, fd 7
webserver-612 [002] ..... 135.300352: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000fbb9b849, op 30, data 0xf3e096f0, flags 8196, non block 1, sq_thread 0
webserver-612 [002] ..... 135.300353: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000ff858bdf, op 15, data 0x0, flags 2097152, non block 1, sq_thread 0
webserver-612 [002] ..... 135.300353: io_uring_link: ring 0000000078f4a859, request 00000000ff858bdf linked after 00000000fbb9b849
webserver-612 [002] ..... 135.300354: io_uring_queue_async_work: ring 0000000078f4a859, request 00000000fbb9b849, flags 798724, normal queue, work 0000000060cd323f
webserver-612 [002] ..... 135.300358: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1
iou-wrk-612-647 [003] ...1. 135.300375: io_uring_complete: ring 0000000078f4a859, user_data 0xf3e096f0, result 4352, cflags 0
iou-wrk-612-647 [003] d..2. 135.300379: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result -125, cflags 0
webserver-612 [002] ..... 135.300388: io_uring_file_get: ring 0000000078f4a859, fd 6
webserver-612 [002] ..... 135.300389: io_uring_file_get: ring 0000000078f4a859, fd 7 0000078f4a859, req 000000008c2bf2be, op 30, data 0xf3e096f0, flags 8196, non block 1, sq_thread 0
webserver-612 [002] ..... 135.300390: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000fe4e50d1, op 15, data 0x0, flags 2097152, non block 1, sq_thread 0
webserver-612 [002] ..... 135.300391: io_uring_link: ring 0000000078f4a859, request 00000000fe4e50d1 linked after 000000008c2bf2be
webserver-612 [002] ..... 135.300392: io_uring_queue_async_work: ring 0000000078f4a859, request 000000008c2bf2be, flags 798724, normal queue, work 00000000f2b434fc
webserver-612 [002] ..... 135.300396: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1
iou-wrk-612-647 [003] ...1. 135.300410: io_uring_complete: ring 0000000078f4a859, user_data 0xf3e096f0, result 4352, cflags 0
iou-wrk-612-647 [003] d..2. 135.300414: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result -125, cflags 0
webserver-612 [002] ..... 135.300423: io_uring_file_get: ring 0000000078f4a859, fd 6
webserver-612 [002] ..... 135.300424: io_uring_file_get: ring 0000000078f4a859, fd 7
webserver-612 [002] ..... 135.300424: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000e48f3098, op 30, data 0xf3e096f0, flags 8196, non block 1, sq_thread 0
webserver-612 [002] ..... 135.300426: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000ec67d53c, op 15, data 0x0, flags 2097152, non block 1, sq_thread 0
webserver-612 [002] ..... 135.300426: io_uring_link: ring 0000000078f4a859, request 00000000ec67d53c linked after 00000000e48f3098
webserver-612 [002] ..... 135.300427: io_uring_queue_async_work: ring 0000000078f4a859, request 00000000e48f3098, flags 798724, normal queue, work 000000009e3701da
webserver-612 [002] ..... 135.300431: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1
iou-wrk-612-647 [003] ...1. 135.300447: io_uring_complete: ring 0000000078f4a859, user_data 0xf3e096f0, result 4352, cflags 0
iou-wrk-612-647 [003] d..2. 135.300452: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result -125, cflags 0
webserver-612 [002] ..... 135.300461: io_uring_file_get: ring 0000000078f4a859, fd 6
webserver-612 [002] ..... 135.300462: io_uring_file_get: ring 0000000078f4a859, fd 7
webserver-612 [002] ..... 135.300462: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000f658f96f, op 30, data 0xf3e096f0, flags 8196, non block 1, sq_thread 0, sq_thread 0
webserver-612 [002] ..... 135.300464: io_uring_link: ring 0000000078f4a859, request 000000006c29e721 linked after 00000000f658f96f
webserver-612 [002] ..... 135.300465: io_uring_queue_async_work: ring 0000000078f4a859, request 00000000f658f96f, flags 798724, normal queue, work 000000007434c68b
webserver-612 [002] ..... 135.300469: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1
iou-wrk-612-647 [003] ...1. 135.300479: io_uring_complete: ring 0000000078f4a859, user_data 0xf3e096f0, result 4608, cflags 0
iou-wrk-612-647 [003] d..2. 135.300483: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result -125, cflags 0
webserver-612 [002] ..... 135.300492: io_uring_file_get: ring 0000000078f4a859, fd 6
webserver-612 [002] ..... 135.300493: io_uring_file_get: ring 0000000078f4a859, fd 7
webserver-612 [002] ..... 135.300494: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000757ef148, op 30, data 0xf3e096f0, flags 8196, non block 1, sq_thread 0
webserver-612 [002] ..... 135.300495: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000e5c82137, op 15, data 0x0, flags 2097152, non block 1, sq_thread 0 8f4a859, request 00000000e5c82137 linked after 00000000757ef148
webserver-612 [002] ..... 135.300496: io_uring_queue_async_work: ring 0000000078f4a859, request 00000000757ef148, flags 798724, normal queue, work 00000000b55630dd
webserver-612 [002] ..... 135.300500: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1
iou-wrk-612-647 [003] ...1. 135.300516: io_uring_complete: ring 0000000078f4a859, user_data 0xf3e096f0, result 4608, cflags 0
iou-wrk-612-647 [003] d..2. 135.300520: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result -125, cflags 0
webserver-612 [002] ..... 135.300529: io_uring_file_get: ring 0000000078f4a859, fd 6
webserver-612 [002] ..... 135.300530: io_uring_file_get: ring 0000000078f4a859, fd 7
webserver-612 [002] ..... 135.300531: io_uring_submit_sqe: ring 0000000078f4a859, req 0000000085e5cac4, op 30, data 0xf3e096f0, flags 8196, non block 1, sq_thread 0
webserver-612 [002] ..... 135.300532: io_uring_submit_sqe: ring 0000000078f4a859, req 0000000008e6a863, op 15, data 0x0, flags 2097152, non block 1, sq_thread 0
webserver-612 [002] ..... 135.300532: io_uring_link: ring 0000000078f4a859, request 0000000008e6a863 linked after 0000000085e5cac4 ng 0000000078f4a859, request 0000000085e5cac4, flags 798724, normal queue, work 0000000036c4ff52
webserver-612 [002] ..... 135.300537: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1
iou-wrk-612-647 [003] ...1. 135.300553: io_uring_complete: ring 0000000078f4a859, user_data 0xf3e096f0, result 4608, cflags 0
iou-wrk-612-647 [003] d..2. 135.300557: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result -125, cflags 0
webserver-612 [002] ..... 135.300566: io_uring_file_get: ring 0000000078f4a859, fd 6
webserver-612 [002] ..... 135.300567: io_uring_file_get: ring 0000000078f4a859, fd 7
webserver-612 [002] ..... 135.300567: io_uring_submit_sqe: ring 0000000078f4a859, req 000000000f7fdd39, op 30, data 0xf3e096f0, flags 8196, non block 1, sq_thread 0
webserver-612 [002] ..... 135.300568: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000741c64e1, op 15, data 0x0, flags 2097152, non block 1, sq_thread 0
webserver-612 [002] ..... 135.300569: io_uring_link: ring 0000000078f4a859, request 00000000741c64e1 linked after 000000000f7fdd39
webserver-612 [002] ..... 135.300570: io_uring_queue_async_work: ring 0000000078f4a859, request 000000000f7fdd39, flags 798724, normal queue, work 00000000fc4accf1
webserver-612 [002] ..... 135.300574: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1
iou-wrk-612-647 [003] ...1. 135.300594: io_uring_complete: ring 0000000078f4a859, user_data 0xf3e096f0, result 4352, cflags 0
iou-wrk-612-647 [003] d..2. 135.300598: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result -125, cflags 0
webserver-612 [002] ..... 135.300607: io_uring_file_get: ring 0000000078f4a859, fd 6
webserver-612 [002] ..... 135.300608: io_uring_file_get: ring 0000000078f4a859, fd 7
webserver-612 [002] ..... 135.300608: io_uring_submit_sqe: ring 0000000078f4a859, req 0000000052b47765, op 30, data 0xf3e096f0, flags 8196, non block 1, sq_thread 0
webserver-612 [002] ..... 135.300610: io_uring_submit_sqe: ring 0000000078f4a859, req 000000003904ded9, op 15, data 0x0, flags 2097152, non block 1, sq_thread 0
webserver-612 [002] ..... 135.300611: io_uring_link: ring 0000000078f4a859, request 000000003904ded9 linked after 0000000052b47765
webserver-612 [002] ..... 135.300612: io_uring_queue_async_work: ring 0000000078f4a859, request 0000000052b47765, flags 798724, normal queue, work 00000000e11c8599
webserver-612 [002] ..... 135.300615: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1
iou-wrk-612-647 [003] ...1. 135.300631: io_uring_complete: ring 0000000078f4a859, user_data 0xf3e096f0, result 768, cflags 0
iou-wrk-612-647 [003] d..2. 135.300634: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result -125, cflags 0
webserver-612 [002] ..... 135.301668: io_uring_file_get: ring 0000000078f4a859, fd 6
webserver-612 [002] ..... 135.301669: io_uring_submit_sqe: ring 0000000078f4a859, req 0000000012863980, op 34, data 0x0, flags 8, non block 1, sq_thread 0
webserver-612 [002] ..... 135.301670: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000f4b07ff9, op 19, data 0x0, flags 0, non block 1, sq_thread 0
webserver-612 [002] ..... 135.301671: io_uring_link: ring 0000000078f4a859, request 00000000f4b07ff9 linked after 0000000012863980
webserver-612 [002] ..... 135.301672: io_uring_queue_async_work: ring 0000000078f4a859, request 0000000012863980, flags 262152, normal queue, work 00000000102270ed
webserver-612 [002] ..... 135.301740: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1
iou-wrk-612-648 [000] ...1. 135.301757: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result 0, cflags 0
iou-wrk-612-648 [000] ...1. 135.301767: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result 0, cflags 0
webserver-612 [002] ..... 135.301769: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1
Userspace said that 768 bytes where missing.
Any ideas for how to debug this or why the async work queue doesn't work with
the CAMM but does with ARM-ASM?
If I can provide more info that might help, I'll try to produce and
provide it. As there are multiple components involved, I'm not sure
where to start or what information is useful. Currently looks like CAAM
is the culprit.
Thanks,
Steffen Trumtrar
--
Pengutronix e.K. | Dipl.-Inform. Steffen Trumtrar |
Steuerwalder Str. 21 | https://www.pengutronix.de/ |
31137 Hildesheim, Germany | Phone: +49-5121-206917-0 |
Amtsgericht Hildesheim, HRA 2686| Fax: +49-5121-206917-5555 |
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: [BUG] Layerscape CAAM+kTLS+io_uring 2022-04-27 8:20 [BUG] Layerscape CAAM+kTLS+io_uring Steffen Trumtrar @ 2022-05-05 19:20 ` Marc Kleine-Budde 2022-05-06 0:10 ` Jakub Kicinski 0 siblings, 1 reply; 5+ messages in thread From: Marc Kleine-Budde @ 2022-05-05 19:20 UTC (permalink / raw) To: Steffen Trumtrar Cc: linux-crypto, io-uring, kernel, Horia Geantă, Pankaj Gupta, Gaurav Jain, Jens Axboe, Pavel Begunkov, Boris Pismenny, John Fastabend, Daniel Borkmann, Jakub Kicinski, netdev [-- Attachment #1: Type: text/plain, Size: 12690 bytes --] Hello, no one seems to care about this problem. :/ Maybe too many components are involved, I'm the respective maintainers on Cc. Cc += the CAAM maintainers Cc += the io_uring maintainers Cc += the kTLS maintainers On 27.04.2022 10:20:40, Steffen Trumtrar wrote: > Hi all, > > I have a Layerscape-1046a based board where I'm trying to use a > combination of liburing (v2.0) with splice, kTLS and CAAM (kernel > v5.17). The problem I see is that on shutdown the last bytes are > missing. It looks like io_uring is not waiting for all completions > from the CAAM driver. > > With ARM-ASM instead of the CAAM, the setup works fine. What's the difference between the CAAM and ARM-ASM crypto? Without looking into the code I think the CAAM is asynchron while ARM-ASM is synchron. Is this worth investigating? > I tried to debug with ftrace and see where it goes wrong. Here is what > seems to be (at least to me) some of the last bytes: > > webserver-612 [002] ..... 135.300350: io_uring_file_get: ring 0000000078f4a859, fd 6 00078f4a859, fd 7 > webserver-612 [002] ..... 135.300352: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000fbb9b849, op 30, data 0xf3e096f0, flags 8196, non block 1, sq_thread 0 > webserver-612 [002] ..... 135.300353: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000ff858bdf, op 15, data 0x0, flags 2097152, non block 1, sq_thread 0 > webserver-612 [002] ..... 135.300353: io_uring_link: ring 0000000078f4a859, request 00000000ff858bdf linked after 00000000fbb9b849 > webserver-612 [002] ..... 135.300354: io_uring_queue_async_work: ring 0000000078f4a859, request 00000000fbb9b849, flags 798724, normal queue, work 0000000060cd323f > webserver-612 [002] ..... 135.300358: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1 > iou-wrk-612-647 [003] ...1. 135.300375: io_uring_complete: ring 0000000078f4a859, user_data 0xf3e096f0, result 4352, cflags 0 > iou-wrk-612-647 [003] d..2. 135.300379: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result -125, cflags 0 > webserver-612 [002] ..... 135.300388: io_uring_file_get: ring 0000000078f4a859, fd 6 > webserver-612 [002] ..... 135.300389: io_uring_file_get: ring 0000000078f4a859, fd 7 0000078f4a859, req 000000008c2bf2be, op 30, data 0xf3e096f0, flags 8196, non block 1, sq_thread 0 > webserver-612 [002] ..... 135.300390: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000fe4e50d1, op 15, data 0x0, flags 2097152, non block 1, sq_thread 0 > webserver-612 [002] ..... 135.300391: io_uring_link: ring 0000000078f4a859, request 00000000fe4e50d1 linked after 000000008c2bf2be > webserver-612 [002] ..... 135.300392: io_uring_queue_async_work: ring 0000000078f4a859, request 000000008c2bf2be, flags 798724, normal queue, work 00000000f2b434fc > webserver-612 [002] ..... 135.300396: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1 > iou-wrk-612-647 [003] ...1. 135.300410: io_uring_complete: ring 0000000078f4a859, user_data 0xf3e096f0, result 4352, cflags 0 > iou-wrk-612-647 [003] d..2. 135.300414: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result -125, cflags 0 > webserver-612 [002] ..... 135.300423: io_uring_file_get: ring 0000000078f4a859, fd 6 > webserver-612 [002] ..... 135.300424: io_uring_file_get: ring 0000000078f4a859, fd 7 > webserver-612 [002] ..... 135.300424: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000e48f3098, op 30, data 0xf3e096f0, flags 8196, non block 1, sq_thread 0 > webserver-612 [002] ..... 135.300426: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000ec67d53c, op 15, data 0x0, flags 2097152, non block 1, sq_thread 0 > webserver-612 [002] ..... 135.300426: io_uring_link: ring 0000000078f4a859, request 00000000ec67d53c linked after 00000000e48f3098 > webserver-612 [002] ..... 135.300427: io_uring_queue_async_work: ring 0000000078f4a859, request 00000000e48f3098, flags 798724, normal queue, work 000000009e3701da > webserver-612 [002] ..... 135.300431: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1 > iou-wrk-612-647 [003] ...1. 135.300447: io_uring_complete: ring 0000000078f4a859, user_data 0xf3e096f0, result 4352, cflags 0 > iou-wrk-612-647 [003] d..2. 135.300452: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result -125, cflags 0 > webserver-612 [002] ..... 135.300461: io_uring_file_get: ring 0000000078f4a859, fd 6 > webserver-612 [002] ..... 135.300462: io_uring_file_get: ring 0000000078f4a859, fd 7 > webserver-612 [002] ..... 135.300462: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000f658f96f, op 30, data 0xf3e096f0, flags 8196, non block 1, sq_thread 0, sq_thread 0 > webserver-612 [002] ..... 135.300464: io_uring_link: ring 0000000078f4a859, request 000000006c29e721 linked after 00000000f658f96f > webserver-612 [002] ..... 135.300465: io_uring_queue_async_work: ring 0000000078f4a859, request 00000000f658f96f, flags 798724, normal queue, work 000000007434c68b > webserver-612 [002] ..... 135.300469: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1 > iou-wrk-612-647 [003] ...1. 135.300479: io_uring_complete: ring 0000000078f4a859, user_data 0xf3e096f0, result 4608, cflags 0 > iou-wrk-612-647 [003] d..2. 135.300483: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result -125, cflags 0 > webserver-612 [002] ..... 135.300492: io_uring_file_get: ring 0000000078f4a859, fd 6 > webserver-612 [002] ..... 135.300493: io_uring_file_get: ring 0000000078f4a859, fd 7 > webserver-612 [002] ..... 135.300494: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000757ef148, op 30, data 0xf3e096f0, flags 8196, non block 1, sq_thread 0 > webserver-612 [002] ..... 135.300495: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000e5c82137, op 15, data 0x0, flags 2097152, non block 1, sq_thread 0 8f4a859, request 00000000e5c82137 linked after 00000000757ef148 > webserver-612 [002] ..... 135.300496: io_uring_queue_async_work: ring 0000000078f4a859, request 00000000757ef148, flags 798724, normal queue, work 00000000b55630dd > webserver-612 [002] ..... 135.300500: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1 > iou-wrk-612-647 [003] ...1. 135.300516: io_uring_complete: ring 0000000078f4a859, user_data 0xf3e096f0, result 4608, cflags 0 > iou-wrk-612-647 [003] d..2. 135.300520: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result -125, cflags 0 > webserver-612 [002] ..... 135.300529: io_uring_file_get: ring 0000000078f4a859, fd 6 > webserver-612 [002] ..... 135.300530: io_uring_file_get: ring 0000000078f4a859, fd 7 > webserver-612 [002] ..... 135.300531: io_uring_submit_sqe: ring 0000000078f4a859, req 0000000085e5cac4, op 30, data 0xf3e096f0, flags 8196, non block 1, sq_thread 0 > webserver-612 [002] ..... 135.300532: io_uring_submit_sqe: ring 0000000078f4a859, req 0000000008e6a863, op 15, data 0x0, flags 2097152, non block 1, sq_thread 0 > webserver-612 [002] ..... 135.300532: io_uring_link: ring 0000000078f4a859, request 0000000008e6a863 linked after 0000000085e5cac4 ng 0000000078f4a859, request 0000000085e5cac4, flags 798724, normal queue, work 0000000036c4ff52 > webserver-612 [002] ..... 135.300537: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1 > iou-wrk-612-647 [003] ...1. 135.300553: io_uring_complete: ring 0000000078f4a859, user_data 0xf3e096f0, result 4608, cflags 0 > iou-wrk-612-647 [003] d..2. 135.300557: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result -125, cflags 0 > webserver-612 [002] ..... 135.300566: io_uring_file_get: ring 0000000078f4a859, fd 6 > webserver-612 [002] ..... 135.300567: io_uring_file_get: ring 0000000078f4a859, fd 7 > webserver-612 [002] ..... 135.300567: io_uring_submit_sqe: ring 0000000078f4a859, req 000000000f7fdd39, op 30, data 0xf3e096f0, flags 8196, non block 1, sq_thread 0 > webserver-612 [002] ..... 135.300568: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000741c64e1, op 15, data 0x0, flags 2097152, non block 1, sq_thread 0 > webserver-612 [002] ..... 135.300569: io_uring_link: ring 0000000078f4a859, request 00000000741c64e1 linked after 000000000f7fdd39 > webserver-612 [002] ..... 135.300570: io_uring_queue_async_work: ring 0000000078f4a859, request 000000000f7fdd39, flags 798724, normal queue, work 00000000fc4accf1 > webserver-612 [002] ..... 135.300574: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1 > iou-wrk-612-647 [003] ...1. 135.300594: io_uring_complete: ring 0000000078f4a859, user_data 0xf3e096f0, result 4352, cflags 0 > iou-wrk-612-647 [003] d..2. 135.300598: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result -125, cflags 0 > webserver-612 [002] ..... 135.300607: io_uring_file_get: ring 0000000078f4a859, fd 6 > webserver-612 [002] ..... 135.300608: io_uring_file_get: ring 0000000078f4a859, fd 7 > webserver-612 [002] ..... 135.300608: io_uring_submit_sqe: ring 0000000078f4a859, req 0000000052b47765, op 30, data 0xf3e096f0, flags 8196, non block 1, sq_thread 0 > webserver-612 [002] ..... 135.300610: io_uring_submit_sqe: ring 0000000078f4a859, req 000000003904ded9, op 15, data 0x0, flags 2097152, non block 1, sq_thread 0 > webserver-612 [002] ..... 135.300611: io_uring_link: ring 0000000078f4a859, request 000000003904ded9 linked after 0000000052b47765 > webserver-612 [002] ..... 135.300612: io_uring_queue_async_work: ring 0000000078f4a859, request 0000000052b47765, flags 798724, normal queue, work 00000000e11c8599 > webserver-612 [002] ..... 135.300615: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1 > iou-wrk-612-647 [003] ...1. 135.300631: io_uring_complete: ring 0000000078f4a859, user_data 0xf3e096f0, result 768, cflags 0 > iou-wrk-612-647 [003] d..2. 135.300634: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result -125, cflags 0 > webserver-612 [002] ..... 135.301668: io_uring_file_get: ring 0000000078f4a859, fd 6 > webserver-612 [002] ..... 135.301669: io_uring_submit_sqe: ring 0000000078f4a859, req 0000000012863980, op 34, data 0x0, flags 8, non block 1, sq_thread 0 > webserver-612 [002] ..... 135.301670: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000f4b07ff9, op 19, data 0x0, flags 0, non block 1, sq_thread 0 > webserver-612 [002] ..... 135.301671: io_uring_link: ring 0000000078f4a859, request 00000000f4b07ff9 linked after 0000000012863980 > webserver-612 [002] ..... 135.301672: io_uring_queue_async_work: ring 0000000078f4a859, request 0000000012863980, flags 262152, normal queue, work 00000000102270ed > webserver-612 [002] ..... 135.301740: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1 > iou-wrk-612-648 [000] ...1. 135.301757: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result 0, cflags 0 > iou-wrk-612-648 [000] ...1. 135.301767: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result 0, cflags 0 > webserver-612 [002] ..... 135.301769: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1 > > > Userspace said that 768 bytes where missing. > > Any ideas for how to debug this or why the async work queue doesn't work with > the CAMM but does with ARM-ASM? > If I can provide more info that might help, I'll try to produce and > provide it. As there are multiple components involved, I'm not sure > where to start or what information is useful. Currently looks like CAAM > is the culprit. Can you provide test code or at least illustrate with code how you plug the components together? Marc -- Pengutronix e.K. | Marc Kleine-Budde | Embedded Linux | https://www.pengutronix.de | Vertretung West/Dortmund | Phone: +49-231-2826-924 | Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917-5555 | [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [BUG] Layerscape CAAM+kTLS+io_uring 2022-05-05 19:20 ` Marc Kleine-Budde @ 2022-05-06 0:10 ` Jakub Kicinski 2022-05-16 10:06 ` Steffen Trumtrar 0 siblings, 1 reply; 5+ messages in thread From: Jakub Kicinski @ 2022-05-06 0:10 UTC (permalink / raw) To: Marc Kleine-Budde Cc: Steffen Trumtrar, linux-crypto, io-uring, kernel, Horia Geantă, Pankaj Gupta, Gaurav Jain, Jens Axboe, Pavel Begunkov, Boris Pismenny, John Fastabend, Daniel Borkmann, netdev On Thu, 5 May 2022 21:20:46 +0200 Marc Kleine-Budde wrote: > Hello, > > no one seems to care about this problem. :/ > > Maybe too many components are involved, I'm the respective maintainers > on Cc. > > Cc += the CAAM maintainers > Cc += the io_uring maintainers > Cc += the kTLS maintainers > > On 27.04.2022 10:20:40, Steffen Trumtrar wrote: > > Hi all, > > > > I have a Layerscape-1046a based board where I'm trying to use a > > combination of liburing (v2.0) with splice, kTLS and CAAM (kernel > > v5.17). The problem I see is that on shutdown the last bytes are > > missing. It looks like io_uring is not waiting for all completions > > from the CAAM driver. > > > > With ARM-ASM instead of the CAAM, the setup works fine. > > What's the difference between the CAAM and ARM-ASM crypto? Without > looking into the code I think the CAAM is asynchron while ARM-ASM is > synchron. Is this worth investigating? Sounds like 20ffc7adf53a ("net/tls: missing received data after fast remote close") ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [BUG] Layerscape CAAM+kTLS+io_uring 2022-05-06 0:10 ` Jakub Kicinski @ 2022-05-16 10:06 ` Steffen Trumtrar 2022-05-17 6:30 ` [EXT] " Gaurav Jain 0 siblings, 1 reply; 5+ messages in thread From: Steffen Trumtrar @ 2022-05-16 10:06 UTC (permalink / raw) To: Jakub Kicinski Cc: Marc Kleine-Budde, linux-crypto, io-uring, kernel, Horia Geantă, Pankaj Gupta, Gaurav Jain, Jens Axboe, Pavel Begunkov, Boris Pismenny, John Fastabend, Daniel Borkmann, netdev Jakub Kicinski <[email protected]> writes: > On Thu, 5 May 2022 21:20:46 +0200 Marc Kleine-Budde wrote: >> Hello, >> >> no one seems to care about this problem. :/ >> >> Maybe too many components are involved, I'm the respective maintainers >> on Cc. >> >> Cc += the CAAM maintainers >> Cc += the io_uring maintainers >> Cc += the kTLS maintainers >> >> On 27.04.2022 10:20:40, Steffen Trumtrar wrote: >> > Hi all, >> > >> > I have a Layerscape-1046a based board where I'm trying to use a >> > combination of liburing (v2.0) with splice, kTLS and CAAM (kernel >> > v5.17). The problem I see is that on shutdown the last bytes are >> > missing. It looks like io_uring is not waiting for all completions >> > from the CAAM driver. >> > >> > With ARM-ASM instead of the CAAM, the setup works fine. >> >> What's the difference between the CAAM and ARM-ASM crypto? Without >> looking into the code I think the CAAM is asynchron while ARM-ASM is >> synchron. Is this worth investigating? > > Sounds like > 20ffc7adf53a ("net/tls: missing received data after fast remote close") That fixes something in tls_sw. I have a kernel that includes this patch. So this sounds right, but can't be it, right? Best regards, Steffen -- Pengutronix e.K. | Dipl.-Inform. Steffen Trumtrar | Steuerwalder Str. 21 | https://www.pengutronix.de/ | 31137 Hildesheim, Germany | Phone: +49-5121-206917-0 | Amtsgericht Hildesheim, HRA 2686| Fax: +49-5121-206917-5555 | ^ permalink raw reply [flat|nested] 5+ messages in thread
* RE: [EXT] Re: [BUG] Layerscape CAAM+kTLS+io_uring 2022-05-16 10:06 ` Steffen Trumtrar @ 2022-05-17 6:30 ` Gaurav Jain 0 siblings, 0 replies; 5+ messages in thread From: Gaurav Jain @ 2022-05-17 6:30 UTC (permalink / raw) To: Steffen Trumtrar, Jakub Kicinski Cc: Marc Kleine-Budde, [email protected], [email protected], [email protected], Horia Geanta, Pankaj Gupta, Jens Axboe, Pavel Begunkov, Boris Pismenny, John Fastabend, Daniel Borkmann, [email protected] Hello Steffen > -----Original Message----- > From: Steffen Trumtrar <[email protected]> > Sent: Monday, May 16, 2022 3:36 PM > To: Jakub Kicinski <[email protected]> > Cc: Marc Kleine-Budde <[email protected]>; [email protected]; > [email protected]; [email protected]; Horia Geanta > <[email protected]>; Pankaj Gupta <[email protected]>; Gaurav Jain > <[email protected]>; Jens Axboe <[email protected]>; Pavel Begunkov > <[email protected]>; Boris Pismenny <[email protected]>; John > Fastabend <[email protected]>; Daniel Borkmann > <[email protected]>; [email protected] > Subject: [EXT] Re: [BUG] Layerscape CAAM+kTLS+io_uring > > Caution: EXT Email > > Jakub Kicinski <[email protected]> writes: > > > On Thu, 5 May 2022 21:20:46 +0200 Marc Kleine-Budde wrote: > >> Hello, > >> > >> no one seems to care about this problem. :/ > >> > >> Maybe too many components are involved, I'm the respective > >> maintainers on Cc. > >> > >> Cc += the CAAM maintainers > >> Cc += the io_uring maintainers > >> Cc += the kTLS maintainers > >> > >> On 27.04.2022 10:20:40, Steffen Trumtrar wrote: > >> > Hi all, > >> > > >> > I have a Layerscape-1046a based board where I'm trying to use a > >> > combination of liburing (v2.0) with splice, kTLS and CAAM (kernel > >> > v5.17). The problem I see is that on shutdown the last bytes are > >> > missing. It looks like io_uring is not waiting for all completions > >> > from the CAAM driver. > >> > > >> > With ARM-ASM instead of the CAAM, the setup works fine. > >> > >> What's the difference between the CAAM and ARM-ASM crypto? Without > >> looking into the code I think the CAAM is asynchron while ARM-ASM is > >> synchron. Is this worth investigating? > > > > Sounds like > > 20ffc7adf53a ("net/tls: missing received data after fast remote > > close") > > That fixes something in tls_sw. I have a kernel that includes this patch. So this > sounds right, but can't be it, right? > To understand more on problem, can you share how you are offloading ktls to CAAM? Gaurav > > Best regards, > Steffen > > -- > Pengutronix e.K. | Dipl.-Inform. Steffen Trumtrar | > Steuerwalder Str. 21 | > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.pe > ngutronix.de%2F&data=05%7C01%7Cgaurav.jain%40nxp.com%7Cc225fa4 > 3da7f4a43f16b08da3723dc4a%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0% > 7C0%7C637882924400875890%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wL > jAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C > %7C&sdata=fI1BHNZIT5RlpjEdEMXVkv1WaHcvgTlZOcaC0LRojac%3D& > reserved=0 | > 31137 Hildesheim, Germany | Phone: +49-5121-206917-0 | > Amtsgericht Hildesheim, HRA 2686| Fax: +49-5121-206917-5555 | ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2022-05-17 6:30 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2022-04-27 8:20 [BUG] Layerscape CAAM+kTLS+io_uring Steffen Trumtrar 2022-05-05 19:20 ` Marc Kleine-Budde 2022-05-06 0:10 ` Jakub Kicinski 2022-05-16 10:06 ` Steffen Trumtrar 2022-05-17 6:30 ` [EXT] " Gaurav Jain
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox