public inbox for [email protected]
 help / color / mirror / Atom feed
* Problem with io_uring splice and KTLS
@ 2023-10-10 14:19 Sascha Hauer
  2023-10-10 14:28 ` Jens Axboe
  0 siblings, 1 reply; 11+ messages in thread
From: Sascha Hauer @ 2023-10-10 14:19 UTC (permalink / raw)
  To: Jens Axboe
  Cc: Pavel Begunkov, io-uring, linux-kernel, kernel, Boris Pismenny,
	John Fastabend, Jakub Kicinski, netdev

Hi,

I am working with a webserver using io_uring in conjunction with KTLS. The
webserver basically splices static file data from a pipe to a socket which uses
KTLS for encryption. When splice is done the socket is closed. This works fine
when using software encryption in KTLS. Things go awry though when the software
encryption is replaced with the CAAM driver which replaces the synchronous
encryption with a asynchronous queue/interrupt/completion flow.

So far I have traced it down to tls_push_sg() calling tcp_sendmsg_locked() to
send the completed encrypted messages. tcp_sendmsg_locked() sometimes waits for
more memory on the socket by calling sk_stream_wait_memory(). This in turn
returns -ERESTARTSYS due to:

        if (signal_pending(current))
                goto do_interrupted;

The current task has the TIF_NOTIFY_SIGNAL set due to:

io_req_normal_work_add()
{
        ...
        /* This interrupts sk_stream_wait_memory() (notify_method == TWA_SIGNAL) */
        task_work_add(req->task, &tctx->task_work, ctx->notify_method)))
}

The call stack when sk_stream_wait_memory() fails is as follows:

[ 1385.428816]  dump_backtrace+0xa0/0x128
[ 1385.432568]  show_stack+0x20/0x38
[ 1385.435878]  dump_stack_lvl+0x48/0x60
[ 1385.439539]  dump_stack+0x18/0x28
[ 1385.442850]  tls_push_sg+0x100/0x238
[ 1385.446424]  tls_tx_records+0x118/0x1d8
[ 1385.450257]  tls_sw_release_resources_tx+0x74/0x1a0
[ 1385.455135]  tls_sk_proto_close+0x2f8/0x3f0
[ 1385.459315]  inet_release+0x58/0xb8
[ 1385.462802]  inet6_release+0x3c/0x60
[ 1385.466374]  __sock_release+0x48/0xc8
[ 1385.470035]  sock_close+0x20/0x38
[ 1385.473347]  __fput+0xbc/0x280
[ 1385.476399]  ____fput+0x18/0x30
[ 1385.479537]  task_work_run+0x80/0xe0
[ 1385.483108]  io_run_task_work+0x40/0x108
[ 1385.487029]  __arm64_sys_io_uring_enter+0x164/0xad8
[ 1385.491907]  invoke_syscall+0x50/0x128
[ 1385.495655]  el0_svc_common.constprop.0+0x48/0xf0
[ 1385.500359]  do_el0_svc_compat+0x24/0x40
[ 1385.504279]  el0_svc_compat+0x38/0x108
[ 1385.508026]  el0t_32_sync_handler+0x98/0x140
[ 1385.512294]  el0t_32_sync+0x194/0x198

So the socket is being closed and KTLS tries to send out the remaining
completed messages.  From a splice point of view everything has been sent
successfully, but not everything made it through KTLS to the socket and the
remaining data is sent while closing the socket.

I vaguely understand what's going on here, but I haven't got the slightest idea
what to do about this. Any ideas?

Sascha

-- 
Pengutronix e.K.                           |                             |
Steuerwalder Str. 21                       | http://www.pengutronix.de/  |
31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

^ permalink raw reply	[flat|nested] 11+ messages in thread

end of thread, other threads:[~2023-10-17 11:51 UTC | newest]

Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-10-10 14:19 Problem with io_uring splice and KTLS Sascha Hauer
2023-10-10 14:28 ` Jens Axboe
2023-10-11 11:21   ` Sascha Hauer
2023-10-12 13:34   ` Sascha Hauer
2023-10-13  1:13     ` Jens Axboe
2023-10-13  1:45     ` Jens Axboe
2023-10-13  5:47       ` Sascha Hauer
2023-10-13 13:45         ` Jens Axboe
2023-10-16  7:26           ` Sascha Hauer
2023-10-16 13:17             ` Jens Axboe
2023-10-17 11:51               ` Sascha Hauer

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox