Ktor `cio-to-nio-writer` exception cannot be caught by caller

[IcarusL]

Current code uses openWriteChannel():

protected var selectorManager = SelectorManager(Dispatchers.IO)

abstract suspend fun onConnectSocket(timeout: Long): ReadWriteSocket

override suspend fun connect() {
    val socket = onConnectSocket(timeout)
    input = socket.openReadChannel()
    output = socket.openWriteChannel(autoFlush = false)
    address = java.net.InetSocketAddress(host, port).address
    this.socket = socket
}

Problem

When the remote socket is closed while data is still being written, the actual write failure may be thrown inside Ktor's internal cio-to-nio-writer coroutine.

This failure is not thrown from the caller's writeFully() / flush() call stack, so caller-side error handling cannot reliably catch it:

runCatching {
    output.writeFully(data)
    output.flush()
}.onFailure {
    // This may not receive the actual cio-to-nio-writer failure.
}

In the observed case, the stack trace only contains Ktor internal classes, which indicates that the failure happens inside Ktor's internal writer coroutine.

Why caller-side catch does not work

openWriteChannel() only returns a ByteWriteChannel. It is not a suspend function that performs the actual socket write in the caller's coroutine chain.

The actual flow is closer to this:

caller try-catch / runCatching
    ↓
output.writeFully(...) / output.flush(...)
    ↓
ByteWriteChannel
    ↓
Ktor internal cio-to-nio-writer coroutine
    ↓
actual socket write fails

Once the failure happens inside cio-to-nio-writer, it is no longer on the caller's call stack. Therefore, wrapping writeFully() / flush() with try-catch or runCatching cannot reliably catch it.

Why external CoroutineContext cannot handle it

From Ktor's implementation, the socket itself is a CoroutineScope.

The simplified inheritance chain is:

SocketImpl(...)
    : NIOSocketImpl(...)

NIOSocketImpl(...)
    : ReadWriteSocket,
      SocketBase(EmptyCoroutineContext)

SocketBase(parent: CoroutineContext)
    : CoroutineScope {
    override val socketContext: CompletableJob = Job(parent[Job])

    override val coroutineContext: CoroutineContext
        get() = socketContext
}

Since NIOSocketImpl creates SocketBase with EmptyCoroutineContext, the socket owns its internal socketContext and does not inherit the caller's CoroutineContext.

As a result, the caller has no public entry point to inject its own CoroutineExceptionHandler into Ktor's internal cio-to-nio-writer coroutine.

So the issue is:

The caller cannot control the CoroutineContext of cio-to-nio-writer, so it cannot handle or consume exceptions thrown inside that internal coroutine with an external CoroutineExceptionHandler.

Why invokeOnCompletion is not enough

Using attachForWriting() instead of openWriteChannel() allows the caller to keep the writer job:

val writeChannel = ByteChannel(autoFlush = false)
val writerJob = socket.attachForWriting(writeChannel)

writerJob.invokeOnCompletion { cause ->
    if (cause != null && cause !is CancellationException) {
        // The writer failure can be observed here.
    }
}

output = writeChannel

However, invokeOnCompletion only observes the job completion cause. It does not catch or consume the exception, and it cannot prevent the uncaught coroutine failure path.

It is useful for logging and cleanup, but it is not equivalent to try-catch.

Conclusion

The key point is:

The exception is thrown inside Ktor's internal cio-to-nio-writer coroutine, not from the caller's writeFully() / flush() call stack.

Therefore:

• caller-side try-catch cannot catch it;
• caller-side runCatching cannot catch it;
• openWriteChannel() is not a suspend function that can propagate this failure through the caller's coroutine chain;
• the caller cannot inject a CoroutineExceptionHandler into cio-to-nio-writer;
• invokeOnCompletion can observe the writer job failure, but cannot consume it.

Correction note

This issue has been updated.

The original version suggested handling this by installing a CoroutineExceptionHandler on SelectorManager. After checking Ktor's socket implementation, that is not considered a valid fix for this failure path.

The actual problem is that the caller has no public API to inject its own CoroutineExceptionHandler into Ktor's internal cio-to-nio-writer coroutine context.

[pedroSG94]

Hello,

Thank you for the report.
I understand that the problem is that ktor crash but, if I'm not able to catch the error and the app crash anyway, the invokeOnCompletion has no sense.

Maybe I miss understand you.
Also, if you can share me the crash error and a way to reproduce it, maybe we can report the error to ktor team and get a fix.

[IcarusL]

After checking the related Ktor source code again, I think the previous conclusion about cio-to-nio-writer needs to be corrected.

The original assumption was: when the remote socket is closed, the write failure may happen inside Ktor's internal cio-to-nio-writer, and caller-side try-catch / runCatching cannot catch it.

This is partially correct: the caller writes to a ByteWriteChannel, while the actual socket write does happen inside Ktor's internal write path, outside the caller's writeFully() / flush() call stack.

However, the more accurate flow is not "cio-to-nio-writer directly crashes with an uncaught exception".

---

1. nioChannel.write(buffer) is the low-level Broken pipe trigger point

In CIOWriter.kt, the real socket write happens here:

rc = nioChannel.write(buffer)

When the remote side has already closed the connection and the local side continues writing, this may throw:

java.io.IOException: Broken pipe

This is the actual low-level network write failure.

---

2. This exception is caught by reader(...) and converted into a failed ByteChannel

cio-to-nio-writer is created through Ktor's reader(...), and reader(...) has an outer catch block:

catch (cause: Throwable) {
    nested.cancel("Exception thrown while reading from channel", cause)
    channel.close(cause)
}

So when nioChannel.write(buffer) throws IOException: Broken pipe, the more accurate flow is:

nioChannel.write(buffer) throws IOException: Broken pipe
        ↓
reader(...) catches the exception
        ↓
reader(...) calls channel.close(cause)
        ↓
the ByteChannel becomes failed/closed with the Broken pipe cause

In other words, the low-level socket write failure is stored as the ByteChannel close cause. It does not directly escape from cio-to-nio-writer as an uncaught exception in this path.

---

3. A later write to the failed ByteChannel throws ClosedWriteChannelException

ByteChannel.writeBuffer checks the closed state:

override val writeBuffer: Sink
    get() {
        if (isClosedForWrite) {
            _closedCause.value?.throwOrNull(::ClosedWriteChannelException)
                ?: throw ClosedWriteChannelException()
        }
        return _writeBuffer
    }

Therefore, once the channel has been closed because of Broken pipe, any later writer using the same channel may receive:

ClosedWriteChannelException: Broken pipe

The basic flow is:

nioChannel.write(buffer) fails
        ↓
reader catches the failure
        ↓
ByteChannel is closed with cause
        ↓
some later writer writes to the same ByteChannel
        ↓
ByteChannel.writeBuffer throws ClosedWriteChannelException

---

4. In the currently saved crash stack, the uncaught exception is reported from cio-tls-closer

The crash stack captured this time is:

UncaughtException
io.ktor.utils.io.ClosedWriteChannelException: Broken pipe
    at io.ktor.utils.io.ByteChannel.getWriteBuffer(ByteChannel.kt:54)
    at io.ktor.utils.io.ByteWriteChannelOperationsKt.writeByte(ByteWriteChannelOperations.kt:19)
    at io.ktor.network.tls.RenderKt.writeRecord(Render.kt:18)
    at io.ktor.network.tls.TLSClientHandshake$output$2$1$1.invokeSuspend(TLSClientHandshake.kt:139)

The coroutine context shows:

CoroutineName(cio-tls-closer)

So in this saved crash stack, the direct reporting coroutine is not cio-to-nio-writer, but the TLS close coroutine: cio-tls-closer.

However, the Caused by still points to the low-level socket write failure:

Caused by: java.io.IOException: Broken pipe
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:512)
    at io.ktor.network.sockets.CIOWriterKt$attachForWritingDirectImpl...

So this stack indicates:

low-level socket write gets Broken pipe
        ↓
reader(...) catches it and closes the ByteChannel
        ↓
TLS closer later writes to this failed ByteChannel
        ↓
ClosedWriteChannelException becomes uncaught in cio-tls-closer

---

5. rawOutput is the raw socket write channel created by openWriteChannel() in Socket.tls()

Socket.tls() creates the raw socket channels:

val reader = openReadChannel()
val writer = openWriteChannel()

return openTLSSession(this, reader, writer, config, coroutineContext)

The writer is passed into TLS as rawOutput.

So TLS rawOutput is essentially the raw socket ByteWriteChannel, backed by this flow:

rawOutput.writeRecord(...)
        ↓
ByteChannel
        ↓
cio-to-nio-writer
        ↓
nioChannel.write(buffer)

After the low-level nioChannel.write(buffer) fails with Broken pipe, this rawOutput is closed as a failed channel by reader(...).

---

6. TLSClientHandshake catches normal encoder failures, but cio-tls-closer has no catch

TLSClientHandshake receives rawOutput and is itself a CoroutineScope:

internal class TLSClientHandshake(
    rawInput: ByteReadChannel,
    rawOutput: ByteWriteChannel,
    private val config: TLSConfig,
    override val coroutineContext: CoroutineContext,
    private val closeTask: CompletableJob = Job(),
) : CoroutineScope

Normal TLS record writing happens inside cio-tls-encoder, which has a catch block:

val output: SendChannel<TLSRecord> = actor(CoroutineName("cio-tls-encoder")) {
    for (rawRecord in channel) {
        try {
            val record = if (useCipher) cipher.encrypt(rawRecord) else rawRecord
            if (rawRecord.type == TLSRecordType.ChangeCipherSpec) useCipher = true

            rawOutput.writeRecord(record)
        } catch (cause: Throwable) {
            channel.close(cause)
            break
        }
    }
}

So normal TLS record write failures are caught by cio-tls-encoder, which then closes the actor channel and exits.

However, when output is closed, it triggers cio-tls-closer:

invokeOnClose {
    launch(CoroutineName("cio-tls-closer")) {
        try {
            val closeRecord = TLSRecord(
                TLSRecordType.Alert,
                packet = buildPacket {
                    writeByte(TLSAlertLevel.WARNING.code.toByte())
                    writeByte(TLSAlertType.CloseNotify.code.toByte())
                }
            )
            val record = if (useCipher) cipher.encrypt(closeRecord) else closeRecord
            rawOutput.writeRecord(record)
            rawOutput.flushAndClose()
        } finally {
            closeTask.complete()
        }
    }
}

The issue here is that cio-tls-closer only has try/finally, but no catch.

If rawOutput has already become a failed ByteChannel because of the lower-level Broken pipe, then this call:

rawOutput.writeRecord(record)

will hit the closed-state check in ByteChannel.writeBuffer and throw:

ClosedWriteChannelException: Broken pipe

Since cio-tls-closer does not catch it, this exception may become an uncaught coroutine exception.

Also, Socket.tls(...) supports passing a CoroutineContext when creating the TLS session:

public actual suspend fun Socket.tls(
    coroutineContext: CoroutineContext,
    config: TLSConfig
): Socket {
    val reader = openReadChannel()
    val writer = openWriteChannel()

    return try {
        openTLSSession(this, reader, writer, config, coroutineContext)
    } catch (cause: Throwable) {
        reader.cancel(cause)
        writer.close(cause)
        close()
        throw cause
    }
}

TLSClientHandshake uses this passed coroutineContext as its own coroutine context:

internal class TLSClientHandshake(
    rawInput: ByteReadChannel,
    rawOutput: ByteWriteChannel,
    private val config: TLSConfig,
    override val coroutineContext: CoroutineContext,
    private val closeTask: CompletableJob = Job(),
) : CoroutineScope

Therefore, for the currently saved cio-tls-closer crash path, a CoroutineExceptionHandler can be passed when creating the TLS socket:

val tlsContext = CoroutineExceptionHandler { _, throwable ->
    // log or report TLS internal coroutine exception
}

socket.tls(tlsContext, config)

This can at least provide a fallback exception handler for TLS internal coroutines and avoid cio-tls-closer going directly through the default uncaught exception path.

---

7. More accurate overall flow

remote server closes socket
        ↓
CIOWriter calls nioChannel.write(buffer)
        ↓
IOException: Broken pipe
        ↓
reader(...) catches the exception
        ↓
reader(...) closes rawOutput ByteChannel with the cause
        ↓
rawOutput becomes failed/closed
        ↓
TLS output is closed
        ↓
TLSClientHandshake launches cio-tls-closer
        ↓
cio-tls-closer tries to write TLS close_notify to rawOutput
        ↓
rawOutput is already failed
        ↓
ByteChannel throws ClosedWriteChannelException: Broken pipe
        ↓
cio-tls-closer has no catch
        ↓
the exception becomes an uncaught coroutine exception

---

8. Correction to the previous conclusion

The previous analysis focused too much on cio-to-nio-writer.

The more accurate conclusion is:

• nioChannel.write(buffer) in CIOWriter is the source of the original IOException: Broken pipe;
• but reader(...) catches this exception and closes the ByteChannel as failed;
• an uncaught exception may happen later when another coroutine writes to this failed ByteChannel;
• in the currently saved crash stack, that later writer is TLSClientHandshake's cio-tls-closer;
• cio-tls-closer tries to send TLS close_notify to an already failed rawOutput;
• because cio-tls-closer has no catch around rawOutput.writeRecord(record) / rawOutput.flushAndClose(), ClosedWriteChannelException: Broken pipe may become an uncaught coroutine exception.

---

9. About the earlier cio-to-nio-writer observation

During the earlier investigation, I did see cio-to-nio-writer in the logs, which is why I initially focused on cio-to-nio-writer.

However, in the currently saved crash stack, cio-to-nio-writer does not appear at all. The crash is clearly reported from:

CoroutineName(cio-tls-closer)

This suggests that the problem may not have a single trigger point.

A more accurate view is:

Earlier observed trigger point: cio-to-nio-writer
Current saved stack trigger point: cio-tls-closer

So TLS closer is only one of the cases that has currently been reproduced and saved. It may not represent all possible cases.

The more general problem is:

After the low-level socket write fails, the related ByteChannel becomes failed/closed.
Any coroutine that later writes to this failed ByteChannel may trigger an exception at its own write call site.

In the currently saved stack, the coroutine writing to the failed ByteChannel is cio-tls-closer.

But since cio-to-nio-writer appeared in earlier logs, there may be other paths or timing conditions that trigger similar failures. Therefore, this should not be attributed only to cio-tls-closer. It is better to treat this as an error propagation issue in the Ktor socket/TLS write path after the remote side closes the connection.

---

10. Reproduction scenario

The current reproduction scenario is:

send malformed H264/FLV data to Facebook over RTMP
        ↓
after about 30 seconds, Facebook closes the connection
        ↓
Ktor socket/TLS write path enters a failed state

The malformed stream can be constructed as follows.

Insert SEI NAL units before encoder-generated keyframes, while allowing the RTMP/FLV packetization logic to handle this input incorrectly.

Example SEI NAL units:

[00 00 00 01 06 05 63 3d 07 6d 45 73 0f 41 a8 b1 c4 25 d7 97 6b f1 ac 64 35 31 39 38 36 30 38 20 61 75 74 68 6f 72 3a 20 59 61 6e 6a 75 6e 20 4c 69 61 6f 20 20 20 32 30 32 35 2d 30 37 2d 32 35 20 66 69 78 5b 68 61 6c 5f 68 32 36 35 65 5d 3a 20 46 69 78 20 6e 61 6c 20 74 79 70 65 20 69 6e 20 74 73 76 63 20 6d 6f 64 65 80]

[00 00 00 01 06 05 50 d7 dc 03 c3 c5 6f 40 e0 8e a9 17 1a d2 ef 5e 23 64 65 66 61 75 6c 74 3a 76 62 72 2d 62 3a 34 31 34 37 32 30 30 5b 32 35 39 32 30 30 3a 34 34 30 36 34 30 30 5d 2d 67 3a 33 30 2d 71 3a 2d 31 3a 5b 31 3a 33 35 5d 3a 5b 31 3a 33 35 5d 3a 32 0a 80]

The first keyframe is usually:

SPS + PPS + IDR

It can be changed to:

SPS + PPS + SEI + SEI + IDR

or:

SEI + SEI + IDR

Later keyframes are usually:

IDR

They can be changed to:

SEI + SEI + IDR

This creates keyframe input containing multiple Annex B NAL units.

If the RTMP/FLV packetization logic does not split these NAL units correctly, but only removes the first start code and writes the remaining content as a single NAL unit into the FLV AVC payload, it produces invalid data such as:

[length][SEI + startCode + SEI + startCode + IDR]

The correct AVC-in-FLV payload should not contain Annex B start codes. It should contain multiple length-prefixed NAL units, for example:

[SEI length][SEI data][SEI length][SEI data][IDR length][IDR data]

Or, if SEI is intentionally dropped:

[IDR length][IDR data]

In this reproduction, after receiving this malformed RTMP video data, Facebook closes the connection after about 30 seconds.

Two behaviors have been observed.

Case 1: occasional crash, lower probability

The crash stack contains:

ClosedWriteChannelException: Broken pipe

The currently saved stack reports it from:

CoroutineName(cio-tls-closer)

Case 2: sender does not fail/exit in time, higher probability

More commonly, the observed behavior is:

no connection failure is reported
the send queue fills quickly
the console keeps printing drop logs

This suggests that the problem may not only be an occasional crash. The failed state may also not be clearly propagated to the upper layer. After the remote side closes the connection, the upper layer does not stop sending in time, so the send queue keeps accumulating data and starts dropping frames.

---

11. About the suspected send blocking / missing failure callback

Besides the occasional crash, the more common behavior is that the send path does not fail or exit in time:

no connection failure is reported
the send queue fills quickly
the console keeps printing drop logs

The actual stream sending code in the library already wraps the write logic with runCatching.

Based on the current understanding of ByteChannel, if the application send path continues writing to a failed ByteChannel, it should theoretically trigger:

ClosedWriteChannelException: Broken pipe

Then the sender-side runCatching should catch it and report the connection failure.

The expected flow would be:

low-level Broken pipe
        ↓
reader(...) catches the exception
        ↓
ByteChannel.close(cause)
        ↓
later application write to ByteChannel
        ↓
ByteChannel.writeBuffer throws ClosedWriteChannelException
        ↓
application runCatching catches it
        ↓
connection failure is reported and sending stops

But the more common observed behavior is:

the underlying connection has failed
        ↓
the application does not receive the failure
        ↓
the send loop does not exit
        ↓
the send queue accumulates
        ↓
frames keep being dropped

So there is still a question to confirm:

Why does the failure state not reliably trigger the sender-side exception handling and connection failure callback after the remote side closes the connection?

I have not fully analyzed the library's send queue and business send flow yet, so this is only recorded as an observation and hypothesis.

---

12. Current suspicion

For the currently saved crash stack, the most suspicious point is cio-tls-closer.

When the lower-level raw output has already failed with Broken pipe, sending TLS close_notify during close may also fail. This is a common connection close race.

The currently confirmed path looks like:

TLS closer writes close_notify to an already failed rawOutput
        ↓
ClosedWriteChannelException
        ↓
no catch
        ↓
uncaught coroutine exception

However, since cio-to-nio-writer appeared in earlier logs, I still cannot rule out other trigger points. The more general issue remains: after a low-level socket write failure, how should later writes to the failed ByteChannel be handled, reported, and cleaned up?

---

Attached: full saved crash stack

UncaughtException
io.ktor.utils.io.ClosedWriteChannelException: Broken pipe
	at io.ktor.utils.io.ByteChannel$writeBuffer$1.invoke(ByteChannel.kt:54)
	at io.ktor.utils.io.ByteChannel$writeBuffer$1.invoke(ByteChannel.kt:54)
	at io.ktor.utils.io.CloseToken.wrapCause(CloseToken.kt:21)
	at io.ktor.utils.io.CloseToken.throwOrNull(CloseToken.kt:26)
	at io.ktor.utils.io.ByteChannel.getWriteBuffer(ByteChannel.kt:54)
	at io.ktor.utils.io.ByteWriteChannelOperationsKt.writeByte(ByteWriteChannelOperations.kt:19)
	at io.ktor.network.tls.RenderKt.writeRecord(Render.kt:18)
	at io.ktor.network.tls.TLSClientHandshake$output$2$1$1.invokeSuspend(TLSClientHandshake.kt:139)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:34)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)
	at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:124)
	at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:89)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:586)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:820)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:717)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:704)
	Suppressed: kotlinx.coroutines.internal.DiagnosticCoroutineContextException: [CoroutineName(cio-tls-closer), StandaloneCoroutine{Cancelling}@6eeb5fa, Dispatchers.IO]
Caused by: java.io.IOException: Broken pipe
	at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
	at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:55)
	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
	at sun.nio.ch.IOUtil.write(IOUtil.java:65)
	at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:512)
	at io.ktor.network.sockets.CIOWriterKt$attachForWritingDirectImpl$1.invokeSuspend$lambda$0(CIOWriter.kt:42)
	at io.ktor.network.sockets.CIOWriterKt$attachForWritingDirectImpl$1.$r8$lambda$2oFIReZ5_z0WfUOcLpz2sy7WYCM(Unknown Source:0)
	at io.ktor.network.sockets.CIOWriterKt$attachForWritingDirectImpl$1$$ExternalSyntheticLambda0.invoke(D8$$SyntheticClass:0)
	at io.ktor.utils.io.core.ByteReadPacketExtensions_jvmKt.read(ByteReadPacketExtensions.jvm.kt:30)
	at io.ktor.network.sockets.CIOWriterKt$attachForWritingDirectImpl$1.invokeSuspend(CIOWriter.kt:77)
	... 8 more