public class InboundMessageHandler
extends io.netty.channel.ChannelInboundHandlerAdapter
FrameDecoder).
Handles small and large messages, corruption, flow control, dispatch of message processing onto an appropriate
thread pool.
# Interaction with FrameDecoder
InboundMessageHandler sits on top of a FrameDecoder in the Netty pipeline, and is tightly
coupled with it.
FrameDecoder decodes inbound frames and relies on a supplied FrameDecoder.FrameProcessor to act on them.
InboundMessageHandler provides two implementations of that interface:
- #process(Frame) is the default, primary processor, and the primary entry point to this class
- UpToOneMessageFrameProcessor, supplied to the decoder when the handler is reactivated after being
put in waiting mode due to lack of acquirable reserve memory capacity permits
Return value of FrameProcessor#process(Frame) determines whether the decoder should keep processing
frames (if true is returned) or stop until explicitly reactivated (if false is). To reactivate
the decoder (once notified of available resource permits), FrameDecoder.reactivate() is invoked.
# Frames
InboundMessageHandler operates on frames of messages, and there are several kinds of them:
1. FrameDecoder.IntactFrame that are contained. As names suggest, these contain one or multiple fully contained
messages believed to be uncorrupted. Guaranteed to not contain an part of an incomplete message.
See #processFrameOfContainedMessages(ShareableBytes, Limit, Limit).
2. FrameDecoder.IntactFrame that are NOT contained. These are uncorrupted parts of a large message split over multiple
parts due to their size. Can represent first or subsequent frame of a large message.
See #processFirstFrameOfLargeMessage(IntactFrame, Limit, Limit) and
#processSubsequentFrameOfLargeMessage(Frame).
3. FrameDecoder.CorruptFrame with corrupt header. These are unrecoverable, and force a connection to be dropped.
4. FrameDecoder.CorruptFrame with a valid header, but corrupt payload. These can be either contained or uncontained.
- contained frames with corrupt payload can be gracefully dropped without dropping the connection
- uncontained frames with corrupt payload can be gracefully dropped unless they represent the first
frame of a new large message, as in that case we don't know how many bytes to skip
See #processCorruptFrame(CorruptFrame).
Fundamental frame invariants:
1. A contained frame can only have fully-encapsulated messages - 1 to n, that don't cross frame boundaries
2. An uncontained frame can hold a part of one message only. It can NOT, say, contain end of one large message
and a beginning of another one. All the bytes in an uncontained frame always belong to a single message.
# Small vs large messages
A single handler is equipped to process both small and large messages, potentially interleaved, but the logic
differs depending on size. Small messages are deserialized in place, and then handed off to an appropriate
thread pool for processing. Large messages accumulate frames until completion of a message, then hand off
the untouched frames to the correct thread pool for the verb to be deserialized there and immediately processed.
See LargeMessage for details of the large-message accumulating state-machine, and ProcessMessage
and its inheritors for the differences in execution.
# Flow control (backpressure)
To prevent nodes from overwhelming and bringing each other to the knees with more inbound messages that
can be processed in a timely manner, InboundMessageHandler implements a strict flow control policy.
Before we attempt to process a message fully, we first infer its size from the stream. Then we attempt to
acquire memory permits for a message of that size. If we succeed, then we move on actually process the message.
If we fail, the frame decoder deactivates until sufficient permits are released for the message to be processed
and the handler is activated again. Permits are released back once the message has been fully processed -
after the verb handler has been invoked - on the Stage for the Verb of the message.
Every connection has an exclusive number of permits allocated to it (by default 4MiB). In addition to it,
there is a per-endpoint reserve capacity and a global reserve capacity ResourceLimits.Limit, shared between all
connections from the same host and all connections, respectively. So long as long as the handler stays within
its exclusive limit, it doesn't need to tap into reserve capacity.
If tapping into reserve capacity is necessary, but the handler fails to acquire capacity from either
endpoint of global reserve (and it needs to acquire from both), the handler and its frame decoder become
inactive and register with a InboundMessageHandler.WaitQueue of the appropriate type, depending on which of the reserves
couldn't be tapped into. Once enough messages have finished processing and had their permits released back
to the reserves, InboundMessageHandler.WaitQueue will reactivate the sleeping handlers and they'll resume processing frames.
The reason we 'split' reserve capacity into two limits - endpoing and global - is to guarantee liveness, and
prevent single endpoint's connections from taking over the whole reserve, starving other connections.
One permit per byte of serialized message gets acquired. When inflated on-heap, each message will occupy more
than that, necessarily, but despite wide variance, it's a good enough proxy that correlates with on-heap footprint.| Modifier and Type | Class and Description |
|---|---|
static interface |
InboundMessageHandler.OnHandlerClosed |
static class |
InboundMessageHandler.WaitQueue
A special-purpose wait queue to park inbound message handlers that failed to allocate
reserve capacity for a message in.
|
| Modifier and Type | Method and Description |
|---|---|
void |
channelInactive(io.netty.channel.ChannelHandlerContext ctx) |
void |
channelRead(io.netty.channel.ChannelHandlerContext ctx,
java.lang.Object msg) |
void |
exceptionCaught(io.netty.channel.ChannelHandlerContext ctx,
java.lang.Throwable cause) |
void |
handlerAdded(io.netty.channel.ChannelHandlerContext ctx) |
boolean |
process(org.apache.cassandra.net.FrameDecoder.Frame frame)
Frame processor that the frames should be handed off to.
|
protected void |
releaseProcessedCapacity(int size,
Message.Header header)
Invoked to release capacity for a message that has been fully, successfully processed.
|
java.lang.String |
toString() |
channelActive, channelReadComplete, channelRegistered, channelUnregistered, channelWritabilityChanged, userEventTriggeredensureNotSharable, handlerRemoved, isSharablepublic void channelRead(io.netty.channel.ChannelHandlerContext ctx,
java.lang.Object msg)
channelRead in interface io.netty.channel.ChannelInboundHandlerchannelRead in class io.netty.channel.ChannelInboundHandlerAdapterpublic void handlerAdded(io.netty.channel.ChannelHandlerContext ctx)
handlerAdded in interface io.netty.channel.ChannelHandlerhandlerAdded in class io.netty.channel.ChannelHandlerAdapterpublic boolean process(org.apache.cassandra.net.FrameDecoder.Frame frame)
throws java.io.IOException
java.io.IOExceptionprotected void releaseProcessedCapacity(int size,
Message.Header header)
releaseCapacity(int), but is necessary for the verifier
to be able to delay capacity release for backpressure testing.public void exceptionCaught(io.netty.channel.ChannelHandlerContext ctx,
java.lang.Throwable cause)
exceptionCaught in interface io.netty.channel.ChannelHandlerexceptionCaught in interface io.netty.channel.ChannelInboundHandlerexceptionCaught in class io.netty.channel.ChannelInboundHandlerAdapterpublic void channelInactive(io.netty.channel.ChannelHandlerContext ctx)
channelInactive in interface io.netty.channel.ChannelInboundHandlerchannelInactive in class io.netty.channel.ChannelInboundHandlerAdapterpublic java.lang.String toString()
toString in class java.lang.ObjectCopyright © 2009-2020 The Apache Software Foundation