Problem description & Simulation
On the online login interface, through monitoring and viewing, there are type conversion exceptions. The specific error reports are shown in the figure below
This error message is displayed on the dubbo consumer side, and most logins are normal, and a few will report type conversion exceptions. Colleagues solve this problem by changing the method name + displaying the specified serialization id, but what is the real reason for this problem? No serialization id specified? Or dubbo method overloading? Why does the server not display this error message?, Next, simulate the situation according to the error.
The online operation shows that the client reporting an error is deployed in the container, jdk version
The service provider is mixed, with virtual machines and containers. The jdk version of the container is the same, and the jdk version of the virtual machine is the same
At first, it was thought that the container called the service of the virtual machine because the specified serialization id was not displayed, and the decoding problem was caused by inconsistent jvm versions. However, after analysis and test, it was found that this was not the case. The simulation is as follows:
Define a dubbo service. The method is overloaded and the input parameter does not display the specified serialization id. the code is as follows
//Defining dubbo services
public interface ProductService {
Result<ProductVO> findProduct(String data);
Result<ProductVO> findProduct(ProductDTO product);
}
//Input parameter
@Data
public class ProductDTO implements Serializable {
//The specified serialization id is not displayed
private Integer productId;
private String sn;
private String code;
}
//Out parameter
@Data
public class ProductVO implements Serializable{
private static final long serialVersionUID = 4529782262922750326L;
private Integer productId;
private String productName;
}
dubbo client calls productservice Findproduct (productdto product) and use jdk1 8.0_ Version 202, the service provider uses jdk1 8.0_ In version 73, after the test (jmeter pressure test), it is found that there is no type conversion exception. Now it is eliminated through code analysis.
Analyze &dubbo provider's request processing process
The reverse order method is adopted, and arthas is used to decompile the proxy class generated by dubbo. The proxy class generated by ProductService is Wrapper2, which is as follows
public Object invokeMethod(Object object, String name, Class[] classArray, Object[] objectArray)
throws InvocationTargetException {
ProductService productService;
try {
productService = (ProductService) object;
} catch (Throwable throwable) {
throw new IllegalArgumentException(throwable);
}
try {
if ("findProduct".equals(name) && classArray.length == 1
&& classArray[0].getName().equals("java.lang.String")) {
return productService.findProduct((String) objectArray[0]);
}
if ("findProduct".equals(name) && classArray.length == 1
&& classArray[0].getName().equals("org.pangu.dto.ProductDTO")) {
return productService.findProduct((ProductDTO) objectArray[0]);
}
} catch (Throwable throwable) {
throw new InvocationTargetException(throwable);
}
throw new NoSuchMethodException(new StringBuffer().append("Not found method \"").append(name)
.append("\" in class org.pangu.api.ProductService.").toString());
}
}
By looking at the decompiled code, we know that the dubbo method is overloaded, and the corresponding target method execution will be found according to the method type and the number of parameters. For my online question, the parameter is ProductDTO. If findProduct(String data) is called, it means that classArray[0], that is, the parameter type is String type. How do you get the parameter type? According to what I wrote before dubbo process analysis , check the source code on COM alibaba. dubbo. rpc. proxy. Abstractproxyinvoker #invoke (invocation), the code is as follows
Method name + method type + aspect parameters are encapsulated in Invocation. Then, find the source of Invocation and find it in DubboProtocol , the anonymous inner class of DubboProtocol. Specifically, in the reply (exchange channel, object message) method, the parameter message is Invocation.
Then look at where to call dubboprotocol $1 The reply (exchangechannel, object message) method is available on COM alibaba. dubbo. remoting. exchange. support. header. Headerexchangehandler #handlerequest (exchange channel, request req) method, com alibaba. dubbo. remoting. exchange. Request. GetData () gets this Invocation, that is, DecodeableRpcInvocation, then look at request and request Source of MDATA;
Then look up at com alibaba. dubbo. remoting. exchange. support. header. The input parameter message of headerexchangehandler #received (channel, object message) is Request;
Keep looking up, com alibaba. dubbo. remoting. transport. The input parameter of decodehandler #received (channel, object message) is Request, where Request MDATA is the Invocation for decoding (it has been decoded by the IO thread by default, and the decoding will not be performed here. DecodeableRpcInvocation#hasDecoded=true).
Keep looking up, com alibaba. dubbo. remoting. transport. dispatcher. ChannelEventRunnable #run() thread, and the message attribute is Request, then you can only find out how the ChannelEventRunnable was created and submitted
Keep looking up at com alibaba. dubbo. remoting. transport. dispatcher. all. Create a ChannelEventRunnable in the allchannelhandler #received (channel, object message) method and submit it to the thread pool for execution.
Keep looking up at com alibaba. dubbo. remoting. exchange. support. header. HeartbeatHandler. Received (channel, channel, object message), the input parameter message is Request
Keep looking up, com alibaba. dubbo. remoting. transport. MultiMessageHandler. received(Channel channel, Object message)
Keep looking up, com alibaba. dubbo. remoting. transport. AbstractPeer. received(Channel ch, Object msg)
Keep looking up, com alibaba. dubbo. remoting. transport. netty4. NettyServerHandler. Channelread (channelhandlercontext CTX, object MSG). Seeing this indicates that it is the work thread of network, and nettyserverhandler is an inbound & outbound event
dubbo service netty starts the added inbound & outbound, i.e. pipeline chain[HeadContext InternalDecoder InternalEncoder NettyServerHandler TailContext], indicating that there must be a channelRead event for executing InternalDecoder in front. At this time, the input parameter message is Request.
The following focuses on the analysis of the channelRead event of the InternalDecoder. The execution stack is as follows:
InternalDecoder(io.netty.handler.codec.ByteToMessageDecoder).channelRead(ChannelHandlerContext ctx, Object msg) InternalDecoder(io.netty.handler.codec.ByteToMessageDecoder).callDecode(ChannelHandlerContext ctx, ByteBuf in, List<Object> out) InternalDecoder.decode(ChannelHandlerContext ctx, ByteBuf input, List<Object> out) DubboCountCodec.decode(Channel channel, ChannelBuffer buffer) DubboCodec(ExchangeCodec).decode(Channel channel, ChannelBuffer buffer) DubboCodec(ExchangeCodec).decode(Channel channel, ChannelBuffer buffer, int readable, byte[] header) DubboCodec.decodeBody(Channel channel, InputStream is, byte[] header) DecodeableRpcInvocation.decode() DecodeableRpcInvocation.decode(Channel channel, InputStream input)
InternalDecoder is the inbound event of netty pipeline and executes channelRead. The specific logic is in InternalDecoder In decode (channelhandlercontext CTX, bytebuf input, list < Object > out), the code is as follows
Then trigger the next inbound channelRead action, and the incoming Request is the Request. The code description is as follows
Then look at dubbocount codec Decode (channel, channelbuffer, buffer). Decode here
//com.alibaba.dubbo.rpc.protocol.dubbo.DubboCountCodec#decode(Channel channel, ChannelBuffer buffer)
public Object decode(Channel channel, ChannelBuffer buffer) throws IOException {
int save = buffer.readerIndex();//Get read location
MultiMessage result = MultiMessage.create();//MultiMessage is a collection of requests
do {
Object obj = codec.decode(channel, buffer);//DubboCodec is used for decoding. Different processing is carried out according to the decoding results
if (Codec2.DecodeResult.NEED_MORE_INPUT == obj) {//It indicates that tcp packet sticking has occurred and exits the loop
buffer.readerIndex(save);
break;
} else {
result.addMessage(obj);//Add obj, that is, Request, to the collection MultiMessage
logMessageLength(obj, buffer.readerIndex() - save);
save = buffer.readerIndex();//Set a new buffer read position and continue to use DubboCodec for decoding
}
} while (true);
if (result.isEmpty()) {
return Codec2.DecodeResult.NEED_MORE_INPUT;
}
if (result.size() == 1) {//If there is only one element in the MultiMessage, it indicates that there is no packet sticking this time
return result.get(0);//Return Request
}
return result;//Return the MultiMessage and get the collection traversal processing of the Request in the subsequent MultiMessagehandler
}
Then look at dubbocodec (exchange code) Decode (channel, channelbuffer) decoding process, how to decode the dubbo protocol, first look at the message structure of the dubbo protocol
Then look at the code and decode the message structure
//DubboCodec(ExchangeCodec).decode(Channel channel, ChannelBuffer buffer)
@Override
public Object decode(Channel channel, ChannelBuffer buffer) throws IOException {
int readable = buffer.readableBytes();
byte[] header = new byte[Math.min(readable, HEADER_LENGTH)];
buffer.readBytes(header);//Store buffer bytes in header
return decode(channel, buffer, readable, header);
}
//DubboCodec(ExchangeCodec).decode(Channel channel, ChannelBuffer buffer, int readable, byte[] header)
@Override
protected Object decode(Channel channel, ChannelBuffer buffer, int readable, byte[] header) throws IOException {
// check magic number.
if (readable > 0 && header[0] != MAGIC_HIGH
|| readable > 1 && header[1] != MAGIC_LOW) {//Non magic number, indicating the beginning of non dubbo message, indicating that tcp unpacking / sticking occurred
int length = header.length;
if (header.length < readable) {
header = Bytes.copyOf(header, readable);
buffer.readBytes(header, length, readable - length);
}
for (int i = 1; i < header.length - 1; i++) {
if (header[i] == MAGIC_HIGH && header[i + 1] == MAGIC_LOW) {
buffer.readerIndex(buffer.readerIndex() - header.length + i);
header = Bytes.copyOf(header, i);
break;
}
}
return super.decode(channel, buffer, readable, header);
}
// check length.
if (readable < HEADER_LENGTH) {//Why is it less than 16? Because magic (2) + fall (1) + status (1) + invokerid (8) + bodylenght (4) of the dubbo message is 16 bytes, which is less than 16 bytes. Unpacking must have occurred. The data received this time has no body
return DecodeResult.NEED_MORE_INPUT;
}
// get data length.
int len = Bytes.bytes2int(header, 12);//The reason for 12 is that magic (2) + fall (1) + status (1) + invokerid (8) of the dubbo message is equal to 12. Take 4 bits after 12 bits and convert them to int, which is the length of the body
checkPayload(channel, len);
int tt = len + HEADER_LENGTH;
if (readable < tt) {//The readable number is less than bodylen+16, indicating that tcp unpacks and needs to continue to read in the network
return DecodeResult.NEED_MORE_INPUT;
}
// limit input stream.
ChannelBufferInputStream is = new ChannelBufferInputStream(buffer, len);
try {
return decodeBody(channel, is, header);//Decoding body content
} finally {
if (is.available() > 0) {
try {
if (logger.isWarnEnabled()) {
logger.warn("Skip input stream " + is.available());
}
StreamUtils.skipUnusedStream(is);
} catch (IOException e) {
logger.warn(e.getMessage(), e);
}
}
}
}
Then look at the decoding dubbo body, on COM alibaba. dubbo. rpc. protocol. dubbo. DubboCodec#decodeBody
//com.alibaba.dubbo.rpc.protocol.dubbo.DubboCodec#decodeBody(Channel channel, InputStream is, byte[] header)
protected Object decodeBody(Channel channel, InputStream is, byte[] header) throws IOException {
byte flag = header[2], proto = (byte) (flag & SERIALIZATION_MASK);
// get request id.
long id = Bytes.bytes2long(header, 4);
if ((flag & FLAG_REQUEST) == 0) {//Yes, response, code
//ellipsis
} else {//Request, decode
// decode request.
Request req = new Request(id);
req.setVersion(Version.getProtocolVersion());
req.setTwoWay((flag & FLAG_TWOWAY) != 0);
if ((flag & FLAG_EVENT) != 0) {
req.setEvent(Request.HEARTBEAT_EVENT);
}
try {
Object data;
if (req.isHeartbeat()) {//heartbeat
data = decodeHeartbeatData(channel, CodecSupport.deserialize(channel.getUrl(), is, proto));
} else if (req.isEvent()) {//event
data = decodeEventData(channel, CodecSupport.deserialize(channel.getUrl(), is, proto));
} else {
DecodeableRpcInvocation inv;
if (channel.getUrl().getParameter(
Constants.DECODE_IN_IO_THREAD_KEY,
Constants.DEFAULT_DECODE_IN_IO_THREAD)) {//The default is decoding in the network thread
inv = new DecodeableRpcInvocation(channel, req, is, proto);
inv.decode();//Decode the dubbo body and save the decoding result in the DecodeableRpcInvocation
} else {
inv = new DecodeableRpcInvocation(channel, req,
new UnsafeByteArrayInputStream(readMessageData(is)), proto);//Otherwise, it is decoded in the business thread ChannelEventRunnable
}
data = inv;
}
req.setData(data);//Save the Invocation to request mData
} catch (Throwable t) {
if (log.isWarnEnabled()) {
log.warn("Decode request failed: " + t.getMessage(), t);
}
// bad request
req.setBroken(true);
req.setData(t);
}
return req;
}
}
Next, look at DecodeableRpcInvocation decoding dubbo body
//com.alibaba.dubbo.rpc.protocol.dubbo.DecodeableRpcInvocation#decode()
@Override
public void decode() throws Exception {
if (!hasDecoded && channel != null && inputStream != null) {
try {
decode(channel, inputStream);//decode
} catch (Throwable e) {
if (log.isWarnEnabled()) {
log.warn("Decode rpc invocation failed: " + e.getMessage(), e);
}
request.setBroken(true);
request.setData(e);
} finally {
hasDecoded = true;//After decoding, the bit has been decoded so that it will not be decoded in the ChannelEventRunnable thread
}
}
}
//com.alibaba.dubbo.rpc.protocol.dubbo.DecodeableRpcInvocation#decode(com.alibaba.dubbo.remoting.Channel, java.io.InputStream)
@Override
public Object decode(Channel channel, InputStream input) throws IOException {
ObjectInput in = CodecSupport.getSerialization(channel.getUrl(), serializationType)
.deserialize(channel.getUrl(), input);//Obtain the deserialization object according to the serialization ID, and the adaptation of dubbo spi
String dubboVersion = in.readUTF();//Read dubbo version from input stream
request.setVersion(dubboVersion);
setAttachment(Constants.DUBBO_VERSION_KEY, dubboVersion);
setAttachment(Constants.PATH_KEY, in.readUTF());//Read path from input stream
setAttachment(Constants.VERSION_KEY, in.readUTF());//Read version from input stream
setMethodName(in.readUTF());//Read the target method name of the call from the input stream
try {
Object[] args;
Class<?>[] pts;
String desc = in.readUTF();//Read the parameter descriptor from the input stream, that is, the type of the parameter, such as [Ljava/lang/String]
if (desc.length() == 0) {//The dubbo calling method does not have an input parameter
pts = DubboCodec.EMPTY_CLASS_ARRAY;
args = DubboCodec.EMPTY_OBJECT_ARRAY;
} else {//There are input parameters in dubbo calling method
pts = ReflectUtils.desc2classArray(desc);//Type descriptors are converted to types, such as [ljava / Lang / String = > ljava.lang.string
args = new Object[pts.length];//Parameter length
for (int i = 0; i < args.length; i++) {
try {
args[i] = in.readObject(pts[i]);//Read parameters from the input stream, here readObject, and perform deserialization
} catch (Exception e) {
if (log.isWarnEnabled()) {
log.warn("Decode argument failed: " + e.getMessage(), e);
}
}
}
}
setParameterTypes(pts);//Save the parameter type to the Invocation object, that is, the parameterTypes property
Map<String, String> map = (Map<String, String>) in.readObject(Map.class);//Read implicit parameters from the input stream and decode them
if (map != null && map.size() > 0) {
Map<String, String> attachment = getAttachments();
if (attachment == null) {
attachment = new HashMap<String, String>();
}
attachment.putAll(map);
setAttachments(attachment);
}
//decode argument ,may be callback
for (int i = 0; i < args.length; i++) {
args[i] = decodeInvocationArgument(channel, this, pts, i, args[i]);
}
setArguments(args);
} catch (ClassNotFoundException e) {
throw new IOException(StringUtils.toString("Read invocation data failed.", e));
} finally {
if (in instanceof Cleanable) {
((Cleanable) in).cleanup();
}
}
return this;
}
It can be seen from the decoding of dubbo body that the target method name, method type, method input parameters and implicit parameters obtained from the input stream decoding are saved to the Invocation object (i.e. decodablerpcinvocation), in which the read input parameters and implicit parameters use serialization decoding (serialization id is required), Getting the method name + parameter type from the input stream does not use the deserialization of the object.
dubbo provider processing receiving summary
After analyzing the calling process from the network to the dubbo business thread pool and how to decode at the dubbo prodiver end, the following is a summary:
The dubbo provider receives and processes consumer requests in two steps
1. Network communication, decode on the io thread, and save the decoding result to Request.
2. The IO thread calls the dubbo business thread, passes in the decoding result Request, calls the target method through the Invoker, and passes in the object, method name, parameter type and parameter of the target method to be executed to call the target method.
Analysis of the problem
Solve 2 problems
Question 1: why is an error ClassCastException reported on the server and there is no error log on the server? There is an error log only on the client
Because calling the target method in Dubbo proxy class Wrapper2 leads to ClassCastException, the exception is captured and encapsulated as InvocationTargetException, thrown up, and then displayed in com alibaba. dubbo. rpc. proxy. The exception in abstractproxyinvoker #invoke is captured and encapsulated as RpcResult, and then the exception information in ExceptionFilter is encapsulated as RuntimeException and returned to the client. There is no log printing, so no error log is generated, so the server can't see it.
Question 2: will dubbo method overloading cause problems?
Conclusion: basically not. The dynamic proxy class WrapperX of dubbo will call the target method according to the methodName + parameter type + parameter of Invocation, so it will not. A big man on the Internet said that dubbo method overloading would lead to problems in some cases, but his sentences were not smooth and messy, and blue and green were traffic isolated and could not be adjusted wrong. I think his examples are inappropriate. You can refer to those who are interested Problems and Reflections on dubbo's homonymy method.
Question 3: is it caused by not explicitly specifying the serialization id?
Through the previous analysis, it is determined that the parameter type is string (originally DTO type), which leads to the exception caused by converting the parameter to string when executing the target method. The parameter type comes from the Invocation object (i.e. RpcInvocation.parameterTypes), and the Invocation comes from Request MDATA, and Request is decoded from network communication, which is on COM alibaba. dubbo. rpc. protocol. dubbo. String desc = in. In decodeablerpcinvocation #decode (COM. Alibaba. Dubbo. Remoting. Channel, Java. Io. InputStream) readUTF(); The byte stream is read from the input stream and decoded into a parameter type descriptor, which does not involve serialization and deserialization of objects.
Look at the client coding code InternalEncoder. The coding parameter type code is shown in the figure below
The client sends the establishment Request on COM alibaba. dubbo. remoting. exchange. support. header. Headerexchangechannel #Request (java.lang.object, int), and the Invocation object encapsulates the method name + parameter in the InvokerInvocationHandler (new RpcInvocation(method, args) at the entrance of Dubbo call to create an Invocation object, and then the parameter type is saved in the Invocation object.
From this analysis, not displaying the specified serialization id does not cause this problem.
The jdk version and the specified serialization ID are excluded. What causes the dubbo method overload to call ClassCastException? The online pre release environment and the production network are interconnected. Is it the pre release environment? The application manually deployed by colleagues can only enter the parameter String method (the version is not synchronized with the production)? My colleagues can't remember clearly and can't check. I can't know the answer to this question for the time being.
According to my guess, the problem may be that the service deployed in the pre launch environment is not synchronized with the production version (due to the lack of findProduct(ProductDTOdata). Our pre launch and generation networks are interconnected. It should be that the production client calls the pre launch environment service, but the service deployed in the pre launch environment does not have findProduct(ProductDTOdata).
Why do I need to display the specified serialization id
The tcp communication used by rpc calls needs to convert the object into a binary stream for sending (encoding) and receiving (decoding), so there needs to be a set of rules to convert the java object in memory into a binary stream. Serialization does this.
When using native serialization, the serialVersionUID plays a role similar to the version number. If the serialVersionUID is different during deserialization, an InvalidClassException will be thrown.
When using the native serialization method, it is strongly recommended to specify a serialVersionUID. If it is not specified, the jvm will automatically calculate a value as the serialVersionUID during the serialization process. Because this method of calculating the serialVersionUID at runtime depends on the implementation method of the jvm, If the jvm implementations of serialization and deserialization are different, the exception InvalidClassException may be thrown, so it is strongly recommended to specify serialVersionUID.
Does not display the specified serialization ID actually cause a problem?
Define a dubbo input parameter, do not display the specified serialization id, the client operation does not change, and the service input parameter adds or deletes fields (the class structure changes). It is found that all requests can be made normally. It is not that the rpc parameter class structure changes when the specified serialization id is not displayed, which does not cause any problems, Of course, I only conducted this test in the jdk8 Version (of course, it's jdk8 now). In this case, it doesn't seem to affect anything if the specified serialization id is not displayed in the actual use process.
It is said on the Internet that not displaying the specified serialization id will lead to a problem: for example, the input parameter does not display the specified serialization id. later, there is a requirement to add a field to the input parameter, and if the specified serialization id is not displayed, a serialization id is added conveniently. In this way, the client application running on the line still references the old jar, When a new service is deployed, it will send serialization failure (the serialization id generated by the client jvm is different from the specified serialization id displayed by the server). It seems that this situation can not be avoided. However, I have tested that adding fields, deleting fields and adding methods to dubbo parameters without displaying the specified serialization id will not cause deserialization problems (jdk8, tested under dubbo 2.6.8), and the requests are normal. The verification results show that the serialization id generated by the jvm has nothing to do with the structure of the class. You can refer to others test result , the same as my test results.
So can we boldly not specify the serialization id? It's not recommended. I don't know how the jvm generates serialization IDs. I don't specify what happens on the line one day.
After verification for a long time, I got an actual verification conclusion that it doesn't matter if I don't specify the serialization id, but I can't rest assured and dare not display the specified serialization id. I'm crazy...
Final conclusion
According to the actual verification (jdk8, tested under dubbo 2.6.8), when the specified serialization id is not displayed, the transmission object of dubbo will not cause deserialization problems in adding fields, deleting fields and adding methods. However, it is strongly recommended to display the specified serialization id in case the serialization id generated by the jvm is incompatible
ending
After analyzing for such a long time, the cause of this problem was not found, but the understanding of dubbo communication layer was deepened. The following article records the summarized dubbo communication layer