Long strings are truncated on packing on Android 4.x

[paddybyers]

The code below is an Android app that packs and then unpacks a map containing a single String member ("data"). On Android 4.4.2 this member is truncated when packing if it is 512 characters or longer; 511 or fewer and it works fine.

The problem does not occur on Android 5.x or 6.x.

package my.app;

import android.app.Activity;
import android.os.Bundle;
import android.widget.Toast;
import android.util.Log;

import java.io.ByteArrayOutputStream;
import java.io.IOException;

import org.msgpack.core.MessageFormat;
import org.msgpack.core.MessagePack;
import org.msgpack.core.MessagePacker;
import org.msgpack.core.MessageUnpacker;

public class MainActivity extends Activity
{
    @Override
    public void onCreate(Bundle savedInstanceState)
    {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.main);

		try {
			TestClass testMessage = new TestClass();
			testMessage.data = testString;

			byte[] encoded = writeMsgpack(testMessage);
			TestClass decoded = readMsgpack(encoded);

			String received = decoded.data;
			if(received.equals(testString)) {
				Toast.makeText(this, "Test success", Toast.LENGTH_LONG).show();
			} else {
				Toast.makeText(this, "Test fail", Toast.LENGTH_LONG).show();
				Log.w(TAG, "Decoded string data: " + received);
			}
		} catch (Exception e) {
			e.printStackTrace();
		}
    }

	static byte[] writeMsgpack(TestClass message) {
		byte[] result = null;
		try {
			ByteArrayOutputStream out = new ByteArrayOutputStream();
			MessagePacker packer = MessagePack.newDefaultPacker(out);
			message.writeMsgpack(packer);
			packer.flush();
			result = out.toByteArray();
		} catch(IOException ioe) {
			ioe.printStackTrace();
		}
		return result;
	}

	static TestClass readMsgpack(byte[] packed) {
		TestClass result = null;
		try {
			MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(packed);
			result = (new TestClass()).readMsgpack(unpacker);
		} catch(IOException ioe) {
			ioe.printStackTrace();
		}
		return result;
	}

	static class TestClass {
		public String data;

		void writeMsgpack(MessagePacker packer) throws IOException {
			packer.packMapHeader(1);
			packer.packString("data");
			packer.packString(data);
		}

		TestClass readMsgpack(MessageUnpacker unpacker) throws IOException {
			unpacker.unpackMapHeader();
			String fieldName = unpacker.unpackString();
			MessageFormat fieldFormat = unpacker.getNextFormat();
			if(fieldFormat.equals(MessageFormat.NIL)) {
				unpacker.unpackNil();
			} else if(fieldName.equals("data")) {
				data = unpacker.unpackString();
			}
			return this;
		}
	}

//	/* succeeds */
//	private static String testString = "0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890";

	/* fails */
 	private static String testString = "01234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901";

	private static final String TAG = "MainActivity";
}

[komamitsu]

@paddybyers Thanks for the report.

I think CharsetEncoderICU#encode and/or CharsetEncoderICU#flush in Android 4 (Kitkat) has a bug (See #405 (comment) for details.)

Can you try this workaround?

-   MessagePacker packer = MessagePack.newDefaultPacker(out);
+   MessagePacker packer = new MessagePack.PackerConfig().withSmallStringOptimizationThreshold(Integer.MAX_VALUE).newPacker(out);

[paddybyers]

Can you try this workaround?

Yes, the test passes with that change.

[mattheworiordan]

Out of interest, if that change works, is there an easy fix to this library that can be made to more broadly address the problem or is the recommendation to use that workaround for now?

[frsyuki]

For ease of use, MessagePacker would need additional workaround for Android. Like this:

diff --git a/msgpack-core/src/main/java/org/msgpack/core/MessagePacker.java b/msgpack-core/src/main/java/org/msgpack/core/MessagePacker.java
index 1cd9a98..2ac6298 100644
--- a/msgpack-core/src/main/java/org/msgpack/core/MessagePacker.java
+++ b/msgpack-core/src/main/java/org/msgpack/core/MessagePacker.java
@@ -658,6 +658,8 @@ public class MessagePacker
 
     private static final int UTF_8_MAX_CHAR_SIZE = 6;
 
+    private static final boolean ANDROID4 = ...;
+
     /**
      * Writes a String vlaue in UTF-8 encoding.
      *
@@ -675,8 +677,10 @@ public class MessagePacker
             packRawStringHeader(0);
             return this;
         }
-        else if (s.length() < smallStringOptimizationThreshold) {
+        else if (ANDROID4 || s.length() < smallStringOptimizationThreshold) {
             // Using String.getBytes is generally faster for small strings
+            // Android 4.x has a bug in CharsetEncoder where offset calculation is wrong (see https://github.com/msgpack/msgpack-java/issues/406).
+            // Android 5 and later and 3.x don't have this bug.
             packStringWithGetBytes(s);
             return this;
         }

[komamitsu]

#409 takes care of this issue.