Class Utf8

java.lang.Object
com.google.protobuf.Utf8
final class Utf8 extends Object
A set of low-level, high-performance static utility methods related to the UTF-8 character encoding. This class has no dependencies outside of the core JDK libraries.

There are several variants of UTF-8. The one implemented by this class is the restricted definition of UTF-8 introduced in Unicode 3.1, which mandates the rejection of "overlong" byte sequences as well as rejection of 3-byte surrogate codepoint byte sequences. Note that the UTF-8 decoder included in Oracle's JDK has been modified to also reject "overlong" byte sequences, but (as of 2011) still accepts 3-byte surrogate codepoint byte sequences.

The byte sequences considered valid by this class are exactly those that can be roundtrip converted to Strings and back to bytes using the UTF-8 charset, without loss: 

Arrays.equals(bytes, new String(bytes, StandardCharsets.UTF_8).getBytes(StandardCharsets.UTF_8))

See the Unicode Standard,
Table 3-6. UTF-8 Bit Distribution,
Table 3-7. Well Formed UTF-8 Byte Sequences.

  • Field Details

    • processor

      private static final Utf8.Processor processor
      UTF-8 is a runtime hot spot so we attempt to provide heavily optimized implementations depending on what is available on the platform. The processor is the platform-optimized delegate for which all methods are delegated directly to.

    • ASCII_MASK_LONG

      private static final long ASCII_MASK_LONG
      A mask used when performing unsafe reads to determine if a long value contains any non-ASCII characters (i.e. any byte >= 0x80).
      See Also:

    • MAX_BYTES_PER_CHAR

      static final int MAX_BYTES_PER_CHAR
      Maximum number of bytes per Java UTF-16 char in UTF-8.
      See Also:

  • Constructor Details

    • Utf8

      private Utf8()

  • Method Details

    • createProcessor

      private static Utf8.Processor createProcessor()

    • isValidUtf8

      static boolean isValidUtf8(byte[] bytes)
      Returns true if the given byte array is a well-formed UTF-8 byte sequence.

      This is a convenience method, equivalent to a call to isValidUtf8(bytes, 0, bytes.length).

    • isValidUtf8

      static boolean isValidUtf8(byte[] bytes, int index, int limit)
      Returns true if the given byte array slice is a well-formed UTF-8 byte sequence. The range of bytes to be checked extends from index index, inclusive, to limit, exclusive.

    • isValidUtf8

      static boolean isValidUtf8(ByteBuffer buffer)
      Determines if the given ByteBuffer is a valid UTF-8 string.

      Selects an optimal algorithm based on the type of ByteBuffer (i.e. heap or direct) and the capabilities of the platform.

      Parameters:
      buffer - the buffer to check.
      See Also:

    • isValidUtf8Internal

      private static boolean isValidUtf8Internal(byte[] bytes, int index, int limit)

    • isValidUtf8NonAscii

      private static boolean isValidUtf8NonAscii(byte[] bytes, int index, int limit)

    • isValidUtf8BufferInternal

      private static boolean isValidUtf8BufferInternal(ByteBuffer buffer, int index, int limit)

    • encodedLength

      static int encodedLength(String string)
      Returns the number of bytes in the UTF-8-encoded form of sequence. For a string, this method is equivalent to string.getBytes(UTF_8).length, but is more efficient in both time and space.

    • encodedLengthGeneral

      private static int encodedLengthGeneral(String string, int start) throws Utf8.UnpairedSurrogateException
      Throws:
      Utf8.UnpairedSurrogateException

    • encode

      static int encode(String in, byte[] out, int offset, int length)

    • decodeUtf8

      static String decodeUtf8(ByteBuffer buffer, int index, int size) throws InvalidProtocolBufferException
      Decodes the given UTF-8 portion of the ByteBuffer into a String.
      Throws:
      InvalidProtocolBufferException - if the input is not valid UTF-8.

    • decodeUtf8

      static String decodeUtf8(byte[] bytes, int index, int size) throws InvalidProtocolBufferException
      Decodes the given UTF-8 encoded byte array slice into a String.
      Throws:
      InvalidProtocolBufferException - if the input is not valid UTF-8.

    • encodeUtf8

      static void encodeUtf8(String in, ByteBuffer out)
      Encodes the given characters to the target ByteBuffer using UTF-8 encoding.

      Selects an optimal algorithm based on the type of ByteBuffer (i.e. heap or direct) and the capabilities of the platform.

      Parameters:
      in - the source string to be encoded
      out - the target buffer to receive the encoded string.
      See Also:

    • estimateConsecutiveAscii

      private static int estimateConsecutiveAscii(ByteBuffer buffer, int index, int limit)
      Counts (approximately) the number of consecutive ASCII characters in the given buffer. The byte order of the ByteBuffer does not matter, so performance can be improved if native byte order is used (i.e. no byte-swapping in ByteBuffer.getLong(int)).
      Parameters:
      buffer - the buffer to be scanned for ASCII chars
      index - the starting index of the scan
      limit - the limit within buffer for the scan
      Returns:
      the number of ASCII characters found. The stopping position will be at or before the first non-ASCII byte.