byte-data

JavaScript binary parser for any browser or environment.
Copyright (c) 2017-2019 Rafael da Silva Rocha.
https://github.com/rochars/byte-data

byte-data is JavaScript binary parser for any browser or environment.

• MIT licensed
• Compatible with IE6+ and any environment with ES3/ES5/ES6+ support
• Tested in little-endian and big-endian machines!
• Zero dependencies

Pack/unpack:

• Integers, unsigned and signed (two's complement)
• 16-bit half-precision floating-point numbers
• 32-bit single-precision floating-point numbers
• 64-bit double-precision floating-point numbers
• Little-endian and big-endian words
• UTF-8 strings (1 to 4 bytes per character, invalid characters are replaced)

Install

npm install byte-data

In the Browser

Use the byte-data.js file in the /dist folder:

<script src="./dist/byte-data.js"></script>
<script>
  // Pack a 32-bit floating-point number
  var packed = byteData.pack(2.1474836, {bits: 32, fp: true});
</script>

Or load it from the jsDelivr CDN:

<script src="https://cdn.jsdelivr.net/npm/byte-data"></script>
<script>
  var packed = byteData.pack(2.1474836, {bits: 32, fp: true});
</script>

Or load it from unpkg:

<script src="https://unpkg.com/byte-data"></script>
<script>
  var packed = byteData.pack(2.1474836, {bits: 32, fp: true});
</script>

Browser compatibility

This module is distributed as a minified UMD transpiled to ES3 and compatible with IE6+. It should work in all modern browsers and environments that support ES3/ES5/ES6+.

The polyfills used in the compilation are distributed with the package in the scripts/ folder. The polyfills are for the defineProperty and getOwnPropertyDescriptor properties of Object, and are not used in case those properties are already defined.

If you are not using a package manager to install this module, you can get the it via CDNs:

<script src="https://cdn.jsdelivr.net/npm/byte-data"></script>
<script src="https://unpkg.com/byte-data"></script>

Cross-browser tests powered by

Node

const byteData = require('byte-data');

// Pack a signed 16-bit integer to a existing byte buffer
// Start writing on index '4' of the buffer
byteData.packTo(1077, {bits: 16, signed: true}, buffer, 4);

// Pack a usigned 8-bit unsigned integer, returns a
// array with the number represented as bytes
let packed = byteData.pack(128, {bits: 8});

Or import just what you need:

import { pack } from 'byte-data';

// Pack a 8-bit unsigned integer
let packed = pack(128, {bits: 8});

About

pack and packTo

pack(num, theType) will return a Array with the bytes of the passed value.

let packed = pack(123, {bits: 16});

packTo(num, theType, buffer, index) will write the bytes of the number to the provided buffer (Uint8Array or Array), start writing on index.

let buffer = new Uint8Array(4);
packTo(402, {bits: 16}, buffer, 2);

index can be ommited and will default to zero:

let buffer = new Uint8Array(4);
packTo(402, {bits: 16}, buffer);

Packing null, false, true and undefined

Packing the following values

• undefined
• null
• true
• false

will values throw a TypeError.

Unpacking and input buffer length

When unpacking values, extra bytes in the end of the buffer are ignored and insufficient bytes will return a empty array by default.

You can unpack in safe mode with the optional safe param set to true. In safe mode insufficient bytes in the input array or extra bytes in the end of the input array will cause a 'Bad buffer length' error:

// throws a 'Bad buffer length' error
byteData.unpackArrayTo([0xff], theType, output, 0, buffer.length, true);

// throws a 'Bad buffer length' error
byteData.unpackArrayTo(
  [0xff, 0xff, 0xff], theType, output, 0, buffer.length, true);

// throws a 'Bad buffer length' error
byteData.unpack([0xff], {bits: 16}, 0, true);

// throws a 'Bad buffer length' error
byteData.unpack([0xff, 0xff, 0xff], {bits: 16}, 2, true);

// do not throw error
byteData.unpack([0xff, 0xff, 0xff], {bits: 16}, 1, true); 

Floating-point numbers

• Floating-point numbers are IEEE 754 standard.
• Overflows are rounded towards Infinity and -Infinity.
• NaN is packed as quiet NaN. Both quiet NaN and signaling NaN can be unpacked.
• Support packing and unpacking negative zeros.
• Support packing and unpacking Infinity and negative Infinity

Minifloats

Currently only 16-bit half-precision.

Integers

• Overflow on integers will throw a RangeError.
• Packing values other than integers will throw a TypeError.

To clamp integers on overflow and avoid RangeError, set the optional clamp param to true:

// Set clamp to true; values will be packed
// as their max and min values on overflow
pack(value, theType, true);
packTo(value, theType, buffer, index, true);
packArrayTo(values, theType, buffer, index, true);

Signed integers

Signed integers are two's complement.

Strings

UTF-8 strings with 1 to 4 bytes per character can be packed and unpacked. BOM is kept untouched if present. Invalid characters are replaced with Unicode Character 'REPLACEMENT CHARACTER' (U+FFFD). Packing values other than strings with packString() or packStringTo() will throw a TypeError.

Reading strings from buffers

Use unpackString(buffer, index, end). The paramters index and end determine a slice of the buffer to read. End is non-inclusive. So to read the first 4 bytes of a buffer:

let str = unpackString(buffer, 0, 4);
// read from buffer[0], buffer[1], buffer[2], buffer[3]

If index and end are ommited unpackString(buffer) will read the entire buffer:

let str = unpackString(buffer);

Writing strings to buffers

packStringTo(str, buffer, index=0) will write the string to the provided buffer (Uint8Array or Array), starting on the index. Index defaults to zero if ommited (start from the beginning of the buffer).

// Will write the string to the buffer, array or Uint8Array
let buffer = [];
packStringTo(str, buffer, 0);

// Will return the bytes of the string in a array
let strBytes = packString(str);

Types

Types are user-defined objects like this:

const binary32 = {
  bits: 32, // required
  signed: true, // optional, defaults to false
  fp: true, // optional, defaults to false, true for floating-point numbers
  be: false // optional, defaults to false, true for big-endian
}

Tests on big-endian systems

Use QEMU with this PowerPC/Debian image:
https://people.debian.org/~aurel32/qemu/powerpc/

API

// Strings
/**
 * Read a string of UTF-8 characters from a byte buffer.
 * @param {!(Uint8Array|Array<number>)} buffer A byte buffer.
 * @param {number} [index=0] The buffer index to start reading.
 * @param {number} [end=buffer.length] The index to stop reading, non inclusive.
 * @return {string}
 */
function unpackString(buffer, index=0, len=buffer.length) {}

/**
 * Write a string of UTF-8 characters as a byte buffer.
 * @param {string} str The string to pack.
 * @return {!Array<number>} The UTF-8 string bytes.
 * @throws {TypeError} If 'str' is not a string.
 */
function packString(str) {}

/**
 * Write a string of UTF-8 characters to a byte buffer.
 * @param {string} str The string to pack.
 * @param {!(Uint8Array|Array<number>)} buffer The output buffer.
 * @param {number} [index=0] The buffer index to start writing.
 * @return {number} The next index to write in the buffer.
 * @throws {TypeError} If 'str' is not a string.
 */
function packStringTo(str, buffer, index=0) {}

// Numbers
/**
 * Pack a array of numbers to a byte buffer.
 * All other packing functions are interfaces to this function.
 * @param {!(Array<number>|TypedArray)} values The values to pack.
 * @param {!{bits:number,
 *   fp: (boolean|undefined),
 *   signed: (boolean|undefined),
 *   be: (boolean|undefined)}} theType The type definition.
 * @param {!(Uint8Array|Array<number>)} buffer The buffer to write on.
 * @param {number} [index=0] The buffer index to start writing.
 * @param {boolean} [clamp=false] True to clamp ints on overflow.
 * @return {number} The next index to write.
 * @throws {Error} If the type definition is not valid.
 * @throws {RangeError} On overflow if clamp is set to false.
 * @throws {TypeError} If 'values' is not a array of numbers.
 * @throws {TypeError} If 'values' is not a array of ints and type is int.
 */
function packArrayTo(values, theType, buffer, index=0, clamp=false) {}

/**
 * Unpack a array of numbers from a byte buffer to a array or a typed array.
 * All other unpacking functions are interfaces to this function.
 * @param {!(Uint8Array|Array<number>)} buffer The byte buffer.
 * @param {!{bits:number,
 *   fp: (boolean|undefined),
 *   signed: (boolean|undefined),
 *   be: (boolean|undefined)}} theType The type definition.
 * @param {!(TypedArray|Array<number>)} output The output array or typed array.
 * @param {number} [start=0] The buffer index to start reading.
 * @param {number} [end=buffer.length] The buffer index to stop reading.
 * @param {boolean} [safe=false] If set to false, extra bytes in the end of
 *   the input array are ignored and input buffers with insufficient bytes will
 *   write nothing to the output array. If safe is set to true the function
 *   will throw a 'Bad buffer length' error on the aforementioned cases.
 * @throws {Error} If the type definition is not valid.
 * @throws {Error} On bad input buffer length if on safe mode.
 */
function unpackArrayTo(
  buffer, theType, output, index=0, end=buffer.length, safe=false) {}

/**
 * Pack a number to a byte buffer.
 * @param {number} value The value.
 * @param {!{bits:number,
 *   fp: (boolean|undefined),
 *   signed: (boolean|undefined),
 *   be: (boolean|undefined)}} theType The type definition.
 * @param {!(Uint8Array|Array<number>)} buffer The byte buffer to write on.
 * @param {number} [index=0] The buffer index to write.
 * @param {boolean} [clamp=false] True to clamp ints on overflow.
 * @return {number} The next index to write.
 * @throws {Error} If the type definition is not valid.
 * @throws {RangeError} On overflow if clamp is set to false.
 * @throws {TypeError} If 'value' is not a number.
 * @throws {TypeError} If 'value' is not a int and type is int.
 */
function packTo(value, theType, buffer, index=0, clamp=false) {}

/**
 * Pack a number as a array of bytes.
 * @param {number} value The number to pack.
 * @param {!{bits:number,
 *   fp: (boolean|undefined),
 *   signed: (boolean|undefined),
 *   be: (boolean|undefined)}} theType The type definition.
 * @param {boolean} [clamp=false] True to clamp ints on overflow.
 * @return {!Array<number>} The packed value.
 * @throws {Error} If the type definition is not valid.
 * @throws {RangeError} On overflow if clamp is set to false.
 * @throws {TypeError} If 'value' is not a number.
 * @throws {TypeError} If 'value' is not a int and type is int.
 */
function pack(value, theType, clamp=false) {}

/**
 * Unpack a number from a byte buffer.
 * @param {!(Uint8Array|Array<number>)} buffer The byte buffer.
 * @param {!{bits:number,
 *   fp: (boolean|undefined),
 *   signed: (boolean|undefined),
 *   be: (boolean|undefined)}} theType The type definition.
 * @param {number} [index=0] The buffer index to read.
 * @param {boolean} [safe=false] If set to false, extra bytes in the end of
 *   the input array are ignored and input buffers with insufficient bytes will
 *   write nothing to the output array. If safe is set to true the function
 *   will throw a 'Bad buffer length' error on the aforementioned cases.
 * @return {number}
 * @throws {Error} If the type definition is not valid.
 * @throws {Error} On bad input buffer length if on safe mode.
 */
function unpack(buffer, theType, index=0, safe=false) {}

/**
 * Pack a array of numbers as a array of bytes.
 * @param {!(Array<number>|TypedArray)} values The values to pack.
 * @param {!{bits:number,
 *   fp: (boolean|undefined),
 *   signed: (boolean|undefined),
 *   be: (boolean|undefined)}} theType The type definition.
 * @param {boolean} [clamp=false] True to clamp ints on overflow.
 * @return {!Array<number>} The packed values.
 * @throws {Error} If the type definition is not valid.
 * @throws {RangeError} On overflow if clamp is set to false.
 * @throws {TypeError} If 'values' is not a array of numbers.
 * @throws {TypeError} If 'values' is not a array of ints and type is int.
 */
function packArray(values, theType, clamp=false) {}

/**
 * Unpack a array of numbers from a byte buffer.
 * @param {!(Uint8Array|Array<number>)} buffer The byte buffer.
 * @param {!{bits:number,
 *   fp: (boolean|undefined),
 *   signed: (boolean|undefined),
 *   be: (boolean|undefined)}} theType The type definition.
 * @param {number} [start=0] The buffer index to start reading.
 * @param {number} [end=buffer.length] The buffer index to stop reading.
 * @param {boolean} [safe=false] If set to false, extra bytes in the end of
 *   the input array are ignored and input buffers with insufficient bytes will
 *   write nothing to the output array. If safe is set to true the function
 *   will throw a 'Bad buffer length' error on the aforementioned cases.
 * @return {!Array<number>}
 * @throws {Error} If the type definition is not valid.
 * @throws {Error} On bad input buffer length if on safe mode.
 */
function unpackArray(buffer, theType, start=0, end=buffer.length, safe=false) {}

Contributing

byte-data welcomes all contributions from anyone willing to work in good faith with other contributors and the community. No contribution is too small and all contributions are valued.

See CONTRIBUTING.md for details.

Style guide

byte-data code should follow the Google JavaScript Style Guide:
https://google.github.io/styleguide/jsguide.html

Code of conduct

This project is bound by a code of conduct: The Contributor Covenant, version 1.4, also available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html

Reporting issues

Use the GitHub issue tracker.

Reporting security issues

Report security issues to this e-mail: rocha.rafaelsilva@gmail.com.

Legal

LICENSE

Copyright (c) 2017-2019 Rafael da Silva Rocha.

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

CHANGELOG

19.0.1 (2020-01-29)

Fix: Ensure original buffer is never touched when unpacking big-endian

19.0.0 (2020-01-28)

API Changes:

• Remove clamp optional param from unpacking functions
• Add safe optional param to unpack()
• packing values other than integer will throw a TypeError if type is integer.
• packing values other than string will throw a TypeError if type is string.
• Does not clamp or check for overflows while unpacking

Other changes

• Zero dependencies

18.1.1 (2020-01-26)

• Fix type definitions in docstrings to ease integration with Closure Compiler:
  • The only required attribute in the type definition is the number of bits
• Add the default value of optional parameters to docstrings

18.1.0 (2020-01-26)

• Add optional boolean param 'clamp' to all packing functions; when set to true, overflows on integers will be clamped instead of throwing a RangeError. Default is false.

18.0.4 (2020-01-04)

• Remove unecessary polyfills
• Use only polyfills that do not pollute the global scope

18.0.3 (2020-01-02)

• Fix docstrings for better integration with Closure Compiler

18.0.2 (2020-01-02)

• Fix externs file
• Fix docstrings for better integration with Closure Compiler

18.0.1 (2020-01-02)

• Fix throwValueError_ docstring

18.0.0 (2020-01-02)

• Use RangeError and TypeError instead of just Error

17.0.0 (2019-12-31)

• Packing true or false result in a "Argument is not a valid number" error
• New package structure:
  • dist file is "./dist/byte-data.js", a UMD served as "main"
  • ES6 source is "./index.js", served as "module"

v16.0.4 (unreleased)

• Fix: TypeScript declaration included in package.json
• Update dependencies to their latest versions

v16.0.3 (2018-08-09)

• Fix: input index on unpackArrayTo error messages
• Error messages include the value that caused the error along with its index
• Faster and safer packing and unpacking
  • Refactor packArrayTo and unpackArrayTo to improve performance
  • Enforce safe comparisons on input validation

v16.0.2 (2018-08-06)

• Remove duplicate validation of integers
• Include String.codePointAt() polyfill in scripts/polyfills
• Use 'ArrayBufferView' type instead of 'any' to represent TypedArray in index.d.ts

v16.0.1 (2018-08-06)

• Faster floating-point parsing
• Fix TypeScript declaration of packArrayTo and unpackArrayTo

v16.0.0 (2018-08-05)

• packString(str) returns a Array; if another type is needed for the output, use packStringTo(str, buffer).
• unpackString(buffer, start, end) end param is now non-inclusive.
• null is not a valid value anymore; null values present in the input will cause a 'not a valid number' error.
• Error messages are more informative and include the index of the input/output that caused the error.
• type objects signature changed to use "fp" instead of "float":

  // instead of 
  let f32 = {float: true, bits: 32}; // will not work
  // you must use
  let f32 = {fp: true, bits: 32}; // will work

v15.1.0 (2018-08-03)

• "safe mode" for unpack array; optional boolean argument 'safe' that defaults to false. If true, a error will be thrown if the input array have extra bytes or not sufficient bytes according to the data type. If false, inputs with insufficient length will generate empty ouputs and extra bytes in the end of the array will be ignored.

  byteData.unpackArray([0xff], {bits: 16}, 0, 1, true); // throws 'Bad buffer length' error
  byteData.unpackArray([0xff, 0xff, 0xff], {bits: 16}, 0, 3, true); // throws 'Bad buffer length' error
  byteData.unpackArray([0xff], {bits: 16}, 0, 1); // return a empty array
  byteData.unpackArray([0xff, 0xff, 0xff], {bits: 16}, 0, 3); // return a array with one 16-bit unsigned int

• type objects signature changed to use "fp" instead of "float"; using "float" still works, but is deprecated and will not work in future releases.

  // instead of 
  let f32 = {float: true, bits: 32};
  // you should use
  let f32 = {fp: true, bits: 32};

v15.0.0 (2018-08-01)

• Fix: throws error when packing NaN as integer
• Fix: unpack binary16 Infinity, -Infinity and NaN
• Fix: pack binary16 Infinity, -Infinity and NaN
• Compatible with IE6+ and all modern browsers that support ES3/ES5/ES6+

unpackString(buffer, index=0, end=null)

• unpackString() now returns a Uint8Array.
• the parameters index and end determine a slice of the buffer to read. So to read the first 4 bytes of a buffer, you would use:

  let str = unpackString(buffer, 0, 3);
  // read from buffer[0], buffer[1], buffer[2], buffer[3]

v14.1.0 (2018-07-19)

• Add countString(string) the API; returns the number of bytes needed to serialize a UTF-8 string.

v14.0.5 (2018-07-19)

• Fix: replace invalid UTF-8 characters with U+FFFD instead of throwing errors
• Fix: packStringTo docstring (remove @throws {Error}).

v14.0.4 (2018-07-19) [DEPRECATED]

• Validation when reading UTF-8.

v14.0.3 (2018-07-18) [DEPRECATED]

• Fix: JSDoc unpack() return signature (remove 'undefined').

v14.0.2 (2018-07-17) [DEPRECATED]

• Fix: unpackArrayTo and unpackArray

v14.0.1 (2018-07-16) [DEPRECATED]

• Fix errors with strings in ES5 dists (transpile String.codePointAt())
• Throw Error if packing anything other than Number, Boolean or null with pack, packTo, packArray and packArrayTo
• Throw Error for bad buffer length on unpack (not unpackArray or unpackArrayTo; see README for details)

v14.0.0 (2018-07-15) [DEPRECATED]

• UTF-8 string support on packString, unpackString and packStringTo
• Remove unpackFrom and unpackArrayFrom from the API; unpack and unpackArray now accept the same optional params as unpackFrom and unpackArrayFrom received.

v13.2.6 (2018-07-13)

• Fix es2015 field in package.json
• Fix documentation issues.

v13.2.5 (2018-07-09)

• Faster 64-bit fp number read/write.

v13.2.4 (2018-07-09)

• Handle big-endian data more efficiently.

v13.2.3 (2018-07-08)

• UMD dist transpiled to ES5.

v13.2.2 (2018-07-08) [DEPRECATED]

• Fix: Support big-endian hosts.

v13.2.1 (2018-07-06) [DEPRECATED]

• Fix: lib name in UMD dist.

v13.2.0 (2018-07-05) [DEPRECATED]

• Allow Array and Uint8Array as output buffer.
• Fix: cases of unpacking extra elements on unpackArrayTo and unpackArrayFrom.

v13.1.3 (2018-07-05) [DEPRECATED]

• Fix types in TypeScript declaration file.
• Fix JSDoc: Typed Arrays as input for packArray and packArrayTo

v13.1.2 (2018-07-05) [DEPRECATED]

• Add validation of strings as ASCII.
• Fix documentation issues.

v13.1.1 (2018-07-04) [DEPRECATED]

• Zero dependencies.

v13.1.0 (2018-07-02) [DEPRECATED]

• Add unpackArrayTo(buffer, type, output) to output to typed arrays.

v13.0.1 (2018-06-27) [DEPRECATED]

• Using dot notation to allow better compilation on hosts.

v13.0.0 (2018-06-26) [DEPRECATED]

• No more standard types; types must be defined by the user.

v12.0.1 (2018-06-26) [DEPRECATED]

• Add TypeScript declaration file.

v12.0.0 (2018-06-26) [DEPRECATED]

• Functions from the old API can handle only numbers
• new string functions: packString, packStringTo, unpackString
• null values are packed as zero

v11.1.0 (2018-06-25) [DEPRECATED]

• Allow better use of this lib as a dependency:
  • package.json refactored with bundlers and ES6 envs in mind
  • Fix inconsistent JSDoc declarations

v11.0.2 (2018-06-24) [DEPRECATED]

• Fix ES6 dist to not rely on Node module path resolution.

v11.0.1 (2018-06-23) [DEPRECATED]

• Fix: type declarations
• Fix: remove unused exports

v11.0.0 (2018-06-22) [DEPRECATED]

• ES6 module
• New API with packTo, packToArray, unpackFrom, unpackArrayFrom

v10.0.0 (2018-06-15) [DEPRECATED]

• New dist file: ./dist/byte-data.min.js.
• Remove 'browser' from package.json

v9.0.2 (2018-06-12) [DEPRECATED]

• fix: validation of null, undefined and string length.

v9.0.1 (2018-06-12) [DEPRECATED]

• fix: dist included in npm package.

v9.0.0 (2018-06-12) [DEPRECATED]

• dist included in npm package.
• bytes only in base 10 for input and output
• types in byteData.types
• throw errors on overflow and underflow
• throw errors when packing null and undefined values
• throw errors for strings with bad length

v8.0.3 (2018-06-11) [DEPRECATED]

• fix: webpack.config so no dependency dist is used in the bundle.

v8.0.1 (2018-05-13) [DEPRECATED]

- better packaging.

v8.0.0 (2018-05-05) [DEPRECATED]

- packStruct() and unpackStruct() are deprecated.
- Validate type when packing/unpacking, throws Error if type not valid.
- Fix: unpackArray of types "char" with more than 8 bits return an array of strings, not a single string with all values.
- Fix: packArray with types "char" and items of length different than the type offset.

v7.0.1 (2018-05-04) [DEPRECATED]

- Fix: check for undefined values on pack().
- Fix: check for null values on unpack().

v7.0.0 (2018-05-03) [DEPRECATED]

- Type class is deprecated. Types should be defined as Object<string, *>.

v6.0.0 (2018-05-03) [DEPRECATED]

- findString() is deprecated.