Important: This documentation covers Yarn 1 (Classic).
For Yarn 2+ docs and migration guide, see yarnpkg.com.

Package detail

omelox-protobuf

linyngfly69deprecated4.2.10TypeScript support: included

Package no longer supported. Contact Support at https://www.npmjs.com/support for more info.

Build Status

readme

Build Status

Omelox-protobuf

Protobuf protocol is a high efficient binary protocol for data encode, this module implement the protobuf protocol, and used in omelox for data transfer. Of course, omelox-protobuf can also be used independently in other projects.

Architecture

Unlike the google protobuf, we provide a universal encoder and decoder in omelox-protobuf. We use protos file as meta data to encode/decode messages, so you do not need to add any code to your project, instead , what you need is to add a protos.json (or two for different encoder and decoder messages) files to define the message need to encode by protobuf.The architecture of omelox-protobuf is as follow:

omelox protobuf

Usage

Define protos

To use omelox-protobuf, you need to write a JSON file to define the message format. The syntax of the file is as the same as the .proto file in protobuf, but in JSON format, here is the example protos.json:

  {
    "onMove" : {
      "required uInt32 entityId" : 1,
      "message Path": {
        "required uInt32 x" : 1,
        "required uInt32 y" : 2
      },
      "repeated Path path" : 2,
      "required uInt32 speed" : 3
    },
    "onAttack" : {
      "required uInt32 attacker" : 1,
      "required uInt32 target" : 2,
      "message AttackResult" : {
        "required uInt32 result" :  1,
        "required uInt32 damage" : 2,
        "optional uInt32 exp" : 3
      },
      "required AttackResult result" : 3
    }
  }

Unlike the google protobuf, we write all the protos in the same file, with a unique key to define the message.

To use the protos, we use a parser to parse the protos file into more machine friendly format, which is also a json format, then you can use the result to decode/encode messages.

RootMessage support

you can write rootMessage in protos for global usage

{
  "message Path": {
    "required double x" : 1,
    "required double y" : 2
  },
  "message Equipment" : {
    "required uInt32 entityId" : 1,
    "required uInt32 kindId" : 2
  },
  "onMove" : {
    "required uInt32 entityId" : 1,
    "repeated Path path" : 2,
    "required float speed" : 3
  },
  "area.playerHandler.enterScene" : {
    "message Player" : {
      "message Bag" : {
        "message Item" : {
          "required uInt32 id" : 1,
          "optional string type" : 2
        },
        "repeated Item items" : 1
      },
      "required uInt32 entityId" : 1,
      "required uInt32 kindId" : 2,
      "required Bag bag" : 3,
      "repeated Equipment equipments" : 4
    },
    "optional Player curPlayer" : 2
  }
}

Server side and Client side

Omelox-protobuf has server code and client code for js.

  • The server code run in Node.JS environment, use Buffer to represent the binary data.
  • The client side code run on browser, use ByteArray to represent the binary data.

On average, the encode/decode speed of Server version is 60% faster than client version, with less memory usage. So we highly recommend that use the server code on Node.JS for better performance.

Example message

  var key = 'onMove';
  var msg = {
    entityId : 14,
    path : [{x : 128,y : 796},{x : 677,y : 895}],
    speed : 160
  };

Server side encode/decode

  //Require proto buf module
  var protobuf = require('protobuf');

  //Set encode protos and decode protos
  var protos = protobuf.parse(require('./protos.json'));
  protobuf.init({encoderProtos:protos, decoderProtos:protos});

  //Encode msg to binary Buffer
  var buffer = protobuf.encode(key, msg);

  //Decode a msg from binary buffer
  var decodeMsg = protobuf.decode(key, buffer);

At server side, the encode result will be a Buffer. The encoderProtos and decodeProtos can be different, in this case we use the same protos for encoder and decoder.

Client side encode/decode

To use the protbuf as browser, you need to include the /client/protobuf.js in your html.

  //Require proto buf
  var protobuf = require('protobuf');

  //Get parsed protos from server
  var protos = getProtos();

  //Init protobuf
  protobuf.init({encoderProtos:protos, decoderProtos:protos});

  //Encode msg to binary Buffer
  var buffer = protobuf.encode(key, msg);

  //Decode a msg from binary buffer
  var decodeMsg = protobuf.decode(key, buffer);

The protobuf will be a global variable, and you need to get the parsed protos from server. The others are the same as in server side, except the encoder result will by a ByteArray instead of Buffer.

Compatibility

For the same message and proto, the encode results are the same for omelox-protobuf and google protobuf .This means you can exchange binary data with google-protobuf.

Some how we has some changes in the proto file, and there are some features we do not support, there are the different:

  • package : The array with simple content (integer, float) are packaged by default.And the complex content(message, string) are not packaged.

  • long : Omelox protocol do not support long type, because there are no long int in javascript.All the integer bigger than 32 bits will be translate to a 64bit float, which has only has 52 bits significant figure. It will lost presion for any integer has more than 52 bits significant figures.

  • default : Omelox-protobuf do not support default keyword, for the default value is only used to initialized the element at the decoder side, which can be done by the constructor.

  • enum : Omelox-protobuf do not support the enum keyword.

changelog

1.4.x

[template] 添加了打包时对配置文件的复制处理

[omelox] DictionaryComponent 添加选项 ignoreAutoRouter

可以自主控制dict的序号id.

// app.set('dictionaryConfig',{dict,ignoreAutoRouter})
export interface DictionaryComponentOptions {
    dict?: string;
    // 不自动按照路由生成router,仅使用 config/dictionary 内的路由.
    // 这样路由的 id 就可以通过dictionary的顺序来控制了,方便proto变更不影响原顺序id (为也热更新考虑)
    // 另外这样也少一次load handler
    ignoreAutoRouter?: boolean;
}

1.4.10

[omelox] 修复win10 mqtt 超时问题

1.4.9

[omelox-protobuf]: 在开启encode缓存时,优化protobuf encode性能. (提升1倍)

 test Protobuf time: 914.453ms
 Protobuf length total: 1780000

 test ProtobufCache time: 416.399ms
 ProtobufCache length total: 1780000

[omelox]: fixed bug:指定服务配置auto-restart失效

[Example]: add nodejs ts client.

1.4.8

fix #128 https://github.com/linyngfly/omelox/issues/128 解决 mqtt-connection

1.4.7

fix #129 https://github.com/linyngfly/omelox/issues/129

1.4.6

revert 1.4.5 -> 1 revert 1.4.5 -> 3

1.4.5

  1. 更新 package.json,转移不需要的包到dev依赖. d58657d523c6ca1782dba1ec4c7d7d5cc62e5e22 https://github.com/linyngfly/omelox/issues/128

  2. [omelox]: manualReloadCrons 添加重载前清除选项. 8c1708ee74f68a9ef583827882a36cb0e59ead28

  3. tsconfig.json add opition skipLibCheck 解决新建的模板项目编译报错问题

1.4.4

修复NPM没有编译dist的问题.

1.4.3

omelox-rpc 修复mqtt调用延迟40~80ms的bug https://github.com/NetEase/pomelo-rpc/pull/33

分布式部署时,servers 配置有 args参数时,自动添加空格。

Merge pull request #125 from lowkeywx/master Modify the log description in the application.start function

Merge pull request #126 from wjt382063576/fix_dict fix(dict): Fix the problem of duplicate handler routing in the dictio…

omelox-protobuf Encoder可选性能优化,添加Encoder缓存选项。 不优化时的逻辑是,对msg进行JSON.stringify 获取长度*2,再分配Buffer。 使用优化时的逻辑是,使用指定大小的预分配Buffer。

// 指定 omelox-protobuf encode使用 buffer缓存大小
// 使用方法  在 connector配置参数
 app.set('protobufConfig', {
    // protobuf Encoder 使用 5m 的缓存 需要保证每个消息不会超过指定的缓存大小,超过了就会抛出异常
    encoderCacheSize: 5 * 1024 * 1024
 });
// 如果缓存大小不够就会有错误日志
// 缓存大小不够 日志示例
 [2020-03-27T10:44:48.752] [ERROR] omelox - [chat-server-1 channelService.js] [pushMessage] fail to dispatch msg to serverId: connector-server-1, err:RangeError [ERR_OUT_OF_RANGE]: The value of "offset" is out of range. It must be >= 0 and <= 0. Received 1
 at boundsError (internal/buffer.js:53:9)
 at writeU_Int8 (internal/buffer.js:562:5)
 at Buffer.writeUInt8 (internal/buffer.js:569:10)
 at Encoder.writeBytes (F:\develop\gong4-server\logicServer\omelox\packages\omelox-protobuf\lib\encoder.ts:195:20)

omelox-protobuf protobufConfig 配置添加一个选项 decodeCheckMsg.

解析客户端消息时校验客户端消息字段是否符合protobuf定义

因为目前发现,有的客户端encode消息没有校验消息字段是否完整,导致服务端收到消息以后字段缺失,会出现逻辑问题.

所以添加了这个选项.

examples/websocket-chat-ts-run 添加开启 选项 decodeCheckMsg 的示例. 并使用 globalBefore 进行捕获.

1.4.2

修复web-server 更新express依赖出现的configure问题。

fix #118 #119

回退 web-server express版本

fix omelox-cli lost dependency.

1.4.1

try fix #63 运行目录问题,先与pomelo的代码行为保持一致。

添加 error handler 和 globalfilter示例 examples/websocket-chat-ts-run/game-server/app.ts

修复 因为修复 #110 导致的 所有日志级别都变为INFO的问题。

1.4.0

更新所有依赖库版本,并修复编译错误。 typescript 版本 3.7.2

fix #110 omelox-logger 的logger对象换成原始的 log4js 对象。

1.3.14

fix #104