Data Event in shopee

原文链接

There are only two hard things in Computer Science: Cache validation and naming things

{
  "command": "update",
  "table": "test_db.test_tab",
  "timestamp": 1347876077,
  "fieldnum": 3,
  "newrow": {
    "0": "30020",
    "1": "username1",
    "2": "0"
  },
  "oldrow": {
    "0": "30020",
    "1": "username2",
    "2": "16"
  }
}

采用zk作为元数据的存储

mysql binlog ->gds -> GDS Kafka -> Consumer-> kafka queue -> Executor -> other db/redis

每个库一个队列

controller + zookeeper来管理，前者用来调度，后者用来存储元数据信息

利用goroutine来进行并发，同时，根据key来分发，也有一定顺序性