Data Event in shopee

原文链接

There are only two hard things in Computer Science: Cache validation and naming things

{
    "command":"update",
    "table":"test_db.test_tab",
    "timestamp":1347876077,
    "fieldnum":3,
    "newrow":{
        "0":"30020",
        "1":"username1",
        "2":"0"
    },
    "oldrow":{
        "0":"30020",
        "1":"username2",
        "2":"16"
    }
}

采用zk作为元数据的存储

mysql binlog ->gds -> GDS Kafka -> Consumer-> kafka queue -> Executor -> other db/redis

每个库一个队列

controller + zookeeper来管理，前者用来调度，后者用来存储元数据信息

利用goroutine来进行并发，同时，根据key来分发，也有一定顺序性