Data Pipeline
Stream Processing as a Service (SPaaS)，allowing them to focus on business application logic while platform provides the scale, operations, and domain expertise.
Fast prototyping of stream processing solutions，提供规范
Provide tooling to enable users to self-serve
failure as first class citizen
some important aspects that impacted our design:
- Assume unreliable network
- Trust underlying runtime infrastructure, but design automatic healing capabilities
- Enforce job level isolation for multi-tenants support
- Reduce blast radius when failure arise
- Design for automatic reconciliation if any components drifts from desired state or even if disaster failure occurs
- Handle & propagate back pressure correctly
Source of Truth Store is a durable, persistent storage that keeps all the desired state information.
关于状态的存储