Eng Cafe

Feb 21, 2022

Ep11. Designing Data-Intensive Applications - Partitioning

这一期我们讨论Designing Data-Intensive Applications书中partitioning这一章的学习笔记。 🔴 这一期偏重技术话题，我们会用很多英文表述技术性专有名词。之前有朋友反馈过中英夹杂对大家收听不方便，希望在意的朋友见谅。如果有不准确或者过时的地方欢迎指正。 # Show Notes * 📕 Designing Data-Intensive Applications * What is partitioning? * A partition is a division of a logical database or its constituent elements into distinct independent parts. * Main reason: scalability - the query load can be distributed across many processors. * Youtube / Vitess scaling story * Single MySQL → Add read replica → Write can’t catchup up → Partition * How to partition? * Partitioning by Key Range (e.g., Bigtable) * Assign a continuous range of keys to each partition * Pro: range scan is easier, data locality * Cons: certain access patterns can lead to hot spots (timestamp) * Cons: finding split points and managing rebalancing is hard * Partitioning by Hash * Good hash function: uniformly distribute keys * Con: no easy range queries * Cassandra does KKV (partitioning key, sort key, value) * Hot spots: 3% of Twitter's Servers Dedicated to Justin Bieber * Secondary indexes: Local index * Efficient write, expensive read * ElasticSearch * Secondary indexes: Global index * Efficient read, expensive write * Using Global Secondary Indexes in DynamoDB (这里说错了，DynamoDB 支持 20 global secondary indexes per table） * Rebalancing partitions * Move loads to other nodes * Fixed number of partitions * New node steals partitions from every existing node * Notion: 480 partitions * Dynamic partitioning * 📈: split partition into 2 * 📉: merge 2 partitions into 1 * Fixed number of partitions per node * https://www.datastax.com/blog/new-token-allocation-algorithm-cassandra-30 * Operations: full automatic (dangerous) / semi-automatic / full manual (tedious) * Request Routing * 3 approaches: nodes talk to each other, separate routing tier, smart client * Separate coordination service such as ZooKeeper * Notes by xg # 联系方式 * 官网: eng.cafe * 微信公众号: Eng Cafe * Twitter: @engcafefm * Youtube: Eng Cafe * 小宇宙播客 * 泛用型播客客户端: eng.cafe/subscribe * Email: hi@eng.cafe

34 min