DevilKing's blog

原文链接

golang部分的锁，主要在sync包里

runtime/sema(信号量)

runtime中的sema.go，信号量的同步功能是作用于goroutine的

1
2
3
4
5
6
7
8
9
10

type semaRoot struct {
    lock  mutex
    head  *sudog
    tail  *sudog
    nwait uint32 // Number of waiters. Read w/o the lock.
}
// 主要结构就是一个sudog的链表和一个nwait。
// 链表用于存储等待的goroutine，nwait表示在该信号量上等待的goroutine数目。
// lock一个互斥量，是在多线程环境中保护链表的。（但是这个mutex不是sync中的mutex，是sema.go
// 内部使用的一个私有版本）

golang设置了可操作信号量个数的最大量是251，相关操作，还是看链表的操作

sync/atomic

这里原子操作，是保证多个cpu（协程）对同一块内存区域的操作是原子的

1
2
3
4
5
6
7
8
9
10

// 互斥量加锁操作
func (m *Mutex) Lock() {
	// Fast path: grab unlocked mutex.
	if atomic.CompareAndSwapInt32(&m.state, 0, mutexLocked) {
		if race.Enabled {
			race.Acquire(unsafe.Pointer(m))
		}
		return
	}
......

sync/mutex（互斥锁）

sync/mutex.go

1
2
3
4
5
6
7
8
9
10
11
12
13

// A Mutex must not be copied after first use.
type Mutex struct {
    state int32    // 一个int32整数，用其中的位来表示
    sema  uint32
}

//iota，特殊常量，可以认为是一个可以被编译器修改的常量。
//在每一个const关键字出现时，被重置为0，然后再下一个const出现之前，每出现一次iota，其所代表的数字会自动增加1。
const (
    mutexLocked = 1 << iota //值为1，表示在state中由低向高第1位，意义：锁是否可用,0可用，1不可用
    mutexWoken // 值为2，表示在state中由低向高第2位，意义：mutex是否被唤醒
    mutexWaiterShift = iota //值为2，表示state中统计阻塞在此mutex上goroutine的数目需要位移的偏移量
)

lock流程

简单来说，如果当前goroutine可以加锁，那么调用原子操作使得mutex中的flag设置成已占用达到互斥；如果当前goroutine发现锁已被占用，那么会有条件的循环尝试获取锁，这里是不用信号量去对goroutine进行sleep和wake操作的（尽可能避免开销），如果循环尝试失败，则最后调用原子操作争抢一次，获取不到则还是得调用runtime_Semacquire去判断阻塞goroutine还是继续争用锁。

sync/RWMutex（读写锁）

1
2
3
4
5
6
7

type RWMutex struct {
	w           Mutex  // held if there are pending writers
	writerSem   uint32 // semaphore for writers to wait for completing readers
	readerSem   uint32 // semaphore for readers to wait for completing writers
	readerCount int32  // number of pending readers
	readerWait  int32  // number of departing readers
}

写锁定

1
2
3
4
5
6
7
8
9
10

func (rw *RWMutex) Lock() {
	// First, resolve competition with other writers.
	rw.w.Lock()
	// Announce to readers there is a pending writer.
	r := atomic.AddInt32(&rw.readerCount, -rwmutexMaxReaders) + rwmutexMaxReaders
	// Wait for active readers.
	if r != 0 && atomic.AddInt32(&rw.readerWait, r) != 0 {
		runtime_Semacquire(&rw.writerSem)
	}
}

原文链接

日志是完全顺序，只接受append操作的数据结构。

I think there are at least three key priorities for the effectiveness of one of these types of systems: performance, high availability, and scalability. If it’s not fast enough, the data becomes decreasingly useful. If it’s not highly available, it means we can’t reliably get our data in or out. And if it’s not scalable, it won’t be able to meet the needs of many enterprises.

原文链接

针对$in操作：

• index early

• index often

• index fully

  queries make use of indexes from left to right.

• index sorts

  If your queries will contain a sort or orderby clause, add the sorted field to your index.

• commands

  ​

The order of fields in an index should be:

1. First, fields on which you will query for exact values.
2. Second, fields on which you will sort.
3. Finally, fields on which you will query for a range of values.

先对sort field部分，进行index，这样能及时保证sort之后的顺序能够得到保证

原文链接

Building Spark

JRE path

Maven installed

• always explicitly set the following in ‘.bashrc’ for ‘root’
• specify support you want explicitly
• Rebuild only modified source code

Running Spark

• always use ‘—verbose’ option on ‘spark-submit’ command to run your workland
• print
  • all default properties
  • command line options
  • settings from spark’ conf file
  • setting from cli
• order of lookup, ‘—package’
  • the local maven repo
  • maven central - web
  • additional remote repositories
• OOM
  • increase ‘—driver-memory’
  • saprk sql and spark streaming need large driver heap size
• GC overhead limit exceeded
  • too much time spent in garbage collection
  • increase executor heapsize
  • modify gc policy
    • -XX:UseG1GC && -XX:UseParallelGC
    • spark default: -XX:UseParallelGC
    • Try overwrite with -XX:G1GC
• has a single SparkContext with multiple sessions supporting
  • concurrency
  • re-useable connections
  • shared cache
• Not all CPUs are busy
  • sart with evenly divided memory and cores. -executor-memroy 2500m --num-executors 200
  • when heap size non-negotiable. --executor-memory 6g --num-executors 80 transfer to --executor-memory 6g --num-executors 80 -executor-cores 2 (Forcing 80% utilization, boosting 33% performance!)
• ‘scratch’ space
  • ‘/tmp’ is full
  • fix spark.local.dir
• max result size exceeded
  • saprk.driver.maxResultSize
• out of space on a few data nodes
  • hdfs balancer to start balancing
  • dfs.datanode.balance.bandwidthPerSec increase to 6GB/s
  • dfs.datanode.balance.max.concurrent.moves set to 5000 concurrent threads

Profiling Spark

• Yarn containers across multiple nodes
• get a full thread dump or get a full heap dump. jstack -l jmap -dump:live,format=b,file=xxx
• step1: find the hostname in the error log; step 2: find the local directory where ‘stderr’ resides, step 3: open the ‘stderr’,

原文链接

microservices allow you to achieve graceful service degradation as components can be set up to fail separately

we need to keep in mind that provider services can be temporarily unavailable by broken releases, configurations, and other changes as they are controlled by someone else and components move independently from each other.

服务降解

In a microservices architecture, services depend on each other. This is why you should minimize failures and limit their negative effect. To deal with issues from changes, you can implement change management strategies and automatic rollouts.

健康检查和负载平衡

To avoid issues, your load balancer should skip unhealthy instances from the routing as they cannot serve your customers’ or sub-systems’ need.

Self-healing

In most of the cases, it is implemented by an external system that watches the instances health and restarts them when they are in a broken state for a longer period.

Implementing an advanced self-healing solution which is prepared for a delicate situation - like a lost database connection - can be tricky. In this case, you need to add extra logic to your application to handle edge cases and let the external system know that the instance is not needed to restart immediately.

failover caching

Failover caches usually use two different expiration dates; a shorter that tells how long you can use the cache in a normal situation, and a longer one that says how long can you use the cached data during failure.

retry logic

In distributed system, a microservices system retry can trigger multiple other requests or retries and start a cascading effect

Using a unique idempotency-key for each of your transactions can help to handle retries.

Rate Limiters and Load Shedders

A fleet usage load shedder can ensure that there are always enough resources available to serve critical transactions

Fail Fast and Independently

bulkhead pattern

We can say that achieving the fail fast paradigm in microservices by using timeouts is an anti-pattern and you should avoid it.Instead of timeouts, you can apply the circuit-breaker pattern that depends on the success / fail statistics of operations.

Bulkheads

By applying the bulkheads pattern, we can protect limited resources from being exhausted.

Circuit Breaks

A circuit breaker opens when a particular type of error occurs multiple times in a short period

Testing for Failures

ChaosMonkey for test

Key Takeways

• Dynamic environments and distributed systems - like microservices - lead to a higher chance of failures.
• Services should fail separately, achieve graceful degradation to improve user experience.
• 70% of the outages are caused by changes, reverting code is not a bad thing.
• Fail fast and independently. Teams have no control over their service dependencies.
• Architectural patterns and techniques like caching, bulkheads, circuit breakers and rate-limiters help to build reliable microservices.