DevilKing's blog

Iceberg Intro

Posted on 2020-11-30 In bigdata

Partitioning is an optimization technique used to divide a table into certain parts based on some attributes.

This includes schema evolution, partition evolution, and table version rollback — all possible without the need for costly table rewrites or table migration.

hidden partitioning

Only data that is written to the table after evolution is partitioned with the new spec, and the metadata for this new set of data is kept separately.

Data partitioned on multiple columns creates multiple layers of folders, with each top-level folder containing one folder for each of the second-level partition values.

spec说明

项目	资源	投入时间段	投入百分比(及周末加班情况)
中银项目	桂权力	12.5 - 12.17	25% (12.6 周日加班半天，12.11-12.12 周末加班1天，12.15 场内出差)

aws lambda power tuning

Posted on 2020-11-23 In aws

repo链接

主旨是找到那些花费少，延迟小的相关的一些配置

原理上，比较简单，设定相应的内存大小，去跑相应的程序，计算

It watches mounted volume dirs and notifies the target process that the config map has been changed. It currently only supports sending an HTTP request, but in future it is expected to support sending OS (e.g. SIGHUP) once Kubernetes supports pod PID namespaces.

主要是利用fsnotify的方式

比较好的基于重试的写法

begun := time.Now()
req, err := http.NewRequest(*webhookMethod, h.String(), nil)
if err != nil {
  setFailureMetrics(h.String(), "client_request_create")
  log.Println("error:", err)
  continue
}
userInfo := h.User
if userInfo != nil {
  if password, passwordSet := userInfo.Password(); passwordSet {
    req.SetBasicAuth(userInfo.Username(), password)
  }
}

successfulReloadWebhook := false

for retries := *webhookRetries; retries != 0; retries-- {
  log.Printf("performing webhook request (%d/%d)", retries, *webhookRetries)
  resp, err := http.DefaultClient.Do(req)
  if err != nil {
    setFailureMetrics(h.String(), "client_request_do")
    log.Println("error:", err)
    time.Sleep(time.Second * 10)
    continue
  }
  resp.Body.Close()
  requestsByStatusCode.WithLabelValues(h.String(), strconv.Itoa(resp.StatusCode)).Inc()
  if resp.StatusCode != *webhookStatusCode {
    setFailureMetrics(h.String(), "client_response")
    log.Println("error:", "Received response code", resp.StatusCode, ", expected", *webhookStatusCode)
    time.Sleep(time.Second * 10)
    continue
  }

  setSuccessMetrics(h.String(), begun)
  log.Println("successfully triggered reload")
  successfulReloadWebhook = true
  break
}

if !successfulReloadWebhook {
  setFailureMetrics(h.String(), "retries_exhausted")
  log.Println("error:", "Webhook reload retries exhausted")
}