k8s service: how to create ELB in a private subnet

For a k8s service in a AWS private subnet. By default it won’t create ELB. In order to get k8s doing all the grunt work of ELB creation, register k8s worker instance with the ELB, add listener, tag it property. there is a magic annotation here.

magic annotation

kubectl annotate the service

kubectl --namespace=guestbook annotate service frontend  service.beta.kubernetes.io/aws-load-balancer-internal=0.0.0.0/0

linux file security bits

• https://docs.oracle.com/cd/E19683-01/816-4883/secfile-69/index.html
• http://foralllinux.blogspot.ca/2013/04/set-setuid-setgid-and-sticky-bit-in.html
• http://thegeekdiary.com/what-is-suid-sgid-and-sticky-bit/ for various use cases.

How to expose docker app on LAN

docker-machine create –driver virtualbox webexposed

docker-machine stop webexposed

VBoxManage modifyvm webexposed –natdnshostresolver1 on

VBoxManage modifyvm webexposed –natpf1 ‘http,tcp,,8080,,80’

docker-machine start webexposed

docker-machine create –driver virtualbox webexposed

docker-compose up

browser to http://lan_ip:8080/

VBoxManage & Docker

 

Popular VBoxManage cmd

• VBoxManage list vms  (list vm’s name or uuid)
• VBoxManage list runningvms
• VBoxManage unregistervm 6621ebbe-0139-40a7-a496-37ac6bcba735 –delete
• VBoxManage controlvm <name or uuid> poweroff
• VBoxManage modifyvm “VM name” –natdnshostresolver1 on
  • http://superuser.com/questions/641933/how-to-get-virtualbox-vms-to-use-hosts-dns
  • http://www.virtualbox.org/manual/ch09.html#nat_host_resolver_proxy

Vagrant

• vagrant package –base   (this creates a new box)
• vagrant box add docker-centos package.box

gpg cheatsheet

• install pgpdump util for viewing the PGP keys.
  • http://www.mew.org/~kazu/proj/pgpdump/en/
• import PGP keys into a custom private key ring:  The public and private are both in private.pgp
  • gpg –import –no-default-keyring –keyring /tmp/public –secret-keyring /tmp/private private.pgp
• list secret keys in a non default ring:
  • gpg -K –secret-keyring /tmp/private
• list  keys in default public and secret rings:
  • gpg -K, gpg -k

A script

# import the public and private key pair prior to the encryption

gpg –import –no-default-keyring –secret-keyring ${SECRET_KEYRING} ${PRIVATE_KEY}

chmod 400 ${SECRET_KEYRING} ${PRIVATE_KEY} ${PASSPHRASE}

# decrypt
cd ${input_dir}

for pgp_file in *.gpg
do
unenc_file=${pgp_file%.gpg}
gpg –secret-keyring ${SECRET_KEYRING} –batch –output ${output_dir}/${unenc_file} –passphrase-fd 0 $pgp_file < ${PASSPHRASE}
done

sensu

Sensu uses the embedded ruby

• /etc/default/sensu
  EMBEDDED_RUBY=true
• /opt/sensu/embedded/bin/gem list
• /opt/sensu/embedded/bin/gem install sensu-plugin

mailer.rb handler set up

• /etc/sensu/handlers/mailer.rb downloaded from sensu-community-plugin.
• /etc/sensu/conf.d/mailer.json for configuration of mailer.rb
• Use this handler “mailer” in a check such as /etc/sensu/conf.d/check_apache.json
• {
  “checks”: {
  “apache_check”: {
  “handlers”: [“mailer”],
  “command”: “/etc/sensu/plugins/check-apache.rb”,
  “interval”: 60,
  “subscribers”: [ “web” ]
  }
  }
  }

high performance python tips

• http://www.jeffknupp.com/blog/2013/06/30/pythons-hardest-problem-revisited/

Quoted from http://ict.swisscom.ch/2015/02/pyperformance/

•  I/O bound problems can make good use of multi-threading (where the GIL is released during I/O) or asynchronous programming.
• CPU-bound problems can be addressed by better algorithms (nothing beats an algorithm with less computational complexity), using array-based programming (NumPy), using various problem-specific packages written in a compiled language, or using Cython, a mix of C and Python.
• Application-level caches are also helpful, because no computation is always faster than the fastest possible computation.

The GIL constraint is removed when multiple processes are used, each with its own Python interpreter and GIL. This works nicely for problems that don’t require massive interaction between data or even massive amounts of read-only data.

In my own work with Quantax, the Swisscom Market Risk System, which is written in Python, we always face demand for increased speed. Using a lot of NumPy and many levels of application caches, we achieve about 25000 valuations of financial instruments per second on one core of a laptop CPU.

However, the price for this is complexity of cache invalidations, and complicated code to map the problem to NumPy.

We use processes at a relative coarse-grained level, as worker processes to calculate reports. The main issue of processes is the massive amount of common data the financial calculations require, leading to rather large memory consumption per process. However, there is rarely more than one logical process that modifies the objects (by changing transactions or rates).

set up sshfs as a non-root user on linux

1. sudo apt-get install sshfs

   sudo usermod -a -G fuse uid: add uid to fuse group


   /etc/fuse.conf: comment out user_allow_other.

   /etc/modules: add fuse module

   sshfs uid@remotehost:/home/uid /home/uid/mnt/desktoplinux  -o allow_other

measure network speed with netcat

• destination: sudo nc -lkv 9999 > /dev/null
• src: dd if=/dev/zero bs=1024K count=512 | nc -v 10.5.5.21 9999 (copy 10meg data with block size 16k)

port forwarding

• TCP port forwarding: SSH port forwarding/reverse port forwarding, iptables.
• SSL: stunnel runs in client mode talks to stunned runs in server mode
• UDP and TCP: socat