Last built 4 months ago using go version go1.7.4 linux/amd64 and 146 seconds of time
elb-instance-status is a small daemon you can run on any instance on an autoscaling group. It periodically executes commands using
/bin/bash and checks for their exit status (0 = fine, everything else = not fine). The collected check results are exposed using an HTTP listener which then can be used by an ELB health check for that machine. This enables your autoscaling-group to react to custom health checks on your machine.
For example given you have a process eating all inodes on your machine and you have no chance to clean up these files you could use this daemon to terminate the instance as soon as the inode usage is too high. Maybe this is a bad example because file system cleanups should be possible all the time but you get the point: Something is wrong on one of your cattle-machines? Remove it.
The checks defined are executed every minute so you should take care not to do too expensive checks as they would stack up and could make your machine unstable. If you have checks taking longer than one minute you should do them using cron and only write a status file read by this daemon.
If the unhealthy threshold (default: 5 checks) is crossed the HTTP status will switch from 200 (OK) to 500 (Internal Server Error) which will cause the ELB to mark your machine unhealthy and the autoscaling-group will remove that machine. Of course you need to ensure there is a starting grace period to give your machine enough time to settle and get all checks green. And you also need to take care the new machines started as a replacement for the unhealthy ones are going to be healthy. Otherwise your whole cluster gets taken out of service.
/statuspath as the check target
# curl -is localhost:3000/status HTTP/1.1 200 OK Date: Fri, 03 Jun 2016 10:56:13 GMT Content-Length: 426 Content-Type: text/plain; charset=utf-8 [PASS] Ensure there are at least 30% free inodes on /var/lib/docker [PASS] Ensure there are at least 30% free inodes on / [PASS] Ensure docker can start a small container [PASS] Ensure volume on /var/lib/docker is mounted [PASS] Ensure there is at least 30% free disk space on /var/lib/docker
The checks are defined in a quite simple yaml file:
--- root_free_inodes: name: Ensure there are at least 30% free inodes on / command: test $(df -i | grep "/$" | xargs | cut -d ' ' -f 5 | sed "s/%//") -lt 70 lib_docker_mounted: name: Ensure volume on /var/lib/docker is mounted command: mount | grep -q /var/lib/docker docker_run: name: Ensure docker can start a small container command: docker run --rm alpine /bin/sh -c "echo testing123" | grep -q testing123
They consist of an unique ID and three keys for each check:
name(required), A descriptive name of the check (do not use the same name twice!)
command(required), The check itself. Needs to have exit code 0 if everything is fine and any other if somthing is wrong.
/bin/bash -c "<command>".
warn-only(optional, default: false), Only put a WARN-line into the output but do not set HTTP status to 500