bash: homemade timeout replacement

So I was young and reckless, and didn't know there is a command called timeout in coreutils. This is how I managed to do it:

#!/bin/sh
{
    ./time_consuming_binary -a param -a notherparam --pleaseblock
} &
CHILDPID=$!
# Kill it after 30 sec
sleep 30
kill -9 $CHILDPID 2>&1 /dev/null

How to check if it had to be killed or not? Measure the wall time of the execution :) I've used it in a Nagios service check, the thing it watched either returned under 5 sec or blocked indefinitely (thanks, NFS), hence the 30 secs.

Shell: Get threaddump directly from the java process

Inspecting the Java source, I found a pretty easy way to skip java in the process of extracting info from another java process :-)

PID=`pgrep java`
SCKT=/tmp/.java_pid$PID
SGNL=/tmp/.attach_pid$PID
CMD='1\0threaddump\0\0\0\0'

if [ ! -r $SCKT ]; then
 touch $SGNL || exit 2
 kill -s SIGQUIT $PID
 sleep 5
 rm $SGNL
 if [ ! -r $SCKT ]; then
  echo Cannot read $SCKT ... either you are not the correct user for this, or the java process does not 'see' our attach request.
  exit 1
 fi
 echo Done
fi

echo -ne "$CMD" | nc -U "$SCKT"

Possible options and variations I know about:

• 1\0threaddump\0-l\0\0\0 small L for the jstack -L option
• 1\0inspectheap\0\0\0\0
• 1\0inspectheap\0-live\0\0\0

For others see the attachListener.cpp (JDK7, JDK8)

kmem russian roulette

sudo dd if=/dev/urandom of=/dev/kmem bs=1 count=1 seek=$RANDOM

I already forgot where and when I read about this first... The theory is that every weekend you shoot this on one of your servers, that should eventually fail. Now, if your HA/failover solution does not compensate inside your SLA, you can fix it outside business time.

About this game on bashorg :-)

Sending emails to addresses and subjects in a CSV with a fixed message in a textfile

The following script was generated to notice approvers that an audit initiated the deletion of some users they did not approved (again). The Subject contains the usernames, and the message is a template with some PC blahblah.

#!/bin/bash

if [ $# -lt 2 ]; then
 echo "No arguments passed! I need a 'mailaddr;subject' .csv and a txt containing the fixed message."
 echo "csvmailer.sh addresses.csv message.txt"
 exit 1
fi

while read p; do
 IFS=';'
 TOKENS=($p) 
 EMAIL=${TOKENS[0]}
 SUBJ=${TOKENS[1]}
 IFS=' '
 mail -s "$SUBJ" "$EMAIL" < $2
done < $1

The CSV is like "jane.doe@company.com;This is a message in the subject about deleting the account of 'johndoe@company.com'\n". Do NOT put a semicolon into the subject column :-D

IFS sets the Internal Field Separator to semicolon (';'), instead of the default whitespace. This way the array generator/tokenizer is dead simple in the next line.

$1 is the CSV file, read into $p line-by-line.

$2 is the fixed message piped into mail.

Linux: Send files in e-mail from console

So I wanted to send some files, but my mailx package did not have support for the famous -a parameter.

#!/bin/bash

function create_attachment_block()
{
        echo -ne "--$BOUNDARY\r\nContent-Transfer-Encoding: base64\r\n"
        echo -ne "Content-Type: $(file -bi "$1"); name=\"$1\"\r\n"
        echo -ne "Content-Disposition: attachment; filename=\"$1\"\r\n\r\n$(base64 -w 0 "$1")\r\n\r\n"
}

if [ $# -lt 2 ]; then
        echo No files specified...
        exit 1;
fi

BOUNDARY="==combine-autogun==_$(date +%Y%m%d%H%M%S)_$$_=="
BODY=""

for a in "$@"
do
        if [ -s "$a" -a -f "$a" -a -r "$a" ]; then
                BODY="$BODY""`create_attachment_block "$a"`"
        fi
done

/usr/sbin/sendmail -oi -t << COMPLEX_MAIL
To: $1
Subject: Please see files attached
MIME-Version: 1.0
User-Agent: $0
Content-Type: multipart/mixed; boundary="$BOUNDARY"

$BODY
--${BOUNDARY}--
COMPLEX_MAIL

Logging IO activity of a process

Okay, your 3rd party app sucks. It sucks big time. Generates heavy disk traffic at seemingly random times, and you just can't think of anything, anymore. The users are revolting, the website is lagging, your boss is raging.

It is time to check what the hell the app is actually doing.

The following script is using strace to catch all IO-related syscalls done by the given process, and dump them in a CSV manner. Later, you can aggregate by seconds, minutes, file systems, or subsystems (like Lucene, etc), create charts, graphs, and pivots.

#!/bin/sh

if [ "x$1" == "x-h" ]; then
 echo "Usage: ./iotrace.sh <pid>"
 exit 0
fi

if [ "$(id -u)" != "0" ]; then
   echo "This script must be run as root " 1>&2
   exit 1
fi

if [ $# -gt 0 ]; then
PID=$1
else
echo "Hey, I need a parameter!"
exit 1
fi

ps -eL | grep $PID | awk '{print"-p " $2}' | xargs strace -q -f -v -ttt -T -s 0 -e trace=open,close,read,write 2>&1 | awk -v pid=$PID '
function output(a, f, r, t)
{
 # a - action
 # f - file descriptor
 # r - result
 # t - time as unix epoch
 if (f in fd)
  file = fd[f];
 else
 {
  ("readlink /proc/" pid "/fd/" f) | getline file;
  fd[f] = file;
 }
 if (file !~ /^(socket|pipe|\/dev|\/proc)/ || r ~ /\d+/)
  print a, file, r, strftime("%Y-%m-%d %H:%M:%S"); #substr(t, 0, index(t, ".")-1));
}

BEGIN { OFS=";"; print "op;path;bytes;epoch";}
{
 if($6 ~ /resumed>/)
 {
  if ($5 ~ /open/){fd[$(NF-1)] = pending[$2];}
  else if ($5 ~ /close/){match($4, /([0-9]+)/, a);delete fd[a[1]];}
  else if ($5 ~ /write/){match($4, /([0-9]+)/, a);output("write", pending[$2], $(NF-1), $3);}
  else if ($5 ~ /read/) {match($4, /([0-9]+)/, a);output("read", pending[$2], $(NF-1), $3);}
  
  delete pending[$2];
 }
 else if ($4 ~ /^open\(/)
 {
  match($4, /\"(.+)\"/, a);
  f = a[1];
  if ($(NF-1) == "<unfinished")
  {
   pending[$2] = f;
  } else {
   fd[$(NF-1)] = f;
  }
 }
 else if ($4 ~ /^close\(/)
 {
  match($4, /([0-9]+)/, a);
  f = a[1];
  if ($(NF-1) == "<unfinished")
  {
   pending[$2] = f;
  } else {
   delete fd[f];
  }
 }
 else if ($4 ~ /^write\(/)
 {
  match($4, /([0-9]+)/, a);
  f = a[1];
  if ($(NF-1) == "<unfinished")
  {
   pending[$2] = f;
  } else {
   output("write", f, $(NF-1), $3);
  }
 }
 else if ($4 ~ /^read\(/)
 {
  match($4, /([0-9]+)/, a);
  f = a[1];
  if ($(NF-1) == "<unfinished")
  {
   pending[$2] = f;
  } else {
   output("read", f, $(NF-1), $3);
  }
 }
}'

What it does?

1. Takes your input of a process ID
2. Reads all the child processes of this process
3. Feeds these into xargs to make strace to attach to all of them, also make strace to only print the four syscalls we are interested in (open, close, read, write), these are used by normal java IO methods
4. Make a dictionary of file descriptors and filenames, and pretty-print the filenames with the acutal number of processed bytes
5. Also, take care of the interrupted syscall printouts.

An average java webapp with Lucene can produce 3.5M rows in an hour. Note, that it cannot be opened in Excel ;-)