This post is the beginning of what will become a group of tips / pointers / sketches of ideas / commands, each individually not worth its own full blog post, but hopefully as a group worth their collective bits. 

 Who killed my process? 

If you’re trying to track what process killed another,  here are two ways::

A)  auditd :
* root@host:# auditctl -a  exit,always -f arch=[b32|b64] -S [kill|tkill|tgkill]  -k KillRule_926  (pick one or more variations)

Running a kill against a process generates 2 audit log messages that look somewhat like this :

root@host:#   ausearch -k “KillRule_926” 
               SYSCALL … syscall=37  success=yes  pid=3936 comm=”killall”  key=”KillRule_926″ ..
               OBJ_PID …  opid=1968  exe=”/bin/bash”  key=”KillRule_926″ 

Where pid is the killer process (invoked by command comm), opid is the victim.  key is an arbitrary audit log “filter key” that appears in the resulting audit log entries to make it easier to extract them.

B) SystemTap :
Instrumenting the entire OS with SystemTap to track down a dying process is surely a judgment call, not your first reaction, but if you choose this route:
* Install SystemTap  perfmon tool
* Run or tweak the   sigmon.stp   script   (these live in /lib/modules/<kernel_version>/systemtap)
* Disable all unnecessary probes.

Linux Daemons 

( Liberal Cloning from here. )

Commands :
/sbin/service   –status-all
/usr/bin/system-config-services           #GUI

Rule Name Type Description Enabled Status
acpid Hardware Power Recommended-On
anacron Jobs Run missed cron jobs Recommended-On
apmd Hardware Use ACPId instead Recommended-Off
atd Jobs Job Scheduling – at Recommended-On
auditd Security Log OS events Recommanded-On
avahi Network Network Sharing Rare
bluetooth / hidd / pand Hardware Bluetooth Devices Rare
dhcdbd Hardware DBus IF to DHCP – NetMgr Optional
gpmd Hardware Mouse in text app Rare
hald Hardware Hardware Abstraction Layer Mandatory
hsqldb DB Used by Open Office, apps Desktop
iptables Security Software Firewall Recommended-on
irqbalance Jobs Distrib SMP jobs Recommended-on
kudzu Hardware Hardware Detection at boot Recommended-on
lisa Network Windows Sharing Recommended-on
lm_sensors Hardware poll sensors Servers
mdmonitor Storage mdadm util Servers
mdmpd Storage multipath healing Servers
messagebus Hardware D-Bus daemon (HW alerts) Recommended-on
ifplugd Network autoup NIC when plugged Desktop
NetworkManager Network wifi, HW NIC switching Desktop
nscd Network DNS Cache Server
portmap Network RPC Manager (NISd or NFSd) Server

Resize an LVM Partition

Now resize with [ pvresize & lvextend & ] resize2fs
root@host:# resize2fs -p /dev/mapper/mpath2

Linux Raid and Volume Mgmt


What is dmraid?
BIOS-assisted, software RAID, with OS-driver. Works with well multipathing.
A better alternative to mdadm – emulates hardware RAID.
pro: Understands more formats of RAID headers than mdadm [ run ‘dmraid -l’ for a list ]
con: Does it handle partitions larger than 2TB?
pro:  Some cases support rebuilding a mirror before the OS boots, which can never happen with software raid.
Note: Sample Device Name :: dm-46

Sample mdadm Commands
HowTo Discover and enable HW Raid sets:   modprobe dm_mod && dmraid -ay && ls -l /dev/mapper
HowTo Disable HW Raid sets :   dmraid -an
HowTo Disable MultiPath :        dmsetup remove_all AND mv /lib/udev/rules.d/64-md-raid.rules /lib/udev/rules.d/DISABLED_RULES/64-md-raid.rules
HowToBypass DM-raid (failed mirror) :  LiveCD boot.  vi /etc/fstab [ remove references to failed RAID ]


What is mdadm? Alternative to dmraid.
mdadm /mdraid – software raid.
con: mdadm devices do not work with multipathing
con: lvm  good volume mgmt, but no raid

* dm-multipathing – handles hot-plug or boot-time multipathing
If you setup RAID via BIOS (say for Windows) and it fails, LiveCD boot to Linux and use mdraid (software raid) to fix the problem. Do not use DMraid (HW).

root@lnxprd00:# dmsetup info
Name:                   mpath2
State:                    ACTIVE
Read Ahead:           256
Tables present:       LIVE
Open count:           1
Event number:       1
Major, minor:         253, 4
Number of targets:  1
UUID:                    mpath-350002ac0004d135d

# HP-UX : create NFS mountpoint and auto-mount it
roo@HP-UX:# mknfsmnt -B -A -S -f /NFS_backup -d /tinleypark -h auspcrpbak03
permanent = -B ;  automnt = -A   ;     soft : -S (uppercase)

# Show OS release on linux
rpm -qa redhat-release   (shows)   “redhat-release-5Server-”

# Set Date on a Linux server
date -d @1358252055 -> “Tue Jan 15, 2013 ….” # convert epoch date to User time

# For ‘ magic keys ‘ to work, you need to add
/etc/sysctl.conf : kernel.sysrq=1
sysctl -p
Alt + SysRq/PrtScrn + S = sync
Alt + SysRq/PrtScrn + B = reBoot   OR   Alt + SysRq/PrtScrn + O = power Off

> IP Tables – based SysRq [ ]
ipt_sysrq module : send a sysrq over the network ( from an IP-restricted host )
# If sent in the last 43200 seconds (to allow for drift if NTP isn’t working)
/etc/modules.conf : modprobe int_SYSRQ passwd=”plaintext_password” tolerance=43200

# Track Listing within CPIO archive
root@host:# cpio -it < apd50_du2_WedOct09_part1.cpio 2>/dev/null

# reset too many password attempts
root@host:# pam_tally2 –user aderidde –reset=0


# Bonding Options

Options are set in /etc/modprobe.conf

* mode 0 = balance-rr
packets are spread across all interfaces in the bonded interface

* mode 1 = active-backup
one interface (the primary) is up until it fails, when it comes back up, failback to it
all interfaces have the same MAC address. After failover/failback, the interface sends
out one ARP packet to alert the switch that it’s active

* mode 2 = balance-xor
Use a randomizer algorithm to decide which port

* mode 4 = 802.3ad
This is described by 802.3ad dynamic link aggregation = LACP
Contrast this to ‘static link aggregation = etherchannel’
This must also be set on the switch ports (negotiates at boot – the entire bonded interface will not come up if one side is down)

* mode 5 = balance-tlb
adaptive transmit load-balancing (no switch-level support needed).
all inbound traffic comes in one interface. outbound is distributed across slaves based on speed

* mode 6 = balance-alb
adaptive load balancing (transmit and receive)
The bonding driver overwrites the sending MAC HW address so that each client is assigned to talk to one of the slaves.
Sometimes the bond interface itself will send out ARPs which the bond driver has to “fix” by having the slaves
update their clients (“nope you’re still supposed to be talking with me, not the active slave”)


Create VLan-Tagged Interface 

# vconfig add bond0 241 (add vlan241)
# brctl addbr

SnOOP Cisco Discovery Protocol / CDP packets

Cisco switches can get set to broadcast info about the switch ports
Show CDP = eth switch port for unix systems
tcpdump -nn -v -i eth1 -s 1500 -c 1 ‘ether[20:2] == 0x2000’|grep -E ‘0x01|0x06|0x03|0x0a’

Show CDP = ethernet interface connections on ESX machines
vmware-vim-cmd hostsvc/net/query_networkhint|grep -E ‘vmnic|devId|portId|hardwarePlatform’|sed -e ‘s/^[ \t]*//’|tr -d ‘\n’|sed ‘s/device/\ndevice/g’; echo;

Show CDP = ethernet interface connections on Solaris
snoop -d bge0 -s 1500 -x0 -c 1 ‘ether[20:2] = 0x2000’

Note:  Brocade/Foundry alternative to  CDP is FDP
Note:  LLDP (link layer discovery protocol) is vendor-independent. Brocade supports it.  protocol for LLDP is 0x88cc
# Switch:   tcpdump -i eth0 -s 1500 -XX -c 1 'ether proto 0x88cc'
# Neighbors : 
 tcpdump -v -s 1500 -c 1 '(ether[12:2]=0x88cc or ether[20:2]=0x2000)'


#Errors on console – I/O blocked for more than 120 seconds.
edit /etc/sysctl.conf
vm.dirty_background_ratio = 5
vm.dirty_ratio = 10


# Tell me all used IP addresses in the subnet:
nmap -sP

# Increase Maximum Postfix message size
postconf -e message_size_limit=60000000 && vi /etc/postfix/
> (add) message_size_limit = 60000000
[root@linuxprd00 postfix]# postconf -n | grep message_size
message_size_limit = 60000000

# xargs on Solaris10  and  list zoned network configs
# -i = replace {} with values -n 1 = one value at a time
root@solaris_ftp00 # zoneadm list | grep -v global | xargs –i zlogin {} “hostname; netstat -rn | grep default; echo ‘ ‘”