This post is the beginning of what will become a group of tips / pointers / sketches of ideas / commands, each individually not worth its own full blog post, but hopefully as a group worth their collective bits.
Who killed my process?
If you’re trying to track what process killed another, here are two ways::
A) auditd :
* root@host:# auditctl -a exit,always -f arch=[b32|b64] -S [kill|tkill|tgkill] -k KillRule_926 (pick one or more variations)
Running a kill against a process generates 2 audit log messages that look somewhat like this :
—
root@host:# ausearch -k “KillRule_926”
SYSCALL … syscall=37 success=yes pid=3936 comm=”killall” key=”KillRule_926″ ..
OBJ_PID … opid=1968 exe=”/bin/bash” key=”KillRule_926″
Where pid is the killer process (invoked by command comm), opid is the victim. key is an arbitrary audit log “filter key” that appears in the resulting audit log entries to make it easier to extract them.
—
B) SystemTap :
Instrumenting the entire OS with SystemTap to track down a dying process is surely a judgment call, not your first reaction, but if you choose this route:
* Install SystemTap perfmon tool
* Run or tweak the sigmon.stp script (these live in /lib/modules/<kernel_version>/systemtap)
* Disable all unnecessary probes.
—
Linux Daemons
( Liberal Cloning from here. )
Commands :
/sbin/service –status-all
/usr/bin/system-config-services #GUI
Rule Name | Type | Description | Enabled Status |
---|---|---|---|
acpid | Hardware | Power | Recommended-On |
anacron | Jobs | Run missed cron jobs | Recommended-On |
apmd | Hardware | Use ACPId instead | Recommended-Off |
atd | Jobs | Job Scheduling – at | Recommended-On |
auditd | Security | Log OS events | Recommanded-On |
avahi | Network | Network Sharing | Rare |
bluetooth / hidd / pand | Hardware | Bluetooth Devices | Rare |
dhcdbd | Hardware | DBus IF to DHCP – NetMgr | Optional |
gpmd | Hardware | Mouse in text app | Rare |
hald | Hardware | Hardware Abstraction Layer | Mandatory |
hsqldb | DB | Used by Open Office, apps | Desktop |
iptables | Security | Software Firewall | Recommended-on |
irqbalance | Jobs | Distrib SMP jobs | Recommended-on |
kudzu | Hardware | Hardware Detection at boot | Recommended-on |
lisa | Network | Windows Sharing | Recommended-on |
lm_sensors | Hardware | poll sensors | Servers |
mdmonitor | Storage | mdadm util | Servers |
mdmpd | Storage | multipath healing | Servers |
messagebus | Hardware | D-Bus daemon (HW alerts) | Recommended-on |
ifplugd | Network | autoup NIC when plugged | Desktop |
NetworkManager | Network | wifi, HW NIC switching | Desktop |
nscd | Network | DNS Cache | Server |
portmap | Network | RPC Manager (NISd or NFSd) | Server |
—
Resize an LVM Partition
Now resize with [ pvresize & lvextend & ] resize2fs
root@host:# resize2fs -p /dev/mapper/mpath2
—
Linux Raid and Volume Mgmt
dmraid
What is dmraid?
BIOS-assisted, software RAID, with OS-driver. Works with well multipathing.
A better alternative to mdadm – emulates hardware RAID.
pro: Understands more formats of RAID headers than mdadm [ run ‘dmraid -l’ for a list ]
con: Does it handle partitions larger than 2TB?
pro: Some cases support rebuilding a mirror before the OS boots, which can never happen with software raid.
Note: Sample Device Name :: dm-46
Sample mdadm Commands
HowTo Discover and enable HW Raid sets: modprobe dm_mod && dmraid -ay && ls -l /dev/mapper
HowTo Disable HW Raid sets : dmraid -an
HowTo Disable MultiPath : dmsetup remove_all AND mv /lib/udev/rules.d/64-md-raid.rules /lib/udev/rules.d/DISABLED_RULES/64-md-raid.rules
HowToBypass DM-raid (failed mirror) : LiveCD boot. vi /etc/fstab [ remove references to failed RAID ]
mdadm
What is mdadm? Alternative to dmraid.
mdadm /mdraid – software raid.
con: mdadm devices do not work with multipathing
LVM
con: lvm good volume mgmt, but no raid
* dm-multipathing – handles hot-plug or boot-time multipathing
Notes:
If you setup RAID via BIOS (say for Windows) and it fails, LiveCD boot to Linux and use mdraid (software raid) to fix the problem. Do not use DMraid (HW).
root@lnxprd00:# dmsetup info
Name: mpath2
State: ACTIVE
Read Ahead: 256
Tables present: LIVE
Open count: 1
Event number: 1
Major, minor: 253, 4
Number of targets: 1
UUID: mpath-350002ac0004d135d
—
# HP-UX : create NFS mountpoint and auto-mount it
roo@HP-UX:# mknfsmnt -B -A -S -f /NFS_backup -d /tinleypark -h auspcrpbak03
permanent = -B ; automnt = -A ; soft : -S (uppercase)
# Show OS release on linux
rpm -qa redhat-release (shows) “redhat-release-5Server-5.8.0.3”
# Set Date on a Linux server
date -d @1358252055 -> “Tue Jan 15, 2013 ….” # convert epoch date to User time
# For ‘ magic keys ‘ to work, you need to add
/etc/sysctl.conf : kernel.sysrq=1
sysctl -p
Alt + SysRq/PrtScrn + S = sync
Alt + SysRq/PrtScrn + B = reBoot OR Alt + SysRq/PrtScrn + O = power Off
> IP Tables – based SysRq [ http://marek.terminus.sk/prog/ipt_sysrq.shtml ]
ipt_sysrq module : send a sysrq over the network ( from an IP-restricted host )
# If sent in the last 43200 seconds (to allow for drift if NTP isn’t working)
/etc/modules.conf : modprobe int_SYSRQ passwd=”plaintext_password” tolerance=43200
# Track Listing within CPIO archive
root@host:# cpio -it < apd50_du2_WedOct09_part1.cpio 2>/dev/null
# reset too many password attempts
root@host:# pam_tally2 –user aderidde –reset=0
Networking
# Bonding Options
Options are set in /etc/modprobe.conf
* mode 0 = balance-rr
packets are spread across all interfaces in the bonded interface
* mode 1 = active-backup
one interface (the primary) is up until it fails, when it comes back up, failback to it
all interfaces have the same MAC address. After failover/failback, the interface sends
out one ARP packet to alert the switch that it’s active
* mode 2 = balance-xor
Use a randomizer algorithm to decide which port
* mode 4 = 802.3ad
This is described by 802.3ad dynamic link aggregation = LACP
Contrast this to ‘static link aggregation = etherchannel’
This must also be set on the switch ports (negotiates at boot – the entire bonded interface will not come up if one side is down)
* mode 5 = balance-tlb
adaptive transmit load-balancing (no switch-level support needed).
all inbound traffic comes in one interface. outbound is distributed across slaves based on speed
* mode 6 = balance-alb
adaptive load balancing (transmit and receive)
The bonding driver overwrites the sending MAC HW address so that each client is assigned to talk to one of the slaves.
Sometimes the bond interface itself will send out ARPs which the bond driver has to “fix” by having the slaves
update their clients (“nope you’re still supposed to be talking with me, not the active slave”)
# Create a VLan-Tagged Interface
# vconfig add bond0 241 (add vlan241)
# brctl addbr
SnooP Cisco Discovery Protocol / CDP packets
Cisco switches can get set to broadcast info about the switch ports
Show CDP = eth switch port for unix systems
tcpdump -nn -v -i eth1 -s 1500 -c 1 ‘ether[20:2] == 0x2000’|grep -E ‘0x01|0x06|0x03|0x0a’
Show CDP = ethernet interface connections on ESX machines
vmware-vim-cmd hostsvc/net/query_networkhint|grep -E ‘vmnic|devId|portId|hardwarePlatform’|sed -e ‘s/^[ \t]*//’|tr -d ‘\n’|sed ‘s/device/\ndevice/g’; echo;
Show CDP = ethernet interface connections on Solaris
snoop -d bge0 -s 1500 -x0 -c 1 ‘ether[20:2] = 0x2000’
Note: Brocade/Foundry alternative to CDP is FDP
Note: LLDP (link layer discovery protocol) is vendor-independent. Brocade supports it. protocol for LLDP is 0x88cc
# Switch:
tcpdump -i eth0 -s 1500 -XX -c 1 'ether proto 0x88cc'
# Neighbors : tcpdump -v -s 1500 -c 1 '(ether[12:2]=0x88cc or ether[20:2]=0x2000)'
Security
OpenSSL Tricks – SSL Certificate Expiration
KyloRen:~ aderidde$ openssl s_client -connect ivzwmdm.iot.motive.com:443 | \
openssl x509 -noout -dates
notBefore=Mar 10 00:00:00 2017 GMT
notAfter=Mar 10 23:59:59 2020 GMT
Miscellaneous
#Error on console – I/O blocked for more than 120 seconds.
Fix –
Edit /etc/sysctl.conf
vm.dirty_background_ratio = 5
vm.dirty_ratio = 10
# Tell me all used IP addresses in the subnet:
nmap -sP 10.151.242.0/24
# Increase Maximum Postfix message size
postconf -e message_size_limit=60000000 && vi /etc/postfix/main.cf
> (add) message_size_limit = 60000000
[root@linuxprd00 postfix]# postconf -n | grep message_size
message_size_limit = 60000000
# xargs on Solaris10 and list zoned network configs
# -i = replace {} with values -n 1 = one value at a time
root@solaris_ftp00 # zoneadm list | grep -v global | xargs –i zlogin {} “hostname; netstat -rn | grep default; echo ‘ ‘”