Process Management

Switch	Description
-H	show process heirarchy (forest)
c	true command name
h	no headers
m	all threads
n	Numeric output for WCHAN and USER
--html	HTML escaped output
--lines	set screen height
--sort	specify the sorting order

These keys specify the data that is too be output for each process that is displayed. There are two forms: the traditional and the standard. Since all of the traditional are subsumed by the standard forms, only those are shown. Altogether, there are over 100 of these options, so only the most important are shown in the table.

• Standard keys and the -o option
• ps -o sort_key_list   -   output the keys specified after the -o option.
• ps -O sort_order_description -o sort_key_list   -   select a list of keys and a sorting sequence

  The major keys are:

  Key	Header	Description
  %cpu	%CPU	CPU percentage
  %mem	%MEM	Memory percentage
  args	COMMAND	Command arguments
  cmd	CMD	The command executed
  etime	ELAPSED	Elapsed wall clock time
  gid	GID	Group ID
  group	GROUP	Group name
  nice	NI	Nice level
  pagein	PAGEIN	Number of page faults
  pid	PID	process ID
  ppid	PPID	Parent process ID
  pri	PRI	Process priority
  rss	RSS	Resident set size (real memory)
  size	SIZE	Size of process memory
  start	START	Start time
  stat	STAT	Process status
  stime	STIME	Process start time
  time	TIME	CPU time used
  tty	TTY	TTY associated with the process
  uid	UID	User id of owner
  user	USER	User name
  vsize	VSZ	Virtual address size
  wchan	WCHAN	Name of kernel process where sleeping

  Examples

  ps ax -o "pid %cpu rss etime"
  
  all processes, show the process id, percentage of memory, real memory used and elapsed clock time.
  ps x -o "pid cmd user group time"
  
  Processes without controlling tty's, show pid, command, user name, group name and CPU time used.
  ps ax -o "%cpu user pid" --sort %cpu,user
  
  All processes, show %cpu, user name and pid, and sort by descending %cpu and secondarily, by increasing user name.

Controlling Processes

Limiting a Process

Processes can be limited on a global basis with kernel parameters, but for user processes, the ability to limit is (unfortunate pun) limited. Bash and its bretheran support the ulimit command and the csh shells support limit. By placing these in the system-wide startup files, or in the skel files you can gain some control, but there is nothing to stop a user from overriding your settings.

The ulimit command has the following format:

ulimit [-SHacdflmnpstuv] [limit]

where (all sizes are in 1024 byte blocks except as noted):

-a		Report all current limits
-c		Maximum core file size
-d		Maximum data segment size
-f		Maximum file size for shell
-l		Maximum locked memory size
-m		Maximum resident set size
-n		Maximum open file descriptors
-p		Maximum pipe size (512 byte blocks)
-s		Maximum stack size
-t		Maximum cpu time in seconds
-u		Maximum number of processes
-v		Maximum amount of virtual memory

For example:

ulimit -t 100 # 100 seconds of CPU time maximum

ulimit -c 1024 # 1 MB core file size max
ulimit -m 2044 # 2 MB real memory

Killing a Process

• kill options [pid]
• Signals
  • 31 different signals representing messages that a process can receive
  • kill -l
  • SIGHUP (1) - hangup
  • SIGQUIT (3) - quit
  • SIGKILL (9) - terminate
  • SIGTERM (15) - software termination
  • SIGTSTP (20) - stop from keyboard (e.g.control-z)
• kill -KILL 5612   -   kill process 5612
• kill -HUP 415   -   send a SIGHUP signal to process 415
• ps -C lpd
  kill -9

• killall [options] -signal command_name
  • If command name includes a "/", it refers to a specific file.
  • options
    • -e - require exact match on long (> 15 chars) names
    • -g - kill the process group to which the process belongs
    • -i - ask for confirmation
    • -w - wait for all to die

    Examples

  • killall -KILL netscape
  • killall -i -TERM /usr/local/java/bin/appletviewer
  • killall -e -KILL /usr/local/java/bin/appletviewer
  • killall -gi -TERM httpd

Setting Process Priority

• Priorities vary from -20 to 20, with -20 being the highest
• nice +n command   -   reduce priority to n (any user)
• nice -n command   -   increase priority to n (root only)
  • nice +5 search     (run search with priority n+5)
  • nice +19 bigsearch     (run search with priority n+19)
  • nice -10 quake     (run quake with priority n-10)

• renice [-+]n [[-p]pid] [[-g]pgrp] [[-u] user]
  • renice +5 1456     (reduce priority of pid 1456 by 5)
  • renice +19 -u starkey     (reduce all user starkey processes priorities by 19)
  • renice +10 -g students     (reduce all student group processes priorities by 10)
  • renice +10 1450 1451 -u schmooz     (reduce priority for all processes for user schmooz and processes with pid's 1450 and 1451.
  • renice -5 -u cpuhog memhog     (increase priorities for all processes for users cpuhog and memhog.

Monitoring Processes

top

top shows an ongoing list of the most cpu-intensive processes on the system. The list content can be adjusted and can be sorted in a variety of ways. In interactive mode, commands can be entered to modify the display. Example Syntax: top [options] Options d delay     change the delay between screen updates. p pid     give a list of up to 20 pids to be monitored q     no delay between updates S     run in secure mode i     ignore zombie or idle processes b     batch mode - no interactive b     batch mode - no interactive Interactive Options space     Update display ^L     Redraw the screen k     Kill a process* i     See -i above n     Change number of processes displayed* r     Renice a process* s     Change delay f     Change fields displayed o     Change order of fields displayed l     Toggle display of load average info m     Toggle display of memory info t     Toggle display of process info N     Sort by pid A     Sort by age P     Sort by CPU usage M     Sort by resident memory usage T     Sort by time W     Write config to ~/.toprc * indicates user prompted for value

vmstat

reports information about process resource usage, including memory, paging, block I/O traps and cpu activity. Example Syntax: vmstat [options] [delay [count]] Options -n     display header only delay     delay in seconds between screen updates count     desired number of updates Field Descriptions Procs r     number of processes waiting to run b     number of processes in uninterruptible sleep w     number of processes swapped out but runnable Memory swpd     Amount of virtual memory used in kB free     Amount of idle memory in kB buff     Amount of memory used as buffers in kB cache     Amount of memory used as cache Swap si     Amount of memory swapped in in kB/s si     Amount of memory swapped out in kB/s IO bi     Blocks sent to a block device in blocks/s bi     Blocks received from a block device in blocks/s System in     Interrupts per second cs     Context switches per second CPU us     User time sy     System time id     Idle time

w

displays information about users on the system and their current activity. Example Syntax: w [options] [user] Options -h     Don't print the header -s     Short format user     Display information only about the given user. Field Descriptions IDLE     Idle time in minutes JCPU     Time used by this process including all attached processes PCPU     Time used just for the command shown in the WHAT field

uptime

uptime shows the first line of the output from w which is the system wide performance information. Example Syntax: uptime

fuser

fuser shows the processes that currently are using the given files or sockets. Example Syntax: fuser [options] names Options -a     Show all files rather than only those being accessed -k     Kill processes accessing the file. -m     name specifies a file system and all accesses on that device should be reported. -n     Change the name space to (file, udp, tcp) -u     Append user names to each PID. Examples fuser -km /hom     Kill all files accessing /home in any way. fuser -s nbsp   Kill processes accessing the file.

Other

• pstree   -   prints a tree of processes
• free   -   information similar to vmstat about memory usage only.

Logging

syslogd

• System event logger
• /etc/syslog.conf
• Example
• Format: selector (Tab) action (Red Hat allows spaces)
• selector = facility.priority-level
• facility names

  authpriv	Security/Authorization
  auth	Security/Authorization
  cron	Clock daemon
  daemon	Other system daemons
  kern	Kernel messages
  local0-7	Local use
  lpr	Printer subsystem
  mail	Mail subsystem
  news	News subsystem
  syslog	Internal to syslog
  user	Generic user
  uucp	UUCP subsystem

• facility.* - all priorities
• priorities (from highest to lowest priority)

  emerg	System is unusable
  alert	Action needed immediately
  crit	Critical conditions
  err	Error conditions
  warning	Warning conditions
  notice	Normal, but signficant
  info	Informational
  debug	Debug message

• *.priorities = all facilities at this priority level
• semicolons separate different selectors
• commas sepearate facilities with the same priority level
• fac.lev;fac.priority-level - multiple pairs
• All rules are applied, so you can have multiple destinations for the same log event.

  fac1.lev1	facility fac1 >= lev1
  fac1,fac2.lev1	fac1 and fac2 >= lev1
  fac1.lev1,fac2.lev2	fac1 >= lev1, fac2 >= lev2
  *.lev1	all >= lev1
  *.lev1;fac2.none	all >= lev1 except fac2

  mail.info	mail at info level and higher
  mail.=info	mail at info level only
  mail.info;mail.!err	Only info, notice and warning
  mail.debug;mail.!warning	All priorities except warning

• actions
  • filename - write a message to the filename
  • @hostname - forward the message to syslogd at hostname
  • @ip_address- forward the message to syslogd at ip_address
  • user1,user2,user3,... - send the message to users logged on
  • * - send the message to all users logged on

• Examples

  *.emerg		* (tell everyone about emergencies)
  *.warning;daemon,auth.=info,user.none /var/log/messages		All warning and above, all daemon messages, auth at level info and no user messages should be logged to the file.

klogd

In order to log data from your iptables, you first need to specify the appropriate logging rule for iptables. The following log non-local traffic to kern.debug.

-A INPUT -p tcp -s ! 153.90.192.0/21 -j LOG --log-level debug

-A INPUT -p udp -s ! 153.90.192.0/21 -j LOG --log-level debug

You could get logging of the debug level messages with these rules in syslog.conf. The trick is to get the iptables log messages sent to iptables and to not have them sent to the console, which is the default.

kern.=debug		/var/log/iptables
*.info;mail.none;authpriv.none;cron.none;kern.!debug		/var/log/messages	<>/var/log/iptables

klogd is typically started along with syslogd as the syslog service.

Scheduling

cron jobs

• cron is a daemon that will run jobs at specified times or intervals.
• at - submit jobs for one time execution
• batch - submit jobs for cron if the load levels are low enough. Typically, 0.8 (see w or top).
• atq - list the jobs in a queue
• atrm - remove jobs from the cron queue
• at [-q queue] [-f file] [-mldv] TIME

  Example

  • Reads the job from stdin (not widely used)
  • -q queue - a = normal cron jobs, b = batch queue
  • -f file - the file containing the job, rather than stdin
  • -m - send mail to the user when it executes (default)
  • -l - same as atq
  • -d - same as atrm
  • -v - shows completed but not yet completed jobs
  • TIME - time to run in one of many forms
    NOW, TODAY, TOMORROW
    AM, PM, NOON, MIDNIGHT, TEATIME
    MINUTE, HOUR, WEEK, DAY, MONTH, YEAR
    SUN, MON, TUE, WED, THU, FRI, SAT
    JAN, FEB, MAR, ... DEC

  • Examples

    11am + 2 days - 11 am 2 days from now
    9 pm Aug 10 - 9 pm on August 10th
    3:30 + 1 MONTH - 3:30, one month hence
    TEATIME TOMORROW - 4:00 tomorrow
    at NOW + 1 MINUTE - in one minute
    at 6:00 am 23.10.00 - 6:00 AM on October 23, 2000
    at NOON MON - Noon, the next time a Monday rolls around
    

crontab

• cron reads your crontab file in /var/spool/cron and executes the commands based on the time interval data found there. By default, it mails the output of the commands executed to you.
• crontab [-l | -r | -e]
• -l = display current crontab file
• -r = remove current crontab file
• -e = edit current crontab file

• crontab file
• file is added to your crontab file.

• Format
  minute hour day month day-of-week shell_command

• Day-of-week is 0-6 for Sunday through Saturday Examples
  

      Find all files changed in the last day by running
        a find command at midnight every day.
     
     00 00 * * * find /bin -ctime -1
      Checksum a directory every ten days or so at
        6:00 AM
     06 00 1,10,20 * * ls /usr/bin | cksum > /root/security/cksum_log
  
     Write the happy hour announcement every
        Friday at 4:00 PM
     0 16 * * 5 /usr/sbin/wall /var/tmp/happy_hour.txt
  
     Submit the files in secure.job to crontab
     crontab -f secure.job
     

swatch

swatch is a tool that will examine new entries as they are added to a log file and watch for specific strings. When something occurs, swatch dumps it to stdout, rings the terminal bell and sends mail to a designated account. swatch requires a configuration file (the default is ~/.swatchrc).

swatch effectively performs a "tail -f" command and watches for particular strings to occur, or it can be used to examine a file for existing strings.

• swatch has options that control a variety of behaviors, but typically the configuration file is sufficient.
  • -c config-file

  • -r restart-time
    +hh:mm     -     restart after the specified time hh:mm     -     restart at the specified time
  • -t filename - file to tail
  • -f filename - file to examine
  • -p program_name - program output to examine
  • -P pattern-separator     -     What pattern to use in parsing the patterns configuration file. The default is a comma.
  • -A action-separator     -     What pattern to use in parsing the actions in the configuration file. The default is a comma.
  • -I input-record-separator     -     What pattern to use as a record boundary in the input file. The default is a carriage return.

  • swatch configuration file - ~/.swatchrc
    Example

    Each line has the form:
    
    Pattern-list tab tab> [Time tabStart-length]
    • Pattern-list is one or more regular expressions specifying a string or strings to search for.

      /login/	login
      /file system full/	"file system full"
      /(login|logout|syslog)/	login, logout or syslog
      /^Oct 12/	start of line is "Oct 12"
      /log*err/	all string log ... err

    • action-list is one or more actions to be taken. The possibilities are:

      echo[=mode]	Echo the matched line; mode = text mode
      bell[=N]	Ring the bell N times
      exec=cmd	Execute the following command. The command can contain variables where $N is the Nth field on the line.
      ignore	Do nothing
      mail=address:address:...	Mail to addresses
      pipe=cmd	Pipe the output to command
      write=user:user:...td>Pipe the output to command

    • The time value provides an interval that can be checked if a time stamp is present. For example, if you get a disk full message, it might reoccur very rapidly. You can specify an interval before it should be reported again. The line must have a time stamp in it, and the fourth parameter specifies the location of the time stamp.
      • Interval times can be SS, MM:SS, HH:MM:SS.
      • 30 - 30 seconds
      • 10:00 - 10 minutes
      • 20:5:10 - 20 hours, 5 minutes and 10 seconds

      • Time stamp location is in the form start:end.
      • 0:20 - the first 20 characters on the line

      Examples

    ---

    Assignment

    Due:
    
    1. Create a few processes by running Netscape or vi or some other processes that will hang around for a while. Try listing them different ways; by pid, by command name.
    2. Build an alias that you could use to list processes from oldest start time to newest. Print out the information you would need to kill processes that have been around too long, but remember that you don't want to kill off root processes.
    3. Think about some common problems you might have with processes. For example, a couple of processes are hogging the CPU, or available memory has gotten dangerously low, or you get a backlog of old processes because users don't properly log off. Come up with three issues, and create a ps command that will help you find the processes.
    4. Design a script to find all processes that have more than 12 hours of idle time and kill them.
    5. Design a script to watch free memory and notify root via a mail message if it gets below 10% of total memory.
    6. Set up a cron job to implement a regular test of some part of your file system. Lets be specific. The script should test the files in /bin, /sbin and /usr/sbin to see if they have been modified in the last 24 hours. If so, send a message to root using the write command.
    7. Use at or batch to execute a command.
    8. Install swatch from the Red Hat CD if it isn't already (or check ftp.stanford.edu/general/security-tools/swatch). Set up /etc/swatchrc to watch for failed logins, su's and sudo's in in /var/log/messages.

    ---