Lecture #4

Administrivia

• Example using shell pipeline: top-ten wordcount
  • combination of filter programs

Our Story So Far

• Process: (essentially) a running program

  • more precise definition: container for running program
    • might be bad choice of terminology because we use "container" for lightweight virtual machines (e.g. Docker)

• In Posix-like systems, a new process is created by fork(2) system call

  • clones current running program

• New program can replace current program in existing process using exec* (sloppy: one system call, several library wrappers) callls

  • if exec is successful, it does not return (because the process is running a new program)
  • exec could fail for several reasons; most common is the replacement program file does not exist or is not executable

• Separation of program initiation & process creation was a design choice

  • pros
    • original program can do some housekeeping after fork before exec (e.g. reopen fd 0, 1, 2)
    • can use fork for divide & conquer
  • cons
    • more compilicated to do the most common use pattern

• Open files stay open across fork, exec

  • unless special system call to set close-on-exec flag (per-file request)
  • especially file descriptors 0, 1, 2 but applies to all open files in process

• Program termination: parent process collects child exit status using wait(2) or waitpid(2)

  • wait(2) is misnamed (shrug)
  • after child has terminated, before parent collects status, child process remains in kernel process table in state Z (zombie)
    • this is a kind of resource leak
      • zombies can fill the process table and degrade system performance
    • no problem if zombie status is short-lived (this is normalish)
      • parent is working but will collect it soon
      • parent is ephemeral
  • if parent dies first, init (process #1) adopts orphans (init reaps orphan zombies)

• Pipes: producer-consumer model

  • 2 endpoints, one-way communication
  • pipe(2) system call
    • pass in array of 2 int to hold file descriptors
      • 0: reader (consumer)
      • 1: writer (producer)
  • file descriptors are opened by a process, but fds stay open across fork/exec
    • two processes can communicate via pipe only if the pipe was created by their common ancestor process
    • typically, producer closes read fd; consumer closes write fd
  • multiple processes can share a pipe (e.g. multiple producers, single consumer)
    • but it's a bit tricky, likely to be brittle (error-prone) in production

• popen(3) (C standard library function) only slightly related to pipe(2)

  • forks a process that runs in a shell (passes string argument to subshell)
  • returns FILE* object that wraps the pipe file descriptor
  • essentially, like system(3) but allows parent process to collect output

• File & pipe descriptors use same read/write system calls

  • predates, but works like C++ virtual functions

• Everything is a file: devices, pseudo devices, network connections

  • can write program so that it doesn't care what it's reading from/writing to

• System calls to determine nature of open file (regular/device/etc.)

  • device-specific operations on open file descriptor: ioctl(2) (generalized interface)
  • most programs should never do this: much more flexible (better system design) if your program doesn't care

• Shell does string processing to determine which program(s) to run, argv (argument vector), file descriptors 0, 1, 2

  • also collects exit status when the program terminates

• Shell syntax: redirect input/outpu

  • < (input)
  • > (output)
  • >> (append output)
  • 2> (standard error)
  • | (pipe output of first to input of second--restricted use of pipe(2) capabilities)

• Pipelines: data-flow style

  • programs act as filters

• Assigment 1: convex hull

Signals

• software analog of hardware interrupts

• signal is small integer with system-defined semantics

• C header file defines symbolic names: man 7 signal

  • SIGUSR1 10: user-defined
  • SIGUSR2 12: user-defined
  • SIGHUP 1: hangup
  • SIGINT 2: keyboard interrupt (control-C)
  • SIGSEGV 11: segmentation violation
  • SIGALRM 14: timer
  • SIGCHLD 17: child process stopped/terminated

• default action depends on specific signal

  • terminate program
  • terminate & dump core
  • ignore
  • continue (SIGCONT)
  • stop (SIGSTOP/SIGTSTP)

• process control block: signal dispatch table

• signal(2): system call to specific action on signal

  • prefer sigaction(2)
    • system gets more complex over time because it has to support legacy applications
    • backwards compatability

• action:

  • SIG_DFL: default behavior (for this signal)
  • SIG_IGN: ignore signal
  • function pointer

• SIGKILL (9): terminate program

  • cannot be caught/trapped

• signal may be due to event (e.g. program executes illegal instruction, child process terminates) or explicitly sent (by another process)

  • can be used for inter-process communication

• kill(2): send signal

  • kill(2) sends signal, does not kill program
    • process's default response (SIG_DFL) is to self-terminate
  • grossly misnamed (YAMSC: yet another misnamed system call)
    • the guys who did the original design were figuring this stuff out for the first time
    • chalk the name up to "seemed like a good idea at the time"

• kill(1): thin wrapper program around kill(2)

• to dispatch signal, kernel manipulates the process's kernel data structure(s) while the process is not running

  • push current value of program counter onto stack
  • set program counter to address of first instruction of signal handler routine
  • schedule the process for it's next CPU timeslice
  • when the function is finished, it returns to the next instruction it was supposed to run before the signal occurred

Interprocess Communication

• cooperating processes

  • can use machine resources more efficiently
  • can simplify programming solutions

• already discussed some ways for processses to commmunicate:

  • shared file: klunky
  • pipe
    • useful for producer-consumer (one-way) dataflow
    • more efficient than file:
      • uses memory buffer
      • processes may proceed concurrently
    • must be created by common ancestor process
      • pass around open file descriptors
      • one of the processes may be pipe creator
  • program exit status
    • slightly more useful than Boolean value
    • shell uses exit status in loops & conditionals
    • only works between child & parent
      • child must finish first
      • parent must explicitly call wait(2) (or variant)
  • signals: "wake up and check something"

Processes

• executable file (on disk): passive data

  • can create using any program that can read/write files, including text editor
  • binary file format
    • text editor is uninteresting
    • data structure (e.g. Executable and Linking Format)
      • #include <elf.h>
      • man 5 elf
        • section 5: file formats
  • "sections"
    • text (code)
    • initialized data
    • bss (uninitialized data): initialized to NUL-value bytes (NUL: ascii 0-value byte)
      • doesn't take up space on disk: it's all zeroes, just say how big the block is
        • run-length encoding (data compression: like fax)
    • relocation symbol table
    • debug symbols

• running process

  • registers (esp. program counter, stack pointer)
  • text (code)
  • static data (initialized/uninitialized)
    • uninitialized data: misnomer--initialized to 0
  • stack ("call stack")
  • heap (arena): space for dynamic memory allocation
  • open files (file descriptors)
  • argv/envp
  • exit status (on process termination)

• kernel view of process: data structure (process control block)

  • register save space (when process is not running)
  • memory map (which blocks of memory are allocated to the process)
  • table of open files
    • indexed by file descriptors
  • other info
    • pid: process id
    • ppid: parent process id
    • uid: user id
    • guid: group id
    • priority (for scheduling)
      • nice value: only root (superuser/admin) may hae negative niceness
    • signal vector

• process status

  • new
  • waiting/not ready/suspended
  • ready/runnable
  • running
  • exited/zombie

• scheduler selects ready process and runs it until

  • process makes a system call
  • hardware interrupt occurs
  • timer expires (timeslice)

• scheduling refinements

  • short-term (e.g. fast system request) vs. long-term waiting (e.g. slow I/O device)
  • priority bands

Interprocess Communication (cont.)

• 2 basic paradigms for communication

  • message passing (aka "shared nothing")
    • e.g. pipes
  • shared memory (stay tuned)

• named pipes

  • create "special" entry in filesystem
  • open by name like any other files
    • may be opened read-only or write-only mode
    • you probably don't want to open in read-write mode
  • works just like pipe, except processes do not require common ancestor
  • more general than regular pipes, but slightly more awkward for most common use case (pipelines)

• Unix-domain sockets

  • like pipes, but uses networking interface (API)
  • works only on same machine (not "real" networking, but looks like networking from the program's POV)

• networking

  • message passing between processes on different machines
    • can also use between processes on same machine (network "loopback" interface)
  • separate topic (rich)
  • "cluster OS": treat entire datacenter as a system that requires resource management/abstraction