levselector.com New York
home > Unix Email

Unix
On this page: Other pages:
intro, manpages, tutorials
books
prompt & terminal
unix editos
more
unix flavors
Linux, Solaris, BSD
SCO, AIX, HP,
Digital, IRIX
commands examples
shell scripting
tcsh
Bourne sh_short
Bourne sh_tutorial
vi_mode_commands.html
emacs_mode_commands
command_line_editing_tips
vi-ref - vi-ref.pdf
less
awk sed grep Tcl/Tk
Learning the UNIX Operating System
unix hints and hacks
my page for linux
more commands (random flow)
network_admin less
make building
 
intro, tutorials home - top of the page - email

 Unix - runs behind ~70% of the Internet, very stable, originally written in C.  It is (was) simple, small and open.  This allowed it to mutated like a virus and to spread onto various platforms.  Why Unix is better or how to hate Unix - read more on Operating Systems page.

Some links:
- www.iki.fi/era/unix/ - History, Useless Use Of ... Awards, etc
- www.iki.fi/era/unix/shell.html - Steve Bourne's original tutorial.
- www.ocean.odu/edu/ug/shell_help.html - quickly written by the looks of it, but seems to have the important stuff.
- www.mpi-sb.mpg.de/~uwe/lehre/unixffb/quoting-guide.html  - a pretty definitive document on quoting and escape characters.

Examples of tutorials:
www.google.com/search?q=unix+tutor+commands
www.emba.uvm.edu/CF/basic.html - - basic unix commands with short descriptions

man pages:
The easiest way to find many man pages for a command is by searching google for:    manpage command
www.engin.umich.edu/htbin/mangate?manpage=perl - perl
www.engin.umich.edu/htbin/mangate?manpage=ls - ls

www.engin.umich.edu/htbin/mangate?manpage=troff - troff
www.engin.umich.edu/htbin/mangate?manpage=groff - groff
www.engin.umich.edu/htbin/mangate?manpage=sed - sed
www.engin.umich.edu/htbin/mangate?manpage=awk - awk
  etc.

from the prompt:
  man time
  man 3 ctime              - ctime(3)
  man -s 3 ctime         - ctime(3)

Reformatting documentaion on unix prompt
- convert manpages to plain text files: 
       man command
       man command | col -b > out.txt
- convert Perl pod documentation:
      pod2man name.pm | nroff -man | less

Good starting tutorials:
www.ora.de/catalog/unixcd/chapter - Unix Commands (from O'Reilly "Unix in a Nutshell, Chapter 2)
www.webreference.com/programming/unix/ - tutorials
www.infojack.com/ - tutorials, books and tests
www.itspace.com/ - unix resources
www.oase-shareware.org/shell/ - all about Unix shells
kulichki.rambler.ru/moshkow/ - many books in Russian

 
unix books home - top of the page - email

 
Unix : Visual Quickstart Guide (Visual Quickstart Guide Series) by Deborah S. Ray, Eric J. Ray
UNIX For Dummies - by John R. Levine, Margaret Levine Young
Learning the UNIX Operating System - (Nutshell Handbook, O'Reilly) - short
Programming With Gnu Software (Nutshell Handbook) 
Unix in a Nutshell: A Desktop Quick Reference for SVR4 and Solaris 7 (3rd Edition) - by Arnold Robbins, Daniel Gilly
UNIX Power Tools  Programming With Gnu Software (Nutshell Handbook) 
UNIX System Administration Handbook - by Evi Nemeth, et al (Paperback - August 2000) 
Essential System Administration : Help for Unix System Administrators (Nutshell Handbook) 
The UNIX Programming Environment - by Brian W. Kernighan, Rob Pike (Contributor), Robert Pike

Advanced Programming in the Unix Environment (Addison-Wesley Professional Computing Series)  W. Richard Stevens 
Unix Network Programming : Networking Apis: Sockets and Xti (Volume 1) by W. Richard Stevens 
Unix Network Programming : Interprocess Communications (Volume 2) - by W. Richard Stevens
Design of the Unix Operating System - by M. Bach
The Magic Garden Explained : The Internals of Unix System V Release 4 : An Open Systems Design - by Berny Goodheart, James Cox 
The Unix Philosophy - by Mike Gancarz 

Interprocess Communications in Unix : The Nooks and Crannies - by John Shapley Gray 
Porting Unix Software : From Download to Debug (Nutshell Handbook) 
Practical Unix and Internet Security 
Unix Backup and Recovery 
Unix Internals: The New Frontiers - by Uresh Vahalia
Unix Shells by Example 
Linux Network Toolkit 
Linux System Administration Black Book 
Unix-haters handbook - by Simson Garfinkel et al, ISBN 1-56884-203-1, IDG Books, $16.59, out of print.  This book is real fun to read, but difficult to find.  Here are some related links: - http://catalog.com/hopkins/unix-haters/handbook.html - more - more - Table of Contents - unixux prompt - rus - Russian translation of parts of it, chapter7, ms_doc.file

See more books listed on pages devoted to shells, misc. tools, sys. administration, network, email, and Linux.

 
editing on the prompt home - top of the page - email

When you start working on the prompt - you want to select the shell to use (usually ksh or bash) and editing mode (vi or emacs).
Ask your sysadmin to set it up for you (edit /etc/passwd file and .profile , .bash_profile , .bashrc in your home directory).

If your terminal is in some strange state - try doing "stty sane". 
Here are some commands in .profile file for Korn shell:
EDITOR=vi
VISUAL=vi   # this overrides EDITOR
set -o vi   # this is the same again
PS1='$LOGNAME'@`uname -n`:'$PWD $ '

Command line editing is usually done in one of 2 modes: vi or emacs.
Here are commands and tips:
vi_mode_commands.html
emacs_mode_commands
command_line_editing_tips

vi-mode is not intuitive for many people. Arrows don't work as you would expect them to work. You have to constantly switch between command and insert modes. Here are some sites to help you love vi:
vi-lovers
vim.org
vi-ref - vi-ref.pdf
www.ungerhu.com/jxh/vi.html - very good one-page reference

If you are not a vi lover - then you probably will be much happier with emacs mode:
   set -o emacs
It allows you to navigate horizontally on the line and vertically (history list) using arrows.

For modern linux-es default is bash in emacs mode.
www.emacs.org/ ,
www.gnu.org/software/emacs/windows/ntemacs.html

Using xterm - examples:
dtterm -d 10.3.3.101:0.0 -fg black -bg white & 
xterm -d daddypc:0.0 -fg green -bg black & 
xemacs -d daddypc:0.0 & 
alias term='xterm -d daddypc:0.0 -fn -*-Fixed-Medium-R-*-*-*-120-75-75-*-*-ISO8859-1 -bg DarkCyan -fg White -geom 90x30 -cr red &' 
alias term0='xterm -d daddypc:0.0 -fg black -bg white &' 
alias term1='xterm -d daddypc:0.0 -fn -*-Fixed-Medium-R-*-*-*-120-75-75-*-*-ISO8859-1 -bg black -fg LemonChiffon -geom 90x30 -cr red' 
alias term2='xterm -d daddypc:0.0 -fn -*-Fixed-Medium-R-*-*-*-140-75-75-*-*-ISO8859-1 -bg black -fg LemonChiffon -geom 100x40 -cr red' 

You can get syntax color coding in vim via CRT-telnet (www.vandyke.com):
  - set CRT-telnet session to ansi emulation (you can change ANSI colors in Global settings).
  - set $TERM to ansi on unix
  - in vim settings file  .vimrc  add command    :syntax on

www.cs.utk.edu/~shuford/terminal/unix_terminal_news.txt - discussion about using terminals
www.unm.edu/cirt/introductions/unix/unix10.html - using X-windows
www.chemie.fu-berlin.de/chemnet/general/aboutx.html - FAQ about X-windows
www2.ncsu.edu/cc/pub/tutorials/unity_intro/31_colors.html - setting colors in X-windows

Unix Editors

vi - exists on all unix boxes. So knowing it is a must for any unix programmer. vi chokes on log files with very long lines - so you better use vim.

vim (vi improved) - http://www.vim.org - great editor. Can read very long files with very long lines without problems (standard vi is known for choking on long lines). Vim exists on many platforms, including unix and windows. It has support for ctags (wa to jump to function definition and back). On linux when you type "vi" - you are actually getting vim.

emacs - another great highly-customizable editor. Editor of choice of many C++ old-timers.

pico - simple text editor (was part of Pine email client).

nano - improved version of pico - http://www.nano-editor.org

joe - http://joe-editor.sourceforge.net/ - great editor (this page also lists many other editors).

more on unix

POSIX - (Portable Operating System Interface) is a set of standard operating system interfaces.  It was based on the UNIX operating system partly because it was "manufacturer-neutral." Read on www.whatis.com

More on Unix:
USENET FAQs
Unix Programming FAQ
Unix Links
The Unofficial Guide to Solaris
OO Network Prgming with Unix
Unix systems programming
sockets
Vi FAQ
Advanced Unix IPC
unix sockets programming in c

See also:
www.gnu.org - GNU software and compiler
www.cerfnet.com/~mpcline/c++-faq-lite/ - C++ FAQ
www.eskimo.com/~scs/C-faq/top.html - C FAQ

Unix Flavors

 Linux
see my page for linux
www.linux.org  - Linux
www.linux.org/dist/english.html - misc distributions (Red Hat, Caldedra, Debian GNU/Linux, Slackware, S.u.S.E. and many-many others).
www.linux.org/vendors/retailers.html - retailers in different countires (long list)
www.redhat.com - Linux
www.suse.com - Linux

  Sun Solaris
www.sun.com - Solaris operating system (available for Intel-PC for $15).
     ( also www.sunfreeware.com - everything you need for operating system )
     info.rutgers.edu/Techdir/solarisbody.html - Solaris system configuration
     www.fwi.uva.nl/pub/solaris/solaris2.html  - Solaris FAQ
     http://docsun.cso.uiuc.edu/cgi-bin/nph-dweb/ab2 - Solaris Online Documentation Library - excellent resource
     http://soldc.sun.com/ - Solaris Developer's Connection

  BSD (Berkeley Systems Distribution)
www.bsd.org - lists several different BSDs (FreeBSD, NetBSD, OpenBSD, BSDI Internet Server (BSD/OS) )
         for example, FreeBSD is used by famous sites like Yahoo and HotMail ( http://freebsd.peon.net/ - tutorials )
www.bsdi.com - commercial expensive BSD

  SCO (The Santa Cruz Operation, Inc.)
www.sco.com - good stable unix on Intel platform:
     www.sco.com/unix/market.html
        1987:  AT&T + Sun = System V, revision 4 (SVR4)
        1993: AT&T sells its unix to Novell
        1995: sold to SCO.

  AIX (An IBM unix)
www.ibm.com/servers/aix/ - unix from IBM

  HP-UX (Hewlett Packard unix)
www.hp.com - unix from Hewlett Packard - good choice (amazon.com uses HP servers)

  DIGITAL UNIX
www.compaq.com/enterprise/  ( read also www.operatingsystems.net/oses/decunix.htm )
   ( ULTRIX - a BSD based + some features from System V and DEC.  It was made by DEC, then replaced by Digital Unix ).

  IRIX
www.sgi.com/ - Silicon Graphics, Inc.

www.microware.com/Products/Software/OS9.html - OS-9 - small, but very nice and robast UNIX-like RTOS

and many-many more types of Unix

 
commands examples home - top of the page - email

You can combine several commands on one line using pipes and redirecting input/output. Here are some examples:
sort < data > junk
tr a-z A-Z < in > out
ls -s | tail +2 | sort -rn | head -5
ps -ef | grep http
tar cvf - . | gzip -9 > mmm.tar.gz
gunzip < mmm.tar.gz | tar -xvf -
ps -ef | grep daemon | cut -c10-14
ls -l | grep "^d" | cut -c55-80
cat tt | sort | uniq -d
 
ps -ef | grep appl
/usr/ucb/ps aux | head
find . -type f -print | xargs fgrep -l 'pattern goes here' /dev/null/
find . -mtime -1 -print
find . -name  '*.pl'  -exec perl -wc {} \;

uname -a
du, du -ks
df, df -k
dmesg | grep mem
cat /var/db/dmesg.boot | more               (on Solaris 7:   cat /var/adm/messages.1 )
vmstat -w 3 (or simply vmstat 3) 
ps -auxrh (BSD) ps -ef  (Solaris)
top 200 | grep 'stronghold' | wc -l
ps -U username | grep 'stronghold' | wc -l

tail -f somelog  - to see the tail of the log as it changes
grep -i  something     - ignore case grep
grep -i -l something - ignore case, print only the names of the files

nslookup `hostname`
netstat
ifconfig
/etc/hosts
/etc/resolv.conf
/var/spool/cron/crontabs/ ...

tar everything in current directory:
   tar   cvf    mmm.tar  .
   gzip -9 mmm.tar
or in 1 line:
   tar cvf - . | gzip -9 > mmm.tar.gz
now in oposite direction:
gunzip mmm.tar.gz
tar xvf mmm.tar
or in 1 line:
gunzip < mmm.tar.gz | tar -xvf -
gzip  -dc  mmm.tar.gz | tar -xvf -
or to just look what's inside:
gzip  -dc  mmm.tar.gz | tar -tvf -
------------------------
gunzip -c mmm.tar.gz | tar xvf - file1 file2 file3
gzip -dc mmm.tar.gz | tar tvf -
gzip -dc mmm.tar.gz | tar -xof - 
zcat < mmm.tar.gz | tar -tf -

to uncompress .bz2 files use:
bzip2, bunzip2 - a block-sorting file compressor, v1.0.2
bzcat - decompresses files to stdout
bzip2recover - recovers data from damaged bzip2 files

bunzip2 somefile.bz2

Recursive operations - use -r ( or -R) option:
cp -r from to
\rm -rf dirname      (backslash is used to un-alias the command - usually it is aliased to ask confirmations for every file)
Do recursive copy preserving all permissions - use tar:
cd fromdir; tar cf - .| (cd todir; tar xfBp -)

one-liners for Unix: 
  rename many files simultaneously (here it renames *.tar into *.tar7): 
      ls -1 *.tar | perl -nle '$old=$_;s/(\.+)tar/$1tar7/;rename($old, $_);' - 
  print help for all commands of perforce (p4) source control system: 
      p4 help commands | perl -nle 'm/^\s+(\w+)\s/; print "$1\n";' - | xargs p4 help 
if you received an  tar.gz file uuencoded in the body of the message between the begin..end lines (begin line contains real file name, for example SelfGOL.tar.gz):
  save it as text file  "file.txt"
  uudecode -p < file.txt  | gunzip | tar -xzvf -

or (step-by-step):
    uudecode file.txt
    ls -goct | sed 2q
    gunzip SelfGOL.tar.gz
    tar xvf SelfGOL.tar

Get gcc (GNU C-compiler) binary - version 2.8.0 or better. 
    /appl/archives/gcc.tar.Z 
    copy it into /appl/Archives directory, uncompress it and untar it (tar xvf ...) 
    This will put all necessary binary files in their places. 

 Test that the compiler is alive: 

    gcc -v      - should show the version 

 When installing the software we will be running scripts which will be looking for a C-compiler.  Some scripts want to see
 C-compiler as /usr/local/bin/cc (or as CC).  To help them, you may execute something like this: 

     ln -s 'whence gcc' /usr/local/bin/cc 

 This may be OK if there are no other C-compilers used on this machine. 
 Otherwise you may need to indicate the compiler explicitly in the command line, for example: 

     make CC=gcc 
     make CC=gcc install 

 Or you can define environment variables before running make or configure script: 

     CC=gcc;export CC (sh) 
     CC=gcc           (ksh) 
     setenv CC gcc    (csh & tcsh) 

 Now you can test: 

     env 
     echo $CC 

 For some scripts it works better if you 
 define variables in one line with the script command: 

     CC=gcc; export CC; ../dist/configure   (sh)
     setenv CC gcc; ../dist/configure        (csh & tcsh) 
     CC=gcc ../dist/configure                (ksh)

To sync directory from old computer to a new computer:

  • log into your new machine and cd to the directory you want for files
  • rsync -avP oldusername@your.old.machine:/your/old/dir .
  • (you may have to set '-e ssh' if they don't have an rsync server running).

sudo - utility to run a command with privileges of another user (typically as root).
( example:  sudo hdparm -t /dev/hdb5  - to show disk speed ).
The permissions are stored in the "sudoers" file (on Linux in /etc/sudoers).
    http://www.sudo.ws/sudo/man/sudoers.html

Some convenient commands for Linux

I have created a file /etc/profile.d/aliases.sh
and put the following into it:
alias l='/bin/ls -alF'
PS1='[\u@\h \W]\$ '

This file is sourced by /etc/profile - look inside for this code:
for i in /etc/profile.d/*.sh ; do
if [ -r "$i" ]; then
. $i
fi
done

chkconfig - command to configure things to start on reboot.

chkconfig sendmail on
or you can explicitly specify the run-levels:
chkconfig --levels 2345 sendmail on

to look at the status:
chkconfig --list
or
chkconfig --list sendmail

the output of the last command:
sendmail 0:off 1:off 2:on 3:on 4:on 5:on 6:off

Crontab - a command to schedule jobs. Each user has a separate crontab. A crontab file contains instructions to the cron daemon. Commands in any given crontab are executed as the user who owns it and using environment of /bin/sh.
- http://www.cs.princeton.edu/cgi-bin/man2html?crontab:1 - manpage
/etc/cron.d/ - cron directory
/etc/default/cron - contains cron default settings
/etc/cron.d/cron.deny - list of denied users
/var/cron/log - accounting information
/var/spool/cron/crontabs - spool area for crontab

export EDITOR=vi
crontab -e (Edits the current crontab or creates a new one)
crontab -l (Lists the contents of the crontab file)

The cron file may contain comments (start with #) or active lines with 6 space-separated fields - minute hour day month weekday command.
Note: format is very rigid - use just one space as a separator.
Format:
Minute - Minutes after the hour (0-59).
Hour - 24-hour format (0-23).
Day - Day of the month (1-31).
Month - Month of the year (1-12).
Weekday - Day of the week. (0-6; the 0 refers to Sunday).

Asterisks (*) specify when commands are to be run in every instance of the value of the field. For instance, an asterisk in the Month field would mean that the command should be run every month. In addition, multiple events can be scheduled within a field by separating all instances with commas - with no space between.

Here are some examples of scheduling:

0 0 * * * -- midnight every day
0 0 * * 1-5 -- midnight every weekday
0 0 1,15 * * -- midnight on 1st and 15th of month
0 0 1 * 5 -- midnight on 1st of month and every Friday

Example of the whole file:

# #################################################
# Crontab Scheduling Explained:
# Minute - Minutes after the hour (0-59).
# Hour - 24-hour format (0-23).
# Day - Day of the month (1-31).
# Month - Month of the year (1-12).
# Weekday - Day of the week. (0-6; the 0 refers to Sunday).
# Examples:
# 0 0 * * * - midnight every day
# 0 13 * * 1-5 - 1 pm every working day
# 0 3 1,15 * * - run at 3 am on 1st and 15th of every month
#
# #################################################
1 * * * * /usr/sbin/ntpdate -s time.apple.com
# #################################################
# uncomment next line to get the default environment into a file
# * * * * * env > /tmp/env_sh_root.txt
# #################################################
# ---------------------------
# remove old backups and logs
# ---------------------------
5 3 * * * ls /backup/mysql/* | perl -nle 'unlink if (-M) > 30;'
6 3 * * * ls /backup/cr/* | perl -nle 'unlink if (-M) > 30;'
#####
# find /somepath -name 'bak*' -type d -print | perl -nle 'system("rm -rf $_") if ((-d) && (-M $_ > 30));'
#####
# ---------------------------
# backup mysql databases and some blog files
# ---------------------------
10 3 * * * /bin/bash -c 'mydate=`date -I`;tar zcvf /backup/mysql/mysql${mydate}.tar.gz /var/lib/mysql > /backup/mysql/log${mydate}.log 2>&1'
15 3 * * * /bin/bash -c 'mydate=`date -I`;tar zcvf /backup/cr/cr${mydate}.tar.gz /home/selector/web/cr > /backup/cr/cr${mydate}.log 2>&1'
# ---------------------------

More examples
# --------------------------------
# Mon-Fri at 3 AM - run the script
# --------------------------------
00 03 * * 1-5 /bin/bash -c 'cd /app/data1/home/stockln/gsl/invmgmt/scripts; /home/someuser/somepath/runLoader.sh somescript > /somepath/runLoader.log 2>&1'

Crontab starts with "sh" shell. So usually it is a good idea to invoke bash,
and may be also to source an environent file. Here is an example:

/usr/bin/bash -c ' source envfile; $PERL $WRAPPER some_executable some_options --dt=$JDSCRONDT >> $logdir/logname_${JDSCRONDATE}.txt 2>&1'

two methods to remove old files:
5 3 * * * ls /somepath/* | perl -nle 'unlink if (-M) > 30;'
6 3 * * * find /somepath -name 'bak*' -type d -print | perl -nle 'system("rm -rf $_") if ((-d) && (-M $_ > 30));'

To have the date on your server synced automatically, ask admins to set up ntpd / ntpdate so that the time is automatically maintained. To do this just set up a cron job as root like this:

1 * * * * /usr/sbin/ntpdate -s time.apple.com

This will sync time with the apple timeserver every hour.
Note - when we use the command as above, it will ignore the ntp config file /etc/ntp.conf

If you don't have auto-sync set up as described above and just need to set the date manually, simply do the date command without option, take the output, change it to your liking - and then feed it to a date command with "-s" option, for example:

date -s "Mon Nov 29 09:31:00 EST 2004"

DNS related commands: nslookup, dig, host

nslookup www.levselector.com ns18.zoneedit.com
nslookup -type=NS levselector.com

Example: looking for MX or SOA record:
nslookup -type=MX yourdomainname.com
nslookup -type=SOA yourdomainname.com

host iguanahelp.com
dig iguanahelp.com

vsftp:

ps auxw | grep ftp
it will show this:
/usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf

Edit the config file:
vi /etc/vsftpd/vsftpd.conf
I added this line to the config file:
use_localtime=YES
and restarted the vsftpd server:
===========================
cd /etc/rc.d/init.d
./vsftpd restart
===========================

The standard log is here:
/var/log/vsftpd.log

Documentation is here
http://vsftpd.beasts.org/vsftpd_conf.html

How to set up a directory for only ftp access:
Attempt was unsuccessfull.
Here are some commands used by an admin
locate sftp_server
updatedb&
locate sftp_server
service vsftpd restart
touch /etc/vsftpd.chroot_list

to add a new user:
useradd myuser
passwd mypasswd

Note: Never make password the same as login - there are common attacks which exploit this security hole.
Next step - add the user to different groups (if necessary) by editing file /etc/group
Next - create subdirectory "web" : /home/bogdanov/web/ and set proper permissions for home directory and subdirectory.

to remove a user:
userdel myuser
cd /home
rm -rf myuser

To create a new group:
groupadd mynewgroup
or manually add to /etc/group

groupmod - modify (rename, etc.) the group
groupdel - delete the group

chmod user:group filename
chgrp gr_id filename - to change the group of the file

Commands to see user and groups:
id
groups myuser
finger myuser

Here is how to see the info about your unix box:

uname -a
cat /proc/cpuinfo
cat /proc/meminfo
cat /proc/version
cat /etc/redhat-release
cat /etc/issue

cd /etc/httpd/conf - here are web configs
# to see cron files of all users:
cd /var/spool/cron

To see what's going on:
top
vmstat
iostat
netstat
dmesg

To reboot the computer remotely, become root and issue the command "reboot".
See also:
man reboot
man halt
man poweroff
man init
man shutdown

Common security threats exploiting commands wget and curl. You can greatly reduce the risk by removing run privileges from users others than root, for example:

chmod 750 /usr/bin/wget
chmod 750 /usr/bin/curl

How to start IPs:
go into this directory:
/etc/sysconfig/network-scripts
and edit the files:
ifcfg-eth0:1
ifcfg-eth0:2
etc.

then do:
./ifup ifcfg-eth0:1
./ifup ifcfg-eth0:2
etc.

-- -- more commands (random flow) -- --