Re-execute commands: At the command prompt press cursor up key and your last command will be shown. You can either re-execute this command or you can edit the command before executing it. Use cursor up and down to scroll through the most recent commands.

Type the command history to see your command history. Bash users can search in their history by pressing CTRL-r.

1.4 The home directory - your personal folder

• Your home directory is a subfolder of the folder /home. The name of your home directory is equal to your username (/home/"username").
• After login you are normally in your home directory.
• Your home directory is daily backed up.
• Disk space in your home directory is limited, see disk quota further down.
• For more information see LinuxHome

1.5 Essential shortcuts

2 Changing your password

https://password.ethz.ch

Please choose a password which is not easy to guess. Use small and capital letters, numbers and special characters like ,.$!.

3 Folders and files

3.1 Create a folder

mkdir new_folder

The following command creates in one step a folder new_folder and a subfolder new_subfolder inside new_folder

mkdir -p new_folder/new_subfolder

3.2 Change directory

cd new_folder

cd new_folder/new_subfolder

cd ..

• .. stands for the parent folder

3.3 Change into your home directory

cd
cd $HOME
cd ~

• ~ and $HOME are variables for your home directory

3.4 List files and folders

ls

ls -l

ls -la

ls -latr

3.5 Rename files and folders

mv old_filename new_filename

3.6 Move folders and files

mv file folder/

mv file ../

3.7 Copy files and folders

cp file1 file2

cp -r folder1 folder2
cp -a folder1 folder2

• -r, --recursive: copy directories recursively
• -a, --archive: includes the option -r. In addition it will also preserve the mode, ownership and timestamps of files and folders

Copy a file /tmp/file into the current folder (.)

cp /tmp/file .

Copy the file file out of the current folder into the parent folder (..)

cp file ../

• . stands for the current folder
• .. stands for the parent folder, i.e. the folder above the current folder

Use wildcards to copy several files at once.

cp file* folder/

cp file? folder/

• ? stands for a single character
• * stands for any number of characters including no character

3.8 Delete a file and folder

rm file
rmdir folder
rm -r folder

• rmdir deletes the folder only in case it is empty
• rm -r deletes the folder recursively, including all its files and subfolders

Delete all files in the current folder

rm *

Delete all files AND all folders in the current folder

rm -r *

Delete all files and folders in the current folder without asking you (-f = force)

rm -rf *

WARNING: Use the option -f with caution!

3.9 Take care of spaces in folder and file names

mv "My File" "My Backup File"

• Using spaces in folder or file names is NOT recommended. For better reading you may want use underscores instead of spaces (My_Backup_File)

4 File and folder permissions

• read (r): File: Read and view. Directories: Read its content.
• write (w): Files: Write or edit. Directories: Can modify its content, i.e. creating/removing files or folders.
• execute (x): Files: Execute or run the file as a program. Directories: Can change into it.

The permissions can be set for the following user groups:

• user: The owner of a file or directory
• group: A user can be part of one or more groups.
• other: Defines the permission for all other users, not being the owner or belonging to the group.

Note:

• Directories have to be executable! Otherwise you get permission denied, if you want to change into it.
• In case you still get permission denied, check whether all directories in the path to the file are read and executable.

4.1 Set file permissions

chmod g+w file           # gives write permission (w) for the group (g)
chmod o+rw file          # gives read/write permission (rw) for anybody/others (o)
chmod u+rwx file         # gives read/write/execute permission (rwx) to the user/owner (u)
chmod a+rwx file         # gives read/write/execute permission (rwx) to all (a)

• u user/owner, g group, o others, a all (= user+group+others)
• r read, w write, x execute
• + set permission, - remove permission

Beside the above modes, you can also use the octal-mode to change the permissions.

• 400 read by user
• 040 read by group
• 004 read by others
• 200 write by user
• 020 write by group
• 002 write by others
• 100 execute by user
• 010 execute by group
• 001 execute by others

The above numeric permissions can be added to set a certain permission. For example to give read/write by the owner and only read by everyone else (400+040+004+200 = 644) (-rw-r--r--)

chmod 644 file

A folder has to be executable and readable for everybody, but should be only writable by the owner (400+040+004+200+100+010+001 = 755) (drwxr-xr-x)

chmod 755 folder

Basically, there are three sets of bits, one each for user, group and other (in that order). Each set has three bits and each bit is an on/off switch for read, write and execute (in that order).

4.2 Some advanced examples

Make all files and folders in the current directory only accessible for you. In other words remove (-) all rights (rwx) for the others (o) and the group (g)

chmod o-rwx *
chmod g-rwx *

With the above command only the permission of the files and folders in the current directory are changed. To change the permission recursively use the option -R

chmod -R o-rwx *
chmod -R g-rwx *

With the above command the permission of hidden files and folders (starting with a .) are not changed in the current directory. An other possibility to change permissions recursively can be done with the find command in combination with the option -exec:

find . -exec chmod o-rwx {} \;
find . -exec chmod g+rw {} \;

Assuming you would like to give to everybody (user, group and others) read permissions to the files and folders in the current directory

find . -type f -exec chmod o+r {} \;
find . -type d -exec chmod o+rx {} \;

Assuming you would like to give to everybody (user, group and others) rw permission (but not x) to all files (-type f) and rwx permission to all directories (-type d) inside the current directory:

find . -type f -exec chmod ugo+rw {} \;
find . -type f -exec chmod ugo-x {} \;
find . -type d -exec chmod ugo+rwx {} \;

Same example as above, but others (o) should not gain write permission (only read permission):

find . -type f -exec chmod ugo+r,o-w,ug+w,ugo-x {} \;
find . -type d -exec chmod ugo+rwx,o-w {} \;

Sometimes it's maybe easier to use the octal-mode for chmod. The following commands do the same as the ones above

find . -type f -exec chmod 664 {} \;
find . -type d -exec chmod 775 {} \;

4.3 Setting your umask

Run the command umask in order to see your setting

umask

umask 0027

4.4 Show my identity (username, groupnames)

4.5 Inherit group from parent folder

chmod g+s folder

5 Text files

5.1 View text files

less file.txt
more file.txt
cat file.txt

5.2 Less command

less file

• The important keys in less are

• Note, the same keys are also valid when viewing a manpage with the man command.

5.3 Write the output into a text file

date > time.txt

• >: writes the standard output to the file. The file will be either created or, if already existing, overwritten.
• >>: appends the output to the file.

A simple example

echo "The current date:" > time.txt
date >> time.txt

cat time.txt will look like

The current date:
Mon Sep  8 12:12:34 CEST 2008

Write standard output AND standard error into the same text file

any_command > output_AND_error.txt 2>&1

• 2>&1: 2 defines the standard error, which is written inot the same file as the standard output (= 1)

5.4 Sort text files

cat file | sort
cat file | sort > file_with_sorted_lines
cat file | sort | uniq

• uniq omits repeated lines

5.5 Search in files using grep

grep <search_pattern> file
grep "Error" file
grep -i "error" file
grep "^Beginning of Line" file
grep "End of Line$" file
grep "pattern1\|pattern2" file

• -i Ignores case distinctions
• ^ matches the beginning of a line
• $ matches the end of a line
• \| separates multiple patterns with OR condition

An example: How to show only the lines that are not comments in a file? Assuming comments start with a #, the regular expression is ^#. Use the grep option -v to invert the sense of matching:

grep -v "^#" file

grep -v "^#" file | grep -v "^$"

More information about regular expression can be found for example at http://www.robelle.com/smugbook/regexpr.html.

A nice grep tutorial can be found at http://www.selectorweb.com/grep_tutorial.html.

Hint: For pdf files use pdfgrep for more info see

man pdfgrep

5.6 Using sed to edit files

To replace the name Peter with Hans in a file run:

sed "s/Peter/Hans/" file > newfile

sed -i "s/Peter/Hans/" file

sed -i "s/Peter/Hans/g" file

To delete (d) all lines which contains the word Peter run

sed -i "/.*Peter.*/d" file

More examples for sed can be found at http://www.cs.hmc.edu/tech_docs/qref/sed.html

6 Editors

6.1 Emacs

• Open file in emacs

  emacs file
  

• Do not open emacs in windows mode (nw = no window mode)

  emacs -nw file
  

• The important keys in emacs are

  key	function
  Esc+w	Copy the text between the last mark and the current position
  Esc+%	Search and replace
  Esc	Quit search mode
  Ctrl+y	Paste
  Ctrl+x s	Save
  Ctrl+x c	Quit Emacs
  Ctrl+w	Cut the text between the last mark and the current position
  Ctrl+space	Set a mark
  Ctrl+s	Search
  Ctrl+E	Jump to the end of the current line
  Ctrl+A	Jump to the beginning of the current line
  Ctrl+_	Undo

• There are many mores, see for example
  • http://www-pool.math.tu-berlin.de/doc/emacs/emacsintro.html

6.2 Further text editors

• kwrite
• kate
• kedit
• vim
• gedit

7 Get info

7.1 Free disk space in your home directory - Disk Quota

homequota

7.2 Shows the current location/folder

pwd

7.3 How much disk space is used by a folder or file

du -chs file
du -chs folder

7.4 Show the usage of disk space

df -h
df -h /lhome
df -h /net/atmos/data

7.5 Get help for a command

"command" --help
"command" -h
man "command"

ls --help
man ls

• To navigate inside a man page use the same key like navigating in less.

7.6 Last entered commands

history
history | less

7.7 Monitor system resources

top

top is an alternative program to monitor CPU and memory usage.

top

top -p <PID>
top -p 9874 -p 30473

top -u $USER

• M: sort processes by resident memory usage
• P: sort processes by CPU usage (default)
• T: sort processes by cumulative time
• W: save sorting state and open that way next time

Sometimes the load is high, but top does not show any processes using a lot of CPU or memory. In order to show the "waiting" processes that are producing the load, run

top -i

htop

htop is a tool to monitor CPU and memory usage. It is "newer" than top (see below) and you can scroll the list vertically and horizontally to see all processes and complete command lines. And htop supports mouse operations.

htop

Use the following commands:

• Help
  • h toggle help
• Sort
  • Shift M sort by memory usage
  • Shift P sort by CPU usage
  • F5 show tree view of processes
• Filter
  • use u and arrows to show single users (alternatively htop -u username)
  • Shift H toggle user process threads

atop

Use atop to get a nice overview which part of the system is the bottleneck (cpu, disk, memory, network). Watch for red text !

atop

Find red entries and check the leftmost column to see what causes the problem:

• MEM memory (RAM) -> memory
• SWP swapped memory -> memory
• PAG Paging frequency -> memory
• LVM logical volumes -> IO (more relevant than DSK below)
• DSK physical hard disks -> IO
• NET network -> too much network traffic (check the row at the bottom with bond0)

The bottleneck may also be on an other server! E.g. when reading data via /net/ from another server!

iotop

Use iotop to find out which users/ processes use high IO:

sudo iotop
sudo iotop -ao

• -a show accumulated I/O instead of bandwidth
• -o only show processes or threads actually doing I/O

[nfsd] means someone reads data from another server. There is no way to find out who/ which server.

ganglia

Servers at IAC are monitored with ganglia: https://ganglia.iac.ethz.ch

xrestop

Use xrestop to monitor server resources used by X11 clients

xrestop

ps

ps aux --no-headers | awk '{arr[$1]+=$4}; END {for (i in arr) {print i,arr[i]}}' | sort -gr -k2

Or to list CPU usage in percent (%) run (please note a system with 64 cores has in total 6400% CPU)

ps aux --no-headers | awk '{arr[$1]+=$3}; END {for (i in arr) {print i,arr[i]}}' | sort -gr -k2

List number of processes running per user

ps aux | awk '{ print $1 }' | sort | uniq -c

8 Processes / running jobs

8.1 Show running processes

ps aux
ps aux | grep firefox
ps aux | grep ^$USER

ps aux | grep ^${USER:0:7}

8.2 Start a job or program with lower priority

nice -n 19 ionice -c 3 my_program

1. nice -n 19 set CPU priority to lowest level (19)
2. ionice -c 3: set disk I/O priority to idle level (-c 3)
   idle level = a program running with idle I/O priority will only get disk time when no other program has asked for disk I/O

More examples for ionice

ionice -p PID                    # get priority of process with process number PID
ionice -c 3 -p 1004              # set process with PID 1004 as an idle io process
ionice -c 2 -n 0 my_script       # run my_script as a best-effort program with highest priority
ionice -c 3 -p $(pgrep -u $USER) # change IO priority of all your running processes to idle 

More examples with nice

nice -n 13 my_script         # run my_script with niceness level 13

For already running jobs you need to use renice:

renice 19 -u $USER  # set all of your running jobs to the lowest priority
renice 19 -p PID    # set jobs with PID to the lowest priority

For more info please see

man nice
man ionice
man renice

8.3 Determinate/kill a process

kill <PID>
kill 395
kill -9 356

• -9 will kill the process hardly
• The PID is the process identification number, which is listed in ps aux or top

killall -u $USER

killall -9 -u $USER

8.4 Determinate/kill a process "graphically"

xkill

The mouse pointer gets a cross or a skull. A mouse click will now determinate the program below the mouse pointer. Press the Ctrl+C to quit the "xkill" mode.

8.5 Run a program and send it to the background

program &
emacs &
firefox &

Or

firefox

bg

In some cases the program will be terminated when you close or exit the terminal from which you have started it - even if you have started the program in the background. The Linux tool screen is maybe a good solution to overcome this problem. For more info see LinuxUseScreen. However, screen only works for programs which run completely in a terminal window.

9 Search

9.1 Search a file in the current folder

find . -name index.html
find . | grep index

Apostrophs needed when using wildcards:

find . -name 'index.*' 

9.2 Find files which have been changed during the last X days

find . -type f -mtime -2

9.3 Find files which have been changed recently

find . -type f -printf "%TY-%Tm-%Td %TH:%TM:%TS %f\n" | sort -n

10 Work on different systems

10.1 Login to an other system

ssh username@hostname
ssh beyerleu@firebolt.ethz.ch
ssh beyerleu@firebolt

10.2 Copy files and folder to other systems

scp file beyerleu@firebolt.ethz.ch:
scp file beyerleu@firebolt.ethz.ch:/home/beyerleu
scp -r folder beyerleu@firebolt.ethz.ch:/home/beyerleu

• The first two commands are equal
• Use -r to copy folders recursively

Instead of scp you can also use rsync over ssh:

scp -r     folder beyerleu@firebolt.ethz.ch:/data/beyerleu/
rsync -av  folder beyerleu@firebolt.ethz.ch:/data/beyerleu/

In order to synchronize two folders use the option --delete which will delete extraneous files from the destination folder:

rsync -avz --delete folder beyerleu@firebolt.ethz.ch:/data/beyerleu/

Please use the option --delete with caution. It can delete all your new files, if you use it the wrong way round. Therefore use the option -n which makes rsync perform a trial run that doesn't make any changes and produces mostly the same output as a real run:

rsync -n -avz --delete folder beyerleu@firebolt.ethz.ch:/data/beyerleu/

More useful rsync options:

-z                      files are compressed before transfer, and uncompressed after transfer
--remove-source-files   sender removes synchronized files, useful if you would like to delete files after the transfer
-u, --update            skip files that are newer on the receiver
-c, --checksum          compare checksum (not only mod-time&size) if the same file exists on source and destination (needs much longer)

In case your connection is not stable, call rsync in a loop until rsync gives you a successful return code:

error=1 
while [[ $error -ne 0 ]]
do
   rsync -av myfolder firebolt.ethz.ch:/path_to_data/
   error=$?
done

For more info about rsync, please see

man rsync

11 Shell environment

11.1 Which shell do I use?

echo $SHELL

11.2 Show shell variables

set

11.3 Show defined aliases

alias

11.4 Define an alias

• Bash shell

  alias ls_nice='ls -lag --color=tty'
  

• TCSH shell

  alias ls_nice 'ls -lag --color=tty'
  

11.5 Customize your shell

• If you're using bash put them into ~/.bashrc
• If you're using tcsh put them into ~/.cshrc

12 X Window System

12.1 Restart X Server

13 Problems and solutions

13.1 I can no longer login to a Linux system

• You have exceeded your diskquota. Try to login over ssh and delete files.
• /tmp is 100% used. Try to login over ssh and delete files in /tmp.
• You have forget your password. The administrator can set a new password.
• Your account was disabled. Ask the administrator to enable it again.
• Quite often your Trash which is under $HOME/.local/share/Trash fills up the whole disk space. Don't forget to delete your trash from time to time.

13.2 How can I find out the size of the folders in my home directory?

To print the size of your folders in the home directory type:

find $HOME -maxdepth 1 -type d -exec du -hs {} \;

13.3 Checkout the Linux FAQ