command-line-fu for web developers

$ [ $(man bash | wc -l) -gt 3000 ] && exit

David Winterbottom | @codeinthehole


Why you should care...

* IDEs can lead to lack of understanding of the tool (eg SVN) * I am a big fan of commandline and am creator of commandlinefu.com, from which many of these tips come from. * Am not a massive expert, more a collector. Am not a sysadmin

Disclaimer

Overview

  1. What everyone should know...
  2. History
  3. MySQL
  4. SSH
  5. BASH Shortcuts
  6. Miscellany
-- basics not really that basic -- talk for anyone who types stuff in linux -- there are lots of bash features that can make your life easier / improve productivity -- sense of power

What everyone
should know

UNIX toolset

- unix philosophy of doing one thing well - basic building blocks - awk is a scripting language - awk and sed are turing complete - computation of string STDIN input

I/O redirection

-- with knowledge of the core tools and the various mechanisms for chaining them together - an awful lot of power is at your disposal -- pipes and filters (sed,grep)

Example

$ head -10000 /var/log/apache2/access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head

Lesser known commands: tee

echo "example" | tee file.txt echo "example" | sudo tee root-owned-file.txt echo "example" | tee >(rev) >(sed 's/e//g')
-- we'll see a clever use of this with vim later on

Lesser known commands: at, batch

$ echo "ps aux > ~/stats.txt" | at 23:00 $ echo "~/put-kettle-on.sh" | at teatime echo "~/heavy-process.sh" | batch

Lesser known commands: locate

$ locate couch
/etc/couchdb
/etc/couchdb/default.d
/etc/couchdb/default.ini
/etc/couchdb/local.d
/etc/couchdb/local.ini

Lesser known commands: apropos

$ apropos couch
couchdb (1) - Apache CouchDB database server
couchdb-dump (1) - a CouchDB dump utility
couchdb-load (1) - a CouchDB load utility
couchjs (1) - Apache CouchDB JavaScript interpreter
couchpy (1) - a CouchDB Python view server

Combining commands: &&, ;

$ (markdown README.md > md.html) && x-www-browser md.html $ ./long-running-local-job.sh ; notify-send "Job is now finished"
$ ./long-running-process.sh ; eject
 $ ./long-running-remote-job.sh | mail -s "Job is finished" root@commandlinefu.com

Beep when server comes back up

$ ping -i 60 -a $IPADDRESS

History

Retrieving commands

-- default configuration can be improved -- hackers don't press UP

Incremental search

More inputrc tips to come

History file tweaks

History expansion 1: Event designators

- beware of getting too cryptic - see man bash for more details - !! is an event designator

Digression: This time I mean it!

$ sudo !! $ say "make me a sandwich"
say: what? make it yourself.
$ sudo !!
$

http://xkcd.com/149/

Digression: Oops - forgot the sudo

:w !sudo tee %

History expansion 2: Word designators

History expansion 3: Modifiers

Magic space

$if Bash
    Space: magic-space
$endif

Alias common commands

history | awk '{print $2}' | sort | uniq -c | sort -n | tail
-- some commands support their own form of aliasing (eg git)

Script common actions

PATH="/home/david/bin:$PATH"
export PATH

MySQL

~/.my.cnf - Easy authentication

$ chmod 600 ~/.my.cnf
- for setting user-specific options

~/.my.cnf - Custom prompt

(root@localhost) [wl_demo]> SELECT User,Host from mysql.user;

Remote backups

$ ssh $HOST 'mysqldump $DATABASENAME | gzip' > /var/backups/backup-`date +%Y-%m-%d`.sql.gz

Copy database between machines

$ mysqldump -p -h $HOST -u $USER $DATABASENAME | mysql -D $DATABASENAME $ mysqldump -p -h $HOST -u $USER $DATABASENAME | sed 's/@[a-zA-Z0-9.]*/@localhost' | mysql -D $DATABASENAME

Monitor queries

$ watch -n 1 "mysqladmin processlist"

Complex SQL queries

$ echo "SHOW TABLES LIKE 'App__%'" | mysql -D $DATABASENAME | sed "1 d" | awk '{print "DROP TABLE `"$1"`;"}' | mysql -D $DATABASENAME
piping is better than mysql -e

Maatkit (mk-*)

$ mk-find --engine MyISAM --exec "ALTER TABLE %D.%N ENGINE=InnoDB"

Secure Shell (SSH)

Secure Shell (SSH)

- replacement for telnet - network protocol using pulic key cryptography - typically used for logging into a remote machine

SSH 101

$ ssh user@host Login to remote host
$ ssh user@host 'ls /tmp' Execute command on remote host
$ scp file.txt user@host: Copy file to remote host

Public keys

$ ssh-keygen -t rsa Generate public key
$ ssh-copy-id user@host Copy public key to host
$ ssh user@host No password prompt when logging in
- eg github - should use a passphrase - gnome keyring manager can take care of it for you - or use ssh-agent to store it in memory

~/.ssh/config - Named hosts

- each option equivalent to a commandline option

Digression: custom prompts

david@earth:~ $ root@mysite-app1:tmp $

Mount using SSH: sshfs

$ sshfs $SERVER:/path/to/folder /path/to/mount/point

Edit a remote file using: vim

$ vim scp://user@host//path/to/file $ vimdiff /etc/fstab scp://user@host//etc/fstab

Connect to unreachable hosts

ssh -t $REACHABLE_HOST ssh $UNREACHABLE_HOST

Cluster SSH

cssh mysite-app1 mysite-app2 mysite-app3

GNU Screen

--like a text version of a window manager

Miscellany

Terminal program

BASH keyboard shortcuts

Change to previous working directory

cd - cd $OLDPWD

Brace expansion

cp /path/to/file{,.bak} cp /path/to/file /path/to/file.bak $ cssh mysite-{app,db}{1..32}
$ ls *.{jpg,jpeg,png}

Follow log files

$ tail -f /var/log/apache2/error.log $ less +F /var/log/apache2/error.log

Flush file

$ > /var/log/apache2/error.log

Search codebase

$ grep -inHr $QUERY ~/workspace/project | grep -v .svn

Use python to server current directory

python -m SimpleHTTPServer sudo python -m smtpd -n -c DebuggingServer localhost:25

Query wikipedia using DNS

dig +short txt $KEYWORD.wp.dg.cx

Watch ASCII Star Wars using telnet

telnet towel.blinkenlights.nl telnet towel.blinkenlights.nl 666

Summary

Summary

Further reading