Some people make a distinction between real programming and scripting but it's a false dichotomy. If a computer runs it, it's a program and that means it's worth taking seriously:
This doesn't mean memorizing the function reference but rather knowing
the basic syntax, idioms and how to find information as you need it
(help, perldoc, pydoc, etc.). The applies particularly to other
people's code: if you're copying an example or taking over someone's
program you'll avoid future grief by taking the time to learn about
anything you don't understand.
This gets a lot of attention as a security measure but it's even more important for robustness: any time you assume something you are introducing an opportunity for a bug.
Because you're protecting against many different things the answer is to code defensively:
Test your assumptions - many languages have an assert statement
which you can use both to check and document what you believe to be
true at the start of a program or function - e.g.
assert(len(files) > 0)
assert(node_number > 0 && node_number < 100)
assert(key in array)
Rule of thumb: any time you're about to copy code substantially the same code it's a sign that you want to move that code into a common location whether that is a common function, class or even a separate program. The latter case is surprisingly easy to forget about: Unix rewards creating small, simple tools and it's trivial to chain them together.
A simple example: ~sysadm/bin/inline-resolver was originally
developed for some cron emails. Since then I've used it for many
different types of log files, reports and a few config files - a handy
return on the initial 10 minute investment.
Most programs will never be worth writing separate documentation for and even for the ones which are, that documentation will tend to drift out of date. A more pragmatic approach is to make the code more explanatory:
Ontology matters: use English names to clearly indicate what
something does. A year from now you're not going to need to remember
what load_users_from_database() or report-spam does - in sharp
contrast with fixup_data() or delme.tmp.
Avoid surprises: try to make sure your code does exactly and only what those clear names suggest. If you find it tempting to do other work, particularly destructive work, in a non-obvious place you might want to reconsider your program design.
Don't waste your users' time with implementation details: unless it's reasonable to expect it don't require knowledge of the underlying features you're using. For example, it would be lazy to simply return the handle for a MySQL result result instead of an array - both because they shouldn't have to know and because you might want store data somewhere else without rewriting everything. On the other hand, it's reasonable to assume that someone using a Unix administration tool should be familiar with syslog, ssh, etc.
Optimize for the default case: if you can do something programmatically, don't be lazy and force the user to do it. The simplest example is the way most Unix programs store their configuration in a default location but allow you to override that in the rare cases when you need to.
Don't be afraid of objects: if you need to maintain a great deal of local data and functions which work with that data, it's much cleaner to create a class even if it's a little more work up front.
My general rule for this is you should try to have each class represent a discrete part of the problem you're solving: users, orders, experiments, etc. The idea is that someone who is familiar with the problem should not require help understanding the basic structure of your classes - even if you don't expect many other users a clean, logical structure will reduce bugs and avoid situations where something "simple" becomes quite hard because your structure does't allow it.
A good step before writing your code is to think about those relationships: "a User has zero or more Orders", "each Simulation will have at least one Task with associated data; the job queue will receive (Task, Data) pairs and submit them to the cluster; when a Task completes it saves its results and notifies the parent Simulation by calling self->Simulation->taskComplete(self, data_id)"
Make your errors unique and Google-able: try to indicate clearly
what happened where (e.g. load_mesh_data(): unable to read foo.dat
rather than "Failed to open file") and include the operating system
or library error message where applicable.
Try to avoid positional variables: you're knee-deep in foo(), is the email address $3 or $4? This is particularly useful in languages which support named arguments because you can avoid having to remember what optional argument goes where and you can add options in the future without breaking legacy code - consider the problem of taking this function which runs a program on a cluster node and adding the ability to specify a different username:
run_command_on_cluster("node5", "/path/to/program", True)
You could simply change it to this:
run_command_on_cluster("node5", "/path/to/program", True, "my_cluster_account")
Next week someone wants you to add priorities and logging and you end up with this:
run_command_on_cluster("node5", "/path/to/program", True, "my_cluster_account", "High", "file.out")
Which works until someone asks why there's a file named "Low" in their home directory and other people are complaining about their job hogging the cluster because they got the order for the last two parameters confused. A week later, someone asks why the detached parameter can't be optional since they always set it to true…
Named parameters provide a nice solution to this by making position irrelevant:
run_command_on_cluster(hostname="node5", command="/path/to/program", detach=True)
run_command_on_cluster(hostname="node5", username="cluster_account", command="/path/to/program", detach=True)
run_command_on_cluster(hostname="node5", command="/path/to/program", username="my_cluster_account", detach=True, priority="High", logfile="/data/debug.log")
Unlike passing a hash or associative array around this approach still allows the compiler to warn about missing or extra parameters and should be faster, too.
It's probably not necessary to write a separate man page but it's pretty easy to make sure your program supports a --help option with some information. The Python optparse module (see below) makes this quite easy; Perl has Pod::Usage which is slightly more work but allows you to generate a full manpage.
There are a number of guides for shell usage (e.g. Bash Pitfalls) and half of the tips are various techniques avoiding problems with embedded spaces and meta-characters, excessive quoting, etc.
Bad:
for foo in `ls -1 *.mp3`; do echo $foo; done
Good:
for foo in *.mp3; do echo "$foo"; done
find . -type \*.mp3 -print0 | xargs -0 rm
TIME_T=`date +%s`
Better:
find . -type \*.mp3 -delete
TIME_T=$(date +%s)
Modern shells support functions and it's always a good idea to move common code into functions, particularly because that makes it easy to add better error-handling without cluttering up the main body of your code:
essh() {
CMD="$@"
if [ ! -z "$CMD" ]; then
CMD="ssh $CMD"
fi
ssh -tA 198.202.70.23 $CMD
}
Once you need to do more than minimal string processing or basic functions it makes sense to switch to a real programming language and avoid the clutter of shell-isms. If your program isn't trivial the richer syntax and libraries will quickly eat up the time spent switching to a real programming language.
You're going to want to know what any non-trivial program is doing and it's nice not to have to uncomment a bunch of print statements when that happens. Taking logging seriously from the beginning offers several advantages:
Variable log levels: you frequently need to distinguish between messages which are only useful when debugging the program, messages which indicate progress and other non-critical information and failures or errors which you always want the user to know about.
Separation of data and log messages: that stray print/echo might
end up in your data (e.g. myscript > outputfile) or might be
discarded entirely if your program is running an environment where the
output isn't recorded. Any real logging system will allow you to
easily log into syslog() or files and do things like maintain separate
logs so you could separate errors caused by invalid input and errors
caused by bugs, misconfigured systems, etc.
Convenience: filtering, formatting and context. Using a standard logging system means you can get out of the business of having to manually synchronize the formatting in all of your print statements. You also get things such as the exception traceback support which Python's logging module includes for free.
logger -t "my-shell-script" -p debug "reached this point in the script"
logger -s -t "my-shell-script" "Copying files…"
(The latter case demonstrates how logger with -s can be a search-and-replace alternative to echo)
open( STDOUT, "|-", "logger -t MyScript" );
open( STDERR, ">&STDOUT" ); # Could also be "logger -t MyScript -p error"
select(STDERR); $| = 1; # Disable buffering for our input and output streams:
select(STDOUT); $| = 1;
use Logger::Syslog;
logger_prefix("MyScript")
notice("Something broke!");
import logging
logging.debug("reach this point for user %s" % username)
…
logging.critical("Couldn't open config file %s - aborting!" % config_file)
from SimpleSyslog import SimpleSyslog
log = SimpleSyslog()
log.warn("foobar")
Because you're smart you're building functions / classes instead of duplicating code. You've probably run into the case where you want to change something but aren't sure what the implications of that change are. This is where the concept of a Unit Test started: by writing simple tests for each component, you can make your changes with impunity because you can easily run your test suite and discover any mistakes in the new code.
The Test-Driven Development school goes a step further and recommends writing the tests before the code since that gives you a solid reference for how you expect your code to behave and it's not unusual for the exercise to suggest design changes.
A unit testing framework simplifies this process by providing all of the test-related structure and handy primitives to use when writing your tests.
A few guidelines:
import unittest
def my_func():
…
class TestMyFunction(unittest.TestCase):
def testResultRange(self):
i = my_func()
self.assertTrue(i <= 5 and i >= 1)
if __name__ == '__main__':
unittest.main()
use Test::Unit;
…
sub test_foo {
my $i = my_func();
assert(($i <=5 and $i >= 1), "my_func() failed to return a value between 1 and 5!");
}
use Getopt::Long;
Getopt::Long::Configure ("bundling"); # Allow single-dash options to be groups: -vvvv = -v -v -v -v
GetOptions(
'v|verbose+' => \$Verbosity,
'max-days=i' => \$MaxAgeInDays,
'test' => \$TestMode,
);
from optparse import OptionParser
parser = OptionParser()
parser.add_option("-v", "--verbose", action="count", dest="verbose", help="provide more detail about what this program is doing", default=0)
parser.add_option("-l", "--log", dest="logfile", metavar="LOGFILE", help="record information in a file instead of displaying it")
(options, args) = parser.parse_args()
if options.verbose > 1:
print "This will be a chatty program"
if options.logfile:
print "Storing diagnostic info in %s" % options.logfile
There's a good intro to the module in Doug Hellmann's Python Module of the Week: optparse
Surprisingly many people do not know about the env(1) utility. It's handy for shell scripts as it allows you to add or delete environmental variables as part of the command-line executed in a different context by something like SSH or sudo.
Here's an example which clones a Unix system disk using rsync over SSH by providing the ssh-agent(1) authentication info for the privileged rsync command to use:
sudo env SSH_AUTH_SOCK=$SSH_AUTH_SOCK rsync -a --delete --progress -x --rsync-path="/usr/bin/sudo /usr/bin/rsync" / USERNAME@CLONE_HOST:/PATH_TO_CLONE_DISK/
Yes, you can try to do this using regular expressions.
No, you probably shouldn't.
import csv
for row in csv.reader(file('BadHardcodedFilename.csv')):
print "Column 2 is %s" % row[1]
use XML::Simple;
my $file = XMLin('users.xml')
print $file->{'Users'}->{'cadams'}->{'email'};
use File::Find;
foreach (@ARGV) {
find( { wanted => \&find_files }, -l $_ ? readlink($_) : $_ );
}
sub find_files {
return unless -f;
open(F, $_);
print "$_ is a Python script\n" if <F> =~ m/python/;
}
import os.path
def find_perl_scripts(arg, dirname, files):
for f in files:
if '#!/usr/bin/perl' in file(os.path.join(dirname, f)).read(15):
print "%s is a Perl script" % f
os.path.walk(os.path.expanduser('~/bin'), find_perl_scripts, None)
Once you outgrow a simple tail -f * something like this comes in
handy if you can't use something like SEC:
use IO::Multiplex;
$mux = IO::Multiplex->new( );
$mux->add($FH1);
$mux->add($FH2); # ... and so on for all the filehandles to manage
$mux->set_callback_object(__PACKAGE__); # or an object
$mux->Loop();
sub mux_input {
my ($package, $mux, $fh, $input) = @_;
if ($input =~ m/…/) …
}
(Courtesy of the Perl Cookbook)
cron has a handy @reboot time specifier which you can use to ensure
that your code will be restarted when a machine reboots:
@reboot /path/to/my/script
Creating a proper Unix daemon requires a fair number of steps to do properly. It may be easy to use a program such as daemontools or daemonize which allows you to run a program as a daemon without modification.
If you need more control other people have done the hard parts:
expect(1) takes a script and uses that to run another program. This allows you to script processes which normally require manual intervention and because it supports branching you can handle tasks with conditional steps such as connecting to a remote system using SSH which might require you to enter a password or accept a host-key if you haven't connected before.
expect has inspired a number of libraries for most major languages and by now I would only recommend using those as the richer language offered by something like Perl is worth it if you need to do anything other than provide canned responses.
use Expect;
my $exp = Expect->spawn("ssh -t $Host $command") or die("Couldn't connect to $Host!");
$exp->expect(
5,
[
'^Are you sure you want to continue connecting \(yes\/no\)\?',
sub { my $self = shift; $self->send("yes\n"); exp_continue; }
],
[ '^Password:', sub { my $self = shift; $self->send("$Password\n"); exp_continue; } ],
…
);
GNU Screen is a tool you need to know. It's handy if you're using an unreliable connection, need to run a terminal application for a long period of time or want to run indepdent applications on multiple hosts:
screen quickstart on the SNL Info Wiki
Screen is also handy because it's scriptable and provides ways for you to create and manage individual windows within a session:
screen -X screen -t'$Hostname' cmd