So, you want to harness the immense power of bioinformatics and computational biology for your science?
Here's some advice that will save you headaches and make your life easier when working in a Linux/UNIX environment.
NOTE: These instructions are for setting up your Linux/UNIX computational environment. This might be on your local machine, but it could also be on a computing cluster like TACC that you log into remotely.
Instructions for setting up your local machine (for example, your laptop) with programs for editing text, accessing remote servers, etc., are covered over at Computer Setup.
You will want to learn basic Unix commands and syntax for navigating your command-line. environment and running commands. These include things like copying files, interrupting a process, redirecting the output/input of a program.
When you log into a remote computer or open a terminal window on your computer, you are entering a "shell" program that interprets Unix commands. There are a few different shells, but generally most are bash derivatives. Several "login scripts" are loaded in the new shell. All that means is that the shell commands in those files are run before it gives you a prompt and lets you start typing your own commands. These generally have names like .bashrc or .zshrc that depend on your shell. (But it can get complicated with some login scripts run globally when anyone logs into a computer and some only run for your user account.)
$PATH
When you invoke commands such as python3 via at the command line, your shell searches all file directories listed in your $PATH in order to execute that command. Errors such as "command not found" when you try to run a program mean you need to add the directory containing that program to your PATH.
To show the current directories that are in your $PATH use this:
echo $PATH
To add a directory to your $PATH you can run this command
PATH=/your/directory/here:$PATH</code>
%COLOR{red}%Be sure that you include the colon and the $PATH part of this. If you leave them off then your shell will not know where to look for built-in commands like ls, cd, etc.!
Generally, you want to add the given directory to the end or the beginning of your PATH variable list, since when you invoke a command, the directories will be searched from beginning to end and the first match will be the one that is run. Because this can lead to confusion, there is even a command you can use that gives you the path to the executable that will be run if you type a command:
which <command>
The current system on TACC that we use for most of our computing is lonestar6. It's address is ls6.tacc.utexas.edu, so to ssh to it you use:
ssh <username>@ls6.tacc.utexas.edu
After you fill in your password and make it past 2FA, you get a shell on the HEAD NODE. This is a machine that is used like the brain of the cluster. It's function is to send tasks to its many COMPUTE NODES*.
DO NOT run any computationally demanding or long tasks on the head node. It will inconvenience others by making the machine slow. Your command will be killed it it uses too many resources and you may be banned from TACC.
Instead, you can get an interactive shell on a COMPUTE NODE using this command:
idev -m 60
The -m 60 is asking for a 60-minute slot on one compute node. Currently, you can make this as high as 120 minutes. For longer jobs, you will need to learn about submitting jobs to the queue.
After some informational messages, your terminal will pop up and now you can run commands on the COMPUTE NODE. They have a lot of cores (processors) and memory (RAM), so you can (and should) be running many jobs in parallel on one of these nodes if you are using it for compute. The idev command is mostly meant for development (that is, writing and testing new code/tools), but it can be used for short tasks, particularly if you are using a job manager like Snakemake that can intelligently use the resources.
If you get lost and can't remember if you are on the HEAD NODE or a COMPUTE NODE, you can use this command:
hostname
If it has "login" in the name it returns, then you are on the HEAD NODE.
$HOME, $WORK, and $SCRATCH
Whether on TACC or your own computer, you'll want to become familiar with the Conda package/environment manager. It makes it easy to install a wide variety of command-line tools in a way that prevents them from interfering with one another or other settings on your system.
Conda is the main framework. Mamba speeds up Conda installs (once it is installed use mamba everywhere you would use conda for running commands. Bioconda makes it possible to install additional packages related to bioinformatics and computational biology. You'll want all three of these working together in your environment.
base environment.
conda install mamba mamba init
Conda environments are a way to:
When you open a new shell, by default the base conda environment will be loaded.
It's OK to install some general-purpose utilities in this environment, but you should generally *install each of your major bioinformatics tools (or sets of tools) in its own environment*.
This sequence of commands creates an environment called breseq-env and installs breseq in it:
mamba env create -n breseq-env mamba activate breseq-env mamba install breseq
Let's say you were trying to reproduce results from an older paper. You may want to install a specific version of breseq in your environment. In this case, you'd use this variant
mamba install breseq=0.36.1
Another very useful set of commands can save your environment to a yaml file:
conda env export > environment.yml
Or load an environment from a yaml file created by someone else, so you can reproduce their work!
conda env create -f environment.yml
Many other possibilities are covered in the official Conda documentation under managing environments.
sshl to connect to lonestar6 using your username.