So, you want to harness the immense power of bioinformatics and computational biology for your science?
Here's some advice that will save you headaches and make your life easier when working in a Linux/UNIX environment.
NOTE: These instructions are for setting up your Linux/UNIX computational environment. This might be on your local machine, but it could also be on a computing cluster like TACC that you log into remotely.
Instructions for setting up your local machine (for example, your laptop) with programs for editing text, accessing remote servers, etc., are covered over at Computer Setup.
$PATH (.bashrc)
The current system on TACC that we use for most of our computing is lonestar6. It's address is ls6.tacc.utexas.edu, so to ssh to it you use:
ssh <username>@ls6.tacc.utexas.edu
After you fill in your password and make it past 2FA, you get a shell on the HEAD NODE. This is a machine that is used like the brain of the cluster. It's function is to send tasks to its many COMPUTE NODES*.
DO NOT run any computationally demanding or long tasks on the head node. It will inconvenience others by making the machine slow. Your command will be killed it it uses too many resources and you may be banned from TACC.
Instead, you can get an interactive shell on a COMPUTE NODE using this command:
idev -m 60
The -m 60 is asking for a 60-minute slot on one compute node. Currently, you can make this as high as 120 minutes. For longer jobs, you will need to learn about submitting jobs to the queue.
After some informational messages, your terminal will pop up and now you can run commands on the COMPUTE NODE. They have a lot of cores (processors) and memory (RAM), so you can (and should) be running many jobs in parallel on one of these nodes if you are using it for compute. The idev command is mostly meant for development (that is, writing and testing new code/tools), but it can be used for short tasks, particularly if you are using a job manager like Snakemake that can intelligently use the resources.
If you get lost and can't remember if you are on the HEAD NODE or a COMPUTE NODE, you can use this command:
hostname
If it has "login" in the name it returns, then you are on the HEAD NODE.
$HOME, $WORK, and $SCRATCH
Whether on TACC or your own computer, you'll want to become familiar with the Conda package/environment manager. It makes it easy to install a wide variety of command-line tools in a way that prevents them from interfering with one another or other settings on your system.
Conda is the main framework. Mamba speeds up Conda installs (once it is installed use mamba everywhere you would use conda for running commands. Bioconda makes it possible to install additional packages related to bioinformatics and computational biology. You'll want all three of these working together in your environment.
base environment.
conda install mamba mamba init
Conda environments are a way to:
When you open a new shell, by default the base conda environment will be loaded.
It's OK to install some general-purpose utilities in this environment, but you should generally *install each of your major bioinformatics tools (or sets of tools) in its own environment*.
This sequence of commands creates an environment called breseq-env and installs breseq in it:
mamba env create -n breseq-env mamba activate breseq-env mamba install breseq
Let's say you were trying to reproduce results from an older paper. You may want to install a specific version of breseq in your environment. In this case, you'd use this variant
mamba install breseq=0.36.1
Another very useful set of commands can save your environment to a yaml file:
conda env export > environment.yml
Or load an environment from a yaml file created by someone else, so you can reproduce their work!
conda env create -f environment.yml
Many other possibilities are covered in the official Conda documentation under managing environments.
sshl to connect to lonestar6 using your username.