Computing Environment Setup for Bioinformatics and Computational Biology

So, you want to harness the immense power of bioinformatics and computational biology for your science?

Here's some advice that will save you headaches and make your life easier when working in a Linux/UNIX environment.

NOTE: These instructions are for setting up your Linux/UNIX computational environment. This might be on your local machine, but it could also be on a computing cluster like TACC that you log into remotely.

Instructions for setting up your local machine (for example, your laptop) with programs for editing text, accessing remote servers, etc., are covered over at Computer Setup.

Introduction to the Shell

You will want to learn basic Unix commands and syntax for navigating your command-line. environment and running commands. These include things like copying files, interrupting a process, redirecting the output/input of a program.

Introduction to Unix
A short intro that describes some key commands and concepts.
Unix Commands Quick Reference
Our Cheat Sheet of useful Shell commands and tricks]]

Login Scripts

When you log into a remote computer or open a terminal window on your computer, you are entering a "shell" program that interprets Unix commands. There are a few different shells, but generally most are bash derivatives. Several "login scripts" are loaded in the new shell. All that means is that the shell commands in those files are run before it gives you a prompt and lets you start typing your own commands. These generally have names like .bashrc or .zshrc that depend on your shell. (But it can get complicated with some login scripts run globally when anyone logs into a computer and some only run for your user account.)

Setting up your `$PATH`

When you invoke commands such as python3 via at the command line, your shell searches all file directories listed in your $PATH in order to execute that command. Errors such as "command not found" when you try to run a program mean you need to add the directory containing that program to your PATH.

To show the current directories that are in your $PATH use this: %CODE{bash}% echo $PATH %ENDCODE%

To add a directory to your $PATH you can run this command %CODE{bash}% PATH=/your/directory/here:$PATH %ENDCODE%

%COLOR{red}%Be sure that you include the colon and the $PATH part of this. If you leave them off then your shell will not know where to look for built-in commands like ls, cd, etc.!

Generally, you want to add the given directory to the end or the beginning of your PATH variable list, since when you invoke a command, the directories will be searched from beginning to end and the first match will be the one that is run. Because this can lead to confusion, there is even a command you can use that gives you the path to the executable that will be run if you type a command:

%CODE{bash}% which %ENDCODE%

Using TACC

Connecting: Head Nodes and Compute Nodes.

The current system on TACC that we use for most of our computing is lonestar6. It's address is ls6.tacc.utexas.edu, so to ssh to it you use:

%CODE{bash}% ssh @ls6.tacc.utexas.edu %ENDCODE%

After you fill in your password and make it past 2FA, you get a shell on the HEAD NODE. This is a machine that is used like the brain of the cluster. It's function is to send tasks to its many COMPUTE NODES*.

DO NOT run any computationally demanding or long tasks on the head node. It will inconvenience others by making the machine slow. Your command will be killed it it uses too many resources and you may be banned from TACC.

Instead, you can get an interactive shell on a COMPUTE NODE using this command:

%CODE{bash}% idev -m 60 %ENDCODE%

The -m 60 is asking for a 60-minute slot on one compute node. Currently, you can make this as high as 120 minutes. For longer jobs, you will need to learn about submitting jobs to the queue.

After some informational messages, your terminal will pop up and now you can run commands on the COMPUTE NODE. They have a lot of cores (processors) and memory (RAM), so you can (and should) be running many jobs in parallel on one of these nodes if you are using it for compute. The idev command is mostly meant for development (that is, writing and testing new code/tools), but it can be used for short tasks, particularly if you are using a job manager like Snakemake that can intelligently use the resources.

If you get lost and can't remember if you are on the HEAD NODE or a COMPUTE NODE, you can use this command:

%CODE{bash}% hostname %ENDCODE%

If it has "login" in the name it returns, then you are on the HEAD NODE.

Filesystems: `$HOME`, `$WORK`, and `$SCRATCH`

Conda 101

Whether on TACC or your own computer, you'll want to become familiar with the Conda package/environment manager. It makes it easy to install a wide variety of command-line tools in a way that prevents them from interfering with one another or other settings on your system.

Set up Conda, Mamba, Bioconda

Conda is the main framework. Mamba speeds up Conda installs (once it is installed use mamba everywhere you would use conda for running commands. Bioconda makes it possible to install additional packages related to bioinformatics and computational biology. You'll want all three of these working together in your environment.

Install Conda (the Miniconda flavor). Using the Quick Command Line Install instructions is probably easiest, esp. on TACC.
Reload your shell (close and open the terminal or logout and log back in) so you are in your conda base environment.
Install Mamba using these commands:
%CODE{bash}%

conda install mamba mamba init%ENDCODE%

Set up Bioconda Run the commands here

Using Conda Environments

Conda environments are a way to:

Insulate different installed tools from one another to prevent incompatibilities and unexpected interactions.
Manage and save exactly which versions of different tools you used for an analysis

When you open a new shell, by default the base conda environment will be loaded.

It's OK to install some general-purpose utilities in this environment, but you should generally *install each of your major bioinformatics tools (or sets of tools) in its own environment*.

This sequence of commands creates an environment called breseq-env and installs breseq in it: %CODE{bash}% mamba env create -n breseq-env mamba activate breseq-env mamba install breseq %ENDCODE%

Let's say you were trying to reproduce results from an older paper. You may want to install a specific version of breseq in your environment. In this case, you'd use this variant %CODE{bash}% mamba install breseq=0.36.1 %ENDCODE%

Another very useful set of commands can save your environment to a yaml file: %CODE{bash}% conda env export > environment.yml %ENDCODE%

Or load an environment from a yaml file created by someone else, so you can reproduce their work! %CODE{bash}% conda env create -f environment.yml %ENDCODE%

Many other possibilities are covered in the official Conda documentation under managing environments.

Miscellaneous Timesavers

Use the alias command to set up shortcuts
For example, type sshl to connect to lonestar6 using your username.
Setting up SSH Public Key Authentication
Save yourself typing your password every time you connect to TACC or another server
Using lftp to copy files to/from UT Box
Copy and mirror files from/to your computer and TACC or other servers.