Bioinformatics Data Skills

The Unix Shell

Topics:

• The Unix philosophy | text streams | pipes and redirection | process control | process substitution

Assignments:

• If you’re using a Mac, you should go ahead and install homebrew
• Next (Mac users only), paste the commands from this script into your terminal. That will use ‘homebrew’ to give you the added functionality of ‘GNU’ commands along with some other stuff you’ll need.
• Read through BDS Chapter 3
• Work through BDS Chapter 3, following along in your own terminal
• Assignment 3 - Running shell scripts, redirecting, pipes, background processes
• Read/watch ALL of the resources below. Be able to write a for-loop.

Resources

• Basic Unix Commands
• Very Useful Tutorial
• On the Value of Command-Line Bullshittery
• On the Annoyance of Command-Line Bullshittery
• So is this stuff even useful for bioinformatics? YES!
• Video walkthroughs of some command line stuff:
  • Part 1 - first commands
  • Part 2 - pipes and wildcards
  • Part 3 - relative filepaths
  • Command line program flags/parameters
  • How to avoid two potentially dangerous command line errors
  • For-loops video walkthrough in BASH
• Bonus tips:
  • BASH aliases video
  • BASH alias examples
• Recorded Lesson (old) Part 1
• Recorded Lesson (old) Part 2 (mostly trying to get things to work on Windows)

Practice

• Practice tasks

For your consideration:

• “This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.” –Doug McIlroy

Back to top of page

Working with Sequence Data

Topics

• fasta and fastq file formats | using existing tools to work with sequence data

Assignments:

• Read through BDS Chapter 10 at least once
• Don’t work through the examples yet (we can return to them once we have more skills)
• Assignment 4 - converting between formats, inspecting and trimming reads, using pre-made command-line tools

Resources

• seqtk
• fasta and fastq formats
• VIDEO on fasta, fastq, and gzip compression
• phred scores
• fasta format for peptide sequences
• Recorded Lesson (old) Part 1
• Recorded Lesson (old) Part 2
• Recorded Lesson (old) Part 3

Practice

• How many sequences are stored (in total) in the fastq files associated with Assignment_4?

• How many sequences end with the seqeunce “AT” in each fastq file?

• Which fastq file associated with Assignment_4 contains the following sequence:

    CCTTCATGCTGTCCTGCAATTACGATAGCATTTCTTTGACGACGAC

For your consideration:

• “Treat data as read-only.” –Vince Buffalo
• Never directly edit any fasta or fastq file! If you have to make edits, redirect them to a new version of the raw file.

Back to top of page

Unix Data Tools

Topics:

• Regular expressions (regex)
• sed, grep
• Chaining together links in a ‘pipeline’
• Intro to process substitution (if we have time)

Assignments:

• Work through BDS Chapter 7
• Spend 3 hours practicing everything we’ve done so far
• Try putting things together in original ways to get new insights on sequnce data
• Ask silly questions like: “What are the counts of Cytosine bases in all seqs that contain the pattern ‘GGCCG’?”

After playing around a bit, I came up with the following for that silly question:

cat Assignment_3_Combined_Files.fasta | grep -B 1 "GGCCG"| seqtk comp | cut -f3

• Playing around like this in freeform is the best way to build your skills between projects.
• Your practice tasks (below) will get you started

Resources

• Introduction to regular expressions video
• Regular expression tester … handy tool
• sed video playlist Definitely worth your time!

Practice

• Practice tasks

Back to top of page

Unix Data Tools, Continued

Topics:

• More handy shell programs: cut, paste, sort, uniq, tr, rename, tee, xargs, awk
• Manipulating text data from one format to another

Assignments:

• Continue working through BDS Chapter 7
• Assignment 5 - using ‘awk’ & ‘process substitution’ to interrogate a table
• Assignment 6 - convert between tabular and fasta formatted data | process/command substitution | advanced paste

Resources

• “Process substitution” vs “command substitution” VIDEO
• Using paste to build fasta from tsv video

Practice

• Here’s an awful-looking one-line command that prints out the phylum from each line of Chapter_7_Practice_File_2.txt along with a number sequence next to it showing which line of the file it came from.

• It uses both process and command substitution, but essentially, it’s just the paste command pasting together the phylum in the first field and the numbers 1-34 in the second field

• I want you to break it apart, looking at each component and understand why it works!

    paste <(cat Chapter_7_Practice_File_2.txt | cut -d ";" -f 2) <(seq $(wc -l Chapter_7_Practice_File_2.txt | cut -d " " -f 1))

• If you wanted to use process substitution again to extend this whole command in order to add a header to the output, what would you do? (i.e., add a first row that is “PHYLUM LINE_NUMBER”)

Back to top of page