Table of Contents

Background
Unix Tutorial
Data Analysis Test Simulations
Post Processing
Non-spinning Simulations

Background

This research project will be related to the LIGO experiment.
LIGO is an international effort to detect gravitational waves.
The project we have in mind will be a significant contribution to LIGO science, so you should be psyched.

Some very brief introductory PowerPoint/Keynote/PDF slides are available here.

Some useful background reading can be found here.

In particular, sections
6 Black holes in aLIGO: the observational payoff! co-authored by our own Dr. Ben Farr
10 Astrophysics of black hole
14 Data analysis: How to find a black hole

Unix

Our simulations will be run on the high performance computing cluster "Quest" at Northwestern.
You speak to Quest in Unix which is a very simple operating system.
Here is a Unix tutorial which you must be familiar with before your official start date.
At the end of the summer add "Proficient in Unix" to your resume.

Mac and Linux operating systems have Unix capabilities built in.
Unix commands are entered in a "Terminal" window.
On a Mac the Terminal app can be bound at
Applications/Utilities/Terminal.app

For Windows users it will be more complicated.
You will first have to install PuTTY which you use to access Unix-speaking computers like Quest.
Using PuTTY you will be able to log in to Quest and learn your way around the Unix commands.
This is a notorious task so if you plan to use your own Windows laptop get this done well before your start date.
We will try to help you troubleshoot as much as possible.

One of our team has some experience getting Windows machines to talk to Quest and has this to offer:

For those of you who may have Windows 8/8.1,
there's an app named Terminal RT available for download in the Windows Store.
Here's a link to a page that will open the store directly for you to download and install the app.

I've attached a basic walkthrough showing how you will need to set up the terminal in order to access quest.
Just open the zip file, extract it, and then you should just be able to double-click it.
If that doesn't work, right-click, select "Open With" and then open it with an internet browser.

Let me know if you have any questions or issues!

Ben Sandeen
A limited number of iMac works stations will be available during the summer.
You can get an account if you do not have your own laptop or would prefer to not use it.

To log on to Quest with a Terminal (PuTTY will have different directions):
> ssh [NetID]@quest.it.northwestern.edu # log in to Quest

Here is a page full of information regarding Quest (Northwestern's supercomputer)
Especially useful is the Running and Managing a Job link under the "Getting Started" section.
This page contains information about managing your runs, and particularly useful is the ability to delete any runs that shouldn't be running.

Once logged in:
> cd /projects/b1011/spinning_runs # go to project directory
> mkdir [NetID] # make a directory named with your NetID (only the first time you log in)
> cd [NetID] # navigate to your NetID directory. This is where you will submit your runs

Data Analysis Test Simulations

We have set up a simple test simulation on Quest.

Once you have set up your directory in spinning_runs go to it and do the following:
> mkdir test # create a test directory
> cd test # enter the test directory
> cp /projects/b1011/spinning_runs/scripts/test.sub . # copy test.sub into your directory. Don't forget the "."

Now you should have the test.sub file in your directory. You can check by running ls at the command line.
This file is used to submit a request to Quest to run the MCMC code. You have to edit the file so that the request is coming from your NetID.

To edit the file we will use a program called VI -- a notoriously un-user-friendly text editor
We will not need to do much with VI so it should be relatively painless
All you will need to know can be found in this simple tutorial

Once you have completed the VI tutorial, open test.sub and at line 9 you should see
#MSUB -d /projects/b1011/spinning_runs/NETID/test/

Change NETID to your NetID, then save and quit VI.

You are now ready to submit your test job using the msub command:
> msub test.sub # submits your job to Quest to be run

A seemingly random 8-digit number should be the only feedback you get.

You can test that it worked and check on the status of your job using the qstat command:
> qstat
Job id                    Name             User            Time Use S Queue
------------------------- ---------------- --------------- -------- - -----
14549177.qsched02         TestRun          tbl987          06:32:10 R ligo

If the S column is Q it means your job is waiting to run.
If it is R that means your job is running! Woo Hoo!
You can see the list of output files by using ls

Your code should finish in a few hours. When qstat returns nothing it is finished.
Now it is time to post process (i.e. make graphs)!

Post Processing

To run the post-processings script:
> source /projects/b1011/spinning_runs/scripts/post.sh

This takes only about a minute to run and should produce a single file.
The name of the file will depend on the specifics of the run that you did, plus your NetID.
For this test example the file should be named TaylorF2_1_[NetID].png where
TaylorF2 is the waveform model we used, 1 denotes we did the first simulated signal
and then your NetID.

Viewing the results is best done by copying the plots from Quest to your computer. For Mac/Linux users we secure copy using scp:
> scp [NetID]@quest.it.northwestern.edu:/projects/b1011/spinning_runs/[NetID]/test/TaylorF2_1_[NetID].png .

This command will copy the file TaylorF2_1_[NetID].png located on Quest at /projects/b1011/spinning_runs/[NetID]/test/
to your current directory (which is what the "." after .png denotes -- you can give it the path to a different location if desired).

Non-spinning Simulations

First set up the directories where you will do these runs:
> mkdir /projects/b1011/spinning_runs/[NetID]/TaylorF2 # Make a direcotry for non-spinning runs using TaylorF2 waveforms
> cd /projects/b1011/spinning_runs/[NetID]/TaylorF2 # Go to that directory
> mkdir [EventNumber] # Make a directory for the simulation number(s) you are assigned
> cd [EventNumber] # Go to that directory

Now we have to set up Python (a programming language) and tell the computer where to find different scripts.
This only needs to be done once each time you log in to Quest.
> source /projects/b1011/spinning_runs/setup_paths.sh

And now we are ready to actually submit the job:
> python /projects/b1011/spinning_runs/scripts/lalinference_mcmc_submit.py --inj /projects/b1011/spinning_runs/injection_set.xml --event [EventNumber] --approx TaylorF2 --walltime 02:00:00:00 # Generate a submit script
> msub submit # Submit the job to Quest

Make an [EventNumber] directory for each job you want to run.
Remember the module load and source /projects/.... lines do not need to be run again unless you have re-logged-in to Quest.