Getting Started - for users new to High Performance Computing
Instructions for obtaining an account on Peregrine are on the
Access to Peregrine is available only from within the NREL firewall, by the Secure Shell (SSH) protocol 2.
To access Peregrine from outside NREL, first create a VPN to NREL.
A big difference between your laptop or personal workstation and Peregrine is that Peregrine is used with a command line interface via a terminal application. If you are using a Mac, your computer already has a terminal application and SSH.
If you are connecting using a Windows systems, a terminal package such as PuTTY, which supports SSH, needs to be installed. You may download PuTTY on their website http://www.putty.org/. Configuring PuTTY to connect to Peregrine is only necessary the first time Peregrine is accessed. You may then log into Peregrine using your user ID and password. Please refer to for instructions of configuring PuTTY to connect to Peregrine.
To access Peregrine from a Mac OS X system at NREL, start the terminal application and enter ssh -Y firstname.lastname@example.org, where username is your NREL user id. Remember to include return at the end of this command.
Once you have entered your password you are logged in to Peregrine and all subsequent commands you type in your terminal application, whether on Windows or Mac, are commands on Peregrine.
In the examples on this and many other web pages, we represent the prompt by the symbol $. On Peregrine, the prompt will include the hostname of the node you have logged into, such as [username@login4 ~]$ . When typing the commands shown in this and other pages on this web site, you do not type the $ symbol. In addition, all commands you type need to end by typing return or enter, which tells the system that you are done typing. In places where you should insert information, we usually represent that information by enclosing it in brackets < >. For example:
$ ssh -Y <username>@peregrine.nrel.gov
When the ssh command completes you will be logged in to one of the login nodes and you will see a prompt that looks like
Sample first Python Program
If you are new to programming, you may want to learn Python. Rather than using a compiler to convert your code to commands the computer can understand, it uses an interpreter to convert each line as the program runs. Using a text editor, such as nano, vi or emacs, create a file called hello.py which contains the single line
print("Hello, world from Python!")
If you don't already know how to use a text editor, we suggest you learn to use nano. To create the file named hello.py enter the following command on the login node on Peregrine:
$ nano hello.py
This will start the nano program, which will allow you to type text into the file. Remember that you don't type the $ sign and you need to hit return or enter at the end of the command.
Commands to tell nano what to do are located at the bottom of your screen. The ^ symbol means to hold the Control key while you type the letter, so ^O (pronounced "control-o") is entered by holding the control key and the o key at the same time.
Enter the following text, being careful to include the spaces and symbols as shown:
print("Hello, world from Python!")
Save this file by entering ^O. Exit nano by entering ^X.
Running your code on Peregrine
Besides the login nodes, Peregrine has a large number of other nodes that are used to run programs. Please do not run programs directly on the login nodes since. Each of the other nodes can only be used by one user at a time. Because Peregrine is shared by many users, it uses software to control which user is using each node. To ask Peregrine to run your program, you give the instructions for how to run it to an application called Torque which keeps track of which nodes are already being used and which may be available to run your job. The set of instructions is put in a file called a "job script".
The command used to submit the job script to be run on the system is qsub.
Besides the name and location of the program, the system needs some additional information. This information can either be provided as options to the qsub command or you can save typing by including it in the script itself (which can be re-used many times). One of the critical pieces of information that is needed is the project handle for your allocation. This is included with the -A option, for example, the command
$ qsub -A <project-handle> <job_script>
provides both the project handle and the name of the job script.
To specify this information inside the script, the options are placed on lines that start with #PBS. To specify the project handle, one of the lines near the beginning of your script will contain #PBS -A project-handle. There are other options to specify resource limits such as the maximum time the job will take, the number of nodes needed, the type of nodes the job should run on, etc. Options that are provided to the qsub command take precedence over the same options that are inside the script.
To run the program you created, we will need to create a job script file. To do this, start your text editor again and give it the name of the file the script will be in. For example, type
$ nano job.sh
To run the sample code, type the following lines into the file using nano.
#PBS -l walltime=3600 # WALLTIME LIMIT
#PBS -N serialjob # Name of job
Save your text (^O) and exit the editor (^X). This created the file called job.sh.
What does this cryptic text mean?
- The first line of the file tells the system that the commands that follow are "bash" commands.
- The next line tells torque that your job will have a maximum runtime of 3600 seconds. If your program takes longer than 3600 seconds, the system will kill your job when you reach this limit.
- The third line gives a name (serialjob) to your job.
- The forth line tells the system to change to the directory (folder) where your files are located.
- The fifth line tells the system to run the program in the file hello.py.
- The text after the second # symbol on lines 2 and 3 are comments to remind you what information the line contains. Comments are ignored when processing the file.
To submit the job, type
$ qsub -A <CSC001> job.sh
but replace <CSC001> with the project handle of an allocation you have access to. The number that appears on your screen is the "job id". This number is handy for checking on the status of the job or reporting a problem. Did the job run and complete? Is it sitting in a queue waiting for nodes to become available? To find out what is going on, you can use the "checkjob" command. Type
$ checkjob <JobID>
An example of output from the checkjob command is shown below:
[icarpent@login2 ~/symm]$ checkjob 1104184
Creds: user:icarpent group:icarpent-upg account:CSC000 class:phi qos:default
WallTime: 00:00:00 of 00:40:00
SubmitTime: Tue Mar 31 09:44:51
(Time Queued Total: 00:00:24 Eligible: 00:00:24)
StartTime: Tue Mar 31 09:45:15
Total Requested Tasks: 1
Req TaskCount: 1 Partition: torque
Opsys: --- Arch: --- Features: phi
Notification Events: JobFail
Reservation '1104184' (00:00:00 -> 00:40:00 Duration: 00:40:00)
We can see that this job is running on a node named "n1575".
To see the output of your programs, look for the output file from the job. The file name will start with the name assigned in the script (serialjob in this case) followed by .o and the JobID. Type ls to see what files exist in your directory (folder).
[jstickel2@login4 ~]$ ls
hello job2.sh serialjob.e1097699 serialjob.o1097699
hello2 job3.sh serialjob.e1097700 serialjob.o1097700
hello.F90 job.sh serialjob.e1098464 serialjob.o1098464
To see what the output file contains, you can use the cat or more command:
[jstickel2@login4 ~]$ cat serialjob.o1097700
Hello, world from Python!
Examples of hello, world programs in Fortran and are also provided.