BYU

Office of Research Computing

Web Services API

The Office of Research Computing's Web Services API allows users to create tools such as a website to remotely manage their files and jobs on FSL systems.

Possible benefits of using the API

  • Lower the barrier to entry for researchers (no need to learn Linux or the batch submission system).
  • Simplify support requirements for CSRs or "the Linux guy/gal" in the group.
  • Provide a simple method to visually manage data through a web browser.
  • Allow the automation of routine tasks.
  • Integrate with existing interfaces that departments already use.

If you have successfully developed an application or website using this API, please let us know about it and how it is used.

Example Use

A professor launches a lot of jobs on FSL systems but constantly has to train new research assistants on the usage of Linux and the batch submission system. Due to the high turnover and constant training needs, the professor decides to assign a research assistant to create a website so researchers can simply upload their job scripts and submit them from a web browser. The researchers can now launch and monitor their jobs from a website and then download the results when they are finished. This can significantly decrease initial training time for researchers with limited Linux experience.

Download

We provide client reference implementation modules in Perl and PHP although you are free to create others. Each version comes with example scripts for using almost every API method call. Please read the included README file for installation instructions.

API Keys

All implementations must have a valid API key. In the future there may be an easier way to create one, but for now you must request a key through the support ticket tracker. The request can be very brief and should contain a short description of what you plan on creating.

The API key is specific to the developer's implementation and should not be shared. If a single developer needs to create multiple websites, separate API keys should be requested for each.

Authentication

Please note that EACH user still needs their own Office of Research Computing account and that accounts should not be shared between users.

This is an in-depth section. It is actually not very complicated but there are a number of steps involved. The most concise summary of what a developer's code needs to do is in the last list of this section.

Most methods require a user to authenticate first. This is a multi-step process that should be understood before using the API. For the example below, there are several parties involved: (1) The FSL API server api.fsl.byu.edu, (2) marylou.byu.edu, (3) the "department" web server that is implementing the API, and (4) the user's web browser. The steps are as follows:

  1. Web browser views department's website.
  2. Department website makes an API call that requires a valid session, such as fsl.file.list or fsl.session.requirelogin. It includes the current session key in the call if a session is established.
  3. The API server checks the session key if it was passed and calls the method if valid.
  4. If the session key is invalid, expired, or missing the API server will send back a special response to the department web server with the method name set to "fsl.control.loginredirect" and the " login_page" field set to a URL.
  5. The department web server then redirects the user's browser to the URL that was returned in the "login_page" field. The URL the browser is directed to is on an Office of Research Computing web server (marylou.byu.edu).
  6. The user will be presented with the Office of Research Computing login page and should enter his credentials.
  7. The Office of Research Computing web server will then redirect the user's web browser back to a page that the API key owner specifies in their preferences page. The setting is the Callback URL. The URL will have the new sessionid appended. For example, if the Callback URL is "https://some.department.byu.edu/~researcherbob/fslapi/index.php" the browser will be redirected to "https://some.deparment.byu.edu/~researcherbob/fslapi/index.php?sessionid=0123456789"
  8. The department web server will then use the sessionid from the query string (in this case "0123456789") and store it somewhere such as in a cookie. Future API calls should now send this sessionid for that user's browser session.

A diagram of these steps is also available.

That may seem very complicated because there are a lot of steps but it is a very industry-standard, straightforward way of handling authentication. FSL reference implementations that you can download already implement all of this. In reality all that the code on a department web server ends up doing is the following:

  1. Check if the "method" value in a method call response is "fsl.control.loginredirect".
  2. If it is, then redirect the user's browser to the URL specified in the "login_page" field from the response.
  3. Have the page specfied by the API key owners's Callback URL store the "sessionid" query parameter in a cookie or something like PHP's $_SESSION.
  4. Send that sessionid in future API calls.

FSL reference implementations already take care of this for you. The only thing you need to do is keep the Callback URL in your preferences page pointed to the right URL for your script.

If you want to test if a user is currently logged in without redirecting them to the login page if they aren't logged in, you can call fsl_session_isvalid() and check the true/false value. If you want to force the user to log in if they aren't already, you can call fsl_session_requirelogin(). This will return true if they are logged in and will redirect them to the login page if not. Any other method that requires a login (virtually all methods) will achieve the same result.

Sessions

With the exception of a few specific methods such as methods.list, a session is required in order to use the API. A session is established when the user is redirected to the FSL website to log in and then sent back to the API's Callback URL. There is a session key that is then created and associated with the user. This key is then passed between the involved parties to maintain the session. There is a timeout whose value can be read from the seconds_left_in_session field. The timeout is reset to the maximum every time a call is made that requires a session.

Non-web usage

It is also possible to use the API in scripts. One of the problems with this is that it makes web-based authentication impossible unless work-arounds are used. One way is to copy and paste the sessionid portion of the redirect URL into the script after logging in. The session timeout is reset each time a call is made that requires a session, thus extending the life of the session. If the script is run regularly through cron, this will effectively make it so the session never expires.

If desired, you may request to be given a permanent session key that never expires. You may request one by opening up a ticket and briefly justifying why you would like to have a non-expiring key. The session should only be used by one user, you. If session key goes unused for a long period of time (several weeks), it may be removed and you must request a new one.

Upload and Download Limits

When reading this section keep in mind that there are two web servers involved: the Office of Research Computing API server and your webserver. This means there are two servers that have different configurations and limits that must be considered.

fsl_file_get has a somewhat limited maximum download size. This is due to the fact that the entire file must be loaded entirely into memory on the API server in order to be serialized and passed back. Your web server must then load the file into memory so it can be unserialized. There are limits on the amount of memory that can be consumed by any one web server process, so this will effectively limit the maximum download size.

However, there is a workaround implemented in the API. When doing a fsl_file_get, the parameter "noencapsulation=1" can be passed (and is by default in the FSL reference implementations) that tells the API server to write the file back directly without serializing it. Your code can then do a loop that reads a certain number of bytes and appends it to a file, writes it out directly to the client, or even add it to a string in memory. This is very efficient but the one caveat is that error handling isn't quite as robust.

fsl_file_put is limited and there are no workarounds. You can call the fsl_file_max_upload_bytes function to see what the upload limit is on the FSL API server. Keep in mind that your own server may have a separate upload limit when a user uploads a file from their browser.

Proxy Problems

Some users have to run their code on a web server that is behind a proxy. We know that this can be an issue but are only aware of one user who has had that problem. If so, please talk to your department CSR for proxy configuration information. Then you will need to add a line to your code such as $ENV{HTTPS_PROXY} = "http://proxy.somedepartment.byu.edu:8080"; if you are using Perl. Please contact us for help.

Examples

The reference implementations that can be downloaded above have the best examples. There are simple examples of using almost all methods in the download files. For the most part, you should be able to drop the files into a directory on your web server, point your Callback URL to theindex.php or index.pl file and have it work.

Listing available methods in Perl

#!/usr/bin/perl

# Change to directory that contains script so that FSLClientAPI.pm
# is in the working directory.  May be necessary when using mod_perl
chdir File::Basename::dirname __FILE__;
require FSLClientAPI;

use strict;
use CGI;

my $cgi = new CGI;

my $session_id = $cgi->cookie("sessionid");
my $fslapi = new FSLClientAPI($session_id);

my $response = $fslapi->methods_list;

$cgi->header;

print q{
<html>
<head>
    <title>FSL API | methods.list</title>
</head>
<body>
    <h1>Methods List</h1>
    <ul>
};

foreach my $i (@$response) {
    print "\t\t<li>$i</li>\n";
}

print q{
    </ul>

</body>
</html>
};
Last changed on Wed May 1 13:51:56 2019