Disaster recovery

There are a number of potential issues that can cause your Mascot Server or other computer infrastructure to stop working. In the worst case, you may need to do a full recovery from backups. This can be an involved procedure, but it is straightforward as long as you have backed up key configuration and data files.

For disk or storage failure, replacing the device will allow you to reinstall Mascot Server and get back up and running quickly. A related issue is hardware failure like the computer’s motherboard, or more problematic for workstations, the RAID controller. If it is a “simple” failure like the motherboard, the hard disks can be moved to another computer. On a computer that suffers RAID controller failure, you will need identical or compatible hardware in order to access the data again. For older systems this can be difficult or expensive to obtain. If you can not source suitable replacement hardware, you may be able to use software like RAID Recovery to read the disks and you’ll need to contact your IT department for further help.

Yet another risk is ransomware and computer viruses. In 2017, ransomware named NotPetya infected tens of thousands of computers around the world including Merck. It was impossible to pay the ransom and NotPetya encrypted hard disks, effectively destroying any data they contained. To an individual lab, these can be treated as multiple disk failures and as long as backups were stored separately, recovery is possible.

What to back up?

We always recommend backing up your raw data files as they are often irreplaceable. One could just back up the raw data and recreate the results as needed.  However, if a lot of data needs to be reprocessed, it might be impossible to keep up with current data production while rerunning all the old searches. The amount of time and work that goes into the data analysis, especially for quantitative data sets, makes it wiser to back up the results, especially as storage is cheap.

In the analysis pipeline, the raw data is converted to the peak lists and then searched on Mascot Server. If you are using Mascot Distiller for the peak picking, we recommend backing up the Distiller rov files as they contain the peak picking parameters, peak lists and, if you have the Distiller Search toolbox, the search results too.  The MGF files only contain the peak lists, so there is no need to save them as they can always be exported from the Mascot results.

For Mascot Daemon, we recommend backing up the search parameter files. The default location is:

C:\ProgramData\Matrix Science\Mascot Daemon\parameters

If Daemon is configured to use Distiller as a data import filter, the rov files are saved in the Daemon MGF directory, whose default location is:

C:\ProgramData\Matrix Science\Mascot Daemon\MGF

We also recommend backing up the Mascot Daemon task database. If you are using the default VistaDB format it will be under:

C:\ProgramData\Matrix Science\Mascot Daemon\taskdb.vdb5

Note that the “ProgramData” directory is hidden by default. You have to type the path into the Windows Explorer address bar.

If you are using one of the alternative database formats like an SQL database, you can either do the back up through the SQL management software or, far easier, directly from Mascot Daemon. Go to the Edit->Preferences dialog and click on the Backup button:

 

With Distiller, we recommend backing up the processing options:

C:\ProgramData\Matrix Science\Mascot Distiller\processing options

And the licence file, which will be in one of these locations:

C:\Program Files\Common Files\Matrix Science\licdb
C:\Program Files (x86)\Common Files\Matrix Science\licdb    (64-bit Windows)
C:\ProgramData\Matrix Science\licdb                         (Vista or later)

Full details of what to back up on the Mascot Server was covered in an older article. The only thing I have to add is that on a Linux system, you may want to back up the web server configuration, config/apache.conf or http.conf.

Recovery

The worst has happened, your Mascot Server is down and maybe other computer systems too. Hopefully you have followed the advice in this and the previous article and have all your data in a secure location. Recovery starts from the Mascot Server and works backwards to Mascot Distiller and Daemon.  If you are currently undergoing a computer disaster and have not made backups as suggested, reading this article right now will probably not help too much, sorry. 

Instructions for moving Mascot Server to a new computer also work for disaster recovery. As you are going to have to reinstall Mascot Server, you can follow the instructions in section 3. If you are reinstalling Mascot Server on the original computer, you should be able to reuse the licence file.  Otherwise contact us for a new product key. In the case of hardware failure, we can also supply temporary product keys so that you can recover to a different computer until the main server is repaired.

One of the issues during the recovery process is the time taken to restore the large amount of data associated with Mascot Server. It may be worthwhile to use a special file copying program to speed up the process. We’ve had success with TeraCopy for Windows, but your IT department may have some specialist solutions as well.

Mascot Distiller recovery is simply a case of running the installer, activating the software and copying back the processing options files. If the computer is unchanged, the old licence file will continue to work. If the hard disk failed and has been replaced or there have been other changes to the computer, you will need to contact us for a new product key.

The last step is to recover Mascot Daemon. The installer is available from your local Mascot Server home page and there are no product keys or serial numbers, so this is a quick process. Install Mascot Daemon and copy the processing options files back to the computer. If you are using the default VistaDB task database, close Mascot Daemon and the Daemon engine in the notification part of the Windows task bar. Rename the taskdb.vdb5 database included with Daemon and copy over your backed up version. Restart Daemon and all your old tasks should be present. If you are using an SQL database, you will need to restore the database either through the SQL management software or from the Mascot Daemon->Edit->Preferences dialog, depending on which method you used for backing up. 

Occasionally things can be a little more complicated or you may get stuck at a certain step, but as long as you have backed up the required files, you will be able to recover Mascot Server, Distiller and Daemon. Please contact us if you need further help.

Keywords: backup, configuration files, Mascot Daemon, Mascot Distiller, sysadmin