Get more search speed with SplitNumberOfQueries
Mascot Server has several parameters and options that have an effect on database search duration. One of them is SplitNumberOfQueries, which specifies how many queries (MS/MS spectra) are searched at a time. The default is 1000 queries, which is reasonable, but like all defaults, it isn’t optimal. On most hardware, increasing it to 2000 will yield an immediate 10-20% performance boost.
What the setting does
When you submit peak lists for a database search, Mascot sorts the spectra by peptide mass and splits them into chunks. There are three reasons for chunking. Firstly, it reduces peak RAM usage during the search. Secondly, limiting the chunk size benefits locality of reference. Most computer programs repeatedly access the same set of memory locations. When this happens, the processor is able to provide very efficient memory caching, which generally improves throughput. Thirdly, in cluster mode, Mascot sends chunks of the same size to each cluster node. Because the search space is identical on each node, chunking provides effective load balancing.
What you might hope to get from increasing SplitNumberOfQueries, and which is often achieved in practice, is reducing the search duration by trading for higher RAM usage. We typically see 10-20% faster searches when doubling SplitNumberOfQueries, while RAM usage approximately doubles.
There is a second, related setting. Chunk size is also limited to at most SplitDataFileSize bytes, by default 107 or around 10MB, so the effective limit is the lower of SplitNumberOfQueries queries or SpliDataFileSize bytes. If your peak lists have little noise, then SplitDataFileSize is hardly ever in effect. Conversely, if your spectra have thousands of peaks, it’s quite likely that SplitDataFileSize will limit the chunk size before SplitNumberOfQueries. It’s typically best to change SplitNumberOfQueries and SplitDataFileSize in tandem.
Recommended procedure
Finding the optimal value is necessarily an experimental, iterative process. There are simply too many factors to guess what might be optimal for any given hardware, software and data combination.
Start by choosing a suitable search for benchmarking. This should be a search that currently takes 5-10 minutes and which represents a typical search in your environment. For example, if you’re doing a lot of iTRAQ, then choose a recent iTRAQ search. If you’re doing many error tolerant searches, choose a recent ET search.
If you don’t have a suitable search, try benchmark_PXD003791.zip from Benchmarking your Mascot Server. This is a straightforward search against SwissProt using a minimal number of variable modifications and has about 100k queries.
- Run the search and monitor memory usage. On Windows, open Task Manager, switch to Details, and observe “Memory (active private working set)” of nph-mascot.exe. On Linux, run ‘top’ and observe the %MEM usage.
- Make a note of the peak memory usage. The search duration can be found from Database Status: click on the database (e.g. SwissProt), which shows the most recent jobs, and click on the job number. The total time is the “Whole process time”, in seconds.
- Run the search a total of three times to get some idea about variation in duration. If there is a lot of variation, run it a couple more times. (The RAM usage pattern should have little variation.)
- Increase the value of SplitNumberOfQueries and SplitDataFileSize in mascot.dat options section. For example, if they were 1000 and 10000000, respectively, set to 1500 and 15000000.
- Run the benchmark search three times. Observe both memory usage and the search duration.
- Continue increasing the option values to 2000, 2500, 3000 and so on. Repeat the search three times for each value until you hit a sweet spot.
The sweet spot depends how much RAM you have and whether you are seeing any effect on search duration. There are diminishing returns in setting SplitNumberOfQueries too high, because you will eventually hit a bottleneck somewhere — disk speed, RAM speed or CPU speed. If Mascot is being used by multiple people, you also need to take care that a single search won’t gobble up all available RAM, leaving none for other concurrent searches.
Example benchmarks
The table below illustrates the effect. The benchmarks were run on a couple Matrix Science test systems using benchmark_PXD003791.dat. Note that these are illustrative only, because the numbers depend strongly on your hardware, Mascot licence size, what else is running at the same time, your typical search space and data file size, and so on. It’s best to repeat the benchmarking on your own system.
| System | Processor | Mascot | Licence | SplitNumber OfQueries |
SplitData FileSize |
Average duration | RAM usage |
|---|---|---|---|---|---|---|---|
| Laptop (2022) | Intel Core i5 | 2.8 | 1-CPU | 1000 | 10000000 | 702s | 300MB |
| 1500 | 15000000 | 645s | 520MB | ||||
| Workstation (2021) | AMD Ryzen 9 | 2.8 | 4-CPU | 1000 | 10000000 | 285s | 1151MB |
| 2000 | 20000000 | 224s | 2067MB | ||||
The laptop is a mid-range Intel Core i5 laptop with an NVMe (SSD) disk. The processor is 12th generation Intel, with 4 “P” cores and 8 “E” cores. The Mascot 1-CPU licence uses only the “P” cores (8 hardware threads). The sweet spot was found at SplitNumberOfQueries 1500, giving 10% speed boost. Increasing the setting to 2000 made little further improvement to the average search duration.
The workstation has a single AMD Ryzen 9 processor from 2021, with Mascot installed on a RAID array. The Mascot licence is 4-CPU and uses all 16 processor cores (32 hardware threads). SplitNumberOfQueries 2000 gives 21% speed boost. Mascot uses approximately 4x the amount of RAM than on the laptop, because 4x more hardware threads are being used and each thread needs roughly the same amount of memory.
We recently heard from a customer who ran the same benchmark search. At SplitNumberOfQueries 1000, the average duration was 201s, and at 2000, it was 146s, giving 27% improvement. Do give it a go and please leave a comment about what you found!
Keywords: benchmark, configuration files, pc hardware, sysadmin
Another customer contacted us by e-mail. He runs Mascot in a VM, where the host server has two Intel Xeon Gold processors. The above benchmark search at the default SplitNumberOfQueries (1000) and SplitDataFileSize (10000000) took 560 seconds. An issue with this VM is that CPU usage never reached 100% during a search.
After trying a few different values, the optimal setting is SplitNumberOfQueries 4000 and SplitDataFileSize 40000000. The benchmark search now takes just 299 seconds, nearly twice as fast. In addition to better speed, changing the setting also fixed the lower than expected CPU usage during the search.