Mitigations against uncontrolled web crawlers and bots

The server running matrixscience.com is being overwhelmed by millions of HTTP requests from distributed botnets, which amounts to a distributed denial of service (DDoS) attack. Due to the high load, matrixscience.com has been periodically unavailable this week as the server does not have enough hardware resources.

We are now forced to restrict access to certain pages of dynamic content:

• Protein Family Summary reports of example searches shipped with Mascot
• Protein View reports of example searches shipped with Mascot
• Peptide View reports of example searches shipped with Mascot

If you submit a database search in the free Mascot service, your search results are not restricted. The above restrictions are designed to only impact bot activity.

The full explanation is below, and if you have any questions, please contact us by email.

Why are these pages unavailable?

We use the Robots Exclusion Protocol (robots.txt) to indicate areas of the website that should not be accessed by automated bots. This is a long-standing mechanism to mitigate server overload, widely recognised as an industry standard.

Since 2024, the world’s largest tech companies have been using large-scale web crawlers to repeatedly and regularly hoover up every page on the public Internet, including websites operated by small companies like Matrix Science. Unlike crawlers operated by search engine companies, these new crawlers (or bots) have been designed to deliberately ignore robots.txt. Because the bots follow every link, they are causing massive processing demand on this server.

Worse, the bots have been stripped of any identifying features and they are operated from botnets of 10,000 or more IP addresses. There is no way to automatically detect or block them. So, we have no choice but to make some dynamic pages unavailable to everyone.

Who is responsible?

We are being hit by distributed botnets run by the world’s largest tech companies. However, because the bots have been stripped of any identifying information, we don’t know who is operating them.

What can be done?

The best thing you can do is contact your elected representative. Lobby them to force governments to regulate tech companies. Lobby them to force these companies to honour the Robots Exclusion Protocol (robots.txt) with any automated web crawler or bot they create or operate.

Change of address for UK office

The physical office of Matrix Science Ltd has moved one block up the road. The new address is:

Matrix Science Ltd
83 Baker Street
London, W1U 6AG
United Kingdom

Matrix Science Ltd has been based in London, UK since 1998; the old address was 64 Baker Street. There is no other change to our contact details, and our phone numbers and email addresses remain the same.

New Mascot distributor in India: BioInnovations

We are pleased to announce a new distributor for Matrix Science products!

Customers in India and neighbouring countries can now buy Mascot Server through BioInnovations, who provide instruments, software, services and consumables for life sciences and analytical and clinical labs. In addition to India, the distribution agreement covers Bangladesh, Bhutan, Maldives, Nepal and Sri Lanka.

BioInnovations are a Bruker channel partner and provide MALDI and timsTOF instruments for proteomics. Bruker BioTools and BioPharma Compass include native integration with Mascot Server for trusted protein identification by peptide mass fingerprinting and LC-MS/MS.

New pricing tiers for Mascot Server

We are pleased to announce a simplification to Mascot Server licensing tiers, which doubles the number of cores at the entry level.

The old licensing tiers were based on the number of CPU units between 1 and 32 CPU. The new licensing tiers are:

• Small: up to 8 cores (16 threads)
• Medium: up to 16 cores (32 threads)
• Large: up to 128 cores (256 threads)

We are also simplifying and reducing the cost of Mascot Server version updates. The current major version is 3.x, and the previous major version was 2.x. When updating from CPU-based licensing to Small/Medium/Large, the Mascot Server 2.x licence is treated as a 50% discount regardless of the number of licensed CPUs and regardless of the age of the licence.

Please contact us for pricing and quotes.

New help entry for Agilent instruments

Agilent have published an application note for recommended MassHunter Qualitative Analysis settings when searching DDA peak lists with Mascot Server. We’ve added a new help page for using Mascot with Agilent instruments, including information about using Mascot Distiller to process Agilent MassHunter files and a link to the Agilent application note.