Measuring Reaction Time of Abuse Desks
Published on 1st October 2018, 10:07:31 UTC
In March 2018, I've launched my most recent project called URLhaus. The goal of URLhaus is to collect and share URLs that are being used for distributing malware. In the first half year, 234 users have contributed and shared more than 50'000 malware distribution sites on URLhaus. That's amazing! But URLhaus does not only collect malware URLs: The project also reports active malware distribution sites to blacklist providers like Spamhaus, SURBL and Google Safe Browsing.
Abuse Reports
Since June 2018, URLhaus is also sending out automated abuse reports to the respective network owners by using Abuseix Abuse Contact DB. The chart below shows the number of active malware distributions sites and abuse reports sent out by URLhaus per day.
When URLhaus started to notify network owners about malware distribution sites in their network, I was pretty surprised to see how many abuse complaints bounced, receiving a Non-Delivery Report (NDR). It appears that it is not uncommon that RIR objects are poorly maintained, pointing to an non-existent (abuse) mailbox. But outdated RIR objects is just one problem I was confronted with when sending out abuse complaints. Here are some of the most common ones:
- Quota exceeded: While it is generally good news when an abuse mailbox exits, it doesn't mean that abuse reports are actually being handled at all. I've had various cases in the past few months where the abuse report could not be delivered because the abuse mailbox was full. This seems to be a good indicator that the network owner doesn't care about abuse problems in their network. Whether you should accept network traffic from / to networks that do not handle abuse reports at all is something that I leave up for discussion.
abuse@kvsolutions.nl LMTP error after RCPT TO:<abuse@kvsolutions.nl>: 552 5.2.2 <abuse@kvsolutions.nl> Quota exceeded (mailbox for user is full) - Spam filters: An abuse mailbox is the point of contact where internet users will report abuse originating from your network. This of course includes abuse reports related to spam. Unfortunately, some hosting providers put their abuse mailbox behind a spam filter. So guess what happens? The abuse report gets classified as spam by the spam filter and hence gets blocked. What I see mostly is that the spam filter detects a malicious URL in the email body (what a surprise!) and hence classifies it as spam. I do therefore believe that it is a bad idea to put an abuse mailbox behind a spam filter. Should your abuse mailbox get overwhelmed by spam, you may want to consider to implement DNSBL filters (checking the sending IP address against common DNSBLs). This should filter out most of the spam emails. However, from what you should definitely refrain, is implementing message content filtering on your abuse mailbox!
Your email to group teknis@qwords.co.id was rejected due to spam classification. The owner of the group can choose to enable message moderation instead of bouncing these emails. - Confirm your address: Some hosting providers came to the glorious idea to implementing sender email address verification. Once you send an abuse report to their designated abuse mailbox, they will send you an automated reply asking you to click on a link to confirm your email address (challenge response). While this might be a way to keep your mailbox free from spam, it doesn't work with automated abuse reports.
- You are not a customer: It seems that some hosting providers do not have a dedicated abuse desk, which means that abuse reports go to their support desk. While this is generally not a problem, it appears that some staff or ticketing systems only accept emails from email addresses that are known to their customer database. For me as sender of an abuse report this means: No way to report abuse!
Your email to our support system could not be accepted because it was not recognized as coming from an email address belonging to one of our customers. If you need assistance, please email from the address you registered with us that you use to login to our client area. - Web based abuse reporting: Very few hosting providers had another glorious idea; forcing the sender of the abuse report to fill out a web form. While this is might be a good way for the hosting provider to ask the reporter certain questions (and hence ensure that the abuse reports contains the necessary information needed to act up on it), it makes automated abuse reporting impossible.
In order for us to successfully process your domain inquiry, please go to https://abuse.web.com/ to complete your inquiry. - We need more information to better understand your issue: It appears that some abuse desk staff are not very well educated for dealing with abuse reports. At least that's what I conclude from the response I received and documented below. Dealing with abuse complaints can be challenging, so network owners should make sure that abuse reports are not being handled that by random support staff rather than people who have been trained to deal with such reports.
Thank you for contacting the Comcast Customer Security Assurance team. Unfortunately, we need more information to better understand your issue. Please select the option below that best describes your question. We’ve provided easy instructions to help us resolve your issue. - Subject overwriting: You as a reporter may send out hundreds of abuse reports per day. So it is essential that the recipient does not overwrite the mail subject when responding to your abuse report. Unfortunately, this seems to be a common practice across major hosting providers like GoDaddy or DreamHost. As they do not include the original abuse report in the email body either, there is not way to map their response to any abuse report you may have sent. Here are a few examples:
Subject: Incident 37181924: We received your feedback. Subject: Thanks for your report. Here are some extra resources. Subject: Thank you for reporting abuse on Google Cloud Platform Subject: Regarding your DreamHost Abuse message... Subject: Action Required Subject: Questions About Our Abuse Policies Subject: Support Ticket Not Opened Subject: Confirmation of Domain Abuse Inquiry
These are just a few examples with that you are confronted with when sending out abuse reports. There are many weird and special cases. What might help to solve at least some of these problems is using of the Abuse Reporting Format (ARF) (but more on this later).
Measuring Abuse Desks Average Reaction Time
In August 2018, I was asked which hosting providers react up on my abuse reports and within which time frame. So I decided to start measuring the reaction time of hosting provider's abuse desks (so called take down time). To do so, I measure the time between when the abuse complaint was sent to the hosting provider and when the reported content went offline. This gives some interesting insights, such as:
- How long malware distribution sites stay active in average once they have been reported to the hosting provider
- Which hosting providers react fast upon abuse reports
- Which hosting providers tend to react slowly to abuse reports
- Geographically distribution
Across 38'924 abuse reports URLhaus has sent out, the average reaction time is 3 days, 2 hours, 33 minutes. But are these good or bad news? I think: it could be better. The problem with most of the malware distribution sites is that they are associated with a Trojan called Emotet (aka Heodo) which is being spammed out with malicious office documents. It is hard to catch a day where there is no Emotet spam run. In case of Emotet and other malspam campaigns, it is important that the malware distribution sites are getting taken down in time, and with in time I mean hours and not days. Otherwise internet users will get infected even days after the spam run. This is bad.
Among the 600+ hosting providers that URLhaus has notified in the past two months, only 104 (or 16%) reacted within 6 hours in average. If we take a look at the number of hosting providers that reacted within the hour when they received the abuse report from URLhaus, we are down to 13 (or 2%).
The table blow shows the top 15 hosting providers with the fastest abuse desks (as per Sept 28th, 2018). It also includes the number of active (online) and offline malware distribution sites. Please consider that the accuracy of these statistics is +/- 1 hour.
| Rank | ASN | Country | Online | Offline | Average Reaction Time |
|---|---|---|---|---|---|
| 1 | AS62240 CLOUVIDER London, United Kingdom |  GB | 0 | 1 | 19 minutes |
| 2 | AS204983 CYBERFUSION |  NL | 0 | 1 | 23 minutes |
| 3 | AS22773 ASN-CXA-ALL-CCI-22773-RDC - Cox Communications Inc. ... |  US | 0 | 1 | 30 minutes |
| 4 | AS17917 QTLTELECOM-AS-AP Quadrant Televentures Limited |  IN | 0 | 4 | 32 minutes |
| 5 | AS3301 TELIANET-SWEDEN Telia Company |  SE | 0 | 2 | 33 minutes |
| 6 | AS45753 NETSEC-HK NETSEC | NL | 3 | 1 | 37 minutes |
| 7 | AS41357 UK-34SP-AS | GB | 0 | 2 | 38 minutes |
| 8 | AS23535 HOSTROCKET - HostRocket.com, Inc. | US | 1 | 5 | 39 minutes |
| 9 | AS197781 AVERS-AS |  UA | 0 | 1 | 44 minutes |
| 10 | AS197226 SPRINT-SDC |  PL | 15 | 1 | 46 minutes |
| 11 | AS42363 PHPNET-AS |  FR | 0 | 5 | 49 minutes |
| 12 | AS15830 TELECITY-LON |  NO | 0 | 2 | 52 minutes |
| 13 | AS6939 HURRICANE - Hurricane Electric LLC | US | 5 | 2 | 59 minutes |
| 14 | AS132680 NET1-AS-AP Net Virtue Pty Ltd |  AU | 0 | 9 | 1 hour, 5 minutes |
| 15 | AS39869 LIVENET- | PL | 0 | 1 | 1 hour, 12 minutes |
Looking at the bottom of the statistics, we will see that the worst hosting providers with the slowest reaction time had more than 10 days in average to react on the abuse report sent by URLhaus (as per Sept 28th, 2018). The table below also includes the number of active (online) and offline malware distribution sites. Please consider that the accuracy of these statistics is +/- 1 hour.
| Rank | ASN | Country | Online | Offline | Average Reaction Time |
|---|---|---|---|---|---|
| 610 | AS49505 SELECTEL |  RU | 8 | 2 | 11 days, 16 hours, 54 minutes |
| 611 | AS21565 AS21565 - Horry Telephone Cooperative, Inc. | US | 0 | 6 | 11 days, 20 hours, 10 minutes |
| 612 | AS42093 INTERRACKS-AS | NL | 2 | 6 | 11 days, 20 hours, 47 minutes |
| 613 | AS200960 PROFESIONALHOSTING |  ES | 11 | 1 | 12 days, 4 hours, 41 minutes |
| 614 | AS48368 COMTEH-RU-AS | RU | 0 | 6 | 12 days, 6 hours, 9 minutes |
| 615 | AS21396 NETCONNEX NetConnex Broadband Ltd. | GB | 0 | 1 | 12 days, 7 hours, 32 minutes |
| 616 | AS47288 FIXNET |  TR | 0 | 1 | 12 days, 10 hours, 55 minutes |
| 617 | AS199968 IWSNET | NL | 2 | 1 | 12 days, 18 hours, 38 minutes |
| 618 | AS41887 PROLOCATION Transit policy pref 100 | NL | 0 | 1 | 13 days, 3 hours, 14 minutes |
| 619 | AS9534 MAXIS-AS1-AP Binariang Berhad |  MY | 0 | 1 | 13 days, 18 hours, 54 minutes |
| 620 | AS55720 GIGABIT-MY Gigabit Hosting Sdn Bhd | MY | 11 | 2 | 14 days, 16 hours, 5 minutes |
| 621 | AS48096 ITGRAD | RU | 0 | 2 | 14 days, 18 hours, 36 minutes |
| 622 | AS9304 HUTCHISON-AS-AP HGC Global Communications Limit ... |  HK | 0 | 2 | 15 days, 7 hours, 56 minutes |
| 623 | AS45287 VARNION-AS-ID Varnion Technology Semesta, PT |  ID | 9 | 1 | 17 days, 12 hours, 37 minutes |
| 624 | AS133120 HNPL-AS-AP Hosted Network Pty. Ltd. | AU | 0 | 1 | 19 days, 20 hours, 42 minutes |
As shown on the table above, there are hosting providers who took more than 2 weeks to remove the malware from their hosting space. But it gets worse: there are malware distribution sites that are active for more than 3 months now and that do not show up in these statistics. The following table documents the top malware hosting networks that are hosting active malware content (counting online malware distribution sites only per 1st Oct 2018):
| Rank | ASN | Country | Average Reaction Time | Active Malware URLs |
|---|---|---|---|---|
| 1 | AS26496 AS-26496-GO-DADDY-COM-LLC - GoDaddy.com, LLC | US | 6 days, 5 hours, 54 minutes | 402 |
| 2 | AS14061 DIGITALOCEAN-ASN - DigitalOcean, LLC | US | 3 days, 14 hours, 27 minutes | 295 |
| 3 | AS48815 CRITICALCASE |  IT | 21 hours, 58 minutes | 151 |
| 4 | AS34619 CIZGI | TR | 4 days, 13 hours, 4 minutes | 134 |
| 5 | AS9891 CSLOX-IDC-AS-AP CS LOXINFO Public Company Limited. |  TH | 1 day, 18 hours, 17 minutes | 132 |
| 6 | AS32244 LIQUIDWEB - Liquid Web, L.L.C | US | 4 days, 22 hours, 14 minutes | 125 |
| 7 | AS13335 CLOUDFLARENET - Cloudflare, Inc. | US | 1 day, 22 hours, 54 minutes | 117 |
| 8 | AS16276 OVH | PL | 2 days, 8 hours, 0 minutes | 112 |
| 9 | AS4134 CHINANET-BACKBONE No.31,Jin-rong Street |  CN | 1 day, 10 hours, 29 minutes | 108 |
| 10 | AS25535 ASN-RUCENTER-HOSTING | RU | 5 days, 6 hours, 57 minutes | 105 |
| 11 | AS37963 CNNIC-ALIBABA-CN-NET-AP Hangzhou Alibaba Advertising Co.,Ltd. | CN | 1 day, 15 hours, 35 minutes | 85 |
| 12 | AS8560 ONEANDONE-AS Brauerstrasse 48 |  DE | 3 days, 22 hours, 39 minutes | 81 |
| 13 | AS46606 UNIFIEDLAYER-AS-1 - Unified Layer | US | 1 day, 5 hours, 33 minutes | 77 |
| 14 | AS36352 AS-COLOCROSSING - ColoCrossing | US | 2 days, 10 hours, 15 minutes | 74 |
| 15 | AS4837 CHINA169-BACKBONE CHINA UNICOM China169 Backbone | CN | 2 days, 6 hours, 29 minutes | 71 |
But measuring the average reaction time of abuse desks does not only allow us to compare how hosting providers perform but also the average reaction time per country, printing some nice geo chart:
The geo chart above (dated: Sept 28th, 2018) lets us compare the average reaction time between countries. It shows that abuse desks in Ukraine (UA), Japan (JP) and Zimbabwe (ZW) tend to be slower than abuse desks located in e.g. Mexico (MX), Iceland (IS) or Pakistan (PK).
What hosting providers can do
If you are a hosting provider, there a few things you can do to make the internet a safer place. It's starts with the general recommendation M3AAWG published in February 2014 in their documented Feedback Reporting Recommendation, which I fully agree on:
- Abuse email addresses should support the unfiltered reception of reports in Abuse Reporting Format (ARF)
- All organizations that own a domain name or are responsible for any segment of the public Internet should maintain an RFC 2142 email address for receiving abuse reports (abuse@domain) and should monitor and act on reports made to that mailbox. This address should be published in WHOIS for all to find.
If you have your own ASN, you are a CERT with national responsibility or you are a ccTLD or gTLD owner, you should subscribe to the appropriate URLhaus feed that is available for free. I do recommend you to fetch it every 15 minutes and act upon it accordingly:
You should also submit your abuse mailbox to abuse.net (not to be mixed with abuse.ch) so that other internet users can find your abuse mailbox more easily.
Last but not least, you should:
- React up on abuse reports within 24 hours
- Educate staff working on abuse reports accordingly
- Refrain from putting your abuse mailbox behind a spam filter (message content filtering)
- Refrain from using web based forms for abuse reporting
- Refrain from using sender verification (challenge response)
- Provide an abuse reporting API where trusted reporters can send bulk abuse reports to in an automated way
Conclusion
Fighting abuse is hard, not only for those who are hunting malware like I do but also for hosting providers. They have to read through hunders of abuse reports every day. Automation is hard, if not impossible. A possible solution for this problem is X-ARF, an email format to report different types of network abuse incidents to network owners. The goal of X-ARF is to make handling abuse reports easier for both, the sender of the abuse report and the receiving party. Unfortunately, I've only seen very few hosting providers that have implement X-ARF yet.
What would be nice to have is a DNS record where a hosting provider can state whether it accepts X-ARF reports and to which email address these should be sent to (as far as I know, such a mechanism does not exist at the moment). By this, the sender of an abuse reports could easily determine whether the receiving party is capable to parse X-ARF report and can send human readable abuse reports to such that do not handle X-ARF (yet).
A breakdown of the reaction time over all hosting providers is available on URLhaus. If you are looking for a new hosting provider, you may want to take these statistics into your consideration before deciding which hosting provider you want to choose.
The question that remains is what we as an internet community should do with network operators that do not care about abuse reports. Should they still have a place in the internet community? This question that is hard to answer. My personal feeling is that there should be more pressure towards network owners that do not care about abuse problems in their network, harming other internet users as well as threatening the reliability and stability of the internet. But as there is (fortunately) no governance over the internet, we must find an answer to this question as a community and not as individuals.
Further reading
- URLhaus Statistics: Average Reaction Time
- URLhaus: Subscribe to a ASN, country or TLD based feed
- x-arf: Network Abuse Reporting 2.0
- RFC 2142: Mailbox Names for Common Services, Roles and Functions
- M3AAWG: Anti-Abuse Best Common Practices for Hosting and Cloud Service Providers
- M3AAWG: Feedback Reporting Recommendation