Measuring Reaction Time of Abuse Desks

Published on 1st October 2018, 10:07:31 UTC


In March 2018, I've launched my most recent project called URLhaus. The goal of URLhaus is to collect and share URLs that are being used for distributing malware. In the first half year, 234 users have contributed and shared more than 50'000 malware distribution sites on URLhaus. That's amazing! But URLhaus does not only collect malware URLs: The project also reports active malware distribution sites to blacklist providers like Spamhaus, SURBL and Google Safe Browsing.

Abuse Reports

Since June 2018, URLhaus is also sending out automated abuse reports to the respective network owners by using Abuseix Abuse Contact DB. The chart below shows the number of active malware distributions sites and abuse reports sent out by URLhaus per day.


When URLhaus started to notify network owners about malware distribution sites in their network, I was pretty surprised to see how many abuse complaints bounced, receiving a Non-Delivery Report (NDR). It appears that it is not uncommon that RIR objects are poorly maintained, pointing to an non-existent (abuse) mailbox. But outdated RIR objects is just one problem I was confronted with when sending out abuse complaints. Here are some of the most common ones:

  • Quota exceeded: While it is generally good news when an abuse mailbox exits, it doesn't mean that abuse reports are actually being handled at all. I've had various cases in the past few months where the abuse report could not be delivered because the abuse mailbox was full. This seems to be a good indicator that the network owner doesn't care about abuse problems in their network. Whether you should accept network traffic from / to networks that do not handle abuse reports at all is something that I leave up for discussion.
      abuse@kvsolutions.nl
        LMTP error after RCPT TO:<abuse@kvsolutions.nl>:
        552 5.2.2 <abuse@kvsolutions.nl> Quota exceeded (mailbox for user is full)
  • Spam filters: An abuse mailbox is the point of contact where internet users will report abuse originating from your network. This of course includes abuse reports related to spam. Unfortunately, some hosting providers put their abuse mailbox behind a spam filter. So guess what happens? The abuse report gets classified as spam by the spam filter and hence gets blocked. What I see mostly is that the spam filter detects a malicious URL in the email body (what a surprise!) and hence classifies it as spam. I do therefore believe that it is a bad idea to put an abuse mailbox behind a spam filter. Should your abuse mailbox get overwhelmed by spam, you may want to consider to implement DNSBL filters (checking the sending IP address against common DNSBLs). This should filter out most of the spam emails. However, from what you should definitely refrain, is implementing message content filtering on your abuse mailbox!
    Your email to group teknis@qwords.co.id was rejected due to spam classification.
    The owner of the group can choose to enable message moderation instead of bouncing these emails.
  • Confirm your address: Some hosting providers came to the glorious idea to implementing sender email address verification. Once you send an abuse report to their designated abuse mailbox, they will send you an automated reply asking you to click on a link to confirm your email address (challenge response). While this might be a way to keep your mailbox free from spam, it doesn't work with automated abuse reports.

  • You are not a customer: It seems that some hosting providers do not have a dedicated abuse desk, which means that abuse reports go to their support desk. While this is generally not a problem, it appears that some staff or ticketing systems only accept emails from email addresses that are known to their customer database. For me as sender of an abuse report this means: No way to report abuse!
    Your email to our support system could not be accepted because it was not recognized as coming from an
    email address belonging to one of our customers. If you need assistance, please email from the address you registered
    with us that you use to login to our client area.
    
  • Web based abuse reporting: Very few hosting providers had another glorious idea; forcing the sender of the abuse report to fill out a web form. While this is might be a good way for the hosting provider to ask the reporter certain questions (and hence ensure that the abuse reports contains the necessary information needed to act up on it), it makes automated abuse reporting impossible.
    In order for us to successfully process your domain inquiry, please go to https://abuse.web.com/
    to complete your inquiry.
  • We need more information to better understand your issue: It appears that some abuse desk staff are not very well educated for dealing with abuse reports. At least that's what I conclude from the response I received and documented below. Dealing with abuse complaints can be challenging, so network owners should make sure that abuse reports are not being handled that by random support staff rather than people who have been trained to deal with such reports.
    Thank you for contacting the Comcast Customer Security Assurance team. Unfortunately, we need more
    information to better understand your issue. Please select the option below that best describes your question.
    We’ve provided easy instructions to help us resolve your issue.
  • Subject overwriting: You as a reporter may send out hundreds of abuse reports per day. So it is essential that the recipient does not overwrite the mail subject when responding to your abuse report. Unfortunately, this seems to be a common practice across major hosting providers like GoDaddy or DreamHost. As they do not include the original abuse report in the email body either, there is not way to map their response to any abuse report you may have sent. Here are a few examples:
    Subject: Incident 37181924: We received your feedback.
    Subject: Thanks for your report. Here are some extra resources.
    Subject: Thank you for reporting abuse on Google Cloud Platform
    Subject: Regarding your DreamHost Abuse message...
    Subject: Action Required
    Subject: Questions About Our Abuse Policies
    Subject: Support Ticket Not Opened
    Subject: Confirmation of Domain Abuse Inquiry

These are just a few examples with that you are confronted with when sending out abuse reports. There are many weird and special cases. What might help to solve at least some of these problems is using of the Abuse Reporting Format (ARF) (but more on this later).

Measuring Abuse Desks Average Reaction Time

In August 2018, I was asked which hosting providers react up on my abuse reports and within which time frame. So I decided to start measuring the reaction time of hosting provider's abuse desks (so called take down time). To do so, I measure the time between when the abuse complaint was sent to the hosting provider and when the reported content went offline. This gives some interesting insights, such as:

  • How long malware distribution sites stay active in average once they have been reported to the hosting provider
  • Which hosting providers react fast upon abuse reports
  • Which hosting providers tend to react slowly to abuse reports
  • Geographically distribution

Across 38'924 abuse reports URLhaus has sent out, the average reaction time is 3 days, 2 hours, 33 minutes. But are these good or bad news? I think: it could be better. The problem with most of the malware distribution sites is that they are associated with a Trojan called Emotet (aka Heodo) which is being spammed out with malicious office documents. It is hard to catch a day where there is no Emotet spam run. In case of Emotet and other malspam campaigns, it is important that the malware distribution sites are getting taken down in time, and with in time I mean hours and not days. Otherwise internet users will get infected even days after the spam run. This is bad.

Among the 600+ hosting providers that URLhaus has notified in the past two months, only 104 (or 16%) reacted within 6 hours in average. If we take a look at the number of hosting providers that reacted within the hour when they received the abuse report from URLhaus, we are down to 13 (or 2%).

The table blow shows the top 15 hosting providers with the fastest abuse desks (as per Sept 28th, 2018). It also includes the number of active (online) and offline malware distribution sites. Please consider that the accuracy of these statistics is +/- 1 hour.

RankASNCountryOnlineOfflineAverage Reaction Time
1AS62240 CLOUVIDER London, United Kingdom- GB0119 minutes
2AS204983 CYBERFUSION- NL0123 minutes
3AS22773 ASN-CXA-ALL-CCI-22773-RDC - Cox Communications Inc. ... - US0130 minutes
4AS17917 QTLTELECOM-AS-AP Quadrant Televentures Limited- IN0432 minutes
5AS3301 TELIANET-SWEDEN Telia Company- SE0233 minutes
6AS45753 NETSEC-HK NETSEC- NL3137 minutes
7AS41357 UK-34SP-AS- GB0238 minutes
8AS23535 HOSTROCKET - HostRocket.com, Inc.- US1539 minutes
9AS197781 AVERS-AS- UA0144 minutes
10AS197226 SPRINT-SDC- PL15146 minutes
11AS42363 PHPNET-AS- FR0549 minutes
12AS15830 TELECITY-LON- NO0252 minutes
13AS6939 HURRICANE - Hurricane Electric LLC- US5259 minutes
14AS132680 NET1-AS-AP Net Virtue Pty Ltd- AU091 hour, 5 minutes
15AS39869 LIVENET-- PL011 hour, 12 minutes

Looking at the bottom of the statistics, we will see that the worst hosting providers with the slowest reaction time had more than 10 days in average to react on the abuse report sent by URLhaus (as per Sept 28th, 2018). The table below also includes the number of active (online) and offline malware distribution sites. Please consider that the accuracy of these statistics is +/- 1 hour.

RankASNCountryOnlineOfflineAverage Reaction Time
610AS49505 SELECTEL- RU8211 days, 16 hours, 54 minutes
611AS21565 AS21565 - Horry Telephone Cooperative, Inc.- US0611 days, 20 hours, 10 minutes
612AS42093 INTERRACKS-AS- NL2611 days, 20 hours, 47 minutes
613AS200960 PROFESIONALHOSTING- ES11112 days, 4 hours, 41 minutes
614AS48368 COMTEH-RU-AS- RU0612 days, 6 hours, 9 minutes
615AS21396 NETCONNEX NetConnex Broadband Ltd.- GB0112 days, 7 hours, 32 minutes
616AS47288 FIXNET- TR0112 days, 10 hours, 55 minutes
617AS199968 IWSNET- NL2112 days, 18 hours, 38 minutes
618AS41887 PROLOCATION Transit policy pref 100- NL0113 days, 3 hours, 14 minutes
619AS9534 MAXIS-AS1-AP Binariang Berhad- MY0113 days, 18 hours, 54 minutes
620AS55720 GIGABIT-MY Gigabit Hosting Sdn Bhd- MY11214 days, 16 hours, 5 minutes
621AS48096 ITGRAD- RU0214 days, 18 hours, 36 minutes
622AS9304 HUTCHISON-AS-AP HGC Global Communications Limit ... - HK0215 days, 7 hours, 56 minutes
623AS45287 VARNION-AS-ID Varnion Technology Semesta, PT- ID9117 days, 12 hours, 37 minutes
624AS133120 HNPL-AS-AP Hosted Network Pty. Ltd.- AU0119 days, 20 hours, 42 minutes

As shown on the table above, there are hosting providers who took more than 2 weeks to remove the malware from their hosting space. But it gets worse: there are malware distribution sites that are active for more than 3 months now and that do not show up in these statistics. The following table documents the top malware hosting networks that are hosting active malware content (counting online malware distribution sites only per 1st Oct 2018):

RankASNCountryAverage Reaction TimeActive Malware URLs
1AS26496 AS-26496-GO-DADDY-COM-LLC - GoDaddy.com, LLC- US6 days, 5 hours, 54 minutes402
2AS14061 DIGITALOCEAN-ASN - DigitalOcean, LLC- US3 days, 14 hours, 27 minutes295
3AS48815 CRITICALCASE- IT21 hours, 58 minutes151
4AS34619 CIZGI- TR4 days, 13 hours, 4 minutes134
5AS9891 CSLOX-IDC-AS-AP CS LOXINFO Public Company Limited.- TH1 day, 18 hours, 17 minutes132
6AS32244 LIQUIDWEB - Liquid Web, L.L.C- US4 days, 22 hours, 14 minutes125
7AS13335 CLOUDFLARENET - Cloudflare, Inc.- US1 day, 22 hours, 54 minutes117
8AS16276 OVH- PL2 days, 8 hours, 0 minutes112
9AS4134 CHINANET-BACKBONE No.31,Jin-rong Street- CN1 day, 10 hours, 29 minutes108
10AS25535 ASN-RUCENTER-HOSTING- RU5 days, 6 hours, 57 minutes105
11AS37963 CNNIC-ALIBABA-CN-NET-AP Hangzhou Alibaba Advertising Co.,Ltd.- CN1 day, 15 hours, 35 minutes85
12AS8560 ONEANDONE-AS Brauerstrasse 48- DE3 days, 22 hours, 39 minutes81
13AS46606 UNIFIEDLAYER-AS-1 - Unified Layer- US1 day, 5 hours, 33 minutes77
14AS36352 AS-COLOCROSSING - ColoCrossing- US2 days, 10 hours, 15 minutes74
15AS4837 CHINA169-BACKBONE CHINA UNICOM China169 Backbone- CN2 days, 6 hours, 29 minutes71

But measuring the average reaction time of abuse desks does not only allow us to compare how hosting providers perform but also the average reaction time per country, printing some nice geo chart:





The geo chart above (dated: Sept 28th, 2018) lets us compare the average reaction time between countries. It shows that abuse desks in Ukraine (UA), Japan (JP) and Zimbabwe (ZW) tend to be slower than abuse desks located in e.g. Mexico (MX), Iceland (IS) or Pakistan (PK).

What hosting providers can do

If you are a hosting provider, there a few things you can do to make the internet a safer place. It's starts with the general recommendation M3AAWG published in February 2014 in their documented Feedback Reporting Recommendation, which I fully agree on:

  • Abuse email addresses should support the unfiltered reception of reports in Abuse Reporting Format (ARF)
  • All organizations that own a domain name or are responsible for any segment of the public Internet should maintain an RFC 2142 email address for receiving abuse reports (abuse@domain) and should monitor and act on reports made to that mailbox. This address should be published in WHOIS for all to find.

If you have your own ASN, you are a CERT with national responsibility or you are a ccTLD or gTLD owner, you should subscribe to the appropriate URLhaus feed that is available for free. I do recommend you to fetch it every 15 minutes and act upon it accordingly:

You should also submit your abuse mailbox to abuse.net (not to be mixed with abuse.ch) so that other internet users can find your abuse mailbox more easily.

Last but not least, you should:

  • React up on abuse reports within 24 hours
  • Educate staff working on abuse reports accordingly
  • Refrain from putting your abuse mailbox behind a spam filter (message content filtering)
  • Refrain from using web based forms for abuse reporting
  • Refrain from using sender verification (challenge response)
  • Provide an abuse reporting API where trusted reporters can send bulk abuse reports to in an automated way

Conclusion

Fighting abuse is hard, not only for those who are hunting malware like I do but also for hosting providers. They have to read through hunders of abuse reports every day. Automation is hard, if not impossible. A possible solution for this problem is X-ARF, an email format to report different types of network abuse incidents to network owners. The goal of X-ARF is to make handling abuse reports easier for both, the sender of the abuse report and the receiving party. Unfortunately, I've only seen very few hosting providers that have implement X-ARF yet.

What would be nice to have is a DNS record where a hosting provider can state whether it accepts X-ARF reports and to which email address these should be sent to (as far as I know, such a mechanism does not exist at the moment). By this, the sender of an abuse reports could easily determine whether the receiving party is capable to parse X-ARF report and can send human readable abuse reports to such that do not handle X-ARF (yet).

A breakdown of the reaction time over all hosting providers is available on URLhaus. If you are looking for a new hosting provider, you may want to take these statistics into your consideration before deciding which hosting provider you want to choose.

The question that remains is what we as an internet community should do with network operators that do not care about abuse reports. Should they still have a place in the internet community? This question that is hard to answer. My personal feeling is that there should be more pressure towards network owners that do not care about abuse problems in their network, harming other internet users as well as threatening the reliability and stability of the internet. But as there is (fortunately) no governance over the internet, we must find an answer to this question as a community and not as individuals.

Further reading

Blog Archive