Published on 1st October 2018, 10:07:31 UTC
In March 2018, I've launched my most recent project called URLhaus. The goal of URLhaus is to collect and share URLs that are being used for distributing malware. In the first half year, 234 users have contributed and shared more than 50'000 malware distribution sites on URLhaus. That's amazing! But URLhaus does not only collect malware URLs: The project also reports active malware distribution sites to blacklist providers like Spamhaus, SURBL and Google Safe Browsing.
Since June 2018, URLhaus is also sending out automated abuse reports to the respective network owners by using Abuseix Abuse Contact DB. The chart below shows the number of active malware distributions sites and abuse reports sent out by URLhaus per day.
When URLhaus started to notify network owners about malware distribution sites in their network, I was pretty surprised to see how many abuse complaints bounced, receiving a Non-Delivery Report (NDR). It appears that it is not uncommon that RIR objects are poorly maintained, pointing to an non-existent (abuse) mailbox. But outdated RIR objects is just one problem I was confronted with when sending out abuse complaints. Here are some of the most common ones:
firstname.lastname@example.org LMTP error after RCPT TO:<email@example.com>: 552 5.2.2 <firstname.lastname@example.org> Quota exceeded (mailbox for user is full)
Your email to group email@example.com was rejected due to spam classification. The owner of the group can choose to enable message moderation instead of bouncing these emails.
Your email to our support system could not be accepted because it was not recognized as coming from an email address belonging to one of our customers. If you need assistance, please email from the address you registered with us that you use to login to our client area.
In order for us to successfully process your domain inquiry, please go to https://abuse.web.com/ to complete your inquiry.
Thank you for contacting the Comcast Customer Security Assurance team. Unfortunately, we need more information to better understand your issue. Please select the option below that best describes your question. We’ve provided easy instructions to help us resolve your issue.
Subject: Incident 37181924: We received your feedback. Subject: Thanks for your report. Here are some extra resources. Subject: Thank you for reporting abuse on Google Cloud Platform Subject: Regarding your DreamHost Abuse message... Subject: Action Required Subject: Questions About Our Abuse Policies Subject: Support Ticket Not Opened Subject: Confirmation of Domain Abuse Inquiry
These are just a few examples with that you are confronted with when sending out abuse reports. There are many weird and special cases. What might help to solve at least some of these problems is using of the Abuse Reporting Format (ARF) (but more on this later).
In August 2018, I was asked which hosting providers react up on my abuse reports and within which time frame. So I decided to start measuring the reaction time of hosting provider's abuse desks (so called take down time). To do so, I measure the time between when the abuse complaint was sent to the hosting provider and when the reported content went offline. This gives some interesting insights, such as:
Across 38'924 abuse reports URLhaus has sent out, the average reaction time is 3 days, 2 hours, 33 minutes. But are these good or bad news? I think: it could be better. The problem with most of the malware distribution sites is that they are associated with a Trojan called Emotet (aka Heodo) which is being spammed out with malicious office documents. It is hard to catch a day where there is no Emotet spam run. In case of Emotet and other malspam campaigns, it is important that the malware distribution sites are getting taken down in time, and with in time I mean hours and not days. Otherwise internet users will get infected even days after the spam run. This is bad.
Among the 600+ hosting providers that URLhaus has notified in the past two months, only 104 (or 16%) reacted within 6 hours in average. If we take a look at the number of hosting providers that reacted within the hour when they received the abuse report from URLhaus, we are down to 13 (or 2%).
The table blow shows the top 15 hosting providers with the fastest abuse desks (as per Sept 28th, 2018). It also includes the number of active (online) and offline malware distribution sites. Please consider that the accuracy of these statistics is +/- 1 hour.
|Rank||ASN||Country||Online||Offline||Average Reaction Time|
|1||AS62240 CLOUVIDER London, United Kingdom||GB||0||1||19 minutes|
|2||AS204983 CYBERFUSION||NL||0||1||23 minutes|
|3||AS22773 ASN-CXA-ALL-CCI-22773-RDC - Cox Communications Inc. ...||US||0||1||30 minutes|
|4||AS17917 QTLTELECOM-AS-AP Quadrant Televentures Limited||IN||0||4||32 minutes|
|5||AS3301 TELIANET-SWEDEN Telia Company||SE||0||2||33 minutes|
|6||AS45753 NETSEC-HK NETSEC||NL||3||1||37 minutes|
|7||AS41357 UK-34SP-AS||GB||0||2||38 minutes|
|8||AS23535 HOSTROCKET - HostRocket.com, Inc.||US||1||5||39 minutes|
|9||AS197781 AVERS-AS||UA||0||1||44 minutes|
|10||AS197226 SPRINT-SDC||PL||15||1||46 minutes|
|11||AS42363 PHPNET-AS||FR||0||5||49 minutes|
|12||AS15830 TELECITY-LON||NO||0||2||52 minutes|
|13||AS6939 HURRICANE - Hurricane Electric LLC||US||5||2||59 minutes|
|14||AS132680 NET1-AS-AP Net Virtue Pty Ltd||AU||0||9||1 hour, 5 minutes|
|15||AS39869 LIVENET-||PL||0||1||1 hour, 12 minutes|
Looking at the bottom of the statistics, we will see that the worst hosting providers with the slowest reaction time had more than 10 days in average to react on the abuse report sent by URLhaus (as per Sept 28th, 2018). The table below also includes the number of active (online) and offline malware distribution sites. Please consider that the accuracy of these statistics is +/- 1 hour.
|Rank||ASN||Country||Online||Offline||Average Reaction Time|
|610||AS49505 SELECTEL||RU||8||2||11 days, 16 hours, 54 minutes|
|611||AS21565 AS21565 - Horry Telephone Cooperative, Inc.||US||0||6||11 days, 20 hours, 10 minutes|
|612||AS42093 INTERRACKS-AS||NL||2||6||11 days, 20 hours, 47 minutes|
|613||AS200960 PROFESIONALHOSTING||ES||11||1||12 days, 4 hours, 41 minutes|
|614||AS48368 COMTEH-RU-AS||RU||0||6||12 days, 6 hours, 9 minutes|
|615||AS21396 NETCONNEX NetConnex Broadband Ltd.||GB||0||1||12 days, 7 hours, 32 minutes|
|616||AS47288 FIXNET||TR||0||1||12 days, 10 hours, 55 minutes|
|617||AS199968 IWSNET||NL||2||1||12 days, 18 hours, 38 minutes|
|618||AS41887 PROLOCATION Transit policy pref 100||NL||0||1||13 days, 3 hours, 14 minutes|
|619||AS9534 MAXIS-AS1-AP Binariang Berhad||MY||0||1||13 days, 18 hours, 54 minutes|
|620||AS55720 GIGABIT-MY Gigabit Hosting Sdn Bhd||MY||11||2||14 days, 16 hours, 5 minutes|
|621||AS48096 ITGRAD||RU||0||2||14 days, 18 hours, 36 minutes|
|622||AS9304 HUTCHISON-AS-AP HGC Global Communications Limit ...||HK||0||2||15 days, 7 hours, 56 minutes|
|623||AS45287 VARNION-AS-ID Varnion Technology Semesta, PT||ID||9||1||17 days, 12 hours, 37 minutes|
|624||AS133120 HNPL-AS-AP Hosted Network Pty. Ltd.||AU||0||1||19 days, 20 hours, 42 minutes|
As shown on the table above, there are hosting providers who took more than 2 weeks to remove the malware from their hosting space. But it gets worse: there are malware distribution sites that are active for more than 3 months now and that do not show up in these statistics. The following table documents the top malware hosting networks that are hosting active malware content (counting online malware distribution sites only per 1st Oct 2018):
|Rank||ASN||Country||Average Reaction Time||Active Malware URLs|
|1||AS26496 AS-26496-GO-DADDY-COM-LLC - GoDaddy.com, LLC||US||6 days, 5 hours, 54 minutes||402|
|2||AS14061 DIGITALOCEAN-ASN - DigitalOcean, LLC||US||3 days, 14 hours, 27 minutes||295|
|3||AS48815 CRITICALCASE||IT||21 hours, 58 minutes||151|
|4||AS34619 CIZGI||TR||4 days, 13 hours, 4 minutes||134|
|5||AS9891 CSLOX-IDC-AS-AP CS LOXINFO Public Company Limited.||TH||1 day, 18 hours, 17 minutes||132|
|6||AS32244 LIQUIDWEB - Liquid Web, L.L.C||US||4 days, 22 hours, 14 minutes||125|
|7||AS13335 CLOUDFLARENET - Cloudflare, Inc.||US||1 day, 22 hours, 54 minutes||117|
|8||AS16276 OVH||PL||2 days, 8 hours, 0 minutes||112|
|9||AS4134 CHINANET-BACKBONE No.31,Jin-rong Street||CN||1 day, 10 hours, 29 minutes||108|
|10||AS25535 ASN-RUCENTER-HOSTING||RU||5 days, 6 hours, 57 minutes||105|
|11||AS37963 CNNIC-ALIBABA-CN-NET-AP Hangzhou Alibaba Advertising Co.,Ltd.||CN||1 day, 15 hours, 35 minutes||85|
|12||AS8560 ONEANDONE-AS Brauerstrasse 48||DE||3 days, 22 hours, 39 minutes||81|
|13||AS46606 UNIFIEDLAYER-AS-1 - Unified Layer||US||1 day, 5 hours, 33 minutes||77|
|14||AS36352 AS-COLOCROSSING - ColoCrossing||US||2 days, 10 hours, 15 minutes||74|
|15||AS4837 CHINA169-BACKBONE CHINA UNICOM China169 Backbone||CN||2 days, 6 hours, 29 minutes||71|
But measuring the average reaction time of abuse desks does not only allow us to compare how hosting providers perform but also the average reaction time per country, printing some nice geo chart:
The geo chart above (dated: Sept 28th, 2018) lets us compare the average reaction time between countries. It shows that abuse desks in Ukraine (UA), Japan (JP) and Zimbabwe (ZW) tend to be slower than abuse desks located in e.g. Mexico (MX), Iceland (IS) or Pakistan (PK).
If you are a hosting provider, there a few things you can do to make the internet a safer place. It's starts with the general recommendation M3AAWG published in February 2014 in their documented Feedback Reporting Recommendation, which I fully agree on:
If you have your own ASN, you are a CERT with national responsibility or you are a ccTLD or gTLD owner, you should subscribe to the appropriate URLhaus feed that is available for free. I do recommend you to fetch it every 15 minutes and act upon it accordingly:
You should also submit your abuse mailbox to abuse.net (not to be mixed with abuse.ch) so that other internet users can find your abuse mailbox more easily.
Last but not least, you should:
Fighting abuse is hard, not only for those who are hunting malware like I do but also for hosting providers. They have to read through hunders of abuse reports every day. Automation is hard, if not impossible. A possible solution for this problem is X-ARF, an email format to report different types of network abuse incidents to network owners. The goal of X-ARF is to make handling abuse reports easier for both, the sender of the abuse report and the receiving party. Unfortunately, I've only seen very few hosting providers that have implement X-ARF yet.
What would be nice to have is a DNS record where a hosting provider can state whether it accepts X-ARF reports and to which email address these should be sent to (as far as I know, such a mechanism does not exist at the moment). By this, the sender of an abuse reports could easily determine whether the receiving party is capable to parse X-ARF report and can send human readable abuse reports to such that do not handle X-ARF (yet).
A breakdown of the reaction time over all hosting providers is available on URLhaus. If you are looking for a new hosting provider, you may want to take these statistics into your consideration before deciding which hosting provider you want to choose.
The question that remains is what we as an internet community should do with network operators that do not care about abuse reports. Should they still have a place in the internet community? This question that is hard to answer. My personal feeling is that there should be more pressure towards network owners that do not care about abuse problems in their network, harming other internet users as well as threatening the reliability and stability of the internet. But as there is (fortunately) no governance over the internet, we must find an answer to this question as a community and not as individuals.