Detecting bots and mobile devices in asp.net

There are a lot of different entities crawling around your web applications in the wild, including web browsers, web crawlers, web spiders, web bots, and automated scripts. Determining the difference between a regular user visiting your site and an automated web bot can help aid you with more accurately recording statistics, customizing content, and optimizing web application performance.

Also, more and more people use mobile devices. It is either a smart-phone or a tablet PC. In many cases, we need that information to show to the visitor the mobile version of our site. This information is more useful when specifically comes to show a video. Mobile browsers cannot play the video stream (lack of flash support) and leave it to the device to play the video. In every case there a few parameters to consider, e.g. bitrate, framerate, codec etc.

In both cases the information exists in the user-agent string. In this article we will see how asp.net provides such information and how to do it with your own code. 

Regarding the bot detection issue, asp.net gives you that information through the Request.Browser object with the .Crawler property.

If(Request.Browser.Crawler) Then
    'this is a web bot
Else
    'this is a regular user
End If

The mechanism for the actual detection is not really hard. All you need to do is parse the user-agent string and search for specific words or phrases. Asp.net gives us the user-agent string in the Request.UserAgent.ToString() property. For example, if you want to detect the Google bot, you can simply write this line of code.

If(Request.UserAgent.ToString().Contains("Googlebot")) Then
    'this is a visit from GoogleBot!
End If

A simple thing to do is keep in an array all the unique words or phrases that can be used to detect each case. If any element of the array is found in the user-agent string then you have a bot and not a regular visitor to your site. For example:

Function IsBot(ByVal Request As System.Web.HttpRequest) As Boolean
   If (Request.Browser.Crawler) Then Return True
   Dim myUserAgent As String = Request.UserAgent.ToLower
 
   Dim myArray() As String = New String() {"teoma", "alexa", "froogle", "gigabot", _
   "inktomi", "looksmart", "url_spider_sql", "firefly", "nationaldirectory",
   "ask jeeves", "tecnoseek", "infoseek", "webfindbot", "girafabot",  "crawler",
   "www.galaxy.com", "googlebot", "scooter", "slurp", "msnbot", "appie", "fast", _
   "webbug", "spade", "zyborg", "rabaz", "baiduspider", "feedfetcher-google", _
   "technoratisnoop", "rankivabot", "mediapartners-google", "sogou web spider", _
   "webalta crawler", "facebookexternalhit"}
 
   For Each _Item As String In myArray
       If (myUserAgent.Contains(_Item)) Then Return True
   Next
   Return False
End Function

Asp.net has a built in mechanism for that and we don’t have to write code for every bot in the planet. Sometimes though, the Request.Browser.Crawler value is always false. This is due to some missing meta-data information. We can add the missing meta-data to our web.config file and get the desired functionality. In “Detecting Browsers, Crawlers, and Web Bots in C# ASP .NET” by Kory Becker, there is a link with an xml file that has this information. Ideally, this should be stored in the machine.config file so that all web applications running in the web server can use this information. If you don’t have access to the machine.config file, you can store it in your Web.config. But the default maximum size of a Web.config file is 250kb (100kb in Vista) so that approach might not be really useful.

Regarding the device detection issue, asp.net also gives you that information through the Request.Browser object with the .IsMobileDevice property.

If(Request.Browser.IsMobileDevice) Then
    'this is a mobile device
Else
    'this is a regular desktop browser
End If

The mechanism to extract this information is basically the same as before. For example you could write the following function:

Function IsMobile(ByVal Request As System.Web.HttpRequest) As Boolean
   If (Request.Browser.IsMobileDevice()) Then Return True
   Dim myUserAgent As String = Request.UserAgent.ToLower
 
   Dim myArray() As String = New String() {"midp", "j2me", "avant", "docomo", _
   "novarra", "palmos", "palmsource", "240x320", "opwv", "chtml", _
   "pda", "windows ce", "mmp/", "blackberry", "mib/", "symbian", "wireless", _
   "nokia", "hand", "mobi", "phone", "cdm", "up.b", "audio", "SIE-", "SEC-", _
   "samsung", "HTC", "mot-", "mitsu", "sagem", "sony", "alcatel", "lg", _
   "eric", "vx", "NEC", "philips", "mmm", "xx", "panasonic", "sharp", "wap", _
   "sch", "rover", "pocket", "benq", "java", "pt", "pg", "vox", "amoi", _
   "bird", "compal", "kg", "voda", "sany", "kdd", "dbt", "sendo", "sgh", _
   "gradi", "jb", "dddi", "moto", "iphone", "android"}
 
   For Each _Item As String In myArray
       If (myUserAgent.Contains(_Item)) Then Return True
   Next
   Return False
End Function

As you can see the approach is very simple and there is not much stuff to think about. One issue is performance, but storing that information in Session is enough. I use the above functions and so far they server me well.  The only problem is that the words and phrases I have must be renewed. The above phrases are what I have picked up so far from the Internet while googling on the subject.

If you are interested in a basic functionality this approach should do just fine. But if you really want to have accuracy, you should consider purchasing a library that does that for you. If you google a bit you will find a few. Having this small database up to date can be a tedious task.

You could check out the “User Agent String.Com” website. It has a good database of crawlers and browsers. Alsi, it provides an API where you can get useful information with POST or GET requests.