Good tools for doing reverse IP and geolocation out of a web server log?

Folks: Not everyone on the planet is cool enough to use Google Analytics. What if you have an old-school HTTP server log and want to get more information about users, especially hostnames and geolocations? What are the most reasonable tools these days, either desktop Windows apps or Unix server-based? I don’t need something scriptable that can run every night.

Alternatively, given a list of IP addresses, what would you do if you wanted to turn that into a CSV file of IP, hostname, location?

9 Comments »

  1. Paddy

    February 12, 2018 @ 1:20 pm

    1

    https://www.maxmind.com/en/geoip-demo is a good example which can be scripted against their DB. Alternatively if you use nginx/apache, you have modules which can do this inline and write it to your server log itself

  2. lion

    February 12, 2018 @ 1:48 pm

    2

    Are you trying to find out where the world’s greatest blog commenter lives?

  3. Justin

    February 12, 2018 @ 2:53 pm

    3

    Might want to check out https://ipinfo.io/. Its free unless you’re a heavy user of IP info.

  4. Justin Thomas

    February 12, 2018 @ 3:15 pm

    4

    Maxmind as said above or route through a CDN which will add the information for you. Visualization is easy with a third party logging tool like Logs.IO (ELK based)

  5. Fazal Majid

    February 12, 2018 @ 6:14 pm

    5

    MaxMind Geolite (free) or GeoIP2 (paid, better coverage) for the database

    GoAccess can use MaxMind, and works in both terminal and web-server modes. You can also use it to do a “top” style running analysis of live log files.
    https://goaccess.io/

    Alternatively if you prefer using SQL to query logs, I have a 255-line Go program I use to convert Apache CLF files into tab-separated files (with geo lookup) suitable for import into PostgreSQL (or more precisely CitusDB, a MPP extension thereof). It needs to be adapted for the specific format used by the web server, as no two servers have exactly the same format string and my tool is too rudimentary to support something as fancy as parsing format strings.

  6. Chris Nahr

    February 13, 2018 @ 1:29 am

    6

    There’s Weblog Expert which has both free and commercial editions.
    http://www.weblogexpert.com/

    The biggest problem is filtering out bots, though. That’s basically impossible today without a continually updated web service (such as Akismet), or else requiring JavaScript code (such as Google Analytics).

    I once tried to do my own referrer spam filtering in conjunction with Weblog Expert but quickly gave up on the attempt. The majority of all Internet traffic is garbage today, so your raw server logs won’t be very useful except to profile bot attacks.
    http://kynosarges.org/ReferFilter.html

  7. Steve

    February 13, 2018 @ 3:02 am

    7

    Phil wants to know more about the Trump supporters who post comments at 3am Boston / 11am Moscow time.

    He probably also wants to know about the WSJ reporters who read this blog for inspiration: https://www.wsj.com/articles/the-high-cost-of-affordable-housing-mandates-1518479107

  8. Reha Gur

    February 13, 2018 @ 9:39 am

    8

    For reasonably small logs (<100Mb) Maxmind free and Perl should do the trick. There are Maxmind Perl modules you can use.

    For larger logs check out the series of wide finder and wide finder2 articles from Tim Bray and combine with Maxmind.

    https://www.tbray.org/ongoing/When/200x/2007/09/20/Wide-Finder
    https://www.tbray.org/ongoing/When/200x/2008/05/01/Wide-Finder-2

  9. Poika

    February 13, 2018 @ 11:28 pm

    9

    For a simple manual search I use the https://abongo.com/ enter the IP or domain name (sans “httpX://”) and press WhoIs button, shows registration info and a map.

    The http://www.all-nettools.com/toolbox/smart-whois.php is a well known site with a lot of online tools and they offer also software, many are shareware but e.g. the “Free IP Tools 4.2” is free and has plenty of tools. This site has always been safe, no badware found there ever (even then I always check each installation packages, that are smaller than their max limit that is 20MB, before installing here: https://www.virustotal.com/#/home/upload they run the files through nearly all of the virus checkers for free).

    In these days, however, one can not draw absolute correct conclusions based on the IP address or domain names because many people are using VPN tubing such as the https://www.f-secure.com/en/web/home_global/freedome (I’m not affiliated with them, that product just is affordable, fast and has good coverage and plenty of output nodes). People who seem to be posting from different IP address each time most likely are using a VPN tube.

Leave a Comment

Log in