A former client of mine, an enterprise architect and a guy I really respect, recommended Splunk (“not just a dirty word”) to me. They bring, more or less, a search engine approach to log file analysis. Now, this is not the sexiest thing in the world, but it’s critically important, especially in large IT shops. A large enterprise generates humongous amounts of log files; my friend said that he’s pinned a big server with just the logs from their domain controllers. And remember, these are just text files.

So the question becomes: how do you analyze all this? Traditionally, people have taken a static reporting approach, which has its place, but you need more when you have to be actively responsive. When was the last time David Hasselhof logged on? Where was he? What systems did he log onto? Did he look at Michael Jackson’s billing records?

Long ago, people thought that some kind of library-like structure was required in order to discover information on the Internet, but it turned out that brute-force searching was better. Likewise in this case, where the end goal is a Google-like interface. Now, this approach has its limitations. You have to know what you’re looking for, first of all. It doesn’t do correlations. It’s got a beautifully simple interface, but it’s not an easy UI for normal, proactive review. It’s not for canned reports. It’s not a SEM (Security Event Manager, or SIEM: Security Information and Event Manager) tool.

But for what it is, it’s great. It’s easy to look at Splunk and say, “you’re just indexing text,” but there is great power in that; look at Google. There been such a huge emphasis on auditability that we’ve generated huge files of events, but mostly they just sit there unloved. Splunk is a good way to leverage that resource.

IMAP makes a difference

Google’s recent announcement that they’re supporting IMAP has finally convinced me to move off of a desktop email client, Thunderbird in my case, and use Gmail exclusively.  Google’s spam filters, enhanced by the many-eyes of collaborative filtering (I mean the “report spam” button), are so much better than T’bird’s.  I was wasting too much time each day sifting through the dross for the pearls.  I’m sure I’m giving up something, but searching is faster and the new features are coming fast and furious.  iPhone integration is great; I don’t know that the IMAP announcement would have made as much of a difference if I didn’t have an iPhone.  Since I travel a lot, keeping POP email sync’d up was a hassle; I’d come home from a week’s trip and have hundreds of spams to deal with after the filters had done their work.  I’m hedging my bets by downloading everything on Gmail to my local machine, but the cloud computing vision just got one giant step closer for me.

Facial recognition

Now, they know what you look like:

Google has quietly added facial recognition to its image search.   If you do an image search and append &imgtype=face at the end of the URL, you get only the faces associated with that search.  So, for example, an image search of “Novell” gives you screenshots, network diagrams, box shots and the like.  But if you append &imgtype=face, you get only people’s faces.  Cool, but scary.

Hunger, data visualization, and the value of clean hands

From Radar O’Reilly:

Another couple of webcasts from our hero, Prof. Hans Rosling of Sweden’s Karolinska Institute and Gapminder (acquired, inevitably, by Google.)

In the first webcast, Rosling using his Trendalyzer visualization tool to describe economic and social change in Sweden over the past three hundred years. If he doesn’t convince you to wash your hands, no one will.

Recently, both the United Nations and OECD (announcment here) have committed to opening up their statistical databases free of charge. In his second webcast, Rosling points out, all of the country-level statistical data — in the whole world, ever — is a smaller download than “Lord of the Rings.”

As before, both are well worth watching, not only for the content of what you learn, but how Rosling delivers the message; it’s a miracle of data visualization and a heartening message about the possibilities of the future.

(Previous post about Rosling here.)

Google to the rescue: FETCH! With Ruff Ruffman

I know they’re improving Maps at an alarming rate; My Maps, full-screen (hide directions), traffic, street view, drag to re-route directions, and so forth. And that’s just in the past few months, in one application. Gears is also significant because it promises to bridge the off-line gap, the so-called airplane problem. They’re supposedly acquiring Grand Central, which I think is very smart. They just released the desktop for Linux, which is also great. And they’re doing something important, which I don’t understand, with package management; but Stephen O’Grady tells me it is, so it is.

But here’s why, today, I think Google’s great, and it cost me two bucks. My fiveRuff Ruffman year old was looking forward to watching a show (“FETCH! With Ruff Ruffman,” whose appeal escapes me) this afternoon on PBS. But it turns out that since we only have old-fashioned over-the-air analog television — which is practically a war crime, I know — that particular show wasn’t on at the promised time. I assume he saw an ad for some PBS cable channel. Anyway, it was his television for the day and he was bitterly disappointed. Sobbing.

Google to the rescue. First, I tried going to the PBS website, but they only had lame games and, frustratingly, trailers for the show. YouTube, nothing. But a quick Google search offered me the entire first season in the Google Video Player (soon to be deprecated in favor of YouTube?) for $1.99/episode. So I got 29 minutes of relief from child care on a Sunday afternoon and Google got my undying gratitude.