Since I read about Splunk on Ben Rockwood's blog, I'm a huge fan. I even got a Splunk baseball cap and a T-Shirt.
At my former employer, I've implemented Splunk to collect system logs for system monitoring and compliance checks/reporting.
Version 3.0 (still in beta) seems to be a huge step forwards. Reporting e.g. is now very sophisticated, allowing one to create many kinds of reports (charts, tables).
To get a quick overview over different environment aspects, it is possible to create user/role dashboards.
In the beginning Splunk was mainly meant for sucking only log files in, the target has now changed to index any kind of unstructured data.
I'm very much interested in loading configuration files and monitor these for changes (security monitoring/audits anyone?). It is also possible to periodically index command outputs. This could be used for recording performance data (output from e.g. iostat, vmstat, etc.). The output from a config file or a command, looks just like any other multi-line event.
I think this feature is what differentiates Splunk from competitors. Collecting log data is one thing, collecting and analyzing any unstructured data is more difficult. Here Splunk has unique features.
In the near future I will write more about, what I think Splunk could be used for.