More Insights

In my previous post, I was talking about the past and especially what I did the last year.

While I am not the guy who celebrates new years eve too much, I do like the beginning of a new year, as January has a much slower pace as December, giving me time to think about what I want to achieve in the new year.

My personal goals are to become a better data scientist.  I like this picture about Data Science found on the Wikipedia Article. I could improve myself in any of the fields.
Attribution: http://commons.wikimedia.org/wiki/User:Calvin.Andrus
Wikipedia: Data Science

In some of the fields I will naturally improve myself doing projects with customers (e.g. domain expertise), others may require self-study (should not have slept in math and statistics classes...).

While I'm practicing data science every day, I think it is good to sometimes take a step backwards and learn more about the theory behind it, especially when there is a lot going on in areas like machine learning and predictive analytics. Applying these algorithms requires to understand the theory behind them.

Machine learning and predictive analytics will be a hot topic for me this year. More and more customers are looking to get more value out of their unstructured data. So far customers have been more in reactive mode,  like doing manual queries on historic data. Some of the more advanced customer have started to write their own algorithms. Giving them the tools to automatically detect anomalies and predict the future is the logical next step. This is one area where we are currently investigating and talking to domain experts, to get more insights and come up with a good solution.

Another area where we are expanding into is Application Performance Monitoring. Systems are getting more and more complex, and it is really hard to find root causes. Last week, we signed a partnership agreement that I am very happy with.

Readers of my blog know that beside Splunk there is another Application I'm a huge fan of. Yes, I'm talking about AppDynamics. I'm very happy that we now have a tool in our portfolio, that perfectly complements Splunk. There are almost no overlaps in functionality. AppDynamics provides the insights into an application, while Splunk has the view from the outside.

Together with the new AppDynamics Splunk App, we now have the perfect tag-team... Stay tuned!

Attribution: http://www.flickr.com/photos/usagj/6301467928/sizes/m/in/photostream/
Splunk-AppDynamics Tag-Team


Comments

Post a Comment

Popular posts from this blog

SLOG Latency

Image
After reading Brendan's newest blog entry, I was curious about what kind of slog latency we can see for our data migration load. To remind you, only synchronous writes go into the slog SSD devices. As this is a migration running, we can see mostly NFS write operations: In our configuration the SSD slog is mirrored (HDD4 and HDD8). Hence the same number of IOPS: The next picture shows us the latency for our SDD SLOG Device HDD 4. We can see here that latencies start at 79 us and are mostly under 200 us. There are some outliers, but approx. 95% are under 500 us: This matches quite well with the values Brendan blogged about (137-181 us), which includes NFSv3 latency. For reference (no picture here), we can see latencies of about 170-500 us mostly for NFSv4. By the way. SLOG Devices are mostly one way devices, as shown here. Only if things go really bad, they are read from...
Read more

Heating up the Data Pipeline (Part 1)

Image
Pre-Processing Data I often hear the question from our customers, how data can be transformed prior to indexing in Splunk. Damien from  Baboonbones  has done a tremendous job in creating add-ons   providing custom inputs for Splunk. Most of his custom inputs provide the means to pre-process data by allowing custom event handlers to be written. Sometimes you still want to pre-process data that gets collected from Splunk's standard input types, like file monitors, Windows EventLogs, scripted inputs etc. Also, not everyone is capable of writing custom event handlers. A requirement these customers have, is that they have rolled out a large number of Splunk Universal Forwarders and they do not want to install another agent. To summarize, the solution capable of pre-processing data, should be easy to use, be easily integrated and be build on top of their existing architecture. How to plumb Splunk Pipelines Splunk has its own fittings to connect a Universal Forwar
Read more

Heating up the Data Pipeline (Part 2)

Image
In Part 1 we went through how to route events from Splunk to a 3rd party system without losing metadata. Now I'll show you how events can be transformed using Apache NiFi and be sent back to Splunk into the  HTTP Event Collector . Note: The following is not a step-by-step documentation. To learn how to use Apache NiFi you should read the G etting Started Guide . Simple Pass-Through Flow As a first exercise we will create a simple flow, that only passes data through NiFi, without applying any complex transformations. The following picture shows a high-level NiFi flow, that receives events in our custom uncooked event format with a TCP listener, then sends the data further into a transformation "black box" (aka Processor Group), which emits events in a format, that can be ingested into a Splunk HTTP Event Collector input. Apache Nifi currently provides a rapidly growing number of processors (currently 266), which can be used for data ingestion, transfo
Read more