Ultra-short guide to Big Data and Security

If you are looking to navigate your way through the complexities of Big Data and its use in Security, here are some links to get you off the ground:

Big Data Basics

1. What is Big Data
2. Which are the major technologies used for Big Data
   1. Hadoop (Apache’s open-source implementation of Google’s MapReduce algo)
   2. Elastic Search, Logstash, Kibana (ELK)
3. Major implementations of Hadoop – Cloudera and Hortonworks
4. How it started to become mainstream – Google publishes its MapReduce algorithm

Big Data For Security?

How can we leverage this capability to deliver security intelligence? There are a number of very interesting implementations that have been done:

1. Here’s a very good take by Splunk on Big Data for Security 
2. Using HP Arcsight for APT Detection
3. 10 Steps to a Better Big Data Security Strategy 
4. How are Cisco and Verizon leveraging ELK for security intelligence

Getting started

Begin by playing with your own ELK setup on AWS or Azure and then see if you can do anomaly detection using ELK. If you need a large set of sample logs, hop over here. Or better yet go straight and get Apache Metron up and running!