Network log analysis with SQL-on-Hadoop

Network log analysis with SQL-on-Hadoop

Blog Article

Toilet Brush With the rapid expansion of network bandwidth,devices and applications,log management is facing the challenge of exploding data volumes.Log analysis platform built on SQL-on-Hadoop is capable of storing and querying hundreds of billions of log entries effectively.Columnar and compressed data formats for Hadoop are benchmarked with real-world multi-TB dataset.

Conditional and statistical querying efficiency of Hive and Impala is tested.With gzipped parquet format,log data can be compressed by 80%,and querying with impala is 5 times faster.On this platform,six security incident analysis and detection applications Ski de fond - Homme - Skis - Classic are already deployed.