Big Data Based Backend

While our new Big Data based backend is explained in detail here, we can just say this is the main reason behind 3.0 announcement. We have made public a modified version of Barnyard2 supporting sending the events as Apache Kafka messages. This messages enter the Manager and are processed and stored in two different places:

• OLAP engine for extremely fast web access on aggregated data with very powerful slice & dice capabilities.
• Hadoop to store all events for post analysis or legal reasons.

This new architecture not only provides us with the capability to infinitely grow the Manager capacity (in the Enterprise version) but also will allow us to include preprocessing stages before the data is actually stored. This will be the main focus of the next redborder release.