Event driven vs big data

 

Let’s compare our description in yesterday's post with the Big Data architectural style of building services.  This can be a vectorized execution environment and typically involving a size of data not seen with the traditional database systems. Both the storage and the message queue handle large volume of data and the execution can be stages as processing and analysis.  The processing can be either batch oriented or stream oriented.  The analysis and reporting can be offloaded to a variety of technology stacks with impressive dashboards. While the processing handles the requirements for batch and real-time processing on the big data, the analytics supports exploration and rendering of output from big data. It utilizes components such as data sources, data storage, batch processors, stream processors, real-time message queue, analytics data store, analytics and reporting stacks, and orchestration.

Some of the benefits of this application include the following: The ability to mix technology choices, achieving performance through parallelism, elastic scale and interoperability with existing solutions.

Some of the challenges faced with this architectural style include: The complexity where numerous components are required to handle the multiple data sources, and the challenge to build, deploy and test big data processes. Different products require as many as skillsets and maintenance with a requirement for data and query virtualization. For example, U-SQL which is a combination of SQL and C# is used with Azure Data Lake Analytics while SQL APIs are used with Hive, HBase, FLink and Spark. With this kind of a landscape, the emphasis on data security gets diluted and spread over a very large number of components.

Some of the best practices with this architectural style leverage parallelism, partition data, apply schema-on read semantics, process data in place, balance utilization and time costs, separate cluster resources, orchestrate data ingestion and scrub sensitive data

Some examples include applications that leverage IoT architecture and edge computing.

Conclusion: Both these styles serve their purpose of a cloud service very well.