From infrastructure requirements to ensuring data security, compliance with the GDPR, and getting the right talent to handle big data, Sunil Paul, Co-founder and COO, Finesse, tells The Integrator, what it takes to make the most out of Big Data Analytics.
Q 1. What measures are businesses adopting to handle big data?
A1. The amount of data, organizations capture daily, has exponentially increased. Recently, with the policy changes implemented using GDPR, it has become critical for companies to use their data responsibly. GDPR laws have enforced that companies notify their data protection authority of any data breach within the first 72 hours when they become aware of it. Similar compliance laws have made it critical for companies to not only monitor their data efficiently, but also increase their infrastructure to store and query their data at the capture stage. Hence, businesses are moving towards tools that can facilitate such capture and query of data in large scale, as well as scale their existing infrastructure to accommodate for this.
Q2. Can businesses use old and new data simultaneously?
A2. A multi-faceted approach is essential to capturing and analyzing data. For example, a lot of security and event management tools use the Suricata engine on top of them to detect any network intrusion or threats to their network. Suricata is an open source network threat detection engine. Hence, any new network data can be compared with Suricata’s threat detection patterns, and suspicious activity can be flagged before it is too late. This kind of preventative security analytics is what most enterprises with sensitive data are implementing. To have such systems in place, it is essential to use both new and old data concurrently. Another area that businesses use old data is while launching updated and enhanced products. They can analyse past data about products and customer feedbacks. The real time (new data) shows shifts in demand and supplies, analyzes consumer needs, preferences, and buying behaviour. They can use these to better the product or launch new ones in the future.
Q3. How can businesses derive maximum benefit from Big Data Analytics?
A3. Big Data Analytics help provide business intelligence that reduces costs and improves efficiency of operations. Big Data Analytics can analyse past data to make predictions for the future. Thus, it helps the present as well as the future. Some of the areas that organizations currently use big data to derive insights include network monitoring, and security and incident management. However, lately there has been a rise in using big data to capture data related to IoT devices and monitor user behavior. These provide a better framework for decision making and risk management. All these results in giving businesses the competitive edge.
Q4. How important is infrastructure for Big Data Analytics?
A4. All businesses want to be part of the big data wave, wanting insights for better decision making. But without the right tools, they all end up wading through enormous data. Big data starts at the infrastructure level. Once the appropriate hardware is in place, businesses can focus on database and applications. Thus today’s infrastructure definitely needs not be the bottle neck, but to be able to handle enormous amounts of data – it has to be high performance in processing as well as have huge storage power. On another level, when it comes to security and event management, there has been a huge spurt in the need for the right infrastructure and tools in place to combat security and data breaches. The recent ransomware attacks, have increased the number of threats coming from unpredictable avenues such as IoT devices that may not have the same level of security protection as most computers.
Q5. What can businesses do to ensure a Big Data Analytics framework that is both agile and scalable?A5. Businesses must continue to build robust frameworks for their IT environments that are flexible for future scalability. These should be able to segment useful data from clusters of data available (both organised as well as unorganised data). This analysis needs to be done by talented personnel who have the skills to make sense out of big data. This is the reason businesses need to invest in getting the right skill set for analysis – right from recruitment to training. All these are useless unless businesses have the right infrastructure to handle and store all the big data.
Q6. What concerns must be addressed about privacy and integrity of big data?
A6. With the recent enforcement of GDPR policies, there is greater awareness of the rights of consumers. Since the advent of the internet until now, most consumers who access certain software or websites blindly think that their information has been in safe hands. But as we all know now, this hasn’t entirely been true. Sometimes, consumers have to waive away any privacy rights when they agree to the software license terms that they accept before using any software. GDPR has pushed companies to be more transparent on these policies, and be clear on how consumer data will be used or maintained by them.