If you listen to enough tech conferences, vendor presentations, and analyst statements, you might believe that every organization on the planet can now detect and respond, in real time, to events that occur in the milliseconds before .
This is not the case. At least not yet. But this quest for real time answers important questions. All the exciting new technologies appearing on the tech scene – artificial intelligence, predictive analytics, embedded systems, application streaming, real-time location monitoring or alert systems – all depend on real-time technology to work. However, these projects are most often still in progress.
Worse: sectoral studies show that real time is more of a dream than a reality. For example, in supply chain management, while 77% of executives seek real-time visibility of shipments, only 25% of them currently use it, according to a Tive survey. Similarly, only 23% of companies surveyed in a survey by Unisphere Research and ChaosSearch say information is available in real time.
“Most businesses don’t need real-time data”
“Most businesses don’t need real-time data,” says Nick Amabile, CEO of DAS42. The question is above all whether the need is operational or analytical.
“Operational systems often need real-time data for information security use cases, such as security threat monitoring, customizations in marketing, logistics, cost optimization , improving customer experience, fraud detection and commercial strategies,” explains Mr. Amabile.
Analytical needs, on the other hand, may have a certain degree of latency.
The question of the cost of infrastructure
“For analytical use cases, we first define a Service Level Agreement (SLA) for acceptable latency,” says Amabile. “User-facing reports may need to be real-time, while management-facing reports may be hours old. Stakeholders often request real-time data and reporting where batch processing may still be acceptable.”
Managers may also want to be selective about the need for real-time, because implementing it also means setting up expensive IT infrastructure.
“There is a wide disparity in the level of readiness for real-time deployments between different organizations,” says Tyson Trautmann, vice president of engineering at Fauna. “Large companies, and particularly those that are technology-driven, such as in finance, e-commerce, and technology services, often have robust infrastructures capable of processing data in real time. But these capabilities have often been built by adding complex layers on top of existing products that did not natively support real-time data.” This also results in a “high operational burden,” he adds.
Moving analytical data at lightning speed from source to system
So does the additional cost of work and budget justify the move to real time? “The infrastructure and complexity of building, running and operating real-time systems often does not match the benefits of moving from batch processing to true real-time,” says Amabile. “Often, near real time is just as useful as real time.
Since real-time also means moving analytical data at lightning speed from source to system, care must be taken to ensure that this data is verified and trustworthy.
“Growing data volumes have created complexities for businesses on the governance, management and measurement side, often with data sets coming from many different sources,” says Sam Pierson, senior vice president at Qlik. “It is essential that organizations have a robust data strategy and infrastructure in place to ensure that the most recent data, from valid and reliable sources, is what is used in real time. decisions may lead to erroneous results.”
The big question of data quality applied to real time
Data quality issues need to be addressed up front. “With real-time data, there is often less time to clean and prepare the data before it is used,” says Trautmann. “This can lead to decisions based on incomplete or inaccurate data, which can lead to poor outcomes.”
The issue of real-time trust “takes on even more importance in a world where there is growing interest and use of generative AI,” Pierson says. “Being able to trust the data provided to employees, knowing with certainty that it is valid and appropriate for how it is used, is essential to maintaining regulatory compliance as well as data security and governance, while enabling decisions in the moment that produce the right impact.”
A real-time or streaming system that works well and is trustworthy “requires complex architecture, infrastructure, and programming skills that are beyond the scope of a typical data science or engineering team “, explains Mr. Amabile. “In addition, there are many other considerations that need to be taken into account regarding production and deployment, monitoring, governance, security and integration between business applications, customer-facing applications and analysis systems.
Going through the cloud for real time?
The good news is that there are tools and platforms that make real-time more tangible, even for small and medium-sized businesses with limited IT budgets. Over the past decade, “the emergence of new real-time infrastructure offerings has enabled a much wider range of organizations to take advantage of real-time capabilities,” says Trautmann.
“Cloud service providers like Amazon Web Services, Google Cloud and Microsoft Azure have implemented managed services tailored to real-time processing, including data streaming and real-time analytics services,” adds -he. “The rise of distributed, in-memory, and time-series databases addresses the need for real-time data workloads. Open-source offerings like Apache Kafka, Apache Flink, and Apache Storm have further enriched the ecosystem real-time data processing.”
Additionally, “the growth of edge computing has also improved real-time processing, particularly for IoT applications, while the potential of 5G technology for lower latency and higher data processing capabilities opens up new frontiers for real-time applications,” adds Trautmann.
Source: “ZDNet.com”