Applications, servers, cloud infrastructures, IoT & mobile devices, DevOps, microservices architectures – the biggest business and IT trends are helping us improve operations and customer experience like never before. But, these trends have also resulted in an explosive growth of machine-generated data, which includes logs and metrics such as user transactions, customer behavior, sensor activity, machine behavior, and security threats. This data is complex, but also the most valuable as it contains operational intelligence for IT, security, and business.
Log analytics involves searching, analyzing, and visualizing machine data generated by your IT systems and technology infrastructure to gain operational insights. Traditional data analytics tools are simply not built to handle the variety and volume of rapidly proliferating machine data.
Why is log analytics important?
Log data is continually growing. Without a holistic, cost-conscious solution, costs will just continue to grow unchecked. Human data and machine generated data is growing with a tremendous speed, where human-generated data has a generally conservative growth rate of 10x over business data. Machine data is projected to grow even more.
What is the history of log analytics?
Since we started creating computer-generated records, we’ve been trying to analyze those records in bulk. The records are emitted by devices, applications, networks (and more), and are then time-sequenced into logs. Many times these logs are not fully documented or consistently formed across applications or devices, which further builds the case for log analytics.
Where will the log analytics market go next?
Log data rarely disappears—it is just stored and accessed differently. As the volume grows, we need ever more sophisticated methods to store that data and make sense of it. Machine learning is being used to find patterns in this growing pile of data, but there is much more that can be done to aid security analytics, fraud detection, anomaly detection, and more.
What are the benefits of log analytics?
Log analytics can answer valuable operational analytics questions, like the list below. Imagine if you had real-time answers to these questions:
- Is my infrastructure working?
- What is the latency and error rate?
- What caused my application issue?
- Is there any suspicious authentication activity?
- What data was accessed by this IP address?
- Are there instances of fraud?
- What content/products are my users interested in?
- Which features are used most or least?
- What user are most active and why?
What are the challenges of log analytics?
Some of the key challenges facing log analytics tools include:
- Data variety and data volumes are increasing rapidly
- Security and cost are critical requirements
- Real-time and predictive results cannot be easily added to older architectures
Who uses log analytics?
Primary users for log analytics are DevOps engineers, site reliability staff and enterprise architects.
What does log analytics do?
Real-time application and infrastructure monitoring
Capture and centralize all logs and metrics from your applications and IT silos to get deep visibility into your application and infrastructure stack and ensure uptime. You need to index the data, making it available for analysis in real time, which allows you to visualize the performance metrics in real-time.
Root-cause analysis
Quickly identify the issues across your environment (servers, code) to reduce the mean-time-to-identification (MTTI) and mean-time-to-resolution (MTTR). With visualization tools, you search through millions of events and correlate across your applications and infrastructure to quickly diagnose the root-cause of the problem, improving uptime.
Clickstream analytics
Get a real-time view of the performance of your web content and user interaction with your applications and websites including user behavior, amount of time spent, popular content, and more. You can aggregate and analyze your clickstream logs effortlessly to gain a deeper understanding of your customers.
Security intelligence and event management (SIEM)
Centralize and analyze events generated across your entire environment including applications, networks, and operating systems to identify any malicious or suspect activity in your network. You can index the data as soon as it is ingested, allowing you to analyze data from multiple sources instantly and find and prevent threats faster.
How to create a log analytics strategy
Your log analytics strategy needs to address data ingest, transformation and enrichment, indexing and sharding strategy, infrastructure planning, and finally data lifecycle and archiving of data. Here are the general steps to take:
- First, you need to identify data motions or data ingest. Essentially, you have to figure out an ingest pathway.
- Second, you need to set up data transformation of log lines or strings. Log analytics many times work with JSON– something has to transform the data appropriately and enrich if needed.
- Third, you have to figure out an indexing and shards strategy. Proper creation of indexes is essential.
- Fourth, you also have to do some infrastructure planning to figure out the instance type and how many of them you need.
- Finally, to control log size and cost, you need a holistic strategy for data life cycle and archiving.
Where should I store my log data?
Amazon OpenSearch Service provides a variety of storage tiers for your log data. You can select a storage tier that aligns with your query requirements (Hot, UltraWarm, and Cold Storage).
What are AWS offerings for log analytics?
Deploy and manage with ease
Amazon OpenSearch Service makes it simple to set up and deploy your cluster, while removing the complexity associated with management tasks, such as hardware provisioning, software installing and patching, failure recovery, backups, and monitoring, allowing you to reduce operational overhead and focus on core business requirements.
Get scale and reliability
With up to 3 PB of instance storage in a single domain, Amazon OpenSearch Service lets you easily add or remove instances without any downtime. The service also offers built-in encryption at-rest and in-transit, user authentication, and VPC support, allowing you to keep your data secure.
Integrate easily with other AWS services
Amazon OpenSearch Service offers built-in integrations with other AWS services such as Kinesis Data Firehose, Managed Streaming for Kafka, IoT, CloudWatch Logs, KMS, Cognito, and IAM, so you can securely ingest, analyze, and visualize data from all sources.
Lower your costs
With Amazon OpenSearch Service, you pay only for what you use. There is no upfront fee or usage requirement. With 24×7 monitoring and AWS support, you don’t need a team of Elasticsearch experts to scale, secure, and monitor your infrastructure, resulting in lower total cost of operations.
How does log analytics with AWS work?
With AWS, you can build different solutions to effectively consolidate, monitor, and analyze your log data. These solutions provide you a streamlined view of your applications, systems, and AWS log information for real-time operational intelligence.
Centralized logging using Amazon OpenSearch Service
In combination with other AWS services, this solution powered by Amazon OpenSearch Service provides you a highly available, turnkey environment to quickly begin logging and analyzing your AWS environment and applications.
The diagram below presents the centralized logging architecture.
Real-time monitoring using Amazon Kinesis
Using Amazon Kinesis along with AWS CloudTrail and Amazon CloudWatch, this solution enables you to build a serverless solution to monitor your applications in real time.
The diagram below presents the real-time application monitoring architecture.
How are customers implementing log analytics?
Real-time application and infrastructure monitoring
Capture and centralize all logs and metrics from your applications and IT silos to get deep visibility into your application and infrastructure stack and ensure uptime. Amazon OpenSearch Service indexes the data, makes it available for analysis in real time, and allows you to visualize the performance metrics in real time using Kibana dashboards.
Expedia Group, one of the world’s leading travel companies, uses Amazon OpenSearch Service for application monitoring. Amazon OpenSearch Service enables Expedia to monitor large volumes of Docker logs cost-effectively, identify and troubleshoot issues in real-time, scale easily to accommodate additional log sources, and offload their operational overhead.
Root-cause analysis
Quickly identify the issues across your environment (servers, code) to reduce the mean-time-to-identification (MTTI) and mean-time-to-resolution (MTTR). With built-in Kibana, Amazon OpenSearch Service lets you search through millions of events and correlate across your applications and infrastructure to quickly diagnose the root-cause of the problem, improving uptime.
Autodesk, a leading provider of 3D design and engineering software, uses AWS services including Amazon OpenSearch Service, Amazon Kinesis Data Firehose, and Amazon Kinesis Data Analytics to build a cost-effective unified logging solution to find and fix application issues faster and improve customer experience.
Clickstream analytics
Get a real-time view of the performance of your web content and user interaction with your applications and websites including user behavior, amount of time spent, popular content, and more. Using Amazon OpenSearch Service and Amazon Kinesis Data Firehose or Amazon Managed Streaming for Kafka, you can aggregate and analyze your clickstream logs effortlessly to gain a deeper understanding of your customers.
Hearst Corporation, a large media company, built a clickstream analytics platform using Amazon OpenSearch Service, Amazon Kinesis Streams, and Amazon Kinesis Firehose to transmit and process 30 terabytes of data per day. With this platform, Hearst is able to make the entire data stream—from website clicks to aggregated data—available to editors in minutes.
Security intelligence and event management (SIEM)
Centralize and analyze events generated across your entire environment including applications, networks, and operating systems to identify any malicious or suspect activity in your network. Amazon OpenSearch Service allows you to index the data as soon as it is ingested, allowing you to analyze data from multiple sources instantly and find and prevent threats faster.
