Building AIOps Pipeline Training in Kuala Lumpur, Malaysia

This course focuses on designing and building an AIOps pipeline using widely available open-source tools. Participants will learn how to collect, process, analyze, correlate, and act upon operational data using a modular, flexible pipeline powered by open tools. The course includes hands-on exercises and practical implementations to give participants experience in constructing a working AIOps architecture.

Designing the AIOps Pipeline and Ingesting Data

AIOps Pipeline Architecture Overview

• Key stages: data ingestion, enrichment, analytics, correlation, response
• Benefits of open source in AIOps development
• Considerations: scalability, interoperability, observability stack

Data Collection and Ingestion Tools

• Collecting logs: Fluentd, Logstash, Filebeat
• Collecting metrics: Prometheus, Telegraf, StatsD
• Working with traces: OpenTelemetry basics
• Pushing data into pipelines using Kafka, MQTT, or HTTP endpoints

Data Normalization and Enrichment

• Structuring unstructured data (logs, alerts)
• Adding context: tags, topology, timestamps, dependencies
• Tools: Elasticsearch for logs, InfluxDB for metrics, TimescaleDB for time-series
• Creating reusable data schemas for downstream analytics

Correlation, Alerting, and Automated Actions

Event Correlation and Noise Reduction

• Event storm scenarios and alert deduplication
• Correlation strategies: rule-based, time-windowed, ML-based
• Tools: ElastAlert, Prometheus Alertmanager, Riemann
• Building correlation rules for multi-source data

Anomaly Detection with Open Source Tools

• Basic anomaly detection using time-series data
• Tools: Prometheus with recording rules, Grafana Machine Learning plugin, Skyline
• Applying dynamic thresholds and contextual detection
• Building alerts from anomaly detection models

Intelligent Alerting and Notification Systems

• Enriching alerts with context (runbooks, metadata, logs)
• Integrating with alert routing tools
• Sending alerts to email, Slack, MS Teams, PagerDuty
• Creating actionable and meaningful alert messages

Automated Incident Response and Final Assessment

• Tools for workflow automation: StackStorm, Rundeck, Ansible
• Triggering remediation scripts based on alert type
• Sample pipeline walkthrough: From ingestion to remediation
• Final hands-on
• Review key learning points and open source best practices