Real-time autoencoder-based anomaly detection for flight operations data streams, with live alerting and Power BI executive dashboards.
This system detects operational anomalies in real-time data streams from flight operations environments using a deep autoencoder neural network. Unlike rule-based systems, the autoencoder learns the normal behavior of the system and flags deviations automatically.
Data streams are processed at scale using PySpark for distributed computing, allowing the system to handle high-throughput operational data. The autoencoder reconstruction error is used as the anomaly score — spikes trigger immediate alerts via a Flask REST API.
Results are visualized in a Power BI executive dashboard that tracks anomaly rates, system health trends, and operational KPIs in real-time. The system was developed using domain knowledge from 10+ years of airport IT operations.
Setting up PySpark streaming pipeline, defining schemas, and handling late/out-of-order data from operations.
Building encoder-decoder architecture in Keras. Choosing latent dimension size and activation functions.
Training on normal-only data. Using statistical methods (3σ, percentile) to set anomaly score thresholds.
Integrating the trained model into the PySpark streaming pipeline for real-time scoring.
Building a Flask REST API that receives anomaly events and dispatches alerts via email/webhook.
Designing executive KPI dashboards connected to the live anomaly stream via DirectQuery.