Streaming project, 1 of 3
Tower Health Stream
A streaming pipeline that watches simulated telemetry from 10,192 real Egyptian cell tower locations.
Live replay in your browser
10,192 real tower locations from OpenCelliD (CC BY-SA 4.0), each reporting every 5 seconds. The event generator and alert rules are ported from the project code.
Problem
Telecom teams often learn about a weak tower only after customers complain. I wanted to see how early a streaming pipeline could flag it, and whether it could tell a hardware fault from bad weather.
What I built
Kafka carries tower events. One Flink job validates them and routes bad records to a dead letter queue, a second classifies anomalies and rolls them up per area in 10 minute tumbling windows. A Python service adds live weather to each alert, Iceberg keeps the history, and Grafana shows it all. Airflow and dbt on DuckDB handle the batch side.
Result
On a local Docker stack it kept up with 2,038 events per second and raised alerts in under 150 ms.
The numbers, and where each one comes from
OpenCelliD export for Egypt (MCC 602), kept the four national operators and dropped rows with missing coordinates or zero range.
from codesimulator/data_cleaner.pyThe simulator sends one event per tower every 5 seconds, so 10,192 / 5 = 2,038 events per second, sustained on a local Docker stack.
calculatedsimulator/tower_simulator.pySum of the measured stage latencies from the simulator to the regional_alerts topic (10 + 50 + 20 + 50 + 20 ms upper bounds).
from READMEREADME.md#latency-breakdownKafka, Kafka UI, Flink jobmanager and taskmanager, Iceberg REST catalog, PostgreSQL, Airflow and Grafana.
from codedocker/docker-compose.ymlArchitecture
CASE
WHEN signal_strength < -90 AND connected_users > 800 THEN 'OVERLOADED'
WHEN signal_strength < -90 THEN 'WEAK_SIGNAL'
WHEN call_drop_rate > 0.05 THEN 'HIGH_DROP_RATE'
WHEN latency_ms > 200 THEN 'HIGH_LATENCY'
ELSE 'DEGRADED'
END AS alert_type
-- regional roll up
TUMBLE(event_time, INTERVAL '10' MINUTE)
GROUP BY area, operator
HAVING COUNT(DISTINCT tower_id) >= 3 -- REGIONAL_OUTAGEWhat the batch layer found
Output of the dbt models on DuckDB over the 10,192 real towers. Every build of this site reconciles these tables with the raw data and refuses to publish if the counts drift.
Share of LTE towers by operator
- WE415 towers100.0%
- Vodafone4,791 towers93.7%
- e&2,774 towers85.7%
- Orange2,212 towers85.2%
Areas by legacy risk
Risk rises with the share of non LTE towers in a location area. 117 areas (934 towers) have no LTE tower at all, a ready short list for an upgrade plan. Source: dbt model regional_risk_index.
Decisions and trade-offs
Tumbling windows for regional outages
Fixed 10 minute buckets give one clean result per window, so an outage is never double counted and the logic stays deterministic.
A dead letter queue, not a silent drop
Events outside the valid range go to their own topic, so bad data stays visible and can be replayed.
Weather as context
Every alert is enriched with live Open-Meteo weather, so the team can tell a hardware fault from a storm and respond differently.
Kafka in KRaft mode
No ZooKeeper to run, one less moving part in a local stack that already has eight services.