From Sensor to Datacenter: Streaming Protocols in Connected Public Transport Networks

Introduction

A tram in Munich updates 2,000 passenger displays. An SNCF station manager remotely opens a platform during a delay. A RATP engineer replays six months of metro telemetry to refine delay-prediction models. These scenarios share a common challenge: a modern transport network is no longer a static collection of vehicles, but a dynamic living information organism, generating a continuous flow of events across devices, stations, and passengers.

No single protocol can address the three core constraints that define this domain—network instability, delivery guarantees, and volume at scale. Instead, transport operators layer protocols, each responding to a specific operational reality. TCP and UDP move bytes across the wire. MQTT adds resilient publish/subscribe messaging at the edge. Kafka provides a durable, replayable backbone at platform scale.

This article follows that architecture from sensor to datacenter. It begins with the limits of the batch-oriented world these systems replaced, then descends into the transport layer, climbs through edge messaging and distributed event streaming, and returns to the operational and business outcomes that justify the design. Throughout, one theme recurs: protocol and architecture choices are dictated by physical and operational constraints, not by trends.

 

The Batch Baseline and the Streaming Shift

Batch processing has long defined data workflows in systems where periodic updates suffice. Nightly ETL jobs reconcile payroll records, monthly reports summarize inventory, and fixed-interval workflows serve audit-heavy processes well. These systems thrive on fixed intervals, where data is processed as a complete set—think of a train schedule updated once per day, or a tracking app that refreshes every hour. For non-time-sensitive tasks, this « snapshot » approach is reliable and auditable, and it remains appropriate for many production workloads today.

The limitations emerge when decisions must be made in motion. A passenger waiting at a bus stop finds an hourly-refreshing app useless. If a train enters a tunnel and loses connectivity, a disconnected display might show outdated departure times until the next batch cycle. A delayed train might only appear on a passenger’s app hours after the fact, leading to missed connections and eroded trust. Batch systems prioritize consistency and completeness; dynamic, event-driven operations demand immediacy and responsiveness.

Streaming addresses this by replacing fixed « buckets of data » with a continuous flow of events—discrete occurrences such as « Bus 10 arrived at Stop A » or « Train 223 delayed by 15 minutes. » More precisely, streaming is the continuous ingestion, processing, and delivery of unbounded event streams, where systems respond to data as it happens rather than waiting for fixed-size batches. It is useful to keep three related ideas distinct: event-driven architecture, in which components react to events; data streaming, the movement of continuous event data; and stream processing, computation over those streams, often stateful and windowed.

This shift is not merely a change of tooling. Streaming introduces a practical problem batch systems did not face: the data is not a fixed set that ends. It is an unbounded series of events, continuous in time. That changes the engineering question from « Can we process this file? » to « How do we handle this flow without letting lag, state, or backpressure collapse the system? » Answering it requires physical transport before any message-level routing can exist—which is where the architecture begins.

 

TCP and UDP: The Invisible Foundation

Before a passenger display can subscribe to a departure update, bytes must move reliably across a network. Modern railway communication systems rely on TCP and UDP for this transport, and railway architectures often distinguish two data types by their delivery needs:

• Process Data (PD): Periodic updates for real-time functions such as speed monitoring and door status.
• Message Data (MD): Event-driven records for alarms, diagnostics, and maintenance.

TCP ensures reliability through acknowledgments, retransmissions, and ordered delivery. This guarantees data integrity at the cost of latency and higher bandwidth usage. Message Data in many railway architectures uses TCP for exactly this reason.

UDP, lacking built-in delivery guarantees, prioritizes speed and suits periodic Process Data on controlled onboard networks, where low latency outweighs strict reliability and a missed sample is quickly superseded by the next one. UDP does not handle congestion control or duplicate suppression, making it unsuitable where guaranteed delivery matters. It is worth qualifying the boundary here: on internal train communication networks (as profiled in standards such as IEC 61375), UDP has a natural role, but edge-to-cloud telemetry over cellular uplinks almost universally uses TCP-based protocols—MQTT over TLS, HTTP, or WebSocket—to navigate carrier NATs, stateful firewalls, and packet fragmentation.

These transport protocols move bytes from one endpoint to another. They do not decide what those bytes mean to a passenger display, a maintenance system, or an analytics pipeline. They cannot answer the questions that matter at the application layer: How does a device rejoining the network after a tunnel blackout obtain its current state? How are outdated messages discarded? For that, a higher-layer protocol is necessary. TCP and UDP are necessary, but not sufficient.

 

MQTT: Surviving the Edge

In 1999, Andy Stanford-Clark and Arlen Nipper designed MQTT to monitor a pipeline over unstable satellite links. Its lightweight publish/subscribe model treats network unreliability as the norm, not the exception—which is precisely the condition public transport devices operate under. Trains enter tunnels, displays lose power, and sensors face intermittent connectivity. As the previous section established, MQTT normally runs over TCP, adding a messaging layer that turns unreliable connections into manageable state.

A departure screen is not a database client. It is a small device on an unstable network that must know the current state quickly, without waiting for the next scheduled message. MQTT provides the mechanisms that make this possible:

• QoS Levels, which describe delivery guarantees at the transport level rather than application-level correctness:
  • QoS 0 (at most once): For high-frequency telemetry such as vehicle position data.
  • QoS 1 (at least once): For alerts, where occasional duplication is acceptable.
  • QoS 2 (exactly once): For critical commands requiring no loss or duplication, at higher overhead.
• Retained Messages: A new subscriber immediately receives the last known value for a topic. This is what allows a screen reconnecting after a tunnel cut to obtain the current departure state without asking every producer to resend its history.
• Message Expiry (MQTT 5): Outdated messages, such as an expired departure time, are discarded automatically rather than delivered stale.
• Persistent Sessions: A reconnecting client resumes its subscriptions and continues from where it left off, without re-publishing all data.

These mechanisms map directly onto the edge’s constraints, as production deployments show:

• SWM Munich: Since 2018, the ZuMPA passenger information system has used MQTT 5 to manage roughly 1,000 displays across roughly 2,000 stops. Message expiry keeps screens current, while retained messages provide instant state recovery after outages. This scale is modest compared with an entire national network, but it is large enough to show why transient client/server polling and ad hoc messaging become hard to operate.
• SNCF Gares & Connexions: MQTT Sparkplug B enables remote station management, allowing platforms to open safely during delays. Sparkplug B standardizes topic namespaces and payload encoding, ensuring interoperability between station systems.

MQTT is built to survive tunnels. But it is not designed to be the company-wide durable backbone for every downstream analytics and operational consumer. For that, the architecture needs a different mechanism.

 

Kafka: The Distributed Log at Scale

MQTT excels at the edge, but its messaging model is transient: once messages are delivered, the broker does not retain them as a long-lived, replayable history. Transport networks increasingly need exactly that—durable event streams that many independent systems can consume, replay, and reprocess. This is Kafka’s design point.

Kafka is often described as an event streaming platform. It can be used as a messaging system, but its core abstraction is a distributed, durable commit log rather than a transient queue that removes messages once consumed:

• Immutable append-only log: Messages are written sequentially and retained for a configurable period.
• Partitions: A topic is split across partitions, enabling parallel processing and horizontal scaling. A « vehicle telemetry » topic might be partitioned by vehicle ID so that consumers process independent streams.
• Offsets and consumer groups: An offset is a position in a partitioned log. Consumer groups decouple producers from consumers—one group can power real-time displays while another feeds analytics, each tracking its own offset.
• Replay: Consumers can rewind and reprocess past events, a capability absent in traditional brokers and essential for training delay-prediction models or debugging historical incidents.

Consider a national operator whose onboard gateways report telemetry from hundreds of vehicles and thousands of sensors. Even after edge filtering, the aggregate stream across vehicles, stations, tickets, sensors, and displays can reach hundreds of thousands of events per second. MQTT is designed for efficient edge pub/sub, not for acting as the long-lived, replayable backbone at this scale. Kafka is commonly introduced as a durable, horizontally scalable buffer that decouples ingestion from processing, letting downstream systems—maintenance, analytics, passenger information—consume data at their own pace.

Kafka does not automatically make a system « real-time. » Consumer lag—how far behind a consumer is relative to the latest event—and processing design determine actual freshness. And Kafka’s power comes with operational complexity, which is where platform engineering begins.

• RATP: The Paris metro operator uses Kafka as a central event streaming backbone for real-time passenger information, with historical replay used to refine delay-prediction models. (This deployment is described in a publicly available Confluent-hosted presentation rather than a peer-reviewed case study; specific production metrics are not independently documented.)

 

Platform Engineering: Scaling and Resilience

A distributed log solves ingestion and fan-out, but a production platform must also connect to legacy systems, tolerate failure, and remain observable. Several patterns recur.

Change Data Capture (CDC) bridges existing databases to event streams. A rail operator’s inventory system, for example, can be turned into a stream of « carriage assigned to train » events without rewriting the underlying database—turning stored state into a flow the streaming platform can consume.

Eventual consistency is the accepted trade-off in distributed systems. If a vehicle’s location is reported by multiple sensors, the platform must reconcile conflicts while keeping downstream consumers such as passenger apps functional. Temporary discrepancies are tolerated in exchange for availability and throughput.

Resiliency is defined by concrete failure modes. What happens when a train enters a tunnel, a display restarts, a consumer crashes, or a consumer group rebalances? Idempotent consumers absorb duplicates: if a display reconnects and receives a repeated « Train 223 delayed » message, it applies the update only once. Local reconciliation and careful offset management keep the system consistent across reconnects and restarts.

Observability replaces the simplistic « Is the server up? » question with operational signals: consumer lag, topic throughput, broker health, connection counts, reconnect rates, publish rate versus consume rate, end-to-end event freshness, and device heartbeat status. At RATP, for instance, consumer-lag monitoring helps keep displays within a bounded delay of real-time data, balancing latency against resource use.

Infrastructure realities flow directly from these patterns rather than a generic spec sheet. Kafka relies heavily on sequential disk writes, operating system page cache, and network throughput; disk speed matters, but so does retention policy. RAM supports caching and stream-processing state. CPU affects compression, serialization, and message transformation. MQTT’s edge footprint is lighter, but its monitoring requirements can be higher, because the network itself is the failure domain.

 

Operational Outcomes and Business Value

When data integrity, offset management, and edge recovery are stabilized at the platform level, real-time data ceases to be an operational liability and becomes a reusable asset. The same event stream that powers departure boards can feed analytics dashboards, drive maintenance schedules, and personalize mobile notifications. Operational success is measured not by server uptime but by whether the right information reaches the right consumer in time.

• Live passenger maps update as vehicles move, reducing missed connections and improving trust in the service.
• Predictive maintenance, triggered by anomaly detection in sensor data, reduces unplanned outages by flagging faults before they escalate.
• Remote platform access during delays, enabled by real-time telemetry, improves operator response times and safety.

These outcomes are the competitive advantage of immediacy: faster decisions, safer operations, and a passenger experience that aligns with the pace of real-world events—reused across many consumers rather than rebuilt for each.

 

The Future: Edge Intelligence and Architectural Constraints

The next phase is not simply « more streaming. » It is moving more processing closer to the source—on vehicles, at station gateways, and in edge nodes. That reduces the volume of raw telemetry that must reach the datacenter and improves response time for safety-critical functions such as anomaly detection or door-status validation. But it also raises new platform questions: firmware distribution, edge observability, secure device identity, and how much state should live outside the central log.

Edge computing redistributes processing without eliminating centralized systems. A tram’s onboard node might preprocess vibration-sensor data to flag anomalies before transmitting summaries over MQTT, preserving network capacity while Kafka retains a durable history for diagnostics. In SNCF’s station IoT systems, edge nodes preprocess HVAC and lighting data, sending only aggregated metrics upstream. This reduces datacenter load but demands robust synchronization to avoid discrepancies. If a vehicle’s edge node loses power, it must reconcile local decisions with the distributed log—which is why the same patterns that stabilize the core, such as idempotent consumers and local write-ahead logs, reappear at the edge.

AI and machine learning extend event-driven architectures into predictive domains, but they inherit the platform’s constraints rather than transcending them. Kafka’s durable, timestamped streams provide training data for models that forecast delay propagation or passenger flow; MQTT’s real-time telemetry feeds inference. The hard part is not model accuracy but operational integration: a model predicting delays must respect signaling capacity, crew availability, and station occupancy limits. Teams must monitor not only consumer lag but also model drift, feature freshness, and the consistency of edge-cloud decisions. AI becomes a constrained decision-support tool, not an autonomous intelligence layer.

 

Conclusion

Protocol choice is not a matter of preference—it is a response to physical and operational constraints.

• MQTT addresses the edge: unstable networks, lightweight clients, and the need for retained state and message expiry. It is built to survive tunnels.
• Kafka scales to platform-level demands: durability, replay, and high-volume fan-out across many independent consumers.

The deployments at SWM Munich, SNCF, and RATP show that these are not theoretical choices but engineering decisions made in production, underpinning systems millions of passengers rely on daily. As distributed intelligence moves outward to the edge and predictive models draw on durable event streams, the same discipline applies: knowing which decisions belong at the edge, which in the core, and which must remain under human control.

The future of these platforms is not defined by more data or more models, but by answering the same fundamental question that shaped their past for every new layer: what constraints does this layer solve, and what new constraints does it create? In public transport, as in all real-time systems, architecture follows constraints.