Where is my bus?
Real-time Maps with Kafka, Python and Leaflet
Building a live map of London showing current bus line locations.
Everybody is talking about real-time data nowadays and indeed the need is real. Real-time availability of data is becoming a crucial success factor and driver for digitalization in nearly all areas like business analytics, financial data, cybersecurity, in the public sector, and so on.
Let’s have a closer look at two examples of which we will implement the second one in this blog series.
(1) Financial data, Real-time fraud detection
Your bank for sure owns algorithms that can protect your account in detecting fraud transactions. In the past, those algorithms were scheduled once a day giving a bulk output of possible fraud transactions. Employees need to take care of reversing malicious transactions with huge manual efforts and attackers may have caused already large damage.
The possibility to detect and stop fraud transactions in real-time as they appear would save huge manual efforts and limit the potential damage attackers could cause. Fortunately, real-time fraud detection is possible nowadays.
(2) Public sector, Real-time maps
Imagine you are on a sightseeing trip in— let’s take London — and you want to take a bus to visit an attraction like Buckingham Palace. Imagine the city offers an App that provides a live map showing the routes & locations of the current bus lines. You could use this map to locate the next best bus line and hop-on-stop to your desired destination.
This is the example we are creating during this series — a map of London with real-time location updates of three different bus lines as shown in the below picture and video.
We’ll make use of the following technologies:
- Apache Kafka as high performant low latency event streaming platform
- Python Pykafka Client to write Kafka Producers and Consumers
- Python Flask as webserver and for API setup
- Javascript Leaflet.JS for visualization of the real-time map
- HTML5 Server-Sent Events to enable the web page to automatically get updates from the Webserver/Kafka Consumer
Additionally, if you are interested in analyzing Twitter data in real-time, you can check out my recent article.
I will split this story into a series of overall 4 stories as shown in the below overview. This allows me to get into more detail of each specific part of the overall solution. As this is my first medium post ever, I would highly appreciate any kind of feedback.
- Part 1 — Introduction — What are we going to build? (this story)
- Part 2— Kafka Producer in Python
- Part 3— Kafka Consumer in Python
- Part 4— The Frontend with Leaflet JS
Have fun :-)
