A crowdsourced map for checking supermarket wait times worldwide
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 115 | |
| Author | ||
| Contributors | ||
| License | CC Attribution - NonCommercial - ShareAlike 4.0 International: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/58808 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
GoogolMassMultiplication signLine (geometry)Data storage deviceProjective planeCodeView (database)Level (video gaming)Auditory maskingData managementCuboidWebsiteSummierbarkeitAnalytic setElectronic mailing listGraph (mathematics)VideoconferencingInstance (computer science)RadiusPoint cloudSurface of revolutionDomain nameDirect numerical simulationThomas BayesStorage area networkRight angleVideo gameMathematical optimizationQuicksortTouchscreenPresentation of a groupSelf-organizationFormal languageBuildingServer (computing)Web 2.0Revision controlFront and back endsArithmetic progressionContent delivery networkMusical ensembleSoftware testingGoogolOpen sourceIntegrated development environmentLink (knot theory)Decision theoryBitMiniDiscSoftwareChannel capacityContext awarenessCASE <Informatik>PlotterProgrammer (hardware)Web pageMereologySemiconductor memoryWeb applicationResultantQuery languageDatabaseComputer virusGodSoftware developerData centerUniform resource locatorExterior algebraError messageTracing (software)LoginCodeInformationInsertion lossMessage passingDigitizingClosed setGoodness of fitProcess (computing)Subject indexingCore dumpGradientMatrix (mathematics)Cartesian coordinate systemPattern languageElectronic program guideEndliche ModelltheorieSpeech synthesisAreaSlide ruleNumberFeedbackAverageComputer architectureFamilyOnline helpDifferent (Kate Ryan album)Inheritance (object-oriented programming)Moment (mathematics)Shared memoryPhysicalismDebuggerMeeting/Interview
08:44
VacuumWeb applicationComputer architectureGraph (mathematics)Level (video gaming)Point cloudOnline helpNumberWebsiteComputer iconElectronic program guideEndliche ModelltheorieIntegrated development environmentTouchscreenField (computer science)Cartesian coordinate systemArithmetic progressionComputer animation
10:19
Point cloudDigital signalMach's principleProjective planePoint cloudServer (computing)Graph (mathematics)Web 2.0Scaling (geometry)Software testingJava appletWeb pageArithmetic progressionContent delivery networkMobile appBitWeb applicationChannel capacityView (database)Integrated development environmentSemiconductor memoryMiniDiscRevision controlStack (abstract data type)GUI widgetComputer animation
14:37
Flash memoryData centerQuicksortData managementInstance (computer science)Different (Kate Ryan album)DigitizingEvoluteDirect numerical simulation
15:48
FeedbackFeedbackOpen sourceCodeProjective planeMultiplication signLevel (video gaming)NumberResultantVideo gameWebsiteSubject indexingProgrammer (hardware)GodCore dumpExterior algebraPoint cloudWeb applicationQuicksortPoint (geometry)Domain nameIntegrated development environmentCoordinate systemSoftware developerMessage passingError messageDatabaseTotal S.A.Tracing (software)Instance (computer science)LoginGoodness of fitThread (computing)Arithmetic progression
24:37
Link (knot theory)Open sourceEndliche ModelltheorieCodeShared memorySlide rule
25:41
Perfect groupMoment (mathematics)Inheritance (object-oriented programming)Projective planeLecture/ConferenceComputer animationMeeting/Interview
Transcript: English(auto-generated)
00:06
So, our next talk has a very different topic. We all know the problem. We go to a grocery store and have to queue in the line. We have to wait long times. And Mickey Lombardi has
00:20
a solution for that. Hi, Mickey. Hi. Hi, everyone. How's it going? How are you? I'm fine. Thank you. And you? I'm fine, too. I'm from Switzerland, streaming from Switzerland here. And where are you streaming from? I'm from Florence, Italy. Okay. So, not that far. Yeah, not that far.
00:41
Okay. So, please start your talk about the supermarket wait times. Thank you. Here we go. Well, I hope that I hope until the end that this talk wanted
01:06
to be physically in front of public. But, unfortunately, we are still in this situation. So, I cannot see a screen in front of me. But I hope to see you next time, presence,
01:21
and to meet you all. I hope to have time to answer your question. Otherwise, you will find me in the room to reply as I can. So, in this talk, I'm going to speak about on my 2020 project, an open source project, to trying to help people during the COVID-19
01:48
global lockdown that everyone affected. So, let's start the talk. Please let me introduce myself first. I am Mihi. I'm from Florence, Italy, as you may know from my accent. Actually,
02:05
I'm a full stack engineer at Growance, a digital company in Italy. And I'm also a co-founder of the Schrodinger podcast and organization in GitHub, where we are having fun on building something open source in Python and other languages. Of course, I'm in love with Python,
02:25
but especially as I am Italian, I'm in love with pizza. I will not post any link, any social link, but you can find me everywhere as the join or the join95. And if you like, you can pass on my website. And the link is mihi.com. Before to start, let's have a look
02:48
what we are going to see in this talk. I divided the talk in four steps. The first one is an intro just to give you the context of everything. And then we are going to start going deeper
03:04
and just to show you how the architecture works and how the cloud was. And then we're going to show you some APIs and especially Redis that saved my life in this project, definitely.
03:20
And then in the end, we're going to have a look into the economical part and the conclusion of these projects. I know that we are just programmers. We don't think about money, but I think that for this project, money is a little, it's something that
03:41
we need to talk about because there is a plot twist. So let's start. As you may know, in 2020, the things just fall apart. I love this GIF that I found on G5 that pretty summed the things up.
04:00
Wear a mask and get the wax is the mantra of this one year and a half. And when the pandemic hit, everyone was like panicking and no one was knowing what to do. No one was having a mask and stuff like that. So not a great big deal.
04:22
And for that everyone, especially in Italy, the first European country to have a lockdown, a national lockdown, we were just panicking and everyone started to take everything from
04:40
the supermarket, the grocery store, the pharmacies, literally everything. And when you go to, when you went to those places, you can have a look at the line and just think about how much time do I need to just enter and just to find something,
05:03
if there is something to take. I think that this issue comes to every places during the pandemic. I talk about Italy because I'm living here, but reading about the newspaper and from the video all around the world, back in the April 2020, we were just in
05:27
the same boat. So the issue was that the day following the global lockdown, every supermarket pharmacy were attacked by people in search of executes. And
05:41
just immediately after the announcement of the lockdown in the country. So just to give you a little context about this project, it wasn't a project for profit, neither for commercial use, but just to help, try to help people on staying safe and stay
06:03
away from the others, unfortunately. And because the longer you stay out and in cocktail with other people, especially in closed and unventilated environments, the more likely you are to contract the virus. So my idea for this question was that.
06:22
I just leave you the link, you can find it in the slide, just that I leave you in the Heropython site website. And of course, the project is now offline, but you can find the open source code that even the front end side and also the back end side. So what was the
06:43
idea? My idea started from Google traffic API, because I was thinking, well, people are like cars and inside the car, there is a person physically. So my idea was to collect some data
07:04
from also Google API for places to just retrieve the time of the average time that one person stay into a place. Like I'm going to the supermarket,
07:20
I stay there for 20 minutes average for Google. And I was thinking, well, that's a good metrics, but yet it's not that much of data. So my thought was, what if I realize a map with those API and also older API that we're going to look after
07:42
and just give the possibility to the end user to just give a feedback, a temporal feedback of the store where he is and just with those feedback, just try to understand how much line,
08:03
how much time we need to wait to enter those places. And this is called cloud sourcing. It's a pattern that now we can find in the modern application. And I managed to
08:21
gives the time almost correctly, at least in my local markets and also around the world by mixing up those data from the Google API and from the feedback user, from the user feedback. And so you may know, you may wonder, what's the map? The map is,
08:46
it was like that. It's showing up with just a full screen map using leaflet and just a model to have a getting started guide to the user and showing up the icon
09:03
that the user can click and how the website working and how the progressive web app were working. Giving also a frequently asked questions just to give the possibility to everyone to navigate the website and find the data that they wanted. It turns out that this website,
09:29
this application went around the globe. And this is a graph. The numbers are no matter for the numbers, but from the late April to mid June, where the website was
09:45
alive for the emergency, it collect those numbers from all around the world. And I'm so thankful to those user. And I hope that I gave a little help to those people.
10:01
Started from a field just for me, my family, my friends in Florence, and then it went kind of viral, I can say around the Europe first and then around the globe worldwide. So let's dive into the architecture and the cloud environment.
10:25
Before doing that, I cannot talk about the technology that I use, the stack. So of course, Python, the version that I use was 3.7. It's not that much important because you can run the
10:40
project in every Python major than 3.4. And of course, Flask for the API layer, Mood widget and Nginx for the web server. Netlify for the front-end side for giving the possibility to go in cloud and find the gem stack using the CDN. Redis, a big thanks to Redis
11:06
and Progressive Web App from Google. Let's back to April 2020 at the start and beginning of this journey. I was using Java Raspberry for put online everything and literally everything
11:27
into this raspberry. I love raspberry, by the way. I was running front-end, Redis server, Nginx, MOSGI, Flask, API, literally everything. And of course, for testing news,
11:46
testing propose, and for beta testing is very good. It was quite fast for me. And also, I mean, the beta testers were just friends, my mom, friends of friends,
12:03
my grandma, the local priest and so on. After that, after the first two weeks of the project life, it started to growing and went viral a little bit without doing anything,
12:22
actually. And I needed to find a way to go worldwide to give the possibility to every user to find out the place nearby them or just find the place by searching it. So I went worldwide and I needed to make a decision if go to AWS for the cloud environment
12:49
and spend all my money on that or try in different way. I choose OVH cloud and also DigitalOcean. You may probably know both of them. I choose, in each case, with a private
13:09
server with pretty the same capacity and performance, 2GB of RAM and 30GB of disk memory.
13:24
Why I decided to go in cloud? Well, because after two weeks, first of all, I got more than 100 contemporary users and more than 20,000 page view per days. So the raspberry for,
13:43
I mean, I love a lot raspberry, but it was dying. I needed to do something. And also because going in cloud, it made me to have much better performance, much better scaling horizontally, of course, availability and have a look into the future.
14:03
Because as you may saw from the graph of Google Analytics around May, where literally the world went in a lockdown, even more users can visit the website, the app. And for the future,
14:23
it means around 1,600 contemporary users and almost 90,000 page views per day from all over the world. So raspberry can't afford that, unfortunately. So let's have a look also to
14:40
a schema and how the management of those instances. As you can see from the schema on the right, I centralized Redis just for a comfortable use for me, just to have all the data into just one Redis and not just in each instance. But I had four APIs instance
15:11
with Mootooji, Nginx and Flash API layer. And of course, when I went worldwide, I decided to
15:22
buy the instance in different data centers in Digital Evolution. One in New York, one in UK and one in San Francisco Bay, just to try to serve with low DNS lookup to those domains. It's just sort of an optimization and performance. Just try to care about the details.
15:50
Maybe I say too much that Redis saved my life on this project, but it's true. And now you maybe see the why. Just to give you a recap of what the end point were, on the right, you can see
16:09
all the end points that I used for geocoding, based on here and Google API. Here is a Microsoft alternative to Google Maps. And the most important end point to get the estimated waiting
16:26
time for a place. And then all the places end points to retrieve data from for the place, such as the name, the location, the coordinates, the geocoding, everything about the place.
16:43
And of course, the feedback end point to the crowdsourcing feature that contributes a lot to make this project very useful for the user and for all the people. And in the end,
17:01
just a little end point to login activities, such as trace error or message info, just for the feedback it proposed. Of course, every user data was anonymous in compliance to the GDPR. And so don't worry about it, because I don't know who was on the website or where.
17:27
And on the API side, thanks to Muvintji, everything was multi-threading. And of course, this is a good improvement for the performance. And this is a very good feature from Muvintji
17:45
that is also implemented in Flask. Talk about Redis. I decided, I tried a lot, different solution to make things workable with a good performance. And we were talking about 30 second
18:02
timeouts for most of the time. When I found out the good solution, it comes with Redis. And Redis saved my life by using the geospatial index and geospatial commands, such as geladd, gel radius, that's for searching purpose, and for setting also a caching layer
18:28
in Redis. And as you can see from the right side, I put a little snippet of code. It's the core of the project. It's basically, getPlacingRadius is basically find
18:45
the waiting times in our radius, starting from our coordinates. And the coordinates were from the user. So even if a place or just the nearby places of the user.
19:05
And Redis is very, very fast. This query, it took just, I don't know, maybe less than a second to retrieve the result. And it can have up to 150 results. And it's a lot
19:21
to make, to put a market into the map. 150 market on the map are quite a lot, trust me. So on every request that the user made to the backend, to the API layer, every request was
19:42
on Redis database and cached in Redis database with a TTL. Just to give the performance at its best for the user. Because with those large amount of data, I never saw
20:01
that amount of user in just one progressive web app or just a website that I made. So I was worried about it and say, okay, well, what if I did not return the correct result or the request starting to go in timeout? Oh my God. So I found this way, this solution,
20:27
it was the correct way to deal with those amount of data and with those amount of traffic from my side. Let's have a look also about the cost. Because I talk about all this data
20:45
and all these performance issue, but also about the traffic. And I know it's not, sometimes it's not a real topic for a programmer to look into the money, but it's something that
21:04
we should take care about to have also more knowledge to build this kind of project. And I hope that I would be a sort of inspiration for you or for the new developer and especially for the open source community.
21:21
So let's have a look of this entire journey from a Raspberry Pi to a cloud environment. Apart from having a couple of Raspberry Pi actually, the cost in 2020 was around $30 maybe because it was before the Raspberry Pi 4 came in. And so I can say that the cost was
21:52
around $30. And also for the domain, I say that it wasn't for commercial use or
22:02
any kind of money proposed. So I reused one of my domain and just create a subdomain on that because it was just for personal use at the beginning and then half the nowhere it went viral and worldwide. This was the prerequisite. And after that,
22:26
for the cloud part, I spent about $25 per month for the cloud usage for the four instances in DigitalOcean and OVH. And the total is around $75 for three months of usage.
22:45
It's quite a lot, if you think, but for those data, for the goal of this project, it seems good and fine to me. The proposal was higher than the money.
23:00
Also, a little note about this. The traffic was quite sustainable for this project, even if there are so many users on the page, maybe for the caching layer. But anyway, even in DigitalOcean and also in OVH, the traffic inbound and outbound was free until
23:24
10 gigabytes. So that was a big advantage for me just to find out and try to make everything better and better every day. And in the end, having also the front-end side that I'm not
23:42
talking about in this speech, but there was. On cloud, on a CDN, it allowed me to satisfy more than 1 million of users and more than 2.1 million of sessions in three months of life with more than 1,500, on average, contemporary users. And those are very big numbers for me,
24:09
no matter, I mean, they're just numbers. The purpose of the project was higher than money
24:21
and number. I'm very happy to have the chance to help the people from all around the world and also people of the open source community help me to build this. And I would like to thanks also one of them, that is Daniel Defoe from Canada for helping me at the beginning of
24:44
this journey for the accessibility on the website and onto the design of the first model that you saw in the previous slide. And I would like also to thank the open source community that is giving me almost everything I can say. And I'm very happy to give it back
25:07
as much as I can. And of course, again, I'm really pleased to have the chance of helping people all over the world. And I hope that with this talk, I can, in my little,
25:22
inspire someone else to do the same and share the code, share the knowledge, help the next one. And yeah, and of course, the code is still available on the GitHub as the link show you into the slide before and after that, thank you for listening. If there are any questions, thank you.
25:55
Yeah, thank you very much. Very interesting talk. Very interesting project. I hate Qs too,
26:02
of course. Perfect solution. So, I see people are typing questions. There is already a first question about geocoding. Why didn't you use geocoding from OpenStreetMap? I think you used Google and what was the second? Here. Here, yeah, exactly. Yeah, OpenStreetMap was having
26:25
some issue for geocoding for the same place compared to Google and here. So, I decided to go safely to those API of Google and Microsoft for just data alignment for the end user. So,
26:45
that's why. Okay, at the moment, there are no more questions coming, but comments like super inspiring, very inspiring talk. Thanks. Oh, thank you guys. So, I think there are no
27:06
more questions at the moment. So, thank you very much again. Thank you too. Have a great day. Have a great day.