Lessons learnt building a medical chatbot in Python
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 160 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/33813 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
ChatterbotSoftwareIntelCloud computingInformationNumberArithmetic progressionLevel (video gaming)Video gameWave packetDegree (graph theory)Disk read-and-write headInternetworkingInformation technology consultingService (economics)ConcentricSmartphoneCartesian coordinate systemMereologyVideoconferencingTelemedizinProcess (computing)Multiplication signTerm (mathematics)Goodness of fitComputer-assisted translationSoftware bugLecture/Conference
03:44
Coma BerenicesSoftwareIntelDemo (music)NeuroinformatikDirected graphShape (magazine)QuicksortScaling (geometry)Mathematical optimizationLattice (order)Virtual machineSoftwareMultiplication signRight angleComputer animationLecture/ConferenceMeeting/Interview
04:55
MappingScaling (geometry)Demo (music)Table (information)Mobile appoutputMultiplication signComputer animation
05:32
Stress (mechanics)Sign (mathematics)Computer chessMeasurementBit rateLimit (category theory)Maxima and minimaMultiplication signDirection (geometry)PlanningDemo (music)Source code
06:22
ChatterbotBuildingNatural numberFormal languageSoftware bugInterface (computing)Client (computing)Sheaf (mathematics)Software bugSet (mathematics)BitLecture/Conference
06:57
Query languageSet (mathematics)Figurate numberPhysical systemSoftware bugBuildingNetwork topologyReal numberNumberDecision theoryCombinational logicExecution unitBranch (computer science)Online helpLecture/Conference
08:00
Software bugComplex (psychology)Decision theorySphereNetwork topologyGoodness of fitMathematicsProjective planeRevision controlDatabaseCanadian Mathematical SocietyLogicGraph (mathematics)DiagramComputer animation
09:00
Graph (mathematics)Software bugSemantics (computer science)Content management systemHierarchyDecision theoryStreaming mediaSoftware bugDatabaseGraph (mathematics)Multiplication signNetwork topologyBranch (computer science)SoftwareGraphical user interfaceData modelCanadian Mathematical SocietySemantics (computer science)BijectionMatching (graph theory)Direction (geometry)CASE <Informatik>CuboidRelational databaseTwitterHierarchyGoodness of fitResultantBuildingQuicksortInformationEndliche ModelltheorieMappingLatent heatComputer animationLecture/Conference
11:25
Data modelProcess modelingDatabaseGraph (mathematics)RoboticsHash functionGroup actionRelational databaseData modelMultiplication signSequelMereologyINTEGRALProcess (computing)Food energySoftware bugSocial classTheory of relativitySubject indexingQuery languageLecture/Conference
13:28
Data conversionElement (mathematics)Graph (mathematics)DatabaseEndliche ModelltheorieGreen's functionBit rateChatterbotData modelChainSoftware bugSpacetimeMultiplication signOperator (mathematics)Message passingMedical imagingMultiplicationMikroblogStructural loadMathematicsEntire functionSingle-precision floating-point formatMiniDiscError messageComputer animation
16:07
Product (business)DatabaseData modelFrequencyGraph (mathematics)Proper mapData conversionReal numberHuman migrationInheritance (object-oriented programming)RoutingLecture/Conference
16:46
DatabaseResultantElement (mathematics)Human migrationLinearizationMathematicsData conversionParticle systemEndliche ModelltheorieGrass (card game)Data modelQuery languageProduct (business)TupleChainDeterminantChatterbotScripting languageEncryptionGraph (mathematics)Software bugDiagram
18:15
Grass (card game)DatabaseConstraint (mathematics)TunisExecution unitStrategy gameRelational databaseHuman migrationQuery languageOpen setArmSoftware bugLecture/Conference
19:22
Sign (mathematics)Digital signalNewsletterComputer networkEvent horizonSoftware bugWindowChatterbotBuildingFormal languageArrow of timeVideo gameNatural languageMassChatterbotDisk read-and-write headProduct (business)DemosceneSoftware bugPattern recognitionContent (media)Library (computing)Virtual machineSpacetimeSemantics (computer science)outputMachine learningRule of inferenceInfinityDifferent (Kate Ryan album)Query languageService (economics)Physical systemEntire functionMappingSet (mathematics)Function (mathematics)InternetworkingTelemedizinCartesian coordinate systemMetropolitan area networkComputer animationLecture/Conference
22:20
Single-precision floating-point formatCartesian coordinate systemMessage passingVirtual machineQuery languageDirected graphMultiplication signoutputXMLUML
23:00
outputMultiplication signDifferent (Kate Ryan album)Endliche ModelltheorieVirtual machineMultiplicationContent (media)BuildingCombinational logicWordFunction (mathematics)FreewareLevel (video gaming)Lecture/Conference
24:01
Computer fontEndliche ModelltheoriePoint (geometry)Dimensional analysis1 (number)Software testingDifferent (Kate Ryan album)MappingQuery languageLevel (video gaming)Right angleSymbol tableInsertion lossProcess (computing)AdditionState of matter
24:56
Scripting languageQuery languageCellular automatonOnline chatOperator (mathematics)Rule of inferenceSimilarity (geometry)Virtual machineOnline helpMereologyPattern recognitionBounded variationMappingDeep WebSoftware testingMultiplication signCombinational logicDifferent (Kate Ryan album)Scripting languageBitChatterbotEndliche ModelltheorieIterationPoint (geometry)Constructor (object-oriented programming)Computer animation
27:27
Wide area networkSoftware testingData modelVirtual machineScripting languageStandard deviationScale (map)Computer fileProduct (business)Connectivity (graph theory)Moment (mathematics)Physical systemElement (mathematics)Complex (psychology)Client (computing)Cartesian coordinate systemEndliche ModelltheorieProcess (computing)Mobile WebSet (mathematics)Standard deviationWave packetPoint (geometry)Web crawlerSoftware testingVirtual machineWordValidity (statistics)1 (number)SpacetimeMultiplication signInfinityCodeMassoutputMathematicsService (economics)Network topologyScaling (geometry)Staff (military)Formal languageBit rateCombinational logicWritingTraffic reportingSearch engine (computing)WebsiteContent (media)Office suiteNatural languageNumberInformation retrievalGame controllerMathematical optimizationSampling (statistics)AuthorizationInteractive televisionMereologyStructural loadData conversionQuery languageChatterbotWrapper (data mining)Electric generatorSoftware bugDisk read-and-write headDependent and independent variablesPerfect graphDifferent (Kate Ryan album)Front and back endsGoodness of fitBinary decision diagramPrototypeLecture/Conference
34:04
Macro (computer science)Connectivity (graph theory)Type theoryValidity (statistics)Object (grammar)String (computer science)CodeArray data structureComputer programmingClient (computing)Mobile WebMusical ensembleSource codeJSONXMLUML
34:51
Cone penetration testRootMassMatching (graph theory)Electronic mailing listMultiplication signMathematicsComputer animationLecture/Conference
35:45
MathematicsSoftware frameworkStructural loadCuboidInternetworkingWhiteboardWeb 2.0Device driverCombinational logicMathematicsDatabaseComplex systemWorkloadAxiom of choiceCAN busElectric generatorFrame problemMassComputer animation
36:33
Physical systemDirected graphWorkloadLevel (video gaming)Video gameError messageChemical equationLastteilungServer (computing)Latent heatFigurate numberLecture/ConferenceComputer animation
37:34
VacuumWechselseitige InformationLine (geometry)Message passingGraph (mathematics)RandomizationMereologySchlüsselverteilungSurvival analysisFigurate numberPublic key certificateServer (computing)Service (economics)EncryptionXML
38:08
Event horizonRead-only memoryLeakModule (mathematics)Beat (acoustics)Rule of inferencePersonal digital assistantPredictionSemiconductor memoryLatent heatElectric generatorMultiplication signImplementationWeb 2.0IterationServer (computing)CoroutineSoftware frameworkPauli exclusion principleFunctional (mathematics)Statement (computer science)MehrprozessorsystemNP-hardData managementBasis <Mathematik>Generating functionError messageSpacetimeContext awarenessPredictabilityDependent and independent variablesBuildingObject (grammar)Characteristic polynomialNatural languageArithmetic meanProteinVideo gameFrequencyEvent horizonService (economics)Remote procedure callJSON
42:15
Moment (mathematics)Context awarenessChatterbotLecture/Conference
43:17
Figurate numberGenderComplex (psychology)Pattern languageLecture/Conference
Transcript: English(auto-generated)
00:05
Brilliant, so my name is Wilhelim, I work at Babylon Health and it's a real privilege to be here, so thank you very much for having me. So to provide an affordable and accessible healthcare to everyone on
00:23
earth, that is the mission of the company I work for and it's been a privilege of the last two years to spend the majority of my waking life trying to solve this problem and when I joined two years ago we were a really small company
00:43
there were like you know 40 of us and we really kind of honed in on the accessibility part of this statement because at the time if you were sick you would somehow have to physically transport your sick body to
01:01
some like health epicenter where there's a high concentration of healthcare professionals that can help you with your illness and in an age where everything from information to cat videos was highly highly accessible it just seemed well within our reach that we could make healthcare a significant
01:25
significant degree more accessible by leveraging the internet smartphones and modern technology. So we went for it and in July 2015 Babylon launched their
01:41
answer to this accessibility question with their telemedicine application that essentially allowed people to have a GP consultation with doctor over the phone and the whole kind of healthcare service that goes around that. So after
02:01
the team kind of recovered from literally doing like blistering hot fixes while our CEO is stepping up onto the stage at the Royal Albert Hall everyone relaxed pat themselves on the back for making healthcare a significant degree more accessible but the affordability part is the tricky part
02:24
because two-thirds of the costs of healthcare go to the salaries of doctors nurses healthcare professionals etc so we need to figure out how do we solve this problem how if we want to make healthcare more affordable we have to
02:44
somehow reduce the cost of healthcare professionals. So one way we can do this is by simply just increasing the number of doctors and that seems like a kind of reasonable approach but some people estimate that we are five million
03:04
doctors short from providing affordable and appropriate health care to everyone on the earth and not just that doctors are expensive to train they take a long time to train it's very it's a very difficult training process so this approach just isn't going to work for us in the short term so
03:24
affordability is really tricky we had kind of made good progress on accessibility and we just kind of put our heads down fixed bugs added features and just kind of went on our merry way but then one day Ali our CEO watched the movie and ever since that fateful day movie this question
03:59
came up again and again and again in meeting after meeting after meeting as
04:05
this kind of idea was starting to take shape in his mind and they say it takes 20 times of repetition before something starts to sink in and after his kind of relentless repetition this idea started to kind of take shape in the and in the imaginations of the engineers as well and the idea
04:26
is basically quite simple right so we know we have lots and lots of doctors short in the world and doctors are just an inherently difficult to scale but we know something that's easy to scale we know that machines and
04:41
software and computers are really you know trivial to scale so if you could just figure out a way to kind of put some of the doctor's brain inside of a computer just one computer then we can just scale it really easily and then iterate on that so this is kind of like you know it was a really kind of
05:02
daunting you know idea and you know how we're gonna do this medicine is so complex so before we did it before we did anything we built a demo we wanted to see kind of how people would take to this idea you know how people would feel like sharing you know their health care concerns with an app so we
05:20
built a demo so I was in the iOS team at the time when we built this demo and
05:43
while I was building this demo I kept saying to Ali like Ali I don't know how we're gonna build this I don't know like the stuff that we're kind of suggesting we can do in this demo like I've got no idea how we are going to do this but the reception of this demo was so amazing people loved
06:01
it people were so encouraged there are people thinking in this direction and the implications it has for health care around the world that was just that that gave us kind of enough motivation and inspiration to not knowing how we're gonna do it just kind of step in and see what we could do so I'm gonna take you through a bunch of the things we we did and we challenges we
06:26
had at the end I'm gonna talk a bit about the role that Python played in this whole journey and at the end of each section I'm going to just share with you some of the lessons we learned because some things we we did we were like oh wow that works surprisingly well and lots of things we
06:42
did we were like wow that was a really bad idea so let's start with medical triage and for those who don't know what medical triage is medical triage is essentially you arrive somewhere with a set of symptoms and
07:01
someone figures out what you should do so should you go to the hospital go to the pharmacy you stay at home you know monitor for a couple of hours like not kind of it's like the step below diagnosis so just kind of suggesting what you should do so we knew we had to do this we knew we had to be able to take medical queries and suggest to people what they should do
07:23
next and building this medical triage system was a real rollercoaster ride because we started off just being like oh you know it's just a basic decision tree you know we just kind of build it up this tree we get some doctors to help out help us and then we just kind of walk down the tree until we
07:42
get where there's no more questions to ask and we can you know give them the outcome at the end of that branch but then it's kind of like we you know we dipped the roller coaster and it was like oh yeah but this question is way more important than this question and when these two questions are answered in combination they kind of nullify each other and our minds were just
08:02
blown with the kind of complexity of this kind of medical sphere and then we kind of like come up again because we had some really good data scientists who helped us kind of like you know blow by some of these these problems and and we built it the first because essentially without kind of going into the details of how we wait questions and that kind of stuff and it
08:24
is just a decision tree that you kind of walk down with some kind of intermediary logic in between and we we built the initial version in PHP with MySQL and it just wasn't working for us like it was clunky it was
08:40
difficult to change some of the changes were quite risky so we scrapped the whole project and we rebuilt it with Python and a graph database so this is a picture of the CMS that we used for this graph database and it's a
09:01
small kind of corner of this this very very large decision stream so why did we choose a graph database for triage I would love to say that there was no hype involved in our decision but the truth is that at the time graph
09:21
databases were all the rage like even now there's a lot of hype about it and and it's solving interesting problems and we were keen to get our hands on this technology but there are some good reasons that we that a graph database is applicable for this use case so in this decision tree there's a
09:43
graph database that is applicable for branches sub-branches sub-trees etc and this kind of data model is normally a good you know probe that maybe a graph database will be applicable the next thing is that a graph database has all the semantics we need to reason about a decision tree just out the box like
10:05
that CMS that I showed you we didn't kind of build that GUI ourselves like those are just like graph database semantics where you have nodes relationships you know relationships have directions all these things that we have in our data model so reasoning about we didn't have to kind of build
10:22
this mapping from like a relational database to our kind of you know this decision tree and because the semantics kind of mostly match one-to-one and stuff like some of the tooling that comes out the box you know really it was really really helpful without a lot of work on our side the CMS being an
10:41
example of that so what are some lessons learnt the first thing I would say and this is kind of just true for all software technology is new technology is is try and try not get carried away by the high because when these new technologies come out there's lots of attention which means there's
11:02
lots of hype on Twitter and etc and if you make an uninformed decision on behalf of your company it can it can end really badly the second thing that I want to say is that and this is specific to graph databases is don't get carried away by the cool animations because I've never seen a technology
11:22
that can sell its technology so well with such like really cool animations that looks like it's solving problems that you can't even imagine solving in your kind of traditional way of working so when should you use graph databases now the single most important question that you need to answer when considering
11:44
whether or not to use a graph database is the following our relationships the most important part of my data model because if the answer to this question is maybe or sometimes or I suppose so you're probably gonna
12:02
have a bad time because if relationships are the most important thing of your data model they are just so much that you can do with a graph database now the reason this question is important is because in traditional databases like
12:21
relational databases relationships are represented or are handled with joins and joins can be expensive and I use can there tentatively because there's many kinds of joins as hash joins and merge joins and index joins and other joins
12:40
that I'm sure I don't know about and databases have done a lot of work to optimize these processes but to get the most out of it you often find yourself writing on a very specific query that makes your data model kind of quite brittle and the integration with the database quite brittle where with graph databases because relationships are first-class citizens as long as that
13:01
remains true you're gonna have a wonderful time in graph databases the second lesson we learned is that data modeling is extremely important and this is this is very true and no sequel databases but especially true in graph databases and and we unfortunately learned this the hard way
13:24
so we did our medical triage with a graph database but because that works so well we also decided to use graph a graph database for our chatbot data model so what you can see here is a a typical thing that you would find in our
13:45
database so the red node is a user the green nodes are conversations and the blue nodes are elements inside that conversation and the element being basically everything that's in the conversation that that is mostly
14:01
messages between users as well as like conversations take this updates and stuff like that ignore the purple nodes we'll just call those hack nodes for now but what's wrong with this so the first thing that's wrong with this as you can't see it from from the image but the the node most closest to
14:26
the conversation node is the most recent element in the conversation and at the end of the chain is the one furthest in the past the reason why this is bad is that every time we add new elements to the
14:42
conversation there's multiple kind of operations happening we have to delete the one relationship change the status of the other relationship and recreate another relationship and it's just a really heavy and error prone operation that happens and not only that is that it exposed some you know bugs in
15:03
Neo4j and which at the time was they fixed this but at the time when you deleted relationships didn't clear up the disk space and because we were deleting relationships like crazy eventually our our this space has got used up so what else is wrong with this and also you can't really see it
15:24
here but so the red nodes are users and initially we wanted to represent the Babylon user like you know the chatbot as a user because it felt like correct so you know everyone comes into our chatbot initially they'll be chatting
15:41
to Babylon the chatbot user and sometimes like a doctor will come into the conversation and you know but what this meant is that because most users were talking to Babylon it meant that the Babylon user node had a relationship to every single conversation node and what this means is
16:02
that every conversation node in our entire graph database was only one node away from every other conversation and in graph databases this is called a super node and super node is like really bad it's really bad for performance and it's a bunch of ways you can do to get around this but none of it feels good and you really just don't want to go down that
16:23
route and all these kind of things we learned like in production and stuff like that which you really don't want to do and we are way way more hesitant now about adopting new technologies and making sure we do kind of proper investigation before we dive into something so another reason why
16:45
data modeling is important is because in graph databases migrations are not trivial by a long shot so what you see here is this should be one linear chain
17:00
of elements in a conversation you know just you said this Babylon said this you said this it should be one chain but what you can see is that it's just craziness happening like like nodes responding with themselves and stuff like that and this is a result of a cipher query someone committed a change to one of the queries and it just messed up the way that we were
17:23
adding new elements to conversation and it was causing problems for users in production so we had to fix it now writing a migration script that fixes this is not easy because it's indeterminate and it's just really
17:40
tricky so you really want to pay a huge amount of attention to your data models you don't ever find yourself in this kind of situation so the third thing we learned is basically that the grass isn't always green on the other side so we now so the medical triage is perfect for graph databases but
18:02
chatbot not so much so we're basically migrating the chatbot data model over to Postgres and as we're kind of coming back to Postgres it kind of feels anonymous to the prodigal son returning because we feel like we've kind of you know been over at Neo4j land promiscuous and Postgres who's
18:22
just always been solid for us you know hasn't done anything wrong and we're returning now and he's just opening you know greeting us with open arms so stuff that we usually you know felt restrictive now feel you know so amazing stuff like constraints on your columns they just like you know
18:41
blowing our mind stuff like familiar querying strategies and you know sharding that is easy to do in relational databases database migrations are like amazing all the tooling that comes around it like SQL alchemy and just the general tooling around Postgres and
19:03
relational databases in Python it's just so great to come back to all that stuff so yes the the grass isn't always green on the other side okay so we have medical triage brilliant we launched it it did really really well
19:22
we pitched it against real-life A&E nurses and doctors and it just did really really well we were really proud of it and you know we kind of felt this massive sense of achievement but as Ali reminded us that we were still a long long way away from the way we wanted to be so we knew we had to build
19:42
chatbots and we have since found that building a chatbot is really difficult and the reason it's really difficult is because natural language is really difficult natural language is like one of those problems that you approach the problem and you just imagine this is gonna be hard you have
20:03
this idea here this is gonna be really difficult but as soon as you're skates touched the ice you realize you've totally underestimated how difficult this is going to be and the reason is difficult is because the scope of it it's just so huge and you enter into it you start solving one problem
20:22
and you realize oh well this means this and this means this and next minute you have the end to try to fit the entire kind of semantics and linguistic rules of and all the things that happens behind the behind the scenes into your head and trying to solve that and the biggest problem we had to overcome initially was figuring out how we could reduce the scope of
20:43
natural language understanding to something that we could take a bite at and ship the production and serve our users so what did we have to do so the first realization we had to make is that we are not interested in helping you decide whether or not you should grow a man bun like we aren't
21:02
helping with that problem we want to hone in on focusing on helping you with medical queries so that's over there over here we have a set of services so we have this very well performing medical triage we have a bunch of content you know content about different diseases and this massive
21:24
library of content we also have doctors who are ready to if our machine can't handle your query to handle your query so we have this very kind of contained scope over here and we have these services over here so when we put it like that all of a sudden it becomes a machine learning problem
21:43
because we have this infinite space of inputs and then we have this kind of set of defined outcomes and this is kind of called internet intent recognition we wanted to take the input and map it to some output that we could
22:00
do if the mapping was not good some generic fallback that would kind of redirect you to you know ask us medical queries so if it's a machine learning problem then we need data and medical data isn't easy to find but we were really lucky because we already had the system running in production called ask in our telemedicine application where you could just send
22:23
us like a single message message to a doctor and he would reply in a couple of hours with some kind of like medical advice on what you should do and the doctors and the nurses who were doing this had the good sense to tag each question and outcome so we had thousands and thousands of these
22:42
queries so by the time that we realized we wanted to do this we had this really really rich data source you know ready for us to train on so rich data source you know some machine learning scientists and engineers that's all you need right not quite because you know what the inputs that
23:06
were coming in were still free text and free text is just inherently unstructured so we had to figure out a way to to take free text and then
23:20
represent it in a way that our models could understand and our machine learning guys did a bunch of work and built building multiple different classifiers to transform the free text into slightly different combinations some were just word-in-the-bag word2vec things other things were like medical entity extracted concepts passed into models and then at the same time
23:45
they did some classifiers around our content and our outputs to to make those easy to map so essentially what they are trying to do is take a sentence and then put it somewhere on a map and then the hope is that
24:01
sentences that are very similar will kind of cluster together in this map so it's very similar to you know a point on a map on some kind of 2d map but the difference is that these maps with 380 and 80 dimensions so this is like you know if a simplified kind of set with some test data and what you can
24:23
see is that in the top left those red ones are all about your neck or something like that on the right we have some ones about hand and foot and stuff like that in the middle those queries are like ones about urination
24:41
and and pooing what I don't know what the politically correct way to say that is but you know it's unclear to our models like is this about digestion or is this about your abdomen or reproduction or anything so there's some parts that are quite hairy but they are you can see that some of the
25:02
things cluster really well together and the guys kind of iterated on their models and it got to a place where it was doing really really well it was handling our queries with 95 percent accuracy and ninety nine point eight percent safety I'll explain a bit more about what this what those percentages
25:21
mean in a bit and they use a bunch of technology a bunch of different technologies that are available to you when you when you code in Python they use tensorflow as well they use a combination of RNNs and other deep nets and I got to a place where they were really happy with their entity
25:43
recognition classifiers so we were doing really well and we were taking some of these medical queries and we were just like knocking them out the park you know we were handling them so so well but then we would get a really simple query and we would just totally bomb out we would look so so
26:05
stupid and then we would get another query that we just had an obligation not to miss but we didn't really have good data for these queries so the engineers told the machine learning guys to to pipe down you know they've
26:20
had their time we're just going to reject this problem so we didn't quite do that we use this tool called drive script which essentially is this linguistic DSL that helps you kind of construct linguistic rules to catch you know huge variations of sentences and map them to a specific outcome so this is an example this is a query about looks like some kind of abuse or child
26:47
abuse or something like that and it handles a huge variation of sentences in this one rule and then maps to a outcome that we want to provide to the users so we wrote like hundreds and hundreds of these rules to kind of
27:02
handle those like non-medical but kind of important queries we're busy actually moving away from that now because the machine learning guys have had had the chance to build things that are more generic that don't need as much data and can handle these simple queries better than these rules
27:20
can so what did we learn we learned that testing chatbots are really difficult since since doing this I've become really interested in how search engines test their search engine because you just have an infinite amount of inputs you know unlike a website where you have a bunch of buttons and it's only so only a certain amount of combinations you know
27:41
of how you can tap the buttons and stuff and you can write comprehensive tests that make you really sure that your system is working but if you have an infinite input space you can make a change and not sure like what your change has done to some corner of this massive massive input space so we couldn't
28:01
come up with like a really good way to test you know to like give us huge assurance so we just went for like many many layers of testing so the first thing we did was machine the model validations which essentially is just you know putting some of your data aside as a test set training your model and your your training set and then passing the test set back to your
28:22
model to see how well it's doing that's where the 95% accuracy comes from we then wrote just thousands of JavaScript examples examples to test that linguistic DSL thing so we just had many many many of these of these
28:43
things we also then created this kind of like client mock system that that pretends it's a client and with kind of BDD approach just just testing loads of different interactions throughout the chatbot we before making major releases
29:07
we did clinical validation tests where we would get a big number of doctors into the office and they would use the chatbots go through examples and generate this report of how clinically safe it is and we would then
29:22
do post-market surveillance so when that when the product is out there we have some doctors who their main job is take a sample each day and run through that example if they were handing that query and then they start at the end whether or not this was safe or not and that's where we get that 99.8% safety from the second lesson we learned is that the chatbot is only as
29:47
good as your content so we initially thought natural language is going to be really difficult and it is and we spent huge effort into kind of solving that problem but then when our first kind of you know prototypes started coming out we realized that this doesn't feel good even though our
30:02
natural language understanding is really good it didn't feel good because our content wasn't like optimized we didn't spend that much effort in our content and we've learned since that you need to kind of improve both at the same time because good content without good understanding is really bad but a good understanding without good content is really bad as well so we now kind of as we're improving the the ML stuff
30:24
we're also improving our content so that the whole chatbot becomes better all together and then I asked machine learning guys like you know I'm doing this talk what what what can I and tell these guys that you guys have learnt that doesn't give away the current cool things you're working on
30:42
that is still kind of under IP controller and they said to me that basically standard classification doesn't scale in the way that we've done it here and the sentence they told me to use is that information retrieval is better than classification and the reason classification doesn't scale is
31:03
because each time we want to make a small change we have to retrain all our models and sometimes the the new feature we want to add doesn't quite fit with that what with that the way that we constructed those models and it's just a very kind of slow process we want to be able to iterate our chatbot really fast so we have a natural language and stuff and we have
31:26
the medical triage we're doing really good but we had no way yet to kind of communicate to mobile clients so we had this really complex stuff happening in the back end but we didn't have like this uniform way to communicate to our mobile clients so we had the dispatcher is where this is like a
31:45
simplified there's more services around this but the dispatcher is where the the understanding happens we have leaflets and medical entity extraction services that it talks to and we have the JavaScript DSL service but we
32:03
needed to build an API that kind of simplified all this complexity happening in the back end to a way that was consistent for our clients mobile clients to understand and again this is just a massively simplified we have like 35 services going in production but it's this kind of piece that was
32:24
particularly difficult to kind of take all this complexity and map it into a simple way for our clients and one of the lessons we learned is that just in this application was just an evil word because we started off like oh we're
32:41
gonna pull this chatbot it's just gonna be easy where it's just you know request response we can just use HTTP it's perfect but it wasn't quite like that because we had to use WebSockets and it wasn't just WebSockets we had to also kind of bring APNs and GCM in there and initially we were like oh it's just you know one person talking another person that isn't just that because there's other people and there's doctors and there's authorization
33:02
of elements and you know our heads just exploded with how much more complex this was than we thought it was going to be a really useful tool we use that you know help us you know build and the clients pull at the same time was connection which basically is this wrapper around floss that helps you generate swagger documentation that your code is dependent on which
33:25
means that your swagger documentation is always up-to-date so how it works is you basically build swagger yaml file if you don't know what swagger is swagger is like a standard for how API document or API design so a standard
33:45
way of representing what your API does and you can do it in JSON or yaml so you build your swagger file connection then will map and the endpoints on that swagger file to the entry point in your code and then it'll do lots of cool
34:01
stuff for you so this is an example of our conversation model so you can see there there's some strings and bulls and arrays referencing to other objects and then connection will do that type validation for you that you don't have to do in your code and it also means because connection is doing that you're
34:21
really tied to making sure that this documentation is correct because it's gonna impact the way that your code runs which meant that our documentation was always very very good which is very useful to our mobile clients and what you get then is documentation and not just documentation you get an
34:41
API client that your your mobile clients can use to test out your API so cool so the last thing I want to talk about is just the role that Python played kind of in all of this and I can really confidently say that we wouldn't
35:04
have done near we wouldn't wouldn't have done as much as we did in the time we did it without something like Python and some of the lessons we learned were like not surprising but we're like you know really welcomed and resoundingly true and some of the lessons were quite difficult and
35:25
not ideal so one of the lessons we learned is that maths and Python are like a match made in heaven there are just so many amazing amazing tools that take in your really complex things and make them you know really easy not easy
35:40
but you know make them really accessible and then there's a huge community around this as well and there's loads of tools and stuff on the internet and tutorials and stuff to kind of get you on board so if you have like maths heavy workloads Python is like an excellent excellent choice for you and not only that Python and the web is also this this really good
36:04
combination because again you have all these amazing tools like you know everything from you know everything out the box frameworks like Django to more simple micro frameworks like flask and kind of everything in between you have like native support to all modern web technologies like Apache spark and
36:24
all the stuff like that you know database drivers and all that support is just really really solid which makes building the stuff you know really confident about building some of these complex systems some of the bad things we learned though is that if you have async heavy workloads in
36:44
Python you're probably gonna you know get caught out at some stage even if you know what you're doing there's probably gonna be something that's gonna catch you out because of all the kind of specific knowledge you need to know about how async works in Python so this is a screenshot from wire shark and why
37:09
were we doing packet sniffing you might ask no one wants to do this but we had this issue where like one in a thousand we were getting this SSL error that our load balancer was just throwing an error and the SSL errors were
37:25
different each time and for the life of us we couldn't figure out what was causing this problem so we then went into the packets to figure out what was happening and basically what's happening here is the server sends like a hello message to the other server the this kind of certificate key
37:45
exchange and cipher and spec and handshakes and stuff and then right in the middle there the server just says hello again and the receiving the service like whoa I wasn't expecting that like a fail and we couldn't
38:00
figure out why this like random kind of initial hello and the SL handshake was just kind of getting in there and we still don't really know exactly why but we did find out that there's the compatibility issue with multi-processing and eventlet and eventlet was the underlying async framework that our web framework was using and it was just causing some
38:24
craziness another problem we ran into was when using AIO HTTP was that you know once a week this kind of internal server we had running it just kind of ran out of memory and crashed and there were just some specific kind
38:44
of implementation details that we didn't kind of you know completely read through our understand where you had to kind of release the object the response object if there was an error otherwise the memory just kind of hangs around and and it's just this stuff that you really don't want to be dealing with
39:01
but when you're doing lots of async stuff in Python you're probably gonna find yourself dealing with this kind of like weird things every now and again so the last thing I want to say and I don't mean to be controversial at like a EuroPython thing is don't look to Python for your like main sense of engineering achievement so my relationship with Python is very much
39:24
like unstable romantic relationship where I move from being infatuated to being totally in love on like a daily basis because I'll be using really cool stuff like generators and decorators and context managers and coroutines and all this stuff but then I'll get something like this so this is just a
39:48
a function that has a return and a yield statement in it and if I was reading this for the first time what I would imagine is that the yield statement will just get ignored and it'll just return 3 and everyone's happy
40:02
and it all makes sense but it doesn't do that it becomes generator function and returns generator so okay whatever I can kind of live with that but if you call next on this generator I guess if I read this for the first time I would imagine that maybe the return is skipped and it yields 2 but no in fact
40:23
it raises stop iteration with the value of the return and if you kind of go into there's a PEP for this and it's got to do with coroutines and sub generators and stuff like that but to a new user it just this just says return doesn't mean one thing in Python and this is kind of the it's stuff like
40:48
this that I have to keep reminding myself that gives Python some of its characteristics some of its personality this is a quote from the
41:02
and this is really important to us because we are way way more concerned about making health care affordable and accessible than we are about doing it in the most pure way and Python has a very same approach where it's way more interested in solving a problem and getting things done than it is about
41:22
doing it in the most pure special way which is very different to when you go to Scala guys or elixir guys and stuff like that and with that approach we've been able to build something that's really really amazing and is affecting people's lives on a daily basis and we'll make way more interested in that
41:41
than we are about doing it in the most pure way and Python allows us to kind of do that which is something that is really amazing to us so that's it from me what's next for us we're moving into diagnostic space we're moving into
42:00
a way more comprehensive health prediction and monitoring space we're making our chatbot international which I can tell you is not trivial so like natural language for English is hard natural language for Arabic is like and Chinese like so that's me are there any questions really interesting talk did
42:40
you for your chatbot take into account social situations or medical history of the patients asking a certain question because it can have a lot of ruins on the questions the doctor will ask after he knows about the history of the patient or about the social context sorry I didn't quite
43:00
catch the question question for example when a patient asks I have headache why should what should I do it can really matter if the patient is a young boy who got all right so at so at the moment let me think so we
43:23
sat like maybe two months ago no but recently yes so recently we take demographics and age and gender all into account into the kind of you know outcome and path that you're going to take that you're going to go through
43:42
and what we're working on now is that not just to take that but also taking everything we know about you you know how many steps you're taking a the medication you take all into the kind of advice that we're giving to you and we have now a pattern that we can start adding these more kind of complex reasoning and deductions too so we're starting kind of down that path
44:07
thank you very much William give a hand