Virtual HLF 2020 – Hot Topic: Health, technology and data: Which is the best way to go?
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 19 | |
| Author | ||
| Contributors | ||
| License | No Open Access License: German copyright law applies. This film may be used for your own use but it may not be distributed via the internet or passed on to external parties. | |
| Identifiers | 10.5446/49578 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
Streaming mediaXML
02:40
Duality (mathematics)Perturbation theoryRow (database)AdditionBit rateOffice suiteMobile appEmailControl flowVirtual realityGroup actionPersonal digital assistantComputer programmingOpticsMereologyVirtual machineMixture modelMachine visionMeasurementIndependence (probability theory)Exception handlingHypermediaVector potentialOcean currentCondition numberDifferent (Kate Ryan album)Machine learningMobile WebMedical imagingElectric generatorLogic gateSampling (statistics)System callStrategy gameOnline helpLevel of measurementCoefficient of determinationArmMeeting/InterviewComputer animation
06:48
Duality (mathematics)Perspective (visual)Moment (mathematics)BitData conversionTouchscreenHasse diagramMultiplication signAlgorithmMedical imagingOrder (biology)Type theoryDoubling the cubeQuicksortAreaArtificial neural networkPoint (geometry)Set (mathematics)Reading (process)Order of magnitudeObservational studyFrame problemContext awarenessGoodness of fitBridging (networking)Meeting/Interview
09:41
CodeLetterpress printingGoodness of fitSparse matrixLevel (video gaming)State of matterAnalytic continuationBitLimit (category theory)MereologyPhysical systemTerm (mathematics)QuicksortMeasurementDependent and independent variablesRange (statistics)PressureSet (mathematics)Interactive televisionMultiplication signRational numberConnected spaceGroup actionLocal ringMachine visionMetropolitan area networkEndliche ModelltheorieWechselseitige InformationProgram slicingMeeting/Interview
12:36
AlgorithmComputer scienceMathematicsNetwork topologyUltrasoundComputer programmingBuildingTask (computing)Prisoner's dilemmaFundamental theorem of algebraProjective planeQuantum stateMachine visionMeasurementRoboticsInstance (computer science)Vector potentialProper mapReading (process)Observational studyFrame problemSign (mathematics)Machine learningMultiplication signDemoscene1 (number)Medical imagingVirtual machineArtificial neural networkSoftware bugNeuroinformatikMeeting/Interview
16:09
Observational studyDisk read-and-write headMultiplication signRight angleAlgorithmSelf-organizationType theoryBuildingWave packetElectric generatorAxiom of choiceDivision (mathematics)Line (geometry)Group actionPhysical systemNumberMeasurementBasis <Mathematik>Process (computing)Inheritance (object-oriented programming)View (database)WebsiteDifferent (Kate Ryan album)Proxy serverSoftware developerSet (mathematics)ThumbnailMeeting/Interview
20:14
Hydraulic jumpAlgorithmMedical imagingComputer sciencePerspective (visual)SoftwareType theoryWave packetLevel (video gaming)Electric generatorVideoconferencingTask (computing)BitSheaf (mathematics)Metric systemPhysical systemResultantLocal ringTerm (mathematics)Revision controlInternetworkingHypermediaCASE <Informatik>BilderkennungPressureFood energyPulse (signal processing)Ocean currentInteractive televisionObservational studyGraph coloringSign (mathematics)Information privacyBit rateIdentical particlesDifferent (Kate Ryan album)Representation (politics)Multiplication signBuildingHeat transferField (computer science)Set (mathematics)Embedded systemNeuroinformatikSpacetimeMobile WebPort scannerMeeting/Interview
23:32
BitPhysical systemProjective planeField (computer science)Context awarenessMobile appQuicksortWordMeeting/InterviewComputer animation
25:11
Perspective (visual)Matrix (mathematics)Projective planeNeuroinformatikSoftware developerMathematicsMetric systemPoint (geometry)AlgorithmWave packetLevel (video gaming)Axiom of choiceReal numberMeasurementSocial classCuboidSet (mathematics)Different (Kate Ryan album)Object (grammar)Mechanism design1 (number)AutomationFrame problemMeeting/Interview
27:42
SkewnessAlgorithmSoftware testingPropositional formulaPhysical systemMedical imagingType theoryGoodness of fitProcess (computing)Point (geometry)Set (mathematics)Information privacyQuantum stateTerm (mathematics)Office suiteMeeting/Interview
30:03
InformationBitContent (media)QuicksortComputer virusShared memoryCuboidRAIDGradientElectronic mailing listAddress spaceBit rateIdentical particlesAuthorizationComputing platformContext awarenessTracing (software)SpacetimeService (economics)Local ringMobile appMeeting/Interview
32:57
Projective planeCuboidBitGroup actionHypermediaPerspective (visual)Error messageSelf-organizationBuildingLevel (video gaming)Decision theoryState of matterTerm (mathematics)AngleScaling (geometry)Water vaporPoint (geometry)Set (mathematics)Interactive televisionChannel capacity40 (number)Context awarenessMultiplication signStandard deviationMechanism designSubsetQuicksortScalabilityWhiteboardMeeting/Interview
36:24
Type theoryZoom lensWeightBitInformation privacyArmComputer programmingTime zoneSinc functionMultiplication signPoint (geometry)Meeting/Interview
38:35
AlgorithmNoise (electronics)Line (geometry)Virtual machineCentralizer and normalizerMeasurementServer (computing)Artificial neural networkInteractive televisionBit rateMultiplication signUniform resource locatorVideo gameNumberEndliche ModelltheorieRow (database)Information privacyPattern languageMetric systemTraffic reportingSet (mathematics)Meeting/Interview
40:32
Information privacyMultiplication sign2 (number)MappingDeterminantLine (geometry)QuicksortPoint (geometry)Different (Kate Ryan album)Computer configurationComa BerenicesElectronic mailing listInternet forumDisk read-and-write headContext awarenessRight angleMeeting/Interview
43:13
Point (geometry)Electronic mailing listLie groupCausalityBitPhysical systemQuantum stateCASE <Informatik>19 (number)Object (grammar)Multiplication signPort scannerX-ray computed tomographyVector potentialAlgorithmSensitivity analysisSoftware testingTask (computing)Decision theoryResultantLatent heatInternet service providerDifferent (Kate Ryan album)Context awarenessLattice (group)Meeting/InterviewJSONComputer animation
45:44
Duality (mathematics)Wave packetUniverse (mathematics)Machine learningPhysical systemBitOnline helpChemical equationArithmetic meanTerm (mathematics)Virtual machineMeeting/Interview
47:14
AlgorithmDimensional analysisTerm (mathematics)QuicksortAdditionSet (mathematics)Cellular automatonBit rateLink (knot theory)Mobile appPhysical law19 (number)MedianContext awarenessMeeting/Interview
48:57
Duality (mathematics)InformationMultiplication signVirtual machinePoint (geometry)Right angleAlgorithmMedical imagingFormal languageTransportation theory (mathematics)Wave packetSubject indexingFunctional (mathematics)Universe (mathematics)Term (mathematics)NumberLink (knot theory)MeasurementSet (mathematics)Observational studySigned measureDomain nameFood energyVapor barrierSinc functionMeeting/Interview
52:17
MathematicsIntegrated development environmentQuicksortObservational studyInclusion mapGoogolMultiplication signRight angleFacebookMeeting/Interview
54:40
Staff (military)FrequencyEntire functionProcess (computing)Field (computer science)Interactive televisionFocus (optics)Domain nameArchaeological field surveyElement (mathematics)Multiplication signFacebookSoftware developerMobile appCoroutineComputer programmingPhysical lawMathematical optimizationArithmetic meanTerm (mathematics)Sign (mathematics)Rule of inferenceGroup actionGoodness of fitLattice (order)Meeting/Interview
56:58
Group action40 (number)Computer virusPoint (geometry)Electronic mailing listMetropolitan area networkTouchscreenMobile appPerturbation theoryReal numberCovering spaceMultiplication signMeeting/Interview
59:50
SkewnessDuality (mathematics)Logic gateOpen setVideo gameComputer programmingProduct (business)Goodness of fitWhiteboardLogicConservation lawAngleMusical ensembleGoogolMixed realityCollaborationismArithmetic meanChainData managementProjective planeComputer engineeringFamilyWater vaporSet (mathematics)Computer scienceSelf-organizationAreaPlanning19 (number)Interactive televisionNeuroinformatikGame controllerComputer animationMeeting/Interview
01:01:58
Self-organizationPhysical systemAreaCartesian coordinate systemInteractive televisionNeuroinformatikUbiquitous computingOpen setOnline helpComputer scienceProcedural programmingRight angleLevel (video gaming)Multiplication signMobile appVideo gameBitProjective planeFamilyPlanningThread (computing)19 (number)Meeting/InterviewComputer animation
01:03:59
SpacetimeVideo gameWave packetLevel (video gaming)Integrated development environmentReduction of orderAreaError messageAdditionPoint (geometry)Peer-to-peerContext awarenessMechanism designMedical imagingLie groupDecision theoryLine (geometry)Process (computing)Exterior algebraOnline helpEvent horizonNP-hardMessage passingMeeting/Interview
01:06:38
Dot productMachine visionSet (mathematics)Software developerBitAreaGradientStudent's t-testMotion captureUbiquitous computingComputing platformBoundary value problemRight angleMobile Web1 (number)Dressing (medical)Field (computer science)Point (geometry)Regulator geneDifferent (Kate Ryan album)CollaborationismService (economics)Computer animationMeeting/Interview
01:08:42
Regulator geneAdditionDifferent (Kate Ryan album)Boundary value problemCollaborationismWordMultiplication signSpacetimePerspective (visual)Self-organizationGoodness of fitInformation privacyUsabilityInformationComputer networkSoftware testingIntegrated development environmentAnalytic setHypothesisNumberQuicksortShared memoryObservational studyInternet service providerWhiteboardChainCASE <Informatik>Block (periodic table)Meeting/Interview
01:12:00
Perspective (visual)ResultantInstance (computer science)CuboidTraffic reportingInformation privacyMobile WebType theoryFocus (optics)Row (database)Group actionQuantum stateSystem callShared memoryData storage deviceGame controllerInformationLevel (video gaming)Computer configurationAxiom of choiceMereologyPhysical systemException handlingPoint (geometry)Variety (linguistics)Different (Kate Ryan album)Multitier architectureMultiplication signUniform resource locatorSpacetimeService (economics)Medical imagingData structureTerm (mathematics)Revision controlProcess (computing)Source codeACIDRight angleTablet computerSoftware developer1 (number)Meeting/Interview
01:16:04
Duality (mathematics)Uniform resource locatorInformation privacyMobile WebTwitterInformationEvent horizonNumberReverse engineeringDirection (geometry)Tablet computerType theoryMoment (mathematics)QuicksortIdentifiabilityFigurate numberPattern language1 (number)Order (biology)VideoconferencingGroup actionProjective planeTerm (mathematics)CASE <Informatik>Right angleMeeting/InterviewJSON
01:17:41
Duality (mathematics)Projective planeGroup actionBitType theoryTerm (mathematics)NumberComputer virusPoint (geometry)Web pageDifferent (Kate Ryan album)IntegerProcess (computing)Formal languageMeeting/InterviewLecture/ConferenceComputer animation
01:19:18
Natural languageValidity (statistics)Loop (music)Process (computing)Point (geometry)Reading (process)Web pageEvent horizonImage resolutionMultiplication signMathematicsSystem callWebsiteArmOrder (biology)AdditionSource codeDifferent (Kate Ryan album)Goodness of fitMonster groupSign (mathematics)Power (physics)AngleDecision support systemPredictabilitySet (mathematics)Endliche ModelltheorieSpacetimeDimensional analysisMass19 (number)Meeting/Interview
01:21:29
MeasurementCross-correlationCodeNumberArmStatisticsBitFeedbackSelf-organizationStatistical hypothesis testingBuildingElectric generatorVariable (mathematics)Computer simulationData flow diagramComplex (psychology)Performance appraisalHypothesisTerm (mathematics)TunisDistanceProcess (computing)Software repositorySet (mathematics)Proper mapTraffic reportingInformation privacyOpen sourceEndliche ModelltheorieDifferent (Kate Ryan album)Multiplication signMobile WebTwitter1 (number)Product (business)Level (video gaming)State of matterArithmetic meanGroup actionPrice indexMereologyVirtual machineSystem callReal numberOnline helpInteractive televisionOffice suiteDemosceneMeeting/Interview
01:26:28
Duality (mathematics)Dimensional analysisVisualization (computer graphics)PredictabilityInformationLevel (video gaming)Decision theoryState of matterBasis <Mathematik>Set (mathematics)Endliche ModelltheorieArmTwitterBitChromosomal crossoverMereologyTerm (mathematics)NumberAreaInheritance (object-oriented programming)SurfaceParticle systemProxy serverCollaborationismComputer-assisted translationIntegrated development environmentLogic gatePrice indexCore dumpLibrary catalogSelectivity (electronic)FreewareSpacetimeMeeting/InterviewComputer animation
01:30:32
Duality (mathematics)Cross-correlationSelectivity (electronic)SpacetimeType theoryAreaLatent heatBitSpring (hydrology)TouchscreenMultiplication signMessage passingFigurate numberEndliche ModelltheorieWave packetResultantFilm editingMeeting/Interview
01:32:05
Endliche ModelltheorieWave packetFunction (mathematics)Right angleBuildingLevel (video gaming)DialectDecision theoryEquivalence relationPhysical systemResultantVirtual machineQuicksortSoftware bugError messagePoint (geometry)Set (mathematics)Discrepancy theoryMultiplication sign2 (number)Group actionSystem callWeightSource codeTouchscreenArmMeeting/InterviewJSON
01:36:25
Row (database)Self-organizationMereologyPoint (geometry)SpacetimeValidity (statistics)Endliche ModelltheorieTouchscreenSoftware testingResultantSet (mathematics)Regulator geneInformationAnalytic continuationLoop (music)Water vaporTimestampMeeting/Interview
01:38:01
InformationLoop (music)Game controllerSequenceOrder (biology)Combinational logicValidity (statistics)State of matterDoubling the cubeTerm (mathematics)AngleQuicksortPoint (geometry)FingerprintEndliche ModelltheorieContext awarenessRight angleAlgorithmMathematicsPerspective (visual)FrequencyLogic gateDecision theoryLetterpress printingSampling (statistics)Water vaporCASE <Informatik>19 (number)Source codeArmReinforcement learningMeeting/Interview
01:40:47
Combinational logicValidity (statistics)FrequencyBuildingState of matterLoop (music)MereologyMultiplicationCASE <Informatik>Latent heatEndliche ModelltheoriePosition operatorAxiom of choiceProjective planeWordObservational studyMultiplication signMeeting/Interview
01:42:16
Duality (mathematics)Theory of relativityProjective planeDataflowWordObservational studyDirection (geometry)Zoom lensMultiplication signTwitterProduct (business)Instance (computer science)GoogolPredictabilityShift operatorDigitizingDialectMereologyResultantData modelQuicksortProcess (computing)Distribution (mathematics)Point (geometry)Set (mathematics)ChecklistReflection (mathematics)Software frameworkDiscrepancy theoryEndliche ModelltheorieProxy serverMessage passingMappingSpacetimeDemosceneUsabilityInformationBuildingLevel (video gaming)Subject indexingExpected valueAdditionFreewareComputer-assisted translation1 (number)Meeting/Interview
01:46:21
Level (video gaming)Subject indexingExpected valueQuery languageTwitterInformationCombinational logicSource codeType theoryBitPredictabilityPoint (geometry)Data streamGauge theoryMultiplicationTerm (mathematics)Inclined planeStreaming mediaWordTouchscreenMeeting/Interview
01:48:03
Duality (mathematics)Large eddy simulationInformationMereologyRevision controlChecklistMultiplication signMobile appMathematicsState of matterSoftware bugPoint (geometry)CircleInteractive televisionView (database)Endliche ModelltheorieContext awarenessRight angleMeeting/Interview
01:49:33
State of matterInformation privacyEndliche ModelltheorieDialectReading (process)Data managementInformationSensitivity analysisMereologyPhysical systemCASE <Informatik>Bit rateWireless LANPosition operatorMultiplication signDesign by contractMeeting/Interview
01:51:32
Duality (mathematics)Level (video gaming)Sensitivity analysisState of matterDependent and independent variablesDifferent (Kate Ryan album)Position operatorSoftware testingNumberCASE <Informatik>Information privacyChannel capacityMobile appMathematicsLocal ringSystem callSimilarity (geometry)19 (number)Multiplication signMeeting/Interview
01:53:48
CASE <Informatik>Mixed realityMultiplication signNeuroinformatikSequenceSoftware testingBit rateType theoryDifferential (mechanical device)Exterior algebraCondition numberDifferent (Kate Ryan album)Right angleAssociative propertyVideo gameRAIDEndliche ModelltheorieComputer animationMeeting/Interview
01:55:34
Duality (mathematics)InformationType theoryLevel (video gaming)Software testingCASE <Informatik>Sign (mathematics)QuicksortOnline helpWhiteboardMultiplication signProjective planeZoom lensMeeting/Interview
01:57:16
Streaming mediaComputer iconMultiplication signUniform resource locator2 (number)Social classTouchscreenExtension (kinesiology)Computer animationMeeting/InterviewXML
Transcript: English(auto-generated)
02:42
everybody to this year's hot topic at HLF. I'm thrilled that we got so many exciting people here on the panel as well as in the audience. I see there are more than 100, we have 111 participants, which is great. Wonderful. Welcome. I think you joined from everywhere
03:01
in the world. And this already feels like we are coming together, although everybody is his or her home or office. A warm welcome. We have an exciting topic. When we started to think about what could be the topic for this year's hot topic, we didn't know that this pandemic is going to arrive. But public health, global health and tech is a big topic anyway. And now,
03:26
in addition, we have the pandemic. And I think we see many aspects why we should think about how to use technology to help us with all the challenges. And of course, there are some examples of things that didn't work out well. I just see that someone wrote me
03:47
that there are more participants in the VR app. Hello, people in VR. That's wonderful. That's another housekeeping thing. Maybe I go on with that. After the two panels, we will have two panels in a row now, each one hour without break. So we'll discuss for two hours. And then we'll
04:01
all meet in virtual reality for the participants of HLF. You all received an email with instructions how to go there. So we'll meet there again with the speakers and all of you for discussions, one-on-one or smaller group discussions. All the questions you are not able to ask now, you will be able to ask later on in the VR solution starting in two hours from now on.
04:23
We have for the hot topic two panels. The first panel is on global health and tech. So we will discuss the conditions in different parts of the world. We will discuss the biggest challenges and the biggest hopes and things we will have in future.
04:41
And the second panel will be more concrete on innovations, apps, and of course, as well, challenges, current solutions, future developments, and with more concrete, what apps are we going to use in the future and will they change public health. So for the first panel, a very warm
05:02
welcome to our wonderful speakers, Stephanie Friedhoff, Arunan Skaneracha, and Ziad Rubemeyer. Stephanie is not only a German-American journalist with 25-plus years of experience in international media and higher education, but is well part of the leadership team at the Harvard Global Health Institute. She executes the Institute's vision through the delivery of
05:24
new programs, initiatives, strategies, and symposia, including a growing portfolio of work on how artificial intelligence is transforming global health. And I think she's a great, great speaker here for our panel to help us to get the global overview of global health.
05:41
Arunan, a very warm welcome, is a program officer at the Bill and Melinda Gates Foundation on innovative technology solutions team. His portfolio spans sensing, imaging, and machine learning for global health. He makes grants to innovators that utilize these technologies to potentially disrupt how community health workers do their work, for example.
06:02
Mobile phones are a big topic here, and we will hear more from him about the potential of these sensors and apps. Arunan completed a PhD in bioengineering at UC Berkeley, where he developed mobile phone-enabled optical measurements for global health. Welcome, Arunan. And then we have Ziad. Ziad Rubemeyer is an assistant professor at UC Berkeley,
06:22
where he does research at the intersection of machine learning, medicine, and health policy. He was named an emerging leader by the National Academy of Medicine, and he has received numerous awards, including the Early Independence Award, the National Institutes of Health, most prestigious award for exceptional junior scientists. Previously, he was an assistant
06:41
professor at Harvard Medical School. He continues to practice emergency medicine in underserved communities, which is, again, a great mixture for our panel today. So a warm welcome to all the speakers and the audience. And I would like to start with a quite open question to all three of the speakers, and I would like to ask Stefanie to answer first.
07:02
So what are, from your perspective, the biggest global challenges in global health at the moment where we could use technology to help to solve them? I'm happy if you want to speak about examples as well, things that worked well, things that we have to work on. Thank you so much, Eva, for having me. Thanks for the great conversations. Fantastic to see you
07:24
all on the screen from all over the world. That's, of course, a good and tough question to start with. So let's do a little bit of framing. Global health, of course, has problems that the world has struggled with, you know, for a really long time. Access to clean water and nutrition are and remain big issues in global health. But then there's also things
07:45
such as what we call the double burden of disease, where we know that chronic diseases will account for about 75% of all deaths around the world in the next 15 years. At the same time, we'll have a deficit of around 30 million skilled healthcare workers. Efforts, of course,
08:04
have been underway to train people, especially in resource-poor settings. But this is one of the areas where we're very hopeful that technology can play a role. Overall, just for sort of layers of magnitude investment in this type of work, especially if we just take artificial intelligence
08:21
in health, has tripled in the last three years alone. It's about a $7 billion market at this point. So there's a lot of interest. And what all of us, I think, who work in global health are a little bit worried about is that it's sort of fairly easy to create an algorithm and really hard to create the data infrastructure and the cultural and community awareness that we
08:45
need in order to make these technologies work. So where are areas where we think technology is moving us forward? Certainly, for example, in the reading of images in radiology.
09:00
It turns out that doctors aren't actually as good at diagnosing diseases or some diseases from images as we hope or think. Some studies show that they're accurate about 50 to 70 percent of the time. And we now have algorithms that can sort of diagnose certain types of lung cancer with an accuracy of 87 percent. So that's where some of the hope is for these technologies.
09:26
And then, of course, there's a lot of concerns that I think our discussion will go into sort of in the details. So I think I should hand it over to my fellow panelists to get us started on those questions. Thank you so much. That was great. I think that was a great overview
09:44
of all the topics we have to discuss today or want to discuss today. Arun, I guess you wanted to say something? Sure, just I think that's an incredibly good framing, Stephanie, and it's a good way to think about both the way that these systems and kind of the very narrow specific
10:03
diagnostic sense have shown incredible performance, but also the fact that prior to COVID and as COVID continues, these larger challenges of what diseases affect whom, or diseases that have been historically in richer parts of the world starting to permeate the rest of the global community. The only kind of nuances I wanted to add are a little bit about
10:28
extending on some of the themes that Shwetak talked about during his talk, this idea of the community healthcare worker as one of the underpinnings of this sort of system. And I think we're seeing both the advantages and the challenges of that model. So as many of you know,
10:46
these community healthcare workers have responsibility for their local geography, could be their village, they could walk a couple miles a day. And because of that intimate connection, they're able to spend time with people, they're able to be called kind of on
11:01
demand, especially when people don't have access to facilities. That I think is an incredible way to extend access. And we're starting to see that in the US and in other parts of the world, again, borrowing from global health. But we also see the limitations. COVID is obviously spread on it by individual interactions. So what does it mean to have that level of personalized care
11:23
when personalized care can be a risk factor? And how do you equip these individuals with new tools? So that maybe the last thing I'll say there is that given that interactions may be challenging, both in the global setting and in the setting of a pandemic, we should really be thinking about mobile devices as a way to make those interactions richer. So when we talk about
11:45
a one-to-one interaction between a worker and an individual, how do we take things that are traditionally subjective like pallor and overall state of health and turn that into a range of quantitative measures? If you can't get a person in front of the healthcare worker
12:02
at all, how do we make that interaction with the mobile phone richer? Could it be continuous measurement of growth of a child so you can understand how malnutrition is affecting that individual? Can you see how things resolve and how diseases are followed up on? So I think while the pressure of both the global health spread access and of the COVID kind of
12:24
interfering with traditional interactions is very real, I think it does push the community forward in terms of what can be done with limited sensors and either sparse or continuous sensing. Great, thank you so much Arunan. I think Stephanie and Arunan raised so many great issues. I'll just pick up on a couple of
12:45
themes that they raised. So I think that the very tempting thing when you look at the big picture of global health and look at what artificial intelligence can do is to view it through a fairly traditional lens of like automation. So, you know, we don't have doctors in all of
13:02
places where we need doctors and artificial intelligence can actually automate very high quality doctors and bring them where there are no doctors. So it really seems like a no-brainer and I think in a lot of ways that vision is fundamentally a good one. But I think that automation and the way we traditionally think about it in robotics and machine learning,
13:25
I think is the wrong frame, both because it is way too ambitious on the one hand, but also because it represents a failure of imagination on the other hand. So why is it way too ambitious? Well, I think when you start disaggregating the big picture into individual
13:43
tasks, you find a consistent challenge that we don't really see in many other places in computer science, which is that we don't have the ground truth labels in medicine. So, you know, Stephanie, you mentioned these great studies where we see that on some measures computers are
14:00
better than doctors at diagnosing lung cancer, but we actually don't have the ground truth for lung cancer. And so, for example, we know that because of incentives, radiologists and other doctors over-diagnose some kinds of lung cancer. We also know that people who don't have access to healthcare are never diagnosed, so they never show up in our data. So when we say that algorithm is better or worse than the doctor, we're building in all of those biases and
14:25
incentives and problems in a way that we don't face when we're designing algorithms for a self-driving car. We know what a pedestrian looks like. We know what a stop sign is. There's no incentives or, you know, under diagnosis of stop signs there in the way that there really is in medicine. And so, you know, that I think brings up the fundamental issue in global health,
14:46
which is, like, what are the data we're using? And I think, Arunan, you pointed out, I think, this huge potential where, again, I think automation is the wrong frame, but for a different reason, which is that currently doctors are not using mobile phone-derived data to make diagnoses
15:03
because there's just too much of it. They're not making full use of ultrasound data to do triage for which women need to give birth in hospital, which ones can can deliver safely at home. We're not getting all we can out of the x-rays that all national tuberculosis programs do
15:20
for every patient that's evaluated for TB. There's just too much data. And at the same time, the data are completely inaccessible. And so a lot of my work is actually getting data and getting those interesting data that we think are underutilizing, that doctors are currently underutilizing, but then formulating those problems in ways that deal with this
15:42
ground truth issue by doing very, very careful work to label the images properly, not just with what the doctor says, because that's just the doctor's opinion, but with actual health outcomes. And so one of the projects that I'm involved most in these days is it's a nonprofit where
16:00
we're trying to actually aggregate data and make them available to researchers around the world. So I heard two really interesting things. And of course, this is always the critical science journalist in me who wants to discuss these two things that are potential challenges is the bias in the data is what you talked about. And the other thing is privacy, of course,
16:26
because you have health data are really sensible data. And especially what you said is that it's a patient doing now and how is he or she doing in 10 years and what was 10 years ago.
16:42
So I would like to elaborate on those two topics with you. I know, Zia, that you had did some really interesting studies around bias in data. Aruna, you know a lot about this topic as well. I would love to hear some examples, because I think this is what people helps to make it better in the future.
17:05
Yeah, so I'll lead off, but Aruna, I'm curious to hear what you have to say. So I'll give you maybe just a thumbnail sketch of two studies that I've done that show, I think, both the perils and the promise of algorithms. So one algorithm that we studied and published on last year was
17:24
an algorithm that's being used in almost every type of algorithm that's being used in almost every health system in the US and actually many in Europe as well. And it's an algorithm that does something that is very, very important for health systems to do. It looks at all of the patients that a health system is taking care of, and then it tries to look into the future and
17:43
say which patients are going to get sick in the future that I can intervene on now. And once I've found those people, I'm going to deliver extra resources to help them. So this can be home visits, extra attention from trained nurses. Now, the problem with that algorithm was that that all sounds very sensible. There was a choice that was made when building that algorithm
18:05
that is an inevitable choice that you have to make, which is how do I measure who gets sick? And so the choice that the algorithm developers made at this company, but at many other companies in academic groups and nonprofits, it's not about an evil company, was to look at
18:20
the amount of healthcare dollars, expenditures that that patient generated over the next year as a proxy measure for their health. Now, again, that's not unreasonable because when you get sick, you generate healthcare expenditures. The problem is that different people generate expenditures in different ways, and poorer people generate fewer expenditures because they have less
18:42
access to care. Non-white people in the US and I think elsewhere are also treated differently by the healthcare system. They're given less care even when they get access to it. And so two people who have the same needs will generate different costs, and that algorithm was building in those biases. And so it was systematically de-prioritizing black patients
19:04
and putting white patients further ahead of them in line to get access to extra care. Now, the nice thing about that is that once we realized that we could work with the manufacturer, and we're working with a number of other health systems on a pro bono basis, to get bias out of that algorithm by changing the label that that algorithm is predicting.
19:24
So instead of predicting measures of cost, we aggregate from the same data sets and the same patients measures of need. And that's harder, but it's fundamentally better because it de-biases this algorithm. So I think that that's a kind of optimistic view of bias because
19:40
it means that bias is not about a biased society producing biased data, and until we fix that biased society, we can't have algorithms. No, it means that by making the right technical choices when you're building your algorithm, you can make the difference between an algorithm that reinforces biases and an algorithm that undoes them. So I have a second one that maybe showcases
20:03
a different way to think about training the algorithm, but maybe I'll stop there and just we can come back to it later, Eva, if you think it's worth it. Yeah, Arun, it looks like you want to... Yeah, I'll jump in with another example. I think one that on the faces is an even more,
20:23
I think, obvious one than the one that Ziad described. But the technical solution is clever, which is something that the community might appreciate, at least we hope it will be. So one of the challenges that we've seen companies, in this case, a Canadian company called Winterlight tackle is dermatology. So looking at skin diseases, sometimes these can be markers
20:44
for things in the global health domain, like parasite infections, and certainly tied to cancer and other diseases. And whether you're looking at things like images, like companies like VisualDx, or if you're looking at voices like Winterlight, what you train on makes a huge
21:04
difference in terms of what you expect out of your algorithm. And while this seems obvious, there is a lot of risk and some very good work that's been done looking at the kind of the classic algorithms and the classic data sets. So whenever you kick off an image analysis study,
21:21
people tend to go to certain image data sets that have now been studied to show, contain predominantly people of lighter skin. So if you then try to supplement that or transfer some of that learning onto a cancer data set, then you already, you started your system with some level of bias. Similar to the kind of systems lens that Zied was taking, this results
21:45
from who initiated the community of computer scientists, whose data is available in public media because of common collection tools like the internet and Hollywood, and how does that feed into kind of the reference that people build from. But I think similarly, I don't think that
22:02
everything is lost here. From a global health perspective, that means that we need to, ahead of any work that a computer scientist does, kind of build towards those data sets, ensure that the time and energy is invested to create those data sets in local ways and in
22:20
more representative settings. One of the clever artificial intelligence approaches to this is around the generation of vital signs by one of our grantees, Conrad Tucker. So his goal is to take a mobile phone, scan someone during interaction, and understand their pulse, potentially other metrics like respiratory rate or blood pressure. But if all of your initial
22:45
data is on people with lighter skin, then these algorithms which happen to use color, the flashing of red in your skin could become occluded. So you have some risk that your algorithm developed in one population doesn't transfer to another. They're actually working on using generative adversarial networks. So a way to basically test, can you generate something
23:05
that's indistinguishable from the real thing to embed pulses in video or individual data. So there's been a lot of great work in creating photorealistic individuals. Can you use that to actually fill in gaps on these data sets? It's not a complete answer. You
23:23
always have to validate with the ground truth and with populations that are representative, but maybe a great way to get the field started in that space. I can just expand on that Eva. I know you asked about bias in data and privacy, and I think some of this is about privacy, but it's also I'd love to expand it a little
23:45
bit into understanding the community and the context and the culture that you work with. Global health, really the field of global health has a large graveyard of well-meaning projects, innovations and interventions that didn't work because people didn't understand their users,
24:01
or in other words, the community they were working with. Just recently in Rwanda, somebody tried to introduce a contact tracing app in this pandemic that asked for people's ethnicity, which the genocide is very well on everybody's mind, not a good question to include, was obviously a system that was transferred from somewhere else where that didn't matter.
24:24
I'd love for us to talk a little bit more about this. This is a big problem in global health in general. We're really advocating for anybody who works on a project or an innovation
24:42
in AI in health or in tech in health to make sure you have somebody on your team who's from the context or the culture that you work with, if at all possible, and if not, what are you doing to mitigate that? I think all three of them are so great and interesting examples where in hindsight, I think all of us think, okay,
25:05
how could that happen? How stupid are those people who had those ideas, who tried to measure, who should get healthcare in the future just by who got in the past? But at the same side, I can understand that if you start a project like this as a researcher or computer scientist,
25:22
developer or innovator, that this is not intuitive from our perspective. So I wonder, and I think this is really important to change these things in the future, is there a matrix or something you have to do or a starting point, how to start projects like this to make sure to avoid all those mistakes and problems? That's the question for all of you
25:43
and whoever wants to jump in first. Yeah, I think I might have one answer to this, which is that it's unlikely that any out-of-the-box solution is going to work. The mechanisms by which bias gets into algorithms and by which algorithms do things that we
26:02
fundamentally don't want them to do, these aren't things that we typically see when we just look at accuracy. When we look at accuracy of an algorithm, we're saying, how well does the algorithm accomplish the goal that I set it to do? Too often the problem is that we've set them to do the wrong goal, and so algorithms are very mechanical. They accomplish the goals
26:24
that we set out for them, and they do that very well, but they can't tell us if those goals are the right ones. And so in a lot of the settings where I think we're increasingly seeing bias, whether that's in hiring, in credit scoring, in criminal justice, the algorithms,
26:43
the problem isn't that accuracy is different or that sometimes it's predicting worse on the minority class. That's a fixable problem, but when we've asked the algorithm to answer the wrong question, that's a much harder thing to run an automated diagnostic against,
27:00
because that's a question of framing and formulating the problem correctly, not one where we can get a quantitative metric back. Now a lot of my work is trying to figure out, here's what we told the algorithm to do, here's the true objective, can we triangulate the difference, and then that's something we can write down. So this is what we've called label choice bias,
27:21
where we've picked the wrong label for the algorithm, and we can quantify the difference between the true label of interest and the one that we've told the algorithm to predict. But that's a semantic exercise that then becomes quantitative, it's not a kind of out-of-the-box diagnostic you can run, and it certainly doesn't show up in accuracy, and accuracy can be very
27:41
misleading. Because accuracy leads you to, in the end, to the wrong question, because then you go do the way backwards in the end. Yeah, and I think, you know, you see this in, for example, if we go back to Stephanie's lung cancer example, the algorithm can be predicting lung cancer extremely well, but what if poorer patients, non-white patients, are never getting
28:06
diagnosed with lung cancer? The algorithm is correct to predict zero for that patient, but it's not because they don't have lung cancer, it's because we never diagnosed it. And so we're never exposing those biases when we just look at accuracy.
28:21
But that means you don't have an unbiased testing data set, right? So it's hard to get rid of these biases just via a technical way. Yeah, so I think that's exactly right, but I don't think it's a hopeless proposition, and I think that gets back to Arunan's point about data. And so, you know, what we would need to do that is just to look at long-term
28:43
follow-up data of the kind that we don't usually have in the same place that we have the x-ray images. And so there's a really important effort around linkage of data to actually overcome a lot of these biases. And so I think, you know, to your earlier question, Eva, about privacy, I mean,
29:01
it's very clear that privacy has to be protected and that people are doing really creepy and not good things with data today. And I think we're very attuned to that set of risks, very reasonably so. I think we as a community tend to be less attuned to the other set of risks, which is not using the data. So a lot of these data are, you know, I think you should
29:26
think of them as like global public goods. There is an enormous amount of knowledge that has to be discovered from these data. And not using them is also risky, but those are risks that are harder to see because there are things that never happened rather than things that,
29:41
you know, might happen. We can all imagine what a privacy breach looks like. How can we imagine an algorithm that was never developed because nobody had access to those data that could have saved thousands or millions of lives? Those are risks too. And I don't think we're currently doing a very good job of trading off those two types of risks that are in tension.
30:03
Thank you so much. It's interesting. Stephanie, I saw you starting. Oh, I have so much. I'm not quite sure. I think I'll do a quick, like there's a couple questions. So we work with a lot of entrepreneurs, right, who are trying to get solutions off the ground. We try to do a lot of this work of connecting the problems, right,
30:26
with the solutions as there's, I think, a tendency to take something that works and try to find a problem as opposed to the other way around. So we have this quick little hit list of questions you can ask yourself if you try to innovate in this space, right?
30:44
The first one is really is the technical solution, the best solution for the problem you're trying to address, right? It's often that we try to throw a technical solution at a behavioral problem. There's an app that puts a sensor into a woman's underwear to alert her friends
31:01
when she is about to get raped. And, you know, aside from the question of how the sensor distinguishes between consensual and non-consensual sex, right, in a society where rape is still stigmatized, is that really the best way to go about the problem? Or, you know, should we just say this is maybe not a problem that gets solved by technology? So I think that's one of the
31:22
key questions to ask. Do you understand the context we're working in, right? That's something that we've already touched on. Then do you actually have the data that you need for your solution and do you understand it? That's what Ziad has been explaining, and both of you have been explaining so beautifully, right? And then sort of there's another aspect of this that's
31:44
about are you making sure that what you do and what you're finding gets shared with local authorities and others working in this space? So, and that was the example I was just thinking about, right? ComCare is a digital platform that was developed by Dimaji for Ebola contact tracing.
32:05
And this, like, in Sierra Leone, it was used heavily in Sierra Leone, and to this day there has been no digital contact tracing in Sierra Leone. It's all done on paper. And if you ask why, it's because after Ebola all the NGOs left and the apps left and nobody ever shared
32:24
this information with the Ministry of Health and Science. So, you know, just to open the box a little bit, right? These are all things we need to be thinking about, and it's quite striking to have these examples. And again, like, these are the typical pitfalls of global health work.
32:43
What if the Ministry of Health is like a partner from the beginning, right? Working with local government and improving the, again, the health, the infrastructure is a key, key issue. Yeah, discuss with them how can you make this sustainable, right? I think this is
33:00
things I often observe in academia, that there's a project and you have, whatever, a grant for three years and when this money is gone, all the work is just back in the box and then that's it and nobody's working with that. And I think this really has to change it. I'm curious, Aruna, maybe you can tell us a bit more about that. Is there anything you can do from your perspective
33:21
or foundation think about to change that to make innovation more sustainable and maybe more, yeah, make innovators to start with these questions Stephanie mentioned right now? No, I think there are things that can be done on the funding side, though a lot of it has to do with how we encourage innovators to design for sustainability. And then I think to add to
33:44
Stephanie's list, and I'm completely on board with the idea that this data, a lot of it should be thought through alongside public health officials so that a subset can be used for that sort of decision making, but it also ties to the kind of incentives question. So one point we haven't talked too much about is what is the,
34:06
what are the ethics of knowing something that's not actionable or knowing something to ZDOT's point around incentives where somebody has a particular outcome that is either financially remunerative or leads to action and then other outcome may not. So in the context of innovation
34:25
for global health, we have as an innovator, you should keep in mind, are you serving the individual clinical interaction and public health or just one or the other? And as soon as it becomes one or the other, no matter how much funding as a grant maker we give them,
34:45
there won't be a sustainability mechanism depending on who pays. So let's take something that lives in the private sector. If you just serve that individual in a clinical setting, that may be sustainable because in a private sector setting, that means you're going to
35:00
be willing to pay for that yourself. On the other hand, if you are only serving the public agency, but you're expecting someone to pay out of their own pocket or to go and seek out care, there's a disconnect there and incentivizing that sort of tool may, up to a certain point, won't solve that long-term incentive issue. So as a funder, I think we can really ask,
35:25
are you going into this alongside the community and alongside the individuals either seeking or providing care? Are you ensuring that you are interacting with the ministry of health? And if you want scalability beyond that country with the World Health Organization or other
35:40
broader standards setting organizations, or do you think this is fully sustainable as a community tool? In that case, let's still try to bring in the public health angle because we'll have to run into the issues we've had with this pandemic where data sets don't connect. So I think those are places where we can encourage. We can ensure that dollars out there
36:02
make sure the data is representative. We can pay for some of that piping, and we can make sure that some of the capacity building is done at the ministerial level so that they can be good partners and so they can make the time to interact with innovators that are outside of their immediate and urgent set of problems. Thank you so much, Arun. Quick announcement.
36:27
I would like to open the questions for the audience now as well. Please just type your name in the chat, only the name, and then I will call you up and then you can unmute yourself and ask a question. We are now more than 140 people in the Zoom room, and I don't know how
36:44
many in the VR room, but I think there are many as well. People in the VR room, you can ask questions just via the chat app. They will be transferred to me in a magical way, so we will answer your question as well. Please just type your names in the chat if you have a question
37:01
for our panelists, either with the name of the panelists or just an open question, and we can decide who wants to answer. And in the meantime, maybe we can talk a bit more about privacy as long as we wait for a question from the audience, because I think this is something I often observe that it would be perfect just to have all available data. Oh, I just see the VR room is
37:27
behind Zoom. Okay, they just have to stay a minute longer in the end, but this is our minute then in the end to join the VR room, and then we will be in the same time zone again. We made it possible to meet here for all different time zones. This is, by the way,
37:42
the reason why we have such a packed program from one hour and one hour without a break, and then again, VR without a break, because there's, of course, people from all over the world, just a little bit of time where most of us are awake and always someone else is sleeping in that time. Okay, since they're still waiting for questions, let's talk a bit about privacy.
38:07
We heard when Shwetak was talking, and as well, Aruna, you talked when we were preparing about this point-of-care technology, the idea to have all kinds of data on your mobile phone,
38:23
so that more or less your mobile phone knows you better than maybe your doctor. Is there a way to make this really privacy friendly? I think there is. There are still open questions, and I encourage people to continue to ask these questions even as
38:41
these tools roll out, but one of the kind of cutting-edge tools in artificial intelligence at point-of-care is the idea of people call it federated machine learning or on-edge or on-device federated learning, and it asks the question, do you really need a large algorithm running on a server or a cluster? Do you really need all of everyone's data all the time
39:05
to answer your specific question, or can a slice, kind of simplified machine learning model, and for the phone folks out there, something like a TensorFlow Lite model, run on-device and ensure that regardless of how much and how rich your data interaction is with that device,
39:25
that what leaves that phone is either very limited, maybe it's just an aggregate number, maybe it's something that actually has some noise added to it or has some some scrambling that's added to it, so that even though your phone may know you well, there's a very clear
39:42
line on what goes back to a central location and that other people have access to. Then one of the, just a quick example here, is around heart rate measurement. Heart rate, especially over time, we think might be identifying, which is kind of interesting because you have these patterns through the day, but if you can add noise to that data and report it, then
40:04
it becomes much harder to work with for privacy reasons, but if you average across a population and that noise that you add is centered around zero and distributed in a normal way, you can actually still pull out interesting population metrics. So we should think creatively
40:20
about how do we take data and allow it to be used for its full value at home, maybe make some trade-offs in these central settings, but still answer some interesting questions there. Well, that means maybe just don't go the easiest way or think about what do I really need and how can I make it privacy-friendly.
40:42
Do you want to answer this as well, otherwise I have... Just one quick thought, which is that I think if all of us introspect about our own behavior, I think all of us are willing to make trade-offs around privacy. I mean, Google Maps knows exactly where I am every time I open it. I'm okay with that because that means I don't have to
41:04
print out a map and bring it with me and I can find a coffee shop when I'm traveling. I think that people understand trade-offs and they're willing to give up some of their privacy when they get something in return and if they know that there's not going to be gross misuse of their data. So I think we should respect our own preferences that all of us
41:27
show. And sure, there's people who should be able to opt out and things like that, but I think many people, if they were getting something back, would actually be willing to contribute to research, contribute to something else. And so I think that right now we're ignoring the
41:42
trade-off and I don't think it's a clear trade-off, but I think people are very aware of the trade-off and they're willing to make it in our own everyday lives. I just want to second that for one second, Eva. I very much agree with what both of you said. I really think data privacy needs a rethink. If you are one of these people who live between
42:07
multiple cultures, you're very aware of how just different contexts, different privacy concerns, exactly as Ziad has laid out. We're already giving up so much privacy around all the, in the end, data that's collected for healthcare in our consumer behaviors as we start thinking
42:26
about the social determinants of health in a new way. So along the lines of what you both laid out earlier that, you know, we can't sort of just look at what happened in the past and then use that to predict the future. I think what we need is a rethink on this question and
42:44
understand what are our values at this point in time and then how do we communicate those or how do we discuss them with our communities so we can come together to some, you know, options, right? And then people can make their trade-offs within that framework so that we
43:01
don't sort of automatically rule out all the opportunities that can come from bringing some of this data together. Thank you so much. That's really interesting. I think we can, in the end, add some more points to your list and maybe give it as a handout to the community
43:21
just to avoid the biggest mistakes. So there's a question from Narinder Pannipun, is the first one. Narinder, can you unmute yourself? Does it work? Hello, everyone. I am Narinder Pannipun from India. So actually, I was just wondering, we have seen a lot of
43:42
potential of deep learning approaches in classification or segmentation. So in COVID, in COVID-19 pandemic diagnosis also, we have seen a lot of research work which have been done to either classify the COVID-19 cases using CT scans or x-rays. So I was just wondering,
44:03
these solutions being live or being like utilized by hospitals or any other organization like in the live? Yeah, I've worked a little bit on this with some of our health system partners. And I think this is a good example of why we need to think a little bit more carefully
44:25
about the labels for our data. So by the time someone is getting a CT scan, the idea that we're going to predict COVID from the CT scan does not actually help any doctor or any provider make any decision. Because at that point, we've got a lab test. We should just do the test.
44:44
So there are many things we'd want to know at the point where we're getting the CT scan. But by the time you're getting the CT scan, you're already worried enough about COVID that if a not very good algorithm predicted negative, you wouldn't take it seriously. If it predicted positive, you wouldn't do anything differently. So I think that there are many
45:04
things doctors might want to know when they're in the ER and faced with a patient who might have COVID. One is, is this patient going to deteriorate? Now, notice that you don't care about that just in people who have COVID. You care about that in anyone who has a pulmonary problem. So I think that this goes back to, I think, Stephanie's point of understanding the
45:24
context of the decision and the person you're trying to help. Predicting the results of a COVID PCR test, which has its own sensitivity and specificity and things like that, from a CT scan or even an X-ray, is actually not a useful task. And that's why it's not being used anywhere.
45:43
Thanks a lot to see it. The next question is from Michael. Can you unmute yourself, Michael? Hi, I'm Michael from University of Bremen in Germany. And you talked earlier about the biases that we have in machine learning because of the bias training data. And that reminded me
46:00
of an interview with Gabriela Kotsis, a newly elected president of ACM. And she suggested to introduce deliberate glitches into the systems to counteract those biases, to get it out of balance, if you will. And I wondered if that would be something for health or if it
46:25
may be critical, on the other hand, to introduce it deliberately here in health systems or medical systems. Can you just say, sorry, I don't know that idea. Can you say a little bit more about what that means? Well, if we have only, I think you gave the example earlier,
46:44
and Stephanie said that you have that kind of cancer that you never diagnosed before. So you will never understand with this trained machine learning because it was never triggered yet. So if we introduce some glitches that introduce some unexpected data in there,
47:09
we might end up having a better system. Maybe I can jump in with a few quick thoughts. So I think one, I haven't heard the term glitches, but we do talk a lot about data
47:21
augmentation, right, which could be an interesting way to look at this. Like where can you, if you have a population that's underrepresented manually or which changes to your segmentation, like add additional features in. I think another dimension that is maybe important to keep in mind
47:41
is if you're afraid that your algorithm is specifically going to learn a bad habit from society, like ignoring people who have a certain skin tone, then don't use the skin tone as one of your predictors. And I mean, that has its pros and cons, but it's sort of making sure you
48:00
don't overfit on a set of behaviors that are more about society than they are about biology. So just two thoughts that maybe are adjacent to that idea of the glitch. Thank you so much. We have, I think this is maybe more a comment than a question from Olutola. Or do you want to ask that yourself, Olutola? You can unmute yourself if you
48:24
want to, maybe the mic doesn't work. So I just read it. So from, Olutola is from Fora Bay College in Sierra Leone, and she was, or he was really interested
48:42
in the contact tracing app for Ebola, which can also be deployed for COVID-19 as well. But I think this is a follow up to what Stefanie said. NGOs that developed that could have made available before leaving, but it looks like they have left. Yeah. So I put a follow up in the chat, the app is called ComCare and there's a link that gives
49:02
you more information about this. Great. Thank you so much. Then there's Remko with a question. Yes. Thank you Eva. Remko Benton the Grave from Newcastle University. I have a question for Ziad. I really appreciate how you brought up the point of, are we optimizing to the right
49:26
goals with machine learning? And you mentioned need as something to optimize to, and I was wondering how do you optimize to need? Yeah. And I think that's the question
49:40
that I basically spend all of my time trying to answer. And I think that one of the traps that I think you can fall into is that I think, my sense is this is what happened with all of the interest in predicting cost, is that we tried to come up with one universal measure of need
50:01
that works for everyone, everywhere, no matter what their medical problem is. And so we just aggregate a bunch of things, but our aggregation function is very weird because the prices are set in arbitrary and bizarre ways. And because of all of the biases that we introduce when we use cost instead of need. So then what do you do? Well, what we did
50:23
in this paper, and I'll post a link to the paper in the chat in a second, is I wouldn't say this was a particularly intelligent solution. It just happened to be better than what people were doing, is we basically tallied up all of the illnesses that you would eventually have to
50:42
see a doctor for over the year after the index date. Now that has a number of the same biases as cost because it still requires you to come in to see a doctor and be diagnosed with something. But since we were dealing with a population that had insurance that lived close to the
51:01
hospital, that didn't face transportation barriers, some of those issues were mitigated. You know, there's another paper that we've worked on where instead of using, for example, this is an algorithm that trains, that looks at someone's knee and tries to decide how severe their arthritis is, rather than training on what the doctor says about the image in terms
51:24
of arthritis, we train the algorithm to predict how painful that knee is because we had a data set that linked the image, the appearance of the knee to the pain score. And that turned out to be far less racially biased than actually what the radiologist had said about the knee.
51:40
Because, you know, the way our scientific knowledge about arthritis has been built up is primarily in white populations that have different problems than non-white populations. So I think, unfortunately, the answer has to be very tailored to the particular problem that you're studying. And I think that's why these issues can be so hard because you need not just the data science language, you need the language of whatever the domain
52:03
that you're studying is. And I think this comes up in criminal justice, it comes up in finance, it comes up in hiring. You really need to know what the real measures of quality are to get your labels. Thank you so much, Ziyad. Then we have Tohoku. And I'm sorry if I don't
52:21
say the names right. It's sometimes hard for me to find out how to say Tohoku. Can you unmute yourself? Yeah, okay. Thanks very much. My name is Tohoku. I'm from Trinity College Dublin. My question is centered on the design of mHealth interventions. And then I would quickly like to
52:44
know how do we actually design to engage users? Because there has been an increase in mHealth interventions to change health outcomes, such as maybe maternal, Ebola, COVID,
53:02
and the treatments are different. So many of these interventions end up being unsuccessful. And I would like to chip in because the topic of this discussion is how to best. So I would
53:27
like to know designing to engage users. So the question is turned up to the panelists. Who wants to answer that? Designing to engage users? Maybe Stephanie, is that something for you?
53:44
Sure. I mean, we call it the mHealth pilotitis, right? There's hundreds of apps, they come and go, like there's lots of investment. And a lot of them struggle from exactly the questions that you mentioned, Tohoku, which is sort of too much hype and not enough reality.
54:05
I guess when you say designing for user, I think that is exactly the right concept, right? They need to include user studies and they need to include an on-the-ground understanding of the environment. I also think a lot of them struggle because they assume that data
54:25
is available that in the end isn't either there or is not compatible or is not yet available. So those are some quick answers. I don't know, Arunan, if you would like to come in with some of your experiences. I think similar to the way that I think
54:43
Seob framed the need question, I think engagement is also a very tailoring question. To give you a sense of how large this field is, this is the time that Google and Facebook and other kind of ad-driven agencies, their entire focus and their entire data science staff
55:04
are looking for these small optimizations, all the way from how do you smooth in the sign-on process, how do you turn that into a single click, to what's the optimal frequency to ping you with notifications. The reason I say that is it's very different if you're driving towards
55:20
adherence and you need engagement over the long term than if you are trying to just get someone through an entire survey during a single interaction. The elements that come into play are going to be different, but I would look closely at kind of what's been learned in the marketing domain, if it's something longitudinal, what's been learned in how people form habits and
55:42
how people establish recurring routines in their lives. All those are, I think, rich social science domains to bring into your app development process. Now it's unfortunately exactly 6 p.m. and I would like to take care of all of your time because I know everybody is
56:03
busy and there is another really exciting panel coming up, which starts right now. So because we have this packed program, all of you will now be able to see how we quickly change our seats or don't do that because we don't have to. I'm really curious how I can find out if my other panelists arrived. I see that Shvetak is already here. So I'm sorry for, there are two or three
56:25
more questions in the chat and some comments. The good thing is that we just go on with the same meeting. So the chat will stay there and you will be able to read the comments until, so one more hour until 7 p.m. Berlin time. And the other good thing is, because I see that
56:44
there are questions still open, we will be meeting afterwards in the VR solution and there we can discuss with the speakers who will be there as well. I will be there as well and I hope all of you will be there. So we can just discuss all the questions we are interested in, just in smaller groups or bigger groups, how many are interested. But now I'm sorry I have to move on
57:04
to the next panel. Thank you so much, Stephanie. Thank you, Syed. Thank you, Aruna. It was really a pleasure talking to you and there are so many interesting things I would like to go on and discuss. Maybe just have to do another panel next year on the same topic. I think it helped people a lot to hear your examples. Stephanie, your list is great or the first few bullet points
57:25
you told us about the list, how to avoid the biggest and yet not intuitive mistakes of innovation in the healthcare sector. And I think it helps all of us to rethink how we deal with
57:40
these topics and I really think that out of this group, really big group of 140 people here and I don't know how many in VR but still many more, there will be some great future innovations and we learned a lot. Thank you so much. So I see Shwetak is here. Welcome to our next panel. Is Catherine, are you here as well? Yes, I am. Hello. I see you here as well.
58:11
And Aisha, are you here as well? Aisha, welcome. Okay, great. I have to figure out how to get you
58:23
all on my first screen but I will find out. This is easier in real life. But now I have you all on the first screen. Please stay there. Don't move. Don't turn off your camera. So yeah, as I promised without a break, I'm sorry for that. Who wants to
58:44
get a drink? Maybe just miss the first question or just wait for another hour. That's the second panel. So since the first panel was more around a general overview of global health and technology and what can be done with it and what are our challenges and chances. The second panel will be on apps. We have some amazing speakers here. We've all
59:07
worked on innovations in global health and technology and have tons of experiences for us to share. And we have one hour to listen to them and to share experiences. I would do it the same way as I did last panel. I would start with the first half,
59:23
first two thirds with a panel discussion with the three of us moderated by me. And then I would open the discussion for the audience and I would like to ask you just to write your name in the chat if you have a question and then you can unmute yourself and ask the questions because this is much more interactive and much more fun. People in VR, you can ask questions as well. Just write them in the chat and they will
59:43
will be transferred to me and magically and will be added here in the chat as well. So everybody who wants to ask something, please, please ask. So welcome, Catherine Chao, Asia Walcott and Shwetak Petal.
01:00:00
Shwetak already gave a keynote talk as an opening for our topic. He had the idea for this topic, and we organized all the panels and topics together and all the questions. So I'm really happy to have him here as a speaker as well.
01:00:21
I would like to introduce to you Catherine Chow. She is the director of research and innovations at Google. She's developing products that apply AI to health care and social good. She previously developed products within Google X Labs for life sciences, which is now Verily, and is also a co-founder and committee chair for the AI for Social Good program at Google,
01:00:41
which is great. Outside of Google, she's a board member and program chair of Lua Wildlife Conservancy, a fellow of the Zoological Society of London, and she collaborates with other wildlife NGOs in the Cambridge Business Sustainable Program in applying the Silicon Valley innovation mindset to new areas, which is really important,
01:01:03
I think. Warm welcome, Catherine. Thank you. Then we have Aysha with us, Aysha Walcott. She leads a team of researchers and engineers at IBM Research in Africa who are working on innovations in global health using advanced AI, blockchain, and other technologies.
01:01:21
She sits in Kenya, by the way. She and her team are engaged in projects in disease intervention planning, including COVID-19 and malaria control, which we will talk about later, maternal newborn child health, family planning, and water management. Aysha has recently worked on a rich data set for tracking non-pharmaceutical interventions
01:01:41
for COVID-19, and she earned her PhD in electrical engineering and computer science from MIT. Very warm welcome, Aysha. And finally, we have Shvetak, who many of you already know and already maybe heard, listened to his talk. He's my fellow organizer. He's directing the UbiComp lab
01:02:02
at the University of Washington. Concurrently, he's also a director of health technologies at Google. His research is in the areas of human computer interaction, ubiquitous computing, and sensor-enabled embedded systems. And I think he talked quite a lot about this in his opening talk. He has a particular emphasis on the application of computing
01:02:20
to health, sustainability, and interaction. Shvetak received his PhD in computer science from Georgia Tech in 2008. He was awarded with multiple prestigious award, and I don't have the time to name them all, but you can find them if you Google his name. But one of them, of course, is the ACM Prize in Computing,
01:02:41
which is why he is a laureate here at the HLF. Warm welcome, Shvetak, back on the stage. I would like to open as well this panel with a kind, very open question, and because this is a question to all of you, because this is a very concrete panel, concrete topic,
01:03:01
our concrete apps and innovations in global health, I would just like to ask you, what are you working on right now, and why do we think it's important for global health and promising? Let's start with you, Aysha. Oh, hi, yeah. So current work on the way is definitely
01:03:21
almost essentially everything you mentioned. So co-leading an effort across a couple of actually continents on the non-pharmaceutical interventions for COVID-19, and also leading the AI for global health efforts from projects with great teams in disease
01:03:44
intervention planning, maternal newborn child health, and a little bit in family planning. So that's the active threads of my life. Thank you. Thanks a lot, and we'll hear much more
01:04:02
about the current work. Katherine, what is your maybe most important project, or why is it important? Thanks, Inge. Yeah, I currently, well, I've been working in the life sciences health care space for six years, and we started in research looking at how data can be
01:04:26
useful and meaningful, and learned a lot through that process. And so what's happened is there's been a lot of innovation. Some of the more clearly established areas now are peer aid diagnostics, and be
01:04:42
able to reduce medical errors and increase accuracy for screening, and things like that, which has a nice global impact, because it does help scale up kind of medical expertise around the world, but definitely has faced challenges when you're looking at distribution, going to market, or these kinds of things. There's additional work that's spun up.
01:05:03
We've been working on it for a while, but actually, it came to a point with COVID-19 around public and environmental health, and how data can end up back in the hands of the communities and the public health agencies so that they can actually be utilizing and making
01:05:20
much more informed decisions, especially when they're trying to steer populations and give these non-pharmaceutical interventions alternatives, and even public messaging, and how can that be leveraged to help increase education awareness.
01:05:41
So there's a lot of work going around on that area. And then there is work that we do with global health that's looking at mobile specifically as a mechanism where we believe that it's hopefully could be a leapfrog technology
01:06:05
compared to where we see some of the developed countries right now very entrenched in facility on-site care, moving it more towards the peripheries and the edges of care through community members and through empowering individuals who are either
01:06:25
at the community health care worker level or requires less training and hopefully can be equipped with technologies. But that's been a promise and dream
01:06:41
that it is actually making it happen. It's fraught with a lot of complications around both incentive alignment and really plugging through the challenges of connecting the dots for infrastructure. So I think that's been interesting to try
01:07:02
and work through. And it requires a whole ecosystem and set of developers and government and everybody to come together to actually have that shared vision to make that happen. Thank you so much, Catherine. You were breaking a bit in the last few sentences. But I think you came back quickly.
01:07:21
So I think we've had it. Greta, maybe just quickly, because I know you already told many of us what you're working on. I'm not sure everybody was at your talk. Maybe just quickly, what's your thoughts? I've had the fortunate opportunity to actually collaborate with some of our panelists, Arun
01:07:41
and Catherine. So it's a big community, but also small communities, so to speak. But my passions have really been around a couple of things. One is looking, as Catherine mentioned, the things around mobile. For me, mobile is just beyond a data collection tool. It's a way to get objective measures out in the field,
01:08:03
so sensors that are on phones and looking at how we can push the boundaries on what the most ubiquitous computing platform out there, which is the phone right now. So that's an area of passion for me. The other area is looking at the intersection of AI and sensors, so looking at how the amazing work that's happening in the AI community,
01:08:20
how can you bring these low-cost sensors together to drive new ways to capture signals and biosignals. I mean, these are things that my teams and my grad students at the end of the day, they're the ones that do the amazing work. I'm just here presenting this stuff. At this point, for me, it's really been also connecting communities together. So I'm really passionate about looking
01:08:40
at how can the research community work with regulators? How can the regulators work with industry? How do industries and NGOs to work together? How do researchers across different sectors work together? So for me, it's looking at how do we reduce the boundaries of collaboration so that we together can push innovation in these spaces? But yeah, and that's kind of what I've been focusing on
01:09:01
as well in addition to some of the basic science research. Thank you so much, Schmitak. Since all of you work with data, a big topic we already started discussing in the last panel is privacy. Because of course, as you say, Schmitak,
01:09:21
and Catherine as well, I see that the opportunities we have with the mobile phone, all the data we can get from there, especially health data are really sensitive data. So how do you work around privacy? What did you learn? What can you do to make sure that these data
01:09:41
don't get in the wrong hands in the end or get hacked or yeah. Who should know everything about myself or does anybody have to know everything about myself or is there a solution that only I know? I need to know. It's a question for all of you who wants to start.
01:10:00
Yeah, I can start. So from, I guess, my organization's perspective, so from an IBM perspective, we usually don't hold data. We usually would license data and have access to it from a different facility or a different organization
01:10:21
and then sort of provide the analytics on top of it. So definitely I think coming from private sector, there's obviously the ethics and the guidelines that have to be adhered to in the regulations based on your environment. So definitely extremely sensitive to that. We do have a study, an IRB study underway in Kenya
01:10:44
working with the Kenya Medical Research Institute, which is exactly looking at the question of patient consent managed and sharing of their health data using blockchain. So in this case, their information is logged on the ledger and so there's a number of hypotheses
01:11:02
around usability from a patient perspective, sharing an exchange from provider to provider and the deployment of such technology here. So that's actively underway and also provides an example to see whether or not people want to bring
01:11:20
both their clinical and nonclinical data into let's say their own sort of personal health data network and then consent to share an exchange within a clinical encounter with a community health worker if you need to go to another institution. So I think there's technology that supports the privacy
01:11:41
that we're testing out. It's whether or not the local communities would adopt it and then there are all the sort of legal and ethics review boards that are in place that need to be adhered to. Thank you. Great, thank you so much.
01:12:01
I think the other big topic, I think that's for Shwetak as well, for Catherine is because you both work for Google as well. Google is at least from a European perspective often in the focus when it comes to privacy and I saw similar discussions in the US when you released this mobility reports, Catherine,
01:12:23
about why does Google know where I am? Isn't that rather surveillance than just helping us to learn something? How do you deal with these critique or these discussions and what is your work to help you understand
01:12:40
or to keep privacy to make sure that you're at the right place? So I think there's different types of data in the healthcare space. There's data that historically is what we call like our medical records and that's actually not within,
01:13:00
typically starts off within the control of, in many countries, I think there's some exceptions here where it's in the control of the patient or the user. It's actually part of the healthcare system and the user has to actually, or the patient has to actually request to be able to get access to that.
01:13:21
Where I think there's a huge opportunity growth is to build out the personal health records. So that's the kind of record that people have had to keep. And you see it happen, especially in situations where people want a variety of information either across healthcare systems or they're trying to collect insights from other sources,
01:13:43
especially if they have like rare diseases and things like that. And so this concept, I've seen patients who walk around with their own like health record. And I think there is a significant difference between these different, even though they're the same data modalities, it's the purposes that they're serving are different.
01:14:02
And so I actually think of data, not by the data type, but by what it's been consented and authorized and permission to do. I think when you're working in the consumer or the user and patient space, it really has to be fully transparent and controlled and the user needs to be aware
01:14:22
and have options to be able to say that I can see what's happening and I wanna participate or I don't, and I want to keep it my own personal vault. And so I think that is, I hope the future for users is that there's certainly data that, again,
01:14:43
the same piece of data could be either inside the healthcare system, serving the purposes of what a healthcare system or let's say a pharmaceutical company that's running clinical trials is, that's the data they've collected. And you have at some point in time signed and agreed to give them that data.
01:15:01
But I think that this is a lot of it has to build the kind of underlying infrastructure to be flexible enough to allow that level of control. And there's different tiers too. There's data that I'm comfortable with only existing on my device. And there's some data that I'm comfortable inside being shared for particular purposes. And there's some data that I'm totally fine
01:15:21
with having aggregated and anonymized for the purposes of population health or public health, like a better understanding of what's happening in the ecosystem. And I think that's up to the individual, but we should put those levers in place in the infrastructure so that people can actually have that choice and we can maximize the benefit of the data as it goes through. So with community mobility reports,
01:15:41
it was, that's an opt-in kind of users choose to opt into that. If they chose to opt into user location history, which we made very transparent when we put it out there, that the community mobility reports was one of the services that was provided through participating in that. And people can either choose to be part of that or not.
01:16:03
Oh, thanks. That's interesting. So people could choose if they want to be part of that. And if they want to have their location tracked or not. Yeah, and because it was aggregated, it's used as differential privacy. Their privacy is completely protected
01:16:20
and it's actually was, there's a whole paper written on how we do that, where you can actually still get the insights of mobility trends and not have to lose that. But there's places where there wasn't enough people opted in and if there are not enough people opted in with that, then we just can't show any information there.
01:16:42
Because then it's not anonymous anymore or people can, you can find out who that one person was. Right, so if you were really clever, you could try and kind of reverse engineer and then you wouldn't be able to figure out the person, but you could kind of figure out the pattern of maybe direction. So one of the things about location information is you don't need to like a person's,
01:17:02
like where they are every single moment in the day is like a type of identifier. So there's all sorts of techniques, really good ones in the privacy sector that are being published and being implemented, which allows for anonymization of the data
01:17:23
and still providing either sequential like relevance or being able to provide sort of like larger aggregate relevance. And so again, differential privacy is one of those techniques. There's many, many others that actually are pretty effective, are very effective actually. Thank you so much.
01:17:41
So all of you, I know that all of you work on projects on COVID as well and I know that AI is very promising here, but this is often the case, right? That AI is promising and then we find out that something went wrong. I would love to hear, but before we discuss, oh, Ayesha, Ayesha, did you leave us?
01:18:01
Just wanted to ask you something. I thought we had a power outage. Before we start talking about the challenges or problems, I would like to hear about your recent projects around AI and COVID and Ayesha, I know that you made this really interesting tool
01:18:21
to find out about non-pharmaceutical interventions and what they do and maybe just tell us a bit about what is it doing and how does it help? Yeah, so basically we wanted to figure out, well, what is everyone doing across the world in the middle of this crisis in terms of limiting the spread of the disease?
01:18:43
And it's obviously a brand new virus, so the different types of interventions that would be proposed could vary from geography. So we used NLP methods to comb a Wikipedia as a starting point corpus,
01:19:01
which has a whole number of COVID-19 pages for every geography that essentially get updated quite frequently. Sorry, I just want to make sure that people know what NLP is, so it's basically text language processing. Yes, yeah. Yes, I'm sorry, I haven't calibrated to the audience.
01:19:24
Yeah, so natural language processing, so it's essentially reading these Wikipedia pages that can change at any point in time that say things like all New York City schools have been closed on April 13th,
01:19:41
or it might say that only the kindergartens are closed, so it can have different resolutions. So that's happening everywhere. So what we did is use NLP natural language processing to so-called read these pages every single day. We do an update, we call the sites,
01:20:00
we check for the changes, and then we have human validators in the loop that take these, I guess, evidence that an event had potentially occurred, and then those validators can basically say, yes, that is a non-pharmaceutical intervention event,
01:20:22
or maybe not, right? And then they also can add additional sources of reference like executive orders and others just to corroborate the source of the interventions. And so the reason why this is very interesting is that it's different from the other efforts that are usually heavy human-based.
01:20:41
This one is more on the AI side, and then limiting the amount of human volunteers that are needed, and that way we can keep a more up-to-date reference. It's publicly available, anybody can use it, so I can share it in the chat here. And the main thing is it's for public good and it's for science.
01:21:00
So the hope is that now we can start to use this data set with the existing COVID-19 modeling efforts, whether they're compartmental models or agent-based models to help strengthen the prediction power of the models and to do what if and decision support. So that's an angle in a way that NLP
01:21:23
actually comes into play in a healthcare space. And what I really love about this project, I really think is interesting that this is a dimension that people, manually or people just with our brains couldn't do just because of the massive amount of data.
01:21:41
And of course, you have to find those data in Wikipedia or news articles. So I really think this is interesting for researchers to work with that data, just to find out correlations between these measures and COVID numbers. And yeah, to find out much more about things we don't know yet about COVID.
01:22:01
Yes, exactly. Catherine, you work with AI and COVID as well by just trying to find out what people are, I'm not sure if it's AI or it's just statistics. Some people say AI and statistics are the same, but that's not the topic today. You work with search words, what people search for,
01:22:22
and I think you try to predict the next outbreak or the next week. Tell us a bit more about that project. Well, so last week we did, well, this is an accumulation of six months of different people within the organization working with external entities and third parties
01:22:43
and researchers to look at what data is actually useful and what we did was we helped bring together a lot of public data sets. So even the public data sets were highly irregular, they were very scattered
01:23:01
and they were formatted very differently, which means that if you're trying to do an analysis, immediately all of our modeling researchers who are extremely keen on helping realize they had no data to work with. And so the data just went through this process of realizing that there was good data actually out there,
01:23:23
just harder to find the public and important kind of covariates, economic indicators, like population statistics, all these things that just needed to kind of come together. So we have this COVID-19 repo that is open source GitHub
01:23:41
that helps kind of bring together across 50 countries data sets that would be useful and hopefully what'll happen is that's just like for the community and they can continue using that for COVID-19 and future epidemiological needs. Then on top of that, we layered in the mobility reports data, which was one of our earlier data releases
01:24:02
and it was just because we had heard so much that people were flying completely blind when trying to figure out what their social interventions or the social distancing interventions, if they needed to do them, if like they were being effective at all.
01:24:20
And that's been actually one of the more highly leveraged data sets, the more recent one, which is as I mentioned, it's kind of search trends data. So we have search trends for a long time now. We had though not actually, I think what we had heard and feedback for search trends, there's this thing called flu trends ages ago, it was kind of interesting,
01:24:41
but it wasn't actually that useful for epidemiological research just because of both the timeliness and the granularity. And so we addressed both of those as well as trying to expand upon this symptom under landscape. So there's about 400 different symptoms now covered,
01:25:02
which kind of, again, this is maybe more useful for our hypothesis generation in our hypothesis testing. But I think that this all needs to be in and of itself, the data is not actually sufficient. It's supplementary to like,
01:25:20
I think what people are already doing today in their research and hopefully additive. Then we provided a bunch of tooling on top of that. One of the ones that is more recent that is based in AI and showing promise is simulations. So we have these agent-based simulators that utilize real-world data, and it allows for like public health organizations
01:25:42
to fine tune particular models. So like if they have an exposure notification model that they're using, and it does allow for them to just consider what's going to happen in these different what if scenarios, if they're gonna have different interventions, what's the impact of those potentially?
01:26:01
I think this is early stages, but it's definitely a potentially better, like in terms of the complexity and real-world variables that happen, this starts accounting for more of that. Also, I think we saw a lot of models go out there without like proper kind of understanding of uncertainty. And so we also have these tools
01:26:22
that help with like uncertainty evaluation when you're building models. Is this uncertainty evaluation about showing people what in which dimensions the prediction could go? Is this visualization or? Yeah, so like error, like I think it's important for people to realize
01:26:43
that like in certain circumstances, the model performs really well. In other circumstances, it's highly uncertain. And you actually probably should either be supplementing additional information or recognizing that your decisions are being made on far more uncertain basis. And so this is particularly critical
01:27:03
if there's only like state and national level data sets and you're trying to gain like city or county level insights and you kind of wanna know how your uncertainty is being distributed. Thank you so much. I would like to come back to this example later again
01:27:22
about new trends and about this sometimes surprising things AI is doing and which nobody thought about or didn't know how that happened. But at first I would like to ask you Shvetak, I heard in your talk, you talked about cuff monitoring
01:27:41
on the mobile phone for COVID-19. Can you tell us a bit more about that? How that works? Is it already, does it already work? Yeah, so that was just an example of one of the crossover opportunities from the TB work we did years ago in the research space. Some of that stuff was in collaboration with the Gates Foundation where some of those models
01:28:02
were really around looking at, could we detect super spreaders? So TB spreads roughly similarly to COVID in the coronavirus in the sense that it's aerosolized, there's particles that can be, that are in the air that you can infect people with both surface contact and in the air.
01:28:22
But just similarly how it's under explore, it's been explored but it's not a good, we don't have a good sense of how it spreads. And so cough was used as a proxy to see if that's an indicator for spread, looking at it, looking at cough as a pre-symptom or even post infection. But it was interesting because that area didn't get a lot of attention because pulmonologists,
01:28:42
you typically think about cough as like, hey, that's a symptom that is part of pretty much every diagnosis. So it has no entropy in that signal, right? A cough is a cough, right? There's no entropy in that at all. But this is kind of an interesting way of thinking about cough from an AI standpoint, that if you look deeper in the cough, the physiology manifests itself
01:29:01
in a different way from the cough. And so that was an opportunity where for TB, we're looking at it and from a standpoint of could you diagnose upper and lower respiratory diseases and those kinds of things. And it turned out that there's a significant portion of the research community jumped on cough when COVID happened. And so some of the data sets or some of the models that were generated in the community,
01:29:20
people are building off of, which has been a interesting way to look at that. But I think some of the crossovers are interesting to look at too, because often you don't need to reinvent the wheel. There's a lot of stuff already out there that we can take advantage of. So that's a particular example I mentioned. But another quick thing to think about for COVID
01:29:41
is also looking at tools that we can start to help for the broader population. A lot of things we're looking at for maybe for older adults, people with risk factors. But if you start to look at the data now and we're seeing unfortunate upticks in terms of the number of people that are infected,
01:30:02
you're seeing the younger population starting to be infected. But the way that they're symptomatic is starting to be different than earlier this year. So I think that's gonna be a moving target for us. So looking at other signals are gonna be really critical. A cat alluded to this, where it might not even be a physiological signal.
01:30:21
It might be an environmental signal, a secondary signal. So I think that's another area I'm interested in right now is looking at other ways that we can help the health community find signal that can tell us where things are going in general. And the idea is to find them just via correlations and big amounts of data. Well, yeah, I mean, just hone in the selection space.
01:30:43
There's just so many places to innovate. What do we focus on? So I think the kind of work that Ayesha and Kat talked about, you can start to hone in on what are the specific types of areas that we can start to dive deeper in that we may have some ways for either diagnostics
01:31:00
or screening or those kinds of things. But I think we just gotta figure out how to get the search space down a little bit. And I think that's what those kinds of approaches can enable. Thanks a lot. So there's already questions coming in. I guess I just open the discussion to the audience as well and start with some questions
01:31:20
because we didn't have enough time for the questions last time. So the first one, so at first, again, my idea was that you just write your name in the chat and then you can ask the questions yourself. Shady, you posted the question already. Should I read it or do you want to unmute yourself
01:31:41
and ask yourself? Shady, maybe I read the question. So the question is, there are many ways to train models in privacy-preserved fashion like federated learning and others. However, I was wondering, what is your take on accountability, Catherine? That's the question, Catherine, for you.
01:32:05
Cut out, can you say that one more time? The accountability, that's what I heard. There are many ways to train models on privacy-preserved fashion like federated learning and others. However, I was wondering, what is your take on accountability?
01:32:22
Um, so is there a way for me to have a dialogue with this person? I just want to confirm for accountability, I believe what's being asked here is who makes the model and what's the output of that model?
01:32:42
Oh yeah. Sure, yeah. Thank you. So my question was, who said what? Like, this is something, this is one of the questions that hospitals would like to know, who actually saved this kind of diagnosis, right? So this is kind of accountability
01:33:01
that I just would like to know your opinion about. Got it. Okay, that's very clarifying. So, there's models that are used, that I've seen used in diagnostic settings and then I've seen it used in screening settings.
01:33:21
And then, well, so I'll start at, there's many levels of accountability that's required, I think, for models. And in fact, I think they're, even for technology as a whole, it might behoove us to think about running trials, the equivalent of like clinical trials for technology
01:33:41
to verify the impact of those technologies. When I think of accountability, the model needs to be accountable to a certain level of peer-reviewed scrutiny that professional kind of people who understand
01:34:02
and are accountable to the greater population protecting patient safety understands what the model's doing and has reviewed and approved that model for that use. And that's how I kind of think of the role of the FDA or CE marks and things like that. But then there is sort of in practice
01:34:22
when you're using it in healthcare systems. And I think my initial preference has always been use it as a kind of second opinion where especially when it's kind of these critical interventions that might happen afterwards
01:34:41
as a result of the decision made. So, using it as a way to help allocate resources more efficiently so that you can triage more effectively, use it in screening situations. I find that to be something we should be heavily leveraging machine learning for. The one where you use it as a final,
01:35:02
like somehow there's not a doctor in the loop. I'm not so sure that we're ready for things like that. So, there's a point in time where there's like low enough errors and there's regions where there's just not enough medical specialists actually to check. But I actually think we should get to a point
01:35:21
where building systems that empower statistically thoughtful medical specialists who are medical practitioners who actually can see a whole population and actually still be doing some level of oversight
01:35:41
or overview of what's going on. So, I think there's many, many levels of accountability that should be built in when using technology. And it's not just AI, it's really all technology should have some of that factored in. And so, yeah, my general thought is that there's more to be done here.
01:36:01
And certainly I think when you build the model, there's where did the data come from and are you being accountable to the sources? You should have a very... And so, we've done a lot to vet to make sure that the data we're working with is if there's discrepancies we've adjudicated for them and all of that. But I think there's just many checkpoints
01:36:22
along the way for accountability. Yeah, I just like to chime in. So, Maria actually said something really interesting in there. Is it really accountability or is it provenance? I mean, that's a great question. If you think about from a health record standpoint or even just a diagnosis or a screen,
01:36:41
provenance is really important from the standpoint of like who validated this or how do we know that it's this thing, it's certified or valid from a trusted organization or an entity or an individual. So, I think that's one of the things that's really fascinating about the AI space is that at some point we have the accuracy
01:37:01
and the efficacy part of the question. And the other part is which model said what and how do we know from a provenance standpoint, how to trust which model gets applied to which scenario? Because if you think about validating test results and those kinds of things, and a provenance is really important, but provenance is fairly straightforward in the sense that, hey, this result came
01:37:22
from these four organizations that we trust. But when you get into the space of AI, it's not just the organization, it's which model, which timestamped model was used on what data set. And that could get very tricky. So, there's some interesting opportunities for researchers to be able to maybe, how do you watermark the models?
01:37:40
How do you watermark the result, which you can backtrack which data set or model was applied? I think these are some really interesting open questions that even regulators are trying to think about. But that's a great comment because I think provenance is related to that. And I think that's something that we don't often think about, but something that is gonna be really important moving forward. And I think maybe to bring Aisha in the discussion as well,
01:38:04
I think one important or maybe first step to accountability is as well the human in the loop, you talked about. Maybe just give us a few more information why you think human in the loop is important or maybe examples what you could detect
01:38:20
or prevent with that human in the loop. Yeah, so to piggybacking off of what was just said, especially around provenance, thinking about our malaria control work, that's with the Gates Foundation. So this was an example where you have longstanding malaria modeling communities
01:38:42
and trust was a big thing. Trust was a big thing from the modelers who spent their PhDs developing the models. What are you gonna use it for? How are you gonna calibrate all sorts of things? Trust was a big thing from policymakers in terms of what model I should use for my context and how do I say that you told me to use this data,
01:39:00
this model to get these outcomes for this grant, right? And so to try to address that, we did build it on top of blockchain. So we allow for the trust and transparency there, but there are like the model fingerprinting and watermarking angles that Swatek also mentioned that can be done as well.
01:39:22
The point is to be completely transparent and traceable. So we weren't developing these models for malaria. They are very strong, rich models that already exist. We wanna convince the modeling community that if we bring AI algorithms, for example, like reinforcement learning,
01:39:42
which allows you to do sequential decision-making and explore, you try a couple of interventions, you look at the future state of your world, have you reduced prevalence, have you reduced mortality? And then that's a sample. And then you try some other combinations. So it's a large search space, but in order to bring all those models in,
01:40:02
you do have to have trust. So I think that question was important question there. The human in the loop, I think, so from a global health perspective, a decision-making perspective, the humans are always there. And then from a validation and trusted data source, like thinking about our COVID-19 data source,
01:40:23
we wanted to have that veracity or that double check there. And it still won't be perfect, but at least there are some things that are very clear to us that right now, AI doesn't have a clear, can't necessarily clearly discern
01:40:41
without a human in the loop. So I do think it's important. And I just like to add to what Aisha said about just the validation with human loop. We had a pretty interesting, like in some cases we saw when we're building period diagnostics
01:41:02
that you could get to kind of state of the art, like on par with medical specialists, but there's still a huge benefit to having human loop on multiple fronts because you want to take into account the holistic care of the individual. There's a lot more to be done. And so these individual specific kind of like detections
01:41:24
of diseases is only just a part of the picture. And then even in cases where we were doing work in pathology for detecting breast cancer, metastases and lymph nodes, we found that while our models could be better
01:41:42
at resulting in catching greater percentage of the cancer lesions, we still had higher false positives than the pathologists who wouldn't have any so the combination actually was far more accurate than either alone. So it's really should be seen as a tool
01:42:01
and without I think somebody looking over how it's being used, it can be used incorrectly and actually detrimentally. So I think there's to me a human should always be in loop somewhere. Thanks a lot. There's another question from the audience this time from VR. It's Jason White asking,
01:42:21
what can be learned from these promising projects where the AI goes wrong or has there been a systematic study of? So is there a word missing? Maybe just use the first sentence because Jason is not in his Zoom room and cannot answer if there's a word missing. So what can be learned from the promising projects
01:42:42
that were where the AI goes wrong? I think this maybe goes in the direction I started or I announced I want to discuss later. There was this discussion about Google flow trends which I know it's a long time ago which as well looked on search trends. And I remember that there was just some,
01:43:01
I think some flu peaks were completely missed or others were predicted and didn't happen. And I think this is at least my guess is this is just some of these typical AI related problems that there are some predictions and you cannot explain how they happen. And I guess there's a lot of work done last
01:43:20
I think that's two years ago, maybe to avoid that or to get an insight what AI, I think partly it's about uncertainty or we had partly it's about accountability. But I guess this is the direction the question goes in and now I ask much more than Jason.
01:43:45
I'll take a first pass at this. I think that I do distinguish in my mind the data can't be, it's not separable from the model. The model is entirely built off of what data you feed it. And so there's a whole set of, I think
01:44:03
growing body of work around like, how do you look at the data that you're learning from make sure that you've done a good job with that. And so flu trends is data more than it is. I consider it a model it's sort of data reflective of what people are searching for. So it's not even, you have to be careful about proxies.
01:44:23
There's this evidence that like really bad models got built because they chose the wrong proxies. So where it actually looked a higher racial discrepancy as a result because they were using kind of monetary or they were using kind of claims data
01:44:43
or they were using transactional data to determine as a proxy for overall care being delivered for a particular region. So that actually has been something where you just have to be really careful about is there bias already and systematically embedded in the data that you're getting.
01:45:02
We're already, because we're in the digital space we're already somewhat biased because it means all people who aren't collecting information or aren't represented by the digital spaces like their data is not getting picked up. So people just have to be aware of the data when they're building the models. But then there's also the failures that happened where I think it was quite preventable
01:45:21
where I had heard that people take models in one region and they don't generalize it to other regions or they aren't doing the distributional shifts and things like that. So there's a lot of techniques as well to make sure that your models are robust. You can also have, this is my explanations around models are so important. Like be able to see and have greater insight.
01:45:43
There's saliency maps to help with this to essentially see what's going on behind the scenes for what is the features that the model's looking at. And so again, these are all just different I think techniques are being built up over time. Sorry about that.
01:46:00
To be able to help with validating models but there's probably, it would be helpful for us to have some sort of framework and checklist that people agree upon and then probably will get built out over time and put into kind of part of the regulatory process.
01:46:21
I think another just point piggybacking on Kat's comment there is that I think we with some of these kind of a noisier population health or population level signals, we have to just keep in mind that we don't wanna over index on some of that stuff. I think sometimes the expectation is that this thing has to be perfect. That search query or the trends of the search query
01:46:41
has to spot on detect when this peak is gonna happen. Really it's a combination of that plus the other sources of information that's critical. And what's happened right now is we don't have those other sources. So if we had some type of sensor data on the ground or if we had some patient reported outcome that could be correlated. I think the community is building out
01:47:01
bits and pieces of that. But when the data streams get put together, I think that's when we'll start to get to better predictions. And so I think flu trends kind of suffered from that from a while ago, which was, it was a good starting point but that was the only data that was available that we're looking at to see what you can glean from it.
01:47:20
But now if you fast forward in terms of to the ability to capture other signals, I think that could help hone in on the accuracy there a little bit more. But yeah, I think it's really the convergence of multiple streams to be able to at least derive some insights from there. So the trends is exactly what it was designed
01:47:41
to be a trends, right? So it's a rough gauge in terms of what are happening and potentially even get a gauge sooner rather than later too. Even though it might be a bit noisy, at least you have some inclination where hotspots might be occurring or things are happening. So it's more of a gauge rather than trying to be a perfect predictor.
01:48:02
Thanks a lot, but that's really interesting to, yeah, to remember these, what Catherine said with proxies. So choose the right data or know which answers you can get from your data. Yeah, and don't think about other answers or, yeah.
01:48:21
This is, by the way, something we had in the first panel as well. One of the parts of the checklist Stephanie gave us about or we did it as the group, we worked on a kind of a checklist, what you have to, how to prepare yourself for innovation in tech when it comes to AI. And that's of course,
01:48:41
find out if your data really can answer your question and of course find out which data you need. So since time is running, there's another question again from VR. It's from Liz De Salinas Pinacho. In Germany, an app was released to inform people if they were in contact with someone that was tested positive by their interaction.
01:49:00
It's about the contact tracing app, I guess COVID-19. One big question was where and how to store such information. Do you have any idea how that can be improved? The information uses only information from the device and not the person. Are there other examples that you know? So I can add that there was, this discussion was about centralized
01:49:21
and decentralized version. And now in Germany, we have the decentralized version, which is from my point of view, quite privacy friendly. Yeah, there's a question. Yeah, that's a great question. I mean, this is the stuff that every country is struggling with right now. You've got the decentralized model
01:49:40
where it's privacy preserving. The advantage of the centralized model is you might be able to think about commuting across, you know, states or provinces, right? Where if you think about some of the spread, if you think about spread across regions, then having something that might be decentralized to a community or maybe something that's very specific to a region,
01:50:02
you lose that ability to be able to manage an outbreak that might be broader than that. The other thing about the decentralized model is that it's privacy preserving the advantage. But one of the other challenges that some communities have seen with the decentralized model is how quickly
01:50:22
can a case manager react to that particular outbreak? So in the centralized model, because there is a way to do that in a privacy preserving way, because no personal identifiable information's out there, you have to consent to push the fact that you were exposed. And if you were exposed, you can say,
01:50:41
yes, let everybody else know. And you're also granting access to talk to a case manager. But the question there is like, how quickly can you actually get to the contact tracing part of it itself? Because it's not a matter of days, it's hours that matter, right? And so that's the other thing is that you can have this lag in the system that could also not be helpful as well.
01:51:04
But then there's this other side of it, which is just independent of the contact tracing is the technology efficacy part of it too, because some of the approaches that these tools use is maybe Bluetooth or wireless signals and those kinds of things, which has sensitivity around that as well, right? So you don't want a system
01:51:21
that cries wolf all the time, something that has a lot of false positives because then you don't have trust in it. But then you also need to have something that has enough true positive rates so that you have some hope that you're actually getting the contact tracing to be effective. So that's the other aspect that I think everybody around the world is struggling with that's trying to implement contact tracing
01:51:41
is that is this actually a supplement to manual contact tracing? I think that's something that in Germany that was answered to some extent, in Korea it was answered, but that's something that is very specific to region too. If you go centralized, decentralized, in the US implementing this stuff is very difficult because a lot of this is managed at the state level.
01:52:02
The Department of Health at the state is what drives a lot of the contact tracing and the public health work. Whereas if you do it at a country level, that's a little bit different. So it's a great question. I don't have a perfect response to that, but just trying to unpack the sensitivities around how this operates within a region.
01:52:23
Thanks a lot. That was really interesting. Yeah, I think there was another discussion as you mentioned about the sensitivity and of this app in the end or the positive, if the app tells you, you met someone who was positive.
01:52:42
There is a problem of course with false positives and you can only work with this if you have enough test capacity because otherwise people don't trust. Or yeah, it's okay for people not to trust it because they cannot verify in the end because of the privacy preserving technology, if they really met this person
01:53:01
or maybe this person was just a neighbor and there was a wall between and things like that. So that's really, there are trade-offs and that's really interesting. Thanks a lot for talking about this. So since it's three minutes before 7 p.m. Berlin time, the last question is from Bula Wai and I would like to ask you if you want to unmute yourself Bula Wai
01:53:21
and ask the question yourself. That's not the case. Then I again read it. Quite a number of diseases have similar symptoms. For example, COVID-19 is seen to present symptoms similar to TB, malaria, HIV, AIDS, et cetera.
01:53:42
Thus, there could be some levels of misdiagnosis. Example, someone could be suspected for COVID-19 due to slight high body temperature. And it could be that the high body temperature is due to exposure to sunny weather condition at that time. What technology or computational technique will be useful for diagnosis in cases of mixed symptoms,
01:54:01
which some referred to as confusible symptoms? Yes, you can. It's a really good question. And at the end of the day, there's a need for a ground truth data. So rapid diagnostic tests are kind of critical
01:54:22
to actually be able to associate the right kind of sequences, symptoms for the right demographics. So there's, especially when it's just, important, it's a complex condition or there's a lot of confusible symptoms going on.
01:54:42
I'm not sure there's an alternative to actually getting the ground truth to be able to start doing the right associations. And so what I hope can happen is more people are enrolled in contributing
01:55:01
and volunteering their data to help with this kind of differential diagnosis that needs to happen. And that can be associated with different types of demographics potentially or different conditions and scenarios. But again, you wouldn't know for sure unless you had the resulting tests.
01:55:21
So that's pretty important, especially now when we have flu and COVID-19, very hard for certain people to tell the difference unless you get the test done. Yeah, I completely agree with Catherine. I think in this case, I mean, in most cases you probably should get tested
01:55:40
for anything that you think you might have to get that as closer to the ground truth as you can. I see that there's kind of a related question to that asking around the AI side and demographics. So for testing or diagnosing, I think we should lean towards ground truth.
01:56:04
Maybe there's a risk level that you associate with someone based on some AI methods, but I wouldn't use the AI methods alone. Let's say you visited two or three places that are reasonably crowded.
01:56:20
So maybe that then you would be encouraged more to sort of get tested or watch out for signs and symptoms versus sitting at home. But yeah, I wouldn't necessarily just rely on it alone. And I know that demographic information, maybe that helps, but also probably genomic information
01:56:41
and many other types of information. So I think the best thing is to get tested and ground truth. Thank you so much. Since now time is over, this was our panel on more concrete innovations, apps, challenges. And I think we learned a lot
01:57:01
and there's so much to be done and to be thought of. So there's much work. And luckily we have many talented people in this Zoom room who will go on with all those important projects. Thank you so much, Shwetak. Thank you, Aisha. Thank you, Catherine. That was really interesting.
01:57:21
Thanks a lot for your time. I know all of you are very busy and I think we learned a lot from you. Thanks a lot. Don't go everybody. Wait for a second. We will be meeting in the VR room now for just a smaller discussions, questions. I think the speakers will be there as well. I will be there as well. We meet in the discussion room
01:57:40
and you'll find it in the VR solution as soon as you enter it. There's a small location icon on the top left corner of your screen. And if you click on that, you can choose a room. And the discussion room is the one where we will be meeting now in a minute. And luckily the VR room is a minute behind. So we have exactly this minute to meet up there again.
01:58:01
So thanks a lot and see you over there. Thanks everyone. Thank you. Bye.