We're sorry but this page doesn't work properly without JavaScript enabled. Please enable it to continue.
Feedback

PIMMI: a command line interface to study image propagation

00:00

Formal Metadata

Title
PIMMI: a command line interface to study image propagation
Alternative Title
PIMMI: Python IMage MIning
Title of Series
Number of Parts
542
Author
Contributors
License
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Identifiers
Publisher
Release Date
Language

Content Metadata

Subject Area
Genre
Abstract
PIMMI is a Python software that performs visual mining in a corpus of images. Its main objective is to find all copies, total or partial, in large volumes of images and to group them together. Our initial goal is to study the reuse of images on social networks (typically, our first use is the propagation of memes on Twitter). However, we believe that its use can be much wider and that it can be easily adapted for other studies. The main features of PIMMI are therefore: - ability to process large image corpora, up to several millions files - robustness to some modifications of the images (crop, zoom, composition, addition of text, ...) - adaptability to different use cases (mainly the nature and volume of the image corpus) A study using PIMMI will generally be broken down into several steps: 1. constitution of a corpus of images (jpg and/or png files) and their metadata 2. choice of PIMMI parameters according to the criteria of the corpus 3. indexing the images with PIMMI and obtaining clusters of reused images 4. exploitation of the clusters by combining them with the descriptive metadata of the images The development of this software is still in progress and we warmly welcome beta-testers, feedback, proposals for new features and even pull requests!
Computer-generated imageryObservational studyInterface (computing)Wireless Markup LanguageObservational studyPropagatorHypermediaTracing (software)Lecture/Conference
Computer-generated imageryObservational studyInterface (computing)Arithmetic meanObservational studyMereologyComputer animation
Identical particlesComputer-generated imageryCluster samplingTransformation (genetics)Characteristic polynomialComputer filePattern recognitionPersonal digital assistantObservational studyComputer networkHypermediaAxiom of choicePairwise comparisonGene clusterDigital photographyTouchscreenPropagatorDifferent (Kate Ryan album)Axiom of choicePattern recognitionDemo (music)Type theoryHoaxNumberComputer-assisted translationBusiness clusterCoefficient of determinationComputer animation
Lecture/Conference
Business clusterComputer-generated imageryData miningExecution unitGamma functionGene clusterSet (mathematics)Filter <Stochastik>Computer animation
Bookmark (World Wide Web)Element (mathematics)Instant MessagingBusiness clusterComputer-generated imageryData miningBusiness clusterResultantGroup actionSource codeQuery languageTwitterComputer animation
UsabilityBusiness clusterDenial-of-service attackProgrammable read-only memoryComputer-generated imageryGamma functionBookmark (World Wide Web)Game theoryBusiness clusterState of matterSlide ruleComputer animationPanel painting
Business cluster
Observational studyComputer-generated imageryInterface (computing)Point (geometry)Invariant (mathematics)Scale (map)Singuläres IntegralLocal ringAlgorithmVector graphicsData structurePrice indexDifferent (Kate Ryan album)DatabaseSimilarity (geometry)Transformation (genetics)Digital filterCluster samplingGraph (mathematics)Gastropod shellFile formatComputerNeuroinformatikVector spacePoint (geometry)DatabaseSimilarity (geometry)Local ringKey (cryptography)Machine visionMereologyComputer animation
Denial-of-service attackBusiness clusterStructural loadRevision controlInformationDemo (music)Cluster samplingQuery languageNumberVertex (graph theory)Component-based software engineeringGraph (mathematics)Maxima and minimaComputer animation
Business cluster
NumberComputer-generated imageryQuery languageInverse problemPrice indexMereologyContext awarenessGraph (mathematics)Similarity (geometry)Electronic visual displayPersonal digital assistantDemo (music)Source codeAstrophysicsLibrary (computing)Vector graphicsInvariant (mathematics)Local ringObject (grammar)Pattern recognitionScale (map)Machine visionSimilarity (geometry)PhysicalismPlotterCASE <Informatik>MetadataProjective planePlanningLink (knot theory)Dependent and independent variablesGraph (mathematics)MereologyTwitterAlgorithmDatabaseActive contour modelType theoryComputer architectureEndliche ModelltheorieOpen setGene clusterStreaming mediaHypermediaSoftware developerComputer animation
Machine visionSource codeInvariant (mathematics)Scale (map)Pattern recognitionLocal ringLibrary (computing)Vector graphicsSimilarity (geometry)Machine visionComputer-generated imageryVector spaceSimilarity (geometry)Visualization (computer graphics)Business clusterDatabaseLink (knot theory)Type theoryResultantHypermediaConnected spaceMereologyLatent heatOrder (biology)Context awarenessSlide ruleGene clusterTwitterServer (computing)NumberGraph (mathematics)Semantics (computer science)Pulse (signal processing)Local ringLecture/Conference
Point cloudProgram flowchart
Transcript: English(auto-generated)
Hi everyone. Well, I'm very impressed to have such a large audience for such a small tool. But I'm Beatrice, I work at the French Media Lab. And today I'm going to present
PIMI, which is a tool to study image propagation. The Media Lab is a lab where social scientists try to, among other things, study the traces that people leave online. And for now, they are quite well equipped with tools to study text. But when they ask
me, OK, how can I study mean propagation? I'm still struggling to give them answers. So what does it mean to study mean propagation? It means being able to recognize that some
parts of an image are copied or partially copied. So what this tool does, it's very clusters of images and grouped together images that are total or partial copies of each
other. It's able to deal with image transformation, so if the image is cropped or zoomed, and it's able to adapt to copper's characteristics. So it will try to make the best of your
data sets, depending on the number of images you have or the type of images you have. What PIMI is not able to do is to cluster semantically similar images. So it's not the tool that you are going to use if you want to create clusters of cats and clusters
of dogs or, I don't know, find images of violence versus images of peace. And it's not able to do some face recognition. So, again, you will not be able to make some
clusters of pictures of Elizabeth II versus clusters of images of Emmanuel Macron. What you could imagine doing, and we could imagine also work together if you are a researcher
working on those subjects, is to study the propagation of MIMO on social networks, as I was saying. But also you could study the usage of the press agency photos in a press corpus or stock photos as well. You could also study the dissemination of fake news
based on image montage. Or you could study the editorial choices between different media, depending on whether they use the same images or not. So let me do a quick demo of how it looks for now. It's not on the screen.
OK. Let's forget about that. I'm very sorry. Well, I'll try to make it work.
OK. Well, still not showing totally all clusters. But so we create clusters of images. So this is a data set that is created by the French INRIA and that is presenting some
degradation on images. So they take an original picture and they apply some filters or they crop the images to see if they are able to put the images together. So we can see that we have pretty correct results on that data set.
And this is our results on some images that we collected ourselves on Twitter using Elon Musk as a query. And so we tried to cluster those images. So as you can see, we have images of Elon Musk. We are able to group together some images that
are crops of others. So this is probably the source image of the montage that has been done here. But we can also see that we have some problems with the tool.
For example, here we have a cluster with two images that have been assembled together and we create a cluster of actually two images. But well, that's the state of the two for now. And now I try to come back to my slides.
OK, so how does it work for people who work in computer vision? I'm going probably to say some things that are quite basic, but I try to make it clear for people who do not do computer vision.
So it is not based on colors at all. It's used like the grayscale of images and it tries to detect points of interest on a picture and then it uses these local key points as vectors.
And then those vectors are indexed in a database that is able to perform some very quick similarity search. Of the tool, as I say, there is that problem of parts of images
that create clusters that are bigger than they should be. So our plan is to be able to detect
images that are actually those links between two clusters. So to be able to detect that this image is actually containing two images and to be able to deal with part of images. And also what we would like to do is to show images in their context,
to be able to show the tweets that contain those images or Instagram posts, etc. Or at least to show additional metadata for the users. And also, we would like to show you the graph of image similarities so that the clusters that
are resulting from that graph are not interpretable. And to improve our tool, we need your use cases because for now we have those two, three databases
but we would be very glad to do some partnerships with other researchers to improve the tool. Thank you very much for your attention.
If you want to look at the slides, we have the references to all the images used and to the papers of the algorithms used by Pini. I'm open for questions.
With the sound stream, but it's back online. So yeah, you should repeat the question for the stream. OK, I'll do it. Yes?
Adam, you said it's actually immediately in mind because research from the physics side, solar cell research, there are some 3D plots where you want to compare similarities, but they're not images as you have right now. It's basically data. Is that the use case that is possible? Or is it 3D plots, contour plots, stuff like that?
Well, trying to find similarities in... Oh, sorry, I have to repeat the question. So the question was, if I understand well, how to reproduce that use case, not on images, but on other types of documents that would be, I guess, some features.
3D contour plots. And I'd say, well, as long as you can represent your data in the shape of vectors,
then you're ready to use face to do some search for nearest neighbor in your database. And then you can go for the whole pipeline, create some graphs, find communities in the graph and go for it. But I'm not sure Pini is your tool, but well,
the architecture of Pini could be, of course, a model. Is there any project current or completely wrong going that Media Labs has used Pini for, or is it still largely in development?
It is largely in the development. Sorry, I repeat the question. So are there some projects at the Media Lab that are currently using Pini? And the response is no.
Have you considered any other ways of presenting picture similarity or
using picture similarity? All the types of image similarity, if I understand well. Well, I'd say that that was what I was saying in my second slide.
There are other types of image similarity, for example, semantical similarity. And well, maybe in a few months, if we have like a robust architecture, we could
maybe include some other types of vectorization of images. But for now, well, there are already tools that do that. Like there is something called Clip Server that helps you find similar images
from clip vectors that are like semantical vectors. So you could use that tool, it's great.
Yes? Yes.
So the question is, is the tool really able to distinguish the thing that is of interest to us, the fact that we are talking about a dog?
So the tool is only able to find partial copies in an image. So the tool would probably be able to say that all those images contain the same part of the face of a dog. So it would probably be able to group all those images together.
The problem is that if there are other images in the database that contain the rest of the images, then they would probably also be grouped in the same cluster. So that's why what we are currently doing about parts of images
would let us improve the cluster so that it's purified from the rest of the images. And we could have a cluster of the face of that specific dog, and then a cluster of that taco in the second cluster.
Yes? What kind of clusterization do you use on the graph? Well, for now we have the best result with, excuse me, what kind of clusterization do you use on the graph?
For now we have our best result using pure connected components. So actually the specification we do on the graph to reduce the number of links between images is enough to have separated connected components in the graph.
And so we take each connected component and it's our cluster. What we would like to do is to try to mix with some move on community detection, but actually for now it's not the thing that works best.
Yes? I'm not sure I understand the question.
Can you try to rephrase it? Okay. What things are you looking at to improve the model?
Well, there are many things where we are looking at. For now, mainly we look at techniques to do a better graph specification in order to find more coherent clusters.
We are not so much working on the local descriptors parts of the tool for now. Yes?
Have you considered, used the direct link to the Twitter images or social media images online?
Did I repeat everything? Yes. Well, yes, we would like people to be able to see images in their context because actually
they won't understand what's happening if they just have images. They need to see, okay, why was this image published, who answered, etc. So this would probably mean that we need to add at least the link to the pulse
or maybe some kind of visualization of it.