The Potential of Using Vision Videos for CrowdRE: Video Comments as a Source of Feedback
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 2 | |
| Author | 0000-0001-5336-6899 (ORCID) | |
| Contributors | 0000-0001-5336-6899 (ORCID) 0000-0001-7674-2930 (ORCID) | |
| License | CC Attribution 3.0 Germany: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/54811 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Producer | 0000-0001-5336-6899 (ORCID) | |
| Production Year | 2021 | |
| Production Place | Celle |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
00:00
InformationRequirements engineeringMachine visionStreaming mediaFeedbackStreaming mediaVector potentialMachine visionXMLUML
00:13
Machine visionForceStreaming mediaGreen's functionRequirements engineeringSoftwareComputing platformSocial softwareElectric currentVector potentialView (database)Internet forumMobile WebFeedbackMaxima and minimaGraphics tabletMachine visionPersonal digital assistantTelecommunicationShared memoryStatement (computer science)MereologyGroup actionFocus (optics)Rule of inferenceMathematical analysisContent (media)MultiplicationComputing platformModal logicReplication (computing)Computing platformView (database)Machine visionStreaming mediaHypermediaEndliche ModelltheorieMathematical analysisOrder (biology)Form (programming)Different (Kate Ryan album)Perspective (visual)Focus (optics)Group actionPhysical systemCategory of beingVisual systemMereologyVector potentialYouTubeComputing platformWind tunnelReplication (computing)Mobile appMultiplication signScripting languageInternet forumContent (media)Computer animation
02:56
ApproximationNumberChemical polarityContent (media)Electronic program guideAbelian categorySign (mathematics)FeedbackNumerical taxonomySupport vector machineDigital object identifierAlgorithmSet (mathematics)ForestRandom numberBinary fileHeegaard splittingStreaming mediaNumerical taxonomyLevel (video gaming)Category of beingTask (computing)WordPoint (geometry)Virtual machineCross-validation (statistics)Lattice (order)Reduction of orderDifferent (Kate Ryan album)View (database)RandomizationForestAdditionPerspective (visual)Traffic reportingMultiplication signAlgorithmResultantHeegaard splittingPolarization (waves)1 (number)NumberMathematical analysisSelectivity (electronic)Support vector machineGoodness of fitContent (media)Observational studyComputer animationProgram flowchart
05:36
Group actionCategory of beingFeedbackNumerical taxonomyPhysical systemMachine visionStreaming mediaPhysical systemNumerical taxonomyStreaming mediaFocus (optics)Element (mathematics)Category of beingResultantMachine visionAuthorizationDifferent (Kate Ryan album)Mobile appInternet forumGroup actionHypermediaMachine visionState observerComputer animation
06:58
Group actionFeedbackNumerical taxonomyPhysical systemMachine visionStreaming mediaComputing platformMachine visionSocial softwareContent (media)Computer configurationAverageBuildingStrategy gameSoftwareEntire functionProcess (computing)Covering spaceOrder (biology)Streaming mediaSoftware developerContent (media)Computer configurationComputing platformEntire functionMachine visionBuildingVector potentialNumberWind tunnelProcess (computing)HypermediaStrategy gameResultantComputer animation
08:06
Address spaceStreaming mediaStatement (computer science)Bit error rateFormal languageSoftware frameworkInternet forumSet (mathematics)Observational studyExtension (kinesiology)Point (geometry)Machine visionStatement (computer science)Streaming mediaMathematical analysisSoftware developerTwitterAdditionExtension (kinesiology)ResultantView (database)HypermediaReplication (computing)Computing platformObservational studySoftware frameworkMobile appAlgorithmMachine learningContent (media)Order (biology)Computer animation
Transcript: English(auto-generated)
00:00
Hello and welcome to my talk with the title The potential of using vision videos for crowdreary video comments as a source of feedback. My name is Oliver Kavas and I'm working at Leibniz Information Center for Science and Technology. Let's start with considering crowdreary and its use of feedback sources. While according to literature there are some frequently used sources like user forums
00:21
or mobile application markets, others and in particular social media platforms are rarely used. Nevertheless, different researchers agreed on the potential of social media platforms since they can provide new motivational opportunities for more than crowd members, for example due to entertaining and enjoyable activities such as watching a video.
00:42
In addition, they provide the opportunity to achieve a wide reach so you can gather millions of views and solicit thousands of comments. We want to further conclude with the idea of using vision videos on social media platforms. For this purpose we need to understand what is a vision video. A vision video is a video that represents a vision or parts of it for achieving shared
01:04
understanding among all parties involved by disclosing, discussing and aligning their mental models of the future system. So far, vision videos are frequently used in individual meetings, focus groups or workshops in order to stimulate discussions and solicit feedback.
01:20
Researchers propose the idea to transfer these benefits of vision videos for crowdreary by using them on social media platforms. In this talk we want to consider the research question what is the potential of using vision videos on social media platforms to solicit feedback in the form of video comments for crowdreary.
01:40
We investigated these research questions from three perspectives regarding the quantity of videos, views and comments, the content of the comments and their suitability for analysis used in crowdreary. In particular, we followed a clear research approach consisting of the four main steps,
02:00
data collection, data cleaning and manual analysis for collected data and first automated analysis. In the manual analysis we spent a lot of effort in order to identify the different classification categories and create at the end a labeled dataset which can be used for the automated analysis. In particular, we selected YouTube as a platform and the visual video tunnels by the Boring company.
02:25
At the time we downloaded the dataset the video had 4500 comments and 2100 replies. The comments are directly related to the video while the replies address some kind of discussions below the respective comment. For this reason we focus only in this paper on the analysis of the comment.
02:43
After the data cleaning the dataset contained 4400 comments for the subsequent analysis. If you are interested in our datasets, scripts and the coding guides, the replication packet is provided on the node. Now let's consider some of the findings of the paper. For the perspective of quantity, how that after 4 days there were already 2770 comments.
03:07
This is interesting that these are 53% of all comments that were collected in 4 years. So we can say that a large number of comments was received in a really short amount of time.
03:21
The result of the content perspective are based on a comprehensive manual analysis of the collected video comments. For this purpose we had two coders who performed for us two classification tasks of the video comments regarding their relevance and polarity as well followed up by the intentional topic. After each classification task we had a discussion meeting to resolve any disagreements between the two coders.
03:44
We found that 764 comments are only relevant from the point of view of crowd RE. The further investigation of the intentional topics showed that the main intentions are feature requests and problem reports which are addressed in the video comments. In addition we found that the topics efficiency and safety are the most frequent topics addressed on the video comments.
04:06
For the classification regarding the intentional topic we used the taxonomy of user classification provided by Santos SI. What we found is that a lot of the content in the comments is related to the taxonomy of user feedback classification categories.
04:21
Regarding the last perspective of suitability we investigated how well the video comments can be classified by machine learning algorithms that are frequently applied in crowd RE. Our selection is based on the overview of user feedback classification approaches provided by Santos SI. We mainly used three algorithms and two features.
04:42
These algorithms are support vector machines and they are the most frequently used ones in crowd RE. In addition we focused on the use of random forest which is rarely used but showed already good performance in different studies. For the features we focused on back-off words and TF-IDF.
05:02
In particular we created for each category a balanced stage set and performed a 10-fold cross-validation for a binary classification task. To reduce data splitting bias we repeated each 10-fold cross-validation 10 times for each category. Our results show that SVM combined with TF-IDF achieved the best results overall.
05:23
In particular we found that the relevant comments and the topic of safety can be classified very well. However the intention of problem report is the most difficult one to classify. Now we would like to discuss some of our findings. First of all considering the feedback on safety.
05:42
If we consider the taxonomy of user feedback classification we see that it was developed with a systematic literature review on the topic of classifying user feedback. In particular the authors investigated 43 different papers and identified 78 different categories for user feedback. However safety is not an element of this taxonomy.
06:03
We assume that this lack of safety in the taxonomy is a result of the currently and frequently used user feedback sources, mainly user forums and mobile application markets. Vision videos are in particular concrete by showing the system and its use. In this way they can empower stakeholders to experience and report their suspected quality news problems
06:25
such as economic health or safety risk, even if they have never actually used the system. This benefit is already known from applying videos in focus groups or workshops for example. This observation and explanation let us conclude that we can see directly that one of the
06:44
most important benefits of vision videos can be transferred on social media platforms, resulting in a currently completely new and even missing aspect of category in the taxonomy of user feedback classification.
07:01
As a second aspect we would like to discuss a selected vision video. We selected the vision video tunnels by The Boring Company. This company was founded by Elon Musk and in this way it already has a strong social media presence, resulting in thousands of comments. However this does not hold for average company. Therefore we need a clear strategy how we can build social media presence for companies in order to solicit thousands of video comments.
07:28
Nevertheless we need even more besides the social media presence we also need to bridge the gap to the development. In this way there is a need for holistic approaches for CrowdRE, covering the entire software development process in order to operationally involve a crowd.
07:43
As an answer to our research question we can summarize that there is a potential of using vision videos on social media platforms. Regarding the number of comments in their content we see promising results for CrowdRE. In this way vision videos can be a suitable option to motivate and crowd numbers to actively participate by writing comments which are a valuable source of feedback.
08:07
Finally I would like to conclude the presentation. What we've seen, we've considered the elicitation and the analysis of views of feedback in CrowdRE. We've seen that frequently used sources are mobile application market and user forms.
08:21
However there are other sources and in particular social media platforms which are rarely used. We've considered the idea of using videos on social media platforms to solicit thousands of comments. We investigated the content of these comments and found interesting results which are promising regarding the use of vision videos for CrowdRE.
08:44
However there is a lot of future work to do. First of all there is a new trend in text classification using deep learning algorithms. We would suggest to apply these algorithms also on our dataset to assess their suitability for CrowdRE.
09:00
In addition we need other replication and extensions of our study including the development of further datasets. Due to the lack of suitable datasets we spent a lot of effort in creating a reliable dataset which hopefully can be further used by other researchers. At the last point we want to emphasize the need to investigate also the replies beside my comments on the video.
09:24
Based on our manual review we saw that these replies do not address the video itself but show some kind of discussion below the associated comment. We would suggest to use for example frameworks like CrowdARG proposed by Khan et al. in order to identify arguments for or against a given statement.
09:41
In this way we could obtain further insight into the source of the crowd. In this way we want now to close our talk and thank you for your interest. Have a nice day.