Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
public:papers:cacm2018 [2017-12-24 11:39] – [A large-scale comparative study of beta testers and standard users [CACM 2018]] xukroppublic:papers:cacm2018 [2020-05-14 11:38] (current) xukrop
Line 2: Line 2:
 ~~NOTOC~~ ~~NOTOC~~
  
 +<grid>
 +<col xs="12" sm="8" lg="8">
 <TEXT size="large"> <TEXT size="large">
-\_{{fa>user}}\_\_//Authors:// Vlasta Stavova, Lenka Dedkova, Martin Ukrop and Vashek Matyas+ 
 +\_{{fa>user}}\_\_//Authors:// [[:publications:authors:vlasta-stavova|Vlasta Stavova]], Lenka Dedkova, [[:publications:authors:martin-ukrop|Martin Ukrop]] and [[:publications:authors:vashek-matyas|Vashek Matyas]]
  
 {{fa>user-circle-o}}\_//Primary contact:// Vlasta Stavova %%<%%<vlasta.stavova@mail.muni.cz>%%>%% {{fa>user-circle-o}}\_//Primary contact:// Vlasta Stavova %%<%%<vlasta.stavova@mail.muni.cz>%%>%%
  
 {{fa>bullhorn}}\_//Journal:// [[https://cacm.acm.org/|Communications of the ACM]] {{fa>bullhorn}}\_//Journal:// [[https://cacm.acm.org/|Communications of the ACM]]
 +
 +\_{{fa>id-badge}}\_\_//DOI:// [[https://doi.org/10.1145/3173570|10.1145/3173570]]
 </TEXT> </TEXT>
 +</col>
  
-<panel type="defaulttitle="Abstract"> +<col xs="12sm="4" lg="4"> 
-There have been many studies exposing poor usability of security software for the common end user. However, only a few inspect the usability challenges faced by more knowledgeable users. +<TEXT align="right"> 
-We conducted an experiment to empirically assess usability of the command line interface of OpenSSL, a well known and widely used cryptographic library. Based on the results, we try to propose specific improvements that would encourage more secure behavior. + 
-We observed 87 developers/administrators at two certificate-related tasks in a controlled environmentFurthermore, we collected participant opinions on both the tool interface and available documentation. +<button type="warning" icon="fa fa-file-pdf-o">[[https://crocs.fi.muni.cz/_media/publications/pdf/2018-cacm-stavova.pdf|Pre-print PDF]]</button> 
-Based on the overall results, we deem the OpenSSL usability insufficient according to both user opinions and standardized measuresMoreover, the perceived usability seems to be correlated with previous experience and used resources. There was a great disproportion between the participant views of a successful task accomplishment and the reality. A general dissatisfaction with both OpenSSL interface and its manual page was shared among the majority of the participants+ 
-As hinted by a participant, OpenSSL gradually "turned into a complicated set of sharp kitchen knives" -- it can perform various jobs very well, but laymen risk stabbing themselves in the processThis highlights the necessity of a usable design even for tools targeted at experienced users. +<button type="info" icon="fa fa-external-link">[[https://dl.acm.org/citation.cfm?id=3173570|Final version (ACM DL)]]</button>
-</panel>+
  
-<popover trigger="focus" title="Not yet available" content="Paper PDF will be added in February 2018."> 
-<button type="warning" icon="fa fa-file-pdf-o">Pre-print PDF</button> 
-</popover> 
-\_ 
 <button collapse="bibtex" icon="fa fa-file-code-o">BiBTeX</button> <button collapse="bibtex" icon="fa fa-file-code-o">BiBTeX</button>
- +</TEXT> 
 +</col> 
 +</grid> 
 <collapse id="bibtex" collapsed="true"> <collapse id="bibtex" collapsed="true">
-  @Article{2017-cacm-stavova, +  @Article{2018-cacm-stavova, 
-    title = {A large-scale comparative study of beta testers and standard users}, +    Title = {A Large-scale Comparative Study of Beta Testers and Regular Users}, 
-    author = {Vlasta Stavova and Lenka Dedkova and Martin Ukrop and Vashek Matyas}, +    Author = {Vlasta Stavova and Lenka Dedkova and Martin Ukrop and Vashek Matyas}, 
-    journal = {To appear in Communications of ACM}, +    Journal = {Communications of the ACM}, 
-    year = {2018}, +    Year = {2018}, 
-    keywords = {usablesecESET}, +    Volume = {61}, 
-    organization = {ACM},+    Number = {2}, 
 +    Pages = {64--71}, 
 +    Publisher = {ACM}, 
 +    Doi = {10.1145/3173570},
   }   }
 </collapse> </collapse>
  
----- +<panel type="default" title="Abstract">
- +
-===== Key insights ===== +
- +
-Our research produced the following actionable takeaways for software developers: +
-  * **Using data**\\ Data you can collect can help you learn who your users and beta testers are. Consider country of origin, software and hardware configuration, and basic demographics. +
-  * **Selecting testers**\\ The fewer testers you have, the pickier you should be about their selection. +
-  * **Identifying usability issues**\\ When testing international products, ensure beta testers are culturally representative of regular users to help identify potential localization and cultural usability issues. +
-  * **Ensuring representation**\\ Most important, testers should be representative of regular users. Keep checking that this is the case or pursue additional rigorous analyses to reach the most credible and applicable conclusions possible. +
-===== Summary ===== +
 Beta testers are the first end users outside a software company to use its product. They have been used for decades and are rightly credited not only with finding and reporting bugs, but also with improving general product usability through their feedback and/or the ways they use the product. Beta testers are the first end users outside a software company to use its product. They have been used for decades and are rightly credited not only with finding and reporting bugs, but also with improving general product usability through their feedback and/or the ways they use the product.
  
Line 53: Line 50:
  
 We conclude that - at least in our study - beta users represent standard users well in terms of hardware and operating system in large scale beta testing. However, populations differ significantly in the distribution of users and testers between countries. This may cause a problem when a testing includes localization and usability issues that may be influenced by regional differences. We conclude that - at least in our study - beta users represent standard users well in terms of hardware and operating system in large scale beta testing. However, populations differ significantly in the distribution of users and testers between countries. This may cause a problem when a testing includes localization and usability issues that may be influenced by regional differences.
 +</panel>
  
 ===== Summarizing video ===== ===== Summarizing video =====
  
 {{ youtube>H7ik9SeAexQ?900x520 |A large-scale comparative study of beta testers and standard users}} {{ youtube>H7ik9SeAexQ?900x520 |A large-scale comparative study of beta testers and standard users}}
 +
 +===== Key insights =====
 +
 +Our research produced the following actionable takeaways for software developers:
 +  * **Using data**\\ Data you can collect can help you learn who your users and beta testers are. Consider country of origin, software and hardware configuration, and basic demographics.
 +  * **Selecting testers**\\ The fewer testers you have, the pickier you should be about their selection.
 +  * **Identifying usability issues**\\ When testing international products, ensure beta testers are culturally representative of regular users to help identify potential localization and cultural usability issues.
 +  * **Ensuring representation**\\ Most important, testers should be representative of regular users. Keep checking that this is the case or pursue additional rigorous analyses to reach the most credible and applicable conclusions possible.
  
 ===== How did we make the research video ===== ===== How did we make the research video =====