Why Johnny the Developer Can't Work with Public Key Certificates [RSA-CT 2018]

   Authors: Martin Ukrop and Vashek Matyas

 Primary contact: Martin Ukrop <mukrop@mail.muni.cz>

 Conference: RSA Cryptographers' Track 2018

   DOI: 10.1007/978-3-319-76953-0_3

Pre-print PDF   Artifacts   Presentation   BiBTeX 

@InProceedings{2018-rsa-ukrop,
  Author        = {Martin Ukrop and Vashek Matyas},
  Title         = {Why Johnny the Developer Can't Work with Public Key Certificates: An Experimental Study of OpenSSL Usability},
  BookTitle     = {Topics in Cryptology -- CT-RSA 2018: The Cryptographers' Track at the RSA Conference 2018},
  Year          = {2018},
  Publisher     = {Springer International Publishing},
  Pages         = {45--64},
  DOI           = {10.1007/978-3-319-76953-0_3},
}

Abstract

There have been many studies exposing poor usability of security software for the common end user. However, only a few inspect the usability challenges faced by more knowledgeable users. We conducted an experiment to empirically assess usability of the command line interface of OpenSSL, a well known and widely used cryptographic library. Based on the results, we try to propose specific improvements that would encourage more secure behavior. We observed 87 developers/administrators at two certificate-related tasks in a controlled environment. Furthermore, we collected participant opinions on both the tool interface and available documentation. Based on the overall results, we deem the OpenSSL usability insufficient according to both user opinions and standardized measures. Moreover, the perceived usability seems to be correlated with previous experience and used resources. There was a great disproportion between the participant views of a successful task accomplishment and the reality. A general dissatisfaction with both OpenSSL interface and its manual page was shared among the majority of the participants. As hinted by a participant, OpenSSL gradually “turned into a complicated set of sharp kitchen knives” – it can perform various jobs very well, but laymen risk stabbing themselves in the process. This highlights the necessity of a usable design even for tools targeted at experienced users.

Several interesting bits...

The usability of OpenSSL interface turned out to be… not so great (no real surprise there). But there are some interesting bits and pieces. To name just a few:

  • 87% participants thought they successfully issued a self-signed certificate, but only 45% did.
  • 42% created a certificate with “Internet Widgits Pty Ltd.” in an organization field.
  • 26% of created certificates were old version 1.
  • 73% participants used Stack Overflow solutions, 40% used the knowledge base of University of Wisconsin-Madison.
  • 28% people tried to open the manual page with 'man openssl verify' (which is incorrect).
  • Only 9% people adjusted the commands after copy-pasting them from the tutorial.

For explanation and further details on there (and other) numbers, see the paper text.

Talk at DevConf 2018

Research artifacts (supplementary material)

The supplementary material to this paper contains of:

  • Informed consent participants had to sign (experiment design approved by Research Ethics Committee of Masaryk University)
  • General questionnaire & System usability scale questionnaire
  • User tasks & certificates to validate
  • Issues changed based on this research:
    • Adding missing OpenSSL online documentation redirects (github issue)
    • Alternative names for OpenSSL manual pages (github issue)

Artifacts (Zenodo)