Will You Trust This TLS Certificate? Perceptions of People Working in IT [ACSAC 2019]

Authors: Martin Ukrop, Lydia Kraus, Vashek Matyas and Heider Wahsheh

Primary contact: Martin Ukrop <mukrop@mail.muni.cz>

Conference: ACSAC 2019

Pre-print PDF Artifacts Presentation BiBTeX

@InProceedings{2019-acsac-ukrop,
  Title         = {Will You Trust This TLS Certificate? Perceptions of People Working in IT},
  Author        = {Martin Ukrop and Lydia Kraus and Vashek Matyas and Heider Ahmad Mutleq Wahsheh},
  BookTitle     = {35rd Annual Computer Security Applications Conference (ACSAC'2019)},
  Year          = {2019},
  Publisher     = {ACM},
  Keywords      = {usablesec, Red-Hat},
  DOI           = {10.1145/3359789.3359800},
}

Abstract

Flawed TLS certificates are not uncommon on the Internet. While they signal a potential issue, in most cases they have benign causes (e.g., misconfiguration or even deliberate deployment). This adds fuzziness to the decision on whether to trust a connection or not. Little is known about perceptions of flawed certificates by IT professionals, even though their decisions impact high numbers of end users. Moreover, it is unclear how much does the content of error messages and documentation influence these perceptions.

To shed light on these issues, we observed 75 attendees of an industrial IT conference investigating, different certificate validation errors. Furthermore, we focused on the influence of re-worded error messages and redesigned documentation. We find that people working in IT have very nuanced opinions regarding the tested certificate flaws with trust decisions being far from binary. The self-signed and the name constrained certificates seem to be over-trusted (the latter also being poorly understood). We show that even small changes in existing error messages and documentation can positively influence resource use, comprehension, and trust assessment. Our conclusions can be directly used in practice by adopting the re-worded error messages and documentation.

Selected conclusions

We investigated perceived trust in five certificate cases: hostname mismatch, self-signed, expired, name constrained and a flawless certificate (as a control case).
When validating certificates, the trust decisions are not binary. Even IT professionals do not completely refuse a certificate just because its validation check fails.
- In case of expired certificates, the expiry duration plays an important role: Certificates expired yesterday were mostly considered as “looking OK”, while a certificate expired 2 weeks ago “looks suspicious” and the one expired a year ago seems “outright untrustworthy”.
- The certificate subject plays a role: Flaws were less likely to be tolerated for big, established companies (Microsoft was mentioned as an example).
We found some certificate cases as over-trusted.
- 21% of the participants considered the self-signed certificate as “looking OK” or better, with a trust mean comparable to that of an expired certificate. We find this concerning as the self-signed certificate never had any identity assurances.
- Similarly, 20% of the participants considered the name constrained certificate as “looking OK” or better, with a trust mean again comparable to that of an expired certificate. We find this concerning as the name constraints violation hints at misconfiguration or even malicious activity at the sub-authority level.
We had half of the participants interact with real OpenSSL error messages and the other half with our re-designed error messages and documentation. Here is the comparison:
- The self-signed case was considered significantly less trustworthy with our error message (which we consider a success).
- The name constrained case was also perceived as less trusted and required less time and less online browsing to understand.
- The other attributes were comparable – thus, we see our documentation in these cases as better than the existing one.
In the redesigned error messages, we included a link to the documentation. To our surprise, 71% of the participants clicked this link. This suggests a nice opportunity of directing the developers to a usable place recommended by the library designers.
As a follow-up work, we started gathering X.509 certificate validation errors and documentation from multiple libraries to consolidate the documentation on a single place.

Visit x509errors.org

Talk at DevConf 2019

The content of this research was partially covered at the DevConf 2019 talk that can be seen below. Presentation

Research artifacts (supplementary material)

The artifacts contain the full experimental setup (as described in Section 2.1 of the paper) and the complete anonymized dataset underlying the evaluation presented in Sections 3 and 4.

The experimental setup contains the documents accompanying the task: the informed consent, pre-task questionnaire, task description, trust scales, and the list of questions posed during the post-task interview (all in PDFs). We further include the custom website with certificate validation documentation for the “redesigned” condition (static HTML). While working on the task, participants in the “redesigned” condition could access this website via a link that was in the redesigned error messages. Furthermore, we provide the software with which the participants interacted. It contains the displayed error messages and validated certificates. These things are available both individually and incorporated in a snapshot of a virtual machine used at the experiment (importable directly into VirtualBox).

The collected data is presented in a single dataset (SPSS format; you can use PSPP as a free alternative). It includes the analysis syntax files to obtain the numerical results presented in the paper. For each participant, the dataset contains: 1) pre-task questionnaire answers, 2) reported trust ratings, 3) sub-task timing, 4) information on whether they browsed the Internet and 5) the interview codes assigned. Note that we do not publish the interview transcripts to preserve participant privacy.

Go to artifacts repository (Zenodo)