Involvement of Domain Experts in the AI Training Does not Affect Adherence: An AutoML Study

Abstract

AutoML is a promising field of Machine Learning (ML) that is supposed to bring the advantages of artificial intelligence to a wide range of organizations in plentiful domains, by automating the process of ML-model creation without requiring prior knowledge in data science or programming. However, AutoML often appears to users as a black-box model created in a black-box process, negatively impacting users’ trust. Additionally, AutoML users are often experts in their respective domains (physicians, engineers, etc.), which are commonly observed to exhibit stronger algorithm aversion than lay people, i.e., having more difficulties trusting and relying on AI recommendations. User non-adherence to AutoML may have high-cost consequences, resulting in inefficient decisions and mitigating the overall progress in the AutoML field. Therefore, we investigate how domain experts’ adherence to AutoML recommendations can be fostered. As involvement of users in product creation processes was shown to positively affect their attitudes towards the product in multiple contexts, we argue that involving domain experts in AutoML-model creation processes may increase their trust and adherence to AutoML. We conduct an experimental laboratory study, in which subjects act as expert engineers and need to foresee machine malfunctions, while being advised by an AutoML-model. We apply three treatments – zero, passive & active involvement – to investigate our hypothesis. We observe that higher involvement leads to a higher perceived influence on the AutoML model and a higher perceived understanding of its functionality. However, these perceptions are not reflected in the actual behavior – subjects across all groups demonstrate similar AI adherence.

Appendices

Appendix A: Additional Tables and Graphics

Table 8. Distribution of study majors among participants

Fig. 6.

Illustration of decision process for each round of maintenance stage

Table 9. Summary statistics and between-group comparison of questionnaire items on treatment perception

This table reports summary statistics of questionnaire items on treatment perception measured on a 5-point scale by treatment. Standard deviations reported in parenthesis. Kruskal-Wallis-H reports the p-values for Kruskal-Wallis H-Tests (KWH) with ties between experimental groups. Includes only subjects with high-accuracy AI.

Please Note.

The following Appendices, B and C, contain the experimental instructions and experimental questionnaire exactly in the form in which they were presented to the participants. Therefore, we abstain from numbering the contained figures and tables as well as referring them in the paper individually. Instead, Appendices B and C are referenced in their entirety.

Appendix B: Experimental Instructions (Active Involvement Group Exemplary)

Scenario

• Over the course of the experiment, you assume the role of a skilled worker in an industrial company. You will be responsible for the operation of a production facility.

• There is a certain probability that a malfunction may occur in the production facility. To avoid potential malfunctions, maintenance can be performed.

• Your task is to evaluate the probability of malfunctions in multiple rounds and then decide whether the production plant should be maintained in a given round. You will be supported in your task by an artificial intelligence (AI).

Malfunction Probability

• The probability of a malfunction is unknown but can be estimated using three indicators.

• These indicators are: Temperature, Speed and Voltage. Each of the indicators can take values between 0 and 100.

• Each of the indicators has its own optimal range. If an indicator is in its optimal range, this is particularly good for the production plant corresponds to a malfunction being less likely.

• The more indicators’ values are located outside their respective optimum ranges, the more likely a malfunction becomes.

• If all three indicators are within their optimal ranges and none are outside, a malfunction is very UNlikely.

• If two indicators are within their optimal ranges and one is outside, a malfunction is UNlikely.

• If one indicator is within their optimal range and two are outside, a malfunction is likely.

• If all three indicators are outside their optimal ranges and none inside, a malfunction is very likely.

  

Example: The graphic above displays an example for the optimal ranges (green bars) for the three indicators (orange dots). In this example, the “Temperature” indicator is located outside its optimal range and the “Speed” and “Voltage” indicators are located inside their respective optimal ranges. Accordingly, a malfunction would be considered unlikely in this case.

• Important: In the experiment, you do not know the optimal ranges. Instead, you must estimate them as accurately as possible, based on the data points of past malfunctions. This estimation is called “acceptable ranges” (see “Procedure”).

Support by an AI

• In each round, the AI predicts the probability of a malfunction and, based on this prediction, gives you a non-binding recommendation as to whether maintenance should be performed.

• The accuracy of the AI predictions can vary. It depends on how the AI has been trained. Training the AI is part of the experiment (see below). You will be informed about the achieved accuracy of the AI (in percent) in the experiment at the end of Stage 3.

Procedure

The experiment consists of four stages that build upon each other.

Stage 1: Comprehension Checks

• In this stage, comprehension checks are conducted about the instructions. Only once you have answered all the control questions correctly the experiment can begin. You have an unlimited number of attempts to answer the questions correctly.

Stage 2: Selection of Acceptable Ranges

• As mentioned, the optimal ranges of the individual indicators are unknown to you. Instead, you must define an acceptable range for each indicator.

• An acceptable range is an approximation of the actual (unknown) optimal range. The closer the acceptable ranges you set are to the optimal range, the better the AI’s advice will be.

• Data about past malfunctions is available for you to set your acceptable ranges:

  • For each indicator individually, you can see at which values there were malfunctions in the past (red dots) and at which there were not (green dots).

  • You are now asked to define a lower limit (minimum) and an upper limit (maximum) of your acceptable range (green dashes). In general, an acceptable range should contain as many points without malfunctions (green) and as few points with malfunctions (red) as possible.

    

  • At the beginning, you will be given an example that you can use to practice setting the limits (technical note: the limit that is closer to your mouse pointer moves in each case).

  • After you have set and confirmed the acceptable ranges, they will be displayed as gray bars in the further course of the experiment for the sake of conciseness (see figure). The individual data points are hidden.

    

• You define a total of three acceptable ranges (one for each indicator), which you will necessarily need in the further course of the experiment.

• Your acceptable ranges will be displayed for all further decisions, so you do not have memorize or note them.

Stage 3: Training the AI

• In this stage, the AI is trained based on your acceptable ranges defined in Stage 2. The AI thus learns how to evaluate the probability of malfunctions for different indicator combinations.

• The training of the AI happens through ten training situations as follows:

  • Each training situation represents a combination of the three indicators’ values. These values are shown together with their acceptable ranges.

  • The following figure provides an example of a training situation. Orange bars represent the indicators’ values. Gray bars represent the acceptable ranges.

    

  • For each training situation, you can see which indicators are within and which are outside your defined acceptable ranges.

  • Your task is to tell the AI how each training situation is to be evaluated regarding the likelihood of a malfunction. In doing so, you help the AI learn.

  • Use your acceptable ranges and your knowledge about the probability of malfunctions for the evaluation:

    • If three indicators are within your acceptable ranges and zero are outside, a malfunction is very UNlikely.

    • If two indicators are within your acceptable ranges and one is outside, a malfunction is UNlikely.

    • If one indicator is within your acceptable ranges and two are outside, a malfunction is likely.

    • If there are zero indicators inside your acceptable ranges and three outside, a malfunction is very likely.

  • Each of your ten malfunction probability assessments is then checked for correctness:

    • If a malfunction has been classified as “very unlikely” or “unlikely” and no malfunction has actually occurred, the assessment is considered correct and otherwise incorrect.

    • If a malfunction was classified as “very likely” or “likely” and a malfunction actually

    • occurred, the evaluation is considered correct and otherwise incorrect.

    • Whether a malfunction actually occurs or not depends on the actual optimum ranges, which remain unknown.

• The results of training, and thus the quality of the AI, depends on how many training situations have been correctly assessed. You will be informed about the result at the end of the training stage. Two results are possible:

  • If at least seven training situations were evaluated correctly, you will receive an AI with the accuracy of 90% (on average it is correct in 9 out of 10 cases and wrong in one out of 10 cases).

  • If less than seven training situations were evaluated correctly, you will receive an AI with the accuracy of 50% (it is correct on average in 5 out of 10 cases and wrong in 5 out of 10 cases).

  • After completing this stage, the AI has learned to evaluate malfunction probabilities in comparable situations through the training situations.

  • In stage 4, you can use the AI for decision support.

Stage 4: Production Plant Surveillance

• This stage consists of 25 rounds.

• In each round you have to make the decision whether to maintain the production facility.

• All rounds are independent of each other, i.e., the decision in one round does not affect other rounds.

• In each round, you will receive a graphic showing the values of the three indicators and your self-defined acceptable ranges (see Stage 3).

• In each round, you make your decision in two steps:

  • In the first step, you evaluate the given situation in terms of the probability of failure and decide whether maintenance should be performed.

  • In the second step, the AI’s recommendation is displayed to you. Afterwards, you are asked again whether you want to perform maintenance.

• Only the decision in the second step is relevant for your payoff in the respective round.

• Whether a malfunction actually occurs or not depends on the optimal ranges, which remain unknown. You will only find out at the end of the experiment how often you were correct and how high your payoff will be.

Payoffs

• During the experiment, all amounts are denoted in the fictitious currency “Taler”.

• Per round, depending on your maintenance decision and the occurrence/non-occurrence of a malfunction, you will receive the following payoffs:

• You decide that maintenance should be performed.

  • Maintenance limits your production capacities. Therefore, your payoff this round is 5 Taler.

• You decide that no maintenance should be performed.

  • If no malfunction occurs and you can therefore produce fully, your payoff from this round is 10 Taler.

  • If a malfunction occurs and therefore you cannot produce, your payoff from this round is 0 Taler.

• The payoffs from all rounds are cumulated.

• At the end of the experiment, you will receive your payoffs at an exchange rate of 1€ per 10 Taler. In addition, you will receive a show-up fee of 2.50 €.

Additional Remarks

• All communication is prohibited for the duration of the experiment except for communication explicitly permitted by the instructions.

• Mobile phones must be turned off for the duration of the experiment.

• All decisions within the scope of the experiment will remain completely anonymous.

• After completing the main part of the experiment, we kindly ask you to answer some additional questions. Answering the questions honestly and in full is very important for the subsequent analysis of the experiment. The answers to the questions remain anonymous and will only be evaluated for scientific purposes. Your answers in this questionnaire have no impact on your payoff achieved in the experiment.

Appendix C: Questionnaire

Please answer the following questions.

What is your age?

What is your gender?

• Male

• Female

• Non-Binary

What is your highest level of education?

• Highschool/GED

• Undergraduate degree

• Graduate degree

• Else/Prefer not to say

What is your current study major?

Please answer the following questions

Please indicate your consent with the following statement on a scale from 1 (= completely disagree) to 7 (= completely agree).

Please indicate your consent with the following statement on a scale from 1 (= Do not consent at all) to 5 (= Fully consent).

Please indicate your consent with the following statement on a scale from 1 (= Do not consent at all) to 5 (= Fully consent).