Artificial Intelligence and Human Subjects Research

What should I consider if I am interested in conducting human subjects research that involves Artificial Intelligence (AI) in my research?

Protocols involving Artificial Intelligence (AI) present unique factors that need to be addressed during the IRB review process. AI includes chatbots, algorithms, deep learning, machine learning/intelligence, predictive analytics, and more.

AI Definitions (from the National Library of Medicine)

Artificial Intelligence: refers to actions that mimic human intelligence displayed by machines and to the field of study focused on this type of intelligence. AI consists of computer programs that are typically built to adaptively update and enhance their own performance over time. They are used to process, analyze, and recognize patterns in large datasets, and they use those patterns to get better at completing tasks or solving problems.

Generative AI: is a system of algorithms or computer processes that can create novel output in text, images or other media based on user prompts. These systems are created by programmers who train them on large sets of data. The AI learns by finding patterns in the data and can then provide novel outputs to users’ queries based on its findings. Generative AI systems like ChatGPT, which employs a Large Language Model, go a step further to generate new combinations of text and ideas for users based on their questions or requests.

Machine Learning: involves sophisticated algorithms which can be trained to sort information, identify patterns, and make predictions within large sets of data. Machine learning algorithms are used by researchers to build models based on sample or training data in order to make predictions or decisions, without being explicitly programmed to do so.

How do I describe the AI system I intend to use?

When submitting a protocol that utilizes AI to CPHS, the following items will need to be described in your protocol:

  • The AI system name(s)/version(s)
  • The type of AI system:
    • An unaltered commercial/public product
    • A commercial/public product with modifications or custom instructions and/or agents added? If so, please explain what the alterations are and how they influence the system’s behavior and/or output.
    • An AI system developed by the research team and, therefore, not a commercial/public system. If so, explain what type of system it is (LLM, rule-based chatbot, image processing algorithm, etc.), how it was developed, what data it was trained on, and will the AI algorithm be continuously updating or is it fixed?
  • If the AI system is also considered Software as a Medical Device (SaMD; if you are uncertain, use the SaMD Regulatory Pathway Navigator tool), make sure to include this determination and reference our guide on FDA-Regulated Research for what to include in the protocol regarding the device. Note that research involving SaMD is not eligible for exempt level review.

How do I describe the ways AI will be used in my research when writing my protocol?

When writing your protocol involving AI, please explain the use case for the AI system in the research:

  • Collecting personal private information from individuals to train an AI model.
  • Development or testing of a new system, algorithm, tool, or device. Describe the intended end-use of system.
  • Behavioral intervention/interaction:
    • How do participants interact with the platform and what are the expected input/outputs, communication mode, etc.
    • Submit any instructions/prompts/guidance participants will receive.
    • If helpful, provide screenshots demonstrating the interaction.
    • If the AI provides unexpected, distressing, or harmful outputs to a participant, describe the safety net (e.g., how is a human researcher notified, and what is the turnaround time for intervention?)

There may be additional risks specific to the use of AI in research. How do I address those risks in my protocol?

Some risks may exist because of the data that will be collected and potential uses of those data, and sometimes the risks are specifically related to the research activity. The protocol must state potential risks or problems that could result from false positives, incorrect outputs, biases, hallucinations, or other unexpected behavior. How could that be minimized (human review of outputs, validation checks, referring participants to expert, etc.)?

  • If the AI system is not licensed by UC Berkeley, clarify whether participant data or other inputs will be used to train, adapt, or refine the system (as part of study or in general) or if the system is sandboxed. Importantly, if identifiable data is shared with a third-party system that is not licensed by UCB for identifiable participant data, the vendor must first agree to UC contract terms via UCB’s Procurement Office, and second, a vendor security assessment from UCB’s Information Security Office (ISO) must be completed. Review the ISO service web page for instructions on how to request an assessment. Note that UC contract terms should first be agreed upon before ISO is contacted. We advise you to reach out to Research Data Management to affirm that the software application you propose has been approved and is okay to use.
  • If using proposed new AI tools or new uses of existing tools (e.g., administrative uses, teaching tools, research tools, etc.), please follow the risk assessment process through the Compliance and Enterprise Risk Committee’s (CERC) Artificial Intelligence Risk Subcommittee (AIR), which is independent of IRB approval. Please include information in the protocol application if you have already undergone this process.

What information about the AI system should Informed Consent documents include?

  • State in plain language (6th–8th grade reading level) what the AI system is, what it is doing, and how it will be used.
  • How participants will be asked to interact with system.
  • Possible risks (e.g., while direct identifiers are removed, AI processing introduces a marginal risk of re-identification through cross-referencing datasets).
  • Privacy and confidentiality related to data input.
  • If and how results generated by the AI will be shared with the participant, particularly if the AI detects unexpected health/behavioral issues.
  • Whether participants’ data will be used to train future commercial or institutional iterations of the tool and whether participants’ data can be withdrawn once entered into the system.
  • If applicable, explain that participants can request alternatives or opt out of AI interactions or analysis.

Refer to our AI consent language guide for examples to include in Consent, Assent, and Permission forms involving AI tools in research.

Resources

If you have any questions about what to include in your protocol, please contact OPHS. If you are unsure what level of review is required for your project, see our Types of Review page and if still unsure, reach out to our office.

Related information