Tools for Effective AI Behavioral Research.

OpenPsy is an open research platform for designing, running and analysing behavioural experiments, including experiments in which language models take part as participants. The goal of OpenPsy is to enable researchers to understand what is happening beneath their results and to have an audit history to understand the progress of their research programs. OpenPsy is presently in internal closed alpha.

Why OpenPsy?

Agentic research can be difficult to parse, monitor, and understand the underlying operations and what agents are doing. OpenPsy was born from this difficulty and aims to assist researchers in agent to agent and agent to human research. The software is designed to improve a researcher's ability to plan, execute, and analyze agentic experiments with or without human participants.

Introspection

OpenPsy enables the effective design and execution of experiments by providing simulation prior to execution, and saved session introspection for auditability.
Replicability

OpenPsy saves every version of a design with its exact wording, so a researcher can export the design or prepare a replication that stays linked to the version it copies.
Context Control

OpenPsy enables researchers to understand exactly what they are serving their agents at each step of an experiment. This understanding is necessary to have confidence in the ground truth experimental operations leading to results.

Building Studies as One Flow

The Flow editor is where a researcher builds an experiment. Using "blocks" the researcher can design the ideal system using a simple node graph interface. All systems read from the same common database, so the plan, the editor and the participant's view are tied to what the protocol ultimately executes. These screenshots show a hypothetical example study, authored through the real OpenPsy interface.

Example Study

1Design

Question: Does a gain or loss framing in a question affect how an LLM responds? Does this response vary across models?
Factors: The sample experiment is a 2x2, two factors with two levels. The first factor is a gain versus loss frame. The second factor is model type, Claude or GPT. Participants are randomized into four conditions.
Instructions: You will read a short scenario about a public health decision. Read it carefully and then answer the questions that follow. There are no right or wrong answers.

Conditions Each session is shown one version, chosen at random.

Gain frame

A new infection is expected to affect 900 people in a region. Health officials are deciding whether to adopt Plan A. If Plan A is adopted, 300 of those people will be protected.

Loss frame

A new infection is expected to affect 900 people in a region. Health officials are deciding whether to adopt Plan A. If Plan A is adopted, 600 of those people will be left unprotected.

The two versions describe the same outcome and differ by one clause. OpenPsy treats that difference as the manipulation, and it checks that every other step of the procedure has exactly the same wording in both conditions.

The OpenPsy Flow graph at 90 percent zoom, and nothing in it is selected. A toolbar above the graph says Version 4 is saved and has a Simulate button, a speed slider set to 4x slower, Subject and Agent tabs and zoom controls. A Random assignment block, which previews one of the 4 conditions with seed 4107 and 4 routes, connects by dashed lines to two boxes marked Independent variable that are stacked one above the other. Each box has 2 planned levels applied at Task instructions. The Frame box holds the levels Gain frame and Loss frame, and each level has a Prompt badge. The Model box says the model taking part varies with the same delivery for every model, and holds the levels Claude and GPT, each with a Model badge. Every level reads Planned, not yet run. The dashed connectors from the two variables meet just before Task instructions and continue as one arrow. Task instructions, labeled Instruction with a Prompt badge, then leads in order to Recommendation Binary (Yes/No) labeled Behaviour, Open-Ended Rationale Explanation labeled Production and Confidence Rating (7 Point Scale) labeled Endorsement. Each of the three measures has an Interface badge, and each of the four cards reads 1 item and Design only. — **The Same Study as a Graph.** A random assignment block gives each session one of the 4 conditions. Frame and Model are both independent variables, stacked in one column: Frame has Gain frame and Loss frame levels, and Model has Claude and GPT levels. Both apply at the task instructions, which are followed by Recommendation Binary (Yes/No), Open-Ended Rationale Explanation and Confidence Rating (7 Point Scale).

The Add a Block panel in OpenPsy. It has a Close button and a search field labeled Search blocks. Under Functions, Participant instructions is listed at version 1. Under IVs, Gain frame and then Loss frame are listed at version 1. Under DVs, Recommendation Binary (Yes/No), Open-Ended Rationale Explanation and Confidence Rating (7 Point Scale) are listed in that order at version 1. Each group has a short description and a Show blocks from menu, which is set to This program for Functions and IVs and to All programs for DVs. Each block has an Add to Flow button. — **Blocks Come From a Shared Library.** The researcher can add blocks from their common pool, using their bank of IVs and DVs.

The OpenPsy editor for the selected block, Confidence Rating (7 Point Scale). It has buttons to add a block after this one, move it earlier and remove it, and the Move later button is unavailable. The Block title field reads Confidence Rating (7 Point Scale), and Delivered to is set to The participant. The wording box, three lines tall, reads: How confident are you in your recommendation, from 1 (not at all confident) to 7 (completely confident)? A note below says the block is pinned to Confidence Rating (7 Point Scale), version 2, that changing the wording creates the next version, and that the saved version is kept. — **Versioning of Blocks Tracks Provenance.** Researchers can modify and track changes and history of experiments to understand their progress.

The header of the experiment Gain and loss framing in a treatment choice. It has Create revision, Prepare replication and Export design buttons and a Version menu set to Version 4. Below it, the Source and attribution section is closed and the Changes from version 3 section is open. The open section shows a shaded panel with a green rule on its left edge. The panel says that Confidence Rating (7 Point Scale) moved from version 1 to version 2 of its wording. — **Every Saved Change Is Recorded.** In this example, the researcher has changed the wording of the confidence rating in version 4, which is now forked from the prior version 3.

2Execution

The execution screens for studies built in the Flow editor are still in progress. This example has not been run, so there is nothing to show at this stage yet.

3Results

The Hypothetical Results section of the example study. A dashed notice reads: These results are hypothetical. They were made up to show how a two-by-two result is reported, and they did not come from running Claude, GPT or any other model. Below it, a table is titled Sessions Recommending Plan A, by Frame and Model (Hypothetical). Its first header row reads Model, Claude with a purple swatch, GPT with a blue swatch, and Total. Its second header row reads Frame, with empty cells beside it. The Total header and the Total row label are set in the same bold type as the Claude, GPT, Gain frame and Loss frame headers. The Gain frame row reads 72% (36 of 50, mean confidence 5.4) for Claude, 74% (37 of 50, mean confidence 5.5) for GPT and 73% (73 of 100, mean confidence 5.5) in the Total column. The Loss frame row reads 46% (23 of 50, mean confidence 5.0) for Claude, 46% (23 of 50, mean confidence 5.1) for GPT and 46% (46 of 100, mean confidence 5.1) in the Total column. The Total row reads 59% (59 of 100, mean confidence 5.2) for Claude and 60% (60 of 100, mean confidence 5.3) for GPT, and its last cell is empty. The Total column and the Total row have a light green fill and a heavier rule on their inner edge. A note under the table says that comparing the row totals shows the main effect of Frame, comparing the column totals shows the main effect of Model, and comparing the four cells shows whether the factors interact. Beneath the table is a grouped bar chart titled Percentage of Sessions Recommending Plan A, by Frame and Model (Hypothetical). It has a legend for Claude in purple and GPT in blue and a vertical axis from 0% to 100%. For each frame it shows a Claude bar and a GPT bar, labeled 72% and 74% for Gain frame and 46% and 46% for Loss frame. At the top of the chart, a dark bracket marked with three stars spans the Gain frame and Loss frame groups. It ends in two short equal ticks, one centred over each group, that reach the 100% gridline and stop well above the other brackets. Below it, a blue bracket marked with one star has long legs that drop to the 74% label of the Gain frame GPT bar and the 46% label of the Loss frame GPT bar. Lower still, a purple bracket marked with one star has legs that drop to the 72% label of the Gain frame Claude bar and the 46% label of the Loss frame Claude bar. The blue bracket's left leg crosses the purple bracket's line just above the 74% label. An APA note under the chart gives the key *p < .05, **p < .01 and ***p < .001. It reports a logistic regression with Wald z tests, with Frame coded +0.5 for Gain frame and −0.5 for Loss frame and Model coded +0.5 for Claude and −0.5 for GPT. The regression showed a significant main effect of Frame, b = 1.16, SE = 0.30, z = 3.83, p < .001, no significant main effect of Model, b = −0.05, SE = 0.30, z = −0.17, p = .87, and no significant interaction, b = −0.10, SE = 0.60, z = −0.17, p = .87. The note explains how each b is read under this coding. It then reports Holm-corrected two-sided two-proportion z tests. Gain frame and Loss frame differed significantly for Claude, z = 2.64, p = .025, and for GPT, z = 2.86, p = .017. Claude and GPT did not differ significantly within Gain frame, z = −0.23, p = 1.00, or within Loss frame, z = 0.00, p = 1.00. A last line says each percentage is the share of sessions in that condition or total that met the outcome, computed from exact counts and rounded once with halves rounded up, and that no confidence interval is shown. — **The Standard Two-by-Two Results Page.** OpenPsy reports a two-by-two design as a table with row totals on the right and column totals along the bottom, so both main effects can be read at a glance. A grouped bar chart follows, with the gain frame on the left, the loss frame on the right and the bars coloured by model. Stars mark the comparisons that the study's declared analysis found significant, and the note under the chart reports each test in APA style. The numbers here are hypothetical and were not produced by running any model, and the page says so in its notice and in both titles.

OpenPsy Roadmap

OpenPsy is in internal closed alpha. It has been tested with invented data only, and no real participant has taken part.

In Progress

A researcher can create an experiment, build it as a graph from Library blocks, edit each block's wording, and save every change as a new version.
Any two saved versions can be compared, and the differences are written out as sentences.
Library wordings are stored as exact versions, and publishing a revision leaves existing experiments on the version they used.
A scripted test participant with invented answers can take an authored study from consent to completion on a phone-sized screen.
Before the software is built, a chain of eight automatic checks runs, and any one failure stops the build.

Coming Soon

Scientific pilots and full data collection are being built and are not enabled yet.
Recruiting human participants will come after ethics and consent review and the other steps that formal data collection requires.
Hosted sign-in for collaborators has not been qualified yet.
Analysis and replication runs inside the web product have not been qualified yet.
Citations produced by the software are provisional for now.

Source Code

The source code, the specification and the product documentation are being prepared for publication on GitHub.

GitHub repository opening soon