Tools for Effective AI Behavioral Research.

OpenPsy is an open research platform for designing, running and analysing behavioural experiments, including experiments in which language models take part as participants. The goal of OpenPsy is to enable researchers to understand what is happening beneath their results and to have an audit history to understand the progress of their research programs. OpenPsy is presently in internal closed alpha.

Why OpenPsy?

Agentic research can be difficult to parse, monitor, and understand the underlying operations and what agents are doing. OpenPsy was born from this difficulty and aims to assist researchers in agent to agent and agent to human research. The software is designed to improve a researcher's ability to plan, execute, and analyze agentic experiments with or without human participants.

Introspection

OpenPsy enables the effective design and execution of experiments by providing simulation prior to execution, and saved session introspection for auditability.
Replicability

OpenPsy saves every version of a design with its exact wording, so a researcher can export the design or prepare a replication that stays linked to the version it copies.
Context Control

OpenPsy enables researchers to understand exactly what they are serving their agents at each step of an experiment. This understanding is necessary to have confidence in the ground truth experimental operations leading to results.

Building Studies as One Flow

The Flow editor is where a researcher builds an experiment. Using "blocks" the researcher can design the ideal system using a simple node graph interface. All systems read from the same common database, so the plan, the editor and the participant's view are tied to what the protocol ultimately executes. These screenshots show a hypothetical example study, authored through the real OpenPsy interface.

Example Study

1Design

Question: Does a gain or loss framing in a question affect how an LLM responds? Does this response vary across models?
Factors: The sample experiment is a 2x2, two factors with two levels. The first factor is a gain versus loss frame. The second factor is model type, Claude Opus 5 or Claude Sonnet 5. Participants are randomized into four conditions.
Instructions: You will read a short scenario about a public health decision. Read it carefully and then answer the questions that follow. There are no right or wrong answers.

Conditions Each session is shown one version, chosen at random.

Gain frame

A new infection is expected to affect 900 people in a region. Health officials are deciding whether to adopt Plan A. If Plan A is adopted, 300 of those people will be protected.

Loss frame

A new infection is expected to affect 900 people in a region. Health officials are deciding whether to adopt Plan A. If Plan A is adopted, 600 of those people will be left unprotected.

The two versions describe the same outcome and differ by one clause. OpenPsy treats that difference as the manipulation, and it checks that every other step of the procedure has exactly the same wording in both conditions.

The OpenPsy Flow graph for the example study on a dotted canvas. On the left, a Random assignment block reads Preview one of the 4 conditions, Seed 4107 and 4 routes. Dashed arrows lead from it to two boxes labeled Independent variable, one above the other. The Frame box says 2 planned levels and Applied at Task instructions, and holds Level 1 Gain frame and Level 2 Loss frame, each with a Prompt badge. The Model box says 2 planned levels and Applied at Task instructions, adds that the model taking part varies with the same delivery for every model, and holds Level 1 Claude Opus 5 and Level 2 Claude Sonnet 5, each with a Model badge. Every level reads Planned, not yet run. Dashed lines from the two boxes join into one arrow to Task instructions, which is labeled Instruction and has a Prompt badge. Arrows then lead in order to Recommendation Binary (Yes/No) labeled Behaviour, Open-Ended Rationale Explanation labeled Production, and Confidence Rating (7 Point Scale) labeled Endorsement, each with an Interface badge. Each of these four cards reads 1 item and Design only. — **The Same Study as a Graph.** A random assignment block gives each session one of the 4 conditions. Frame and Model are the two independent variables, and both apply at the task instructions. Frame has Gain frame and Loss frame as its levels, and Model has Claude Opus 5 and Claude Sonnet 5. The task instructions are followed by Recommendation Binary (Yes/No), Open-Ended Rationale Explanation and Confidence Rating (7 Point Scale). Every level is marked as planned and not yet run.

The Add a Block panel in OpenPsy. It has a Close button and a search field labeled Search blocks. Under Functions, Participant instructions is listed at version 1. Under IVs, Gain frame and then Loss frame are listed at version 1. Under DVs, Recommendation Binary (Yes/No), Open-Ended Rationale Explanation and Confidence Rating (7 Point Scale) are listed in that order at version 1. Each group has a short description and a Show blocks from menu, which is set to This program for Functions and IVs and to All programs for DVs. Each block has an Add to Flow button. — **Blocks Come From a Shared Library.** The researcher can add blocks from their common pool, using their bank of IVs and DVs.

The OpenPsy editor for the selected block, Confidence Rating (7 Point Scale). It has buttons to add a block after this one, move it earlier and remove it, and the Move later button is unavailable. The Block title field reads Confidence Rating (7 Point Scale), and Delivered to is set to The participant. The wording box, three lines tall, reads: How confident are you in your recommendation, from 1 (not at all confident) to 7 (completely confident)? A note below says the block is pinned to Confidence Rating (7 Point Scale), version 2, that changing the wording creates the next version, and that the saved version is kept. — **Versioning of Blocks Tracks Provenance.** Researchers can modify and track changes and history of experiments to understand their progress.

The header of the experiment Gain and loss framing in a treatment choice. It has Create revision, Prepare replication and Export design buttons and a Version menu set to Version 4. Below it, the Source and attribution section is closed and the Changes from version 3 section is open. The open section shows a shaded panel with a green rule on its left edge. The panel says that Confidence Rating (7 Point Scale) moved from version 1 to version 2 of its wording. — **Every Saved Change Is Recorded.** In this example, the researcher has changed the wording of the confidence rating in version 4, which is now forked from the prior version 3.

2Execution

The Coverage & integrity panel of the Execution Monitor. Under the heading Pilot, a line reads "This phase is recorded in simulated mode." Below it is a green progress bar labelled 23/40 terminal. A section headed Cell coverage (counts only) holds a card titled Session Coverage by Frame and Model. The table has Model across the top, with a purple swatch for Claude Opus 5 and a blue swatch for Claude Sonnet 5, and Frame down the side. It has four cells, each with a green progress strip: Gain frame with Claude Opus 5 shows 6 of 10 sessions complete, Gain frame with Claude Sonnet 5 shows 6 of 10, Loss frame with Claude Opus 5 shows 6 of 10, and Loss frame with Claude Sonnet 5 shows 5 of 10. There is no Total column and no totals row. A note under the table reads "Each cell counts the scheduled sessions of one condition that have finished. The table shows how many sessions are done, never how they turned out." Below the card, the terminal taxonomy shows valid-observation 23. Three tiles at the bottom show ledger freshness (evidence arriving now), envelope closure (23/23 terminal units sealed by envelope) and scoring closure (23/40 scheduled units scored). — **Collection Is Tracked Without Showing Results.** The Execution Monitor shows how many sessions in each condition have finished, in the same table layout as the results, and it keeps results hidden until collection closes. The run shown here is simulated, and no model was called. Running studies built in the Flow editor with model participants is still in development.

3Results

The Hypothetical Results section of the example study. A dashed notice reads: These results are hypothetical. They were made up to show how a two-by-two result is reported, and they did not come from running any model. Below it, a table is titled Sessions Recommending Plan A, by Frame and Model (Hypothetical). Its first header row reads Model, Claude Opus 5 with a purple swatch, Claude Sonnet 5 with a blue swatch and Total, and its second header row reads Frame with empty cells beside it. The Total header and the Total row label are set in the same bold type as the Claude Opus 5, Claude Sonnet 5, Gain frame and Loss frame headers. The Gain frame row reads 72% (36 of 50, mean confidence 5.4) for Claude Opus 5, 74% (37 of 50, mean confidence 5.5) for Claude Sonnet 5 and 73% (73 of 100, mean confidence 5.5) in the Total column. The Loss frame row reads 46% (23 of 50, mean confidence 5.0) for Claude Opus 5, 46% (23 of 50, mean confidence 5.1) for Claude Sonnet 5 and 46% (46 of 100, mean confidence 5.1) in the Total column. The Total row reads 59% (59 of 100, mean confidence 5.2) for Claude Opus 5 and 60% (60 of 100, mean confidence 5.3) for Claude Sonnet 5, and its last cell is empty. The Total column and the Total row have a light green fill and a heavier rule on their inner edge. A note under the table says that comparing the row totals shows the main effect of Frame, comparing the column totals shows the main effect of Model, and comparing the four cells shows whether the factors interact. Beneath the table, a grouped bar chart is titled Percentage of Sessions Recommending Plan A, by Frame and Model (Hypothetical), with a legend for Claude Opus 5 in purple and Claude Sonnet 5 in blue and a vertical axis from 0% to 100%. It shows a Claude Opus 5 bar and a Claude Sonnet 5 bar for each frame, labeled 72% and 74% for Gain frame and 46% and 46% for Loss frame. At the top of the chart, a dark bracket marked with three stars spans the Gain frame and Loss frame groups and ends in two short equal ticks, one centred over each group, that stop well above the other brackets. Below it, a blue bracket marked with one star has long legs that drop to the 74% label of the Gain frame Claude Sonnet 5 bar and the 46% label of the Loss frame Claude Sonnet 5 bar, and lower still a purple bracket marked with one star has legs that drop to the 72% label of the Gain frame Claude Opus 5 bar and the 46% label of the Loss frame Claude Opus 5 bar. The only text under the chart is the star key, which reads *p < .05, **p < .01 and ***p < .001. Below the chart, a separate card headed Statistical Tests reports a logistic regression with Frame coded +0.5 for Gain frame and −0.5 for Loss frame and Model coded +0.5 for Claude Opus 5 and −0.5 for Claude Sonnet 5, with Wald z tests computed from the four cell counts, which showed a significant main effect of Frame, b = 1.16, SE = 0.30, z = 3.83, p < .001, no significant main effect of Model, b = −0.05, SE = 0.30, z = −0.17, p = .87, and no significant interaction, b = −0.10, SE = 0.60, z = −0.17, p = .87. It explains how each b is read under this coding. A second paragraph reports Holm-corrected pairwise two-proportion z tests: Gain frame and Loss frame differed significantly for Claude Opus 5, z = 2.64, p = .025, and for Claude Sonnet 5, z = 2.86, p = .017, while Claude Opus 5 and Claude Sonnet 5 did not differ significantly within Gain frame, z = −0.23, p = 1.00, or within Loss frame, z = 0.00, p = 1.00. A last line under that card says each percentage is the share of sessions in that condition or total that met the outcome, computed from exact counts and rounded once with halves rounded up, and that no confidence interval is shown. — **The Standard Two-by-Two Results Page.** OpenPsy reports a two-by-two design as a table with row totals on the right and column totals along the bottom, so both main effects can be read at a glance. A grouped bar chart follows, with the gain frame on the left, the loss frame on the right and the bars coloured by model. Stars mark the comparisons that the study's declared analysis found significant, and the note under the chart reports each test in APA style. The numbers here are hypothetical and were not produced by running any model, and the page says so in its notice and in both titles.

OpenPsy Roadmap

OpenPsy is in internal closed alpha. It has been tested with invented data only, and no real participant has taken part.

In Progress

A researcher can create an experiment, build it as a graph from Library blocks, edit each block's wording, and save every change as a new version.
Any two saved versions can be compared, and the differences are written out as sentences.
Library wordings are stored as exact versions, and publishing a revision leaves existing experiments on the version they used.
A scripted test participant with invented answers can take an authored study from consent to completion on a phone-sized screen.
Before the software is built, a chain of eight automatic checks runs, and any one failure stops the build.

Coming Soon

Scientific pilots and full data collection are being built and are not enabled yet.
Recruiting human participants will come after ethics and consent review and the other steps that formal data collection requires.
Hosted sign-in for collaborators has not been qualified yet.
Analysis and replication runs inside the web product have not been qualified yet.
Citations produced by the software are provisional for now.

Source Code

The source code, the specification and the product documentation are being prepared for publication on GitHub.

GitHub repository opening soon