Assessment services for companion AI operators

What this is

An independent evaluation of a companion chatbot's response conduct, structured to support the risk assessment required by California SB 1119 section 21812, which operators must perform annually beginning 1 July 2027.

The evaluation is performed against the published Companion AI Conduct Measurement Protocol. Because that protocol is public and versioned, it satisfies section 21812(a)(1)(B)(i)'s requirement to cite publicly available benchmarks.

What it is not

It is not a section 21814 child safety audit. That provision requires an audit of internal controls, processes and systems, and is work for an assurance firm. The evaluation sits upstream of it and produces the technical findings such an audit would examine.

Independence

Suoja makes the following commitments to every operator it assesses.

• Suoja sells no runtime product to operators. No guardrail, no filter, no content moderation service, no monitoring software deployed on an operator's systems. Selling the mitigation would foreclose assessing the deployment.
• No compensation is contingent on findings.
• Suoja does not assess any system it materially designed or operated.
• Suoja does not publish grades, scores or rankings of named products, so an engagement cannot be influenced by the prospect of a public rating.
• Suoja takes no custody of client conversation data. Evaluation is performed against the operator's deployed product using the published protocol, not against the operator's user records.

How the evaluation is performed

The method is the one set out in the published protocol, applied to a single deployment rather than a cohort. In outline:

• Scripted adult personas only. Every persona turn is written in advance, so runs are reproducible and comparable.
• Two arms. In one, no vulnerability is disclosed. In the other, the persona discloses a condition in one scripted line at a fixed turn, and every other turn is identical. The difference between the arms shows whether conduct changes in response to a disclosed vulnerability.
• Each assistant turn is labelled for conduct, outcome for the persona, and whether context was gathered before advising. Refusal is scored like any other response, not counted as a pass.
• Automated labels are checked against a qualified clinical reviewer, with agreement reported per dimension.

There is no pass mark, because no statute sets one. Rates are reported with their denominators and confidence intervals, not converted into a grade.

What an engagement produces

A written evaluation report covering:

• conduct measured against the published protocol;
• the methodology description section 21812(a)(1)(B) requires;
• denominators and confidence intervals for each reported rate;
• the limitations of the measurement.

Named results are provided to the operator and are not published.

What the public protocol withholds

A portion of the probe set is not published, so an evaluation cannot be prepared for in advance by tuning against published probes.

Contact

Enquiries from operators and their counsel: use the contact form.