“Auditor-in-a-Box: Tools for Third-Party Auditing” by Roy Rinberg, Ben Penchas

Introduction:  There is a need for untrusting parties to share information. In the world before LLMs (and even today) this need has largely been satisfied using legal contracts (and sometimes through cryptography and blockchain technologies). However the scale and pace of things to keep track of and monitor has grown immensely and legal contracts appear insufficient. LLMs can help give auditors the right tools to enable information sharing with untrusted parties. In this post, I outline the shape of the problems that need to be resolved to enable using LLMs for 3rd party auditing, and present one concrete solution we have attempted. We designed a scheme for third party auditing: an open-source LLM, running inside a trusted execution environment (TEE), which executes commands that the two parties have agreed on over private data. In this post, I highlight that this type of tooling can directly be applied to enabling 3rd party monitoring for regular governance concerns, it has applications to enabling improved monitoring for customers who want Zero-Data-Retention, and it also can be one of the tools that enable a verified slowdown. Code: We release an open-source implementation that runs in a real TEE. The code is [...] ---Outline:(00:12) Introduction:(02:26) 1. Secure Multi-party computation(05:05) 2. The primitive: a trusted LLM in a trusted box(08:52) 3. Process-level problems(09:04) 3.1. Plan negotiation (How do you agree on a plan)(10:59) 3.2. False positives and appeals(14:03) 3.3. Out-of-distribution inputs and prompt injection(16:20) 4. Applications(16:28) 4.1. Application: Verifiably Scoped Monitoring (reconciling safety monitoring with zero-data retention)(20:10) 4.2. Application: recurring third-party audit(22:03) 5. Limitations and trust assumptions(23:23) 6. A Reference Implementation(25:39) 7. Current and Future States(26:55) Appendix(26:58) Example Computation Graph for Privacy Preserving Monitoring(28:10) Ledgers for handling abusing the false positive appeals process The original text contained 1 footnote which was omitted from this narration. --- First published: July 28th, 2026 Source: https://www.lesswrong.com/posts/uWYk7MM9hAf9GEbGe/auditor-in-a-box-tools-for-third-party-auditing --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.