Project Aura
Architecture, Working Plan & Current Status
Last Updated: July 14, 2026
← Return to Project OverviewView Interview Behavior Coverage →Current Phase
🟢 End-to-End Prototype & Reviewer Calibration
Current Milestone
Orientation Core v0.1
Next Milestone
Reviewer Policy & Integration Alignment
Next Review
American Councils Demo — July 15, 2026
Project Overview
Project Aura is an AI-assisted conversational assessment platform designed to help American Councils collect richer, more structured behavioral evidence through adaptive scenario-based interviews.
Aura does not ask a language model to make an unexplained applicant decision. It separates language interpretation, state management, adaptive probing, factor assessment, and recommendation policy into explicit and inspectable system components.
The purpose of the system is to make reviewer reasoning more observable, consistent, explainable, and easier to navigate.
Human reviewers remain the final decision makers.
Current Product Understanding
1. Evidence before inference
Aura extracts only behavior that is grounded in the student's response. Every retained observation must be traceable to an exact supporting quote.
2. Only assess what is observed
A student is not penalized because a factor did not naturally surface. Unobserved factors remain insufficient rather than becoming negative.
3. State over isolated answers
Aura maintains a cumulative orientation state across the interview instead of treating every response as an independent score.
4. Adaptive but bounded probing
Aura selects follow-up questions from an approved scenario-specific probe bank. The planner chooses the probe expected to reduce the most useful remaining uncertainty.
5. Policy remains explicit
Factor aggregation and recommendation rules are deterministic, inspectable, and separate from language-model evidence extraction.
6. Continuous calibration
Evidence definitions, probe policies, factor aggregation, recommendation thresholds, and reviewer outputs will improve through reviewer feedback, staff testing, pilot results, and future cohort data.
Current Prototype Capability
Live Student Experience
Two scenarios, each with one baseline question and two adaptive follow-up probes.
Grounded Evidence Extraction
LLM-assisted extraction of quoted behavioral evidence across the full eight-factor rubric.
Orientation State
Persistent factor-level evidence, confidence, observed status, and behavioral flags across turns.
Adaptive Probe Planning
Approved probes ranked according to the uncertainty they are expected to reduce.
Factor Assessment
Deterministic aggregation with diminishing returns, contradiction tracking, confidence bands, and scenario breadth.
Recommendation Policy
Auditable recommendation scores from -5 to +3, accompanied by confidence, coverage, reasons, and limitations.
Reviewer Packet
Recommendation, factor assessments, evidence quotes, behavioral flags, and complete transcript.
Calibration Playground
Internal interface for comparing expected evidence, extracted evidence, probe ranking, factor assessments, and recommendation behavior.
Product Workflow — Reviewer View
This view explains what Aura does from the perspective of the student experience and reviewer workflow.
Student completes an assigned scenario
↓
Aura records the response, prompt, scenario, probe, and turn metadata
↓
Aura acknowledges the student's stated approach with a neutral conversational bridge
↓
Aura selects the next approved follow-up probe
↓
The process repeats across two scenarios and six total turns
↓
Aura prepares factor-level assessments from grounded evidence
↓
Recommendation policy generates a provisional score, confidence, coverage, reasons, and limitations
↓
Reviewer receives the recommendation, evidence, factors, flags, and complete transcript
↓
Human reviewer interprets the packet and retains final decision authority
Orientation Core Reference Architecture
The Orientation Core is the reasoning layer underneath the student and reviewer experiences. It does not score the raw response directly. It progressively converts language into evidence, state, uncertainty, policy, and reviewer-facing output.
Student Response
Natural-language input submitted in response to the current scenario or approved follow-up probe.
Evidence Extraction
The language model identifies distinct behaviors supported by exact response quotes and maps them to the best-fitting rubric dimension.
Orientation State Update
Validated evidence and behavioral flags are added to the cumulative session state.
Uncertainty Mapping
The system identifies factors that remain unobserved, lightly supported, contradictory, or strategically valuable to investigate.
Probe Selection
Approved scenario probes are ranked according to expected uncertainty reduction, coverage, prior use, and remaining interview capacity.
Factor Assessment
Evidence is aggregated within each factor using diminishing returns, directional signal, scenario breadth, contradiction status, and evidence sufficiency.
Recommendation Policy
Factor-level orientation is converted into a provisional recommendation with confidence, coverage, reasons, limitations, and policy overrides.
Reviewer Packet
The final reviewer-facing structure contains recommendation context, factor assessments, evidence, flags, and the complete transcript.
JSON Export / AIS Handoff
A structured payload can be returned through a controlled integration boundary for American Councils systems.
Separation of Responsibilities
The current architecture deliberately assigns different responsibilities to the language model, Orientation Core, domain configuration, application, and human reviewer.
| Layer | Owns | Does Not Own |
|---|---|---|
| Language Model | Language understanding, exact-quote evidence extraction, concise behavioral descriptions, and conversational bridge generation. | Orientation state, probe policy, factor aggregation, final scoring thresholds, or reviewer decisions. |
| Aura Domain Pack | Factor definitions, scenario content, approved probes, probe objectives, and domain-specific calibration cases. | Runtime state, general orchestration, or applicant decisions. |
| Orientation Core | State accumulation, uncertainty representation, evidence history, probe ranking inputs, and factor-level orientation. | Free-form question invention or opaque LLM scoring. |
| Recommendation Policy | Deterministic score bands, confidence, coverage, reasons, limitations, and policy overrides. | Semantic interpretation of raw text or final human judgment. |
| Application | Student experience, reviewer interface, API boundaries, transcript display, and integration payloads. | Reviewer policy definitions. |
| Human Reviewer | Interpretation, policy calibration, contextual judgment, and final decision authority. | System implementation details. |
Current System Assumptions
Factor definitions are the source of truth for evidence extraction.
Scenario and probe factor mappings guide planning but do not restrict what evidence may be observed.
One distinct behavior should normally map to one best-fitting factor.
Absence of evidence is not negative evidence.
Neutral evidence may indicate uncertainty or self-awareness without supporting a positive or negative conclusion.
Repeated evidence within one factor receives diminishing returns.
Evidence across multiple scenarios increases confidence in cross-context interpretation.
Contradictory evidence remains visible rather than silently canceling out.
Confidence describes evidence sufficiency, not a statistical probability about the student.
Strong recommendations require breadth, consistency, and cross-scenario evidence.
Current score thresholds are provisional policy hypotheses for reviewer calibration.
Human reviewers retain authority over interpretation and final decisions.
Recommendation Policy v0.1
The recommendation layer is currently deterministic and auditable. It evaluates the shape of the factor-level orientation rather than adding raw evidence counts.
| Score | Current Provisional Meaning |
|---|---|
| +3 | Strong positive orientation supported broadly and consistently. |
| +2 | Positive orientation supported across multiple factors and contexts. |
| +1 | Limited positive orientation; evidence is constructive but narrow, incomplete, or accompanied by limited concern. |
| 0 | No clear positive or negative orientation; evidence may be absent, weak, balanced, or unresolved. |
| -1 | Limited concern. |
| -2 | Material concerns across multiple factors or patterns. |
| -3 | Strong, broad, or repeated concern. |
| -5 | Decision-critical conduct requiring immediate reviewer attention under the current provisional policy. |
Important calibration question
Current Integration Boundary
The prototype currently demonstrates the interview and reviewer workflow independently of direct AIS database access. The intended production boundary should minimize participant data and use narrowly scoped interfaces.
AIS or American Councils system assigns a participant
↓
Aura receives only the minimum required participant identifier (future: interview configuration)
↓
Aura conducts the adaptive interview
↓
Aura produces a structured reviewer packet
↓
American Councils receives a JSON result through an approved API, webhook, or controlled file exchange
↓
American Councils retains control of the participant record and final reviewer decision
Preferred Pilot Boundary
- Pseudonymous participant ID
- No broad AIS database access
- Least-privilege service account
- Sandbox or test integration first
- Explicit retention and deletion window
- No raw student content in ordinary application logs
Potential Export Fields
- Participant ID
- Scenario and probe IDs
- Transcript and turn metadata
- Grounded evidence
- Factor assessments
- Recommendation and limitations
Pilot Security Posture
PanoSight does not currently maintain its own SOC 2 attestation. The pilot is deployed on infrastructure providers with established security and compliance programs, but their attestations do not automatically extend to the Aura application or PanoSight organization.
The proposed pilot posture is based on proportionate controls and a deliberately narrow data boundary.
Minimum necessary participant data
Least-privilege access
Environment variables for secrets
Encrypted network transport
Restricted production access
No production credentials in source control
Defined data retention and deletion
Documented subprocessors
Incident notification process
Human review of all consequential outputs
Security question for American Councils
Working Backwards Timeline
| Target | Window | Deliverable | Purpose |
|---|---|---|---|
| Orientation Core v0.1 | Completed | Evidence, state, planning, factor assessment, and recommendation pipeline | Demonstrate the categorical value of an explicit orientation architecture. |
| Student Experience | Completed | Two scenarios and four live adaptive probes | Validate an end-to-end student interaction using approved question banks. |
| Reviewer Workflow | Completed | Live recommendation, factors, evidence, flags, and transcript | Make reviewer reasoning inspectable and evidence-backed. |
| Reviewer & CTO Alignment | July 15 | Architecture, scoring, security, and integration review | Identify policy assumptions, technical requirements, and pilot boundaries. |
| Calibration Iteration | After July 15 | Updated factor definitions, flag policy, score bands, and probe strategy | Replace engineering assumptions with reviewer-calibrated policy. |
| Staff Testing | Mid-to-late July | Internal test cohort and structured reviewer feedback | Measure extraction consistency, scoring agreement, usability, and edge cases. |
| Pilot Integration | To be confirmed | Approved JSON contract and controlled AIS handoff | Connect Aura to the operational reviewer workflow with an agreed data boundary. |
Current Dependencies
Dependency
Reviewer Policy Calibration
Owner
Joe / Elena / Reviewer Team
Status
In Discussion
Blocks / Informs
- Score-band meanings
- Factor weighting
- Attention-flag policy
- Strong versus weak candidate interpretation
- Treatment of contradictions
Dependency
Security & Data Boundary
Owner
Shawn / Technical Team / Richard
Status
Initial Review
Blocks / Informs
- SOC 2 expectations
- Data Access Agreement
- OpenAI and Vercel subprocessor expectations
- Participant data minimization
- Retention and deletion requirements
- Incident-response expectations
Dependency
AIS Integration Contract
Owner
American Councils Technical Team
Status
Not Yet Defined
Blocks / Informs
- Authentication method
- API or webhook structure
- JSON payload format
- Read and write permissions
- Sandbox availability
- Error-handling and retry expectations
Dependency
Golden Calibration Dataset
Owner
Joe / Elena / Reviewers
Status
Planned
Blocks / Informs
- Extractor agreement measurement
- Factor-level calibration
- Recommendation policy refinement
- Future statistical or machine-learning models
Current Reviewer Calibration Questions
- □What should each score band mean operationally?
- □Should a decision-critical flag automatically produce -5, or create a separate mandatory-review status?
- □Which behaviors qualify as decision-critical?
- □Are all eight factors equally important?
- □Are any factors gating or disproportionately important?
- □How much breadth is required for +2 or +3?
- □Should strong evidence in one scenario ever support a strong recommendation?
- □How should later evidence relate to earlier contradictory evidence?
- □When does emotional distress become concern evidence rather than a neutral reaction?
- □How should support-seeking be interpreted across age, culture, and context?
- □What response patterns count as nonresponsive or insufficient?
- □Which reviewer feedback fields can be collected during staff testing?
- □Should cohort-relative ranking complement the criterion-based score later?
Backend & Module Map
1. Aura Domain Pack
- Eight behavioral factor definitions
- Scenario prompts
- Approved probe banks
- Probe objectives and target dimensions
- Calibration cases and expected evidence
2. Evidence Layer
- OpenAI structured evidence extraction
- Exact quote validation
- Rubric-wide factor mapping
- Behavioral flag extraction
- Neutral conversational bridge
- Extraction diagnostics
3. Orientation Core
- Initial orientation state
- Evidence accumulation
- Factor-level state updates
- Observed and unresolved dimensions
- Behavioral flag state
4. Interview Planner
- Uncertainty-aware probe ranking
- Approved question-bank selection
- Used-probe tracking
- Coverage scoring
- Planner rationale
5. Factor & Recommendation Layer
- Diminishing-return evidence aggregation
- Positive, concern, mixed, and insufficient states
- Confidence bands
- Contradiction tracking
- Recommendation metrics
- Auditable recommendation policy
6. Application & Integration
- Student interview UI
- Reviewer workflow UI
- Orientation Core Playground
- Working plan and architecture surface
- Orientation turn API
- Calibration endpoint
- Future JSON / AIS integration
Mid-July Pilot Goal
Demonstrated Today
- Live adaptive student conversations
- Grounded evidence extraction
- Orientation state updates
- Uncertainty-aware probe selection
- Factor-level assessments
- Recommendation reasons and limitations
- Reviewer transcript and evidence auditability
Requires Alignment
- Reviewer-approved score semantics
- Decision-critical flag policy
- Production data boundary
- AIS integration contract
- Security control requirements
- Golden dataset generation
- Staff-testing and calibration workflow
Design Principle
Orientation before automation
Project Aura is not intended to automate evaluator judgment. Its purpose is to continuously reduce uncertainty, preserve the evidence behind every conclusion, and present the right information at the right time while maintaining human decision-making throughout the evaluation process.