Automated Legal Folder Structure Analysis
A .NET 10 desktop application that analyzes legal folder structures, classifies folders by role, detects anomalies, extracts entities, and provides an AI-powered chat interface for exploring results — all while keeping client data fully pseudonymized.
ML-powered classification of folders into roles: Client, Matter, DocumentCategory, SubCategory, AdHoc, TemporalGroup. Confidence scoring with rule-based + ML hybrid approach.
Detects 10 anomaly types: spelling errors, case inconsistencies, semantic duplicates, redundant nesting, role mismatches, path-too-long, and more. Severity scoring 0.0-1.0.
Conversational Q&A about analysis results using a local LLM (Qwen2.5:3b via Ollama, CPU-only). All data pseudonymized before reaching the model, then compacted with per-session ID aliases (Folder_9991607 → f12) to keep prompts fast. Full conversation history support.
Client names, folder names, and all PII are replaced with pseudonyms (Folder_XXXXXXX) before any data leaves the machine. Pseudonym mappings stored locally in SQLite.
Clean architecture with a Core project holding only interfaces and models. Host applications (WPF, API, Server, CLI) consume Core through DI. Satellite projects provide specific implementations.
Each host provides its name via IAppNameProvider. AppDataPaths uses it to isolate
SQLite databases per host, preventing concurrent-access conflicts when API and WPF run on the same machine.
The API exposes lightweight health probes for post-deploy verification. Each returns 200 when healthy and 503 with a diagnostic JSON payload when not.
| Endpoint | Checks | 503 hint when unhealthy |
|---|---|---|
GET /health/db | EF Core can connect to the training database | Connection error detail |
GET /health/llm | Python bridge ready, Ollama reachable, model pulled, test prompt round-trips | ollama pull qwen2.5:3b |
GET /health/logging | Log directory is writable (catches the IIS systemprofile path issue) | Real resolved path + error |
GET /health/server new | The DocTreeRetrainer Windows service is installed and Running (queried via ServiceController) | sc.exe create "DocTreeRetrainer" binPath=... |
/health/server works because the API and the retraining Server are installed on the same
machine, so the API queries the Windows Service Control Manager directly — no extra HTTP surface on the worker.
Two execution modes: Progressive (live UI updates, two-phase) and Performance (single-pass parallel). The orchestrator manages the full lifecycle including feedback application and training data submission.
Every analysis follows this six-stage pipeline. Data flows left to right; each stage feeds the next.
| Output | Description | Count (typical) |
|---|---|---|
| Folder Classifications | Each folder assigned a role (Client, Matter, DocumentCategory, etc.) with confidence 0.0-1.0 | ~1,800 folders |
| Anomalies | 10 types: SpellingError, CaseInconsistency, SemanticDuplicate, RoleMismatch, RedundantNesting, etc. | ~1,100 anomalies |
| Entity Extraction | Detected entities: Person, Corporation, Partnership with confidence scores | ~20 entities |
| Patterns | Naming conventions and structural patterns (Client > Matter > Category) | ~5 patterns |
| DMS Migration Plan | Generated migration mapping for NetDocuments | Per workspace |
All sensitive data is pseudonymized before leaving the user's machine. The pseudonym mapping is stored
locally in SQLite and never transmitted. The LLM only ever sees Folder_XXXXXXX.
Names are first reduced to non-identifying features, the key that could reverse a token never leaves the
device, and four privacy levels let each firm tune the privacy/utility trade-off.
The real folder name is read once by IFeatureExtractor to compute structural signals, then
the name itself is replaced with a token. The model learns from the features — never the text.
A token is meaningless on its own. The only thing that can reverse it — the
pseudonyms.db mapping — stays on the user's machine and is never transmitted.
Chosen per firm in Settings (PrivacyLevel). Higher privacy means less training signal.
Pseudonymized is the default.
| Data | Action | Reason |
|---|---|---|
| Folder names | Pseudonymized | Contains client names, personal info |
| Entity names | Pseudonymized | PII — person/company names |
| Free-text descriptions | Dropped entirely | May contain real names in context |
| Anomaly suggestions | Dropped | References original folder names |
| Folder role (Client, Matter...) | Preserved | Structural — no PII |
| Confidence scores | Preserved | Numeric — no PII |
| Depth, counts, severity | Preserved | Statistical — no PII |
| Anomaly type | Preserved | Enum value — no PII |
After each analysis, obfuscated training data is queued locally, then sent to the API. The Server periodically retrains ML models from accumulated submissions and uploads model bundles.
Users ask natural-language questions about their analysis results. The entire pipeline ensures no real client data ever reaches the LLM — all names are pseudonymized before transmission.
| Section | Always Sent | Content |
|---|---|---|
| Summary | Yes | Total folders, max depth, confidence breakdown, anomaly totals by type |
| Client Breakdown | Yes | Per-client rollup: subfolder count + anomaly count with type breakdown |
| Anomalies | Yes | Grouped by type with total counts, 5 distinct examples per type |
| Classifications | Yes | All folders for small roles (Client, Matter), top 15 for large roles |
| Entities | Yes | Top 20 extracted entities with type and confidence |
| Tree Structure | On demand | Depth distribution, leaf folder count |
| Patterns | On demand | Detected naming conventions with prevalence |
Each section is then bound to a safety-net cap of 40 lines / 2500 chars by the
ContextSectionTrimmer, and every pseudonym is swapped for a short per-session alias
(Folder_9991607 → f12) before it is sent — so far more real data fits in the same token budget.
We don't have a powerful LLM — so we keep the prompt small.
The model runs on a modest CPU-only server (6 vCPU, 12 GB RAM, no GPU) with a small 3B-parameter model. A large prompt is the bottleneck: the time to process the input (prefill) grows with its size, and on this hardware a full analysis dumped into the prompt took ~7 minutes per question.
Our mitigations:
PromptAliasMap swaps each long pseudonym for a tiny code (Folder_9991607 → f12), ~5 tokens down to ~1. A 60–70% cut on every identifier, so far more real data fits in the same budget instead of being thrown away.qwen2.5:3b instead of a 7B model that the CPU can't keep up with.
Aliasing replaced trimming as the main size lever. Aliases are assigned per chat session and
reversed on the way back (ToPseudonym) before pseudonyms are un-masked to real names —
so the user still sees real names while the model only ever saw f12. The Python
SYSTEM_PREAMBLE was updated to tell the model how to read the codes.
Folder_9991607 costs ~5 tokens — almost all of it wasted on random digits. PromptAliasMap assigns each pseudonym a tiny per-session code (f12, c3, m5) just for the LLM round-trip, ~1 token each — a 60–70% cut per identifier. The map is reversed on the response (ToPseudonym) before un-masking, so it's invisible to the user; unknown codes the model invents are left untouched. This let us relax the trimmer from 12 lines / 800 chars to 40 / 2500 (now just a safety net). The Python SYSTEM_PREAMBLE was updated to tell the model how to read the codes.Two database systems: SQL Server (API/Server — EF Core) for centralized training data, and local SQLite databases (WPF/CLI) for pseudonyms, submission queue, and analysis history.
Centralized on the API server. Stores all anonymized training submissions received from clients.
Used by the Server's RetrainingService to retrain ML models periodically.
Parent table. One row per analysis run submitted by any client.
| Column | Type | Description |
|---|---|---|
| Id | int (PK) | Auto-increment primary key |
| SubmissionId | string | Unique GUID per analysis run |
| ClientId | string | Anonymous client identifier (persisted locally) |
| SchemaVersion | int | Submission format version for backward compatibility |
| SubmittedAt | DateTime | When the client ran the analysis |
| ReceivedAt | DateTime | When the API received it |
| AppVersion | string | Client app version (e.g. "v7.0") |
| TotalFolders | int | Number of folders in the analyzed tree |
| MaxDepth | int | Maximum nesting depth |
One row per classified folder. Contains ML features for retraining. FK → TrainingSubmissions.
| Column | Type | Description |
|---|---|---|
| Id | int (PK) | Auto-increment |
| SubmissionId | int (FK) | Parent submission |
| NamePseudonym | string | Folder pseudonym (e.g. "Folder_9991607") |
| ParentNamePseudonym | string | Parent folder pseudonym |
| Depth | float | Nesting level in tree |
| ChildCount | float | Number of direct children |
| SiblingCount | float | Number of siblings at same level |
| HasDraftsSubfolder | bool | Feature: has a "Drafts" child |
| ContainsPersonName | bool | Feature: name matches person pattern |
| ContainsCorpIdentifier | bool | Feature: name contains Inc/Ltd/Corp/LLC |
| MatchesLegalCategory | bool | Feature: name is a known legal category |
| NameLength | float | Feature: character count of folder name |
| IsAllCaps | bool | Feature: name is all uppercase |
| IsLeaf | bool | Feature: has no children |
| PredictedRole | string | Classification result (Client, Matter, etc.) |
| Confidence | float | Classification confidence 0.0-1.0 |
User feedback on detected anomalies. Trains severity and auto-dismiss models. FK → TrainingSubmissions.
| Column | Type | Description |
|---|---|---|
| Id | int (PK) | Auto-increment |
| SubmissionId | int (FK) | Parent submission |
| AnomalyType | int | Enum: SpellingError=0, CaseInconsistency=1, ... |
| Severity | float | Severity score 0.0-1.0 |
| IsValidated | bool | Whether user confirmed the anomaly |
| Resolution | string? | User action: "Consolidate", "Dismissed", etc. |
| FolderDepth | int | Context feature for ML |
| FolderRole | int | Context feature for ML |
| FolderConfidence | float | Context feature for ML |
| FolderNameLength | int | Context feature for ML |
Extracted entities with features for entity-type classifier retraining. FK → TrainingSubmissions.
| Column | Type | Description |
|---|---|---|
| Id | int (PK) | Auto-increment |
| SubmissionId | int (FK) | Parent submission |
| NamePseudonym | string | Entity pseudonym |
| DetectedType | int | Enum: Person, Corporation, Partnership, ... |
| Confidence | float | Detection confidence |
| ContainsComma | bool | Feature: "Last, First" pattern |
| ContainsCorpIdentifier | bool | Feature: Inc/Ltd/Corp |
| ContainsPersonName | bool | Feature: person name pattern |
| ContainsTrustKeyword | bool | Feature: "Trust", "Estate" |
| ContainsPartnershipKeyword | bool | Feature: "LLP", "Partners" |
| ContainsGovernmentKeyword | bool | Feature: "Ministry", "Crown" |
| NameLength / WordCount / IsAllCaps | float/bool | Additional features |
Semantic duplicate pairs with user validation. Trains the duplicate detector. FK → TrainingSubmissions.
| Column | Type | Description |
|---|---|---|
| Id | int (PK) | Auto-increment |
| SubmissionId | int (FK) | Parent submission |
| NamePseudonymA / B | string | The two folder pseudonyms in the pair |
| NameSimilarity | float | String similarity score |
| LengthDifference | float | Absolute length difference |
| SharePrefix / ShareSuffix | bool | Whether names share a common prefix/suffix |
| IsDuplicate | bool | User-confirmed: are they actually duplicates? |
| Resolution | string? | User action taken |
Retrained ML model bundles stored as binary BLOBs. Only one is active at a time.
| Column | Type | Description |
|---|---|---|
| Id | int (PK) | Auto-increment |
| Version | int | Sequential version number |
| CreatedAt | DateTime | When the bundle was generated |
| AppVersion | string | Server version that created it |
| FileName | string | Original ZIP filename |
| FileData | byte[] | Complete ZIP bundle (BLOB) |
| TrainingSubmissionCount | int | How many submissions were used to train |
| ETag | string | HTTP ETag for client caching |
| IsActive | bool | Whether this is the current active bundle |
Stored on the user's machine under %LocalAppData%\DocTreeAnalyzer\{host}\.
Each host (WPF, API, CLI) gets its own isolated copy.
Maps real names to pseudonyms. Deterministic: same input always gets same pseudonym across sessions.
| Column | Type | Description |
|---|---|---|
| id | INTEGER (PK) | Auto-increment |
| real_value | TEXT | Original name (e.g. "FTI Professional Grade Inc") |
| pseudonym | TEXT (UNIQUE) | Generated pseudonym (e.g. "Folder_9991607") |
| type | TEXT | Category: "Folder", "Person", "Entity" |
| first_seen | TEXT | ISO timestamp of first mapping |
Offline queue for training data. Persists submissions until successfully sent to the API.
| Column | Type | Description |
|---|---|---|
| submission_id | TEXT (PK) | Unique submission GUID |
| payload_json | TEXT | Full serialized ObfuscatedTrainingSubmission |
| created_at | TEXT | When the submission was enqueued |
| retry_count | INTEGER | Number of failed send attempts |
All local files are organized under %LocalAppData%\DocTreeAnalyzer\{host}\.
Host isolation ensures the API and WPF never touch each other's files.
Root: %LocalAppData%\DocTreeAnalyzer\ — each host gets its own subdirectory
| File | Format | Created By | Description |
|---|---|---|---|
| pseudonyms.db | SQLite | SqlitePseudonymService | Stores real_value ↔ pseudonym mappings. Deterministic: "ACME Corp" always becomes "Folder_9991607". Used for both masking (before sending to LLM/API) and unmasking (when displaying LLM responses). Each host has its own copy to prevent SQLite locking conflicts. |
| submissions.db | SQLite | SubmissionQueue | Offline-first queue for training data submissions. When the API is unreachable, submissions persist here and retry on next DrainAsync(). Max 10 retries, max age 30 days. Automatically cleaned up after successful send. |
| analysis.db | SQLite | AnalysisRepository | Stores complete analysis sessions for the history feature. Each session includes the full tree, classifications, anomalies, entities, and patterns. Users can reload any previous analysis from the history panel. |
| universal.dtamodel | ZIP | ModelUpdater | Cached ML model bundle downloaded from the API. Contains 5 ML.NET model files: folder-classifier.zip, semantic-duplicate.zip, entity-type.zip, anomaly-severity.zip, anomaly-dismiss.zip, plus a manifest.json with version info. Updated via ETag-based conditional GET. |
| model-etag.txt | Text | FileModelUpdateSettings | Single-line file storing the HTTP ETag of the last downloaded model bundle. ModelUpdater sends this in If-None-Match header — if the server returns 304, the cached bundle is still current and no download is needed. |
| projects.json | JSON | ProjectRepository | Index of all named projects. Each project has an ID, name, source file path, and creation date. Projects organize analyses by client engagement or matter. |
| projects\default.db | SQLite | ProjectRepository | Fallback project database used when no named project is selected. Stores analysis sessions that aren't assigned to a specific project. |
| Logs\log_YYYYMMDD.txt | Text | FileLogger | Daily rotating log files. Format: [timestamp] [LEVEL] message. Exceptions include type, message, stack trace, and inner exception. Thread-safe via SemaphoreSlim. One file per day per host. |
| python.log | Text | chat_engine.py | Python-side diagnostic log (API host only). Records LLM provider creation, model loading, chat calls, response lengths, and errors. DEBUG level to file, INFO to console. Silences noisy httpcore/httpx logs. |
| {name}.feedback.json | JSON | FeedbackStore | Saved alongside the source tree file (not in AppData). Stores user's active-learning answers (role overrides, anomaly validations). Re-applied when the same file is re-analyzed to preserve user feedback. |
A single ZIP archive containing all 5 retrained ML.NET models plus a manifest. Downloaded via conditional GET (ETag).
.NET 10 C# 13 Python 3.12
WPF CommunityToolkit.Mvvm MVVM Pattern
ASP.NET Minimal APIs Swagger/OpenAPI
ML.NET Qwen2.5:3b Ollama pythonnet
SQLite EF Core SQL Server
MSTest Moq 196 Tests
| Package | Project | Purpose |
|---|---|---|
| Microsoft.Extensions.DependencyInjection.Abstractions | Core | Only NuGet in Core — DI registration |
| Microsoft.Data.Sqlite | Sqlite | Pseudonym DB + Submission Queue |
| Microsoft.ML | ML | Folder classifier training/inference |
| pythonnet | Api | C# ↔ Python bridge for LLM |
| Microsoft.EntityFrameworkCore.Sqlite | Database | Training data persistence |
| CommunityToolkit.Mvvm | Wpf | MVVM source generators |
| ollama (pip) | PythonAi | Ollama client for LLM inference |