Self-Hosted LLM Chatbot for Insurance Integration Process

Project Overview

We partnered with an innovative insurance provider to build a fully self-hosted AI assistant for claims processing and customer onboarding. The client operated under strict data protection requirements — customer data including health information, financial details, and personal identifiers could not leave their infrastructure under any circumstances. Cloud-hosted LLM APIs were categorically excluded. Our mandate was to deliver production-grade AI capabilities with complete data sovereignty, GDPR Article 28 processor compliance, and audit-ready documentation.

Key Challenges

• Absolute Data Sovereignty: Insurance claims contain special category data under GDPR (health information) and financial PII. No data could be transmitted to external APIs — requiring fully on-premises model inference with no external dependencies.
• Regulatory Explainability: FCA conduct rules require that customer-facing communications are clear, fair, and not misleading. AI-generated responses needed documented reasoning, source citations, and human oversight for edge cases.
• High Manual Workload: Claims handlers spent 80% of their time on repetitive document extraction, data entry, and standard response generation — creating processing backlogs and customer frustration.
• Integration Complexity: The solution needed to integrate with legacy CRM, document management, and policy administration systems — many with limited API capabilities.

Solution Architecture

We deployed a fully air-gapped LLM infrastructure within the client's data centre, with zero external network dependencies during inference:

• On-Premises Model Serving: Deployed Llama 3 70B on dedicated GPU infrastructure using vLLM for high-throughput inference. Model weights stored encrypted (AES-256) with offline update capability via signed model packages.
• Insurance-Specific Fine-Tuning: Fine-tuned on 25,000+ anonymised claim interactions, policy documents, and approved response templates. Training conducted on isolated infrastructure with synthetic data augmentation.
• RAG with Policy Grounding: Retrieval-augmented generation using indexed policy documents, terms & conditions, and approved response templates. Every response includes source citations for audit traceability.
• Human-in-the-Loop Governance: Confidence thresholds trigger automatic escalation. Complex claims, disputed amounts, and edge cases route to human handlers with AI-prepared context summaries.
• CRM Integration: Bidirectional sync with existing CRM via secure internal APIs. Chatbot extracts structured data (policy numbers, claim types, customer details) and updates records with full change logging.

Security & Compliance Architecture

• Complete Data Sovereignty: Zero external API calls during operation. All inference, embedding generation, and retrieval happens within the client's network perimeter. No customer data ever leaves the premises.
• GDPR Processor Compliance: Documented data processing activities per Article 30. PII detection and handling policies. Data retention controls with automated purging. Subject access request (SAR) workflow integration.
• Encryption Throughout: TLS 1.3 for all internal service communication. AES-256 encryption for data at rest including conversation logs and model weights. Database-level encryption with customer-managed keys.
• Audit Trail: Every interaction logged with: timestamp, user context, input hash, model version, retrieved documents, generated response, confidence score, and any human interventions. Logs retained for 7 years per insurance record-keeping requirements.
• Access Control: LDAP integration with existing identity management. Role-based access (Claims Handler, Supervisor, Admin, Auditor). All access logged and reviewable.
• Prompt Injection Prevention: Input sanitisation, output filtering, and system prompt protection. Regular adversarial testing against known attack vectors.

Technologies Used

• Llama 3 70B + vLLM: Open-source foundation model with optimised inference serving. PagedAttention for efficient memory utilisation. Throughput: ~640 tokens/second on 1x A100 GPU at 100 concurrent users.
• Unsloth: Parameter-efficient fine-tuning framework delivering 2x faster training with 70% less VRAM usage on insurance-specific data. Training completed on isolated GPU cluster with no external connectivity.
• FastAPI + Pydantic: Type-safe API layer with request validation, structured error handling, and OpenAPI documentation for internal developers.
• PostgreSQL 16 + pgvectorscale: Hybrid search combining structured policy queries with vector similarity for document retrieval. Row-level security for multi-tenant data isolation.
• Agentic Framework: Lightweight orchestration layer with minimal overhead and OpenAI-compatible APIs, enabling seamless future integrations with different tools and services. Persistent state management, conditional routing, and built-in checkpointing for audit reconstruction with custom nodes handling approval workflows and human-in-the-loop escalation.
• Neo4j: Graph database modelling policy relationships, claim dependencies, and customer journey mapping for contextual response generation.

Results

• 50% Reduction in Manual Processing: Claims handlers now focus on complex cases and customer relationships. First-response time reduced from 24 hours to under 2 hours for standard enquiries.
• 100% Data Sovereignty: Zero customer data transmitted externally. Successfully passed client's internal security audit and third-party penetration testing.
• Audit-Ready Documentation: Every AI interaction fully traceable with source citations, confidence scores, and human oversight records — satisfying both internal compliance and regulatory requirements.
• 94% Response Accuracy: Measured against human-validated golden dataset of 2,000 claim scenarios. Continuous evaluation maintains quality with automated alerts on regression.
• Investment Milestone: The demonstrable AI capability and compliance posture contributed to the client securing Series A funding, with investors citing the technology platform as a key differentiator.

Scalability & Roadmap

The architecture is designed for expansion as the client's business grows:

• Multi-Product Expansion: The same infrastructure now supports motor, home, and pet insurance products with product-specific fine-tuning and retrieval namespaces.
• Claims Validation Pipeline: Phase 2 deployment adds AI-powered document verification, coverage checking, and preliminary risk assessment — with human approval gates for final decisions.
• Fraud Detection Integration: Planned integration with anomaly detection models to flag suspicious patterns for investigation, maintaining human-in-the-loop for all escalations.
• Model Evolution: Architecture supports model upgrades (Llama 4, Mistral Large) with A/B testing, evaluation comparison, and controlled rollout — all without external dependencies.

Why NodeNova

This project exemplifies our approach to AI engineering in regulated industries: we don't compromise on data sovereignty, security, or compliance to deliver AI capabilities. Our expertise in on-premises LLM deployment, fine-tuning, and production-grade infrastructure means organisations can modernise with AI while maintaining complete control over their data and meeting their regulatory obligations.

• Data Sovereignty Expertise: Proven experience deploying LLMs in air-gapped and highly restricted environments.
• Regulatory Understanding: We design for audit requirements, explainability, and human oversight from the start — not as an afterthought.
• Production Engineering: Not prototypes — production systems with observability, evaluation pipelines, and documented incident response.