How to Design an Enterprise Document Management System
Building a Production-Grade SharePoint/Google Drive for Enterprise — Storage, Versioning, Search, Workflows, Compliance
1. System Overview and Requirements
What is an Enterprise Document Management System?
An Enterprise Document Management System (DMS) is a centralized platform for storing, organizing, managing, securing, and retrieving organizational documents throughout their lifecycle. Think SharePoint, Google Drive for Enterprise, Box, or OpenText — but designed for regulated industries that demand versioning, audit trails, retention policies, and granular access control.
This is not a simple file server. A production DMS must handle millions of documents across thousands of users, support concurrent editing, enforce compliance, provide full-text search across terabytes of content, integrate with enterprise identity providers, and maintain an unbreakable audit trail. Every design decision must account for security, scalability, and regulatory requirements simultaneously.
Functional Requirements
- Document CRUD: Upload, download, view, and delete documents with support for 500+ file types
- Folder Hierarchy: Nested folder structures with libraries, sites, and workspaces
- Versioning: Major and minor version tracking with check-in/check-out to prevent edit conflicts
- Content Types: Define custom metadata schemas per document type (contracts, invoices, SOPs)
- Full-Text Search: Search across document content, metadata, and file names with sub-second latency
- Document Preview: Render PDFs, Office documents, images, and videos in-browser without download
- Co-Authoring: Multiple users editing the same document simultaneously with operational transforms
- Workflows: Configurable approval, review, and electronic signature pipelines
- Access Control: ACL-based permissions with role inheritance and external sharing links
- Retention Policies: Time-based archival and deletion with legal hold capabilities
- Sensitivity Labels: Classification labels (Public, Internal, Confidential, Restricted) with enforcement
- Audit Trail: Immutable log of every access, modification, share, and deletion event
- Templates: Predefined document templates with auto-populated metadata fields
- Metadata Views: List, gallery, calendar, and board views driven by document metadata
- OCR: Extract text from scanned PDFs, images, and photographs
- Barcode/QR Scanning: Extract and index barcode data for document categorization
- Records Management: Declare records, apply legal holds, and support eDiscovery exports
- Content Deduplication: Block-level deduplication to reduce storage costs
- Large File Support: Chunked, resumable uploads for files up to 250 GB
- Template Generation: Create new documents from templates with dynamic field population
Non-Functional Requirements
| Attribute | Target | Rationale |
|---|---|---|
| Availability | 99.99% (52 min downtime/year) | Enterprise-critical; legal teams depend on document access |
| Latency | <200ms for metadata ops, <2s for search | Users expect snappy file browsing |
| Throughput | 10,000+ uploads/min, 50,000+ downloads/min | Peak usage during month-end filing |
| Storage Scale | 10 PB+ across all tenants | Enterprise customers accumulate petabytes over decades |
| Document Count | 500M+ documents per large tenant | Legal and healthcare orgs have billions of files |
| Durability | 99.999999999% (11 nines) | Document loss is catastrophic for compliance |
| Consistency | Strong for metadata, eventual for search index | Metadata must be immediately consistent; search can lag |
| Security | SOC 2 Type II, GDPR, HIPAA, FedRAMP | Enterprise compliance is non-negotiable |
Capacity Estimation
Assume 10,000 enterprises with an average of 5,000 users each. That is 50 million total users. Average document size: 2 MB. Average documents per user: 200. Total documents: 10 billion. Total storage: 20 PB.
Daily uploads: 50 million users x 5 documents/day = 250 million uploads/day. Peak upload rate: ~5,000 per second. Daily downloads (view + edit): 250 million x 10 = 2.5 billion downloads/day. Peak download rate: ~30,000 per second.
Search index size: assuming 1 KB metadata + 50 KB extracted text per document = 51 KB x 10 billion = 510 TB of index data.
2. High-Level Architecture
React SPA"] MOB["Mobile Apps
iOS / Android"] DESKTOP["Desktop Sync
Windows / macOS"] API_CLIENT["Third-Party
API Consumers"] end subgraph Gateway["API Gateway & Auth"] GW["API Gateway
Rate Limiting, Routing"] AUTH["Identity Provider
Azure AD / Okta / SAML"] RBAC["Authorization Service
ACL Engine"] end subgraph Core["Core Services"] DOC["Document Service
CRUD, Metadata"] VER["Versioning Service
Major/Minor, Check-In/Out"] SEARCH_SVC["Search Service
Elasticsearch Cluster"] PREVIEW_SVC["Preview Service
Rendering Pipeline"] WORKFLOW_SVC["Workflow Engine
Approval, Review, Signature"] COLLAB_SVC["Collaboration Service
OT / CRDT Co-Authoring"] RETENTION_SVC["Retention Service
Policies, Archival"] CLASSIFY_SVC["Classification Service
Sensitivity Labels"] AUDIT_SVC["Audit Service
Event Logging"] TEMPLATE_SVC["Template Service
Document Generation"] VIEW_SVC["View Service
Metadata-Driven Views"] DEDUP_SVC["Deduplication Service
Block-Level Dedup"] OCR_SVC["OCR Service
Text Extraction"] BARCODE_SVC["Barcode Service
Scanning, Indexing"] end subgraph Storage["Storage Layer"] BLOB["Blob Storage
Azure Blob / S3"] DB["Metadata DB
Cosmos DB / PostgreSQL"] CACHE_SVC["Cache Layer
Redis Cluster"] QUEUE["Message Queue
Kafka / Service Bus"] GRAPH["Graph DB
Neo4j (Permissions)"] end subgraph Pipeline["Async Pipelines"] INDEX_P["Indexing Pipeline
Content Extraction"] THUMB_P["Thumbnail Pipeline
Preview Generation"] SCAN_P["Malware Scan
ClamAV / Defender"] DLP_P["DLP Pipeline
Data Loss Prevention"] end WEB --> GW MOB --> GW DESKTOP --> GW API_CLIENT --> GW GW --> AUTH GW --> RBAC GW --> DOC GW --> SEARCH_SVC GW --> PREVIEW_SVC GW --> WORKFLOW_SVC GW --> COLLAB_SVC GW --> RETENTION_SVC GW --> CLASSIFY_SVC GW --> AUDIT_SVC GW --> TEMPLATE_SVC GW --> VIEW_SVC GW --> OCR_SVC GW --> BARCODE_SVC DOC --> BLOB DOC --> DB DOC --> CACHE_SVC DOC --> QUEUE VER --> DB VER --> BLOB SEARCH_SVC --> DB RBAC --> GRAPH WORKFLOW_SVC --> QUEUE QUEUE --> INDEX_P QUEUE --> THUMB_P QUEUE --> SCAN_P QUEUE --> DLP_P INDEX_P --> SEARCH_SVC THUMB_P --> BLOB
Service Decomposition
The architecture follows a service-oriented design where each major capability is an independently deployable service. The Document Service is the central orchestrator — it handles CRUD operations, delegates to blob storage for binary content, and coordinates metadata in the database. All other services subscribe to events from the Document Service via Kafka topics.
The API Gateway is the single entry point. It handles authentication via JWT tokens validated against the enterprise's identity provider, enforces rate limits per tenant, and routes requests to the appropriate backend service. We use a claims-based authorization model where the token carries role and permission claims.
Data Flow for Upload
- Client initiates upload via
POST /api/v1/documents/uploadwith file stream and metadata JSON - API Gateway authenticates the request and validates the JWT token
- Document Service checks quotas and permissions
- File stream is written to a temp blob location via chunked upload
- Malware scan is triggered asynchronously via Kafka
- Content is hashed for deduplication; if duplicate exists, reference is created instead of new blob
- Metadata record is created in the database with
PENDING_SCANstatus - Indexing pipeline extracts text content and indexes it in Elasticsearch
- Thumbnail pipeline generates preview images
- Document status transitions to
ACTIVEand becomes searchable
3. Document Storage — Blobs and Metadata
Two-Tier Storage Architecture
Every document in the system is split into two components: the blob (the actual file binary) and the metadata record (the descriptive information about the file). These are stored in fundamentally different systems optimized for their respective access patterns.
Blob Storage
We use Azure Blob Storage (or S3 on AWS) organized into containers with a hierarchical namespace. The blob key follows a deterministic path structure:
plaintext
/{tenant_id}/{library_id}/{year}/{month}/{day}/{document_id}/{version_number}/{filename}
Example:
/acme-corp/contracts-lib/2026/07/12/doc-8f3a-92bc/v2/Service-Agreement-v2.pdf
This path structure provides several benefits: tenant isolation at the storage level, temporal locality for archival operations, and deterministic paths that can be regenerated from metadata alone. We use the document ID (a UUID) rather than the filename to avoid collisions and path traversal issues.
Metadata Storage
Metadata is stored in a globally-distributed database like Cosmos DB (multi-master) or a sharded PostgreSQL cluster. The core document metadata record contains:
C#
public class DocumentMetadata
{
public Guid DocumentId { get; set; }
public Guid TenantId { get; set; }
public Guid LibraryId { get; set; }
public Guid? ParentFolderId { get; set; }
public string FileName { get; set; }
public string ContentType { get; set; } // MIME type
public long FileSize { get; set; }
public string BlobPath { get; set; }
public string ContentHash { get; set; } // SHA-256 for dedup
public int MajorVersion { get; set; }
public int MinorVersion { get; set; }
public DocumentStatus Status { get; set; }
public Guid CreatedBy { get; set; }
public DateTime CreatedAt { get; set; }
public Guid ModifiedBy { get; set; }
public DateTime ModifiedAt { get; set; }
public Guid? CheckedOutBy { get; set; }
public DateTime? CheckedOutAt { get; set; }
public Guid ContentTypeId { get; set; }
public Guid? RetentionPolicyId { get; set; }
public SensitivityLabel Sensitivity { get; set; }
public bool IsDeleted { get; set; }
public DateTime? DeletedAt { get; set; }
public string ExternalId { get; set; } // For barcode/QR mapping
}
public enum DocumentStatus
{
PendingScan,
Active,
CheckedOut,
Archived,
OnHold,
Deleted
}
public enum SensitivityLabel
{
Public,
Internal,
Confidential,
Restricted
}
The metadata database uses a composite partition key of (TenantId, LibraryId) to ensure all documents within a library are colocated for efficient range queries. The primary index is on DocumentId, with secondary indexes on (TenantId, ModifiedAt), (TenantId, ContentTypeId), and (ParentFolderId).
Blob Storage Tiers
| Tier | Access Pattern | Latency | Cost (per GB/month) | Use Case |
|---|---|---|---|---|
| Hot | Frequent read/write | <10ms | $0.018 | Active documents, recent uploads |
| Cool | Infrequent access | <100ms | $0.010 | Documents older than 90 days |
| Cold | Rare access | <150ms | $0.0045 | Archived documents, retention period |
| Archive | Compliance only | Hours | $0.001 | Legal hold, regulatory retention |
Write Path — How Uploads Work
For files under 100 MB, we use a single PUT operation to blob storage. For larger files, the upload is chunked into 4 MB blocks. Each block is uploaded independently (and can be retried), then committed atomically. The upload session is tracked in Redis with a TTL of 24 hours, allowing resumable uploads if the client disconnects.
C#
public class BlobStorageService
{
private readonly IBlobContainer _container;
private readonly IDeduplicationService _dedup;
public async Task<BlobUploadResult> UploadDocumentAsync(
Stream fileStream, DocumentMetadata metadata)
{
var contentHash = await ComputeSha256Async(fileStream);
fileStream.Position = 0;
var existingBlob = await _dedup.FindByHashAsync(
metadata.TenantId, contentHash);
if (existingBlob != null)
{
return new BlobUploadResult
{
BlobPath = existingBlob.BlobPath,
IsDeduplicated = true,
StorageSaved = metadata.FileSize
};
}
var tier = DetermineStorageTier(metadata);
var blobPath = GenerateBlobPath(metadata);
var accessTier = tier switch
{
StorageTier.Hot => AccessTier.Hot,
StorageTier.Cool => AccessTier.Cool,
_ => AccessTier.Hot
};
if (metadata.FileSize < 100 * 1024 * 1024)
{
await _container.UploadAsync(
blobPath, fileStream, accessTier);
}
else
{
await _container.ChunkedUploadAsync(
blobPath, fileStream,
chunkSize: 4 * 1024 * 1024,
accessTier);
}
return new BlobUploadResult
{
BlobPath = blobPath,
IsDeduplicated = false,
StorageSaved = 0
};
}
}
4. Folder and Library Hierarchy
The Entity Hierarchy
Enterprise DMS platforms organize content in a strict hierarchy: Tenant > Site > Library > Folder > Document. Each level has its own access control, settings, and metadata schemas.
(Enterprise Account)
acme-corp"] --> SITE1["Site: Legal"] TENANT --> SITE2["Site: Engineering"] TENANT --> SITE3["Site: Human Resources"] SITE1 --> LIB1["Library: Contracts"] SITE1 --> LIB2["Library: NDAs"] SITE1 --> LIB3["Library: Litigation"] LIB1 --> F1["Folder: 2026"] F1 --> F2["Folder: Q1"] F1 --> F3["Folder: Q2"] F2 --> D1["Document: Service-Agreement.pdf"] F2 --> D2["Document: SOW.docx"] LIB2 --> F4["Folder: Active"] LIB2 --> F5["Folder: Expired"]
Entity Model
C#
public class Tenant
{
public Guid TenantId { get; set; }
public string Name { get; set; }
public string Domain { get; set; }
public StorageQuota Quota { get; set; }
public TenantSettings Settings { get; set; }
public List<Site> Sites { get; set; }
}
public class Site
{
public Guid SiteId { get; set; }
public Guid TenantId { get; set; }
public string Name { get; set; }
public string Description { get; set; }
public Guid OwnerUserId { get; set; }
public List<DocumentLibrary> Libraries { get; set; }
}
public class DocumentLibrary
{
public Guid LibraryId { get; set; }
public Guid SiteId { get; set; }
public string Name { get; set; }
public Guid DefaultContentTypeId { get; set; }
public VersioningPolicy VersioningPolicy { get; set; }
public List<ContentType> AllowedContentTypes { get; set; }
public List<Folder> RootFolders { get; set; }
}
public class Folder
{
public Guid FolderId { get; set; }
public Guid? ParentFolderId { get; set; } // null = root
public Guid LibraryId { get; set; }
public string Name { get; set; }
public string Path { get; set; } // Materialized path
public int Depth { get; set; }
}
Materialized Path for Fast Queries
Every folder stores a materialized path — a denormalized string containing the full hierarchy from root to the current folder. For example: /contracts/2026/Q1/. This enables efficient subtree queries without recursive CTEs. When moving a folder, we update the materialized path of all descendants in a single batch operation.
SQL
SELECT d.* FROM Documents d
JOIN Folders f ON d.ParentFolderId = f.FolderId
WHERE f.MaterializedPath LIKE '/contracts/2026/%'
AND f.TenantId = @tenantId
AND d.IsDeleted = 0
ORDER BY d.FileName;
Site Templates and Provisioning
When a new enterprise tenant is created, we provision a set of default sites (Legal, Engineering, HR, Finance) each with preconfigured libraries, content types, and retention policies. The provisioning process runs as an asynchronous workflow that creates all database records, configures blob containers, sets up search indexes, and applies initial ACLs.
5. File Versioning — Major/Minor and Check-In/Check-Out
Versioning Model
Enterprise DMS systems use a dual version numbering scheme: major versions (1.0, 2.0, 3.0) for published/approved documents and minor versions (1.1, 1.2, 2.1) for drafts and working copies. This mirrors how legal and compliance teams think about document lifecycle — a contract in negotiation might be at version 2.3 (draft 2, revision 3) before being finalized as version 3.0.
| Version Type | Example | Visibility | Use Case |
|---|---|---|---|
| Major (published) | 1.0, 2.0, 3.0 | All users with read access | Approved, finalized documents |
| Minor (draft) | 1.1, 1.2, 2.1 | Only author + edit permissions | Work in progress, revisions |
| Check-out draft | 2.0 (locked) | Only the person who checked out | Exclusive editing to prevent conflicts |
Version Storage
When a new version is uploaded, we create a new version record in the database pointing to the new blob. Previous version blobs are NOT deleted — they remain in storage for the lifetime of the document (and beyond, if retention policies require it). This means version 1.0 and version 2.0 are both fully accessible at any time.
C#
public class DocumentVersion
{
public Guid VersionId { get; set; }
public Guid DocumentId { get; set; }
public int MajorVersion { get; set; }
public int MinorVersion { get; set; }
public string BlobPath { get; set; }
public long FileSize { get; set; }
public string ContentHash { get; set; }
public Guid CreatedBy { get; set; }
public DateTime CreatedAt { get; set; }
public string Comment { get; set; }
public VersionStatus Status { get; set; }
}
public enum VersionStatus
{
Draft,
PendingApproval,
Published,
Archived,
Superseded
}
Check-In / Check-Out Workflow
Check-out provides pessimistic locking for documents that cannot tolerate concurrent edits. When a user checks out a document, a lock record is created and the document becomes exclusively editable by that user. Other users see a "Checked out by [user]" badge and can only view the last published version.
Version Cleanup Policies
Keeping every version forever is impractical at scale. We implement configurable version limits per library:
- Keep last N major versions: Default is 50. Older major versions are soft-deleted.
- Keep last N minor versions per major: Default is 10.
- Expiration: Versions older than N days are purged (except published versions protected by retention policy).
- Legal hold override: Documents under legal hold retain ALL versions regardless of cleanup policy.
C#
public class VersionCleanupService
{
public async Task CleanupVersionsAsync(Guid libraryId)
{
var policy = await _db.GetVersioningPolicyAsync(libraryId);
var libraries = await _db.GetDocumentsInLibraryAsync(libraryId);
foreach (var doc in libraries)
{
var versions = await _db.GetVersionsAsync(doc.DocumentId);
if (doc.RetentionPolicy?.IsOnHold == true) continue;
var publishedVersions = versions
.Where(v => v.Status == VersionStatus.Published)
.OrderByDescending(v => v.MajorVersion)
.ToList();
var versionsToKeep = publishedVersions
.Take(policy.MajorVersionsToKeep)
.Select(v => v.MajorVersion)
.ToHashSet();
var versionsToArchive = versions
.Where(v => !versionsToKeep.Contains(v.MajorVersion))
.ToList();
foreach (var version in versionsToArchive)
{
await _blob.TierBlobAsync(version.BlobPath, BlobTier.Archive);
version.Status = VersionStatus.Superseded;
await _db.UpdateVersionAsync(version);
}
}
}
}
6. Content Types and Metadata Schemas
What Are Content Types?
Content types are the mechanism that makes a DMS more than a glorified file server. A content type defines a template for a class of documents — its required metadata fields, validation rules, default workflows, retention policies, and allowed templates. For example, the "Contract" content type might require fields like Contract Number, Party Name, Effective Date, Expiration Date, and Contract Value.
C#
public class ContentType
{
public Guid ContentTypeId { get; set; }
public Guid TenantId { get; set; }
public string Name { get; set; }
public string Description { get; set; }
public string IconUrl { get; set; }
public Guid? ParentContentTypeId { get; set; }
public List<MetadataField> MetadataFields { get; set; }
public List<WorkflowDefinition> DefaultWorkflows { get; set; }
public RetentionPolicy DefaultRetention { get; set; }
public List<DocumentTemplate> Templates { get; set; }
}
public class MetadataField
{
public Guid FieldId { get; set; }
public string InternalName { get; set; }
public string DisplayName { get; set; }
public FieldType Type { get; set; }
public bool IsRequired { get; set; }
public bool IsIndexed { get; set; }
public bool IsSearchable { get; set; }
public string DefaultValue { get; set; }
public List<ChoiceOption> Choices { get; set; }
public FieldValidation Validation { get; set; }
}
public enum FieldType
{
Text, MultiLineText, Number, Currency, DateTime,
Choice, MultiChoice, YesNo, Person, Hyperlink,
Calculated, ManagedMetadata
}
Example Content Types
| Content Type | Required Fields | Workflow | Retention |
|---|---|---|---|
| Contract | Contract#, Party, Start Date, End Date, Value, Signatory | Legal Review > Manager Approval > eSignature | 10 years after expiration |
| Invoice | Invoice#, Vendor, Amount, Due Date, PO Reference | AP Review > Manager Approval > Payment | 7 years (tax requirement) |
| Employee Record | Employee ID, Name, Department, Start Date, Position | HR Review > Manager Acknowledgment | 7 years after termination |
| SOP | SOP#, Department, Effective Date, Review Cycle | Peer Review > Quality Approval > Publish | Permanent (regulatory) |
| Meeting Minutes | Meeting Date, Attendees, Action Items | Auto-archive after 30 days | 3 years |
Content Type Inheritance
Content types support inheritance. A "Master Service Agreement" content type inherits from "Contract" and adds fields specific to MSAs. This allows organizations to define a base set of metadata fields and progressively specialize them. The inheritance tree is stored as an adjacency list in the database, and the full field set for a content type is computed by traversing the tree and merging fields.
Managed Metadata and Taxonomy
Enterprises often maintain a centralized taxonomy — a hierarchy of terms like "Legal > Contracts > Vendor Agreements" or "Engineering > Design Documents > Architecture Specs". This taxonomy is stored in a managed metadata service and referenced by choice fields. When a user tags a document with a taxonomy term, they are creating a link to a globally unique term that can be used for cross-site navigation, aggregation, and governance.
SQL
CREATE TABLE TaxonomyTerms (
TermId UNIQUEIDENTIFIER PRIMARY KEY,
TenantId UNIQUEIDENTIFIER NOT NULL,
ParentTermId UNIQUEIDENTIFIER NULL,
Name NVARCHAR(256) NOT NULL,
Path NVARCHAR(4000) NOT NULL,
Depth INT NOT NULL DEFAULT 0,
IsAvailableInNewForms BIT DEFAULT 1,
IsDeprecated BIT DEFAULT 0
);
CREATE TABLE DocumentTerms (
DocumentId UNIQUEIDENTIFIER NOT NULL,
TermId UNIQUEIDENTIFIER NOT NULL,
FieldId UNIQUEIDENTIFIER NOT NULL,
PRIMARY KEY (DocumentId, TermId, FieldId)
);
7. Full-Text Search with Elasticsearch
Search Architecture
Full-text search is arguably the most critical feature in a DMS. Users need to find documents by content, metadata, author, date range, file type, and sensitivity label — all with sub-second latency across billions of documents. We use Elasticsearch as the primary search engine, with a custom indexing pipeline that processes documents asynchronously after upload.
document-indexing"] KAFKA --> EXTRACT["Content Extractor
PDF, DOCX, TXT"] EXTRACT --> NLP["NLP Pipeline
NER, Tokenization"] NLP --> INDEX["Elasticsearch Index
per tenant"] INDEX --> SEARCH_API["Search API
Query, Filter, Facet"] SEARCH_API --> RESULTS["Ranked Results
with Snippets"]
Elasticsearch Index Mapping
JSON
{
"mappings": {
"properties": {
"document_id": { "type": "keyword" },
"tenant_id": { "type": "keyword" },
"library_id": { "type": "keyword" },
"file_name": {
"type": "text",
"analyzer": "standard",
"fields": {
"keyword": { "type": "keyword" },
"autocomplete": {
"type": "text",
"analyzer": "autocomplete_analyzer"
}
}
},
"content": {
"type": "text",
"analyzer": "english",
"term_vector": "with_positions_offsets",
"store": true
},
"content_type_name": { "type": "keyword" },
"author": { "type": "keyword" },
"created_at": { "type": "date" },
"modified_at": { "type": "date" },
"file_size": { "type": "long" },
"sensitivity": { "type": "keyword" },
"metadata": {
"type": "nested",
"properties": {
"field_name": { "type": "keyword" },
"field_value": { "type": "text", "fields": {
"keyword": { "type": "keyword" }
}}
}
}
}
}
}
Content Extraction Pipeline
Extracting text from documents is a non-trivial problem. Different file types require different parsers:
| File Type | Parser | Notes |
|---|---|---|
| Apache Tika + PDFBox | Handles both text-based and scanned PDFs | |
| DOCX/XLSX/PPTX | Open XML SDK / Apache POI | Extracts text from XML payloads inside ZIP |
| Plain Text | Direct read | UTF-8 encoding detection |
| Images (JPG/PNG) | Tesseract OCR | Text extraction from photos/scans |
| Email (EML/PST) | Apache Tika + Mail API | Extracts body, attachments, headers |
| CSV/TSV | Custom parser | Extracts header row as metadata, body as content |
| Video/Audio | Speech-to-text (Azure Speech) | Extracts transcript for indexing |
C#
public class ContentExtractionService
{
private readonly ITikaClient _tika;
private readonly IOcrService _ocr;
private readonly IOpenXmlParser _openXml;
public async Task<ExtractedContent> ExtractAsync(
Stream fileStream, string mimeType)
{
var result = new ExtractedContent();
switch (mimeType)
{
case "application/pdf":
var pdfContent = await _tika.ExtractTextAsync(fileStream);
if (string.IsNullOrWhiteSpace(pdfContent))
{
fileStream.Position = 0;
var pages = await _tika.ExtractPdfPagesAsImagesAsync(fileStream);
var ocrTexts = new List<string>();
foreach (var page in pages)
{
var ocrText = await _ocr.RecognizeTextAsync(page);
ocrTexts.Add(ocrText);
}
result.Text = string.Join("\n\n", ocrTexts);
result.IsOcrExtracted = true;
}
else
{
result.Text = pdfContent;
}
break;
case "application/vnd.openxmlformats-officedocument.wordprocessingml.document":
var docx = await _openXml.ExtractDocxAsync(fileStream);
result.Text = docx.BodyText;
break;
case "image/jpeg":
case "image/png":
result.Text = await _ocr.RecognizeTextAsync(fileStream);
result.IsOcrExtracted = true;
break;
default:
using var reader = new StreamReader(fileStream);
result.Text = await reader.ReadToEndAsync();
break;
}
return result;
}
}
Search API Design
C#
public class SearchRequest
{
public string Query { get; set; }
public Guid? LibraryId { get; set; }
public string ContentType { get; set; }
public SensitivityLabel? Sensitivity { get; set; }
public DateTime? ModifiedAfter { get; set; }
public DateTime? ModifiedBefore { get; set; }
public string Author { get; set; }
public List<MetadataFilter> MetadataFilters { get; set; }
public SearchSort Sort { get; set; }
public int Page { get; set; } = 1;
public int PageSize { get; set; } = 20;
public bool IncludeContent { get; set; }
public List<string> HighlightFields { get; set; }
}
public class SearchResponse
{
public int TotalResults { get; set; }
public double QueryTimeMs { get; set; }
public List<SearchResult> Results { get; set; }
public List<FacetGroup> Facets { get; set; }
}
public class SearchResult
{
public Guid DocumentId { get; set; }
public string FileName { get; set; }
public string ContentType { get; set; }
public string Author { get; set; }
public DateTime ModifiedAt { get; set; }
public long FileSize { get; set; }
public float Score { get; set; }
public List<SearchHighlight> Highlights { get; set; }
public Dictionary<string, object> Metadata { get; set; }
}
8. Document Preview and Rendering
Preview Pipeline
When a user clicks on a document in the web app, they expect to see an inline preview without downloading the file. The preview pipeline converts documents into web-friendly formats (HTML, images, or PDF) and caches the results. The conversion happens asynchronously after upload, and the results are stored alongside the original blob.
Office Online"] CONVERT --> IMG["ImageMagick /
Sharp"] CONVERT --> VIDEO["FFmpeg"] PDF --> STORE["Store in
Preview Cache"] IMG --> STORE VIDEO --> STORE STORE --> SERVE
Conversion Strategies by File Type
| Source Format | Conversion Tool | Output Format | Notes |
|---|---|---|---|
| DOCX/XLSX/PPTX | LibreOffice Headless / Word Online | PDF + PNG pages | Fidelity is critical for enterprise docs |
| PDF.js renderer | HTML canvas | Server-side: page images for thumbnails | |
| JPG/PNG/SVG | Sharp / ImageMagick | Optimized thumbnail | Resize for preview, keep original for download |
| Video (MP4) | FFmpeg | Thumbnail sprite + HLS | Generate preview frames at intervals |
| Audio (MP3/WAV) | FFmpeg | Waveform visualization | SVG waveform for inline display |
| Markdown | Markdig | HTML | Render inline with syntax highlighting |
| Code files | Highlight.js | Syntax-highlighted HTML | Auto-detect language |
C#
public class PreviewService
{
private readonly IBlobStorage _blobStorage;
private readonly IRedisCache _cache;
private readonly ILibreOfficeConverter _libreOffice;
private readonly IImageProcessor _imageProcessor;
public async Task<PreviewResult> GetPreviewAsync(
Guid documentId, int? pageNumber = null)
{
var cacheKey = $"preview:{documentId}:{pageNumber ?? 0}";
var cached = await _cache.GetAsync<PreviewResult>(cacheKey);
if (cached != null) return cached;
var metadata = await _db.GetDocumentAsync(documentId);
var originalBlob = await _blobStorage.GetBlobAsync(metadata.BlobPath);
PreviewResult result = metadata.ContentType switch
{
var ct when ct.Contains("wordprocessingml") =>
await ConvertOfficeDocumentAsync(originalBlob, metadata),
var ct when ct.Contains("pdf") =>
await RenderPdfAsync(originalBlob, pageNumber),
var ct when ct.StartsWith("image/") =>
await GenerateImagePreviewAsync(originalBlob, metadata),
var ct when ct.StartsWith("video/") =>
await GenerateVideoPreviewAsync(originalBlob, metadata),
_ => await GenerateTextPreviewAsync(originalBlob, metadata)
};
await _cache.SetAsync(cacheKey, result, TimeSpan.FromHours(24));
return result;
}
private async Task<PreviewResult> ConvertOfficeDocumentAsync(
IBlob file, DocumentMetadata metadata)
{
var pdfBytes = await _libreOffice.ConvertToPdfAsync(
file.Content, metadata.ContentType);
var pages = new List<PreviewPage>();
var pageCount = await _libreOffice.GetPageCountAsync(pdfBytes);
for (int i = 1; i <= pageCount; i++)
{
var pageImage = await _libreOffice.RenderPageAsync(pdfBytes, i, width: 1200);
var thumbImage = await _imageProcessor.ResizeAsync(pageImage, width: 300);
var fullPath = await _blobStorage.UploadAsync(
$"previews/{metadata.DocumentId}/page-{i}.png", pageImage);
var thumbPath = await _blobStorage.UploadAsync(
$"previews/{metadata.DocumentId}/thumb-{i}.png", thumbImage);
pages.Add(new PreviewPage
{
PageNumber = i,
FullImageUrl = fullPath,
ThumbnailUrl = thumbPath
});
}
return new PreviewResult
{
Type = PreviewType.Paginated,
Pages = pages,
TotalPages = pageCount
};
}
}
10. Document Workflows — Approval, Review, Signature
Workflow Engine Design
Document workflows encode business processes as state machines attached to documents. A typical contract might go through Draft → Legal Review → Manager Approval → eSignature → Published. Each transition has conditions, assignees, deadlines, and escalation rules. The workflow engine must support branching logic (parallel approvals), delegation, rework loops, and timeout handling.
C#
public class WorkflowDefinition
{
public Guid WorkflowId { get; set; }
public string Name { get; set; }
public List<WorkflowState> States { get; set; }
public List<WorkflowTransition> Transitions { get; set; }
public List<WorkflowVariable> Variables { get; set; }
public TimeSpan? DefaultTimeout { get; set; }
}
public class WorkflowTransition
{
public string TransitionId { get; set; }
public string FromState { get; set; }
public string ToState { get; set; }
public string TriggerAction { get; set; }
public List<TransitionCondition> Conditions { get; set; }
public List<AssigneeRule> Assignees { get; set; }
public TimeSpan? Timeout { get; set; }
public string TimeoutAction { get; set; }
}
public class WorkflowInstance
{
public Guid InstanceId { get; set; }
public Guid DocumentId { get; set; }
public Guid WorkflowDefinitionId { get; set; }
public string CurrentState { get; set; }
public List<WorkflowAction> History { get; set; }
public DateTime CreatedAt { get; set; }
public DateTime? CompletedAt { get; set; }
}
public class WorkflowAction
{
public Guid ActionId { get; set; }
public Guid InstanceId { get; set; }
public string TransitionId { get; set; }
public Guid PerformedBy { get; set; }
public string Comment { get; set; }
public DateTime PerformedAt { get; set; }
}
Approval Chain Patterns
| Pattern | Description | Use Case |
|---|---|---|
| Sequential | Each approver acts in order; document advances on approval | Contract signing, policy approval |
| Parallel | Multiple approvers act independently; all must approve | HR onboarding checklist, multi-dept sign-off |
| Majority Vote | Document advances when more than N% of approvers approve | Committee decisions, board resolutions |
| Conditional Branch | Next approver depends on document metadata (e.g., contract value) | Tiered approval thresholds |
| Delegation | Approver delegates to another user (manager on leave) | Availability-aware routing |
| Escalation | If no action within SLA, escalate to next level | Time-sensitive approvals |
11. Retention Policies and Archival
Why Retention Matters
Every document has a lifecycle. Retention policies define how long a document must be kept, what happens when that period expires, and whether the document is subject to legal hold. Organizations in regulated industries (healthcare, finance, legal) face fines for destroying documents too early, and face liability for keeping them too long (privacy violations). An automated retention system removes human error from this equation.
C#
public class RetentionPolicy
{
public Guid PolicyId { get; set; }
public Guid TenantId { get; set; }
public string Name { get; set; }
public RetentionTrigger Trigger { get; set; }
public int RetentionDays { get; set; }
public ExpirationAction Action { get; set; }
public bool AllowExtension { get; set; }
public int MaxExtensionDays { get; set; }
public List<LitigationHold> ActiveHolds { get; set; }
}
public enum RetentionTrigger
{
DocumentCreated,
DocumentModified,
DocumentApproved,
ContractExpired,
EmployeeTerminated,
CustomDateField
}
public enum ExpirationAction
{
Delete,
Archive,
NotifyOwner,
ReviewRequired,
MoveToRecordCenter
}
Legal Hold Mechanism
A legal hold overrides ALL retention policies and cleanup operations. When a document is placed under legal hold (typically due to pending litigation or regulatory investigation), it cannot be deleted, modified, or moved to archive — regardless of its retention period. The hold is stored as a separate entity linked to the document, and every delete/archive operation checks for active holds before proceeding.
SQL
CREATE TABLE LitigationHolds (
HoldId UNIQUEIDENTIFIER PRIMARY KEY,
TenantId UNIQUEIDENTIFIER NOT NULL,
CaseNumber NVARCHAR(100) NOT NULL,
CaseName NVARCHAR(500) NOT NULL,
Description NVARCHAR(MAX),
CreatedBy UNIQUEIDENTIFIER NOT NULL,
CreatedAt DATETIME2 NOT NULL,
ExpiresAt DATETIME2 NULL,
IsActive BIT DEFAULT 1
);
CREATE TABLE DocumentHolds (
DocumentId UNIQUEIDENTIFIER NOT NULL,
HoldId UNIQUEIDENTIFIER NOT NULL,
PlacedBy UNIQUEIDENTIFIER NOT NULL,
PlacedAt DATETIME2 NOT NULL,
RemovedAt DATETIME2 NULL,
PRIMARY KEY (DocumentId, HoldId)
);
Archival Pipeline
The retention service runs a daily batch job that evaluates every document against applicable retention policies. Documents past their retention period (and not under legal hold) trigger their configured expiration action. The archival pipeline moves blobs from Hot/Cool tier to Cold/Archive tier, generates a deletion schedule for documents that must be permanently destroyed, and logs every retention action for audit purposes.
12. Sensitivity Labels and Classification
Classification Framework
Sensitivity labels define the confidentiality level of a document and drive downstream enforcement — encryption, access restrictions, watermarking, and DLP policies. A "Restricted" document is encrypted at rest and in transit, accessible only to explicitly named users, watermarked with the viewer's identity, and logged in a separate audit trail. A "Public" document has no restrictions and may be cached on CDN edges worldwide.
C#
public class SensitivityLabelDefinition
{
public SensitivityLabel Label { get; set; }
public string DisplayName { get; set; }
public string Description { get; set; }
public bool EncryptAtRest { get; set; }
public bool EncryptInTransit { get; set; }
public bool ApplyWatermark { get; set; }
public bool BlockExternalSharing { get; set; }
public bool RequireMfa { get; set; }
public int MaxAccessLogRetentionDays { get; set; }
public List<AccessControlRule> AccessRules { get; set; }
}
public class AccessControlRule
{
public string PrincipalType { get; set; }
public string PrincipalId { get; set; }
public PermissionLevel Permission { get; set; }
public bool RequiresApproval { get; set; }
}
| Label | Encryption | External Sharing | Watermark | Audit | Example |
|---|---|---|---|---|---|
| Public | No | Allowed | No | Standard | Marketing brochure, press release |
| Internal | In-transit | Blocked | No | Standard | Internal memo, team handbook |
| Confidential | At-rest + In-transit | Blocked | Yes | Extended (1 year) | Financial reports, HR records |
| Restricted | AES-256 + Key Vault | Blocked | Yes (dynamic) | Indefinite | M&A docs, trade secrets, legal strategy |
Auto-Classification with Machine Learning
Manual classification is unreliable — users forget, misclassify, or skip it. We deploy a classification model trained on labeled document samples that analyzes content patterns, metadata, and file characteristics to predict the appropriate sensitivity label. The model runs as part of the indexing pipeline and pre-populates the label, which the user can then confirm or override. For "Restricted" labels, we require manual confirmation — the system never auto-assigns the highest classification level.
13. Access Control — ACLs, Inheritance, Sharing Links
Permission Model
Enterprise DMS requires hierarchical access control where permissions are inherited down the folder tree but can be broken at any level. A user with "Contribute" access on a library automatically gets "Contribute" on all subfolders — unless a subfolder has broken inheritance and explicitly grants or denies access to a different set of users.
C#
public class AccessControlEntry
{
public Guid AceId { get; set; }
public Guid ResourceId { get; set; }
public ResourceType ResourceType { get; set; }
public PrincipalType PrincipalType { get; set; }
public Guid PrincipalId { get; set; }
public PermissionLevel Permission { get; set; }
public bool IsInherited { get; set; }
public bool IsDenied { get; set; }
public DateTime GrantedAt { get; set; }
public Guid GrantedBy { get; set; }
}
public enum PermissionLevel
{
None = 0,
Read = 1,
Contribute = 2,
Edit = 3,
Design = 4,
FullControl = 5,
Owner = 6
}
Permission Evaluation Algorithm
When a user requests access to a document, the system walks up the folder hierarchy collecting all applicable ACLs (both inherited and explicit), evaluates explicit denies first (deny always wins), then checks for the required permission level. The result is cached in Redis with a TTL of 5 minutes to avoid repeated tree walks on every request.
ACL: [All Staff: Read]"] --> F1["Folder: Contracts
ACL: [Legal Team: FullControl]
Inherits: Read from Root"] F1 --> F2["Subfolder: Active
ACL: [Partners: Contribute]
Inherits: FullControl from Contracts"] F1 --> F3["Subfolder: Expired
ACL: BREAK INHERITANCE
[Archivists: Read]"] F2 --> D1["Document: NDA.pdf
Inherits: Contribute from Active"] style F3 fill:#f85149,color:#fff style D1 fill:#3fb950,color:#fff
External Sharing Links
Sharing links allow external users to access specific documents without a full account. Links can be scoped to a single document, a folder, or an entire library. Each link has an expiration date, optional password protection, and an access limit (number of unique visitors). All access through sharing links is logged to the audit trail with the link creator's identity.
C#
public class SharingLink
{
public Guid LinkId { get; set; }
public Guid DocumentId { get; set; }
public Guid CreatedBy { get; set; }
public SharingLinkType Type { get; set; }
public PermissionLevel Permission { get; set; }
public DateTime ExpiresAt { get; set; }
public string PasswordHash { get; set; }
public int? MaxAccessCount { get; set; }
public int CurrentAccessCount { get; set; }
public bool IsRevoked { get; set; }
public List<SharingLinkAccess> AccessLog { get; set; }
}
public enum SharingLinkType
{
SpecificPeople,
AnyoneWithLink,
OrganizationOnly
}
14. Audit Trail and Activity Logging
Immutable Audit Log
An enterprise DMS must produce an immutable, tamper-evident log of every significant action: who accessed which document, when, from where, and what they did. This is non-negotiable for compliance with SOX, HIPAA, GDPR, and FedRAMP. The audit log is append-only — no updates or deletes are permitted — and is stored in a write-once, read-many (WORM) storage tier.
C#
public class AuditEvent
{
public long EventId { get; set; }
public Guid TenantId { get; set; }
public Guid UserId { get; set; }
public string UserEmail { get; set; }
public string UserAgent { get; set; }
public string IpAddress { get; set; }
public AuditEventType EventType { get; set; }
public Guid? DocumentId { get; set; }
public string DocumentPath { get; set; }
public string Action { get; set; }
public DateTime Timestamp { get; set; }
public Dictionary<string, object> Details { get; set; }
public string PreviousHash { get; set; }
public string EventHash { get; set; }
}
public enum AuditEventType
{
DocumentCreated,
DocumentViewed,
DocumentDownloaded,
DocumentModified,
DocumentDeleted,
DocumentShared,
PermissionChanged,
WorkflowAction,
LabelChanged,
SearchPerformed,
ExportRequested,
LegalHoldPlaced,
LegalHoldRemoved,
RetentionAction
}
Hash Chain for Tamper Detection
Each audit event includes the SHA-256 hash of the previous event, forming a blockchain-like chain. If anyone modifies a historical event, the hash chain breaks and the tampering is detectable. Verification runs daily as a background job that re-hashes every event and compares it to the stored hash.
15. Document Templates
Template Engine
Document templates allow users to create new documents pre-populated with structure, formatting, and dynamic fields. The template engine uses the Open XML SDK for Office documents and Handlebars/Razor for HTML templates. Templates are stored as versioned blobs and can include conditional sections, repeating tables, and field calculations.
C#
public class DocumentTemplate
{
public Guid TemplateId { get; set; }
public Guid TenantId { get; set; }
public string Name { get; set; }
public string Description { get; set; }
public Guid ContentTypeId { get; set; }
public string TemplateBlobPath { get; set; }
public string OutputMimeType { get; set; }
public List<TemplateField> Fields { get; set; }
public int Version { get; set; }
public bool IsDefault { get; set; }
}
public class TemplateField
{
public string FieldName { get; set; }
public string DisplayName { get; set; }
public TemplateFieldType Type { get; set; }
public bool IsRequired { get; set; }
public string Placeholder { get; set; }
public string DefaultValue { get; set; }
public string Pattern { get; set; }
}
When a user creates a document from a template, they fill in a form with the required field values. The engine replaces placeholders in the template document, evaluates any conditional sections, and generates a new document blob. The generated document inherits the content type, default workflow, and retention policy of the template.
16. Metadata-Driven Views
Flexible View System
Views allow users to browse library contents through different lenses — a list view sorted by modified date, a gallery view with thumbnails, a calendar view grouped by due date, or a Kanban board grouped by status. Views are stored as query definitions and rendered client-side, with search results from Elasticsearch providing the backing data.
C#
public class LibraryView
{
public Guid ViewId { get; set; }
public Guid LibraryId { get; set; }
public string Name { get; set; }
public ViewType Type { get; set; }
public List<ViewFilter> Filters { get; set; }
public List<ViewSort> SortColumns { get; set; }
public List<ViewGroup> GroupBy { get; set; }
public List<ViewColumn> Columns { get; set; }
public bool IsDefault { get; set; }
public bool IsPublic { get; set; }
public Guid CreatedBy { get; set; }
}
public enum ViewType
{
List,
Gallery,
Calendar,
Board,
Timeline,
Custom
}
public class ViewColumn
{
public string FieldName { get; set; }
public string DisplayName { get; set; }
public int Order { get; set; }
public int? Width { get; set; }
public bool IsVisible { get; set; }
public string Format { get; set; }
}
Views are resolved to Elasticsearch queries at runtime. A "Board" view grouped by document status sends a terms aggregation query to Elasticsearch and renders the results in columns. A "Calendar" view sends a date histogram aggregation. This approach avoids maintaining a separate query engine — Elasticsearch's aggregation framework handles all view types natively.
17. Content Deduplication
Hash-Based Deduplication
Enterprises often have many copies of the same document — emailed versions, backup copies, and renamed duplicates. Content deduplication eliminates redundant storage by detecting identical files and storing only one blob, referenced by multiple metadata records. We use SHA-256 content hashing at the file level and optionally at the block level for large documents.
C#
public class DeduplicationService
{
private readonly IDatabase _db;
private readonly IBlobStorage _blob;
public async Task<DeduplicationResult> CheckAndDeduplicateAsync(
Guid tenantId, string contentHash, string blobPath, long fileSize)
{
var existing = await _db.QueryAsync<ContentHash>(
"SELECT * FROM ContentHashes " +
"WHERE TenantId = @tenant AND Hash = @hash",
new { tenant = tenantId, hash = contentHash });
if (existing != null)
{
await _blob.DeleteAsync(blobPath);
return new DeduplicationResult
{
IsDuplicate = true,
OriginalDocumentId = existing.DocumentId,
StorageSavedBytes = fileSize
};
}
await _db.ExecuteAsync(
"INSERT INTO ContentHashes " +
"(TenantId, Hash, BlobPath, FileSize, FirstDocumentId, CreatedAt) " +
"VALUES (@tenant, @hash, @path, @size, @docId, @now)",
new
{
tenant = tenantId,
hash = contentHash,
path = blobPath,
size = fileSize,
docId = Guid.NewGuid(),
now = DateTime.UtcNow
});
return new DeduplicationResult { IsDuplicate = false };
}
}
Deduplication Metrics
| Metric | Typical Value | Impact |
|---|---|---|
| Duplicate rate (enterprise) | 15-30% of uploads | 15-30% storage cost reduction |
| Average storage saved per tenant | 2-5 TB/year | $400-$1,000/month in Azure Blob costs |
| Hash computation time | <50ms for 100 MB file | Negligible latency impact |
| False positive rate | 0% (cryptographic hash) | No risk of incorrect deduplication |
18. Large File Handling — Chunked and Resumable Uploads
Chunked Upload Protocol
Enterprise users upload large files — CAD drawings (500 MB), video files (2 GB), forensic images (250 GB). A single HTTP PUT cannot handle these reliably. We implement a chunked upload protocol where the client splits the file into 4 MB blocks, uploads each block independently, and the server assembles them into the final blob. If the connection drops, the client can resume from the last successful block.
C#
public class UploadSession
{
public Guid SessionId { get; set; }
public Guid TenantId { get; set; }
public Guid UserId { get; set; }
public string FileName { get; set; }
public string ContentType { get; set; }
public long TotalFileSize { get; set; }
public int ChunkSize { get; set; } = 4 * 1024 * 1024;
public int TotalChunks => (int)Math.Ceiling(
(double)TotalFileSize / ChunkSize);
public HashSet<int> CompletedChunks { get; set; } = new();
public DateTime CreatedAt { get; set; }
public DateTime ExpiresAt { get; set; }
public UploadSessionStatus Status { get; set; }
}
19. Barcode, QR Code, and OCR Processing
OCR Pipeline
Many enterprise documents are scanned paper records — contracts, invoices, medical records, legal filings. OCR (Optical Character Recognition) extracts text from these images, making them searchable and enabling automated data extraction. Our OCR pipeline uses Tesseract for general-purpose recognition and Azure Computer Vision for high-accuracy scenarios.
C#
public class OcrProcessingService
{
private readonly ITesseractEngine _tesseract;
private readonly IAzureComputerVision _azureVision;
private readonly IPdfRenderer _pdfRenderer;
public async Task<OcrResult> ProcessDocumentAsync(
Stream fileStream, string mimeType, OcrQuality quality)
{
var pages = new List<OcrPage>();
if (mimeType == "application/pdf")
{
var images = await _pdfRenderer.RenderPagesToImagesAsync(
fileStream, dpi: 300);
foreach (var img in images)
{
var ocrPage = quality == OcrQuality.High
? await _azureVision.RecognizeTextAsync(img)
: await _tesseract.RecognizeAsync(img);
pages.Add(ocrPage);
}
}
else if (mimeType.StartsWith("image/"))
{
var ocrResult = quality == OcrQuality.High
? await _azureVision.RecognizeTextAsync(fileStream)
: await _tesseract.RecognizeAsync(fileStream);
pages.Add(ocrResult);
}
return new OcrResult
{
FullText = string.Join("\n\n", pages.Select(p => p.Text)),
Confidence = pages.Average(p => p.Confidence),
DetectedLanguage = pages.First().DetectedLanguage,
WordCount = pages.Sum(p => p.Words.Count),
Pages = pages
};
}
}
Barcode and QR Code Scanning
Barcodes and QR codes on scanned documents serve as machine-readable document identifiers. The barcode service extracts codes, maps them to document metadata (e.g., a barcode on a contract might encode the contract number), and uses this mapping to auto-populate metadata fields during upload. We support Code 128, Code 39, QR Code, Data Matrix, and PDF417 formats.
| OCR Engine | Accuracy | Speed | Cost | Best For |
|---|---|---|---|---|
| Tesseract (local) | 85-95% | Fast (local GPU) | Free | Standard text documents |
| Azure Computer Vision | 95-99% | Medium (API call) | $1/1000 pages | Handwriting, low-quality scans |
| Google Cloud Vision | 94-98% | Medium (API call) | $1.50/1000 pages | Multi-language documents |
20. Compliance — Records Management, Legal Hold, eDiscovery
Records Management
Records management goes beyond simple retention. A "record" is a document that has been officially declared as a business record — it cannot be modified, only appended with a reason code. Records management requires a chain of custody (who declared it, when, under what authority), a records schedule (when it must be destroyed), and an audit trail of all access.
C#
public class DocumentRecord
{
public Guid RecordId { get; set; }
public Guid DocumentId { get; set; }
public string RecordCategory { get; set; }
public string RecordSchedule { get; set; }
public DateTime DeclaredAt { get; set; }
public Guid DeclaredBy { get; set; }
public DateTime RetentionExpiry { get; set; }
public DestructionStatus DestructionStatus { get; set; }
public List<ChainOfCustodyEntry> CustodyChain { get; set; }
}
public class ChainOfCustodyEntry
{
public DateTime Timestamp { get; set; }
public Guid UserId { get; set; }
public string Action { get; set; }
public string Reason { get; set; }
public string Location { get; set; }
}
eDiscovery
eDiscovery (electronic discovery) is the process of identifying, preserving, and producing documents relevant to legal proceedings. The eDiscovery module allows legal teams to create cases, define search queries (custodian-based, date-range, keyword, content-type), review search results, and export matched documents with a complete chain of custody and metadata for court submission.
21. Security — Encryption at Rest and In Transit
Encryption Architecture
All document content is encrypted at rest using AES-256. For standard documents, we use platform-managed keys (Azure/S3 default encryption). For "Confidential" and "Restricted" documents, we use customer-managed keys stored in Azure Key Vault or AWS KMS, giving the enterprise full control over key rotation and revocation. In transit, all communication uses TLS 1.3 with certificate pinning on mobile clients.
C#
public class EncryptionService
{
private readonly IKeyVaultClient _keyVault;
private readonly IKeyResolver _keyResolver;
public async Task<EncryptedContent> EncryptAsync(
Stream content, SensitivityLabel label, Guid tenantId)
{
var keyId = label switch
{
SensitivityLabel.Restricted =>
await _keyVault.GetKeyAsync($"restricted-{tenantId}"),
SensitivityLabel.Confidential =>
await _keyVault.GetKeyAsync($"confidential-{tenantId}"),
_ => PlatformManagedKey
};
var encrypted = await Aes256.EncryptAsync(
content, keyId.Value, keyId.IV);
return new EncryptedContent
{
EncryptedStream = encrypted,
KeyId = keyId.Kid,
Algorithm = "AES-256-GCM",
EncryptedAt = DateTime.UtcNow
};
}
}
Zero-Trust Architecture
Every service-to-service call within our architecture is mutually authenticated via mTLS (mutual TLS). The API Gateway validates JWT tokens with short expiry (15 minutes) and issues refresh tokens. Sensitive operations (deleting documents, changing permissions, exporting data) require step-up authentication — a second factor challenge — even if the user is already authenticated. All service identities are managed through a service mesh (Istio) that enforces authorization policies at the network layer.
22. Monitoring, Logging, and Observability
Observability Stack
We instrument every service with OpenTelemetry for distributed tracing, Prometheus for metrics, and structured logging to a centralized sink (ELK or Datadog). The key metrics we track include:
| Metric | Target | Alert Threshold |
|---|---|---|
| Upload success rate | >99.9% | <99.5% |
| Search latency (p99) | <2 seconds | >3 seconds |
| Preview generation time | <10 seconds | >30 seconds |
| OCR processing queue depth | <1,000 | >10,000 |
| Audit log write latency | <50ms | >200ms |
| Active collaboration sessions | Tracked | Alert at 10K concurrent |
| Workflow completion rate | >95% | <80% |
| Storage utilization per tenant | <90% quota | >85% |
23. API Design
RESTful API Surface
The DMS exposes a comprehensive RESTful API following OpenAPI 3.0 specification. All endpoints are versioned (v1, v2), accept JSON request bodies, and return standard HTTP status codes. Authentication is via Bearer token (JWT) in the Authorization header.
| Method | Endpoint | Description |
|---|---|---|
| POST | /api/v1/documents/upload | Upload a new document |
| GET | /api/v1/documents/{id} | Get document metadata |
| GET | /api/v1/documents/{id}/content | Download document binary |
| PUT | /api/v1/documents/{id} | Update document metadata |
| DELETE | /api/v1/documents/{id} | Soft-delete a document |
| POST | /api/v1/documents/{id}/versions | Create a new version |
| GET | /api/v1/documents/{id}/versions | List all versions |
| POST | /api/v1/documents/{id}/checkout | Check out for editing |
| POST | /api/v1/documents/{id}/checkin | Check in with new version |
| POST | /api/v1/documents/{id}/share | Create a sharing link |
| POST | /api/v1/search | Full-text search across documents |
| GET | /api/v1/documents/{id}/preview | Get inline preview |
| POST | /api/v1/documents/{id}/workflow/start | Start a workflow |
| POST | /api/v1/workflows/{id}/action | Perform workflow action |
| GET | /api/v1/libraries/{id}/views | List views for a library |
| POST | /api/v1/discovery/cases | Create eDiscovery case |
| GET | /api/v1/audit/events | Query audit log |
API Request/Response Example
JSON
// POST /api/v1/documents/upload
// Request (multipart/form-data):
// file: [binary stream]
// metadata: {
// "libraryId": "a1b2c3d4-...",
// "parentFolderId": "e5f6g7h8-...",
// "contentTypeId": "i9j0k1l2-...",
// "metadata": {
// "ContractNumber": "MSA-2026-0042",
// "PartyName": "Acme Corp",
// "EffectiveDate": "2026-07-01"
// },
// "sensitivity": "Confidential"
// }
// Response: 201 Created
{
"documentId": "m3n4o5p6-...",
"fileName": "Service-Agreement-Acme.pdf",
"version": "1.0",
"status": "Active",
"size": 2048576,
"sensitivity": "Confidential",
"createdBy": "user-123",
"createdAt": "2026-07-13T10:30:00Z",
"previewAvailable": true,
"searchIndexed": true
}
Pagination and Rate Limiting
All list endpoints support cursor-based pagination (not offset-based) for consistent results when data is changing. Rate limits are applied per-tenant at the API Gateway: 10,000 requests/minute for standard endpoints, 1,000 requests/minute for search, and 100 requests/minute for bulk export operations. Rate limit headers (X-RateLimit-Remaining, X-RateLimit-Reset) are included in every response.
C#
[ApiController]
[Route("api/v1/[controller]")]
[Authorize]
public class DocumentsController : ControllerBase
{
private readonly IDocumentService _documents;
private readonly IAuditService _audit;
[HttpPost("upload")]
[RequestSizeLimit(256L * 1024 * 1024 * 1024)]
public async Task<ActionResult<DocumentResponse>> Upload(
IFormFile file, [FromForm] UploadMetadataRequest metadata)
{
var userId = User.GetUserId();
var tenantId = User.GetTenantId();
var result = await _documents.UploadAsync(
file.OpenReadStream(), metadata, userId, tenantId);
await _audit.LogAsync(new AuditEvent
{
EventType = AuditEventType.DocumentCreated,
UserId = userId,
DocumentId = result.DocumentId,
Action = "Upload",
Details = new Dictionary<string, object>
{
["fileName"] = file.FileName,
["fileSize"] = file.Length,
["sensitivity"] = metadata.Sensitivity
}
});
return CreatedAtAction(
nameof(GetDocument),
new { id = result.DocumentId },
result);
}
}
24. Testing Strategy
Multi-Layer Testing Approach
A production DMS demands rigorous testing across every layer. We follow the testing pyramid with heavy emphasis on integration and contract tests, given the number of external dependencies (blob storage, Elasticsearch, Key Vault, Identity Provider).
| Layer | Tool | Coverage Target | What We Test |
|---|---|---|---|
| Unit Tests | xUnit + Moq | 80%+ | Business logic, permission evaluation, version numbering, hash computation |
| Integration Tests | Testcontainers | All service boundaries | Database queries, blob storage operations, Elasticsearch indexing |
| Contract Tests | Pact | All API consumers | API request/response schemas between frontend and backend |
| E2E Tests | Playwright | Critical user flows | Upload, search, preview, workflow, sharing |
| Load Tests | k6 | N/A | 10K concurrent uploads, 50K search queries/min |
| Chaos Tests | Chaos Mesh | N/A | Blob storage outage, DB failover, Kafka partition |
| Security Tests | OWASP ZAP | OWASP Top 10 | XSS, CSRF, path traversal, privilege escalation |
C#
public class DocumentUploadIntegrationTest : IAsyncLifetime
{
private readonly PostgreSqlContainer _db;
private readonly AzuriteContainer _blobStorage;
private DocumentService _service;
public async Task InitializeAsync()
{
_db = new PostgreSqlBuilder().Build();
await _db.StartAsync();
_blobStorage = new AzuriteBuilder().Build();
await _blobStorage.StartAsync();
_service = new DocumentService(
_db.GetConnectionString(),
_blobStorage.GetBlobServiceClient());
}
[Fact]
public async Task Upload_ShouldCreateBlobAndMetadata()
{
var stream = new MemoryStream(
Encoding.UTF8.GetBytes("test document content"));
var result = await _service.UploadAsync(
stream, "test.pdf", "application/pdf",
tenantId: Guid.NewGuid(),
userId: Guid.NewGuid());
Assert.NotNull(result.BlobPath);
Assert.Equal("test.pdf", result.FileName);
Assert.True(result.FileSize > 0);
var metadata = await _service.GetAsync(result.DocumentId);
Assert.Equal(DocumentStatus.Active, metadata.Status);
}
public async Task DisposeAsync()
{
await _db.DisposeAsync();
await _blobStorage.DisposeAsync();
}
}
25. Cost Estimation
Infrastructure Cost Breakdown
The following table estimates monthly costs for a mid-size deployment serving 500 enterprises with 2.5 million total users and 5 PB of stored documents. Costs assume Azure pricing in US East region.
| Component | Specification | Monthly Cost | Notes |
|---|---|---|---|
| Blob Storage (Hot) | 1 PB (20% of total) | $18,000 | $0.018/GB/month |
| Blob Storage (Cool) | 2 PB (40%) | $20,000 | $0.010/GB/month |
| Blob Storage (Archive) | 2 PB (40%) | $2,000 | $0.001/GB/month |
| Cosmos DB (Metadata) | 500 TB RU/s, 50 TB data | $75,000 | Multi-master, 400 RU/s per 1 GB |
| Elasticsearch Cluster | 200 TB, 120 data nodes | $45,000 | D48s v5 instances |
| Redis Cache | 1 TB, 6 nodes | $8,000 | Premium P6 tier |
| Kafka Cluster | 6 brokers, 50 TB | $12,000 | Azure Event Hubs |
| Compute (AKS) | 80 pods, D16s v5 | $30,000 | All microservices combined |
| Key Vault | 10,000 operations/day | $500 | Managed HSM for Restricted keys |
| Cognitive Services (OCR) | 500K pages/month | $500 | At $1/1000 pages |
| CDN (Previews) | 50 TB egress/month | $4,000 | $0.08/GB egress |
| Monitoring (Datadog) | 200 hosts | $8,000 | Pro tier |
| DNS & SSL | Global DNS + Certificates | $500 | Traffic Manager + App Gateway |
| Backup & DR | Cross-region geo-redundancy | $15,000 | 2x storage replication cost |
| Total | $238,500 |
Cost Per User
Total monthly cost: $238,500 / 2.5 million users = $0.095 per user per month. If we price the DMS at $15 per user per month (comparable to SharePoint Online), the gross margin is approximately 99.4%. However, this does not include engineering team costs (20 engineers at $200K average = $333K/month), sales/marketing, or support infrastructure.
26. Interview Q&A
Question 1: How would you handle a scenario where a user uploads a 200 GB forensic video file?
Question 2: How do you ensure that search results respect document-level permissions?
viewer_ids field containing all user and group IDs with read access. The search query is wrapped with a DLS filter that restricts results to documents where the requesting user's ID appears in viewer_ids. For performance, we also precompute a flattened permission set for each user (updated on ACL changes) and pass it as a runtime field filter. For tenants with fewer than 10,000 documents, we use a shared index with DLS. For larger tenants, we use index-per-tenant isolation, which provides stronger security guarantees and independent shard management.
Question 3: What happens when the approval chain requires 5 approvers but one is on leave for 2 weeks?
Question 4: How would you migrate 5 million documents from a legacy file server to this DMS?
Question 5: How do you prevent a rogue administrator from deleting audit logs?
Question 6: Explain the trade-offs between storing metadata in Cosmos DB versus PostgreSQL.
Question 7: How would you design the real-time notification system for co-authoring?
Question 8: How do you handle concurrent uploads of the same document by two different users?
If-Match header with the version hash it read. If the hash has changed since the read, the server returns 409 Conflict and the client must re-read and retry.