مرحباً بكم في الجزء 9! في الأجزاء السابقة، بنينا نظاماً قوياً RAG الذي يقوم بعمليات وضع العلامات على نقاط المدونات ويجعلها قابلة للبحث من خلال عمليات التضمين المائلة. الآن حان الوقت لتوسيع قدراتنا مع **** - مكتبة قوية لمعالجة الوثائق يمكنها أن تُضفي طابعاً شاملاً على الـ PDFs، DOCX، وغير ذلك من الأشكال، مما يجعل محامينا GPT شاملاً حقاً.
ملاحظة: هذا جزء من تجاربي مع AI (يساعد في الصياغة) + تحريري الخاص. نفس الصوت، نفس العملية العملية؛ مجرد أصابع أسرع.
ليس أنني بحاجة إلى هذا من أجل المدونة (كلها علامة إلى أسفل)، ولكن للكمال فكرت في أن أعرض كيف يمكن بسهولة إضافة قدرات ابتلاع الوثائق إلى خط أنابيبك RAG. هذا مفيد بشكل خاص إذا كنت تبني نظام يحتاج إلى معالجة الوثائق القانونية، العقود، أو غير ذلك من وثائق الأعمال.
وتتناول الممارسات القانونية الحديثة مجموعة متنوعة من أشكال الوثائق التي لا تقتصر على مجرد النص، إذ يتعين على المحامين الرجوع إلى ما يلي:
ويتيح التطويع لمحامينا GPT تجهيز هذه الأشكال وجعلها قابلة للبحث، مما ينشئ قاعدة معارف شاملة حقاً تعكس الطريقة التي تستخدم بها شركات القانون الحديثة هذه المعايير.
هو عبارة عن مجموعة أدوات لمعالجة الوثائق من مصادر مفتوحة مستمدة من نظام IBM:
تقديم الخدمات هي أسهل طريقة لمعالجة الطلب كخدمة. دعونا نجهزها:
# Using the official container image from Quay.io
docker run -p 5001:5001 quay.io/docling-project/docling-serve
# Or with the UI enabled for testing
docker run -p 5001:5001 -e DOCLING_SERVE_ENABLE_UI=1 quay.io/docling-project/docling-serve
صور الحاويات المتاحة:
□ الصورة / الوصف / الحجم /
|-------|-------------|------|
| quay.io/docling-project/docling-serve □ صورة قاعدية (حزم PyPIP) _ 8.7GB (md64)
| quay.io/docling-project/docling-serve-cpu x 4.4GB □ 4.4 GB □
| quay.io/docling-project/docling-serve-cu126 12.6 لـ GUU
| quay.io/docling-project/docling-serve-cu128 12.8 لـ GUU 12.8 لـ
نقاط النهاية:
http://localhost:5001http://localhost:5001/docshttp://localhost:5001/ui (عندما يُمكَل)للانتاج ، اضف Docling إلى الموجود docker-compose.yml:
services:
docling:
image: quay.io/docling-project/docling-serve:latest
ports:
- "5001:5001"
environment:
- DOCLING_SERVE_ENABLE_UI=0
- DOCLING_SERVE_MAX_WORKERS=4
volumes:
- docling_cache:/root/.cache
restart: unless-stopped
volumes:
docling_cache:
قبل الإندماج مع C، دعونا نختبر AI:
# Convert a PDF from URL
curl -X 'POST' \
'http://localhost:5001/v1/convert/source' \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-d '{
"sources": [{"kind": "http", "url": "https://arxiv.org/pdf/2501.17887"}],
"options": {
"to_formats": ["md"]
}
}'
# Convert a local file (upload)
curl -X 'POST' \
'http://localhost:5001/v1/convert/file' \
-H 'accept: application/json' \
-F '[email protected]'
بما أن Docling هو خدمة بايثون، سوف ندمج عن طريق HTTP. دعونا نبني عميلاً قوياً C#.
using System.Text.Json.Serialization;
namespace Mostlylucid.BlogLLM.Core.Models
{
/// <summary>
/// Request to convert documents from URLs or base64 content
/// </summary>
public class DoclingConvertRequest
{
[JsonPropertyName("sources")]
public List<DoclingSource> Sources { get; set; } = new();
[JsonPropertyName("options")]
public DoclingOptions? Options { get; set; }
}
public class DoclingSource
{
[JsonPropertyName("kind")]
public string Kind { get; set; } = "http"; // "http", "base64", "file"
[JsonPropertyName("url")]
public string? Url { get; set; }
[JsonPropertyName("base64")]
public string? Base64Content { get; set; }
[JsonPropertyName("filename")]
public string? Filename { get; set; }
}
public class DoclingOptions
{
[JsonPropertyName("to_formats")]
public List<string> ToFormats { get; set; } = new() { "md" }; // "md", "json", "text"
[JsonPropertyName("ocr")]
public bool Ocr { get; set; } = true;
[JsonPropertyName("table_mode")]
public string TableMode { get; set; } = "accurate"; // "fast", "accurate"
}
/// <summary>
/// Response from Docling conversion
/// </summary>
public class DoclingConvertResponse
{
[JsonPropertyName("document")]
public DoclingDocument? Document { get; set; }
[JsonPropertyName("status")]
public string Status { get; set; } = string.Empty;
[JsonPropertyName("errors")]
public List<string>? Errors { get; set; }
}
public class DoclingDocument
{
[JsonPropertyName("md_content")]
public string? MarkdownContent { get; set; }
[JsonPropertyName("json_content")]
public object? JsonContent { get; set; }
[JsonPropertyName("text_content")]
public string? TextContent { get; set; }
[JsonPropertyName("metadata")]
public DoclingMetadata? Metadata { get; set; }
}
public class DoclingMetadata
{
[JsonPropertyName("filename")]
public string? Filename { get; set; }
[JsonPropertyName("page_count")]
public int? PageCount { get; set; }
[JsonPropertyName("file_type")]
public string? FileType { get; set; }
}
/// <summary>
/// Our internal document model
/// </summary>
public class ProcessedDocument
{
public string DocumentId { get; set; } = Guid.NewGuid().ToString();
public string FileName { get; set; } = string.Empty;
public string OriginalFormat { get; set; } = string.Empty;
public string MarkdownContent { get; set; } = string.Empty;
public string? TextContent { get; set; }
public DateTime ProcessedDate { get; set; } = DateTime.UtcNow;
public string[] Categories { get; set; } = Array.Empty<string>();
public int? PageCount { get; set; }
public string ContentHash { get; set; } = string.Empty;
}
}
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using Mostlylucid.BlogLLM.Core.Models;
using System.Net.Http.Json;
using System.Security.Cryptography;
using System.Text;
using System.Text.Json;
namespace Mostlylucid.BlogLLM.Core.Services
{
public class DoclingClientOptions
{
public string BaseUrl { get; set; } = "http://localhost:5001";
public int TimeoutSeconds { get; set; } = 300; // 5 minutes for large documents
public bool EnableOcr { get; set; } = true;
public string TableMode { get; set; } = "accurate";
}
public class DoclingClient : IDisposable
{
private readonly HttpClient _httpClient;
private readonly ILogger<DoclingClient> _logger;
private readonly DoclingClientOptions _options;
public DoclingClient(
HttpClient httpClient,
ILogger<DoclingClient> logger,
IOptions<DoclingClientOptions> options)
{
_httpClient = httpClient;
_logger = logger;
_options = options.Value;
_httpClient.BaseAddress = new Uri(_options.BaseUrl);
_httpClient.Timeout = TimeSpan.FromSeconds(_options.TimeoutSeconds);
}
/// <summary>
/// Convert a document from a URL
/// </summary>
public async Task<ProcessedDocument?> ConvertFromUrlAsync(
string url,
string[]? categories = null,
CancellationToken cancellationToken = default)
{
_logger.LogInformation("Converting document from URL: {Url}", url);
var request = new DoclingConvertRequest
{
Sources = new List<DoclingSource>
{
new() { Kind = "http", Url = url }
},
Options = new DoclingOptions
{
ToFormats = new List<string> { "md", "text" },
Ocr = _options.EnableOcr,
TableMode = _options.TableMode
}
};
return await SendConversionRequestAsync(request, categories, cancellationToken);
}
/// <summary>
/// Convert a local file
/// </summary>
public async Task<ProcessedDocument?> ConvertFileAsync(
string filePath,
string[]? categories = null,
CancellationToken cancellationToken = default)
{
if (!File.Exists(filePath))
{
_logger.LogError("File not found: {FilePath}", filePath);
return null;
}
_logger.LogInformation("Converting local file: {FilePath}", filePath);
// Read file and convert to base64
var fileBytes = await File.ReadAllBytesAsync(filePath, cancellationToken);
var base64Content = Convert.ToBase64String(fileBytes);
var fileName = Path.GetFileName(filePath);
var request = new DoclingConvertRequest
{
Sources = new List<DoclingSource>
{
new()
{
Kind = "base64",
Base64Content = base64Content,
Filename = fileName
}
},
Options = new DoclingOptions
{
ToFormats = new List<string> { "md", "text" },
Ocr = _options.EnableOcr,
TableMode = _options.TableMode
}
};
return await SendConversionRequestAsync(request, categories, cancellationToken);
}
/// <summary>
/// Convert a file using multipart form upload (more efficient for large files)
/// </summary>
public async Task<ProcessedDocument?> ConvertFileUploadAsync(
string filePath,
string[]? categories = null,
CancellationToken cancellationToken = default)
{
if (!File.Exists(filePath))
{
_logger.LogError("File not found: {FilePath}", filePath);
return null;
}
_logger.LogInformation("Uploading and converting file: {FilePath}", filePath);
try
{
using var fileStream = File.OpenRead(filePath);
using var content = new MultipartFormDataContent();
using var streamContent = new StreamContent(fileStream);
var fileName = Path.GetFileName(filePath);
content.Add(streamContent, "files", fileName);
var response = await _httpClient.PostAsync(
"/v1/convert/file",
content,
cancellationToken);
if (!response.IsSuccessStatusCode)
{
var errorContent = await response.Content.ReadAsStringAsync(cancellationToken);
_logger.LogError("Docling conversion failed: {StatusCode} - {Error}",
response.StatusCode, errorContent);
return null;
}
var result = await response.Content.ReadFromJsonAsync<DoclingConvertResponse>(
cancellationToken: cancellationToken);
return MapToProcessedDocument(result, fileName, categories);
}
catch (Exception ex)
{
_logger.LogError(ex, "Error uploading file to Docling: {FilePath}", filePath);
return null;
}
}
private async Task<ProcessedDocument?> SendConversionRequestAsync(
DoclingConvertRequest request,
string[]? categories,
CancellationToken cancellationToken)
{
try
{
var response = await _httpClient.PostAsJsonAsync(
"/v1/convert/source",
request,
cancellationToken);
if (!response.IsSuccessStatusCode)
{
var errorContent = await response.Content.ReadAsStringAsync(cancellationToken);
_logger.LogError("Docling conversion failed: {StatusCode} - {Error}",
response.StatusCode, errorContent);
return null;
}
var result = await response.Content.ReadFromJsonAsync<DoclingConvertResponse>(
cancellationToken: cancellationToken);
var filename = request.Sources.FirstOrDefault()?.Filename
?? request.Sources.FirstOrDefault()?.Url
?? "unknown";
return MapToProcessedDocument(result, filename, categories);
}
catch (HttpRequestException ex)
{
_logger.LogError(ex, "HTTP error calling Docling API");
return null;
}
catch (TaskCanceledException ex)
{
_logger.LogError(ex, "Docling conversion timed out");
return null;
}
catch (Exception ex)
{
_logger.LogError(ex, "Unexpected error calling Docling API");
return null;
}
}
private ProcessedDocument? MapToProcessedDocument(
DoclingConvertResponse? response,
string filename,
string[]? categories)
{
if (response?.Document == null)
{
_logger.LogWarning("Docling returned empty document");
return null;
}
var markdownContent = response.Document.MarkdownContent ?? string.Empty;
return new ProcessedDocument
{
DocumentId = Guid.NewGuid().ToString(),
FileName = Path.GetFileName(filename),
OriginalFormat = Path.GetExtension(filename).TrimStart('.').ToLower(),
MarkdownContent = markdownContent,
TextContent = response.Document.TextContent,
ProcessedDate = DateTime.UtcNow,
Categories = categories ?? Array.Empty<string>(),
PageCount = response.Document.Metadata?.PageCount,
ContentHash = ComputeHash(markdownContent)
};
}
private static string ComputeHash(string content)
{
using var sha256 = SHA256.Create();
var bytes = sha256.ComputeHash(Encoding.UTF8.GetBytes(content));
return Convert.ToBase64String(bytes);
}
public void Dispose()
{
_httpClient?.Dispose();
}
}
}
الآن دعونا ندمج الدلالة مع خط أنابيب RAG الحالي:
using Microsoft.Extensions.Logging;
using Mostlylucid.BlogLLM.Core.Models;
namespace Mostlylucid.BlogLLM.Core.Services
{
public class DocumentIngestionService
{
private readonly ILogger<DocumentIngestionService> _logger;
private readonly DoclingClient _doclingClient;
private readonly MarkdownParserService _markdownParser;
private readonly ChunkingService _chunker;
private readonly BatchEmbeddingService _embedder;
private readonly QdrantVectorStore _vectorStore;
public DocumentIngestionService(
ILogger<DocumentIngestionService> logger,
DoclingClient doclingClient,
MarkdownParserService markdownParser,
ChunkingService chunker,
BatchEmbeddingService embedder,
QdrantVectorStore vectorStore)
{
_logger = logger;
_doclingClient = doclingClient;
_markdownParser = markdownParser;
_chunker = chunker;
_embedder = embedder;
_vectorStore = vectorStore;
}
/// <summary>
/// Process a document file and add to vector store
/// </summary>
public async Task<bool> ProcessDocumentAsync(
string filePath,
string[]? categories = null,
CancellationToken cancellationToken = default)
{
try
{
_logger.LogInformation("Processing document: {FilePath}", filePath);
// Step 1: Convert with Docling
var document = await _doclingClient.ConvertFileUploadAsync(
filePath, categories, cancellationToken);
if (document == null)
{
_logger.LogError("Failed to convert document: {FilePath}", filePath);
return false;
}
return await ProcessConvertedDocumentAsync(document, cancellationToken);
}
catch (Exception ex)
{
_logger.LogError(ex, "Error processing document {FilePath}", filePath);
return false;
}
}
/// <summary>
/// Process a document from URL and add to vector store
/// </summary>
public async Task<bool> ProcessDocumentFromUrlAsync(
string url,
string[]? categories = null,
CancellationToken cancellationToken = default)
{
try
{
_logger.LogInformation("Processing document from URL: {Url}", url);
// Step 1: Convert with Docling
var document = await _doclingClient.ConvertFromUrlAsync(
url, categories, cancellationToken);
if (document == null)
{
_logger.LogError("Failed to convert document from URL: {Url}", url);
return false;
}
return await ProcessConvertedDocumentAsync(document, cancellationToken);
}
catch (Exception ex)
{
_logger.LogError(ex, "Error processing document from URL {Url}", url);
return false;
}
}
private async Task<bool> ProcessConvertedDocumentAsync(
ProcessedDocument document,
CancellationToken cancellationToken)
{
_logger.LogInformation("Document converted: {FileName} ({PageCount} pages)",
document.FileName, document.PageCount ?? 0);
// Step 2: Parse the markdown content
var post = _markdownParser.ParseMarkdownFromContent(
document.MarkdownContent,
document.FileName);
post.Categories = document.Categories;
// Step 3: Chunk the content
var chunks = _chunker.ChunkBlogPost(post);
_logger.LogInformation("Created {ChunkCount} chunks from {FileName}",
chunks.Count, document.FileName);
if (chunks.Count == 0)
{
_logger.LogWarning("No chunks created for document: {FileName}", document.FileName);
return false;
}
// Step 4: Generate embeddings
var progress = new Progress<int>(processed =>
{
_logger.LogDebug("Embedded {Processed}/{Total} chunks",
processed, chunks.Count);
});
await _embedder.GenerateEmbeddingsAsync(chunks, progress, cancellationToken);
// Step 5: Store in vector database
await _vectorStore.UpsertChunksAsync(chunks);
_logger.LogInformation("Successfully processed document: {FileName} ({ChunkCount} chunks)",
document.FileName, chunks.Count);
return true;
}
/// <summary>
/// Batch process multiple documents
/// </summary>
public async Task<(int success, int failed)> ProcessDocumentBatchAsync(
IEnumerable<string> filePaths,
string[]? categories = null,
CancellationToken cancellationToken = default)
{
int success = 0;
int failed = 0;
foreach (var filePath in filePaths)
{
if (cancellationToken.IsCancellationRequested)
break;
var result = await ProcessDocumentAsync(filePath, categories, cancellationToken);
if (result)
success++;
else
failed++;
}
_logger.LogInformation("Batch processing complete: {Success} succeeded, {Failed} failed",
success, failed);
return (success, failed);
}
}
}
using Microsoft.Extensions.DependencyInjection;
using Mostlylucid.BlogLLM.Core.Services;
public static class ServiceCollectionExtensions
{
public static IServiceCollection AddDoclingServices(
this IServiceCollection services,
Action<DoclingClientOptions>? configureOptions = null)
{
// Configure options
if (configureOptions != null)
{
services.Configure(configureOptions);
}
else
{
services.Configure<DoclingClientOptions>(options =>
{
options.BaseUrl = "http://localhost:5001";
options.TimeoutSeconds = 300;
options.EnableOcr = true;
});
}
// Register HttpClient with configuration
services.AddHttpClient<DoclingClient>((serviceProvider, client) =>
{
var options = serviceProvider
.GetRequiredService<IOptions<DoclingClientOptions>>().Value;
client.BaseAddress = new Uri(options.BaseUrl);
client.Timeout = TimeSpan.FromSeconds(options.TimeoutSeconds);
});
// Register services
services.AddScoped<DocumentIngestionService>();
return services;
}
}
إلى appsettings.json:
{
"Docling": {
"BaseUrl": "http://localhost:5001",
"TimeoutSeconds": 300,
"EnableOcr": true,
"TableMode": "accurate"
}
}
// PDF Processing
await documentIngestionService.ProcessDocumentAsync(
"C:\\documents\\client_contract.pdf",
new[] { "legal", "contracts", "client-agreements" });
// DOCX Processing
await documentIngestionService.ProcessDocumentAsync(
"C:\\documents\\motion_to_dismiss.docx",
new[] { "legal", "briefs", "motions" });
// URL Processing (great for public documents)
await documentIngestionService.ProcessDocumentFromUrlAsync(
"https://arxiv.org/pdf/2501.17887",
new[] { "research", "ai", "docling" });
// Batch Processing
var files = Directory.GetFiles("C:\\documents\\legal", "*.pdf");
var (success, failed) = await documentIngestionService.ProcessDocumentBatchAsync(
files,
new[] { "legal", "batch-import" });
Console.WriteLine($"Processed {success} files, {failed} failures");
public class DocumentWatcherService : BackgroundService
{
private readonly IServiceProvider _serviceProvider;
private readonly ILogger<DocumentWatcherService> _logger;
private FileSystemWatcher? _watcher;
private readonly string _watchPath;
public DocumentWatcherService(
IServiceProvider serviceProvider,
ILogger<DocumentWatcherService> logger,
IConfiguration configuration)
{
_serviceProvider = serviceProvider;
_logger = logger;
_watchPath = configuration["DocumentWatch:Path"] ?? "C:\\documents\\incoming";
}
protected override Task ExecuteAsync(CancellationToken stoppingToken)
{
if (!Directory.Exists(_watchPath))
{
Directory.CreateDirectory(_watchPath);
}
_watcher = new FileSystemWatcher(_watchPath)
{
NotifyFilter = NotifyFilters.FileName | NotifyFilters.LastWrite,
IncludeSubdirectories = false
};
// Watch for common document types
_watcher.Filters.Add("*.pdf");
_watcher.Filters.Add("*.docx");
_watcher.Filters.Add("*.doc");
_watcher.Filters.Add("*.pptx");
_watcher.Created += OnFileCreated;
_watcher.EnableRaisingEvents = true;
_logger.LogInformation("Watching for documents in: {Path}", _watchPath);
return Task.CompletedTask;
}
private async void OnFileCreated(object sender, FileSystemEventArgs e)
{
_logger.LogInformation("New document detected: {FileName}", e.Name);
// Wait for file to be fully written
await Task.Delay(1000);
using var scope = _serviceProvider.CreateScope();
var ingestionService = scope.ServiceProvider
.GetRequiredService<DocumentIngestionService>();
await ingestionService.ProcessDocumentAsync(e.FullPath);
}
public override void Dispose()
{
_watcher?.Dispose();
base.Dispose();
}
}
نوع وثيقة من نوع الوثائق |--------------|-------------|-------------------|----------------| PDF البسيط (1-5 صفحات) / 100KB- 500KB ° 2-5 ثوان □ 0.5-1 ثانية □ PDF) (20+ صفحات) □ 1-5 ميغابايت □ 10-30 ثانية □ 2-5 ثوان □ PDF الممسوحة مع OCR o 1-10 ميغابايت o 30 - 120 ثانية o 2 - 5 ثوان o وثيقة DOCX وثيقة DOCX عرض عرض PPPTX عرض عرض 1-20 ميغابايم - 1 - 20 ميغابايت 5 - 30 ثانية □ 1 - 3 ثا
docling-serve-cu126 أو docling-serve-cu128)fast لا شيء** عندما تكون دقة الجدول غير حرجةلقد نجحنا في دمج التعميم في نظام محامينا GPT، مما يمكّن من:
وهذا يوسع قاعدة معرفتنا بما يتجاوز المدونات لتشمل الوثائق القانونية والعقود والورقات البحثية وغيرها من المواد الهامة.
© 2026 Scott Galloway — Unlicense — All content and source code on this site is free to use, copy, modify, and sell.