欢迎加入 第九部分! 在前几部分, 我们建立了强大的RAG 系统, 处理标记博客文章, 并可以通过语义嵌入搜索它们。 现在是时候扩大我们的能力了。 停 停 停 停 停 停 停 停 停 停 停 停 - 一个强大的文件处理图书馆,能够吸收PDFs、DOCX和其他格式,使我们的律师GPT真正包罗万象。

注:这是我在AI(协助起草)+我自己编辑的实验的一部分。同一声音,同样的务实;只是更快的手指。

这并不是说我需要这个来写博客(这是所有的标记),而是要完成,我想我应该展示如何在你的RAG管道中轻松添加文件接收能力。如果你正在建立一个需要处理法律文件、合同或其他商业文件的系统,这特别有用。

为何文件接收事项须提交法律大赦国际

现代法律惯例不仅涉及文本,还涉及各种文件格式。

• PDF 合同 及法律协定或法律协定
• DOCX书状 动议和动议
• 电子邮件线索 来往公文和来往公文
• 扫描文件 (通过OCR)
• 电子表格 与案例数据

文档使我们的律师GPT能够处理这些格式,使其可以搜索,创造一个真正全面的知识库,反映现代律师事务所如何使用大赦国际。

什么是多林号?

停 停 停 停 停 停 停 停 停 停 停 停 是一个来自IBM的开放源码文件处理工具包,用于:

• 将 PDFs、 DOCX、 PPTX、 HTML 和其他格式转换成结构标记或 JSON 格式
• 保存格式、表格和文档结构
• 支持文件扫描扫描文件的OCR
• 作为 docker 服务运行通过 文档服务
• 展出REST API,方便与任何语言融合

设置备注服务

Docker 服务部署

DoclingService是运行 Docling服务的最简单的方式。让我们设置它:

# Using the official container image from Quay.io
docker run -p 5001:5001 quay.io/docling-project/docling-serve

# Or with the UI enabled for testing
docker run -p 5001:5001 -e DOCLING_SERVE_ENABLE_UI=1 quay.io/docling-project/docling-serve

现有集装箱图像:

|-------|-------------|------| | quay.io/docling-project/docling-serve 基本图像( PyPI 软件包) ~ 8. 7GB (amd64) | quay.io/docling-project/docling-serve-cpu CPU - 唯一的变式 ~4. 4GB | quay.io/docling-project/docling-serve-cu126 -=YTET -伊甸园字幕组=-伊甸园字幕组=-伊甸园字幕组=-伊甸园字幕组=-伊甸园字幕组=- 翻译: | quay.io/docling-project/docling-serve-cu128 -=YTET -伊甸园字幕组=- 翻译组=- 翻译组=- 翻译组=- 翻译组=- 翻译组=- 翻译组=- 翻译组=- 翻译组=- 翻译组=- 翻译组=- 翻译组=- 翻译组=- 翻译组=- 翻译组=- 翻译组=- 翻译组=- 翻译组=- 翻译组=- 翻译组=- 翻译组=- 翻译组=- 翻译组=- 翻译组=- 翻译组=

终点 :

• API : http://localhost:5001
• API 文件( 斯瓦格): http://localhost:5001/docs
• UI 游戏场 : http://localhost:5001/ui (一旦启用)

嵌入器合成设置

生产时,在您现有产品中添加 docling docker-compose.yml:

services:
  docling:
    image: quay.io/docling-project/docling-serve:latest
    ports:
      - "5001:5001"
    environment:
      - DOCLING_SERVE_ENABLE_UI=0
      - DOCLING_SERVE_MAX_WORKERS=4
    volumes:
      - docling_cache:/root/.cache
    restart: unless-stopped
    
volumes:
  docling_cache:

使用卷卷纸做试验

在与 C# 整合之前, 让我们测试 API :

# Convert a PDF from URL
curl -X 'POST' \
  'http://localhost:5001/v1/convert/source' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "sources": [{"kind": "http", "url": "https://arxiv.org/pdf/2501.17887"}],
    "options": {
      "to_formats": ["md"]
    }
  }'

# Convert a local file (upload)
curl -X 'POST' \
  'http://localhost:5001/v1/convert/file' \
  -H 'accept: application/json' \
  -F '[email protected]'

C# 融合

多克林是一个Python服务, 我们将通过 HTTP 整合。 让我们建立一个强大的 C# 客户端 。

使用 API 模型

using System.Text.Json.Serialization;

namespace Mostlylucid.BlogLLM.Core.Models
{
    /// <summary>
    /// Request to convert documents from URLs or base64 content
    /// </summary>

    public class DoclingConvertRequest
    {
        [JsonPropertyName("sources")]
        public List<DoclingSource> Sources { get; set; } = new();

        [JsonPropertyName("options")]
        public DoclingOptions? Options { get; set; }
    }

    public class DoclingSource
    {
        [JsonPropertyName("kind")]
        public string Kind { get; set; } = "http"; // "http", "base64", "file"

        [JsonPropertyName("url")]
        public string? Url { get; set; }

        [JsonPropertyName("base64")]
        public string? Base64Content { get; set; }

        [JsonPropertyName("filename")]
        public string? Filename { get; set; }
    }

    public class DoclingOptions
    {
        [JsonPropertyName("to_formats")]
        public List<string> ToFormats { get; set; } = new() { "md" }; // "md", "json", "text"

        [JsonPropertyName("ocr")]
        public bool Ocr { get; set; } = true;

        [JsonPropertyName("table_mode")]
        public string TableMode { get; set; } = "accurate"; // "fast", "accurate"
    }

    /// <summary>
    /// Response from Docling conversion
    /// </summary>

    public class DoclingConvertResponse
    {
        [JsonPropertyName("document")]
        public DoclingDocument? Document { get; set; }

        [JsonPropertyName("status")]
        public string Status { get; set; } = string.Empty;

        [JsonPropertyName("errors")]
        public List<string>? Errors { get; set; }
    }

    public class DoclingDocument
    {
        [JsonPropertyName("md_content")]
        public string? MarkdownContent { get; set; }

        [JsonPropertyName("json_content")]
        public object? JsonContent { get; set; }

        [JsonPropertyName("text_content")]
        public string? TextContent { get; set; }

        [JsonPropertyName("metadata")]
        public DoclingMetadata? Metadata { get; set; }
    }

    public class DoclingMetadata
    {
        [JsonPropertyName("filename")]
        public string? Filename { get; set; }

        [JsonPropertyName("page_count")]
        public int? PageCount { get; set; }

        [JsonPropertyName("file_type")]
        public string? FileType { get; set; }
    }

    /// <summary>
    /// Our internal document model
    /// </summary>

    public class ProcessedDocument
    {
        public string DocumentId { get; set; } = Guid.NewGuid().ToString();
        public string FileName { get; set; } = string.Empty;
        public string OriginalFormat { get; set; } = string.Empty;
        public string MarkdownContent { get; set; } = string.Empty;
        public string? TextContent { get; set; }
        public DateTime ProcessedDate { get; set; } = DateTime.UtcNow;
        public string[] Categories { get; set; } = Array.Empty<string>();
        public int? PageCount { get; set; }
        public string ContentHash { get; set; } = string.Empty;
    }
}

客户服务

using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using Mostlylucid.BlogLLM.Core.Models;
using System.Net.Http.Json;
using System.Security.Cryptography;
using System.Text;
using System.Text.Json;

namespace Mostlylucid.BlogLLM.Core.Services
{
    public class DoclingClientOptions
    {
        public string BaseUrl { get; set; } = "http://localhost:5001";
        public int TimeoutSeconds { get; set; } = 300; // 5 minutes for large documents
        public bool EnableOcr { get; set; } = true;
        public string TableMode { get; set; } = "accurate";
    }

    public class DoclingClient : IDisposable
    {
        private readonly HttpClient _httpClient;
        private readonly ILogger<DoclingClient> _logger;
        private readonly DoclingClientOptions _options;

        public DoclingClient(
            HttpClient httpClient,
            ILogger<DoclingClient> logger,
            IOptions<DoclingClientOptions> options)
        {
            _httpClient = httpClient;
            _logger = logger;
            _options = options.Value;

            _httpClient.BaseAddress = new Uri(_options.BaseUrl);
            _httpClient.Timeout = TimeSpan.FromSeconds(_options.TimeoutSeconds);
        }

        /// <summary>
        /// Convert a document from a URL
        /// </summary>

        public async Task<ProcessedDocument?> ConvertFromUrlAsync(
            string url,
            string[]? categories = null,
            CancellationToken cancellationToken = default)
        {
            _logger.LogInformation("Converting document from URL: {Url}", url);

            var request = new DoclingConvertRequest
            {
                Sources = new List<DoclingSource>
                {
                    new() { Kind = "http", Url = url }
                },
                Options = new DoclingOptions
                {
                    ToFormats = new List<string> { "md", "text" },
                    Ocr = _options.EnableOcr,
                    TableMode = _options.TableMode
                }
            };

            return await SendConversionRequestAsync(request, categories, cancellationToken);
        }

        /// <summary>
        /// Convert a local file
        /// </summary>

        public async Task<ProcessedDocument?> ConvertFileAsync(
            string filePath,
            string[]? categories = null,
            CancellationToken cancellationToken = default)
        {
            if (!File.Exists(filePath))
            {
                _logger.LogError("File not found: {FilePath}", filePath);
                return null;
            }

            _logger.LogInformation("Converting local file: {FilePath}", filePath);

            // Read file and convert to base64
            var fileBytes = await File.ReadAllBytesAsync(filePath, cancellationToken);
            var base64Content = Convert.ToBase64String(fileBytes);
            var fileName = Path.GetFileName(filePath);

            var request = new DoclingConvertRequest
            {
                Sources = new List<DoclingSource>
                {
                    new()
                    {
                        Kind = "base64",
                        Base64Content = base64Content,
                        Filename = fileName
                    }
                },
                Options = new DoclingOptions
                {
                    ToFormats = new List<string> { "md", "text" },
                    Ocr = _options.EnableOcr,
                    TableMode = _options.TableMode
                }
            };

            return await SendConversionRequestAsync(request, categories, cancellationToken);
        }

        /// <summary>
        /// Convert a file using multipart form upload (more efficient for large files)
        /// </summary>

        public async Task<ProcessedDocument?> ConvertFileUploadAsync(
            string filePath,
            string[]? categories = null,
            CancellationToken cancellationToken = default)
        {
            if (!File.Exists(filePath))
            {
                _logger.LogError("File not found: {FilePath}", filePath);
                return null;
            }

            _logger.LogInformation("Uploading and converting file: {FilePath}", filePath);

            try
            {
                using var fileStream = File.OpenRead(filePath);
                using var content = new MultipartFormDataContent();
                using var streamContent = new StreamContent(fileStream);

                var fileName = Path.GetFileName(filePath);
                content.Add(streamContent, "files", fileName);

                var response = await _httpClient.PostAsync(
                    "/v1/convert/file",
                    content,
                    cancellationToken);

                if (!response.IsSuccessStatusCode)
                {
                    var errorContent = await response.Content.ReadAsStringAsync(cancellationToken);
                    _logger.LogError("Docling conversion failed: {StatusCode} - {Error}",
                        response.StatusCode, errorContent);
                    return null;
                }

                var result = await response.Content.ReadFromJsonAsync<DoclingConvertResponse>(
                    cancellationToken: cancellationToken);

                return MapToProcessedDocument(result, fileName, categories);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error uploading file to Docling: {FilePath}", filePath);
                return null;
            }
        }

        private async Task<ProcessedDocument?> SendConversionRequestAsync(
            DoclingConvertRequest request,
            string[]? categories,
            CancellationToken cancellationToken)
        {
            try
            {
                var response = await _httpClient.PostAsJsonAsync(
                    "/v1/convert/source",
                    request,
                    cancellationToken);

                if (!response.IsSuccessStatusCode)
                {
                    var errorContent = await response.Content.ReadAsStringAsync(cancellationToken);
                    _logger.LogError("Docling conversion failed: {StatusCode} - {Error}",
                        response.StatusCode, errorContent);
                    return null;
                }

                var result = await response.Content.ReadFromJsonAsync<DoclingConvertResponse>(
                    cancellationToken: cancellationToken);

                var filename = request.Sources.FirstOrDefault()?.Filename
                    ?? request.Sources.FirstOrDefault()?.Url
                    ?? "unknown";

                return MapToProcessedDocument(result, filename, categories);
            }
            catch (HttpRequestException ex)
            {
                _logger.LogError(ex, "HTTP error calling Docling API");
                return null;
            }
            catch (TaskCanceledException ex)
            {
                _logger.LogError(ex, "Docling conversion timed out");
                return null;
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Unexpected error calling Docling API");
                return null;
            }
        }

        private ProcessedDocument? MapToProcessedDocument(
            DoclingConvertResponse? response,
            string filename,
            string[]? categories)
        {
            if (response?.Document == null)
            {
                _logger.LogWarning("Docling returned empty document");
                return null;
            }

            var markdownContent = response.Document.MarkdownContent ?? string.Empty;

            return new ProcessedDocument
            {
                DocumentId = Guid.NewGuid().ToString(),
                FileName = Path.GetFileName(filename),
                OriginalFormat = Path.GetExtension(filename).TrimStart('.').ToLower(),
                MarkdownContent = markdownContent,
                TextContent = response.Document.TextContent,
                ProcessedDate = DateTime.UtcNow,
                Categories = categories ?? Array.Empty<string>(),
                PageCount = response.Document.Metadata?.PageCount,
                ContentHash = ComputeHash(markdownContent)
            };
        }

        private static string ComputeHash(string content)
        {
            using var sha256 = SHA256.Create();
            var bytes = sha256.ComputeHash(Encoding.UTF8.GetBytes(content));
            return Convert.ToBase64String(bytes);
        }

        public void Dispose()
        {
            _httpClient?.Dispose();
        }
    }
}

文件摄取服务

现在让我们把杜克林 和我们现有的RAG输油管融合起来:

using Microsoft.Extensions.Logging;
using Mostlylucid.BlogLLM.Core.Models;

namespace Mostlylucid.BlogLLM.Core.Services
{
    public class DocumentIngestionService
    {
        private readonly ILogger<DocumentIngestionService> _logger;
        private readonly DoclingClient _doclingClient;
        private readonly MarkdownParserService _markdownParser;
        private readonly ChunkingService _chunker;
        private readonly BatchEmbeddingService _embedder;
        private readonly QdrantVectorStore _vectorStore;

        public DocumentIngestionService(
            ILogger<DocumentIngestionService> logger,
            DoclingClient doclingClient,
            MarkdownParserService markdownParser,
            ChunkingService chunker,
            BatchEmbeddingService embedder,
            QdrantVectorStore vectorStore)
        {
            _logger = logger;
            _doclingClient = doclingClient;
            _markdownParser = markdownParser;
            _chunker = chunker;
            _embedder = embedder;
            _vectorStore = vectorStore;
        }

        /// <summary>
        /// Process a document file and add to vector store
        /// </summary>

        public async Task<bool> ProcessDocumentAsync(
            string filePath,
            string[]? categories = null,
            CancellationToken cancellationToken = default)
        {
            try
            {
                _logger.LogInformation("Processing document: {FilePath}", filePath);

                // Step 1: Convert with Docling
                var document = await _doclingClient.ConvertFileUploadAsync(
                    filePath, categories, cancellationToken);

                if (document == null)
                {
                    _logger.LogError("Failed to convert document: {FilePath}", filePath);
                    return false;
                }

                return await ProcessConvertedDocumentAsync(document, cancellationToken);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error processing document {FilePath}", filePath);
                return false;
            }
        }

        /// <summary>
        /// Process a document from URL and add to vector store
        /// </summary>

        public async Task<bool> ProcessDocumentFromUrlAsync(
            string url,
            string[]? categories = null,
            CancellationToken cancellationToken = default)
        {
            try
            {
                _logger.LogInformation("Processing document from URL: {Url}", url);

                // Step 1: Convert with Docling
                var document = await _doclingClient.ConvertFromUrlAsync(
                    url, categories, cancellationToken);

                if (document == null)
                {
                    _logger.LogError("Failed to convert document from URL: {Url}", url);
                    return false;
                }

                return await ProcessConvertedDocumentAsync(document, cancellationToken);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error processing document from URL {Url}", url);
                return false;
            }
        }

        private async Task<bool> ProcessConvertedDocumentAsync(
            ProcessedDocument document,
            CancellationToken cancellationToken)
        {
            _logger.LogInformation("Document converted: {FileName} ({PageCount} pages)",
                document.FileName, document.PageCount ?? 0);

            // Step 2: Parse the markdown content
            var post = _markdownParser.ParseMarkdownFromContent(
                document.MarkdownContent,
                document.FileName);
            post.Categories = document.Categories;

            // Step 3: Chunk the content
            var chunks = _chunker.ChunkBlogPost(post);
            _logger.LogInformation("Created {ChunkCount} chunks from {FileName}",
                chunks.Count, document.FileName);

            if (chunks.Count == 0)
            {
                _logger.LogWarning("No chunks created for document: {FileName}", document.FileName);
                return false;
            }

            // Step 4: Generate embeddings
            var progress = new Progress<int>(processed =>
            {
                _logger.LogDebug("Embedded {Processed}/{Total} chunks",
                    processed, chunks.Count);
            });

            await _embedder.GenerateEmbeddingsAsync(chunks, progress, cancellationToken);

            // Step 5: Store in vector database
            await _vectorStore.UpsertChunksAsync(chunks);

            _logger.LogInformation("Successfully processed document: {FileName} ({ChunkCount} chunks)",
                document.FileName, chunks.Count);

            return true;
        }

        /// <summary>
        /// Batch process multiple documents
        /// </summary>

        public async Task<(int success, int failed)> ProcessDocumentBatchAsync(
            IEnumerable<string> filePaths,
            string[]? categories = null,
            CancellationToken cancellationToken = default)
        {
            int success = 0;
            int failed = 0;

            foreach (var filePath in filePaths)
            {
                if (cancellationToken.IsCancellationRequested)
                    break;

                var result = await ProcessDocumentAsync(filePath, categories, cancellationToken);
                if (result)
                    success++;
                else
                    failed++;
            }

            _logger.LogInformation("Batch processing complete: {Success} succeeded, {Failed} failed",
                success, failed);

            return (success, failed);
        }
    }
}

依赖注射设置

using Microsoft.Extensions.DependencyInjection;
using Mostlylucid.BlogLLM.Core.Services;

public static class ServiceCollectionExtensions
{
    public static IServiceCollection AddDoclingServices(
        this IServiceCollection services,
        Action<DoclingClientOptions>? configureOptions = null)
    {
        // Configure options
        if (configureOptions != null)
        {
            services.Configure(configureOptions);
        }
        else
        {
            services.Configure<DoclingClientOptions>(options =>
            {
                options.BaseUrl = "http://localhost:5001";
                options.TimeoutSeconds = 300;
                options.EnableOcr = true;
            });
        }

        // Register HttpClient with configuration
        services.AddHttpClient<DoclingClient>((serviceProvider, client) =>
        {
            var options = serviceProvider
                .GetRequiredService<IOptions<DoclingClientOptions>>().Value;
            client.BaseAddress = new Uri(options.BaseUrl);
            client.Timeout = TimeSpan.FromSeconds(options.TimeoutSeconds);
        });

        // Register services
        services.AddScoped<DocumentIngestionService>();

        return services;
    }
}

配置配置配置

添加到您的 appsettings.json:

{
  "Docling": {
    "BaseUrl": "http://localhost:5001",
    "TimeoutSeconds": 300,
    "EnableOcr": true,
    "TableMode": "accurate"
  }
}

使用率示例

处理不同文件类型

// PDF Processing
await documentIngestionService.ProcessDocumentAsync(
    "C:\\documents\\client_contract.pdf",
    new[] { "legal", "contracts", "client-agreements" });

// DOCX Processing
await documentIngestionService.ProcessDocumentAsync(
    "C:\\documents\\motion_to_dismiss.docx",
    new[] { "legal", "briefs", "motions" });

// URL Processing (great for public documents)
await documentIngestionService.ProcessDocumentFromUrlAsync(
    "https://arxiv.org/pdf/2501.17887",
    new[] { "research", "ai", "docling" });

// Batch Processing
var files = Directory.GetFiles("C:\\documents\\legal", "*.pdf");
var (success, failed) = await documentIngestionService.ProcessDocumentBatchAsync(
    files,
    new[] { "legal", "batch-import" });
Console.WriteLine($"Processed {success} files, {failed} failures");

自动摄入文件监视器

public class DocumentWatcherService : BackgroundService
{
    private readonly IServiceProvider _serviceProvider;
    private readonly ILogger<DocumentWatcherService> _logger;
    private FileSystemWatcher? _watcher;
    private readonly string _watchPath;

    public DocumentWatcherService(
        IServiceProvider serviceProvider,
        ILogger<DocumentWatcherService> logger,
        IConfiguration configuration)
    {
        _serviceProvider = serviceProvider;
        _logger = logger;
        _watchPath = configuration["DocumentWatch:Path"] ?? "C:\\documents\\incoming";
    }

    protected override Task ExecuteAsync(CancellationToken stoppingToken)
    {
        if (!Directory.Exists(_watchPath))
        {
            Directory.CreateDirectory(_watchPath);
        }

        _watcher = new FileSystemWatcher(_watchPath)
        {
            NotifyFilter = NotifyFilters.FileName | NotifyFilters.LastWrite,
            IncludeSubdirectories = false
        };

        // Watch for common document types
        _watcher.Filters.Add("*.pdf");
        _watcher.Filters.Add("*.docx");
        _watcher.Filters.Add("*.doc");
        _watcher.Filters.Add("*.pptx");

        _watcher.Created += OnFileCreated;
        _watcher.EnableRaisingEvents = true;

        _logger.LogInformation("Watching for documents in: {Path}", _watchPath);

        return Task.CompletedTask;
    }

    private async void OnFileCreated(object sender, FileSystemEventArgs e)
    {
        _logger.LogInformation("New document detected: {FileName}", e.Name);

        // Wait for file to be fully written
        await Task.Delay(1000);

        using var scope = _serviceProvider.CreateScope();
        var ingestionService = scope.ServiceProvider
            .GetRequiredService<DocumentIngestionService>();

        await ingestionService.ProcessDocumentAsync(e.FullPath);
    }

    public override void Dispose()
    {
        _watcher?.Dispose();
        base.Dispose();
    }
}

绩效考量

文件大小影响

• • 文件类型 * * 典型大小 * * 处理文件 * * 嵌入时间 * * * |--------------|-------------|-------------------|----------------| 简单 PDF(1-5页) 100KB-500KB 2-5秒 0.5-1秒 复杂PDF (20+页) 1-5MB 10-30 秒 2-5 秒 1 -10 MB 30 -120秒 2 -5秒 -=YTET -伊甸园字幕组=- 翻译: PPTX 演示 = 1 -20MB = 5 -30秒 = 1 -3秒

优化提示

1. 使用 GPU 图像 重度工作负担(重度工作负担(重度工作负担docling-serve-cu126 或 docling-serve-cu128)
2. 禁用 OCR 数字本地PDF 加速处理
3. 使用使用 fast 表格表格模式 当表格精确度不关键时
4. 批次处理过程 大型文档集超峰值时数
5. 缓存结果 - 储存内容散装物以避免后处理无改动的文档

摘要摘要摘要

我们成功地将Docling 融入了我们的律师GPT系统 使:

1. PDFs、DOCX、PPTX和其他文件格式的处理
2. OCR 扫描文件支持
3. 与我们现有的RAG管道进行清洁整合
4. 用于缩放的嵌入部署
5. 批次处理能力
6. 自动文件, 监视亲手摄入

包括法律文件、合同、研究论文及其他重要材料。

系列导航

• 第1部分:导言和建筑
• 第2部分:C#中的GPU设置和CUDA CUDA
• 第三部分:了解嵌入和矢量数据库
• 第4部分:建设吸收管道
• 第5部分:视窗客户端
• 第6部分:地方LLM整合
• 第7部分:内容生成和即时工程
• 第8部分:先进地物和生产部署
• 第9部分:文件对文件页数的摄取 (本员额)

资源资源资源 资源资源资源 资源资源 资源资源

• 文档 GitHub 仓库
• KitHub 储存库
• 文档服务文件
• API文件服务 (当当地运行时)
• ArXiv 纸