# 视频缩影@: 减少了视频的 RAG @ (#Shots @→场景 @MS K3}证据=) > **状态状况现状现况***:* 发展是其中的一部分 [更清晰](https://www.lucidrag.com). > **资料来源来源**: [GithubMS .com/scottgalMS K2卢西德拉格](https://github.com/scottgal/lucidrag) **何处适合**:视频拼音器是 **管弦管** 会 议 会 排 排 会 会 会 议 会 会总 会 会 [更清晰](https://www.lucidrag.com) 将三条管道合并成统一的视频分析引擎 @: - **[DocSummamer 缩写器](/blog/building-a-document-summarizer-with-rag)** - 文档 * @(_实体提取 *,}知识图表=)* - **[图像合成器](/blog/constrained-fuzzy-image-intelligence)** @-_图像 @ (22-_ 波浪视觉智能@,_ CLIP 嵌入,}(OCRQZ)) @ - **[音频合成器](/blog/audiosummarizer-forensic-audio-characterization)** @-音响@(声波剖析 ,喇叭喇叭 - **视频合成器** @(_这篇文章@)}#-}影片 @(orchestrates 全部三个{,}都添加了枪声*/scene结构 *)} - **[数据合成器](/blog/datasummarizer-how-it-works)** - 數據(schema inferenceMS K2 剖析_)} 都跟着一样 **[降低的RAG模式](/blog/reduced-rag)**“:” 提取信号,一旦“,” 存储证据, 用捆绑的 LLM 输入来合成 --- 在CLIP嵌入的电影框中处理两个-小时 -}%by-}框架需要数小时和数百美元来计算 **VideoSummarizer 以三个密钥优化来解析此选项 @ :** 1. **感知性散列体衰变** 在昂贵的ML (40%前跳过视觉类似的框架 2. **批量 CLIP 嵌入** @ - process {% 1} 每个 GPU 通道的图像, 而不是一个 * MS K2x creedup_ )} 3. **管道成分** -链图像拼写器,用于各个实体的键盘@,音频缩写器 @,NER @: @ a magium principle as as [图像合成器](/blog/constrained-fuzzy-image-intelligence) 和 [音频合成器](/blog/audiosummarizer-forensic-audio-characterization)由统一的视频分析管道组成 > **核心洞察力** 视频是图片“+”音频 “+”文本“MSSK2”处理每个有专用工具的域@,}将结果合并到一致场景\. > > - **第一个进程结构** (cuts,I-framesMS K3}音频段) > - **解压缩跨-}模式信号一次** @(_embeddings@,_rought\ ,}实体_) **名词@:** - **射击** 从FFmpeg现场探测到的 ) - **场景** 从 Group_) }组组成一个一致单位的相毗连一组射线 @ (semantic,\ {\ IMSK2 - **证据** `(start_time, end_time)` + 信号 @+#指针@+}来源 **使用的密钥 ML 模型@:** - **[CLIP](https://openai.com/research/clip)** OpenAI' 维基嵌入,将视觉语义编码 - **[耳语](https://openai.com/research/whisper)** OpenAI's 语音识别模式@; 将音频记录为带有时间戳的文本 - **[BERT-纳](https://huggingface.co/dslim/bert-base-NER)** 命名实体识别@; @% 从文本中提取人们 @,}组织_,}位置 - **[NONX 运行时间](https://onnxruntime.ai/)** Crossó-platform ML 发酵;在没有框架锁定的情况下运行 CPU @/GPU 模型@-in 本篇文章涵盖 @: - 视频成像器如何搭配三条输油管线 @ (ImaageSummarizer @,音频成像机@,NER) - 从正常化到证据生成的波浪 - **能力系统**:Lazy 模型下载@,GPU检测 @,反应性路由 - 用于键盘嵌入的批量 CLIP 优化@ (3-5 @x faster @ MS K1} - 缩放框@) - 多@ - @ 信号场景群集 @ MS K1_ 编队 * +} 抄本 @ MPK3 剪切类型 @ I+ 时间 @ O) - 从记录誊本中提取实体的NER集成 - **时空原子** 限制时间估计率@,}%,}和后压 - 输出@:片段 @ ,_ shools @ MPK2}正本\ ,}作为RAG证据的文本轨迹 **相关条款**: - **[减少的RAG](https://www.mostlylucid.net/blog/reduced-rag)** 核心模式 - **[受限制的模糊模式](/blog/constrained-fuzziness-pattern)** *-* 基础模式 - **减少RAG执行量** - [DocSummamer 缩写器](/blog/building-a-document-summarizer-with-rag) - 具有实体提取功能的文件RAG - [图像合成器](/blog/constrained-fuzzy-image-intelligence) @-图像RAG 带有 @MS K1波直观情报 - [音频合成器](/blog/audiosummarizer-forensic-audio-characterization) 音频法证鉴定 - **影片缩写器 @ ( @ this article @) @** - 视频情报管弦乐队 [TOC] --- ## 问题#:视频昂贵 > **基准基准**@:}以下數字以AMD 9950X MS K2Core )/NIVIDA ANSK5#(16GB□)/{96GBRA MASK10NMMEMSKO11MS K12p H.264,Whisper base+.你的里程将会不同 一部典型的电影包含着: - **~170,000_框架** @(2_BARBAR_ 时间在#24_FPS_)> - **~2-小时音频** @(_speech},}音乐 , 效果 *)* - **多文本层** @(_Sub标题@,}incredit*\ ,}在-}屏幕文字=)} 稻草人的方法 @ ( @ nobody does this,}但是它设定了缩放Q):@ - CLIP 按框架嵌入@ : @ @ MS K1 @ ms @ I× @ * 170,000} @ MPK4# **# 9.4+#小时** - 每个框架的视野LLM= : MS K1 NSK2 I170,000 MPK4 **# 94+#小时** 闪烁的视觉API球场 马斯克1号 即便有键盘提取@( @say @,}500-1000_Brame{),}(即使按键框架提取%S'>秒的CLIP连环直导数 <.\Q} **传统方法**@"#Extrap keyrames@,_发送到VisionLLM%,}希望最佳的"* **问题**:_此刻度计算在冗余框架上 @(}%many keybrames 是视觉相似的 <),}处理它们时序@(}GPU 在框中空闲 *),}♪错过音频=/* text 信号 完全{. **解决方案**: 多-阶段过滤 *,批量处理*,和管道构成*MS K4 --- ## 视频放大器结构Name 视频放大器执行工具 **[减少的RAG](/blog/reduced-rag)** 3- 阶段缩减MSC1 ```mermaid flowchart TB subgraph Input["Video File (.mp4, .mkv, etc.)"] V[Video Stream] A[Audio Stream] end subgraph Stage1["Stage 1: Structural Analysis"] N[NormalizeWave
FFprobe metadata] SD[ShotDetectionWave
Scene cuts via FFmpeg] KE[KeyframeExtractionWave
I-frame + dedup] end subgraph Stage2["Stage 2: Content Extraction"] IS[ImageSummarizer
CLIP, OCR, Vision] AS[AudioSummarizer
Whisper, Diarization] NER[NER Service
Entity extraction] end subgraph Stage3["Stage 3: Scene Assembly"] SC[SceneClusteringWave
CLIP similarity] EV[EvidenceGenerationWave
RAG chunks] end V --> N --> SD --> KE KE --> IS A --> AS AS --> NER IS --> SC NER --> SC SC --> EV style Stage1 stroke:#22c55e,stroke-width:2px style Stage2 stroke:#3b82f6,stroke-width:2px style Stage3 stroke:#8b5cf6,stroke-width:2px ``` ### 产生的证据 在跳入执行前 @, @% 这里 @ '} 您可以得到输出的 schema@ MS K2 @ @ |Artifact | 键字段 @| 來源MS K3 |----------|------------|--------| | **场景** | `id`, `start_time`, `end_time`, `key_terms[]`, `speaker_ids[]`, `embedding[512]` @|# 场景清新瓦夫 |# | **射击** | `id`, `start_time`, `end_time`, `cut_type`, `keyframe_path` | 射擊偵測 | | **偏差** | `id`, `text`, `start_time`, `end_time`, `speaker_id`, `confidence` @|_Transcription 维系 #|# | **文本跟踪** | `id`, `text`, `start_time`, `text_type` “( ”标题“/ Credit/ 字幕“MS K3 ocr}) | 子标题除去“| ” | **关键框架** | `id`, `timestamp`, `frame_path`, `dhash`, `clip_embedding[512]` | 键盘提取量 已保存 *| 每件艺术品包括 **来源出处**:源波,处理时间戳,置信度*. 这是下游RAG查询操作的“.” ### 信号@- @Aware波管道 视频合成器使用 a **基于 signal-基波结构** 每一波都明确声明其信号合同 ```csharp public interface ISignalAwareVideoWave { /// Signals this wave requires before it can run. IReadOnlyList RequiredSignals { get; } /// Signals this wave can optionally use if available. IReadOnlyList OptionalSignals { get; } /// Signals this wave emits on successful completion. IReadOnlyList EmittedSignals { get; } /// Cache keys this wave produces for downstream waves. IReadOnlyList CacheEmits { get; } /// Cache keys this wave consumes from upstream waves. IReadOnlyList CacheUses { get; } } ``` 启用此功能 **动态波协调**: - 如果缺少所需的信号, 波会自动跳过 - 运行时依赖性解析@ ( @ no hardcoded order * ) - 部分重新运行是用信号键 < ( cache > 复制的{ )\ } - 用户界面进步颗粒度=:}每波都独立发出进展 ### @ 16-+# 已跳出管道 Keyframe 提取作为7颗粒波进行,以更好地平行和缓存效率@: | 优先需要 |Q Emits *|Q时间*|* |------|----------|----------|-------|------| | **平整平原** | 1000 | - | `video.duration`, `video.fps`, `video.normalized` # | @ @ MS K1 @ @ * |# | **FFmpeg 热探测器** | 900 | `video.normalized` | `shots.detected`, `shots.count` # | @ @ MS K1 @ @ * |# | **IFrame检测站** | 850 | `video.normalized` | `keyframes.iframes_detected`, `keyframes.iframes_count` # | @ @ MS K1 @ @ * |# | **键框架选择除** | 840 | `shots.detected`, `keyframes.iframes_detected` | `keyframes.selected`, `keyframes.selected_count` # | @ @ MS K1 @ @ * |# | **缩略图提取除** | 830 | `keyframes.selected` | `keyframes.thumbnails_extracted` # | @ @ MS K1 @ @ * |# | **关键字框应用已用** | 820 | `keyframes.thumbnails_extracted` | `keyframes.deduplicated`, `keyframes.duplicates_skipped` # | @ @ MS K1 @ @ * |# | **键框架FurllResrespeave 边边** | 810 | `keyframes.deduplicated` | `keyframes.extracted`, `keyframes.count` # | @ @ MS K1 @ @ * |# | **剪贴床边** | 800 | `keyframes.extracted` | `clip.embeddings_ready`, `clip.embeddings_count` # | @ @ MS K1 @ @ * |# | **图像分析边** | 790 | `keyframes.deduplicated` | `keyframes.analyzed`, `ocr.extracted` # | @ @ MS K1 @ @ * |# | **标题缩放** | 750 | `shots.detected` | `title.detected`, `credits.detected` # | @ @ MS K1 @ @ * |# | **音速提取除** | 650 | `video.normalized` | `audio.extracted`, `audio.path` # | @ @ MS K1 @ @ * |# | **翻译服务** | 600 | `audio.extracted` | `transcription.complete`, `transcription.utterance_count` # | @ @ MS K1 @ @ * |# | **字幕标题减号** | 550 | `video.normalized` | `subtitles.extracted` # | @ @ MS K1 @ @ * |# | **分节减法** | 500 | `video.normalized` | `chapters.extracted` # | @ @ MS K1 @ @ * |# | **现场清理** | 400 | `shots.detected` | `scenes.detected`, `scene.count` # | @ @ MS K1 @ @ * |# | **证据废地** | 100 | `scenes.detected` | `evidence.generated` # | @ @ MS K1 @ @ * |# **注::** - 图像分析网用途 `keyframes.deduplicated` @(not full@-res):OCR 运行在缩略图MS K3 视觉字幕使用全=-res, 如果能力通过路由#.}提供的话 - “3-7”是用于缓存效率的“"”键盘提取“MS K2” 子“MSSK3”管线“(7”颗粒波@).} **@2-#小时电影总数**@: ~10-15#分钟 @ ( @vs_.}没有优化的时数 @MS K4 @ ### {\fn黑体\fs22\bord1\shad0\3aHBE\4aH00\fscx67\fscy66\2cHFFFFFF\3cH808080}好的- 信号被定义为一致性的常数@: @% ```csharp public static class VideoSignals { // NormalizeWave signals public const string VideoDuration = "video.duration"; public const string VideoFps = "video.fps"; public const string VideoNormalized = "video.normalized"; // Shot detection signals public const string ShotsDetected = "shots.detected"; public const string ShotsCount = "shots.count"; // Keyframe signals public const string IframesDetected = "keyframes.iframes_detected"; public const string KeyframesSelected = "keyframes.selected"; public const string KeyframesDeduplicated = "keyframes.deduplicated"; public const string KeyframesExtracted = "keyframes.extracted"; // CLIP embedding signals public const string ClipEmbeddingsReady = "clip.embeddings_ready"; // Scene clustering signals public const string ScenesDetected = "scenes.detected"; public const string SceneCount = "scene.count"; // Transcription signals public const string TranscriptionComplete = "transcription.complete"; } ``` --- ## 能力系统@:懒惰模型 @&路由 视频合成器使用 a **以能力为主的架构@-**在启动时, : 检测一次 GPU #, 下载模型 lazily}, 路线工作到可用的组件\ MS K3# ### 型号声明@( @YAML @+}类型 @MS K2#Safe Constants_) 模型的定义如下: `models.yaml`代码中没有魔法字符串@ : @ ```yaml # models.yaml (excerpt) models: clip-vit-b32: name: "CLIP ViT-B/32" download_url: "https://huggingface.co/openai/clip-vit-base-patch32/resolve/main/onnx/visual_model.onnx" preferred_providers: [CUDAExecutionProvider, DmlExecutionProvider, CPUExecutionProvider] components: ClipEmbeddingWave: models: [clip-vit-b32] fallback_chain: [ImageAnalysisWave] ``` ```csharp // Type-safe constants (no raw strings) await coordinator.EnsureModelAsync(ModelIds.ClipVitB32); await coordinator.ActivateWaveAsync(ComponentIds.TranscriptionWave); // Route with fallback var route = await coordinator.RouteWorkAsync(new[] { ComponentIds.ClipEmbeddingWave, // Primary (GPU) ComponentIds.ImageAnalysisWave // Fallback (CPU) }); ``` ### 管道效率原子 利率限制@,时间估计 @, 和适应性回压保持UI反应,同时尽量扩大通过量\: ```csharp // Time estimation from actual data var estimator = CapabilityAtoms.CreateTimeEstimator(); using (estimator.Time("clip_embedding")) { await ProcessAsync(); } var eta = estimator.GetEstimate("clip_embedding", remaining: 50); // eta.Estimated, eta.Optimistic, eta.Pessimistic, eta.Confidence ``` > **全能力系统 docs***:* 见 `Mostlylucid.Summarizer.Core/Capabilities/` 用于 GPU 检测@,信号棒 @/sub},后压控制器 @MS K3和网状地形设计 --- ## 键优化 *1:\ 感知散装物应用 在使用视觉类似的框架运行昂贵的 CLIP 嵌入器前@ , @ VideoSummarizer 过滤器 **hash=(=dHash= )=**. ### DHash 如何工作 ```csharp public class KeyframeDeduplicationService { // dHash parameters: 9x8 grayscale = 64 bits private const int HashWidth = 9; private const int HashHeight = 8; private const int DefaultHammingThreshold = 10; public async Task ComputeDHashAsync(string imagePath, CancellationToken ct) { using var image = Image.Load(imagePath); // Resize to 9x8 (one extra column for gradient comparison) image.Mutate(x => x .Resize(HashWidth, HashHeight) .Grayscale()); ulong hash = 0; int bit = 0; // Compare adjacent pixels horizontally for (int y = 0; y < HashHeight; y++) { for (int x = 0; x < HashWidth - 1; x++) { var left = image[x, y].R; var right = image[x + 1, y].R; // Set bit if left pixel is brighter than right if (left > right) { hash |= (1UL << bit); } bit++; } } return hash; } public static int HammingDistance(ulong a, ulong b) => BitOperations.PopCount(a ^ b); } ``` **示例输出:** ``` Input: 50 keyframe candidates (from codec I-frames) Deduplication (Hamming threshold 10): Frame 0: hash=0x8f3a2c1d → KEEP (first frame) Frame 1: hash=0x8f3a2c1e → SKIP (distance=1 from frame 0) Frame 2: hash=0x8f3a2c1f → SKIP (distance=2 from frame 0) Frame 3: hash=0xc7e1b4a2 → KEEP (distance=28 from frame 0) ... Result: 50 → 30 frames (40% reduction) Processing saved: ~8 seconds of CLIP inference ``` **为什么这重要?** - ***~40% 框架缩减** 典型内容 - **<1%s 每个框架** 用于计算 cLIP ) 的 hash 计算 ( @vs@.} @ MS K2\% ] - 过滤冗余框架 **之前** 昂贵的 GPU 操作 --- ## 密钥优化 *2:批次 CLIP 嵌入 而不是在一个时间里处理一个图像@, @ VideoSummerizer 批量 @8}每个 GPU pass @ . ### 批处理架构 ```csharp public class BatchClipEmbeddingService { private const int ClipImageSize = 224; private const int DefaultBatchSize = 8; // 8 images per GPU pass public async Task> GenerateBatchEmbeddingsAsync( Dictionary framePaths, int batchSize = DefaultBatchSize, CancellationToken ct = default) { var session = await GetOrLoadClipModelAsync(ct); var results = new Dictionary(); // Pre-index batch for O(1) lookup (not batch.IndexOf!) var batches = framePaths .Select((kvp, idx) => (idx, kvp.Key, kvp.Value)) .Chunk(batchSize); foreach (var batch in batches) { // Create batch tensor [batchSize, 3, 224, 224] var tensor = new DenseTensor(new[] { batch.Length, 3, ClipImageSize, ClipImageSize }); // Preprocess images in parallel (simplified; production uses vectorised span copy) Parallel.ForEach(batch, item => { var (batchIdx, frameIndex, path) = item; var localIdx = batchIdx % batchSize; PreprocessImageToTensor(path, tensor, localIdx); // ImageSharp pixel buffers }); // Single GPU pass for entire batch var inputs = new List { NamedOnnxValue.CreateFromTensor("input", tensor) }; using var outputResults = session.Run(inputs); // Extract embeddings from batch output... } return results; } } ``` **性能比较*:** ``` Input: 30 keyframes (after deduplication) Serial processing (1 frame at a time): 30 × 200ms = 6,000ms (6.0 seconds) Batch processing (8 frames per pass): 4 batches × 350ms = 1,400ms (1.4 seconds) Speedup: 4.3x ``` **批量处理为何有效 @:** - GPU 平行法没有被充分利用, - 批次电压 `[8, 3, 224, 224]` 使用与单个图像相同的 GPU 内存 *% 1 - 运行时间优化内部的批量作业 --- ## 密钥优化 *3:管道构成 VideoSummarizer does't 重塑图像合成器或音频合成器 **连锁链** {\fn黑体\fs22\bord1\shad0\3aHBE\4aH00\fscx67\fscy66\2cHFFFFFF\3cH808080}他们... ### Keyframe Sub% -Pipeline:图像合成器集成 键盘提取法被分割成“7” 颗粒波 ( *请参见上方的浪表*).}*这里='}*协调模式显示它们是如何连接在一起的\:} ```csharp // IFrameDetectionWave → KeyframeSelectionWave → ThumbnailExtractionWave // → KeyframeDeduplicationWave → KeyframeFullResExtractionWave → ClipEmbeddingWave // ClipEmbeddingWave coordinates with ImageSummarizer public class ClipEmbeddingWave : IVideoWave, ISignalAwareVideoWave { public IReadOnlyList RequiredSignals => [VideoSignals.KeyframesExtracted]; public IReadOnlyList EmittedSignals => [VideoSignals.ClipEmbeddingsReady]; public async Task ProcessAsync(VideoContext context, CancellationToken ct) { var keyframes = context.GetCached>("keyframes.paths"); // Batch CLIP embedding (3-5x faster than serial) var embeddings = await _batchClipService.GenerateBatchEmbeddingsAsync( keyframes, batchSize: 8, ct); foreach (var (frameIndex, embedding) in embeddings) context.KeyframeEmbeddings[frameIndex] = embedding; } } // ImageAnalysisWave runs ImageSummarizer on deduplicated frames public class ImageAnalysisWave : IVideoWave, ISignalAwareVideoWave { public IReadOnlyList RequiredSignals => [VideoSignals.KeyframesDeduplicated]; public async Task ProcessAsync(VideoContext context, CancellationToken ct) { var keyframePaths = context.GetCached>("keyframes.deduplicated_paths"); foreach (var path in keyframePaths) { // Run ImageSummarizer for OCR, vision, captions var result = await _imageOrchestrator.AnalyzeAsync(path, ct); context.SetCached($"image_analysis.{Path.GetFileName(path)}", result); } } } ``` ### 音频合成器集成 音频提取和转录现在是分开的信号 -aware 波 : ```csharp // AudioExtractionWave runs first (extracts audio track from video) public class AudioExtractionWave : IVideoWave, ISignalAwareVideoWave { public IReadOnlyList RequiredSignals => [VideoSignals.VideoNormalized]; public IReadOnlyList EmittedSignals => ["audio.extracted", "audio.path"]; public async Task ProcessAsync(VideoContext context, CancellationToken ct) { var audioPath = await _ffmpegService.ExtractAudioAsync( context.VideoPath, context.WorkingDirectory, ct); context.SetCached("audio.path", audioPath); } } // TranscriptionWave depends on audio.extracted signal public class TranscriptionWave : IVideoWave, ISignalAwareVideoWave { public IReadOnlyList RequiredSignals => ["audio.extracted"]; public IReadOnlyList EmittedSignals => [ VideoSignals.TranscriptionComplete, "transcription.utterance_count" ]; public async Task ProcessAsync(VideoContext context, CancellationToken ct) { var audioPath = context.GetCached("audio.path"); // Run AudioSummarizer pipeline (Whisper + diarization) var audioProfile = await _audioOrchestrator.AnalyzeAsync(audioPath, ct); // Extract utterances with speaker info var turns = audioProfile.GetValue>("speaker.turns"); foreach (var turn in turns ?? []) { context.Utterances.Add(new Utterance { Id = Guid.NewGuid(), Text = turn.Text, StartTime = turn.StartSeconds, EndTime = turn.EndSeconds, SpeakerId = turn.SpeakerId, Confidence = turn.Confidence }); } // Run NER on full transcript for entity extraction var transcript = audioProfile.GetValue("transcription.full_text"); if (!string.IsNullOrEmpty(transcript)) { var entities = await _nerService.ExtractEntitiesAsync(transcript, ct); context.SetCached("transcript_entities", entities); // Emit entity signals by type (PER, ORG, LOC, MISC) foreach (var group in entities.GroupBy(e => e.Type)) { context.AddSignal($"transcript.entities.{group.Key.ToLowerInvariant()}", group.Select(e => e.Text).Distinct().ToList()); } } } } ``` --- ## NER 整合:命名实体识别 使用 BERT-}以 NER 为基地的 NER {(}ONNXMS K2}从笔录中提取命名实体。 ### OnnxNer 服务 ```csharp public class OnnxNerService { // Model: dslim/bert-base-NER (ONNX exported) // Entities: PER (Person), ORG (Organization), LOC (Location), MISC (Miscellaneous) public async Task> ExtractEntitiesAsync(string text, CancellationToken ct) { var entities = new List(); // Chunk long text (BERT max 512 tokens) foreach (var chunk in ChunkText(text, maxTokens: 400, overlap: 50)) { // Tokenize with WordPiece var tokens = _tokenizer.Tokenize(chunk); // Run ONNX inference var inputs = PrepareInputs(tokens); using var results = _session.Run(inputs); // Decode BIO tags var predictions = DecodePredictions(results); var chunkEntities = ExtractEntitySpans(tokens, predictions); entities.AddRange(chunkEntities); } // Deduplicate entities return entities .GroupBy(e => (e.Text.ToLowerInvariant(), e.Type)) .Select(g => g.First()) .ToList(); } } ``` **示例输出:** ``` Transcript: "Today we're speaking with John Smith from Microsoft about their new AI lab in Seattle. The project, codenamed Phoenix, builds on research from Stanford University." Entities extracted: PER: John Smith ORG: Microsoft, Stanford University LOC: Seattle MISC: Phoenix Signals emitted: transcript.entities.per = ["John Smith"] transcript.entities.org = ["Microsoft", "Stanford University"] transcript.entities.loc = ["Seattle"] transcript.entities.misc = ["Phoenix"] ``` **为什么NER对视频很重要?** - 启用像 @ " find 视频这样的查询, 提及微软@ MS K1 { - 与 DocSummerizer 实体图的链接 - 提供结构化元数据,不推断LLM --- ## 多@ - @ 信号场景集束 使用 CLIP 嵌入器单独进行现场检测, 的确“' @t 工作精密框架”在设计上是稀疏的。 “(_one per shot change@ ),}但镜头很稠密。 在 &39Q 拍摄时, “MS K4}” 插入了“1881Q shouse” (MSK5Q) 中 [(~2%+ 覆盖] @),}纯嵌入集成只产生“MS K8% 场景”,用于“MSKO9_ hour movine @MS K10 ” 。 视频合成器使用 a **mount -_signal 方针** 结合了“4”加权信号,以进行稳健的现场边界探测 “:” ### 场景 CloenClustering Wave @ : @ 多@ 多@ MS K1 @ 信号建筑 ```csharp public class SceneClusteringWave : IVideoWave, ISignalAwareVideoWave { // Signal weights for boundary scoring private const double EmbeddingWeight = 0.4; // CLIP embedding dissimilarity private const double TranscriptWeight = 0.3; // Semantic shift in transcript private const double CutTypeWeight = 0.2; // Fade/dissolve detection private const double TemporalWeight = 0.1; // Time since last scene // Temporal constraints private const double MinSceneDuration = 15.0; // Don't split scenes < 15s private const double MaxSceneDuration = 300.0; // Force split at 5 minutes private const double TargetSceneDuration = 90.0; // Prefer ~90s scenes public IReadOnlyList RequiredSignals => [VideoSignals.ShotsDetected]; public IReadOnlyList OptionalSignals => [ VideoSignals.ClipEmbeddingsReady, VideoSignals.TranscriptionComplete, VideoSignals.KeyframesDeduplicated ]; public IReadOnlyList EmittedSignals => [ VideoSignals.ScenesDetected, "scene.count", "scene.avg_duration", "scene.clustering_method" ]; private List<(int shotIndex, double score)> ComputeBoundaryScores(VideoContext context) { var shots = context.Shots.OrderBy(s => s.StartTime).ToList(); var scores = new List<(int, double)>(); // Build embedding map with nearest-neighbor interpolation var shotEmbeddings = PropagateEmbeddingsToNearbyShots(context, shots); // Build transcript windows for semantic shift detection var transcriptWindows = BuildTranscriptWindows(context, shots, windowSeconds: 10); for (int i = 0; i < shots.Count - 1; i++) { double score = 0; var currentShot = shots[i]; var nextShot = shots[i + 1]; // 1. Embedding dissimilarity (40%) if (shotEmbeddings.TryGetValue(i, out var currentEmbed) && shotEmbeddings.TryGetValue(i + 1, out var nextEmbed)) { var similarity = CosineSimilarity(currentEmbed, nextEmbed); score += (1.0 - similarity) * EmbeddingWeight; } // 2. Transcript semantic shift (30%) if (transcriptWindows.TryGetValue(i, out var currentWords) && transcriptWindows.TryGetValue(i + 1, out var nextWords)) { var overlap = currentWords.Intersect(nextWords).Count(); var union = currentWords.Union(nextWords).Count(); var jaccard = union > 0 ? (double)overlap / union : 0; score += (1.0 - jaccard) * TranscriptWeight; } // 3. Cut type signal (20%) - fades/dissolves suggest scene boundaries if (currentShot.CutType is "fade" or "dissolve") { score += CutTypeWeight; } // 4. Temporal pressure (10%) - encourage splits near target duration var timeSinceLastScene = currentShot.EndTime - GetLastSceneBoundary(); if (timeSinceLastScene > TargetSceneDuration) { var pressure = Math.Min(1.0, (timeSinceLastScene - TargetSceneDuration) / 60); score += pressure * TemporalWeight; } scores.Add((i, score)); } return scores; } } ``` ### 关键创新 1. **近邻内嵌入式孕育**“: ” 只有“~2% 」 有直接CLIP嵌入的 CLIP}. 新的方法通过时间接近重量,在“MS K3 秒内将嵌入附近的镜头。 2. **标定语义窗口**: 建構10- 第二個單字視窗, 透過 Jaccar 低廉的語言流動代理 *(low 重複 *MS K3 主题變更*).BMZK5當可用時可以使用或嵌入漂移 3. **认识切剪类型**@:_Fade-_to_-_Black and unformissions 强力表示场景边界@,}提升边界评分=.{ 4. **适应性推进权**“:”不是固定的阈值,而是从“25%”和“(”中分数的顶端选择边界。 5. **时间制约因素**“:”强制实施最小的“15”场景和部队边界(在 @5-”) **实例::** ``` Input: 1881 shots from a 2-hour movie 39 keyframes with CLIP embeddings 2302 utterances from transcript Boundary scoring per shot: Shot 45-46: embedding=0.15, transcript=0.32, cut=0.0, temporal=0.0 → score=0.156 Shot 46-47: embedding=0.08, transcript=0.12, cut=0.0, temporal=0.0 → score=0.068 Shot 47-48: embedding=0.35, transcript=0.41, cut=0.2, temporal=0.05 → score=0.388 ← BOUNDARY ... Adaptive threshold (top 25%): 0.25 Natural boundaries found: 45 Output: 47 scenes (avg 2.6 minutes per scene) - Min scene: 15.2s - Max scene: 298.4s - Total coverage: 100% Signals: scenes.detected = true scene.count = 47 scene.avg_duration = 156.3 scene.clustering_method = "multi_signal_weighted" ``` --- ## 视频信号合同 VideoSummarizer 延长了图像合成器和音频合成器的信号合同 ```csharp public record VideoSignal { public required string Key { get; init; } // "scene.count", "transcript.entities.per" public object? Value { get; init; } public double Confidence { get; init; } = 1.0; public required string Source { get; init; } // "SceneClusteringWave" // Video-specific: time range public double? StartTime { get; init; } public double? EndTime { get; init; } public DateTime Timestamp { get; init; } public Dictionary? Metadata { get; init; } public List? Tags { get; init; } // ["visual", "scene"] } public static class VideoSignalTags { public const string Visual = "visual"; public const string Audio = "audio"; public const string Speech = "speech"; public const string Ocr = "ocr"; public const string Motion = "motion"; public const string Scene = "scene"; public const string Shot = "shot"; public const string Metadata = "metadata"; } ``` **关键信号发出@: @%** -=YTET -伊甸园字幕组=- 翻译: |--------|--------|-------------| | `video.duration` 以秒计的总持续时间@| | `video.resolution` “| 正常生活”“|+Width @×QH8 |” | `video.fps` “|” 正常生活“|” 框架速率“MS K2” | `shots.count` 侦测到的射击次数 | `keyframes.count` “| ” 键盘提取除去“|” 后唯一的键盘“MSC2” | `keyframes.duplicates_skipped` 由 dHash 过滤的 | 框架 | `scene.count` @| 场景清新 {|} 相合场景片段| | `transcript.entities.per` @|_TrannprificingWave@|}来自NER的名人 {|} | `transcript.entities.org` 组织名称 | | `transcript.word_count` | Transcription Wave @| 抄本中总单词@| --- ## 视频 Pipeline:RAG 输出 缩略 `VideoPipeline` 将视频信号转换为 `ContentChunk` 对于 RAG 指数化 < :\ } ```csharp public class VideoPipeline : PipelineBase { public override string PipelineId => "video"; public override IReadOnlySet SupportedExtensions => new HashSet { ".mp4", ".mkv", ".avi", ".mov", ".wmv", ".webm", ".flv", ".m4v", ".mpeg", ".mpg" }; private List BuildContentChunks(VideoContext context, string filePath) { var chunks = new List(); // 1. Scene-based chunks (best for video retrieval) foreach (var scene in context.Scenes) { var sceneText = BuildSceneText(context, scene); var embedding = context.GetCached($"scene_centroid.{scene.Id}"); chunks.Add(new ContentChunk { Text = sceneText, ContentType = ContentType.Summary, Embedding = embedding, // Proper vector column, not metadata Metadata = new Dictionary { ["source"] = "video_scene", ["scene_id"] = scene.Id, ["key_terms"] = scene.KeyTerms, ["speakers"] = scene.SpeakerIds, ["start_time"] = scene.StartTime, ["end_time"] = scene.EndTime } }); } // 2. Transcript chunks (1-minute windows) var transcriptChunks = BuildTranscriptChunks(context, filePath); chunks.AddRange(transcriptChunks); // 3. Text track chunks (on-screen text/subtitles) foreach (var textTrack in context.TextTracks) { chunks.Add(new ContentChunk { Text = $"On-screen text: {textTrack.Text}", ContentType = ContentType.ImageOcr, Metadata = new Dictionary { ["source"] = "video_ocr", ["text_type"] = textTrack.TextType.ToString(), ["start_time"] = textTrack.StartTime } }); } return chunks; } private string BuildSceneText(VideoContext context, SceneSegment scene) { var parts = new List(); if (!string.IsNullOrEmpty(scene.Label)) parts.Add($"Scene: {scene.Label}"); parts.Add($"[{FormatTime(scene.StartTime)} - {FormatTime(scene.EndTime)}]"); if (scene.KeyTerms.Count > 0) parts.Add($"Topics: {string.Join(", ", scene.KeyTerms)}"); // Add utterances in this scene var sceneUtterances = context.Utterances .Where(u => u.StartTime >= scene.StartTime && u.EndTime <= scene.EndTime) .OrderBy(u => u.StartTime); if (sceneUtterances.Any()) parts.Add($"Speech: {string.Join(" ", sceneUtterances.Select(u => u.Text))}"); return string.Join("\n", parts); } } ``` **电影的示例输出@:** ```json { "chunks": [ { "text": "Scene: Opening montage\n[0:00 - 2:34]\nTopics: city, night, traffic\nSpeech: The year is 2049. The world has changed.", "contentType": "Summary", "metadata": { "source": "video_scene", "scene_id": "abc123", "key_terms": ["city", "night", "traffic"], "start_time": 0.0, "end_time": 154.0 } }, { "text": "The detective arrived at the crime scene. Forensics had already processed the area.", "contentType": "Transcript", "metadata": { "source": "video_transcript", "time_window": "2:34 - 3:34", "utterance_count": 4 } }, { "text": "On-screen text: LOS ANGELES 2049", "contentType": "ImageOcr", "metadata": { "source": "video_ocr", "text_type": "Title" } } ] } ``` --- ## 性能特点 ### 处理时间 @ (2-_ hour movie@ MS K1} @ I1080p) -=YTET -伊甸园字幕组=- 翻译: |-------|------|-------| | FFprobe元数据 @| @#~2#s@|} @MS K4# -=YTET -伊甸园字幕组=- 翻译: | 键盘提取@| @ @~30 @s @s|}(I500) |#dHash dedudition *|} @~0.5%s *|* *MS K4* *NSK5* *MSSK6* 框架@|* | 批量CLIP 嵌入 @| _~60s *|# MS K4框_,批量@8{|} |图像放大器 OCR | ~120s @|+50键盘和文本@|# @| @音频提取@|}#~30 @s | @FFmpeg @s|# @|_whyworld 发言时间#|_ -=YTET -伊甸园字幕组=- 翻译: |N NER提取 I|Q MS K2QS |BERT-NER 在笔录@|QZ @| @ 场景群集 @ |# @ @ ~5 @s @s| @s|} @sk4@ -=YTET字幕组=- 翻译: | **共计共计** | **分钟** | | ### 没有优化 | 优化 MS K1 储蓄 *| |--------------|---------| | dHash dedudition | MS K2 框架过滤@= @~24s CLIP 保存 *|# | 批量CLIP @| MS K2x 更快@=#~180s saved #|# |管成份@|重新使用图像Summarizer @/AudioSummerizer 波浪+| | **节余共计** | **分钟** | ### 内存使用 |% 元件 @ | @ memory # MSC2# |-----------|--------| |CLIPVIT-B/32ONNX NSK3 MS K4MB □| |耳语基地 MS K1 ~500MB| | ANAPA- TDNN| MS K3MB| |BERT-NER NSK2 MS K3MB| | **峰值** | **-=YTET -伊甸园字幕组=- 翻译:** | --- ## 与清晰合成集成 视频合成器登记为 `IPipeline` 自动路由@: @% ```csharp // In Program.cs builder.Services.AddDocSummarizer(builder.Configuration.GetSection("DocSummarizer")); builder.Services.AddDocSummarizerImages(builder.Configuration.GetSection("Images")); builder.Services.AddVideoSummarizer(); // NEW builder.Services.AddPipelineRegistry(); // Must be last // Auto-routing by extension var registry = services.GetRequiredService(); var pipeline = registry.FindForFile("movie.mp4"); // Returns VideoPipeline var result = await pipeline.ProcessAsync("movie.mp4"); ``` **支持的扩展@: @%** - `.mp4`, `.mkv`, `.avi`, `.mov`, `.wmv`, `.webm`, `.flv`, `.m4v`, `.mpeg`, `.mpg` --- ## 你得到的 - **场景-级RAG块**:集成片段,有笔录{,}关键名词_,喇叭 - **多重-}模式证据**: 視覺 K1 Keyframe 嵌入 *, 音效 (speaker diarization *MS K4 文字 *MSKO5OCR, 字幕*) - **命名实体**@:_People_,_Organizations\ ,}来自抄本 NER - **可审计来源**每个信号都有源波 , 信任 MS K2 时间戳 - **高效处理**@2-@hour 电影时间: @(_nonot#)> ## 成本多少 - **~1.5GB GPU 内存** 适合所有ONNX模型 - **~8-10 @ 分钟处理** 每部2- 小时电影 - **磁盘空间** 自动清理@)}% - **复杂程度**需要了解波的依附关系 --- ## 结论结论 视频合成器显示 **管道构成** 缩放: 1. **专用管道再利用**:唐't 重塑图像合成器或音频合成链 2. **在昂贵操作前过滤**:dHash解码成本 3. **批次 GPU 操作**@ : @ @ I8 图像/ 每過程@ MS K2 @ *3-5x crapup 4. **内容前的抽取结构**-=YTET -伊甸园字幕组=- 翻译: 5. **懒惰模式管理**@ : 下载模型只在需要时@ MS K1 自动检测 GPU 6. **反反应路由**@: 现有部件的路线工程@, 优雅地后退 结果是“:+”一部ZMK1Qhour电影变成了一个结构化的信号分类账,其中显示场景@,+Criptration @,+Peblicity_,}和嵌入功能可以用于RAG查询,如:} - John Smith在讨论微软的"时, - @"_Show 剪辑,上面有#-}关于凤凰计划的屏幕文字 200)}" - @"_Find video 类似此场景的视频@"}{(}CLIP 嵌入搜索中=)} **视频“ :” 减少的 RAG 模式Name** ``` Ingestion: Video → 16 waves → Signals + Evidence (scenes, transcripts, entities) Storage: Signals (indexed) + Embeddings (CLIP, voice) + Evidence (chunks) Query: Filter (SQL) → Search (BM25 + vector) → Synthesize (LLM, ~5 results) ``` **能力系统@: @%** ``` Startup: Detect GPU → Load ModelManifest (YAML) → Initialize SignalSink Activation: Component requests model → Lazy download → Signal "ModelAvailable" Routing: Route to best provider → Fallback chain → Backpressure control Atoms: Rate limiting + Time estimation + Pipeline balancing ``` 这是 **受限制的模糊** 缩放时@: @% - **概率组成部分提出信号**:嵌入中 @ MS K1CLIP),OCR假想@,diarization 猜測 - **确定性评分组装结构**:加权边界分数,门槛选择MS K2时间限制 - **LLM {(}可选})根据证据合成**:绑定的上下文 @,可审计来源 LLM使用预设的-* computedevid证据而不是原始视频}. --- ## 资源资源资源 资源和资源资源资源 ### 清晰的RAG 文件 - **[视频成像器库](https://github.com/scottgal/lucidrag/tree/main/src/VideoSummarizer.Core)** - 源代码 - **[NNER 文件](https://github.com/scottgal/lucidrag/blob/main/docs/NER_EXTRACTION_DEDUPLICATION.md)** -实体采掘结构 ### 相关图书馆 - **[图像合成器](https://github.com/scottgal/lucidrag/tree/main/src/ImageSummarizer.Core)** 视觉情报管道 - **[音频合成器](https://github.com/scottgal/lucidrag/tree/main/src/AudioSummarizer.Core)** 音频法证管道 - **[DocSummamer 缩写器](https://github.com/scottgal/lucidrag/tree/main/src/Mostlylucid.DocSummarizer.Core)** -文件编审中 ### ONNX模型 - **[CLIP Vit-B/32](https://huggingface.co/openai/clip-vit-base-patch32)** - 视觉嵌入器 - **[ECAPA-TDNN(坦桑尼亚)](https://huggingface.co/Wespeaker/wespeaker-ecapa-tdnn512-LM)** @- 发言人嵌入 - **[BERT-纳](https://huggingface.co/dslim/bert-base-NER)** - 命名实体识别 - **[耳语](https://github.com/openai/whisper)** - 语音转录 ### 相关条款 **核心模式** - **[减少的RAG](https://www.mostlylucid.net/blog/reduced-rag)** 核心模式 - **[受限制的模糊模式](/blog/constrained-fuzziness-pattern)** 基础模式 **减少RAG执行量** - **[DocSummamer 缩写器](/blog/building-a-document-summarizer-with-rag)** - 具有实体提取功能的文件RAG - **[图像合成器](/blog/constrained-fuzzy-image-intelligence)** @-图像RAG 带有 @MS K1波直观情报 - **[3-Tier OCR管道](/blog/constrained-fuzzy-image-ocr-pipeline)** OCR升级 - **[音频合成器](/blog/audiosummarizer-forensic-audio-characterization)** 音频法证鉴定 - **影片缩写器 @ ( @ this article @) @** - 视频情报管弦乐队 - **[数据合成器](/blog/datasummarizer-how-it-works)** - 数据剖析和计划推论 --- ## 系列丛书 @ | @ part * MPK1_ 模式 @ I|} 聚焦@ MSC3 |------|---------|-------| | 1 | [受限制的模糊](/blog/constrained-fuzziness-pattern) @|单元件 @ |# | 2 | [限制的模糊MOM](/blog/constrained-mom-mixture-of-models) | 多元件| | 3 | [环境拖累](/blog/constrained-fuzzy-context-dragging) *|Q时间* */Q内存* | 4 | [图像情报](/blog/constrained-fuzzy-image-intelligence) |波形结构 MS K122浪 MSC3 | 4.1 | [3-Tier OCR管道](/blog/constrained-fuzzy-image-ocr-pipeline) -=YTET -伊甸园字幕组=- 翻译: | 4.2 | [音频合成器](/blog/audiosummarizer-forensic-audio-characterization) -=YTET -伊甸园字幕组=- 翻译: | **4.3** | **影片缩写器 @ ( @ this article @) @** | **视频節奏,批量 CLIPMS K1NER** | **下一頁**: multi-modal 图RAG 用清晰的RAG将所有四个总结器组成一个统一的知识图,由连接\-MQ3的跨MSK 2modial实体链接起来 所有部件都沿着相同的变数 : **建议的概率元件**.