大多数系统超载后会退化。 内存会填充, 询问慢, 用户抱怨, 服务器崩溃 。
几个不同寻常的 更好.
本条说明: 以 LRU为基础的行为内存 当它击中能力时,自我优化成为自我优化- 以及这种模式如何驱动我的学习系统 机器人检测引擎。这也是尽可能最小的版本 DISE 建筑结构 - 通过资源压力控制进化
如果你读过我的文章 CQRS CQRS 和事件检验这是CQRS被剥下来的骨头,没有事件商店,没有预测,没有Marten。只是一个记忆缓存,一个背景工人,还有SQLite。
在潜入之前,让我定义一个词,我会在下面使用: 签名签名。签名是指代表一种行为模式的任何稳定关键 - IP +用户代理的散列、页眉组合的指纹、检测器对“此项请求看起来像X”的分类。缓存存储这些签名以及随时间变化的学习重量。
如果你能建立一个系统 来:
这正是 IMemoryCache 滑动失效使你... 如果你了解你正在建造的建筑
LRU( LRU (最小最近使用) 缓存缓存将最近没有访问的条目驱逐。 当缓存填满后, 最冷的条目会被丢出, 以让热条目有空间 。
大多数开发商认为这是一个限制。 "哦,不,我的缓存满了, 数据丢失了!"
但对于行为系统来说,这是一个特点。
// From WeightStore.cs - the bounded memory window
_cache = new MemoryCache(new MemoryCacheOptions
{
SizeLimit = _cacheSize, // e.g., 1000 entries
CompactionPercentage = 0.25 // Remove 25% when limit reached
});
这: SizeLimit 不仅仅是一个记忆力限制 选择选择压力它决定了系统“成员”的数量,迫使它专注于什么重要。
与滑动失效结合, 你会自动忘记:
// From WeightStore.cs:254-259
private MemoryCacheEntryOptions GetCacheEntryOptions()
{
return new MemoryCacheEntryOptions()
.SetSlidingExpiration(_slidingExpiration) // 30 minutes
.SetSize(1); // Each entry counts as 1 toward size limit
}
如果30分钟内无法取得签名,它就会被逐出。不是因为它错了,而是因为它不再相关。
由此创建 自然遗忘:
静态块状列表变淡。 滑动失效会让记忆更新 。
这就是它赖以运作的模式。 SqliteWeightStore 类 :
/// <summary>
/// SQLite implementation of the weight store with sliding expiration memory cache.
/// Uses a CQRS-style pattern with write-behind:
/// - Reads: Hit memory cache first (fast path), fall back to SQLite on miss
/// - Writes: Update cache immediately, queue SQLite writes for background flush
/// Sliding expiration provides automatic LRU-like eviction behavior.
/// </summary>
public class SqliteWeightStore : IWeightStore, IAsyncDisposable
{
// Memory cache with sliding expiration - auto-evicts least recently used entries
private readonly MemoryCache _cache;
// Write-behind queue for batched SQLite persistence
private readonly ConcurrentDictionary<string, PendingWrite> _pendingWrites = new();
private readonly Timer _flushTimer;
private readonly TimeSpan _flushInterval = TimeSpan.FromMilliseconds(500);
这是 CQRS 非正式的CQRS 而不是在写文章后撤销缓存条目 缓存是写模型SQLite只是永久的帐本
flowchart LR
subgraph Cache["In-Memory Behaviour Store"]
A[Hot Signatures] --- B[Sliding Expiry]
end
subgraph DB["SQLite Ledger"]
C[(Durable Write-Behind)]
end
A --Periodic Flush--> C
B --Eviction--> D[Forgotten]
style Cache fill:none,stroke:#10b981,stroke-width:2px
style DB fill:none,stroke:#6366f1,stroke-width:2px
关键洞察力: 阅读并写作到内存数据库最终是一致的,这很好。
当探测器需要学习重量时, 它会到达缓存处:
// From WeightStore.cs:421-472
public async Task<double> GetWeightAsync(
string signatureType,
string signature,
CancellationToken ct = default)
{
var key = CacheKey(signatureType, signature);
// Check cache first (fast path - no DB access)
if (_cache.TryGetValue(key, out LearnedWeight? cached) && cached != null)
{
_metrics?.RecordCacheHit(signatureType);
return cached.Weight * cached.Confidence;
}
_metrics?.RecordCacheMiss(signatureType);
// Cache miss - load from DB
await EnsureInitializedAsync(ct);
await using var conn = new SqliteConnection(_connectionString);
await conn.OpenAsync(ct);
var sql = $@"
SELECT weight, confidence, observation_count, first_seen, last_seen
FROM {TableName}
WHERE signature_type = @type AND signature = @sig
";
// ... execute query ...
if (await reader.ReadAsync(ct))
{
// Cache the result for future reads
var learnedWeight = new LearnedWeight { /* ... */ };
_cache.Set(key, learnedWeight, GetCacheEntryOptions());
return weight * confidence;
}
return 0.0; // No learned weight exists
}
热路 从未访问过数据库SQLite只是备份存储。
当系统学习新东西时, 它会立即更新缓存, 并排队数据库写入 :
// From WeightStore.cs:551-582
public Task UpdateWeightAsync(
string signatureType,
string signature,
double weight,
double confidence,
int observationCount,
CancellationToken ct = default)
{
var key = CacheKey(signatureType, signature);
// Update cache immediately (source of truth for reads)
var learnedWeight = new LearnedWeight
{
SignatureType = signatureType,
Signature = signature,
Weight = weight,
Confidence = confidence,
ObservationCount = observationCount,
FirstSeen = DateTimeOffset.UtcNow,
LastSeen = DateTimeOffset.UtcNow
};
_cache.Set(key, learnedWeight, GetCacheEntryOptions());
// Queue for async SQLite persistence (write-behind)
QueueWrite(signatureType, signature, weight, confidence, observationCount);
return Task.CompletedTask;
}
通知 : UpdateWeightAsync 返回返回返回 Task.CompletedTask 写作排队,没有执行,这意味着:
每500米, 待定的写作会分批冲到SQLite:
// From WeightStore.cs:274-357
public async Task FlushPendingWritesAsync(CancellationToken ct = default)
{
if (_pendingWrites.IsEmpty) return;
// Only one flush at a time
if (!await _flushLock.WaitAsync(0, ct)) return;
try
{
await EnsureInitializedAsync(ct);
// Snapshot and clear pending writes atomically
var writes = new List<PendingWrite>();
foreach (var key in _pendingWrites.Keys.ToList())
{
if (_pendingWrites.TryRemove(key, out var write))
{
writes.Add(write);
}
}
if (writes.Count == 0) return;
await using var conn = new SqliteConnection(_connectionString);
await conn.OpenAsync(ct);
await using var transaction = await conn.BeginTransactionAsync(ct);
try
{
var sql = $@"
INSERT INTO {TableName}
(signature_type, signature, weight, confidence,
observation_count, first_seen, last_seen)
VALUES (@type, @sig, @weight, @conf, @count, @now, @now)
ON CONFLICT(signature_type, signature) DO UPDATE SET
weight = @weight,
confidence = @conf,
observation_count = @count,
last_seen = @now
";
foreach (var write in writes)
{
await using var cmd = new SqliteCommand(sql, conn, transaction);
// ... add parameters and execute ...
}
await transaction.CommitAsync(ct);
_logger.LogDebug("Flushed {Count} pending writes in {Duration:F1}ms",
writes.Count, sw.ElapsedMilliseconds);
}
catch
{
await transaction.RollbackAsync(ct);
throw;
}
}
finally
{
_flushLock.Release();
}
}
这是 活动外包-灯光。你得到:
当新观测到达时, 系统使用指数移动平均值来更新重量 :
// From WeightStore.cs:584-635
public Task RecordObservationAsync(
string signatureType,
string signature,
bool wasBot,
double detectionConfidence,
CancellationToken ct = default)
{
var key = CacheKey(signatureType, signature);
// Calculate new weight using EMA in memory
var alpha = 0.1; // Learning rate
var weightDelta = wasBot ? detectionConfidence : -detectionConfidence;
double newWeight;
double newConfidence;
int newObservationCount;
if (_cache.TryGetValue(key, out LearnedWeight? existing) && existing != null)
{
// Apply EMA: new_weight = old_weight * (1-α) + delta * α
newWeight = existing.Weight * (1 - alpha) + weightDelta * alpha;
newConfidence = Math.Min(1.0, existing.Confidence + detectionConfidence * 0.01);
newObservationCount = existing.ObservationCount + 1;
}
else
{
// First observation
newWeight = weightDelta;
newConfidence = detectionConfidence;
newObservationCount = 1;
}
// Update cache immediately
var learnedWeight = new LearnedWeight
{
SignatureType = signatureType,
Signature = signature,
Weight = newWeight,
Confidence = newConfidence,
ObservationCount = newObservationCount,
FirstSeen = existing?.FirstSeen ?? DateTimeOffset.UtcNow,
LastSeen = DateTimeOffset.UtcNow
};
_cache.Set(key, learnedWeight, GetCacheEntryOptions());
// Queue for persistence
QueueWrite(signatureType, signature, newWeight, newConfidence, newObservationCount);
return Task.CompletedTask;
}
EMA公式的流畅学习: new_weight = old_weight × (1 - α) + new_value × α
α=0.1:
这是大多数人错过的关键洞察力
当缓存填满时:
想想看,如果5万个独有的签名 击中你的机器人探测器, 但你只有1万个记忆, 哪个签名重要?
最热的一万 - 通常占实际交通量的99%。
在制作过程中,我每天看到大约40,000个一次性签名(摩擦手尝试一次,随机探测,合法用户再也不会回来 ) , 以及可能5—10,000个连续不断的签名。 这5—10,000个是风险生命的99%。 长尾签名?噪音。 消灭它们并不影响检测准确性 — — 甚至可以通过减少低信任模式的假阳性来改善它。
flowchart TB
subgraph Input["50,000 Unique Signatures"]
Hot[Hot Signatures\n~10,000]
Cold[Cold Signatures\n~40,000]
end
subgraph Cache["Bounded Cache (10,000)"]
Kept[Kept in Memory]
end
subgraph Evicted["Evicted"]
Lost[Forgotten\nNoise Traffic]
end
Hot --> Kept
Cold --> Lost
style Hot fill:none,stroke:#10b981,stroke-width:2px
style Cold fill:none,stroke:#94a3b8,stroke-width:2px
style Kept fill:none,stroke:#10b981,stroke-width:2px
style Lost fill:none,stroke:#ef4444,stroke-width:2px
溢 溢 溢 流 尖尖 行为内存 系统自我限制
这不是魔法,有些边缘情况 路运联盟的压力会影响你
缓存太小, 缓存太小:如果您的缓存只持有100个条目,但您有1000个真正重要的签名, 你会在它们积累足够的证据之前不断发作, 失去有用的模式。 大小您的缓存可舒适地保存您的“ 热集 ” 。
统一运输:如果你运行在一个微小的内部系统中, 几乎所有东西都是“热”的( 发热的独特签名, 重复出现) , 溢出会减少你的利益。 选择压力没有选择的余地 。
冷启动问题:新部署的系统没有学习的重量。一切都同样寒冷。在热信号建立之前,最初几小时的假正率会更高。
该模式在您拥有时最有效 具有权力法分布的高度基本线 - 很多独特的签名,但一个小子集 主导交通。
缓存是读取真理的来源,但数据库是持久分类账。当它们漂移时会怎样?
当数据库重量衰减时,缓存需要跟随。 DecayOldWeightsAsync 方法同时处理 :
// From WeightStore.cs:725-766
public async Task DecayOldWeightsAsync(TimeSpan maxAge, double decayFactor, CancellationToken ct = default)
{
await EnsureInitializedAsync(ct);
await using var conn = new SqliteConnection(_connectionString);
await conn.OpenAsync(ct);
var cutoff = DateTimeOffset.UtcNow.Subtract(maxAge).ToString("O");
// Decay old weights in the database
var sql = $@"
UPDATE {TableName}
SET weight = weight * @decay,
confidence = confidence * @decay
WHERE last_seen < @cutoff
";
await using var cmd = new SqliteCommand(sql, conn);
cmd.Parameters.AddWithValue("@decay", decayFactor);
cmd.Parameters.AddWithValue("@cutoff", cutoff);
var updated = await cmd.ExecuteNonQueryAsync(ct);
// Delete weights that have decayed below threshold
var deleteSql = $@"
DELETE FROM {TableName}
WHERE confidence < 0.01 OR (ABS(weight) < 0.01 AND observation_count < 5)
";
await using var deleteCmd = new SqliteCommand(deleteSql, conn);
var deleted = await deleteCmd.ExecuteNonQueryAsync(ct);
if (updated > 0 || deleted > 0)
{
_logger.LogInformation(
"Weight decay: {Updated} decayed, {Deleted} deleted",
updated, deleted);
// Compact cache to remove stale entries
_cache.Compact(0.25);
}
}
在数据库记录衰减后,我们打电话给 _cache.Compact(0.25) - 迫使 - 迫使 - 迫使 MemoryCache 将25%的条目排出, 优先排序最近使用最少的条目。 下一个读取将会从数据库中重新装入新值 。
有时您需要废除整个类缓存条目 - 例如当再培训探测器或当外部数据发生变化时 :
// From WeightStore.cs:768-777
/// <summary>
/// Evicts all cached entries for a specific signature type (tag-based eviction).
/// </summary>
public void EvictByTag(string signatureType)
{
// MemoryCache doesn't natively support tag-based eviction, but we can compact
// For now, just compact - sliding expiration will handle stale entries
_cache.Compact(0.1);
_logger.LogDebug("Compacted cache for signature type: {SignatureType}", signatureType);
}
NET 网络 MemoryCache 没有像Redis那样的本地标签式驱逐, 但压缩效果是一样的: 强制删除陈旧的条目, 让我们从数据库中重新读取。
关键的观点是 不需要完全同步系统容许漂移,因为:
这是最终一致的正确方法。 缓存会“ 足够接近” 数据库, 不需要复杂的无效逻辑 。
flowchart TB
subgraph Sync["Cache-Database Synchronisation"]
D[Database Decay] --> C[Cache Compact]
E[Tag Eviction] --> C
S[Sliding Expiration] --> M[Cache Miss]
M --> R[Reload from DB]
end
style Sync fill:none,stroke:#6366f1,stroke-width:2px
没有缓存失效地狱,没有复杂的酒吧/活动,只是压缩和自然失效。
注: 如果您想要 LRU + 写入后模式, 您可以在这里停止 。 此文章的其余部分显示我如何将同样的想法应用到完全模式的名声上 国家机器、歇斯底里和时间衰减。 这是那些建设适应系统的“ 外英里 ” 。
对于典型的声誉(跟踪签字是机器人还是人,随时间推移),适用同样的原则,但具有更多精密性:
// From PatternReputation.cs:42-108
public record PatternReputation
{
public required string PatternId { get; init; }
public required string PatternType { get; init; }
public required string Pattern { get; init; }
/// <summary>Current bot probability [0,1]. 0 = human, 1 = bot, 0.5 = neutral</summary>
public double BotScore { get; init; } = 0.5;
/// <summary>Effective sample count - decays over time, increases with observations</summary>
public double Support { get; init; } = 0;
/// <summary>Current reputation state - determines fast-path behavior</summary>
public ReputationState State { get; init; } = ReputationState.Neutral;
// Computed properties
public double Confidence => Math.Min(1.0, Support / 100.0);
public bool CanTriggerFastAbort =>
State is ReputationState.ConfirmedBad or ReputationState.ManuallyBlocked;
public bool CanTriggerFastAllow =>
State is ReputationState.ConfirmedGood or ReputationState.ManuallyAllowed;
}
模式不会直接从中立转向确认的巴德。有歇斯底里来防止拍耳光:
// From PatternReputation.cs:367-421 - simplified
public PatternReputation EvaluateStateChange(PatternReputation reputation)
{
if (reputation.IsManual)
return reputation;
var newState = reputation.State;
var score = reputation.BotScore;
var support = reputation.Support;
switch (reputation.State)
{
case ReputationState.Neutral:
// Can promote to Suspect or ConfirmedGood
if (score >= 0.6 && support >= 10)
newState = ReputationState.Suspect;
else if (score <= 0.1 && support >= 100)
newState = ReputationState.ConfirmedGood;
break;
case ReputationState.Suspect:
// Can promote to ConfirmedBad or demote to Neutral
if (score >= 0.9 && support >= 50)
newState = ReputationState.ConfirmedBad;
else if (score <= 0.4 || support < 10)
newState = ReputationState.Neutral;
break;
case ReputationState.ConfirmedBad:
// Can demote to Suspect (requires MORE evidence to forgive)
if (score <= 0.7 && support >= 100)
newState = ReputationState.Suspect;
break;
}
// ... log state change and return ...
}
注意不对称性:被封锁比被封锁更容易。 ConfirmedBad → Suspect 需要100个支持,而 Neutral → Suspect 这是故意的 原谅比怀疑更难
当模式变得安静时,它们会向中立方向衰落:
// From PatternReputation.cs:334-361
public PatternReputation ApplyTimeDecay(PatternReputation reputation)
{
if (reputation.IsManual)
return reputation;
var hoursSinceLastSeen = (DateTimeOffset.UtcNow - reputation.LastSeen).TotalHours;
if (hoursSinceLastSeen < 1)
return reputation; // Too recent to decay
// Score decay toward prior (0.5 = neutral)
// new_score = old_score + (prior - old_score) × (1 - e^(-Δt/τ))
var scoreDecayFactor = 1 - Math.Exp(-hoursSinceLastSeen / _options.ScoreDecayTauHours);
var newScore = reputation.BotScore + (0.5 - reputation.BotScore) * scoreDecayFactor;
// Support decay
// new_support = old_support × e^(-Δt/τ)
var supportDecayFactor = Math.Exp(-hoursSinceLastSeen / _options.SupportDecayTauHours);
var newSupport = reputation.Support * supportDecayFactor;
return reputation with
{
BotScore = Math.Clamp(newScore, 0, 1),
Support = newSupport
};
}
默认时间常数 :
这意味着一个被确认为坏的IP,在一个月内保持安静,最终会回到中立。 这并不是因为改革 — — 因为它的证据变得僵化。
缩略 ReputationMaintenanceService 运行三个定期任务 :
// From ReputationMaintenanceService.cs:48-129
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
_logger.LogInformation("Reputation maintenance service starting");
// Load persisted reputations on startup
await _cache.LoadAsync(stoppingToken);
var decayInterval = TimeSpan.FromMinutes(60); // Hourly decay sweep
var gcInterval = TimeSpan.FromHours(24); // Daily garbage collection
var persistInterval = TimeSpan.FromMinutes(5); // Persist every 5 minutes
var lastDecay = DateTimeOffset.UtcNow;
var lastGc = DateTimeOffset.UtcNow;
var lastPersist = DateTimeOffset.UtcNow;
while (!stoppingToken.IsCancellationRequested)
{
await Task.Delay(TimeSpan.FromMinutes(1), stoppingToken);
var now = DateTimeOffset.UtcNow;
// Decay sweep: push stale scores toward neutral
if (now - lastDecay >= decayInterval)
{
await _cache.DecaySweepAsync(stoppingToken);
lastDecay = now;
}
// Garbage collection: remove old neutral patterns
if (now - lastGc >= gcInterval)
{
await _cache.GarbageCollectAsync(stoppingToken);
lastGc = now;
var stats = _cache.GetStats();
_logger.LogInformation(
"Reputation stats: {Total} patterns, {Bad} bad, {Suspect} suspect",
stats.TotalPatterns, stats.ConfirmedBadCount, stats.SuspectCount);
}
// Persistence: save to SQLite
if (now - lastPersist >= persistInterval)
{
await _cache.PersistAsync(stoppingToken);
lastPersist = now;
}
}
// Final persist on shutdown
await _cache.PersistAsync(CancellationToken.None);
}
垃圾收集器除去下列模式:
这使记忆中的字典无法不受限制地成长,同时保留宝贵的学习模式。
PostgreSQL 或 Redis 的许多开发商反射达标。 但对于这个模式, SQLite 是理想的 :
最起码的策略是:
CREATE TABLE IF NOT EXISTS learned_weights (
signature_type TEXT NOT NULL,
signature TEXT NOT NULL,
weight REAL NOT NULL,
confidence REAL NOT NULL,
observation_count INTEGER NOT NULL DEFAULT 1,
first_seen TEXT NOT NULL,
last_seen TEXT NOT NULL,
PRIMARY KEY (signature_type, signature)
);
CREATE INDEX IF NOT EXISTS idx_signature_type ON learned_weights(signature_type);
CREATE INDEX IF NOT EXISTS idx_confidence ON learned_weights(confidence);
CREATE INDEX IF NOT EXISTS idx_last_seen ON learned_weights(last_seen);
如果您需要更大的比例, 请换成 PostgreSQL。 如果您需要HA 或复制, 请换成 Redis 或分布的缓存 。 结构不会改变 - 只是连接字符串。 SQLite 是边缘部署的默认值, 不是宗教 。
如果(如果) DiSE 日志 这是完整的进化引擎, 这种缓存模式是 mitochondria - 最小的一块 仍然行为像进化 在制约下。
这种模式在最起码的层次上执行DISE原则:
整个暗藏处不是失败 而是失败 进进压力系统自选性:热信号停留,冷信号驱离,行为记忆与实际重要性趋同。
没有ML训练,没有外部模型,只是像生活系统一样的建筑
整个图案归结为:
你的小行为商店的行为 更像一个活的系统 而不是CROUD。它记住什么重要,忘记什么不重要, 并在压力下变得更好。
最微小的建筑 即突发的正确性 溢出 更好地关注。 压力稳定。
如果你想在行动中看到这一点,请查看 多数为lucid.bott 检测 - 以及 DISE 建筑建筑系列 更深层的哲学
© 2026 Scott Galloway — Unlicense — All content and source code on this site is free to use, copy, modify, and sell.