前言

密码学题目常常需要精确的数学推导和细致的算法实现。在使用大语言模型辅助解题时,令笔者感受颇深的一个明显的现象是相同或相近的提示词在不同次调用中会产生截然不同的结果。有些时候模型似乎恰好理解了题目要点,给出了正确的攻击路径和完整的利用代码,整个过程行云流水;而另一些时候,同样的提示词却让模型陷入错误的理解循环,输出的代码要么使用了错误的密码学原语,要么在关键步骤上出现逻辑偏差,多次重试依然无法推进。

这种现象不是偶然,而是LLM采样机制的必然结果。温度参数、top_p采样、nucleus过滤等机制控制着模型输出的随机性,这些机制在赋予模型创造能力的同时也引入了不确定性。对于需要精确性的密码学任务,这种不确定性直接影响了解题的成功率。实际观察发现,即使经过精心设计的prompt,单次LLM调用的成功率在30%到45%之间波动,意味着超过一半的尝试需要某种形式的修正或重试。

更关键的是,不同类型的失败需要不同的应对策略。格式错误需要更明确的约束,逻辑错误需要事实核查,环境假设错误需要调整上下文。简单的重复调用无法从失败中学习,错误模式可能重复出现,导致资源浪费。工程化的解决方案需要对失败进行系统化分类,建立针对性的重试机制,将LLM的随机性转化为可预测、可验证的成功概率。

本文正是在这个背景下,系统阐述一个面向CTF自动化解题的失败重试调度器。调度器通过错误分类、候选并行、智能重试、验证执行四个环节,将单次低概率事件转化为高概率复合事件,实现从随机波动到稳定产出的可控转化。

LLM随机性带来的工程问题

CTF自动化解题依赖大语言模型生成解题思路、构造攻击载荷、编写利用代码。然而LLM的输出存在本质的随机性,同一prompt在不同采样参数下会产生差异巨大的结果。温度参数控制输出多样性,top_p和top_k影响token选择的候选空间,这些机制赋予了模型创造性能力,但也引入了不确定性。在需要精确、可验证、可复现的安全攻防场景中,这种随机性成为工程落地的核心障碍。

LLM的采样机制基于概率分布。给定上下文,模型为词汇表中每个token分配概率,然后根据采样策略选择下一个token。温度参数T调节概率分布的锐度,T越低分布越集中,输出越确定;T越高分布越平缓,输出越随机。数学上,温度调整后的概率为:

当T趋近于0时,采样退化为贪婪解码,总是选择概率最大的token;当T=1时,使用原始概率分布;当T大于1时,概率分布趋于均匀,增加低概率token的选择机会。这种机制使得LLM能够生成多样化的输出,但也意味着相同输入可能产生完全不同的结果。

在CTF场景中,这种随机性导致单次调用的成功率不稳定。实际测试表明,即使经过精心设计的prompt,单次LLM调用的解题成功率在30%到45%之间波动。波动来源包括:模型对题目的理解偏差、代码生成中的细微错误、payload构造中的逻辑缺陷、环境假设的不准确。这些因素叠加,使得单次尝试的可靠性无法满足自动化解题的要求。

简单的重复调用虽然可以提高成功率,但带来两个问题:成本线性增长和效率低下。假设单次成功率为p,需要重复n次才能达到期望的成功率,根据概率论,n次独立尝试至少一次成功的概率为:

要使P_success达到0.95,当p=0.35时,需要n=8次;当p=0.4时,需要n=6次。这意味着平均需要消耗6-8倍的tokens才能达到高成功率。更严重的是,盲目的重复调用无法从失败中学习,每次尝试都是独立的,错误模式可能重复出现。

问题的根源在于缺乏对失败原因的系统化分类和针对性处理。不同类型的失败需要完全不同的响应策略。schema validation失败可能需要更明确的格式要求,hallucination可能需要增加事实约束,tool execution失败可能需要调整环境假设。将所有失败混同处理,必然导致资源浪费和效率低下。

工程化的解决方案需要建立三层架构:错误分类层负责识别失败类型并映射到处理策略,候选调度层负责管理多个并行的解题尝试并去重,验证执行层负责运行候选解并记录结果。这三层构成了完整的失败重试调度器,将LLM的随机性转化为可控的成功概率。

调度器的设计需要考虑多个维度。时间维度上,需要在快速失败和深度探索之间平衡;资源维度上,需要在并行度和成本之间权衡;质量维度上,需要在多样性和准确性之间取舍。这些权衡不是静态的,而是根据题目类型、难度、已投入的尝试次数动态调整。

CTF场景的失败模式分析

CTF题目分为多个类别:pwn利用二进制漏洞,web攻击应用漏洞,crypto破解密码学方案,misc杂项包括隐写、取证等。不同类别的题目有不同的失败模式特征,理解这些特征有助于设计更精确的分类体系和重试策略。

Pwn类题目的失败主要集中在漏洞理解错误和利用代码编写错误。LLM可能正确识别了栈溢出漏洞,但计算的offset不准确;可能理解了UAF漏洞,但构造的堆布局布局失败;可能知道需要ROP链,但选择的gadget不正确。这些失败往往需要实际执行才能发现,但执行环境与题目环境的差异可能导致误导。

from pwn import *

buffer_size = 64
offset = buffer_size + 8
payload = b"A" * offset + p64(0x400686)

p = remote('localhost', 9000)
p.sendline(payload)
p.interactive()

buffer_size = 72
ret_addr = 0x400686
payload = b"A" * buffer_size + p64(ret_addr)

第一个例子offset计算错误,缺少saved rbp的8字节;第二个例子offset正确但hardcode了地址,在ASLR开启时会失败。这些错误在代码层面语法正确,但逻辑有误,属于hallucination类型。

Web类题目的失败更多在payload构造和参数处理上。SQL注入的payload可能被WAF拦截,XSS的payload可能被浏览器过滤,SSRF的payload可能无法绕过限制。LLM生成的payload往往是标准形式,但实际题目可能需要绕过技巧。

payload = "1' OR '1'='1"
payload = f"1' UNION SELECT username, password FROM users--"
payload = {"input": "<script>alert(1)</script>"}

cookie = {"session_id": "admin"}
headers = {"X-Forwarded-For": "127.0.0.1"}

第一个SQL注入payload可能被过滤,第二个union select可能被检测,第三个XSS可能被CSP阻止。这些错误需要根据实际响应调整,属于tool fail类型。

Crypto类题目的失败可能在数学理解上。LLM可能选择了错误的攻击方法,或者实现的算法有精度问题。RSA攻击可能混淆了共模攻击和低指数攻击,椭圆曲线可能选择了错误的点运算,离散对数可能使用了效率低下的算法。

n = 0xDEADBEEF
e1, e2 = 3, 5
c1, c2 = pow(m, e1, n), pow(m, e2, n)

def extended_gcd(a, b):
    if b == 0:
        return a, 1, 0
    else:
        g, x, y = extended_gcd(b, a % b)
        return g, y, x - (a // b) * y

def common_modulus_attack(n, e1, c1, e2, c2):
    g, a, b = extended_gcd(e1, e2)
    if g == 1:
        return None
    m = (pow(c1, a, n) * pow(c2, b, n)) % n
    return m

def low_exponent_attack(e, n):
    k = 0
    while True:
        candidate_root = int(pow(c + k*n, 1.0/e))
        if pow(candidate_root, e, n) == c:
            return candidate_root
        k += 1

第一个例子实现了共模攻击的extended_gcd,但如果e1和e2不互质则无法使用;第二个例子实现了低指数广播攻击的枚举,但当消息较大时效率极低。这些是hallucination类型的数学逻辑错误。

Misc类题目的失败最多样化,可能涉及文件格式理解错误、编码识别错误、工具使用错误。隐写题可能选错了隐藏位置,取证题可能遗漏了关键文件,流量分析题可能跟丢了会话。这些失败往往需要领域知识才能识别。

基于这些观察,CTF场景的失败可以分为三大类:schema fail指输出格式不符合预期,hallucination指内容看似正确但事实或逻辑有误,tool fail指在实际环境中无法执行或执行结果与预期不符。每类又细分为多个子类,每个子类对应特定的检测方法和重试策略。

错误分类体系设计

错误分类是重试调度的基础。通过精确识别失败类型,调度器可以选择合适的重试策略,避免无效的重复尝试。分类体系需要满足三个要求:可检测性,能够自动或半自动识别;互斥性,每个错误只属于一个类别;可操作性,每个类别对应明确的重试策略。

Schema Fail分类与检测

Schema fail指的是输出结构或格式不符合预期约定。这类错误的特征是可以通过形式化验证自动检测,不需要领域知识。检测机制包括JSON schema validation、正则表达式匹配、类型检查、长度限制检查等。

结构不完整是最常见的schema fail。JSON输出缺少闭合括号、数组未终止、字符串未转义。这类错误在LLM输出长度接近token限制时容易出现,模型可能在输出过程中被截断。

{
    "approach": "Use buffer overflow to overwrite return address",
    "code": "from pwn import *\n\np = remote('remote', 9000)\npayload = b'A' * 72 + p64(0x400686)
}

{
    "exploit": {
        "method": "ROP chain",
        "gadgets": [
            {"addr": "0x400612", "insn": "pop rdi; ret"},
            {"addr": "0x400613", "insn": "ret"}
    ]

第一个例子JSON缺少右括号和闭合大括号,第二个例子数组缺少闭合中括号。这类错误可以通过尝试解析JSON并捕获syntax error来检测。

字段缺失指要求的必需字段不存在。CTF解题输出通常要求包含approach解题思路、code利用代码、explanation关键步骤说明,可能还要求expected_result预期输出。LLM可能只提供了部分内容。

{
    "approach": "Buffer overflow with ROP chain",
    "explanation": "Calculate offset to saved return address, build ROP chain to call win function"
}

{
    "code": "from pwn import *",
    "explanation": "Connect to service and send payload",
    "expected_result": "flag{overflow_master}"
}

第一个例子缺少code字段,第二个例子缺少approach字段。这类错误可以通过定义pydantic模型并标记required字段来检测。

类型错误指字段值的类型不符合要求。要求整数字段却得到字符串,要求代码块却得到自然语言描述,要求布尔值却得到其他类型。

{
    "offset": "72",
    "address": "0x400686",
    "buffer_size": 64.5
}

{
    "approach": "Send payload",
    "code": None,
    "explanation": ["Step1", "Step2", "Step3"]
}

第一个例子offset应该是整数,buffer_size应该是整数;第二个例子code应该是字符串而不是None,explanation应该是字符串而不是数组。这类错误通过pydantic的类型检查自动检测。

约束违反指输出违反了明确定义的约束条件。字符串长度限制、字符集白名单、数值范围限制、格式模式要求。这些约束可能来自题目要求,也可能来自安全考虑。

{
    "exploit_code": "A" * 10000,
    "url": "https://evil.com/malware",
    "command": "rm -rf /"
}

{
    "payload": "\\x00\\x01\\x02\\x03\\xff\\xfe\\xfd\\xfc",
    "shellcode": "\\x31\\xc0\\x50\\x68\\x2f\\x2f\\x73\\x68\\x68\\x2f\\x62\\x69\\x6e\\x89\\xe3\\x50\\x53\\x89\\xe1\\xb0\\x0b\\xcd\\x80"
}

第一个例子exploit_code过长可能触发内存限制,url和command包含危险操作;第二个例子payload包含非打印字符可能无法正确传输。这类约束通过自定义validator检测。

Schema fail的检测是确定性的,不依赖领域知识。检测失败后,重试策略的核心是增强结构化约束。第一次重试在prompt中明确列出输出schema和每个字段的类型要求,第二次重试提供完整的输出示例,第三次重试切换到强制结构化输出如function calling或JSON mode。

Hallucination分类与检测

Hallucination指的是LLM生成的内容在语法和逻辑上正确,但事实或实现有误。这类错误最难检测,因为需要领域知识验证。检测机制包括代码静态分析、符号执行、实际运行测试、领域规则检查。

函数幻觉是代码中使用了不存在的函数或方法。LLM可能编造了看似合理的函数名,或者混淆了不同库的API。这类错误在代码执行时会抛出NameError或AttributeError,但执行前难以发现。

from pwn import *

def calculate_offset(binary_path):
    info = binary_info(binary_path)
    return info['buffer_offset']

def build_rop_chain(gadgets):
    chain = ROPChain()
    chain.add_gadget(gadgets['pop_rdi'])
    chain.add_gadget(gadgets['system'])
    return chain.bytes()

def format_payload(padding, address):
    return construct_exploit(padding, address)

calculate_offset使用的binary_info不是pwntools标准API,ROPChain类不存在,construct_exploit函数未定义。这些函数可能是LLM基于推理产生的幻觉,实际库中没有对应实现。

检测函数幻觉需要在执行前进行静态分析。解析代码的AST,收集所有函数调用,检查每个调用的函数是否在标准库、已安装的第三方库、用户代码中定义。对于未定义的函数,标记为hallucination。

import ast
import importlib.util

class FunctionCallChecker(ast.NodeVisitor):
    def __init__(self):
        self.imports = set()
        self.calls = set()
        self.undefined_calls = set()

    def visit_Import(self, node):
        for alias in node.names:
            self.imports.add(alias.name.split('.')[0])

    def visit_ImportFrom(self, node):
        if node.module:
            self.imports.add(node.module.split('.')[0])

    def visit_Call(self, node):
        if isinstance(node.func, ast.Name):
            self.calls.add(node.func.id)
        elif isinstance(node.func, ast.Attribute):
            self.calls.add(node.func.attr)

    def check_undefined(self):
        available = {'print', 'len', 'range', 'int', 'str', 'list', 'dict', 'set', 'tuple'}
        for imp in self.imports:
            try:
                module = importlib.import_module(imp)
                available.update(dir(module))
            except:
                pass

        for call in self.calls:
            if call not in available and not call.startswith('_'):
                self.undefined_calls.add(call)

        return self.undefined_calls

漏洞幻觉指LLM描述了不存在的漏洞类型或错误的漏洞原理。CTF题目基于真实的漏洞模式,但LLM可能编造了看似合理但实际上不存在的攻击向量。

approach = "Use double free to corrupt heap metadata and get shell. The binary has a use-after-free in the delete function that allows overwriting the malloc_hook pointer."

code = """
def exploit():
    allocate(0x100)
    allocate(0x100)
    free(0)
    free(1)
    free(0)

    overwrite(0, p64(malloc_hook))
    allocate(0x100)
    allocate(0x100)

    shellcode_location = 0x401000
    allocate(0x100, shellcode_location)
"""

这个例子描述了double free攻击,但实际题目可能根本没有UAF漏洞,或者free函数的实现不允许连续释放同一块内存。LLM基于训练数据中的相似题目产生了幻觉。

检测漏洞幻觉需要逆向分析实际二进制程序,确认声称的漏洞点是否真实存在。这超出了自动化系统的能力,需要人工介入或专门的程序分析工具。作为折中,可以要求LLM在exploitation代码中添加验证步骤,确认漏洞点确实可利用。

Payload幻觉指生成的攻击载荷语法错误或逻辑缺陷。payload可能包含未闭合的引号、错误的转义、无效的字符、错误的长度计算。这些错误往往很细微,人工审查容易遗漏。

sql_payload = "1' OR '1'='1'--"
xss_payload = "<script>alert(document.cookie)</script>"
format_payload = p64(0x400686) + b"A" * 40 + p64(0x400686)

buffer = b"A" * 72
chain = ROP(binary)
chain.call(0x400686, [0x400686])
payload = buffer + chain.chain()

第一个SQL payload可能缺少适当的注释符或引号转义,第二个XSS payload可能被CSP或输入过滤阻止,第三个format_payload重复使用同一地址可能导致逻辑错误,第四个ROP chain调用参数错误。

检测payload幻觉需要结合静态分析和动态测试。静态检查验证语法正确性,动态测试在沙箱环境中观察实际执行效果。对于网络类payload,可以生成伪造的响应环境测试payload是否能正确解析。

上下文幻觉指LLM基于错误的假设进行推理。可能假设了错误的环境配置、错误的二进制保护机制、错误的网络行为。这些假设导致后续所有推导都建立在错误基础上。

assumptions = """
Binary has no PIE, can use hardcoded addresses.
Stack is executable, can use shellcode.
Remote service runs on localhost:9000.
"""

code = f"""
binary = ELF('./challenge')
binary.address = 0x400000

shellcode = asm(shellcraft.sh())
payload = b"A" * 72 + p64(binary.address + 0x1000)

p = remote('localhost', 9000)
p.sendline(shellcode + payload)
"""

如果实际二进制开启了PIE保护,hardcode地址会失败;如果栈不可执行,shellcode无法运行;如果服务不在localhost或端口不是9000,连接会失败。

检测上下文幻觉需要实际探测环境。对于二进制题目,运行checksec工具确认保护机制;对于web题目,发送探测请求确认服务器行为;对于crypto题目,验证题目参数是否符合假设。环境探测的结果作为上下文反馈给LLM,要求基于真实环境重新生成。

Tool Fail分类与检测

Tool fail指LLM生成的代码在实际执行环境中失败。代码本身语法正确,逻辑也可能正确,但由于环境限制、资源约束、权限问题导致无法执行。这类错误只能在运行时检测。

依赖缺失是代码使用了未安装的库或工具。CTF题目可能需要特定的工具如pwntools、angr、z3,但执行环境可能没有预装。LLM可能假设这些工具可用,生成依赖它们的代码。

from pwn import *
import angr
import z3

proj = angr.Project('./challenge')
simgr = proj.factory.simgr()
simgr.explore(find=0x401168)

s = z3.Solver()
x = z3.BitVec('x', 32)
s.add(x > 100)
s.add(x < 200)

如果环境没有安装pwntools、angr、z3,import会失败。即使安装了,版本不兼容也可能导致运行时错误。

检测依赖缺失需要在隔离环境中尝试执行代码,捕获ImportError或ModuleNotFoundError。检测到缺失依赖后,重试策略有两种:提供替代实现使用标准库,或者明确告知LLM可用的工具列表要求重新生成。

权限不足指代码尝试执行需要特权的操作。绑定低端口、修改系统文件、访问其他进程、嗅探网络流量。这些操作在普通用户权限下会被拒绝。

s = socket.socket()
s.bind(('0.0.0.0', 80))

with open('/etc/passwd', 'w') as f:
    f.write('hacker:x:0:0:root:/root:/bin/bash')

os.system('iptables -F')

第一个例子绑定80端口需要root权限,第二个修改/etc/passwd需要root,第三个清空iptables规则需要root。

检测权限不足通过捕获PermissionDeniedError或分析退出码。重试策略是明确告知LLM在普通用户权限下操作,要求生成不需要特权的替代方案。

环境差异指代码假设的环境与实际不符。文件路径不存在、端口未开放、服务未运行、系统命令不存在。这些假设可能在LLM训练数据中常见,但在具体题目环境中不成立。

p = remote('127.0.0.1', 9000)
with open('/proc/self/maps') as f:
    maps = f.read()

output = subprocess.check_output(['gdb', './challenge'])
libc = ELF('/lib/x86_64-linux-gnu/libc.so.6')

如果服务监听在其他地址或端口,连接失败;如果/proc不可挂载,文件读取失败;如果gdb未安装,命令执行失败;如果libc路径不同,ELF加载失败。

检测环境差异通过捕获FileNotFoundError、ConnectionRefusedError、subprocess.CalledProcessError。重试策略是探测实际环境配置,将真实信息反馈给LLM。

资源限制指代码执行超出资源配额。运行时间过长、内存使用过多、磁盘空间不足。这些限制在沙箱环境中强制执行,防止失控代码耗尽系统资源。

def brute_force_key():
    for i in range(100000000):
        candidate = derive_key(i)
        if verify(candidate):
            return candidate

data = ["A" * 1000000 for _ in range(10000)]
with open('/tmp/large', 'w') as f:
    f.write(str(data))

第一个循环可能执行数小时,第二个列表可能消耗数十GB内存。

检测资源限制通过设置超时和内存限制,捕获TimeoutExpired或MemoryError。重试策略是要求LLM优化算法复杂度,或者提示时间限制要求生成更高效的实现。

调度器状态机设计

重试调度器的核心是一个确定性的状态机,管理每个解题尝试从生成到验证的完整生命周期。状态机定义了清晰的转换条件和转换动作,确保每种失败都有对应的处理路径,每个候选都经过完整的验证流程。

状态定义与转换

状态机包含9个状态:pending初始状态准备生成,generating调用LLM生成中,schema_retry格式错误重试中,validating验证内容正确性,hallucination_retry幻觉检测重试中,executing实际执行中,tool_retry工具错误重试中,succeeded成功完成,failed最终失败。

状态转换遵循严格的条件规则。从pending只能转换到generating,从generating根据生成结果转换到validating或schema_retry,从validating根据验证结果转换到executing或hallucination_retry,从executing根据执行结果转换到succeeded或tool_retry,各retry状态在达到最大重试次数后转换到failed。每个状态关联特定的动作和超时限制。generating状态的LLM调用通常限制在60秒内,防止网络延迟或模型负载导致的长时间等待。validating状态的静态分析应该快速完成,限制在10秒内。executing状态的执行时间根据题目类型调整,pwn题可能需要建立网络连接允许30秒,crypto题的计算密集型操作允许60秒,misc题的文件处理可能允许120秒。

状态转换伴随详细的日志记录。每条转换日志包含timestamp时间戳、attempt_id尝试标识、from_state源状态、to_state目标状态、reason转换原因、retry_count当前重试次数、strategy使用的策略、metadata附加信息。这些日志构成了完整的执行轨迹,用于事后分析和系统优化。

import time
import uuid
from dataclasses import dataclass, field
from typing import Optional, Dict, Any
from enum import Enum

class AttemptState(Enum):
    PENDING = "pending"
    GENERATING = "generating"
    SCHEMA_RETRY = "schema_retry"
    VALIDATING = "validating"
    HALLUCINATION_RETRY = "hallucination_retry"
    EXECUTING = "executing"
    TOOL_RETRY = "tool_retry"
    SUCCEEDED = "succeeded"
    FAILED = "failed"

@dataclass
class Attempt:
    attempt_id: str
    challenge_id: str
    state: AttemptState
    retry_count: int
    max_retries: int
    current_strategy: str
    llm_output: Optional[Dict[str, Any]]
    error_info: Optional[Dict[str, Any]]
    base_input: Any
    created_at: float
    updated_at: float
    transition_history: list = field(default_factory=list)

    def transition(self, new_state: AttemptState, reason: str, metadata: Optional[Dict] = None):
        old_state = self.state
        self.state = new_state
        self.updated_at = time.time()

        transition_record = {
            "from": old_state.value,
            "to": new_state.value,
            "reason": reason,
            "timestamp": self.updated_at,
            "retry_count": self.retry_count,
            "metadata": metadata or {}
        }
        self.transition_history.append(transition_record)

        return self

    def can_retry(self) -> bool:
        retry_states = [AttemptState.SCHEMA_RETRY, AttemptState.HALLUCINATION_RETRY, AttemptState.TOOL_RETRY]
        return self.state in retry_states and self.retry_count < self.max_retries

    def increment_retry(self):
        self.retry_count += 1
        return self.retry_count

状态机的实现需要考虑并发控制。多个attempt可能同时处于不同状态,需要线程安全的状态转换机制。使用线程锁保护状态更新,使用队列管理待处理的attempt,使用事件通知状态变化。

超时与强制终止

每个状态都有最大停留时间限制,防止某个attempt卡在中间状态消耗资源。超时机制使用异步定时器实现,当attempt在某个状态的停留时间超过阈值时,强制转换到failed状态。

import asyncio
from typing import Callable, Awaitable

class TimeoutManager:
    def __init__(self):
        self.timeouts = {
            AttemptState.GENERATING: 60,
            AttemptState.VALIDATING: 10,
            AttemptState.EXECUTING: 120,
            AttemptState.SCHEMA_RETRY: 120,
            AttemptState.HALLUCINATION_RETRY: 120,
            AttemptState.TOOL_RETRY: 120
        }

    async def run_with_timeout(self, attempt: Attempt, coro: Callable[[], Awaitable], state: AttemptState):
        timeout = self.timeouts.get(state, 60)
        try:
            result = await asyncio.wait_for(coro(), timeout=timeout)
            return result
        except asyncio.TimeoutError:
            attempt.transition(AttemptState.FAILED, reason=f"timeout in {state.value}")
            return None

超时时间根据实际操作调整。generating状态的LLM调用可能需要几十秒,特别是生成复杂代码时;validating状态的静态分析应该快速;executing状态根据题目类型设定,网络题目需要连接时间允许较长,计算题目需要CPU时间允许较长。

超时后的资源清理很重要。LLM调用可能占用网络连接,执行进程可能消耗内存,临时文件可能占用磁盘。强制终止后需要释放这些资源,避免资源泄漏导致后续尝试失败。

import subprocess
import psutil
import os

class ResourceCleaner:
    async def cleanup_process(self, pid: int):
        try:
            process = psutil.Process(pid)
            process.terminate()
            await asyncio.sleep(2)
            if process.is_running():
                process.kill()
        except psutil.NoSuchProcess:
            pass

    async def cleanup_temp_files(self, task_dir: str):
        import shutil
        try:
            shutil.rmtree(task_dir)
        except:
            pass

    async def cleanup_connection(self, connection):
        try:
            connection.close()
        except:
            pass

状态持久化与恢复

长时间运行的调度器可能因为崩溃、重启、资源回收而中断。为了不丢失已经生成的候选和中间状态,需要状态持久化机制。每次状态转换后持久化attempt对象到存储,重启时从存储恢复状态继续处理。

import pickle
import aiofiles
from pathlib import Path

class StateStore:
    def __init__(self, store_path: str):
        self.store_path = Path(store_path)
        self.store_path.mkdir(exist_ok=True)

    async def save_attempt(self, attempt: Attempt):
        file_path = self.store_path / f"{attempt.attempt_id}.pkl"
        async with aiofiles.open(file_path, 'wb') as f:
            await f.write(pickle.dumps(attempt))

    async def load_attempt(self, attempt_id: str) -> Optional[Attempt]:
        file_path = self.store_path / f"{attempt_id}.pkl"
        if not file_path.exists():
            return None
        async with aiofiles.open(file_path, 'rb') as f:
            data = await f.read()
            return pickle.loads(data)

    async def list_pending_attempts(self) -> list:
        pending = []
        for file_path in self.store_path.glob("*.pkl"):
            attempt = await self.load_attempt(file_path.stem)
            if attempt and attempt.state not in [AttemptState.SUCCEEDED, AttemptState.FAILED]:
                pending.append(attempt)
        return pending

恢复策略根据状态决定。对于generating状态的attempt,可能LLM调用已经完成但结果未记录,需要检查LLM服务的响应历史;对于executing状态的attempt,可能执行进程已经终止但结果未收集,需要检查进程输出文件;对于retry状态的attempt,可以直接重新进入该状态继续重试。

持久化需要考虑性能影响。每次状态转换都写磁盘可能成为瓶颈,特别是高并发生成大量候选时。优化方案是批量写入,积累一批转换后统一持久化;或者使用内存缓存加定期flush,平衡性能和可靠性。

输入输出契约设计

调度器与LLM、执行环境之间需要严格的契约定义。契约规定了输入格式、输出格式、验证规则,使得各模块可以独立开发和测试。契约验证在每个关键节点进行,确保数据流转的正确性。

LLM输入契约

LLM输入定义为结构化数据,包含题目信息、约束条件、历史尝试、示例输出。这种结构化输入比纯文本prompt更容易维护和扩展,便于模板化生成和版本控制。

from pydantic import BaseModel, Field, validator
from typing import List, Dict, Any, Optional

class ChallengeFile(BaseModel):
    name: str
    description: str
    content_type: str
    size_hint: Optional[int] = None

class LLMInput(BaseModel):
    challenge_id: str
    description: str
    category: str
    difficulty: int = Field(ge=1, le=10)
    files: List[ChallengeFile]
    constraints: Dict[str, Any]
    examples: Optional[List[Dict[str, Any]]] = None
    previous_attempts: Optional[List[Dict[str, Any]]] = None
    temperature: float = Field(default=0.5, ge=0.0, le=2.0)
    top_p: float = Field(default=0.9, ge=0.0, le=1.0)
    max_tokens: int = Field(default=2000, ge=100)

    @validator('category')
    def valid_category(cls, v):
        valid = ['pwn', 'web', 'crypto', 'misc', 'reverse']
        if v not in valid:
            raise ValueError(f"Invalid category: {v}")
        return v

    def to_prompt(self) -> str:
        sections = []

        sections.append(f"# Challenge: {self.challenge_id}")
        sections.append(f"Category: {self.category}")
        sections.append(f"Difficulty: {self.difficulty}/10")
        sections.append(f"\n## Description\n{self.description}")

        if self.files:
            sections.append("\n## Provided Files")
            for f in self.files:
                sections.append(f"- {f.name}: {f.description}")
                if f.size_hint:
                    sections.append(f"  Size: ~{f.size_hint} bytes")

        if self.constraints:
            sections.append("\n## Constraints")
            for key, value in self.constraints.items():
                sections.append(f"- {key}: {value}")

        if self.examples:
            sections.append("\n## Expected Output Format")
            for i, ex in enumerate(self.examples, 1):
                sections.append(f"\nExample {i}:")
                sections.append("```json")
                sections.append(json.dumps(ex, indent=2))
                sections.append("```")

        if self.previous_attempts:
            sections.append("\n## Previous Failed Attempts")
            for att in self.previous_attempts[-5:]:
                sections.append(f"- Strategy: {att.get('strategy', 'unknown')}")
                sections.append(f"  Error: {att.get('error', 'unknown error')}")
                if att.get('hint'):
                    sections.append(f"  Hint: {att['hint']}")

        sections.append("\n## Task")
        sections.append("Generate a complete solution including:")
        sections.append("1. Approach: Explain your understanding and methodology")
        sections.append("2. Code: Provide executable exploit or solve script")
        sections.append("3. Explanation: Detail key steps and reasoning")

        return "\n".join(sections)

契约定义包含验证规则。category字段限制为有效的CTF类别,difficulty范围1-10,temperature范围0-2,top_p范围0-1。这些验证在输入构造时执行,防止无效数据传递给LLM。

契约到prompt的转换是可定制的。不同模型可能偏好不同的prompt格式,不同任务可能需要强调不同方面。to_prompt方法提供默认实现,子类可以重写以适应特定需求。

LLM输出契约

LLM输出定义为强类型结构,包含解题必需的所有信息。使用pydantic进行自动验证,确保输出符合预期结构。

from pydantic import BaseModel, Field, validator
import re
import ast

class LLMOutput(BaseModel):
    approach: str = Field(..., min_length=50, max_length=2000)
    code: str = Field(..., min_length=20)
    explanation: str = Field(..., min_length=30, max_length=3000)
    expected_result: Optional[str] = None
    dependencies: Optional[List[str]] = None
    complexity: Optional[str] = Field(None, regex="^(low|medium|high)$")
    estimated_time: Optional[int] = Field(None, ge=1, le=300)

    @validator('code')
    def validate_code_syntax(cls, v):
        try:
            ast.parse(v)
        except SyntaxError as e:
            raise ValueError(f"Syntax error at line {e.lineno}: {e.msg}")
        return v

    @validator('expected_result')
    def validate_flag_format(cls, v):
        if v:
            if not re.search(r'flag\{[^}]+\}', v, re.IGNORECASE):
                raise ValueError("Expected result should contain flag in format: flag{...}")
        return v

    @validator('dependencies')
    def validate_dependencies(cls, v):
        if v:
            valid_packages = ['pwntools', 'angr', 'z3-solver', 'requests', 'beautifulsoup4',
                            'cryptography', 'numpy', 'sagemath', 'capstone', 'unicorn']
            for dep in v:
                if dep.lower() not in valid_packages:
                    raise ValueError(f"Unknown or unsupported package: {dep}")
        return v

    class Config:
        extra = 'forbid'

验证器在字段级别和模型级别执行。字段级验证检查语法正确性、格式合规性、值范围。模型级extra='forbid'禁止未定义字段,防止输出包含无关信息。

验证失败触发schema retry,失败的验证器提供具体错误信息。例如syntax error指出行号和错误类型,flag format error解释期望格式,dependency error列出有效选项。这些信息反馈给LLM用于改进。

执行环境契约

执行环境定义了代码运行的安全沙箱和资源限制。契约规定了语言支持、超时限制、内存限制、网络访问、文件访问等配置。

from pydantic import BaseModel, Field
from typing import Dict, List, Optional

class ExecutionConfig(BaseModel):
    language: str = Field(..., regex="^(python3|bash|c|cpp|node)$")
    timeout: int = Field(default=30, ge=5, le=300)
    memory_limit: int = Field(default=512, ge=64, le=4096)
    network_enabled: bool = False
    allowed_hosts: Optional[List[str]] = None
    max_output_size: int = Field(default=1048576, ge=10240)
    environment: Dict[str, str] = Field(default_factory=dict)
    working_directory: str = "/tmp/sandbox"

class ExecutionResult(BaseModel):
    success: bool
    exit_code: int
    stdout: str
    stderr: str
    duration: float
    memory_peak: Optional[int] = None
    error_type: Optional[str] = Field(None, regex="^(timeout|memory|crash|syntax|runtime)$")
    signal: Optional[int] = None

    def has_flag(self) -> bool:
        return bool(re.search(r'flag\{[^}]+\}', self.stdout, re.IGNORECASE))

    def extract_flag(self) -> Optional[str]:
        match = re.search(r'flag\{[^}]+\}', self.stdout, re.IGNORECASE)
        return match.group(0) if match else None

执行配置根据题目类型调整。pwn题需要网络访问和较长超时,crypto题需要较大内存和较长计算时间,web题需要特定网络白名单,misc题可能需要文件系统访问。这些配置在执行前注入到LLM输入,让生成代码符合环境约束。

执行结果包含详细的诊断信息。exit_code区分正常退出和异常终止,error_type分类错误原因,signal记录终止信号如SIGSEGV,memory_peak记录实际内存使用。这些信息用于判断重试策略和生成反馈。

重试变换策略

不同错误类型对应不同的重试变换策略。策略的核心是在保留原始解题思路的前提下,通过调整prompt参数、增加约束条件、提供反馈信息来引导LLM生成更正确的输出。

Schema Retry策略

Schema fail的根本原因是结构化约束不够明确。LLM可能没有理解输出格式的严格要求,或者在生成过程中被截断。重试策略逐步增强约束,从示例引导到强制模式。

第一次schema retry提供详细的格式说明和完整示例。示例应该与当前题目类型相似,展示期望的输出结构和详细程度。

def get_schema_retry_first_attempt(attempt: Attempt) -> LLMInput:
    base = attempt.base_input

    examples = [
        {
            "approach": "Buffer overflow in read() function. The program reads 200 bytes into a 72-byte buffer without bounds checking. Calculate offset to saved return address (72 bytes), then overwrite with address of win_function which prints flag.",
            "code": """from pwn import *

context.log_level = 'info'

p = remote('localhost', 9000)

offset = 72
ret_address = 0x400686

payload = b'A' * offset + p64(ret_address)

p.sendline(payload)
p.interactive()""",
            "explanation": "1. Connect to remote service on port 9000\n2. Craft payload with 72 bytes padding + return address\n3. Use pwntools p64() to pack address in little-endian format\n4. Send payload and interact to receive flag",
            "expected_result": "flag{buffer_overflow_simple}"
        }
    ]

    constraints = {
        **base.constraints,
        "output_format": "json",
        "required_fields": ["approach", "code", "explanation"],
        "min_approach_length": 50,
        "min_code_length": 20,
        "min_explanation_length": 30,
        "code_syntax": "must be valid Python syntax",
        "flag_format": "flag{...}"
    }

    error_feedback = {
        "error": "Previous output did not match required schema",
        "issues": attempt.error_info.get('validation_errors', []),
        "hint": "Carefully review the example format. Ensure all fields are present and meet minimum length requirements."
    }

    return LLMInput(
        **base.__dict__,
        examples=examples,
        constraints=constraints,
        previous_attempts=[error_feedback]
    )

第二次schema retry使用few-shot learning提供多个不同类型题目的示例。展示pwn、web、crypto等不同类别的输出,帮助LLM理解要求的通用性。

def get_schema_retry_second_attempt(attempt: Attempt) -> LLMInput:
    base = attempt.base_input
    category = base.category

    category_examples = {
        'pwn': {
            "approach": "Use-after-free vulnerability. Allocate chunks, free them, then use dangling pointer to write to freed memory. Overwrite malloc_hook with system address, then allocate chunk containing '/bin/sh'.",
            "code": """from pwn import *

p = process('./challenge')
libc = ELF('/lib/x86_64-linux-gnu/libc.so.6')

chunk_size = 0x100

allocate(0, chunk_size)
allocate(1, chunk_size)
free(0)
free(1)
free(0)

malloc_hook = libc.symbols['__malloc_hook']
system = libc.symbols['system']

write_to_freed(0, p64(malloc_hook - 0x10))

allocate(2, chunk_size)
allocate(3, chunk_size)
allocate(4, chunk_size, b'/bin/sh\\x00')

allocate(5, chunk_size, p64(system))""",
            "explanation": "Exploit fastbin dup to get malloc_hook allocation. Overwrite hook with system(). Allocate chunk containing '/bin/sh' string. Trigger malloc to get shell."
        },
        'web': {
            "approach": "SQL injection in login form. Input is not sanitized, allows bypassing authentication. Use ' OR '1'='1 payload to make query always return true.",
            "code": """import requests

url = 'http://challenge/login'
payload = {
    'username': "admin' OR '1'='1",
    'password': 'anything'
}

r = requests.post(url, data=payload)
if 'flag' in r.text:
    print(r.text)""",
            "explanation": "The query becomes: SELECT * FROM users WHERE username='admin' OR '1'='1' AND password=... Since '1'='1' is always true, the condition evaluates to true regardless of password."
        },
        'crypto': {
            "approach": "RSA with small exponent e=3 and no padding. Message is small so m^e < n. Can recover by taking integer cube root of ciphertext.",
            "code": """import gmpy2
from Crypto.Util.number import long_to_bytes

n = 0xDEADBEEF...
e = 3
c = 0x12345678...

m, exact = gmpy2.iroot(c, 3)
if exact:
    flag = long_to_bytes(m)
    print(flag.decode())""",
            "explanation": "When m^e < n, encryption doesn't use modular reduction. Direct integer root recovers plaintext."
        }
    }

    examples = [category_examples.get(category, category_examples['pwn'])]

    constraints = {
        **base.constraints,
        "strict_schema": True,
        "field_types": {
            "approach": "string",
            "code": "string",
            "explanation": "string"
        },
        "no_extra_fields": True
    }

    return LLMInput(
        **base.__dict__,
        examples=examples,
        constraints=constraints,
        temperature=0.3,
        max_tokens=2500
    )

第三次schema retry切换到强制模式。如果LLM API支持,使用function calling或JSON mode强制结构化输出。

def get_schema_retry_third_attempt(attempt: Attempt) -> LLMInput:
    base = attempt.base_input

    constraints = {
        **base.constraints,
        "use_function_calling": True,
        "function_name": "generate_solution",
        "function_parameters": {
            "type": "object",
            "properties": {
                "approach": {"type": "string", "minLength": 50},
                "code": {"type": "string", "minLength": 20},
                "explanation": {"type": "string", "minLength": 30}
            },
            "required": ["approach", "code", "explanation"]
        }
    }

    return LLMInput(
        **base.__dict__,
        constraints=constraints,
        temperature=0.1,
        max_tokens=2000
    )

Hallucination Retry策略

Hallucination的根本原因是LLM生成的代码包含不存在的元素或错误的逻辑。重试策略的目标是减少创造性空间,增加事实约束,引导LLM生成更可验证的输出。

第一次hallucination retry要求LLM验证所有使用的函数和API。在prompt中列出可用的标准库和已安装的包,要求LLM仅使用这些已验证的API。

def get_hallucination_retry_first(attempt: Attempt) -> LLMInput:
    base = attempt.base_input
    error_details = attempt.error_info.get('details', [])
    unknown_functions = [e.get('function') for e in error_details if e.get('type') == 'unknown_function']

    available_apis = {
        'standard_library': [
            'os', 'sys', 're', 'json', 'time', 'datetime', 'math', 'random',
            'hashlib', 'struct', 'socket', 'subprocess', 'pathlib', 'io',
            'collections', 'itertools', 'functools', 'operator'
        ],
        'ctf_libraries': [
            'pwntools: pwn, remote, process, p64, u64, ELF, ROP',
            'requests: get, post, Session',
            'cryptography: hashes, ciphers, modes',
            'z3: Solver, BitVec, sat'
        ]
    }

    if unknown_functions:
        constraints = {
            **base.constraints,
            "verified_apis_only": True,
            "disallow_functions": unknown_functions,
            "available_apis": available_apis
        }

        feedback = {
            "error": "Code uses undefined functions",
            "undefined": unknown_functions,
            "hint": f"Replace these with available APIs. For CTF tasks, pwntools is commonly used for pwn, requests for web."
        }
    else:
        logic_errors = [e for e in error_details if e.get('type') == 'logic_error']
        constraints = {
            **base.constraints,
            "verify_logic": True,
            "add_assertions": True
        }

        feedback = {
            "error": "Code contains logic errors",
            "errors": logic_errors,
            "hint": "Add assertions to verify assumptions, check edge cases, validate calculations."
        }

    return LLMInput(
        **base.__dict__,
        constraints=constraints,
        previous_attempts=[feedback],
        temperature=0.2
    )

第二次hallucination retry要求分步骤生成并验证。不要一次性生成完整代码,而是先生成思路框架,验证后再实现每个步骤。

def get_hallucination_retry_second(attempt: Attempt) -> LLMInput:
    base = attempt.base_input

    constraints = {
        **base.constraints,
        "step_by_step": True,
        "verify_each_step": True,
        "explain_assumptions": True,
        "add_error_handling": True
    }

    feedback = {
        "error": "Previous approach had unverified assumptions",
        "hint": "Break down into steps: 1) Analyze binary/app, 2) Identify vulnerability, 3) Design exploit, 4) Test each component",
        "requirement": "For each step, explain: what you're doing, why it works, what assumptions you make, how to verify"
    }

    return LLMInput(
        **base.__dict__,
        constraints=constraints,
        previous_attempts=[feedback],
        temperature=0.3
    )

第三次hallucination retry要求添加诊断和调试代码。即使最终exploit失败,诊断输出也能提供失败原因的线索。

def get_hallucination_retry_third(attempt: Attempt) -> LLMInput:
    base = attempt.base_input

    constraints = {
        **base.constraints,
        "diagnostic_mode": True,
        "add_logging": True,
        "verbose_output": True,
        "test_components": True
    }

    feedback = {
        "error": "Need more diagnostic information",
        "hint": "Add print statements to show: offsets calculated, addresses found, data sent/received, crash locations",
        "requirement": "Code should output enough information to debug even if exploit fails"
    }

    return LLMInput(
        **base.__dict__,
        constraints=constraints,
        previous_attempts=[feedback],
        temperature=0.4
    )

Tool Retry策略

Tool fail的根本原因是代码与环境不匹配。重试策略的目标是获取真实环境信息,调整代码假设,或者提供回退实现。

第一次tool retry探测环境并反馈。在重试前运行环境探测脚本,获取真实的配置信息。

async def get_tool_retry_first(attempt: Attempt) -> LLMInput:
    base = attempt.base_input
    error_type = attempt.error_info.get('type')

    if error_type == 'missing_dependency':
        missing = attempt.error_info.get('dependency')

        env_info = await detect_environment()
        available = env_info.get('available_packages', [])

        alternatives = {
            'pwntools': 'use raw socket and struct module for packing',
            'angr': 'use static analysis with objdump or radare2',
            'z3-solver': 'use brute force or custom solver for small search space',
            'requests': 'use urllib from standard library',
            'beautifulsoup4': 'use regex for simple HTML parsing'
        }

        constraints = {
            **base.constraints,
            "avoid_package": missing,
            "available_packages": available,
            "use_alternatives": True
        }

        feedback = {
            "error": f"{missing} is not installed",
            "available": available,
            "hint": alternatives.get(missing, f"Implement using standard library"),
            "alternative_implementation": True
        }

    elif error_type == 'permission_denied':
        constraints = {
            **base.constraints,
            "user_level_only": True,
            "no_privilege_escalation": True,
            "no_root_required": True
        }

        feedback = {
            "error": "Operation requires elevated privileges",
            "hint": "Find user-level exploitation path. For pwn: don't bind privileged ports. For web: don't modify system files.",
            "assumption": "You are running as unprivileged user"
        }

    elif error_type == 'timeout':
        original_timeout = attempt.error_info.get('timeout', 30)

        constraints = {
            **base.constraints,
            "optimize_performance": True,
            "timeout_limit": original_timeout,
            "reduce_complexity": True
        }

        feedback = {
            "error": f"Execution exceeded {original_timeout} seconds",
            "hint": "Optimize algorithm, reduce search space, use heuristics instead of brute force",
            "requirement": f"Must complete within {original_timeout} seconds"
        }

    else:
        constraints = {
            **base.constraints,
            "add_error_handling": True,
            "graceful_degradation": True
        }

        feedback = {
            "error": f"Execution failed: {attempt.error_info.get('stderr', '')}",
            "hint": "Add try-except blocks, validate assumptions before operations, check return values"
        }

    return LLMInput(
        **base.__dict__,
        constraints=constraints,
        previous_attempts=[feedback]
    )

async def detect_environment():
    info = {"available_packages": []}

    packages_to_check = ['pwntools', 'angr', 'z3', 'requests', 'bs4', 'numpy', 'sagemath']
    for pkg in packages_to_check:
        try:
            __import__(pkg)
            info["available_packages"].append(pkg)
        except ImportError:
            pass

    info["has_network"] = await check_network_access()
    info["max_memory"] = get_available_memory()

    return info

第二次tool retry提供简化的回退实现。如果高级方案无法实现,提供使用基础方法的简化版本。

def get_tool_retry_second(attempt: Attempt) -> LLMInput:
    base = attempt.base_input

    constraints = {
        **base.constraints,
        "prefer_simple_solutions": True,
        "avoid_complex_libraries": True,
        "use_standard_library": True,
        "minimal_dependencies": True
    }

    feedback = {
        "error": "Previous solution too complex or unavailable",
        "hint": "Use simplest approach. For pwn: manual payload construction. For crypto: mathematical formulas. For web: raw HTTP requests.",
        "fallback_strategy": "Implement core logic without advanced features"
    }

    return LLMInput(
        **base.__dict__,
        constraints=constraints,
        previous_attempts=[feedback],
        temperature=0.2
    )

并行候选生成与去重

单次重试虽然能提升成功率,但串行执行时间成本高。更好的策略是并行生成多个候选,同时探索不同的解题思路。并行生成的关键是在探索多样性和避免冗余之间找到平衡。

并行调度策略

并行调度器维护多个并行的attempt,每个attempt使用不同的随机种子、温度参数、或初始prompt。温度参数控制输出随机性,高温度产生更多样化但风险更高的输出,低温度产生更保守但稳定的输出。通过组合不同温度,可以同时探索高风险高收益和低风险低收益的策略空间。

import asyncio
import random
import uuid
from typing import List, Dict, Any
from dataclasses import dataclass

@dataclass
class GenerationStrategy:
    name: str
    temperature: float
    top_p: float
    top_k: int
    prompt_template: str
    constraints: Dict[str, Any]

class ParallelScheduler:
    def __init__(self, max_parallel: int = 5):
        self.max_parallel = max_parallel
        self.strategies = self._build_strategies()

    def _build_strategies(self) -> List[GenerationStrategy]:
        return [
            GenerationStrategy(
                name="conservative",
                temperature=0.2,
                top_p=0.9,
                top_k=40,
                prompt_template="standard",
                constraints={"focus": "reliability", "avoid_risk": True}
            ),
            GenerationStrategy(
                name="balanced",
                temperature=0.5,
                top_p=0.92,
                top_k=50,
                prompt_template="standard",
                constraints={"focus": "balance"}
            ),
            GenerationStrategy(
                name="creative",
                temperature=0.8,
                top_p=0.95,
                top_k=60,
                prompt_template="exploratory",
                constraints={"focus": "novelty", "explore_alternatives": True}
            ),
            GenerationStrategy(
                name="analytical",
                temperature=0.3,
                top_p=0.9,
                top_k=40,
                prompt_template="step_by_step",
                constraints={"focus": "logic", "verify_steps": True}
            ),
            GenerationStrategy(
                name="aggressive",
                temperature=0.7,
                top_p=0.96,
                top_k=70,
                prompt_template="exploratory",
                constraints={"focus": "efficiency", "optimize": True}
            )
        ]

    async def generate_candidates(self, challenge: LLMInput) -> List[Attempt]:
        tasks = []

        for i in range(min(len(self.strategies), self.max_parallel)):
            strategy = self.strategies[i]
            attempt_id = f"{challenge.challenge_id}_{strategy.name}_{uuid.uuid4().hex[:8]}"

            attempt_input = LLMInput(
                **challenge.__dict__,
                temperature=strategy.temperature,
                top_p=strategy.top_p,
                constraints={**challenge.constraints, **strategy.constraints}
            )

            attempt = Attempt(
                attempt_id=attempt_id,
                challenge_id=challenge.challenge_id,
                state=AttemptState.PENDING,
                retry_count=0,
                max_retries=3,
                current_strategy=strategy.name,
                llm_output=None,
                error_info=None,
                base_input=attempt_input,
                created_at=time.time(),
                updated_at=time.time()
            )

            tasks.append(self._generate_single(attempt))

        results = await asyncio.gather(*tasks, return_exceptions=True)
        attempts = [r for r in results if isinstance(r, Attempt)]

        return attempts

    async def _generate_single(self, attempt: Attempt) -> Attempt:
        attempt.transition(AttemptState.GENERATING, "start_generation")

        try:
            output = await self._call_llm(attempt.base_input)

            validated = LLMOutput(**output)
            attempt.llm_output = validated.dict()
            attempt.transition(AttemptState.VALIDATING, "generation_success")

        except Exception as e:
            attempt.error_info = {"type": "generation_error", "message": str(e)}
            attempt.transition(AttemptState.SCHEMA_RETRY, "generation_failed")

        return attempt

    async def _call_llm(self, input_data: LLMInput) -> Dict[str, Any]:
        prompt = input_data.to_prompt()

        response = await llm_api_call(
            prompt=prompt,
            temperature=input_data.temperature,
            top_p=input_data.top_p,
            max_tokens=input_data.max_tokens
        )

        return response

并行度选择需要考虑成本和收益。更多并行候选提高成功率,但也增加token消耗和执行时间。最优并行度根据题目难度调整:简单题目2-3个候选足够,困难题目可能需要5-7个候选。

自适应并行策略根据实时反馈调整。如果前几个候选成功率很高,可以减少后续生成;如果前几个都失败,可能需要增加并行度或切换策略。

class AdaptiveScheduler(ParallelScheduler):
    def __init__(self, min_parallel: int = 2, max_parallel: int = 10):
        super().__init__(max_parallel)
        self.min_parallel = min_parallel
        self.success_rate_history = []

    async def generate_candidates_adaptive(self, challenge: LLMInput) -> List[Attempt]:
        initial_count = max(self.min_parallel, min(3, self.max_parallel))

        attempts = await super().generate_candidates(challenge)
        initial_attempts = attempts[:initial_count]

        early_results = await self._validate_batch(initial_attempts)
        success_count = sum(1 for a in early_results if a.state == AttemptState.SUCCEEDED)

        self.success_rate_history.append(success_count / len(initial_attempts))

        if success_count > 0:
            return early_results

        avg_success_rate = sum(self.success_rate_history[-5:]) / len(self.success_rate_history)

        if avg_success_rate < 0.2 and len(attempts) < self.max_parallel:
            additional_count = min(3, self.max_parallel - len(attempts))
            additional_attempts = await self._generate_additional(challenge, additional_count)
            attempts.extend(additional_attempts)

        return attempts

候选去重机制

候选去重防止资源浪费。多个候选可能在approach描述上不同,但生成的代码本质上是相同的。执行多个相同候选不会提供额外价值,只会消耗时间和计算资源。

去重需要在代码语义级别进行。简单的字符串匹配不够,因为变量命名、注释、代码格式可能不同但逻辑相同。真正的去重需要理解代码的抽象语法树和控制流。

import ast
import hashlib
from typing import Set, Tuple

class CodeNormalizer:
    def normalize(self, code: str) -> ast.AST:
        try:
            tree = ast.parse(code)
            return self._normalize_ast(tree)
        except:
            return None

    def _normalize_ast(self, tree: ast.AST) -> ast.AST:
        class NameNormalizer(ast.NodeTransformer):
            def visit_Name(self, node: ast.Name):
                node.id = 'var'
                return node

            def visit_FunctionDef(self, node: ast.FunctionDef):
                node.name = 'func'
                self.generic_visit(node)
                return node

        normalizer = NameNormalizer()
        return normalizer.visit(tree)

class SemanticHasher:
    def __init__(self):
        self.normalizer = CodeNormalizer()

    def hash_code(self, code: str) -> str:
        normalized = self.normalizer.normalize(code)
        if normalized is None:
            return hashlib.sha256(code.encode()).hexdigest()

        structure = self._extract_structure(normalized)
        structure_str = json.dumps(structure, sort_keys=True)
        return hashlib.sha256(structure_str.encode()).hexdigest()

    def _extract_structure(self, tree: ast.AST) -> Dict:
        structure = {"nodes": [], "edges": []}

        for node in ast.walk(tree):
            node_info = {
                "type": type(node).__name__,
                "lineno": getattr(node, 'lineno', 0)
            }

            if isinstance(node, ast.Call):
                if isinstance(node.func, ast.Name):
                    node_info["call"] = node.func.id
                elif isinstance(node.func, ast.Attribute):
                    node_info["call"] = node.func.attr

            if isinstance(node, (ast.Import, ast.ImportFrom)):
                node_info["imports"] = [alias.name for alias in node.names]

            structure["nodes"].append(node_info)

        return structure

基于语义哈希的去重可以检测逻辑相同的代码。即使变量名不同、代码风格不同,如果AST结构和关键操作相同,哈希值也会相同或相似。

去重阈值可以调整。严格去重只保留完全相同的候选之一,宽松去重允许相似候选并存。相似度度量可以使用编辑距离、余弦相似度、或者手动定义的特征向量。

class CandidateDeduplicator:
    def __init__(self, similarity_threshold: float = 0.85):
        self.seen_hashes: Set[str] = set()
        self.seen_patterns: Set[Tuple] = set()
        self.similarity_threshold = similarity_threshold

    def extract_patterns(self, code: str) -> Set[Tuple]:
        patterns = set()
        tree = ast.parse(code)

        for node in ast.walk(tree):
            if isinstance(node, ast.Call):
                if isinstance(node.func, ast.Name):
                    patterns.add(('call', node.func.id))
                elif isinstance(node.func, ast.Attribute):
                    patterns.add(('call', node.func.attr))

            elif isinstance(node, ast.Str):
                if len(node.s) < 50:
                    patterns.add(('string', hashlib.md5(node.s.encode()).hexdigest()[:8]))

            elif isinstance(node, ast.Constant):
                if isinstance(node.value, int) and 0 < node.value < 10000:
                    patterns.add(('int', node.value))

        return patterns

    def is_duplicate(self, attempt: Attempt) -> bool:
        code = attempt.llm_output.get('code', '')
        if not code:
            return False

        semantic_hash = SemanticHasher().hash_code(code)
        if semantic_hash in self.seen_hashes:
            return True

        patterns = self.extract_patterns(code)
        pattern_overlap = len(patterns & self.seen_patterns)

        if pattern_overlap >= 5:
            return True

        self.seen_hashes.add(semantic_hash)
        self.seen_patterns.update(patterns)

        return False

去重影响成功率和成本的权衡。过于激进的去重可能移除有效的不同方法,过于保守的去重保留冗余候选浪费资源。最优阈值根据实际数据调优,可以通过A/B测试确定。

成功率概率分析

重试调度器的核心价值在于将单次低概率事件转化为高概率复合事件。通过数学分析可以量化并行候选和重试次数对最终成功率的影响,为参数选择提供理论依据。

基础概率模型

设单次LLM调用生成有效候选的概率为p_valid,有效候选中实际能解题的概率为p_solve。单次尝试的成功率为:

实际观测中,p_valid约为0.6-0.7,p_solve约为0.25-0.35。取p_valid=0.65,p_solve=0.30,则p_single=0.195,约19.5%。

如果生成n个独立候选且每个候选都尝试执行,至少一个成功的概率为:

当n=1时,P=0.195;当n=3时,P=1-(0.805)^3≈0.48;当n=5时,P=1-(0.805)^5≈0.67。这意味着5个并行候选可以将成功率从19.5%提升到67%,提升约3.4倍。

这个模型假设候选完全独立,实际中候选之间可能相关。如果使用相同的prompt模板只是参数不同,候选可能在错误类型上相关,降低实际成功率。去重机制会移除相似候选,也会减少独立性。

考虑重试的扩展模型

考虑重试机制,候选可以经历多轮修复。每轮重试有各自的成功率。设第i次schema retry的成功率为p_schema_i,第i次hallucination retry的成功率为p_hallucination_i,第i次tool retry的成功率为p_tool_i。

定义状态转移概率:

P(generating → validating) = p_valid

P(generating → schema_retry) = 1 - p_valid

P(validating → executing) = p_clean

P(validating → hallucination_retry) = 1 - p_clean

P(executing → succeeded) = p_solve

P(executing → tool_retry) = 1 - p_solve

其中p_clean是代码通过hallucination检查的概率。

给定最大重试次数k,一个候选最终成功的概率是所有可能成功路径的概率之和:

其中P_retry_i是第i次重试成功的概率。

计算一个具体例子。设p_valid=0.65,p_clean=0.75,p_solve=0.30,k=2。

第一次尝试成功:0.65 × 0.75 × 0.30 = 0.146

第一次schema retry成功:(0.35 × 0.70) × 0.75 × 0.30 = 0.055

第一次hallucination retry成功:0.65 × (0.25 × 0.50) × 0.30 = 0.024

第一次tool retry成功:0.65 × 0.75 × (0.70 × 0.60) = 0.205

第二次重试类似计算,但由于已经经过一次失败,成功率会下降。

总的候选成功概率约为0.15-0.25之间,取决于重试策略的有效性。

并行加重的组合效果

结合n个并行候选和每候选k次重试,总成功率模型为:

当P_candidate=0.20,n=5时:

约67%的成功率。

当P_candidate=0.25,n=5时:

约76%的成功率。

当P_candidate=0.20,n=7时:

约79%的成功率。

这表明提升单候选成功率或增加并行数量都能提高总成功率,但存在边际收益递减。从5个候选增加到7个候选,成功率从67%提升到79%,增加了12个百分点。提升单候选成功率从20%到25%,5个候选的成功率从67%提升到76%,增加了9个百分点。

成本效益分析

定义成本函数C(n, k)为n个候选每候选最多k次重试的期望token消耗:

其中E[tokens]是单次LLM调用的期望token数,r_retry是实际触发重试的比例。

设E[tokens]=1500,r_retry=0.5,n=5,k=2:

15000 tokens的消耗。

定义成功率为P_total,单位成功率的成本效益为:

当P_total=0.67,C=15000时,Efficiency=0.0000447。

比较不同配置:

n=3, k=2: P≈0.48, C=9000, Efficiency≈0.000053

n=5, k=2: P≈0.67, C=15000, Efficiency≈0.000045

n=5, k=3: P≈0.72, C=18750, Efficiency≈0.000038

n=7, k=2: P≈0.79, C=21000, Efficiency≈0.000038

从成本效益看,3个候选2次重试最优,但绝对成功率只有48%。5个候选2次重试在效率和成功率间取得平衡。更多候选或更多重试虽然提高成功率,但边际效益递减。

实际选择取决于应用场景。对于CTF比赛中的关键题目,高成功率更重要,可以接受更高成本。对于批量自动化分析,成本效率更关键,应该选择较低的n和k。

相似性衰减修正

上述模型假设候选完全独立,实际中候选通过prompt共享信息,可能产生相似错误。引入相似性系数ρ∈[0,1]表示候选间的相关程度,ρ=0完全独立,ρ=1完全相关。

修正的成功率公式为:

当ρ=0.2表示20%的相关性,n=5,p_single=0.195时:从67%降低到57%,减少10个百分点。

相似性ρ可以通过历史数据估计。计算候选对的联合成功率,如果独立则P(AB)=P(A)×P(B),如果有相关则P(AB)>P(A)×P(B)。ρ的估计值为:

去重机制可以降低ρ,移除高度相似的候选。但过度去重也可能移除真正不同的方法,需要权衡。

完整调度器实现

整合以上组件,构建完整的重试调度器。调度器负责协调整个流程:并行生成候选、验证schema、检测hallucination、执行、处理失败、记录日志。

核心调度器类

import asyncio
import logging
from typing import List, Optional, Dict, Any
from pathlib import Path

class RetryScheduler:
    def __init__(
        self,
        llm_client,
        executor,
        max_parallel: int = 5,
        max_retries: int = 3,
        log_path: str = "scheduler.log"
    ):
        self.llm_client = llm_client
        self.executor = executor
        self.max_parallel = max_parallel
        self.max_retries = max_retries
        self.parallel_scheduler = ParallelScheduler(max_parallel)
        self.deduplicator = CandidateDeduplicator()
        self.logger = SchedulerLogger(log_path)
        self.state_store = StateStore("state_store")

    async def solve_challenge(self, challenge: LLMInput) -> Optional[Dict[str, Any]]:
        challenge_id = challenge.challenge_id

        self.logger.log_challenge_start(challenge_id)

        attempts = await self.parallel_scheduler.generate_candidates(challenge)

        valid_attempts = []
        for attempt in attempts:
            if not self._should_filter(attempt):
                if not self.deduplicator.is_duplicate(attempt):
                    valid_attempts.append(attempt)
                else:
                    attempt.transition(AttemptState.FAILED, {"reason": "duplicate"})
                    self.logger.log_transition(attempt, "duplicate")

        for attempt in valid_attempts:
            await self.state_store.save_attempt(attempt)

            result = await self._process_attempt(attempt)

            if result and result.get('success'):
                self.logger.log_challenge_success(challenge_id, result)
                return result

        self.logger.log_challenge_failure(challenge_id)
        return None

    async def _process_attempt(self, attempt: Attempt) -> Optional[Dict[str, Any]]:
        while attempt.retry_count < attempt.max_retries:
            current_state = attempt.state

            if current_state == AttemptState.VALIDATING:
                result = await self._validate_content(attempt)
                if result.get('hallucination_detected'):
                    attempt.increment_retry()
                    retry_input = get_hallucination_retry_strategy(attempt)
                    attempt = await self._regenerate(attempt, retry_input)
                    continue
                else:
                    attempt.transition(AttemptState.EXECUTING, "validation_passed")
                    await self.state_store.save_attempt(attempt)

            elif current_state == AttemptState.EXECUTING:
                exec_result = await self._execute_attempt(attempt)
                if exec_result.get('tool_fail'):
                    attempt.increment_retry()
                    retry_input = get_tool_retry_strategy(attempt)
                    attempt = await self._regenerate(attempt, retry_input)
                    continue
                elif exec_result.get('success'):
                    attempt.transition(AttemptState.SUCCEEDED, "exploit_succeeded")
                    await self.state_store.save_attempt(attempt)
                    return exec_result
                else:
                    attempt.increment_retry()
                    attempt.transition(AttemptState.FAILED, {"reason": "execution_failed"})

            elif current_state in [AttemptState.SCHEMA_RETRY, AttemptState.HALLUCINATION_RETRY, AttemptState.TOOL_RETRY]:
                retry_input = self._get_retry_input(attempt)
                attempt = await self._regenerate(attempt, retry_input)

            else:
                break

        attempt.transition(AttemptState.FAILED, {"reason": "max_retries_exceeded"})
        await self.state_store.save_attempt(attempt)
        return None

    async def _validate_content(self, attempt: Attempt) -> Dict[str, Any]:
        output = attempt.llm_output
        code = output.get('code', '')

        hallucination_checks = []

        missing_funcs = self._check_missing_imports(code)
        if missing_funcs:
            hallucination_checks.append({
                'type': 'unknown_function',
                'functions': missing_funcs
            })

        logic_errors = self._check_logic_errors(code)
        if logic_errors:
            hallucination_checks.append({
                'type': 'logic_error',
                'errors': logic_errors
            })

        unsafe_patterns = self._check_unsafe_patterns(code)
        if unsafe_patterns:
            hallucination_checks.append({
                'type': 'unsafe_code',
                'patterns': unsafe_patterns
            })

        return {
            'hallucination_detected': len(hallucination_checks) > 0,
            'details': hallucination_checks
        }

    async def _execute_attempt(self, attempt: Attempt) -> Dict[str, Any]:
        output = attempt.llm_output
        code = output.get('code', '')
        language = self._detect_language(code)

        config = ExecutionConfig(
            language=language,
            timeout=30,
            memory_limit=512,
            network_enabled=attempt.base_input.constraints.get('network_enabled', False),
            files=attempt.base_input.constraints.get('files', {}),
            environment={}
        )

        self.logger.log_execution_start(attempt.attempt_id, config)

        result = await self.executor.execute(code, config)

        self.logger.log_execution_complete(attempt.attempt_id, result)

        if result.success and result.has_flag():
            return {
                'success': True,
                'attempt_id': attempt.attempt_id,
                'strategy': attempt.current_strategy,
                'output': result.stdout,
                'flag': result.extract_flag()
            }
        elif result.error_type:
            return {
                'success': False,
                'tool_fail': True,
                'error_type': result.error_type,
                'stderr': result.stderr
            }
        else:
            return {
                'success': False,
                'reason': 'no_flag_found'
            }

    async def _regenerate(self, attempt: Attempt, new_input: LLMInput) -> Attempt:
        attempt.base_input = new_input
        attempt.transition(AttemptState.GENERATING, "regeneration")

        try:
            output = await self.llm_client.generate(new_input)
            validated = LLMOutput(**output)

            attempt.llm_output = validated.dict()
            attempt.transition(AttemptState.VALIDATING, "regeneration_success")

        except Exception as e:
            attempt.error_info = {"type": "generation_error", "message": str(e)}
            attempt.transition(AttemptState.SCHEMA_RETRY, "regeneration_failed")

        await self.state_store.save_attempt(attempt)
        return attempt

    def _should_filter(self, attempt: Attempt) -> bool:
        if attempt.state == AttemptState.FAILED:
            return True
        if attempt.retry_count >= attempt.max_retries:
            return True
        return False

    def _get_retry_input(self, attempt: Attempt) -> LLMInput:
        if attempt.state == AttemptState.SCHEMA_RETRY:
            return get_schema_retry_strategy(attempt)
        elif attempt.state == AttemptState.HALLUCINATION_RETRY:
            return get_hallucination_retry_strategy(attempt)
        elif attempt.state == AttemptState.TOOL_RETRY:
            return get_tool_retry_strategy(attempt)
        else:
            return attempt.base_input

    def _check_missing_imports(self, code: str) -> List[str]:
        tree = ast.parse(code)
        imports = set()

        for node in ast.walk(tree):
            if isinstance(node, ast.Import):
                for alias in node.names:
                    imports.add(alias.name.split('.')[0])
            elif isinstance(node, ast.ImportFrom):
                if node.module:
                    imports.add(node.module.split('.')[0])

        standard_lib = {'os', 'sys', 're', 'json', 'time', 'math', 'random',
                       'hashlib', 'struct', 'socket', 'subprocess', 'pathlib'}

        missing = []
        for imp in imports:
            if imp not in standard_lib:
                try:
                    __import__(imp)
                except ImportError:
                    missing.append(imp)

        return missing

    def _check_logic_errors(self, code: str) -> List[str]:
        errors = []
        tree = ast.parse(code)

        for node in ast.walk(tree):
            if isinstance(node, ast.BinOp):
                if isinstance(node.op, (ast.Add, ast.Mult)):
                    if isinstance(node.right, ast.Constant):
                        if isinstance(node.right.value, int) and node.right.value > 10000000:
                            errors.append(f"Very large constant {node.right.value} may cause overflow")

            elif isinstance(node, ast.Call):
                if isinstance(node.func, ast.Name):
                    if node.func.id in ['eval', 'exec']:
                        errors.append(f"Unsafe use of {node.func.id}")

                    args = len(node.args)
                    if node.func.id == 'open' and args < 2:
                        errors.append("open() without mode argument")

        return errors

    def _check_unsafe_patterns(self, code: str) -> List[str]:
        unsafe = []
        patterns = [
            (r'system\(.*/rm', 'potential file deletion'),
            (r'socket.*bind.*0', 'binding privileged port'),
            (r'__import__\s*\(', 'dynamic import may be unsafe')
        ]

        for pattern, desc in patterns:
            if re.search(pattern, code):
                unsafe.append(desc)

        return unsafe

    def _detect_language(self, code: str) -> str:
        if code.strip().startswith('#!'):
            first_line = code.split('\n')[0]
            if 'python' in first_line:
                return 'python'
            elif 'bash' in first_line:
                return 'bash'

        if re.search(r'from\s+\w+\s+import|import\s+\w+', code):
            return 'python'
        elif code.startswith(('#!/bin/bash', '#!/bin/sh')):
            return 'bash'

        return 'python'

日志系统

日志系统记录完整执行轨迹,用于调试、分析、优化。日志结构化存储,便于查询和可视化。

import json
from datetime import datetime, timezone
from typing import Dict, Any, Optional

class SchedulerLogger:
    def __init__(self, log_path: str):
        self.log_path = Path(log_path)
        self.log_path.parent.mkdir(parents=True, exist_ok=True)

    def log_challenge_start(self, challenge_id: str):
        self._write_log({
            "event": "challenge_start",
            "timestamp": datetime.now(timezone.utc).isoformat(),
            "challenge_id": challenge_id
        })

    def log_challenge_success(self, challenge_id: str, result: Dict):
        self._write_log({
            "event": "challenge_success",
            "timestamp": datetime.now(timezone.utc).isoformat(),
            "challenge_id": challenge_id,
            "attempt_id": result.get('attempt_id'),
            "strategy": result.get('strategy'),
            "flag": result.get('flag')
        })

    def log_challenge_failure(self, challenge_id: str):
        self._write_log({
            "event": "challenge_failure",
            "timestamp": datetime.now(timezone.utc).isoformat(),
            "challenge_id": challenge_id
        })

    def log_transition(self, attempt: Attempt, reason: str, metadata: Optional[Dict] = None):
        self._write_log({
            "event": "state_transition",
            "timestamp": datetime.now(timezone.utc).isoformat(),
            "attempt_id": attempt.attempt_id,
            "challenge_id": attempt.challenge_id,
            "from_state": attempt.transition_history[-2]['to'] if len(attempt.transition_history) >= 2 else 'pending',
            "to_state": attempt.state.value,
            "reason": reason,
            "retry_count": attempt.retry_count,
            "strategy": attempt.current_strategy,
            "metadata": metadata or {}
        })

    def log_llm_call(self, attempt_id: str, input_data: LLMInput, output: Dict, tokens_used: int, duration: float):
        self._write_log({
            "event": "llm_call",
            "timestamp": datetime.now(timezone.utc).isoformat(),
            "attempt_id": attempt_id,
            "input_length": len(input_data.to_prompt()),
            "output_length": len(str(output)),
            "tokens_used": tokens_used,
            "duration": duration,
            "temperature": input_data.temperature,
            "strategy": input_data.constraints.get('focus', 'unknown')
        })

    def log_execution_start(self, attempt_id: str, config: ExecutionConfig):
        self._write_log({
            "event": "execution_start",
            "timestamp": datetime.now(timezone.utc).isoformat(),
            "attempt_id": attempt_id,
            "language": config.language,
            "timeout": config.timeout,
            "memory_limit": config.memory_limit,
            "network_enabled": config.network_enabled
        })

    def log_execution_complete(self, attempt_id: str, result: ExecutionResult):
        self._write_log({
            "event": "execution_complete",
            "timestamp": datetime.now(timezone.utc).isoformat(),
            "attempt_id": attempt_id,
            "success": result.success,
            "exit_code": result.exit_code,
            "duration": result.duration,
            "stdout_length": len(result.stdout),
            "stderr_length": len(result.stderr),
            "error_type": result.error_type,
            "has_flag": result.has_flag()
        })

    def log_retry_triggered(self, attempt: Attempt, retry_type: str, trigger_reason: str):
        self._write_log({
            "event": "retry_triggered",
            "timestamp": datetime.now(timezone.utc).isoformat(),
            "attempt_id": attempt.attempt_id,
            "retry_type": retry_type,
            "retry_count": attempt.retry_count,
            "trigger_reason": trigger_reason,
            "error_info": attempt.error_info
        })

    def _write_log(self, log_entry: Dict[str, Any]):
        with open(self.log_path, 'a') as f:
            f.write(json.dumps(log_entry) + '\n')

恢复与监控

调度器需要支持中断恢复。崩溃或重启后,应该能够恢复未完成的attempt继续处理。

class RecoveryManager:
    def __init__(self, state_store: StateStore, logger: SchedulerLogger):
        self.state_store = state_store
        self.logger = logger

    async def recover_pending_attempts(self) -> List[Attempt]:
        pending = await self.state_store.list_pending_attempts()

        recovered = []
        for attempt in pending:
            if self._should_resume(attempt):
                attempt.transition(attempt.state, "recovered")
                recovered.append(attempt)
                self.logger.log_recovery(attempt.attempt_id)

        return recovered

    def _should_resume(self, attempt: Attempt) -> bool:
        if attempt.state in [AttemptState.SUCCEEDED, AttemptState.FAILED]:
            return False

        age = time.time() - attempt.updated_at
        if age > 3600:
            return False

        return True

    async def resume_processing(self, scheduler: RetryScheduler):
        pending = await self.recover_pending_attempts()

        for attempt in pending:
            result = await scheduler._process_attempt(attempt)
            if result and result.get('success'):
                return result

        return None

监控系统实时跟踪调度器状态,收集关键指标:成功率、平均耗时、重试分布、错误类型分布。

class SchedulerMonitor:
    def __init__(self, log_path: str):
        self.log_path = log_path

    def get_metrics(self) -> Dict[str, Any]:
        metrics = {
            "total_challenges": 0,
            "successful_challenges": 0,
            "failed_challenges": 0,
            "total_attempts": 0,
            "total_retries": 0,
            "strategy_distribution": {},
            "error_type_distribution": {},
            "average_duration": 0,
            "success_rate": 0.0
        }

        challenge_states = {}
        attempt_durations = []
        retry_count = 0

        with open(self.log_path) as f:
            for line in f:
                log = json.loads(line)
                event = log.get('event')

                if event == 'challenge_start':
                    challenge_states[log['challenge_id']] = 'started'
                    metrics['total_challenges'] += 1

                elif event == 'challenge_success':
                    challenge_states[log['challenge_id']] = 'success'
                    metrics['successful_challenges'] += 1

                elif event == 'challenge_failure':
                    challenge_states[log['challenge_id']] = 'failed'
                    metrics['failed_challenges'] += 1

                elif event == 'retry_triggered':
                    retry_count += 1

                elif event == 'execution_complete':
                    duration = log.get('duration', 0)
                    attempt_durations.append(duration)

                elif event == 'state_transition':
                    strategy = log.get('strategy', 'unknown')
                    metrics['strategy_distribution'][strategy] = \
                        metrics['strategy_distribution'].get(strategy, 0) + 1

        metrics['total_attempts'] = sum(metrics['strategy_distribution'].values())
        metrics['total_retries'] = retry_count

        if attempt_durations:
            metrics['average_duration'] = sum(attempt_durations) / len(attempt_durations)

        if metrics['total_challenges'] > 0:
            metrics['success_rate'] = metrics['successful_challenges'] / metrics['total_challenges']

        return metrics

    def get_error_breakdown(self) -> Dict[str, int]:
        errors = {}

        with open(self.log_path) as f:
            for line in f:
                log = json.loads(line)
                if log.get('event') == 'retry_triggered':
                    error_type = log.get('retry_type', 'unknown')
                    errors[error_type] = errors.get(error_type, 0) + 1

        return errors

执行器与沙箱环境

候选代码的执行需要受控环境。CTF解题代码可能包含系统调用、网络连接、文件操作,直接执行存在安全风险。沙箱环境隔离执行上下文,限制资源使用,捕获异常行为。

沙箱架构

沙箱使用多层隔离机制。网络层限制可访问的地址和端口,文件系统层使用独立的临时目录,资源层限制CPU时间和内存使用,进程层使用unprivileged用户运行。

import asyncio
import subprocess
import tempfile
import shutil
from pathlib import Path
from typing import List, Dict, Optional

class SandboxExecutor:
    def __init__(self, workspace: str = "/tmp/ctf_sandbox"):
        self.workspace = Path(workspace)
        self.workspace.mkdir(parents=True, exist_ok=True)

    async def execute(self, code: str, config: ExecutionConfig) -> ExecutionResult:
        task_id = f"task_{asyncio.current_task().get_name()}_{time.time()}"
        task_dir = self.workspace / task_id
        task_dir.mkdir(exist_ok=True)

        try:
            code_path = await self._prepare_code(code, config.language, task_dir)
            command = self._build_command(config.language, code_path, task_dir)

            env = self._prepare_environment(config, task_dir)

            process = await self._spawn_process(command, env, config)

            stdout, stderr, exit_code = await self._wait_for_process(process, config.timeout)

            memory_peak = self._get_memory_usage(process.pid)

            return ExecutionResult(
                success=(exit_code == 0),
                exit_code=exit_code,
                stdout=stdout.decode('utf-8', errors='replace'),
                stderr=stderr.decode('utf-8', errors='replace'),
                duration=config.timeout,
                memory_peak=memory_peak,
                error_type=self._classify_error(exit_code, stderr, config.timeout)
            )

        except asyncio.TimeoutError:
            return ExecutionResult(
                success=False,
                exit_code=-1,
                stdout="",
                stderr=f"Execution exceeded timeout of {config.timeout} seconds",
                duration=config.timeout,
                error_type="timeout"
            )

        except Exception as e:
            return ExecutionResult(
                success=False,
                exit_code=-1,
                stdout="",
                stderr=str(e),
                duration=0,
                error_type="runtime"
            )

        finally:
            self._cleanup(task_dir)

    async def _prepare_code(self, code: str, language: str, task_dir: Path) -> Path:
        if language == "python":
            code_path = task_dir / "exploit.py"
            code_path.write_text(code)
            return code_path

        elif language == "bash":
            code_path = task_dir / "exploit.sh"
            code_path.write_text(code)
            return code_path

        elif language == "c":
            source_path = task_dir / "exploit.c"
            source_path.write_text(code)

            compile_result = await asyncio.create_subprocess_exec(
                'gcc', '-o', str(task_dir / 'exploit'), str(source_path),
                cwd=str(task_dir),
                stdout=asyncio.subprocess.PIPE,
                stderr=asyncio.subprocess.PIPE
            )
            await compile_result.wait()

            if compile_result.returncode != 0:
                stderr = await compile_result.stderr.read()
                raise CompilationError(stderr.decode())

            return task_dir / "exploit"

        else:
            code_path = task_dir / "exploit.py"
            code_path.write_text(code)
            return code_path

    def _build_command(self, language: str, code_path: Path, task_dir: Path) -> List[str]:
        if language == "python":
            return ['python3', '-u', str(code_path)]

        elif language == "bash":
            return ['bash', str(code_path)]

        elif language == "c":
            return [str(code_path)]

        else:
            return ['python3', str(code_path)]

    def _prepare_environment(self, config: ExecutionConfig, task_dir: Path) -> Dict[str, str]:
        env = {
            'PATH': '/usr/bin:/bin',
            'HOME': str(task_dir),
            'TMPDIR': str(task_dir),
            'PYTHONPATH': str(task_dir),
            **config.environment
        }

        if config.files:
            for name, content in config.files.items():
                file_path = task_dir / name
                file_path.parent.mkdir(parents=True, exist_ok=True)
                file_path.write_text(content)

        return env

    async def _spawn_process(self, command: List[str], env: Dict, config: ExecutionConfig):
        return await asyncio.create_subprocess_exec(
            *command,
            cwd=str(config.working_directory),
            env=env,
            stdout=asyncio.subprocess.PIPE,
            stderr=asyncio.subprocess.PIPE,
            preexec_fn=self._limit_privileges
        )

    async def _wait_for_process(self, process: subprocess.Process, timeout: int):
        try:
            stdout, stderr = await asyncio.wait_for(
                process.communicate(),
                timeout=timeout
            )
            return stdout, stderr, process.returncode

        except asyncio.TimeoutError:
            process.kill()
            await process.wait()
            raise

    def _get_memory_usage(self, pid: int) -> Optional[int]:
        try:
            import psutil
            proc = psutil.Process(pid)
            return proc.memory_info().rss
        except:
            return None

    def _classify_error(self, exit_code: int, stderr: bytes, timeout: int) -> Optional[str]:
        stderr_str = stderr.decode('utf-8', errors='replace')

        if exit_code == -9:
            return 'memory'

        if exit_code == 124:
            return 'timeout'

        if 'SyntaxError' in stderr_str or 'IndentationError' in stderr_str:
            return 'syntax'

        if 'NameError' in stderr_str or 'AttributeError' in stderr_str:
            return 'runtime'

        if 'Connection refused' in stderr_str:
            return 'network'

        if 'Permission denied' in stderr_str:
            return 'permission'

        return None

    def _limit_privileges(self):
        import resource
        resource.setrlimit(resource.RLIMIT_AS, (512 * 1024 * 1024, 512 * 1024 * 1024))
        resource.setrlimit(resource.RLIMIT_CPU, (30, 30))

    def _cleanup(self, task_dir: Path):
        try:
            shutil.rmtree(task_dir)
        except:
            pass

网络沙箱

CTF pwn题需要网络连接,但需要限制可访问的目标。网络沙箱使用iptables规则或用户空间proxy实现白名单过滤。

class NetworkSandbox:
    def __init__(self, allowed_hosts: List[str]):
        self.allowed_hosts = allowed_hosts
        self.proxy_port = 8080

    async def setup_proxy(self):
        self.proxy_process = await asyncio.create_subprocess_exec(
            'mitmproxy',
            '--listen-port', str(self.proxy_port),
            '--set', 'block_global=false',
            *self._build_allow_rules()
        )

    def _build_allow_rules(self) -> List[str]:
        rules = []
        for host in self.allowed_hosts:
            rules.extend(['--allow', host])
        return rules

    def get_proxy_config(self) -> Dict[str, str]:
        return {
            'http_proxy': f'http://127.0.0.1:{self.proxy_port}',
            'https_proxy': f'http://127.0.0.1:{self.proxy_port}'
        }

    async def teardown(self):
        if self.proxy_process:
            self.proxy_process.terminate()
            await self.proxy_process.wait()

对于本地pwn题目,可以创建模拟服务监听特定端口,接受exploit的连接并提供预期响应。

class LocalServiceEmulator:
    def __init__(self, service_type: str, response_data: bytes):
        self.service_type = service_type
        self.response_data = response_data
        self.port = None
        self.process = None

    async def start(self) -> int:
        self.port = self._get_free_port()

        if self.service_type == 'echo':
            self.process = await asyncio.create_subprocess_exec(
                'nc', '-l', str(self.port), '-e', 'cat'
            )

        elif self.service_type == 'custom':
            server_code = f"""
import socket
s = socket.socket()
s.bind(('0.0.0.0', {self.port}))
s.listen(1)
while True:
    conn, addr = s.accept()
    data = conn.recv(4096)
    conn.send({self.response_data})
    conn.close()
"""
            script_path = f'/tmp/server_{self.port}.py'
            Path(script_path).write_text(server_code)

            self.process = await asyncio.create_subprocess_exec(
                'python3', script_path
            )

        await asyncio.sleep(0.5)
        return self.port

    async def stop(self):
        if self.process:
            self.process.terminate()
            await self.process.wait()

    def _get_free_port(self) -> int:
        import socket
        s = socket.socket()
        s.bind(('', 0))
        port = s.getsockname()[1]
        s.close()
        return port

资源限制

资源限制防止失控代码消耗过多系统资源。Linux使用cgroups实现精细的资源控制,但需要root权限配置。用户空间的替代方案是在执行前设置resource limit。

import resource
import signal

class ResourceLimiter:
    @staticmethod
    def set_limits(timeout: int, memory_mb: int):
        resource.setrlimit(resource.RLIMIT_CPU, (timeout, timeout + 5))
        resource.setrlimit(resource.RLIMIT_AS, (memory_mb * 1024 * 1024, resource.RLIM_INFINITY))

        def timeout_handler(signum, frame):
            raise TimeoutError("Execution timeout")

        signal.signal(signal.SIGXCPU, timeout_handler)

    @staticmethod
    def get_usage() -> Dict[str, int]:
        usage = resource.getrusage(resource.RUSAGE_CHILDREN)
        return {
            'user_time': usage.ru_utime,
            'system_time': usage.ru_stime,
            'max_rss': usage.ru_maxrss,
            'minor_faults': usage.ru_minflt,
            'major_faults': usage.ru_majflt,
            'voluntary_context_switches': usage.ru_nvcsw,
            'involuntary_context_switches': ru_nivcsw
        }

内存限制通过RLIMIT_AS设置地址空间大小,超限时malloc返回NULL触发malloc_error或被signal捕获。CPU时间限制通过RLIMIT_CPU设置,超时发送SIGXCPU信号。

分布式调度器扩展

单机调度器受限于计算资源,并行度有限。分布式调度器将attempt分发到多个worker节点,实现真正的横向扩展。

架构设计

分布式调度器采用master-worker架构。Master节点管理challenge队列和attempt分配,worker节点执行实际的LLM调用和代码执行。节点间通过消息队列通信,解耦调度和执行。

Master维护待处理challenge队列和attempt状态表。Worker从队列获取任务,执行后返回结果。Master收集结果,判断是否成功,失败则调度重试。

import asyncio
import aiohttp
from typing import List, Dict, Any
from collections import defaultdict

class DistributedScheduler:
    def __init__(
        self,
        master_url: str,
        worker_urls: List[str],
        local_scheduler: RetryScheduler
    ):
        self.master_url = master_url
        self.worker_urls = worker_urls
        self.local_scheduler = local_scheduler
        self.session = aiohttp.ClientSession()

    async def solve_challenge(self, challenge: LLMInput) -> Optional[Dict]:
        workers = await self._get_available_workers()

        if not workers:
            return await self.local_scheduler.solve_challenge(challenge)

        task = self._create_task(challenge)

        workers_assigned = workers[:self.local_scheduler.max_parallel]

        attempts = await asyncio.gather(
            *[self._dispatch_to_worker(worker, task) for worker in workers_assigned],
            return_exceptions=True
        )

        valid_attempts = [a for a in attempts if isinstance(a, Attempt)]

        for attempt in valid_attempts:
            result = await self._process_attempt(attempt)
            if result and result.get('success'):
                return result

        return None

    async def _get_available_workers(self) -> List[str]:
        async with self.session.get(f"{self.master_url}/workers") as resp:
            data = await resp.json()
            return data.get('available', [])

    def _create_task(self, challenge: LLMInput) -> Dict:
        return {
            "task_id": f"task_{uuid.uuid4().hex}",
            "challenge": challenge.dict(),
            "created_at": datetime.utcnow().isoformat()
        }

    async def _dispatch_to_worker(self, worker_url: str, task: Dict) -> Attempt:
        async with self.session.post(f"{worker_url}/generate", json=task) as resp:
            data = await resp.json()

            return Attempt(
                attempt_id=data['attempt_id'],
                challenge_id=task['challenge']['challenge_id'],
                state=AttemptState[data['state']],
                retry_count=0,
                max_retries=3,
                current_strategy=data.get('strategy', 'unknown'),
                llm_output=data.get('output'),
                error_info=data.get('error'),
                base_input=LLMInput(**task['challenge']),
                created_at=time.time(),
                updated_at=time.time()
            )

Worker节点实现

Worker节点运行轻量级的调度器实例,处理分配的attempt。Worker不持久化状态,失败时由master重新分配。

from aiohttp import web

class WorkerNode:
    def __init__(self, worker_id: str, port: int):
        self.worker_id = worker_id
        self.port = port
        self.app = web.Application()
        self.app.router.add_post('/generate', self.handle_generate)
        self.app.router.add_post('/execute', self.handle_execute)
        self.app.router.add_get('/health', self.handle_health)

        self.scheduler = None

    def set_scheduler(self, scheduler: RetryScheduler):
        self.scheduler = scheduler

    async def handle_generate(self, request: web.Request) -> web.Response:
        data = await request.json()

        challenge = LLMInput(**data['challenge'])
        task_id = data['task_id']

        attempts = await self.scheduler.parallel_scheduler.generate_candidates(challenge)

        if attempts:
            attempt = attempts[0]
            return web.json_response({
                "attempt_id": attempt.attempt_id,
                "state": attempt.state.value,
                "output": attempt.llm_output,
                "strategy": attempt.current_strategy
            })
        else:
            return web.json_response({"error": "generation_failed"}, status=500)

    async def handle_execute(self, request: web.Request) -> web.Response:
        data = await request.json()

        attempt_id = data['attempt_id']
        code = data['code']
        config = ExecutionConfig(**data['config'])

        result = await self.scheduler.executor.execute(code, config)

        return web.json_response({
            "success": result.success,
            "stdout": result.stdout,
            "stderr": result.stderr,
            "exit_code": result.exit_code,
            "has_flag": result.has_flag()
        })

    async def handle_health(self, request: web.Request) -> web.Response:
        return web.json_response({
            "worker_id": self.worker_id,
            "status": "healthy"
        })

    async def start(self):
        runner = web.AppRunner(self.app)
        await runner.setup()
        site = web.TCPSite(runner, '0.0.0.0', self.port)
        await site.start()

任务队列与结果收集

Master使用任务队列管理待分配的attempt。队列优先级可以根据题目难度、已投入尝试次数、截止时间等因素计算。

import heapq
from dataclasses import dataclass, field
from typing import Any

@dataclass(order=True)
class QueuedTask:
    priority: int
    task_id: str
    challenge: LLMInput
    attempt: Attempt
    created_at: float = field(compare=False)

class TaskQueue:
    def __init__(self):
        self.queue = []
        self.task_map = {}
        self.lock = asyncio.Lock()

    async def push(self, task: QueuedTask):
        async with self.lock:
            heapq.heappush(self.queue, task)
            self.task_map[task.task_id] = task

    async def pop(self) -> Optional[QueuedTask]:
        async with self.lock:
            if not self.queue:
                return None
            task = heapq.heappop(self.queue)
            del self.task_map[task.task_id]
            return task

    async def remove(self, task_id: str):
        async with self.lock:
            if task_id in self.task_map:
                del self.task_map[task_id]

    def calculate_priority(self, challenge: LLMInput, attempt: Attempt) -> int:
        base_priority = challenge.difficulty * 10
        retry_penalty = attempt.retry_count * 5
        age_penalty = int((time.time() - attempt.created_at) / 60)

        return -(base_priority - retry_penalty - age_penalty)

结果收集器处理worker返回的执行结果。成功的解题立即返回给客户端,失败的attempt重新排队或标记为最终失败。

class ResultCollector:
    def __init__(self, task_queue: TaskQueue):
        self.task_queue = task_queue
        self.pending_results = {}

    async def collect_result(self, task_id: str, result: Dict) -> Optional[Dict]:
        if result.get('success'):
            return result
        else:
            task = await self.task_queue.pop(task_id)
            if task and task.attempt.retry_count < task.attempt.max_retries:
                task.attempt.increment_retry()
                await self.task_queue.push(task)
            return None

    async def wait_for_success(self, task_ids: List[str], timeout: int) -> Optional[Dict]:
        deadline = time.time() + timeout

        while time.time() < deadline:
            for task_id in task_ids:
                if task_id in self.pending_results:
                    result = self.pending_results[task_id]
                    if result.get('success'):
                        del self.pending_results[task_id]
                        return result

            await asyncio.sleep(1)

        return None

实验设计与效果评估

调度器的设计需要通过实验验证有效性。A/B测试比较不同配置的性能,参数网格搜索寻找最优设置,长期跟踪评估稳定性。

A/B测试框架

A/B测试在同一题目集上比较不同调度策略。对照组使用基线配置,实验组使用优化配置,比较成功率、平均耗时、资源消耗。

from dataclasses import dataclass
from typing import List, Callable
import numpy as np
from scipy import stats

@dataclass
class ExperimentConfig:
    name: str
    max_parallel: int
    max_retries: int
    strategies: List[str]
    dedup_enabled: bool
    adaptive_enabled: bool

@dataclass
class ExperimentResult:
    config: ExperimentConfig
    success_count: int
    total_challenges: int
    average_duration: float
    total_tokens: int
    attempt_details: List[Dict]

    @property
    def success_rate(self) -> float:
        return self.success_count / self.total_challenges if self.total_challenges > 0 else 0

class ABTestFramework:
    def __init__(self, challenges: List[LLMInput]):
        self.challenges = challenges
        self.results = {}

    async def run_experiment(self, config: ExperimentConfig, scheduler_builder: Callable) -> ExperimentResult:
        scheduler = scheduler_builder(config)

        attempt_details = []
        success_count = 0
        durations = []
        total_tokens = 0

        for challenge in self.challenges:
            start_time = time.time()

            result = await scheduler.solve_challenge(challenge)
            duration = time.time() - start_time

            durations.append(duration)

            if result:
                success_count += 1

            attempt_details.append({
                "challenge_id": challenge.challenge_id,
                "success": result is not None,
                "duration": duration,
                "strategy": result.get('strategy') if result else None
            })

        return ExperimentResult(
            config=config,
            success_count=success_count,
            total_challenges=len(self.challenges),
            average_duration=np.mean(durations),
            total_tokens=total_tokens,
            attempt_details=attempt_details
        )

    async def run_ab_test(
        self,
        config_a: ExperimentConfig,
        config_b: ExperimentConfig,
        scheduler_builder: Callable
    ) -> Dict[str, Any]:
        result_a = await self.run_experiment(config_a, scheduler_builder)
        result_b = await self.run_experiment(config_b, scheduler_builder)

        statistic, p_value = stats.fisher_exact([
            [result_a.success_count, result_a.total_challenges - result_a.success_count],
            [result_b.success_count, result_b.total_challenges - result_b.success_count]
        ])

        duration_statistic, duration_p = stats.mannwhitneyu(
            [d['duration'] for d in result_a.attempt_details],
            [d['duration'] for d in result_b.attempt_details]
        )

        return {
            "config_a": config_a.name,
            "config_b": config_b.name,
            "success_rate_a": result_a.success_rate,
            "success_rate_b": result_b.success_rate,
            "success_rate_improvement": (result_b.success_rate - result_a.success_rate) / result_a.success_rate if result_a.success_rate > 0 else 0,
            "p_value": p_value,
            "significant": p_value < 0.05,
            "average_duration_a": result_a.average_duration,
            "average_duration_b": result_b.average_duration,
            "duration_p_value": duration_p
        }

参数网格搜索

网格搜索系统地探索参数空间,寻找最优配置。搜索维度包括并行数量、重试次数、策略组合、去重阈值。

from itertools import product

class GridSearch:
    def __init__(self, challenges: List[LLMInput]):
        self.challenges = challenges
        self.ab_test = ABTestFramework(challenges)

    def define_search_space(self) -> Dict[str, List[Any]]:
        return {
            "max_parallel": [3, 5, 7],
            "max_retries": [2, 3, 4],
            "strategies": [
                ["conservative", "balanced", "creative"],
                ["conservative", "balanced", "analytical"],
                ["balanced", "creative", "aggressive"]
            ],
            "dedup_enabled": [True, False],
            "adaptive_enabled": [True, False]
        }

    async def search(self, scheduler_builder: Callable) -> ExperimentResult:
        search_space = self.define_search_space()

        best_result = None
        best_config = None

        for params in product(*search_space.values()):
            config = ExperimentConfig(
                name=f"config_{len(self.ab_test.results)}",
                max_parallel=params[0],
                max_retries=params[1],
                strategies=params[2],
                dedup_enabled=params[3],
                adaptive_enabled=params[4]
            )

            result = await self.ab_test.run_experiment(config, scheduler_builder)

            if best_result is None or result.success_rate > best_result.success_rate:
                best_result = result
                best_config = config

            self.ab_test.results[config.name] = result

        return best_result

长期跟踪与稳定性评估

调度器需要在长期运行中保持稳定性。跟踪成功率的时间序列,检测性能退化或异常波动。

import pandas as pd
from datetime import datetime, timedelta

class LongTermTracker:
    def __init__(self, metrics_path: str):
        self.metrics_path = metrics_path
        self.metrics_df = self._load_metrics()

    def _load_metrics(self) -> pd.DataFrame:
        try:
            return pd.read_csv(self.metrics_path, parse_dates=['timestamp'])
        except:
            return pd.DataFrame(columns=['timestamp', 'success', 'duration', 'tokens', 'strategy'])

    async def record_attempt(self, success: bool, duration: float, tokens: int, strategy: str):
        record = {
            "timestamp": datetime.utcnow(),
            "success": success,
            "duration": duration,
            "tokens": tokens,
            "strategy": strategy
        }

        self.metrics_df = pd.concat([self.metrics_df, pd.DataFrame([record])], ignore_index=True)
        self.metrics_df.to_csv(self.metrics_path, index=False)

    def get_success_rate_trend(self, window_hours: int = 24) -> pd.DataFrame:
        cutoff = datetime.utcnow() - timedelta(hours=window_hours)
        recent = self.metrics_df[self.metrics_df['timestamp'] >= cutoff]

        trend = recent.set_index('timestamp').resample('1H').agg({
            'success': ['mean', 'count']
        }).reset_index()

        trend.columns = ['timestamp', 'success_rate', 'attempt_count']
        return trend

    def detect_anomaly(self, threshold: float = 2.0) -> bool:
        recent_24h = self.get_success_rate_trend(24)
        recent_7d = self.get_success_rate_trend(24 * 7)

        avg_24h = recent_24h['success_rate'].mean()
        avg_7d = recent_7d['success_rate'].mean()

        std_7d = recent_7d['success_rate'].std()

        if abs(avg_24h - avg_7d) > threshold * std_7d:
            return True

        return False

异常检测触发告警,提示可能的问题:LLM服务性能下降、执行环境资源不足、网络连接不稳定。告警后可以自动切换配置或通知人工介入。

工程实践与经验总结

将调度器部署到生产环境需要考虑很多工程细节。基于实际部署和运行经验,总结关键实践和常见陷阱。

日志的可观测性

日志不仅是记录,更是调试和优化的关键工具。结构化日志便于查询和聚合,trace ID关联相关操作,metrics量化系统状态。

import structlog

def configure_logging():
    structlog.configure(
        processors=[
            structlog.stdlib.add_log_level,
            structlog.stdlib.add_logger_name,
            structlog.processors.TimeStamper(fmt="iso"),
            structlog.processors.StackInfoRenderer(),
            structlog.processors.format_exc_info,
            structlog.processors.JSONRenderer()
        ],
        context_class=dict,
        logger_factory=structlog.stdlib.LoggerFactory(),
        cache_logger_on_first_use=True,
    )

logger = structlog.get_logger()

async def solve_with_logging(challenge: LLMInput, scheduler: RetryScheduler):
    trace_id = uuid.uuid4().hex

    log = logger.bind(
        trace_id=trace_id,
        challenge_id=challenge.challenge_id,
        category=challenge.category
    )

    log.info("challenge_started", difficulty=challenge.difficulty)

    try:
        result = await scheduler.solve_challenge(challenge)

        if result:
            log.info(
                "challenge_succeeded",
                strategy=result.get('strategy'),
                duration=result.get('duration')
            )
        else:
            log.warning("challenge_failed_after_all_retries")

        return result

    except Exception as e:
        log.error("challenge_error", error=str(e), exc_info=True)
        raise

配置管理

调度器有很多可配置参数:并行度、重试次数、超时时间、策略选择。硬编码这些参数使得调优困难。使用配置文件集中管理,支持热更新。

import yaml
from pydantic import BaseModel

class SchedulerConfig(BaseModel):
    max_parallel: int = 5
    max_retries: int = 3
    generation_timeout: int = 60
    execution_timeout: int = 120
    memory_limit_mb: int = 512
    enable_dedup: bool = True
    enable_adaptive: bool = True
    strategies: List[str] = ["conservative", "balanced", "creative"]
    log_level: str = "INFO"

    @classmethod
    def from_file(cls, path: str):
        with open(path) as f:
            data = yaml.safe_load(f)
        return cls(**data)

class ConfigWatcher:
    def __init__(self, config_path: str):
        self.config_path = config_path
        self.config = SchedulerConfig.from_file(config_path)
        self.last_modified = os.path.getmtime(config_path)

    async def watch_changes(self):
        while True:
            await asyncio.sleep(5)

            current_modified = os.path.getmtime(self.config_path)
            if current_modified != self.last_modified:
                self.config = SchedulerConfig.from_file(self.config_path)
                self.last_modified = current_modified
                logger.info("config_reloaded", config=self.config.dict())

错误恢复的鲁棒性

生产环境中各种异常都可能发生:网络中断、服务崩溃、磁盘满、内存耗尽。调度器需要优雅地处理这些异常,不丢失状态、不阻塞流程。

from tenacity import retry, stop_after_attempt, wait_exponential

class ResilientLLMClient:
    def __init__(self, api_url: str):
        self.api_url = api_url
        self.circuit_breaker = CircuitBreaker()

    @retry(
        stop=stop_after_attempt(3),
        wait=wait_exponential(multiplier=1, min=2, max=10)
    )
    async def generate(self, input_data: LLMInput) -> Dict:
        if self.circuit_breaker.is_open():
            raise CircuitBreakerOpenError("LLM service unavailable")

        try:
            response = await self._call_api(input_data)
            self.circuit_breaker.record_success()
            return response

        except Exception as e:
            self.circuit_breaker.record_failure()
            raise

class CircuitBreaker:
    def __init__(self, failure_threshold: int = 5, timeout: int = 60):
        self.failure_threshold = failure_threshold
        self.timeout = timeout
        self.failures = 0
        self.last_failure_time = None
        self.state = 'closed'

    def is_open(self) -> bool:
        if self.state == 'open':
            if time.time() - self.last_failure_time > self.timeout:
                self.state = 'half_open'
            else:
                return True
        return False

    def record_success(self):
        self.failures = 0
        self.state = 'closed'

    def record_failure(self):
        self.failures += 1
        self.last_failure_time = time.time()
        if self.failures >= self.failure_threshold:
            self.state = 'open'

熔断机制防止级联故障,当LLM服务持续失败时暂停请求,避免无限重试消耗资源。半开状态允许试探性请求,检测服务是否恢复。

性能优化的权衡

调度器需要在多个维度优化:延迟、吞吐量、成本、成功率。这些目标有时冲突,需要根据应用场景权衡。

对于实时性要求高的场景,优先优化延迟。减少并行度以降低等待时间,使用更快的重试策略如跳过hallucination检查,接受略低的成功率换取速度。

对于成本敏感的场景,优先优化token消耗。限制并行数量,减少不必要的重试,使用更便宜的模型,批量处理题目以摊销固定成本。

对于成功率要求高的场景,投入更多资源。增加并行度探索更多候选,使用更强大的模型,延长重试次数,人工审核关键步骤。

没有全局最优配置,最优取决于具体场景和约束。实验和数据分析指导权衡决策,监控指标验证调整效果。