九哥聊Kestrel网络编程第二章：开发一个Fiddler

InCerry

发布于 2023-03-08 16:06:55

5080

发布于 2023-03-08 16:06:55

文章被收录于专栏：InCerryInCerry

1 文章目的

本文讲解基于 kestrel 开发类似 Fiddler 应用的过程，让读者了解 kestrel 网络编程里面的 kestrel 中间件和 http 应用中间件。由于最终目的不是输出完整功能的产品，所以这里只实现 Fiddler 最核心的 http 请求和响应内容查看的功能。本文章是KestrelApp 项目[2]里面的一个 demo 的讲解，希望对您有用。

2 开发顺序

代理协议 kestrel 中间件
tls 协议侦测 kestrel 中间件
隧道和 http 协议侦测 kestrel 中间件
请求响应分析 http 中间件
反向代理 http 中间件
编排中间件创建服务器和应用

3 传输层与 kestrel 中间件

所谓传输层，其目的是为了让应用协议数据安全、可靠、快速等传输而存在的一种协议，其特征是把应用协议的报文做为自己的负载，常见的 tcp、udp、quic、tls 等都可以理解为传输层协议。比如 http 协议，常见有如下的传输方式：

http over tcp
http over tls over tcp
http over quic over udp

3.1 Fiddler 的传输层

Fiddler 要处理以下三种 http 传输情况：

http over tcp：直接 http 请求首页
http over proxy over tcp：代理 http 流量
http over tls over proxy over tcp：代理 https 流量

3.2 Kestrel 的中间件

kestrel 目前的传输层基于 tcp 或 quic 两种，同时内置了 tls 中间件，需要调用ListenOptions.UseHttps()来使用 tls 中间件。kestrel 的中间件的表现形式为：Func<ConnectionDelegate, ConnectionDelegate>，为了使用读者能够简单理解中间件，我在KestrelFramework里定义了 kestrel 中间件的变种接口，大家基于此接口来实现更多的中间件就方便很多：

/// <summary>
/// Kestrel的中间件接口
/// </summary>
public interface IKestrelMiddleware
{
    /// <summary>
    /// 执行
    /// </summary>
    /// <param name="next"></param>
    /// <param name="context"></param>
    /// <returns></returns>
    Task InvokeAsync(ConnectionDelegate next, ConnectionContext context);
}

4 代理协议 kestrel 中间件

Filddler 最基础的功能是它是一个 http 代理服务器，我们需要为 kestrel 编写代理中间件，用于处理代理传输层。http 代理协议分两种：普通的 http 代理和 Connect 隧道代理。两种的报文者是遵循 http1.0 或 1.1 的文本格式，我们可以使用 kestrel 自带的HttpParser<>来解析这些复杂的 http 文本协议。

4.1 代理特征

在中间件编程模式中，Feature是一个很重要的中间件沟通桥梁，它往往是某个中间件工作之后，留下的财产，让之后的中间件来获取并受益。我们的代理中间件，也设计了 IProxyFeature，告诉之后的中间件一些代理特征。

/// <summary>
/// 代理Feature
/// </summary>
public interface IProxyFeature
{
    /// <summary>
    /// 代理主机
    /// </summary>
    HostString ProxyHost { get; }

    /// <summary>
    /// 代理协议
    /// </summary>
    ProxyProtocol ProxyProtocol { get; }
}

/// <summary>
/// 代理协议
/// </summary>
public enum ProxyProtocol
{
    /// <summary>
    /// 无代理
    /// </summary>
    None,

    /// <summary>
    /// http代理
    /// </summary>
    HttpProxy,

    /// <summary>
    /// 隧道代理
    /// </summary>
    TunnelProxy
}

4.2 代理中间件的实现

/// <summary>
/// 代理中间件
/// </summary>
sealed class KestrelProxyMiddleware : IKestrelMiddleware
{
    private static readonly HttpParser<HttpRequestHandler> httpParser = new();
    private static readonly byte[] http200 = Encoding.ASCII.GetBytes("HTTP/1.1 200 Connection Established\r\n\r\n");
    private static readonly byte[] http400 = Encoding.ASCII.GetBytes("HTTP/1.1 400 Bad Request\r\n\r\n");

    /// <summary>
    /// 解析代理
    /// </summary>
    /// <param name="next"></param>
    /// <param name="context"></param>
    /// <returns></returns>
    public async Task InvokeAsync(ConnectionDelegate next, ConnectionContext context)
    {
        var input = context.Transport.Input;
        var output = context.Transport.Output;
        var request = new HttpRequestHandler();

        while (context.ConnectionClosed.IsCancellationRequested == false)
        {
            var result = await input.ReadAsync();
            if (result.IsCanceled)
            {
                break;
            }

            try
            {
                if (ParseRequest(result, request, out var consumed))
                {
                    if (request.ProxyProtocol == ProxyProtocol.TunnelProxy)
                    {
                        input.AdvanceTo(consumed);
                        await output.WriteAsync(http200);
                    }
                    else
                    {
                        input.AdvanceTo(result.Buffer.Start);
                    }

                    context.Features.Set<IProxyFeature>(request);
                    await next(context);

                    break;
                }
                else
                {
                    input.AdvanceTo(result.Buffer.Start, result.Buffer.End);
                }

                if (result.IsCompleted)
                {
                    break;
                }
            }
            catch (Exception)
            {
                await output.WriteAsync(http400);
                break;
            }
        }
    }


    /// <summary>
    /// 解析http请求
    /// </summary>
    /// <param name="result"></param>
    /// <param name="request"></param>
    /// <param name="consumed"></param>
    /// <returns></returns>
    private static bool ParseRequest(ReadResult result, HttpRequestHandler request, out SequencePosition consumed)
    {
        var reader = new SequenceReader<byte>(result.Buffer);
        if (httpParser.ParseRequestLine(request, ref reader) &&
            httpParser.ParseHeaders(request, ref reader))
        {
            consumed = reader.Position;
            return true;
        }
        else
        {
            consumed = default;
            return false;
        }
    }


    /// <summary>
    /// 代理请求处理器
    /// </summary>
    private class HttpRequestHandler : IHttpRequestLineHandler, IHttpHeadersHandler, IProxyFeature
    {
        private HttpMethod method;

        public HostString ProxyHost { get; private set; }

        public ProxyProtocol ProxyProtocol
        {
            get
            {
                if (ProxyHost.HasValue == false)
                {
                    return ProxyProtocol.None;
                }
                if (method == HttpMethod.Connect)
                {
                    return ProxyProtocol.TunnelProxy;
                }
                return ProxyProtocol.HttpProxy;
            }
        }

        void IHttpRequestLineHandler.OnStartLine(HttpVersionAndMethod versionAndMethod, TargetOffsetPathLength targetPath, Span<byte> startLine)
        {
            method = versionAndMethod.Method;
            var host = Encoding.ASCII.GetString(startLine.Slice(targetPath.Offset, targetPath.Length));
            if (versionAndMethod.Method == HttpMethod.Connect)
            {
                ProxyHost = HostString.FromUriComponent(host);
            }
            else if (Uri.TryCreate(host, UriKind.Absolute, out var uri))
            {
                ProxyHost = HostString.FromUriComponent(uri);
            }
        }

        void IHttpHeadersHandler.OnHeader(ReadOnlySpan<byte> name, ReadOnlySpan<byte> value)
        {
        }
        void IHttpHeadersHandler.OnHeadersComplete(bool endStream)
        {
        }
        void IHttpHeadersHandler.OnStaticIndexedHeader(int index)
        {
        }
        void IHttpHeadersHandler.OnStaticIndexedHeader(int index, ReadOnlySpan<byte> value)
        {
        }
    }
}

5 tls 协议侦测 kestrel 中间件

Fiddler 只监听了一个端口，要同时支持非加密和加密两种流量，如果不调用调用ListenOptions.UseHttps()，我们的程序就不支持 https 的分析；如果直接调用ListenOptions.UseHttps()，会让我们的程序不支持非加密的 http 的分析，这就要求我们有条件的根据客户端发来的流量分析是否需要开启。

我已经在KestrelFramework内置了TlsDetection中间件，这个中间件可以根据客户端的实际流量类型来选择是否使用 tls。在 Fiddler 中，我们还需要根据客户端的tls握手中的sni使用 ca 证书来动态生成服务器证书用于 tls 加密传输。

/// <summary>
/// 证书服务
/// </summary>
sealed class CertService
{
    private const string CACERT_PATH = "cacert";
    private readonly IMemoryCache serverCertCache;
    private readonly IEnumerable<ICaCertInstaller> certInstallers;
    private readonly ILogger<CertService> logger;
    private X509Certificate2? caCert;


    /// <summary>
    /// 获取证书文件路径
    /// </summary>
    public string CaCerFilePath { get; } = OperatingSystem.IsLinux() ? $"{CACERT_PATH}/fiddler.crt" : $"{CACERT_PATH}/fiddler.cer";

    /// <summary>
    /// 获取私钥文件路径
    /// </summary>
    public string CaKeyFilePath { get; } = $"{CACERT_PATH}/fiddler.key";

    /// <summary>
    /// 证书服务
    /// </summary>
    /// <param name="serverCertCache"></param>
    /// <param name="certInstallers"></param>
    /// <param name="logger"></param>
    public CertService(
        IMemoryCache serverCertCache,
        IEnumerable<ICaCertInstaller> certInstallers,
        ILogger<CertService> logger)
    {
        this.serverCertCache = serverCertCache;
        this.certInstallers = certInstallers;
        this.logger = logger;
        Directory.CreateDirectory(CACERT_PATH);
    }

    /// <summary>
    /// 生成CA证书
    /// </summary>
    public bool CreateCaCertIfNotExists()
    {
        if (File.Exists(this.CaCerFilePath) && File.Exists(this.CaKeyFilePath))
        {
            return false;
        }

        File.Delete(this.CaCerFilePath);
        File.Delete(this.CaKeyFilePath);

        var notBefore = DateTimeOffset.Now.AddDays(-1);
        var notAfter = DateTimeOffset.Now.AddYears(10);

        var subjectName = new X500DistinguishedName($"CN={nameof(Fiddler)}");
        this.caCert = CertGenerator.CreateCACertificate(subjectName, notBefore, notAfter);

        var privateKeyPem = this.caCert.GetRSAPrivateKey()?.ExportRSAPrivateKeyPem();
        File.WriteAllText(this.CaKeyFilePath, new string(privateKeyPem), Encoding.ASCII);

        var certPem = this.caCert.ExportCertificatePem();
        File.WriteAllText(this.CaCerFilePath, new string(certPem), Encoding.ASCII);

        return true;
    }

    /// <summary>
    /// 安装和信任CA证书
    /// </summary>
    public void InstallAndTrustCaCert()
    {
        var installer = this.certInstallers.FirstOrDefault(item => item.IsSupported());
        if (installer != null)
        {
            installer.Install(this.CaCerFilePath);
        }
        else
        {
            this.logger.LogWarning($"请根据你的系统平台手动安装和信任CA证书{this.CaCerFilePath}");
        }
    }


    /// <summary>
    /// 获取颁发给指定域名的证书
    /// </summary>
    /// <param name="domain"></param>
    /// <returns></returns>
    public X509Certificate2 GetOrCreateServerCert(string? domain)
    {
        if (this.caCert == null)
        {
            using var rsa = RSA.Create();
            rsa.ImportFromPem(File.ReadAllText(this.CaKeyFilePath));
            this.caCert = new X509Certificate2(this.CaCerFilePath).CopyWithPrivateKey(rsa);
        }

        var key = $"{nameof(CertService)}:{domain}";
        var endCert = this.serverCertCache.GetOrCreate(key, GetOrCreateCert);
        return endCert!;

        // 生成域名的1年证书
        X509Certificate2 GetOrCreateCert(ICacheEntry entry)
        {
            var notBefore = DateTimeOffset.Now.AddDays(-1);
            var notAfter = DateTimeOffset.Now.AddYears(1);
            entry.SetAbsoluteExpiration(notAfter);

            var extraDomains = GetExtraDomains();

            var subjectName = new X500DistinguishedName($"CN={domain}");
            var endCert = CertGenerator.CreateEndCertificate(this.caCert, subjectName, extraDomains, notBefore, notAfter);

            // 重新初始化证书，以兼容win平台不能使用内存证书
            return new X509Certificate2(endCert.Export(X509ContentType.Pfx));
        }
    }

    /// <summary>
    /// 获取域名
    /// </summary>
    /// <param name="domain"></param>
    /// <returns></returns>
    private static IEnumerable<string> GetExtraDomains()
    {
        yield return Environment.MachineName;
        yield return IPAddress.Loopback.ToString();
        yield return IPAddress.IPv6Loopback.ToString();
    }
}

6 隧道和 http 协议侦测 kestrel 中间件

经过KestrelProxyMiddleware后的流量，在 tls 解密(如果可能)之后，一般情况下都是 http 流量了，但如果你在 qq 设置代理到我们这个伪 Fildder 之后，会发现部分流量流量不是 http 流量，原因是 http 隧道也是一个通用传输层，可以传输任意 tcp 或 tcp 之上的流量。所以我们需要新的中间件来检测当前流量，如果不是 http 流量就回退到隧道代理的流程，即我们不跟踪不分析这部分非 http 流量。

6.1 http 流量侦测

/// <summary>
/// 流量侦测器
/// </summary>
private static class FlowDetector
{
    private static readonly byte[] crlf = Encoding.ASCII.GetBytes("\r\n");
    private static readonly byte[] http10 = Encoding.ASCII.GetBytes(" HTTP/1.0");
    private static readonly byte[] http11 = Encoding.ASCII.GetBytes(" HTTP/1.1");
    private static readonly byte[] http20 = Encoding.ASCII.GetBytes(" HTTP/2.0");

    /// <summary>
    /// 传输内容是否为http
    /// </summary>
    /// <param name="context"></param>
    /// <returns></returns>
    public static async ValueTask<bool> IsHttpAsync(ConnectionContext context)
    {
        var input = context.Transport.Input;
        var result = await input.ReadAtLeastAsync(1);
        var isHttp = IsHttp(result);
        input.AdvanceTo(result.Buffer.Start);
        return isHttp;
    }

    private static bool IsHttp(ReadResult result)
    {
        var reader = new SequenceReader<byte>(result.Buffer);
        if (reader.TryReadToAny(out ReadOnlySpan<byte> line, crlf))
        {
            return line.EndsWith(http11) || line.EndsWith(http20) || line.EndsWith(http10);
        }
        return false;
    }
}

6.2 隧道回退中间件

/// <summary>
/// 隧道传输中间件
/// </summary>
sealed class KestrelTunnelMiddleware : IKestrelMiddleware
{
    private readonly ILogger<KestrelTunnelMiddleware> logger;

    /// <summary>
    /// 隧道传输中间件
    /// </summary>
    /// <param name="logger"></param>
    public KestrelTunnelMiddleware(ILogger<KestrelTunnelMiddleware> logger)
    {
        this.logger = logger;
    }

    /// <summary>
    /// 执行中间你件
    /// </summary>
    /// <param name="next"></param>
    /// <param name="context"></param>
    /// <returns></returns>
    public async Task InvokeAsync(ConnectionDelegate next, ConnectionContext context)
    {
        var feature = context.Features.Get<IProxyFeature>();
        if (feature == null || feature.ProxyProtocol == ProxyProtocol.None)
        {
            this.logger.LogInformation($"侦测到http直接请求");
            await next(context);
        }
        else if (feature.ProxyProtocol == ProxyProtocol.HttpProxy)
        {
            this.logger.LogInformation($"侦测到普通http代理流量");
            await next(context);
        }
        else if (await FlowDetector.IsHttpAsync(context))
        {
            this.logger.LogInformation($"侦测到隧道传输http流量");
            await next(context);
        }
        else
        {
            this.logger.LogInformation($"跳过隧道传输非http流量{feature.ProxyHost}的拦截");
            await TunnelAsync(context, feature);
        }
    }

    /// <summary>
    /// 隧道传输其它协议的数据
    /// </summary>
    /// <param name="context"></param>
    /// <param name="feature"></param>
    /// <returns></returns>
    private async ValueTask TunnelAsync(ConnectionContext context, IProxyFeature feature)
    {
        var port = feature.ProxyHost.Port;
        if (port == null)
        {
            return;
        }

        try
        {
            var host = feature.ProxyHost.Host;
            using var socket = new Socket(SocketType.Stream, ProtocolType.Tcp);
            await socket.ConnectAsync(host, port.Value, context.ConnectionClosed);
            Stream stream = new NetworkStream(socket, ownsSocket: false);

            // 如果有tls中间件，则反回来加密隧道
            if (context.Features.Get<ITlsConnectionFeature>() != null)
            {
                var sslStream = new SslStream(stream, leaveInnerStreamOpen: true);
                await sslStream.AuthenticateAsClientAsync(feature.ProxyHost.Host);
                stream = sslStream;
            }

            var task1 = stream.CopyToAsync(context.Transport.Output);
            var task2 = context.Transport.Input.CopyToAsync(stream);
            await Task.WhenAny(task1, task2);
        }
        catch (Exception ex)
        {
            this.logger.LogError(ex, $"连接到{feature.ProxyHost}异常");
        }
    }
}

7 请求响应分析 http 中间件

这部分属于 asp.netcore 应用层内容，关键点是制作可多次读取的 http 请求 body 流和 http 响应 body 流，因为每个分析器实例都可以会重头读取一次请求内容和响应内容。

7.1 http 分析器

为了方便各种分析器的独立实现，我们定义 http 分析器的接口

/// <summary>
/// http分析器
/// 支持多个实例
/// </summary>
public interface IHttpAnalyzer
{
    /// <summary>
    /// 分析http
    /// </summary>
    /// <param name="context"></param>
    /// <returns></returns>
    ValueTask AnalyzeAsync(HttpContext context);
}

这是输到日志的 http 分析器

public class LoggingHttpAnalyzer : IHttpAnalyzer
{
    private readonly ILogger<LoggingHttpAnalyzer> logger;

    public LoggingHttpAnalyzer(ILogger<LoggingHttpAnalyzer> logger)
    {
        this.logger = logger;
    }

    public async ValueTask AnalyzeAsync(HttpContext context)
    {
        var builder = new StringBuilder();
        var writer = new StringWriter(builder);

        writer.WriteLine("[REQUEST]");
        await context.SerializeRequestAsync(writer);

        writer.WriteLine("[RESPONSE]");
        await context.SerializeResponseAsync(writer);

        this.logger.LogInformation(builder.ToString());
    }
}

7.2 分析 http 中间件

我们把请求 body 流和响应 body 流保存到临时文件，在所有分析器工作之后再删除。

/// <summary>
/// http分析中间件
/// </summary>
sealed class HttpAnalyzeMiddleware
{
    private readonly RequestDelegate next;
    private readonly IEnumerable<IHttpAnalyzer> analyzers;

    /// <summary>
    /// http分析中间件
    /// </summary>
    /// <param name="next"></param>
    /// <param name="analyzers"></param>
    public HttpAnalyzeMiddleware(
        RequestDelegate next,
        IEnumerable<IHttpAnalyzer> analyzers)
    {
        this.next = next;
        this.analyzers = analyzers;
    }

    /// <summary>
    /// 分析代理的http流量
    /// </summary>
    /// <param name="context"></param>
    /// <returns></returns>
    public async Task InvokeAsync(HttpContext context)
    {
        var feature = context.Features.Get<IProxyFeature>();
        if (feature == null || feature.ProxyProtocol == ProxyProtocol.None)
        {
            await next(context);
            return;
        }

        context.Request.EnableBuffering();
        var oldBody = context.Response.Body;
        using var response = new FileResponse();

        try
        {
            // 替换response的body
            context.Response.Body = response.Body;

            // 请求下个中间件
            await next(context);

            // 处理分析
            await this.AnalyzeAsync(context);
        }
        finally
        {
            response.Body.Position = 0L;
            await response.Body.CopyToAsync(oldBody);
            context.Response.Body = oldBody;
        }
    }

    private async ValueTask AnalyzeAsync(HttpContext context)
    {
        foreach (var item in this.analyzers)
        {
            context.Request.Body.Position = 0L;
            context.Response.Body.Position = 0L;
            await item.AnalyzeAsync(context);
        }
    }


    private class FileResponse : IDisposable
    {
        private readonly string filePath = Path.GetTempFileName();

        public Stream Body { get; }

        public FileResponse()
        {
            this.Body = new FileStream(filePath, FileMode.Open, FileAccess.ReadWrite);
        }

        public void Dispose()
        {
            this.Body.Dispose();
            File.Delete(filePath);
        }
    }
}

8 反向代理 http 中间件

我们需要把请求转发到真实的目标服务器，这时我们的应用程序是一个 http 客户端角色，这个过程与 nginx 的反向代理是一致的。具体的实现上，我们直接使用 yarp 库来完成即可。

/// <summary>
/// http代理执行中间件
/// </summary>
sealed class HttpForwardMiddleware
{
    private readonly RequestDelegate next;
    private readonly IHttpForwarder httpForwarder;
    private readonly HttpMessageInvoker httpClient = new(CreateSocketsHttpHandler());

    /// <summary>
    /// http代理执行中间件
    /// </summary>
    /// <param name="next"></param>
    /// <param name="httpForwarder"></param>
    public HttpForwardMiddleware(
        RequestDelegate next,
        IHttpForwarder httpForwarder)
    {
        this.next = next;
        this.httpForwarder = httpForwarder;
    }

    /// <summary>
    /// 转发http流量
    /// </summary>
    /// <param name="context"></param>
    /// <returns></returns>
    public async Task InvokeAsync(HttpContext context)
    {
        var feature = context.Features.Get<IProxyFeature>();
        if (feature == null || feature.ProxyProtocol == ProxyProtocol.None)
        {
            await next(context);
        }
        else
        {
            var scheme = context.Request.Scheme;
            var destinationPrefix = $"{scheme}://{feature.ProxyHost}";
            await httpForwarder.SendAsync(context, destinationPrefix, httpClient, ForwarderRequestConfig.Empty, HttpTransformer.Empty);
        }
    }

    private static SocketsHttpHandler CreateSocketsHttpHandler()
    {
        return new SocketsHttpHandler
        {
            Proxy = null,
            UseProxy = false,
            UseCookies = false,
            AllowAutoRedirect = false,
            AutomaticDecompression = DecompressionMethods.None,
        };
    }
}

9 编排中间件创建服务器和应用

9.1 kestrel 中间件编排

这里要特别注意顺序，传输层套娃。

/// <summary>
///  ListenOptions扩展
/// </summary>
public static partial class ListenOptionsExtensions
{
    /// <summary>
    /// 使用Fiddler的kestrel中间件
    /// </summary>
    /// <param name="listen"></param>
    public static ListenOptions UseFiddler(this ListenOptions listen)
    {
        // 代理协议中间件
        listen.Use<KestrelProxyMiddleware>();

        // tls侦测中间件
        listen.UseTlsDetection(tls =>
        {
            var certService = listen.ApplicationServices.GetRequiredService<CertService>();
            certService.CreateCaCertIfNotExists();
            certService.InstallAndTrustCaCert();
            tls.ServerCertificateSelector = (context, domain) => certService.GetOrCreateServerCert(domain);
        });

        // 隧道代理处理中间件
        listen.Use<KestrelTunnelMiddleware>();
        return listen;
    }
}

9.2 http 中间件的编排

public static class ApplicationBuilderExtensions
{
    /// <summary>
    /// 使用Fiddler的http中间件
    /// </summary>
    /// <param name="app"></param>
    public static void UseFiddler(this IApplicationBuilder app)
    {
        app.UseMiddleware<HttpAnalyzeMiddleware>();
        app.UseMiddleware<HttpForwardMiddleware>();
    }
}

9.3 创建应用

我们可以在传统的 MVC 里创建伪 fiddler 的首页、下载证书等 http 交互页面。

public static void Main(string[] args)
{
    var builder = WebApplication.CreateBuilder(args);

    builder.Services
        .AddFiddler()
        .AddControllers();

    builder.WebHost.ConfigureKestrel((context, kestrel) =>
    {
        var section = context.Configuration.GetSection("Kestrel");
        kestrel.Configure(section).Endpoint("Fiddler", endpoint => endpoint.ListenOptions.UseFiddler());
    });

    var app = builder.Build();
    app.UseRouting();
    app.UseFiddler();

    app.MapControllerRoute(
        name: "default",
        pattern: "{controller=Home}/{action=Index}/{id?}");

    app.Run();
}