Real-time Linux kernel patch set with Thomas Gleixner

RT linux patches 经过漫长的等待终于要见曙光了。

Jason Perlow, Editorial Director at theLinux Foundation interviews Thomas Gleixner, Linux Foundation Fellow, CTO ofLinutronix GmbH, and project leader of the PREEMPT_RT real-timekernel patch set.

JP: Greetings, Thomas! It’s great to have you here thismorning — although for you, it’s getting late in the afternoon in Germany.So PREEMPT_RT, the real-time patch set for the kernel is a fascinatingproject because it has some very important use-cases that most people who useLinux-based systems may not be aware of. First of all, can you tell me what“Real-Time” truly means?

JP：Thomas 你好！很高兴今天早上能请到你，虽然对你来说，德国已经是下午了。关于今天的主题，内核的实时补丁集 PREEMPT_RT 是一个很吸引人的项目，因为它有一些非常重要的用例，而使用基于 Linux 系统的大多数人可能没有意识到。首先，你能告诉我“实时”是什么意思吗？

TG: Real-Timein the context of operating systems means that the operating system providesmechanisms to guarantee that the associated real-time task processes an eventwithin a specified period of time. Real-Time is often confused with “reallyfast.” The late Prof. Doug Niehaus explained it this way: “Real-Time is not asfast as possible; it is as fast as specified.”

TG：在操作系统上下文中的实（Real-Time）是指操作系统提供了保证相关的实时任务在指定时间内处理事件的机制。实时常常与“非常快”相混淆。已故的 Doug Niehaus 教授这样解释：“实时不是尽可能快的；它是和指定的一样快。”

The specified time constraint isapplication-dependent. A control loop for a water treatment plant can havecomparatively large time constraints measured in seconds or even minutes, whilea robotics control loop has time constraints in the range of microseconds. Butfor both scenarios missing the deadline at which the computation has to befinished can result in malfunction. For some application scenarios, missing thedeadline can have fatal consequences.

In the strict sense of Real-Time, theguarantee which is provided by the operating system must be verifiable, e.g.,by mathematical proof of the worst-case execution time. In some applicationareas, especially those related to functional safety (aerospace, medical,automation, automotive, just to name a few), this is a mandatory requirement.But for other scenarios or scenarios where there is a separate mechanism forproviding the safety requirements, the proof of correctness can be morerelaxed. But even in the more relaxed case, the malfunction of a real-timesystem can cause substantial damage, which obviously wants to be avoided.

指定的时间约束和应用程序相关。水处理厂的控制回路可以有相对较松的时间限制，以秒甚至分钟为单位，而机器人控制回路的时间限制在微秒范围内。但是对于这两种情况，超时完成可能导致故障。对于某些应用程序场景，超时可能会产生致命的后果。

在严格意义上的实时，由操作系统提供的保证必须是可验证的，例如，通过数学证明最坏情况下的执行时间。在一些应用领域，特别是与功能安全相关的领域（航空航天、医疗、自动化、汽车等），这是一项强制性要求。但是对于其他场景或有提供安全需求的独立机制的场景，实时性可以更宽松一些。但即使在更宽松的情况下，实时系统的故障也会造成实质性的损害，这显然是要避免的。

JP: What is the history behind the project? How did it getstarted?

JP：这个项目背后的历史是什么？它是怎么开始的？

TG: Real-TimeLinux has a history that goes way beyond the actual PREEMPT_RT project.

TG：Real-Time Linux 的历史远远超出了实际的 PREEMPT_RT 项目

Linux became a research vehicle very earlyon. Real-Time researchers set out to transform Linux into a Real-Time Operatingsystem and followed different approaches with more or less success. Still, noneof them seriously attempted a fully integrated and perhaps upstream-ablevariant. In 2004 various parties started an uncoordinated effort to get somekey technologies into the Linux kernel on which they wanted to build properReal-Time support. None of them was complete, and there was a lack of anoverall concept.

Linux 很早就成为了研究工具。实时研究人员着手将 Linux 转变为实时操作系统，并采用了不同的方法，或多或少都取得了成功。尽管如此，他们中没有人认真尝试过一个完全整合的、或许可以提交到upstream的变种。在 2004 年，各方开始了一项不协调的努力，将一些关键技术引入 Linux 内核，他们希望在 Linux 内核上构建适当的实时支持。没有一个是完整的，并且缺少一个整体的概念。

Ingo Molnar, working for RedHat, started topick up pieces, reshape them and collect them in a patch series to build thegrounds for the real-time preemption patch set PREEMPT_RT. At thattime, I worked with thelate Dr. Doug Niehaus to port a solution we hadworking based on the 2.4 Linux kernel forward to the 2.6 kernel. Our work wasboth conflicting and complimentary, so I teamed up with Ingo quickly to getthis into a usable shape. Others like Steven Rostedt brought in ideas andexperience from other Linux Real-Time research efforts. With a quickly formingloose team of interested developers, we were able to develop a halfway usableReal-Time solution that was fully integrated into the Linux kernel in a shortperiod of time. That was far from a maintainable and production-ready solution.Still, we had laid the groundwork and proven that the concept of making theLinux Kernel real-time capable was feasible. The idea and intent of fullyintegrating this into the mainline Linux kernel over time were there from thevery beginning.

在 RedHat 工作的 Ingo Molnar开始check这些patches，将它们重新组合并收集成一个补丁系列，为实时抢占补丁集 PREEMPT_RT 奠定基础。当时，我与已故的 Doug Niehaus 博士[2]合作，将我们现有的一个基于 2.4 Linux 内核的解决方案移植到 2.6 内核。我们的工作既有矛盾又有互补，所以我很快就和 Ingo 合作了，让它变得有用。Steven Rostedt 等人从其他 Linux Real-Time 研究工作中带来了一些想法和经验。有了一个由感兴趣的开发人员组成的团队，我们能够在很短的时间内开发一个可用的实时解决方案，该解决方案完全集成到 Linux 内核中。这与可维护和生产就绪的解决方案相去甚远。尽管如此，我们已经打下了基础，并证明了使 Linux 内核具有实时性的概念是可行的。从一开始就有将其完全集成到主线 Linux 内核中的想法和意图。

JP: Why is it still a separate project from the Mainline kerneltoday?

JP：为什么现在它仍然是一个独立于主线内核的项目？

TG: Tointegrate the real-time patches into the Linux kernel, a lot of preparatorywork, restructuring, and consolidation of the mainline codebase had to be donefirst. While many pieces that emerged from the real-time work found their wayinto the mainline kernel rather quickly due to their isolation, the moreintrusive changes that change the Linux kernel’s fundamental behavior needed(and still need) a lot of polishing and careful integration work.

TG：为了将实时补丁集成到 Linux 内核中，必须首先做大量的准备工作、重组和巩固主线代码库。许多隔离（与mainline的code互不影响）的实时patch很快就进入了主线内核，但与mainline code 有关联，并且可能需要改变 Linux 内核基本行为的patch需要大量重构和仔细的集成工作。

Naturally, this has to be coordinated withall the other ongoing efforts to adopt the Linux kernel to the different usecases ranging from tiny embedded systems to supercomputers.

This also requires carefully designing theintegration so it does not get in the way of other interests and imposesroadblocks for further developing the Linux kernel, which is something thecommunity and especially Linus Torvalds, cares about deeply.

当然，这必须与所有其他正在进行的工作协调，以便在从微型嵌入式系统到超级计算机的不同用例中采用Linux 内核。

这也需要仔细设计集成，这样它就不会妨碍其他利益，也不会对进一步开发Linux 内核造成障碍，这是社区，特别是Linus Torvalds 非常关心的事情。

As long as these remaining patches are outof the mainline kernel, this is not a problem because it does not put anyburden or restriction on the mainline kernel. The responsibility is on thereal-time project, but on the other side, in this context, there is norestriction to take shortcuts that would never be acceptable in the upstreamkernel.

The real-time patches are fundamentallydifferent from something like a device driver that sits at some corner of thesource tree. A device driver does not cause any larger damage when it goesunmaintained and can be easily removed when it reaches the final state bit-rot.Conversely, the PREEMPT_RT core technology is in the heart ofthe Linux kernel. Long-term maintainability is key as any problem in that areawill affect the Linux user universe as a whole. In contrast, a bit-rotteddriver only affects the few people who have a device depending on it.

只要这些剩余的补丁不在主线内核中，这就不是问题，因为它不会给主线内核带来任何负担或限制。责任在实时项目上，但在另一方面，在这个上下文中，没有任何限制可以走上游内核永远不能接受的捷径。

实时补丁从根本上不同于位于源代码某个角落的设备驱动程序。设备驱动程序不会造成任何更大的损坏，当它在未被维护时，可以很容易地删除它。相反，PREEMPT_RT是 Linux 内核的心脏。长期的可维护性是关键，因为这方面的任何问题都会影响到整个Linux 用户世界。

JP: Traditionally, when I think about RTOS, I think of legacysolutions based on closed systems. Why is it essential we have an open-sourcealternative to them?

JP：传统上，当我想到 RTOS 时，我想到的是基于封闭系统的遗留解决方案。为什么我们有一个开源的替代品是必要的？

TG: TheRTOS landscape is broad and, in many cases, very specialized. As I mentioned onthe question of “what is real-time,” certain application scenarios require afully validated RTOS, usually according to an application space-specificstandard and often regulatory law. Aside from that, many RTOSes are limited toa specific class of CPU devices that fit into the targeted application space.Many of them come with specialized application programming interfaces whichrequire special tooling and expertise.

TG：实时操作系统的领域非常广泛，而且在很多情况下非常专一。正如我在“什么是实时”的问题上提到的，某些应用程序场景需要完全验证的RTOS，通常根据特定于应用程序领域的标准和规则。除此之外，许多RTOS 被限制在适合目标应用程序领域的特定类别的CPU 设备中。其中许多都提供了需要特殊工具和专业知识的专用API。

The Real-Time Linux project never aimed atthese narrow and specialized application spaces. It always was meant to be thesolution for 99% of the use cases and to be able to fully leverage theflexibility and scalability of the Linux kernel and the broader FOSS ecosystemso that integrated solutions with mixed-criticality workloads can be handledconsistently.

Real-Time Linux 项目从来没有把这些狭窄和专门的应用程序领域作为目标。它一直是99%用例的解决方案，并且能够充分利用Linux 内核和更广泛的FOSS 生态系统的灵活性和可伸缩性，以便能够一致地处理具有混合临界工作负载的集成解决方案。

Developing real-time applications on areal-time enabled Linux kernel is not much different from developingnon-real-time applications on Linux, except for the careful selection of systeminterfaces that can be utilized and programming patterns that should beavoided, but that is true for real-time application programming in generalindependent of the RTOS.

The important difference is that the toolsand concepts are all the same, and integration into and utilizing the largerFOSS ecosystem comes for free.

在支持实时的Linux 内核上开发实时应用程序与在Linux 上开发非实时应用程序没有太大的区别，除了要仔细选择可以利用的系统接口和应该避免的编程模式之外，但这对于一般独立于RTOS 的实时应用程序编程来说是正确的。

重要的区别在于，工具和概念都是相同的，集成到更大的FOSS 生态系统中并加以利用是免费的。

The downside of PREEMPT_RT isthat it can’t be fully validated, which excludes it from specific applicationspaces, but there are efforts underway, e.g., the LF ELISA project, to fillthat gap. The reason behind this is, that large multiprocessor systems havebecome a commodity, and the need for more complex real-time systems in variousapplication spaces, e.g., assisted / autonomous driving or robotics, requires amore flexible and scalable RTOS approach than what most of the specialized andvalidated RTOSes can provide.

That’s a long way down the road. Still,there are solutions out there today which utilize external mechanisms toachieve the safety requirements in some of the application spaces whileleveraging the full potential of a real-time enabled Linux kernel along withthe broad offerings of the wider FOSS ecosystem.

PREEMPT_RT 的缺点是它不能被完全验证，这将它排除在特定的应用程序空间之外，但目前正在进行一些工作，例如LF ELISA 项目，以填补这一空白。其背后的原因是，大型多处理器系统已成为一种商品，并且各种应用（例如，辅助/自动驾驶或机器人技术）中需要更复杂的实时系统，这需要比大多数经过验证的专业RTOS 可以提供更灵活和可扩展的RTOS 方法。

这是一条很长的路。尽管如此，现在仍然有一些解决方案利用外部机制来实现某些应用程序领域的安全需求，同时利用支持实时的Linux 内核的全部潜力以及更广泛的FOSS 生态系统的广泛产品。

JP: What are examples of products and systems that use thereal-time patch set that people depend on regularly?

JP：有哪些产品和系统使用人们经常依赖的实时补丁集的例子？

TG: It’sall over the place now. Industrial automation, control systems, robotics, medicaldevices, professional audio, automotive, rockets, and telecommunication, justto name a few prominent areas.

TG：现在到处都是。工业自动化、控制系统、机器人、医疗设备、专业音频、汽车、火箭和电信，这只是几个突出的领域。

JP: Who are the major participants currently developingsystems and toolsets with the real-time Linux kernel patch set?

JP：目前开发实时 Linux 内核补丁集的系统和工具集的主要参与者是谁？

TG: Listingthem all would be equivalent to reciting the “who’s who” in the industry. Onthe distribution side, there are offerings from, e.g., RedHat, SUSE, Mentor,and Wind River, which deliver RT to a broad range of customers in differentapplication areas. There are firms like Concurrent, National Instruments,Boston Dynamics, SpaceX, and Tesla, just to name a few on the products side.

RedHat and National Instruments are alsomembers of the LF collaborative Real-Time project.

TG：把它们都列出来就等于在背诵行业中的“名人录”。在发行端，有来自 RedHat、SUSE、Mentor 和 WindRiver 等公司的产品，它们将RT 交付给不同应用领域的广泛客户。在产品方面，有Concurrent、NationalInstruments、BostonDynamics、SpaceX和 Tesla 等公司。

RedHat 和National Instruments 也是LF 协作实时项目的成员。

JP: What are the challenges in developing a real-timesubsystem or specialized kernel for Linux? Is it any different than how otherprojects are run for the kernel?

JP：为 Linux 开发实时子系统或专门的内核有什么挑战？它与内核的其他项目运行有什么不同吗？

TG: Notreally different; the same rules apply. Patches have to be posted, arereviewed, and discussed. The feedback is then incorporated. The loop startsover until everyone agrees on the solution, and the patches get merged into therelevant subsystem tree and finally end up in the mainline kernel.

TG：没有什么不同；同样的规则适用。补丁必须发布、审查和讨论。然后反馈被合并。循环开始，直到每个人都同意解决方案，补丁被合并到相关的子系统树中，最后在主线内核中结束。

But as I explained before, it needs a lotof care and effort and, often enough, a large amount of extra work torestructure existing code first to get a particular piece of the patchesintegrated. The result is providing the desired functionality but is at thesame time not in the way of other interests or, ideally, provides a benefit foreveryone.

但正如我之前解释的那样，它需要大量的注意和努力，通常还需要大量的额外工作来重构现有代码，以便集成特定的补丁。结果是提供了期望的功能，但同时不是以其他利益的方式，或者，理想情况下，为每个人提供好处。

The technology’s complexity that reachesinto a broad range of the core kernel code is obviously challenging, especiallycombined with the mainline kernel’s rapid change rate. Even larger changeshappening at the related core infrastructure level are not impacting ongoingdevelopment and integration work too much in areas like drivers or filesystems. But any change on the core infrastructure can break a carefullythought-out integration of the real-time parts into that infrastructure andsend us back to the drawing board for a while.

该技术的复杂性涉及到广泛的核心内核代码，这显然是一个挑战，特别是结合主线内核的快速变化率。即使在相关的核心基础设施级别发生更大的变化，也不会过多地影响正在进行的开发和集成工作，比如驱动程序或文件系统。但是，对核心基础架构的任何更改都可能破坏对实时部分的仔细考虑的集成，并使我们暂时回到绘图板上。

JP: Whichcompanies have been supporting the effort to get the PREEMPT_RT Linuxkernel patches upstream?

JP：哪些公司一直在支持 PREEMPT_RT Linux 内核补丁的上游工作？

TG: Forthe past five years, it has been supported by the members of the LF real-timeLinux project, currently ARM, BMW, CIP, ELISA, Intel, National Instruments,OSADL, RedHat, and Texas Instruments. CIP, ELISA, and OSADL are projects ororganizations on their own which have member companies all over the industry.Former supporters include Google, IBM, and NXP.

I personally, my team and the broader Linuxreal-time community are extremely grateful for the support provided by thesemembers.

TG：在过去的5 年里，它得到了 LF 实时 Linux 项目成员的支持，目前包括 ARM、BMW、CIP、ELISA、Intel、NationalInstruments、OSADL、RedHat 和 TexasInstruments。CIP、ELISA 和 OSADL 是独立的项目或组织，其成员公司遍布整个行业。以前的支持者包括谷歌、IBM和 NXP。

我个人、我的团队和更广泛的Linux 实时社区非常感谢这些成员提供的支持。

However, as with other key open sourceprojects heavily used in critical infrastructure, funding always was and stillis a difficult challenge. Even if the amount of money required to keep suchlow-level plumbing but essential functionality sustained is comparativelysmall, these projects struggle with finding enough sponsors and often lacklong-term commitment.

然而，就像在关键基础设施中大量使用的其他关键开源项目一样，资金一直是一个困难的挑战，现在仍然是。即使维持这种低水平的基础功能所需的资金相对较少，这些项目也很难找到足够的赞助商，而且往往缺乏长期的承诺。

The approach to funding these kinds ofprojects reminds me of the MikadoGame, which is popular in Europe, where thefirst player who picks up the stick and disturbs the pile often is the one wholoses.

这种为此类项目提供资金的方法让我想起了在欧洲流行的 Mikado 游戏，即第一个拿起棍子并扰乱游戏堆的玩家往往就是输家。

That’s puzzling to me, especially as manycompanies build key products depending on these technologies and seem to takethe availability and sustainability for granted up to the point where such aproject fails, or people stop working on it due to lack of funding. Suchcompanies should seriously consider supporting the funding of the Real-Timeproject.

这让我很困惑，特别是许多公司依靠这些技术开发关键产品，似乎把可用性和可持续性视为理所当然，直到项目失败，或者人们因为缺乏资金而停止工作。这些公司应该认真考虑支持 Real-Time 项目的资金。

It’s a lot like the Jenga game,where everyone pulls out as many pieces as they can up until the point where itcollapses. We cannot keep taking; we have to give back to these communitiesputting in the hard work for technologies that companies heavily rely on.

I gave up long ago trying to make sense ofthat, especially when looking at the insane amounts of money thrown at theover-hyped technology of the day. Even if critical for a large part of theindustry, low-level infrastructure lacks the buzzword charm that attractsattention and makes headlines — but it still needs support.

这很像层层叠游戏，每个人都拿出尽可能多的棋子，直到它崩溃为止。我们不能继续索取；我们必须回报这些社区，他们为公司严重依赖的技术付出了艰苦的努力。

很久以前，我就放弃了对这一现象的理解，尤其是当我看到大量资金投入到当今过度炒作的技术上时。尽管对该行业的很大一部分来说至关重要，但低水平的基础设施缺乏吸引关注和成为头条新闻的流行词魅力——但它仍然需要支持。

JP: One of the historicalconcerns was that Real-Time didn’t have a community associated with it; whathas changed in the last five years?

JP：历史问题之一是，Real-Time 并没有一个与之相关的社区；过去五年发生了什么变化？

TG: Thereis a lively user community, and quite a bit of the activity comes from the LFproject members. On the development side itself, we are slowly gaining morepeople who understand the intricacies of PREEMPT_RT and alsopeople who look at it from other angles, e.g., analysis and instrumentation.Some fields could be improved, like documentation, but there is alwayssomething that can be improved.

TG：这里有一个活跃的用户社区，其中很多活动都来自于 LF 项目成员。就开发本身而言，我们正在慢慢地吸引更多的人来理解 PREEMPT_RT 的复杂之处，以及从其他角度来看待它的人，比如分析和工具。有些领域可以改进，比如文档，但总有一些地方可以改进。

JP: What will theReal-Time Stable team be doing once the patches are accepted upstream?

JP：一旦上游接受了这些补丁，Real-Time Stable 团队会做什么？

TG: Thestable team is currently overseeing the RT variants of the supported mainlinestable versions. Once everything is integrated, this will dry out to someextent once the older versions reach EOL. But their expertise will still berequired to keep real-time in shape in mainline and in the supported mainlinestable kernels.

TG：稳定团队目前正在监督受支持的主线稳定版本的 RT 变体。一旦所有东西都集成了，一旦旧版本达到 EOL，这种情况就会在某种程度上消失。但是他们的专业知识仍然需要在主线和支持主线的稳定内核中保持实时特性。

JP: So once the upstreaming activity is complete, whathappens afterward?

JP：那么，一旦上游操作完成，会发生什么呢？

TG: Onceupstreaming is done, efforts have to be made to enable RT support for specificLinux features currently disabled on real-time enabled kernels. Also, for quitesome time, there will be fallout when other things change in the kernel, andthere has to be support for kernel developers who run into the constraints ofRT, which they did not have to think about before.

TG：一旦完成上游操作，就必须努力为当前在实时启用的内核上禁用的特定 Linux 功能启用 RT 支持。此外，在相当长的一段时间内，当内核中其他东西发生变化时，将会有附带影响，并且必须支持那些遇到 RT 限制的内核开发人员，而这些限制是他们以前没有考虑过的。

The latter is a crucial point for thiseffort. Because there needs to be a clear longer-term commitment that thepeople who are deeply familiar with the matter and the concepts are not goingto vanish once the mainlining is done. We can’t leave everybody else with thetask of wrapping their brains around it in desperation; there cannot beinstitutional knowledge loss with a system as critical as this.

后者是这一努力的关键。因为需要有一个明确的长期承诺，让那些对事情和概念非常熟悉的人不会在主线完成后就消失。我们不能让其他人在绝望中绞尽脑汁地思考这个问题；在一个如此重要的体系中，不可能存在机构知识的流失。

The lack of such a commitment would be ashowstopper on the final step because we are now at the point where the notablechanges are focused on the real-time only aspects rather than welcomingcleanups, improvements, and features of general value. This, in turn, circlesback to the earlier question of funding and industry support — for this finalstep requires several years of commitment by companies using the real-timekernel.

缺乏这样的承诺将成为最后一步的阻碍，因为我们现在所处的位置是，显著的变化只集中在实时方面，而不是欢迎清理、改进和具有普遍价值的特性。反过来，这又回到了之前关于资金和行业支持的问题——因为这最后一步需要使用实时内核的公司几年的承诺。

There’s not going to be a shortage ofthings to work on. It’s not going to be as much as the current upstreamingeffort, but as the kernel never stops changing, this will be interesting for along time.

不会缺少需要努力的东西。它不会像当前的努力那样多，但由于内核永远不会停止改变，这在长期以来将很有趣。