前往小程序,Get更优阅读体验!
立即前往
首页
学习
活动
专区
工具
TVP
发布
社区首页 >专栏 >SMP多核启动

SMP多核启动

作者头像
233333
发布2018-07-30 11:00:53
3.4K0
发布2018-07-30 11:00:53
举报

在 Linux系统中,对于多核的ARM芯片而言,在Biotron代码中,每个CPU都会识别自身ID,如果ID是0,则引导Bootloader和 Linux内核执行,如果ID不是0,则Biotron一般在上电时将自身置于WFI或者WFE状态,并等待CPU0给其发CPU核间中断或事件(一般通过SEV指令)以唤醒它。一个典型的多核 Linux启动过程如图20.6所示。 被CPU0唤醒的CPUn可以在运行过程中进行热插拔,譬如运行如下命令即可卸载CPU1,并且将CPUI上的任务全部迁移到其他CPU中:

代码语言:javascript
复制
# echo 0 > /sys/devices/system/cpu/cpu1/online

同理,运行如下命令可以再次启动CPU1:

代码语言:javascript
复制
# echo 1 > /sys/devices/system/cpu/cpu1/online

之后CPU1会主动参与系统中各个CPU之间的运行任务的负载均衡工作;

CPUO唤醒其他CPU的动作在内核中被封装为一个 smp_operations的结构体,对于ARM而言,它定义于 arch/arm/include/asm/smp.h中。该结构体的成员函数如代码清单所示。

代码语言:javascript
复制
struct smp_operations {
#ifdef CONFIG_SMP
    /*
     * Setup the set of possible CPUs (via set_cpu_possible)
     */
    void (*smp_init_cpus)(void);
    /*
     * Initialize cpu_possible map, and enable coherency
     */
    void (*smp_prepare_cpus)(unsigned int max_cpus);

    /*
     * Perform platform specific initialisation of the specified CPU.
     */
    void (*smp_secondary_init)(unsigned int cpu);
    /*
     * Boot a secondary CPU, and assign it the specified idle task.
     * This also gives us the initial stack to use for this CPU.
     */
    int  (*smp_boot_secondary)(unsigned int cpu, struct task_struct *idle);
#ifdef CONFIG_HOTPLUG_CPU
    int  (*cpu_kill)(unsigned int cpu);
    void (*cpu_die)(unsigned int cpu);
    int  (*cpu_disable)(unsigned int cpu);
#endif
#endif
};

CPUO唤醒其他CPU的动作在内核中被封装为一个 smp_operations 的结构体,对于ARM而言,它定义于 arch/arm/include/asm/smp.h中。该结构体的成员函数如代码清单所示。

代码语言:javascript
复制
DT_MACHINE_START(VEXPRESS DT,"ARM-Versatile Express)
.dt_compat = v2m_dt_match,
.smp = smp_ops(express_smp_ops),
.map_io = v2m_dt_map_io,
MACHINE_END

通过 arch/arm/mach-vexpress/platsmp.c的实现代码可以看出, smp_operations的成员函数smp_init_cpus(),即 vexpress_smp_init_cpus调用的ct_ca9x4_init_cpu_map(会探测SoC内CPU核的个数,并通过 set_cpu_possible设置这些CPU可见。 而 smp_operations的成员函数 smp_prepare_cpus,即 vexpress_smp_prepare_cpus则会通过v2m_flags_set( virt_to_phys( versatile_secondary_startup)设置其他CPU的启动地址为versatile_secondary_startup,如代码清单所示。 在smp_prepare_cpus()设置CPU1...n启动地址:

代码语言:javascript
复制
static void __init vexpress_smp_prepare_cpus(unsigned int max_cpus)
{
    /*
     * Initialise the present map, which describes the set of CPUs
     * actually populated at the present time.
     */
    if (ct_desc)
        ct_desc->smp_enable(max_cpus);
    else
        vexpress_dt_smp_prepare_cpus(max_cpus);

    /*
     * Write the address of secondary startup into the
     * system-wide flags register. The boot monitor waits
     * until it receives a soft interrupt, and then the
     * secondary CPU branches to this address.
     */
    vexpress_flags_set(virt_to_phys(versatile_secondary_startup));
}

注意这部分具体实现方式是与SOC相关的,由芯片设计及芯片内部的Bootrom决定。对于VEXPRESS来讲,设置方法如下:

代码语言:javascript
复制
void __init v2m_flags_set(u32 data)
{
    writel(~0, v2m_sysreg_base + V2M_SYS_FLAGSCLR);
    writel(data, v2m_sysreg_base + V2M_SYS_FLAGSCLR);
}

即填充v2m_sysreg_base+V2M_SYS_FLAGSCLR标记清除寄存器为0xFFFFFFFF,将CPU1...n初始启动执行的指令地址填入v2m_sysreg_base+V2M_SYS_FLAGSSET寄存器。这两个地址由芯片内部的Bootrom程序设定的。填入的CPU1...n的起始地址都通过virt_to_phys转化为物理地址,因为此时CPU1...n的MMU尚未开启; 比较关键的是smp_operations的成员函数smp_boot_secondary(),它是完成CPU最终唤醒的工作,对于本例而言,versatile_boot_secondary(); CPU0通过终端唤醒其他CPU:

代码语言:javascript
复制
/*
 * Write pen_release in a way that is guaranteed to be visible to all
 * observers, irrespective of whether they're taking part in coherency
 * or not.  This is necessary for the hotplug code to work reliably.
 */
static void __cpuinit write_pen_release(int val)
{
    pen_release = val;
    smp_wmb();
    __cpuc_flush_dcache_area((void *)&pen_release, sizeof(pen_release));
    outer_clean_range(__pa(&pen_release), __pa(&pen_release + 1));
}
代码语言:javascript
复制
int __cpuinit versatile_boot_secondary(unsigned int cpu, struct task_struct *idle)
{
    unsigned long timeout;

    /*
     * Set synchronisation state between this boot processor
     * and the secondary one
     */
    spin_lock(&boot_lock);

    /*
     * This is really belt and braces; we hold unintended secondary
     * CPUs in the holding pen until we're ready for them.  However,
     * since we haven't sent them a soft interrupt, they shouldn't
     * be there.
     */
    write_pen_release(cpu_logical_map(cpu));

    /*
     * Send the secondary CPU a soft interrupt, thereby causing
     * the boot monitor to read the system wide flags register,
     * and branch to the address found there.
     */
    arch_send_wakeup_ipi_mask(cpumask_of(cpu));

    timeout = jiffies + (1 * HZ);
    while (time_before(jiffies, timeout)) {
        smp_rmb();
        if (pen_release == -1)
            break;

        udelay(10);
    }

    /*
     * now the secondary core is starting up let it run its
     * calibrations, then wait for it to finish
     */
    spin_unlock(&boot_lock);

    return pen_release != -1 ? -ENOSYS : 0;
}

调用的 write_pen_release会将 pen_release变量设置为要唤醒的CPU核的CPU号 cpu_logical_map(cpu),而后通过 arch_send_wakeup_ipi mask给要唤醒的CPU发IPI中断,这个时候,被唤醒的CPU会退出WFI状态并从前面 smp_operations中的smp_prepare_cpus成员函数,即 vexpress_smp_prepare_cpus里通过 v2m_flags_set()设置的起始地址 versatile_secondary_startup开始执行,如果顺利的话,该CPU会将原先为正数的pen_release写为-1,以便CPU0从等待pen_release成为-1的循环跳出;

versatile_secondary_startup实现于arch/arm/plat-versatile/headsmp.S中,是一段汇编,如下代码所示:

代码语言:javascript
复制
/*
 * Realview/Versatile Express specific entry point for secondary CPUs.
 * This provides a "holding pen" into which all secondary cores are held
 * until we're ready for them to initialise.
 */
ENTRY(versatile_secondary_startup)
    mrc p15, 0, r0, c0, c0, 5
    bic r0, #0xff000000
    adr r4, 1f
    ldmia   r4, {r5, r6}
    sub r4, r4, r5
    add r6, r6, r4
pen:    ldr r7, [r6]
    cmp r7, r0
    bne pen

    /*
     * we've been released from the holding pen: secondary_stack
     * should now contain the SVC stack for this core
     */
    b   secondary_startup

    .align
1:  .long   .
    .long   pen_release
ENDPROC(versatile_secondary_startup)

上述循环代码的循环是等待pen_release变量称为CPU0设置的cpu_logical_map(cpu),一般就直接成立了。第16行调用内核通用的secondary_startup()函数,经过一系列初始化(如MMU等),最终新的被唤醒的CPU将调用smp_operationssmp_secondary_init()的成员函数,对于本例为versatile_secondary_init()

代码语言:javascript
复制
void __cpuinit versatile_secondary_init(unsigned int cpu)
{
    /*
     * let the primary processor know we're out of the
     * pen, then head off into the C entry point
     */
    write_pen_release(-1);

    /*
     * Synchronise with the boot thread.
     */
    spin_lock(&boot_lock);
    spin_unlock(&boot_lock);
}

上述代码会将pen_release写为-1,于是CPU0还在执行代码的versatile_boot_secondary()函数中的如下循环就退出了:

代码语言:javascript
复制
timeout = jiffies + (1 * HZ);
while (time_before(jiffies, timeout)) {
    smp_rmb();
    if (pen_release == -1)
        break;

    udelay(10);
}

这样CPU0就知道目标CPU已经被正确地唤醒,此后CPU0和新唤醒的其他CPU各自运行。整个系统在运行过程中会进行实时进程和正常进程的动态负载均衡。

下图总结了前文提到的vexpress_smp_prepare_cpus()versatile_boot_secondary()write_pen_release()versatile_secondary_startup()versatile_secondary_init()这些函数的执行顺序;

本文参与 腾讯云自媒体同步曝光计划,分享自作者个人站点/博客。
原始发表:2018-07-26 ,如有侵权请联系 cloudcommunity@tencent.com 删除

本文分享自 作者个人站点/博客 前往查看

如有侵权,请联系 cloudcommunity@tencent.com 删除。

本文参与 腾讯云自媒体同步曝光计划  ,欢迎热爱写作的你一起参与!

评论
登录后参与评论
0 条评论
热度
最新
推荐阅读
相关产品与服务
负载均衡
负载均衡(Cloud Load Balancer,CLB)提供安全快捷的流量分发服务,访问流量经由 CLB 可以自动分配到云中的多台后端服务器上,扩展系统的服务能力并消除单点故障。负载均衡支持亿级连接和千万级并发,可轻松应对大流量访问,满足业务需求。
领券
问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档