Skip to content

FF-A Notifications - #6394

Merged
jforissier merged 13 commits into
OP-TEE:masterfrom
jenswikl:notif
Nov 29, 2023
Merged

FF-A Notifications#6394
jforissier merged 13 commits into
OP-TEE:masterfrom
jenswikl:notif

Conversation

@jenswikl

Copy link
Copy Markdown
Contributor

No description provided.

@b49020

b49020 commented Oct 30, 2023

Copy link
Copy Markdown
Member

As I mentioned here [1], it would be good to unify notifications feature to use secure world donated SGI and get rid of platform specific SPI or PPI for SMC ABI too.

[1] https://lists.trustedfirmware.org/archives/list/op-tee@lists.trustedfirmware.org/message/VL7FPBS2LNT3OCZOPCSIGCPARFOMNRD7/

@etienne-lms

Copy link
Copy Markdown
Contributor

As I mentioned here [1], it would be good to unify notifications feature to use secure world donated SGI and get rid of platform specific SPI or PPI for SMC ABI too.

stm32mp platforms will continue to use a PPI over SMC ABI.

@b49020

b49020 commented Nov 6, 2023

Copy link
Copy Markdown
Member

stm32mp platforms will continue to use a PPI over SMC ABI.

Although I don't see ST platforms enabling notifications in upstream OP-TEE but any particular reason for this choice? I suppose there would be free SGI available there too.

@etienne-lms

Copy link
Copy Markdown
Contributor

stm32mp13 does, since not that long.
As for stm32mp15, there is some legacy upstream kernel contrainst, but I still hope to overcome them, some day.

Comment thread core/drivers/gic.c Outdated
SHIFT_U64(mask_aff2, 32) | SHIFT_U64(mask_aff3, 48);

if (cpu_mask & ITR_CPU_MASK_TO_THIS_CPU)
target_list = BIT32(mpidr & 0xf);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't the whole bitfield of affinity 0 be covered? I see no limitations on the number of CPU per cluster

target_list = BIT32(mpidr & 0xff)?

More than 4 core per cluster is quite unusual though.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The target list bitfield is only 16 bits wide so we need to mask the cpu_id to max 15 or we'll overflow.

@etienne-lms etienne-lms left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed-by: Etienne Carriere <etienne.carriere@foss.st.com> for commits
"qemu_armv8a: enable testing of notifications using the console",
"core: separate async notification implementation",
"core: notif: remove interrupt assumptions",

Acked-by: Etienne Carriere <etienne.carriere@foss.st.com> for commits
"core: mobj_ffa.c: add reassuring comment in mobj_ffa_unregister_by_cookie()",
"qemu_armv8a: add GIC v3 redistributor base address",
"core: ffa: special treatment for FFA_ERROR",

For commit "core: gic: add gic_init_donate_sgi_to_ns()":
The commit message states "Secure world currently doesn't use any SGIs at all" which may not be true with #6440.

Commit "core: ffa: add notifications with SPMC at S-EL1" is tough, I'm not done with it, event for a shallow review :)

Comment thread core/drivers/gic.c Outdated
#define GICD_CTLR_ENABLEGRP1 (1 << 1)

#define GICD_PIDR2_ARCHREV_SHIFT (4)
#define GICD_PIDR2_ARCHREV_MASK (0xF)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could remove parentheses (really need to be consistent with the useless ones above?)

Maybe add an inline comment stating on GICv2 the GICD register offset and bit mapping is the same despite the register is named ICPIDR2. Suggestion:
/* GICD IDR2 name differs on GICv3 and GICv2 but uses same offset and bit map */

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed that the offset is different for GICv2 (0xfe8 vs 0xffe8). I'll update.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks you to have cross checked that! I've misread both to specs as I failed to see they differ for an 'f'.
My apologies.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No worries, it was my mistake. Thanks for reviewing.

Comment thread core/drivers/gic.c Outdated
struct gic_data *gd = &gic_data;
vaddr_t gicc_base = 0;
vaddr_t gicd_base = 0;
uint32_t vers = 0;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

__maybe_unused

Comment thread core/drivers/gic.c Outdated
#else
uint32_t mask_id = it & 0xf;
uint32_t mask_group = group & 0x1;
uint32_t mask_group = group;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why this change on GICv2 implemenation?

Comment thread core/drivers/gic.c Outdated
#define GICD_CTLR_ARE_NS BIT32(5)

#define GICD_PIDR2_ARCHREV_SHIFT (4)
#define GICD_PIDR2_ARCHREV_MASK (0xF)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

already define by a previous commit (see lines 72-73).

Comment thread core/drivers/gic.c

/* GIC configuration is initialized from TF-A when embedded */
#ifndef CFG_WITH_ARM_TRUSTED_FW
#if defined(CFG_WITH_ARM_TRUSTED_FW)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you move the above inline comment inside this #ifdef block?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK

Comment thread core/drivers/gic.c

/* GIC configuration is initialized from TF-A when embedded */
#ifndef CFG_WITH_ARM_TRUSTED_FW
#if defined(CFG_WITH_ARM_TRUSTED_FW)

This comment was marked as resolved.

void thread_spmc_set_async_notif_intid(int intid);
#else
static inline void __noreturn
thread_spmc_set_async_notif_intid(int intid __unused)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nitpicking: would fit in a single line with s/intid/id/.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree, but I prefer using the same name for the argument.

* w3: Error code, 0 on success
* w4-w7: Not used (MBZ)
*/
#define OPTEE_FFA_ENABLE_ASYNC_NOTIF OPTEE_FFA_BLOCKING_CALL(5)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not using ID 4?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what happened with ID 4, but this has to be kept in synch with the kernel patches so changing this now might be more trouble than it's worth.

Comment thread core/arch/arm/kernel/thread_spmc.c Outdated
switch (args->a1 & FFA_FEATURES_FEATURE_ID_MASK) {
case FFA_FEATURE_SCHEDULE_RECV_INTR:
if (!spmc_notif_is_ready)
goto err;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be better to have explicit return values here:

		case FFA_FEATURE_SCHEDULE_RECV_INTR:
			if (spmc_notif_is_ready) {
				ret_fid = FFA_SUCCESS_32;
				ret_w2 = notif_intid;
			} else {
				ret_fid = FFA_ERROR;
				ret_w2 = FFA_NOT_SUPPORTED;
			}
			goto out;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This turned out a bit more complicated than I liked. I'll refactor this function to see if you prefer that.

@jenswikl

Copy link
Copy Markdown
Contributor Author

Update

@etienne-lms etienne-lms left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comments for commit "core: ffa: add notifications with SPMC at S-EL1"

Comment thread core/arch/arm/include/optee_ffa.h Outdated
* w5: Bitfield of secure world capabilities OPTEE_FFA_SEC_CAP_* below,
* unused bits MBZ.
* w6-w7: Not used (MBZ)
* w5: bitfield of OP-TEE capabilities OPTEE_FFA_SEC_CAP_*

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/bitfield/Bitfield/

cpu_spin_unlock_xrestore(&spmc_notif_lock, old_itr_status);
}

spmc_set_args(args, ret_fid, 0, ret_val, 0, 0, 0);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

generic comment: at many places is FFA_PARAM_MBZ is used instead of 0.

Comment thread core/arch/arm/kernel/thread_spmc.c Outdated

old_itr_status = cpu_spin_lock_xsave(&spmc_notif_lock);

if (!notif_async_is_started() && notif_vm_id_valid &&

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From the FF-A spec, return code should be DENIED "if the Notification bitmap is not registered or is
registered but not in a masked and non-pending state.".

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I'll update.

Comment thread core/arch/arm/kernel/thread_spmc.c Outdated
if (args->a5 || args->a6 || args->a7)
goto out;
if (args->a2) {
/* We only deal with global notifications for now. */

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nitpicking: remove period for inline comments consistency

Comment thread core/arch/arm/kernel/thread_spmc.c Outdated
goto out;
}

/* DST as in the destination of the eventual notification */

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"as in" ?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll rephrase.

Comment thread core/arch/arm/kernel/thread_spmc.c Outdated

/*
* Note we're only supporting physical OS kernel in normal world
* with Global Notifications at that.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"at that moment" ?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll rephrase it.

Comment thread core/arch/arm/kernel/thread_spmc.c Outdated
return TEE_SUCCESS;
}
#else /* !defined(CFG_CORE_SEL1_SPMC) */

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unrelated change?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure

Comment thread mk/config.mk
else
# CFG_CORE_ASYNC_NOTIF is defined by the platform to enable support
# for sending asynchronous notifications to normal world. Note that an
# for sending asynchronous notifications to normal world.

This comment was marked as resolved.

@jenswikl

Copy link
Copy Markdown
Contributor Author

Addressing comments. This PR needs to be rebased. OK if I squash in the review updates and rebase?

@etienne-lms

Copy link
Copy Markdown
Contributor

ok for rebasing (sorry for the delayed answer)

@jenswikl

Copy link
Copy Markdown
Contributor Author

Squashed and rebased.

@jenswikl

Copy link
Copy Markdown
Contributor Author

Tags applied and commit message of "core: gic: add gic_init_donate_sgi_to_ns()" with regards to used SGIs in the secure world.

@jenswikl

Copy link
Copy Markdown
Contributor Author

I've picked up the kernel counterpart of this, so the ABI is frozen.

@jockebech

jockebech commented Nov 20, 2023

Copy link
Copy Markdown
Contributor

This fails at IBART at the same way as @etienne-lms 's #6361 .
So, now we have

  • Two PR's failing in the same way.
  • I can easily reproduce the error locally when running with @etienne-lms branch in core interrupt: use device tree data #6361 on two different machines.
    repo init -u https://github.com/OP-TEE/manifest.git -m default.xml
    repo sync -j4
    cd build
    make toolchains -j2
    make -j28 CFG_LOCKDEP=y CFG_LOCKDEP_RECORD_STACK=n CFG_ENABLE_EMBEDDED_TESTS=n CFG_IN_TREE_EARLY_TAS=pkcs11/fd02c9da-306c-48c7-a49c-bbd827ae86ee CFG_PKCS11_TA=y BR2_PER_PACKAGE_DIRECTORIES=n CFG_EMBED_DTB=y
  • When I run tip on our master, I cannot reproduce it.

@jenswikl

Copy link
Copy Markdown
Contributor Author

Rebased on master

@etienne-lms etienne-lms left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed-by: Etienne Carriere <etienne.carriere@foss.st.com> for commits
"core: gic: check gic version" and
"core: gic: use redistributor CPU interface" (with minor suggestion).

Acked-by: Etienne Carriere <etienne.carriere@foss.st.com> for commit
"plat-vexpress: donate a secure SGI to normal world".

Question for commit "core: gic: add gic_init_donate_sgi_to_ns()" otherwise it looks good to me.

Comment thread core/drivers/gic.c Outdated

/* Offsets from gic.gicr_base[core_pos] */
#define GICR_V3_PCPUBASE_SIZE (2 * 64 * 1024)
#define GICR_SGIBASE_OFFSET (64 * 1024)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the spec uses SGI_base worinf, maybe rename to GICR_SGI_BASE_OFFSET?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, I'll update.

Comment thread core/drivers/gic.c Outdated
write_icc_igrpen1(1);
#else
io_write32(gd->gicc_base + GICC_PMR, 0x80);
io_write32(gd->gicc_base + GICC_PMR, 0x80);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before this change, when CFG_WITH_ARM_TRUSTED_FW is enable,
for GICv3 we called write_icc_pmr(0x80); + write_icc_igrpen1(1);
for GICv2 we called write_icc_igrpen1(1); + io_write32(gd->gicc_base + GICC_CTLR, ...)
Now its not the case. Is it on purpose?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that's on purpose. Should I do that in a separate commit or perhaps mention it in the commit?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Either a specific commit or a comment in the commit message would help.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's actually no change in behaviour since gic_cpu_init() is normally not called with CFG_WITH_ARM_TRUSTED_FW=y, but with donate SGI it will be needed. I'll make a separate commit and update all the platforms.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are many platforms that enable CFG_WITH_ARM_TRUSTED_FW and call gic_cpu_init() from boot_secondary_init_intc().

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems tricky to update all platforms in one go. How about deprecating gic_cpu_init() and add a new gic_init_per_cpu()? gic_cpu_init() is kept unchanged with regards to PMR and igrpen1/GICC_CTLR and gic_init_per_cpu() is expected by be by all secondary CPUs regardless of CFG_WITH_ARM_TRUSTED_FW. I'll submit separate PRs to tidy up the various platforms after this PR.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds a good comprise. We should notify platform maintainers to update their implementation so that we can get rid of gic_cpu_init() afterward.

@etienne-lms

Copy link
Copy Markdown
Contributor

You can rebase to solve confict and IBART test failure.

When asynchronous notifications are enabled the console driver in
qemu_armv8a is configured as a top half and bottom half driver allowing
basic testing of the notification framework.

Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
Reviewed-by: Etienne Carriere <etienne.carriere@foss.st.com>
Separates the implementation of sending asynchronous notifications from
the part managing reception of events. This makes room for an
alternative implementation based on FF-A.

Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
Reviewed-by: Etienne Carriere <etienne.carriere@foss.st.com>
Removes the assumption in notif_register_driver() that OP-TEE is using
interrupts to notify the normal world. The coming FF-A based
implementation will use FFA_NOTIFICATION_SET for when notifying using an
SPMC at S-EL2 or EL3.

Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
Reviewed-by: Etienne Carriere <etienne.carriere@foss.st.com>
…okie()

Adds a reassuring comment in mobj_ffa_unregister_by_cookie() to explain
why it may fail if the cookie hasn't been used yet. Updates the error
message to include inactive_refs.

Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
Acked-by: Etienne Carriere <etienne.carriere@foss.st.com>
Adds a check to see that the version of the GIC matches the expected
version.

Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
Reviewed-by: Etienne Carriere <etienne.carriere@foss.st.com>
GICv3 has a redistributor CPU interface that until now hasn't been used.
To prepare for coming patches that need to access the redistributor
initialize a list with each CPU specific redistributor address. A new
function gic_init_v3() is added with a parameter for the redistributor
base address.

Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
Reviewed-by: Etienne Carriere <etienne.carriere@foss.st.com>
Adds and configures the GIC v3 redistributor base address.

Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
Acked-by: Etienne Carriere <etienne.carriere@foss.st.com>
@jenswikl

Copy link
Copy Markdown
Contributor Author

Rebased to resolve the conflict. Tags applied.
I'm going to rework "core: gic: add gic_init_donate_sgi_to_ns()" as discussed above.

@jenswikl

Copy link
Copy Markdown
Contributor Author

Added "core: deprecate gic_cpu_init()" and "plat-vexpress: use gic_init_per_cpu()".

@jenswikl

Copy link
Copy Markdown
Contributor Author

Fixing the compile error.

@etienne-lms etienne-lms left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed-by: Etienne Carriere <etienne.carriere@foss.st.com> for commits
"core: deprecate gic_cpu_init()" (with typo fixed. Feel free to discard the suggestion)
and "plat-vexpress: use gic_init_per_cpu()".

Comment thread core/include/drivers/gic.h Outdated
* then used by the other functions.
* The two gic_init() and gic_init_v3() functions initializes the struct
* gic_data which is then used by the other functions. These two functions
* also initlized the GIC and are only supposed to be called from the

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/initlized/initialized/

Comment thread core/drivers/gic.c
void gic_cpu_init(void)
{
struct gic_data *gd = &gic_data;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe add IMSG("gic_cpu_init() is deprecated, please use gic_init_per_cpu()")

@jenswikl
jenswikl force-pushed the notif branch 2 times, most recently from f0ebf67 to 86efdbd Compare November 27, 2023 08:32
@jenswikl

Copy link
Copy Markdown
Contributor Author

Comments addressed and tag applied.

Deprecates gic_cpu_init() in favor of the new function
gic_init_per_cpu().  gic_cpu_init() is only supposed to be called by
secondary CPUs in non-TF-A configurations while gic_init_per_cpu()
should be called by all secondary CPUs. gic_init_per_cpu() itself takes
CFG_WITH_ARM_TRUSTED_FW into account instead of having each platform
doing that.

Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
Reviewed-by: Etienne Carriere <etienne.carriere@foss.st.com>
Calls gic_init_per_cpu() instead of the now deprecated gic_cpu_init().

Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
Reviewed-by: Etienne Carriere <etienne.carriere@foss.st.com>
@jenswikl

Copy link
Copy Markdown
Contributor Author

Fixing the checkpatch warning.

@jenswikl

Copy link
Copy Markdown
Contributor Author

Two commits remain to be reviewed:

  • core: ffa: add notifications with SPMC at S-EL1
  • core: gic: add gic_init_donate_sgi_to_ns()

@etienne-lms etienne-lms left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For commit "core: gic: add gic_init_donate_sgi_to_ns()":

Should core ensures there is no secure interrupt handler registered for donated SGIs, and counter part that secure interrupt handler cannot be registered for a donated SGI?

Typo in commit message:
"In case another non-secure SGI is needed secure world can donate one of its ..."

Comment thread core/drivers/gic.c Outdated
/* Ignore matching bits */
if (!(BIT32(n) & (grp0 ^ gd->per_cpu_group_status)))
continue;
need_sync = true;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe add this instruction below the below inline comment.

@etienne-lms etienne-lms left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comments for commit "core: ffa: add notifications with SPMC at S-EL1".

Comment thread mk/config.mk
ifeq ($(filter y, $(CFG_CORE_SEL1_SPMC) $(CFG_CORE_SEL2_SPMC) \
$(CFG_CORE_EL3_SPMC)),y)
# FF-A case, handled via the FF-A ABI
CFG_CORE_ASYNC_NOTIF ?= y

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

instad of the $(call force,...) in arm.mk and _CFG_CORE_ASYNC_NOTIF_DEFAULT_IMPL ?= $(CFG_CORE_ASYNC_NOTIF) in core.mk directives that are a bit confusing, I would suggest here:

 ifeq ($(filter y, $(CFG_CORE_SEL1_SPMC) $(CFG_CORE_SEL2_SPMC) \
 		  $(CFG_CORE_EL3_SPMC)),y)
 # FF-A case, handled via the FF-A ABI
 CFG_CORE_ASYNC_NOTIF ?= y
+$(call force,_CFG_CORE_ASYNC_NOTIF_DEFAULT_IMPL,n)`
 else
 # CFG_CORE_ASYNC_NOTIF is defined by the platform to enable support
 # for sending asynchronous notifications to normal world.
 # Interrupt ID must be configurged by the platform too. Currently is only
 # CFG_CORE_ASYNC_NOTIF_GIC_INTID defined.
 CFG_CORE_ASYNC_NOTIF ?= n
+$(call force,_CFG_CORE_ASYNC_NOTIF_DEFAULT_IMPL,$(CFG_CORE_ASYNC_NOTIF))`
 endif

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Much better, thanks.

Comment thread core/kernel/notif_default.c Outdated
*/

#include <bitstring.h>
#include <drivers/gic.h>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

replace with #include <kernel/interrupt.h>
add inclusion of assert.h and trace.h.

@jenswikl

Copy link
Copy Markdown
Contributor Author

For commit "core: gic: add gic_init_donate_sgi_to_ns()":

Should core ensures there is no secure interrupt handler registered for donated SGIs, and counter part that secure interrupt handler cannot be registered for a donated SGI?

Perhaps, but we should then also prevent registering a handler for a non-secure SGI. Can we take this later in another PR?

@jenswikl

Copy link
Copy Markdown
Contributor Author

Force push to update the commit message of "core: gic: add gic_init_donate_sgi_to_ns", the rest of the changes went into "[review]" commits.

@etienne-lms

etienne-lms commented Nov 29, 2023

Copy link
Copy Markdown
Contributor

Perhaps, but we should then also prevent registering a handler for a non-secure SGI. Can we take this later in another PR?

Sure.

Build errors:

 core/kernel/notif_default.c:8:10: fatal error: drivers/interrupt.h: No such file or directory
    8 | #include <drivers/interrupt.h>
      |          ^~~~~~~~~~~~~~~~~~~~~

s/drivers/interrupt.h/kernel/interrupt.h/
Maybe should we move kernel/interrupt.h to drivers/interrupt.h?

@etienne-lms etienne-lms left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed-by: Etienne Carriere <etienne.carriere@foss.st.com> for commit
"core: gic: add gic_init_donate_sgi_to_ns()" (edited: with comment addressed :)

Acked-by: Etienne Carriere <etienne.carriere@foss.st.com> for commit
"core: ffa: add notifications with SPMC at S-EL1".

Comment thread core/drivers/gic.c
* SGI-n differs from primary CPU configuration,
* let's sync up.
*/
need_sync = true;

@etienne-lms etienne-lms Nov 29, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I forgot to mention my comment also applied to gicv3_sync_sgi_config() line 225.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll fix it.

@jenswikl

Copy link
Copy Markdown
Contributor Author

I'll squash and apply tags once this has passed CI.

Adds gic_init_donate_sgi_to_ns() which changes a secure SGI to become
non-secure.

Secure world currently has a few SGIs to spare while normal world
typically have reserved each of the non-secure SGIs for different
purposes.

In case another non-secure SGI is needed secure world can donate one of
its unused SGIs. This configuration will then deviate from the standard
GIC configuration where SGI ID0-ID7 are non-secure and ID8-ID15 are
secure.

Platforms using gic_init_donate_sgi_to_ns() should also use
gic_init_per_cpu() instead of the deprecated gic_cpu_init().

Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
Reviewed-by: Etienne Carriere <etienne.carriere@foss.st.com>
If FFA_ERROR is received print the error code. If the FFA_ERROR is from
the SPMC panic, else return back FFA_ERROR(FFA_NOT_SUPPORTED).

Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
Acked-by: Etienne Carriere <etienne.carriere@foss.st.com>
Adds support for asynchronous notifications via FF-A with SPMC at S-EL1.

The OP-TEE FF-A ABI is extended to report support for asynchronous
notifications during OPTEE_FFA_EXCHANGE_CAPABILITIES.

The SPMC at S-EL1 is extended to provide the FF-A notifications ABI to a
normal world VM.

The notifications depends on having a non-secure SGI interrupt ID
available to notify normal world that a notification is pending.
Notifications becomes available once platform code has called
thread_spmc_set_async_notif_intid() with a designed SGI ID.

Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
Acked-by: Etienne Carriere <etienne.carriere@foss.st.com>
With OP-TEE acting as SPMC in S-EL1 donate a secure SGI to normal world
to be used to signal asynchronous notifications for FF-A.

Signed-off-by: Jens Wiklander <jens.wiklander@linaro.org>
Acked-by: Etienne Carriere <etienne.carriere@foss.st.com>
@jenswikl

Copy link
Copy Markdown
Contributor Author

Squashed and tags applied. All commits are reviewed and should be ready to merge.

@jforissier
jforissier merged commit d876c67 into OP-TEE:master Nov 29, 2023
@jenswikl
jenswikl deleted the notif branch November 29, 2023 11:00
paddr_t gicr_base_pa);
static inline void gic_init(paddr_t gicc_base_pa, paddr_t gicd_base_pa)
{
gic_init_v3(gicc_base_pa, gicd_base_pa, 0);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does this work? Any platform with CFG_ARM_GICV3 set will panic with a 0 for gicr_base_pa. I'm guessing you will need to convert all those plats to use gic_init_v3(), but none are converted so all will panic..

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm sorry, I missed that. Would it work to only check for the redistributor address if affinity routing is enabled? Or should we rather let it be optional and only panic or fail if a function that needs it is called?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@glneo, will this work? #6516

@glneo glneo mentioned this pull request Nov 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants