This post is part 2 of the "SBSA Reference Platform in QEMU" series:
- Versioning of sbsa-ref machine
- SBSA Reference Platform update
- Testing *BSD on SBSA Reference Platform
- Running SBSA Reference Platform
- DT-free EDK2 on SBSA Reference Platform
- ConfigurationManager in EDK2: just say no
There were several changes done since my previous post on the topic. So after some discussions I decided to write a post about it.
There are improvements, fixes and even issues with BSA specification.
Versioning related changes
SBSA Reference Platform (“sbsa-ref” in short) is now at version 0.3 one. Note that this is internal number. Machine name is still the same.
First bump was adding GIC data into (minimalistic) device-tree so firmware can configure it without using any magic numbers (as it was before).
Second update added GIC ITS (Interrupt Translation Services) support. Which means that we can have MSI-X interrupts and complex PCI Express setup.
Third time we said goodbye to USB 2.0 (EHCI) host controller. It never worked and only generated kernel warnings. XHCI (USB 3) controller is used instead now. EDK2 enablement is still work in progress.
Firmware improvements
Most of versioning updates involved firmware changes. Information about hardware details gets passed from virtual hardware level to operating system via standard defined ways:
- Trusted Firmware (TF-A) gets minimalistic Device-Tree from QEMU
- UEFI (EDK2) uses Secure Monitor Calls to get information from TF-A
- operating system uses ACPI tables
This way we were able to get rid of part of “magic numbers” from firmware components.
CPU updates
We can use Neoverse V1 cpu core now. It uses Arm v8.4 architecture and brings SVE and a bunch of other interesting features. You may need to update Trusted Firmware to make use of it.
QEMU got Arm Cortex-A710 cpu core support. It is first Arm v9.0 core there. Due to 240 address space we cannot use it for sbsa-ref. But it prepares code for Neoverse N2/V2 cores.
PCI Express changes and disputes
SBSA Reference Platform passes most of BSA ACS tests from PCI Express module:
*** Starting PCIe tests ***
Operating System View:
801 : Check ECAM Presence : Result: PASS
802 : PE - ECAM Region accessibility check : Result: PASS
803 : All EP/Sw under RP in same ECAM Region : Result: PASS
804 : Check RootPort NP Memory Access : Result: PASS
805 : Check RootPort P Memory Access : Result: PASS
806 : Legacy int must be SPI & lvl-sensitive
Checkpoint -- 2 : Result: SKIPPED
808 : Check all 1's for out of range : Result: PASS
809 : Vendor specfic data are PCIe compliant : Result: PASS
811 : Check RP Byte Enable Rules : Result: PASS
817 : Check Direct Transl P2P Support
Checkpoint -- 1 : Result: SKIPPED
818 : Check RP Adv Error Report
Checkpoint -- 1 : Result: SKIPPED
819 : RP must suprt ACS if P2P Txn are allow
Checkpoint -- 1 : Result: SKIPPED
820 : Type 0/1 common config rule : Result: PASS
821 : Type 0 config header rules : Result: PASS
822 : Check Type 1 config header rules
BDF 0x400 : SLT attribute mismatch: 0xFF020100 instead of 0x20100
BDF 0x500 : SLT attribute mismatch: 0xFF030300 instead of 0x30300
BDF 0x600 : SLT attribute mismatch: 0xFF040400 instead of 0x40400
BDF 0x700 : SLT attribute mismatch: 0xFF050500 instead of 0x50500
BDF 0x800 : SLT attribute mismatch: 0xFF060600 instead of 0x60600
BDF 0x900 : SLT attribute mismatch: 0xFF080700 instead of 0x80700
BDF 0x10000 : SLT attribute mismatch: 0xFF020201 instead of 0x20201
Failed on PE - 0
Checkpoint -- 7 : Result: FAIL
824 : Device capabilities reg rule : Result: PASS
825 : Device Control register rule : Result: PASS
826 : Device cap 2 register rules : Result: PASS
830 : Check Cmd Reg memory space enable
BDF 400 MSE functionality failure
Failed on PE - 0
Checkpoint -- 1 : Result: FAIL
831 : Check Type0/1 BIST Register rule : Result: PASS
832 : Check HDR CapPtr Register rule : Result: PASS
833 : Check Max payload size supported : Result: PASS
835 : Check Function level reset : Result: PASS
836 : Check ARI forwarding enable rule : Result: PASS
837 : Check Config Txn for RP in HB : Result: PASS
838 : Check all RP in HB is in same ECAM : Result: PASS
839 : Check MSI support for PCIe dev : Result: PASS
840 : PCIe RC,PE - Same Inr Shareable Domain : Result: PASS
841 : NP type-1 PCIe supp 32-bit only
NP type-1 pcie is not 32-bit mem type
Failed on PE - 0
Checkpoint -- 1 : Result: FAIL
842 : PASID support atleast 16 bits
Checkpoint -- 3 : Result: SKIPPED
One or more PCIe tests failed or were skipped.
-------------------------------------------------------
Total Tests run = 30 Tests Passed = 22 Tests Failed = 3
-------------------------------------------------------
As you see some of them require work.
Root ports SLT issue
I reported problem with test 822 to QEMU developers and turned out that it is a bug there. I got patch from Michael S. Tsirkin (one of QEMU PCI maintainers) and it made test pass. I hope it will be merged soon.
PCIe to PCI bridge issue
I wonder how many SBSA physical platforms will use one of those. Probably none, but my testing setup has one.
And it makes test 841 fail. This time problem requires more discussion because BSA specification writes (chapter E.2 PCI Express Memory Space):
When PCI Express memory space is mapped as normal memory, the system must support unaligned accesses to that region. PCI Type 1 headers, used in PCI-to-PCI bridges, and therefore in root ports and switches, have to be programmed with the address space resources claimed by the given bridge. For non-prefetchable (NP) memory, Type 1 headers only support 32-bit addresses. This implies that endpoints on the other end of a PCI-to-PCI bridge only support 32-bit NP BARs.
On the other side we have PCI Express Base Specification Revision 6.0 which, in chapter 7.5.1.2.1, says that BAR can be either 32 or 64-bit long:
Base Address registers that map into Memory Space can be 32 bits or 64 bits wide (to support mapping into a 64-bit address space) with bit 0 hardwired to 0b. For Memory Base Address registers, bits 2 and 1 have an encoded meaning as shown in Table 7-9. Bit 3 should be set to 1b if the data is prefetchable and set to 0b otherwise. A Function is permitted to mark a range as prefetchable if there are no side effects on reads, the Function returns all bytes on reads regardless of the byte enables, and host bridges can merge processor writes into this range 150 without causing errors. Bits 3-0 are read-only.
Table 7-9 Memory Base Address Register Bits 2:1 Encoding
Bits 2:1(b) Meaning 00 Base register is 32 bits wide and can be mapped anywhere in the 32 address bit Memory Space. 01 Reserved 10 Base register is 64 bits wide and can be mapped anywhere in the 64 address bit Memory Space. 11 Reserved
And pcie-pci-bridge device in QEMU uses 64-bit BAR.
I opened support ticket for it at Arm. Will see how it ends.
Non-secure EL2 virtual timer
Arm v8.1 architecture brought Virtual Host Extension (VHE in short). And it added one more timer: non-secure EL2 virtual timer.
BSA ACS checks for it and we were failing:
226 : Check NS EL2-Virt timer PPI Assignment START
NS EL2 Virtual timer interrupt 28 not received
Failed on PE - 0
B_PPI_02
Checkpoint -- 4 : Result: FAIL
END
Turned out that everything to make it pass was already present in QEMU. Except code to enable it for our platform. Two lines of code were enough.
After I sent my small patch, Leif Lindholm extracted timer definitions to separate include file and cleaned code around it to make it easier to compare QEMU code with BSA specification.
Result? Test passes:
226 : Check NS EL2-Virt timer PPI Assignment START
Received vir el2 interrupt
B_PPI_02
: Result: PASS
END
Summary
SBSA Reference Platform in QEMU gets better and better with time. We can emulate more complex systems, information about hardware details gets passed from virtual hardware level to operating system via standard defined ways.
Still have test failures but less than it was in past.