Benefits of using PCI Express with RedHawk

Reduced communication jitter

The SISCI API provides true kernel bypass communication, enabling user space applications to safely access remote memory. The Dolphin Express products are implemented using standard PCI Express bridges and switches; the data flow is managed by PCI Express ASICs. The hardware will instantly and automatically forward data to the addressed remote memory. The transfer does not depend on extensive software communication protocols, regular DMA operations or system interrupts that will add an undeterminstic communication delay and data delivery jitter.

The typical average communication latency between two systems connected using Dolphins PCI Express cards typically less than 600 nanoseconds on all supported operating systems. As long as there is enough free CPU resources in a system, the applications will be allocated to a free CPU and enabled to run virtually uninterrupted until it completes its tasks.

In practice, the application will be re-scheduled by the system scheduler (another application is given the ability to run for some time) or interrupted by the the operating system when it issues an internal inter-CPU interrupt etc. These execution delays are hardly noticeable for regular applications, but will cause hard real-time applications to fail to meet their timing requirements.

CPU shielding

CPU Shielding is a feature offered by real-time operating systems like RedHawk and it enables applications to be completely "shielded" from being interrupted by some or all system events and from being rescheduled. The benefits are obvious for real-time processes - the application will be allowed to always use 100% of the allocated CPU(s) to execute its real time process.

Performance

The SISCI ping pong benchmark scipp is included in the Dolphin software distribution and can be used to measure best, average and worst case latency for a sequence of ping pong data transfer operations.

The following results were generated when the tool was executed with and without RedHawk's CPU shielding techniques. The latency is for a full round-trip between two systems (the time it takes to send a message of "size" bytes to the remote system, receive it and send back a similar size message). High precision timers are used to measure the round-trip latency for each transfer:

The latency for a full round-trip is around 2 microseconds. The average latency is slightly higher - this means that the application often is able to run without being interrupted. The worst case latency is significantly higher - up to 30 microseconds - for a small transfer. By using the CPU Shielding technique - the worst case is reduced down to slightly above the average latency.

By using the combination of Dolphin Express products and RedHawk Linux, the worst case communication latency and jitter can be reduced down to an absolute minimum. This enables hard real-time applications to meet their real-time constraint.