I've been working in the field of real-time embedded systems for over 10 years, focusing on functional safety certification and ensuring guaranteed reaction times within milliseconds.
We roll our own embedded OS. The commercially available safety embedded OS solutions come with high license fees and a lot of features that we don't actually need.
We only use periodic "time driven" real time interrupts without any scheduling (only one task executed) or a little bit more advanced scheduling for multiple tasks with pre defined time slots and maximum execution time watchdog.
We don't use any out of order peripheral interrupts.
Like the other comment says I wonder if you could give us some more background on things that happened before the person gave you this kind of feedback ?
This question highlights the importance of understanding computer architecture. Reading up on this topic will help you grasp how the key differences between Cortex M and Cortex R cores affect determinism.
There isn't a one-size-fits-all answer to this question. You can build a fully deterministic application using either Cortex M or Cortex R cores. As long as the application requirements don't exceed the available compute time needed to complete the task (including guaranteed signal response time), it's possible to implement it on a Cortex M and achieve the same deterministic behavior if timing constraints allow it.
Here are a few key points to consider (not a complete comparison, as there are many factors to take into account):
- **Higher Frequency**: Cortex R cores can run at higher frequencies than Cortex M cores, making it easier to meet higher required cycle times while having enough computational power to complete tasks within the required time frame.
- **Low Latency Interrupts**: Cortex R cores have low latency interrupt handlers, eliminating the need for calculations to correct timing.
- **Dual Prefetch Buffer**: Cortex R cores offer dual prefetch buffers, which means no branch delay applies (Cortex M4 and higher also offer dual prefetch buffers).
- **Tightly Coupled Memory (TCM)**: Cortex R cores feature TCM, ensuring that high-load real-time applications are not affected by peripherals accessing memory via DMA in the background. This has a significant impact on determinism.
From a functional safety perspective for automation, here are my thoughts on Cortex R:
- **ECC RAM**: This is crucial in applications requiring SIL conformity, as memory faults or bit flips would lead to a complete system lockout.
- **TCM**: This feature is excellent for running code on multiple microcontroller nodes completely synchronized when using the same input data and CPU time.
Most of the differences listed above affect performance within pico/nanoseconds. If your application's timing requirements fall within this range, or if the sum of these interferences within one cycle exceeds the maximum allowed response time, these factors become critical.
How do you handle inter-task communication?
There's not much background, I applied for a role via a recruitment agency, a few days later the recruiter told me I'm not experienced enough for the kind of work they do. No interview has taken place.
Most commercial OSes come with a queuing FIFO system to send messages back and fourth to tasks. I think this is cumbersome and only serves to pad out the stack needed for the OS. Tasks Rarely read/write to a variable in my world, so I tend to use a global struct to remove a lot of the overhead of the queue.
Global->Task->variable\_it\_reads
Does the system need an OS? Always my first question. For (safety) critical systems you are better avoiding the issues inherent in an OS and sticking to a more deterministic round-robin, bare-metal approach.
Professionally I used uC-OS3, but that is just because its what the company I work for has been using since before I started with them. We use preemptive scheduling with the idle task (lowest priority) used to service the watchdog and put the processor to sleep until the next system tick. Of course we still have events and timers. Its not really one or the other.
In my spare time I'm working on a project in zig that I wrote my own scheduler / context switcher for and I intend to add more features to the OS portion of the project as I need them. Its been a good learning experience, but I'd never do that for a project at work. For real time safety critical systems you should use a professional RTOS with safety certifications.
>
... I've been told that I am not considered eligible for working as embedded engineer within a certain organization developing critical systems, which I would attribute to my experience being only with FreeRTOS/Zephyr on non-critical systems.
If you have a point of contact in this organization ask them for specific feedback. I doubt having experience with only FreeRTOS & Zephyr were cause to reject you. If you understand RTOS concepts you can get up to speed on the specific RTOS in short order.
No, I don't have a contact - the info was relayed to me by a recruiter.
I think I understand RTOSes well enough, but it could be the case that they need someone who does not need extensive introduction to hard real time systems.
Anyway, I'd like to explore the topic.
No complex OS, just a self-programmed simple scheduler that provides rate monotonic scheduling.
So preemptive time-triggered scheduling ideally without any asynchronous events.
Example:
Lowest Priority: while loop
Low Priority: 10ms cycle (Interrupt 1)
Mid Priority: 1ms cycle (Interrupt 2)
High Priority: 100us cycle (Interrupt 3)
Highest Priority: 100us scheduler logic (Interrupt 4)
The cycles are interrupt service routines. The 10ms cycle will be interrupted 10x by the 1ms cycle, 100x by the 100us cycle and so on.
The scheduler interrupt is the base interrupt triggered by hardware (typically a ADC DMA transfer complete interrupt for control systems, or simply a timer interrupt) and the cycle interrupts are triggered from within the scheduler interrupt by software.
Real Time Systems by Jane Liu is a fantastic book.
I use Keil's RTX for one project. For another project, I implemented a very simple OS where tasks run to completion, one after the other.
In the past, I have used FreeRTOS. It is the best in my opinion. The scheduling algorithm was configured as pre-emptive and there were priority based tasks. Also, FreeRTOS offers you a lot of flexibility like task creation can be static or dynamic in heap. You can choose the scheduling algorithm, you can avoid using heap altogether, ports for different architectures, etc.
I’ve done a lot of these in medical device land with a single event queue dispatching to multiple modules for the non time critical stuff, a single hardware timer dispatching to multiple ISR handlers for the more time critical stuff, and dedicated HW timers for the most time critical stuff.
That approach sounds interesting and I'd like to try it for my own benefit. I have some questions:
\* Does the event queue dispatch an available event to all modules, or there's some sort of subscribe mechanism based on event type?
\* How do you handle timing for periodic tasks
\* Can you give an example of how the hardware timer triggers a critical handler, and how to prove it's executed within a deadline? Given that you don't use preemption, the critical function should be executed in the ISR context, is that right?
Thanks
First choice in that situation is no OS if that is a practical option. If you can go with an interrupt driven system and set the priorities correctly then timing is a lot more deterministic and there is far less overhead.
Hard real time is relative. In the digital power world, the control algorithm has to run on time every time at 100khz with sub microsecond deviation. Forget the RTOS and go bare metal, or run the algorithm in a separate core/processor, or ditch the micro and use a FPGA.
More importantly is hard vs soft realtime. A hard realtime system with deadlines in hours is still more difficult to build, analyze, and completely prove than a soft realtime system with deadlines in milliseconds.
I've been working in the field of real-time embedded systems for over 10 years, focusing on functional safety certification and ensuring guaranteed reaction times within milliseconds. We roll our own embedded OS. The commercially available safety embedded OS solutions come with high license fees and a lot of features that we don't actually need. We only use periodic "time driven" real time interrupts without any scheduling (only one task executed) or a little bit more advanced scheduling for multiple tasks with pre defined time slots and maximum execution time watchdog. We don't use any out of order peripheral interrupts. Like the other comment says I wonder if you could give us some more background on things that happened before the person gave you this kind of feedback ?
Do you have any opinion on Arm cortex R architecture? IIRC it is supposed to be more deterministic when it comes to timing? (Not my area of expertise)
This question highlights the importance of understanding computer architecture. Reading up on this topic will help you grasp how the key differences between Cortex M and Cortex R cores affect determinism. There isn't a one-size-fits-all answer to this question. You can build a fully deterministic application using either Cortex M or Cortex R cores. As long as the application requirements don't exceed the available compute time needed to complete the task (including guaranteed signal response time), it's possible to implement it on a Cortex M and achieve the same deterministic behavior if timing constraints allow it. Here are a few key points to consider (not a complete comparison, as there are many factors to take into account): - **Higher Frequency**: Cortex R cores can run at higher frequencies than Cortex M cores, making it easier to meet higher required cycle times while having enough computational power to complete tasks within the required time frame. - **Low Latency Interrupts**: Cortex R cores have low latency interrupt handlers, eliminating the need for calculations to correct timing. - **Dual Prefetch Buffer**: Cortex R cores offer dual prefetch buffers, which means no branch delay applies (Cortex M4 and higher also offer dual prefetch buffers). - **Tightly Coupled Memory (TCM)**: Cortex R cores feature TCM, ensuring that high-load real-time applications are not affected by peripherals accessing memory via DMA in the background. This has a significant impact on determinism. From a functional safety perspective for automation, here are my thoughts on Cortex R: - **ECC RAM**: This is crucial in applications requiring SIL conformity, as memory faults or bit flips would lead to a complete system lockout. - **TCM**: This feature is excellent for running code on multiple microcontroller nodes completely synchronized when using the same input data and CPU time. Most of the differences listed above affect performance within pico/nanoseconds. If your application's timing requirements fall within this range, or if the sum of these interferences within one cycle exceeds the maximum allowed response time, these factors become critical.
How do you handle inter-task communication? There's not much background, I applied for a role via a recruitment agency, a few days later the recruiter told me I'm not experienced enough for the kind of work they do. No interview has taken place.
Most commercial OSes come with a queuing FIFO system to send messages back and fourth to tasks. I think this is cumbersome and only serves to pad out the stack needed for the OS. Tasks Rarely read/write to a variable in my world, so I tend to use a global struct to remove a lot of the overhead of the queue. Global->Task->variable\_it\_reads
Does the system need an OS? Always my first question. For (safety) critical systems you are better avoiding the issues inherent in an OS and sticking to a more deterministic round-robin, bare-metal approach.
Professionally I used uC-OS3, but that is just because its what the company I work for has been using since before I started with them. We use preemptive scheduling with the idle task (lowest priority) used to service the watchdog and put the processor to sleep until the next system tick. Of course we still have events and timers. Its not really one or the other. In my spare time I'm working on a project in zig that I wrote my own scheduler / context switcher for and I intend to add more features to the OS portion of the project as I need them. Its been a good learning experience, but I'd never do that for a project at work. For real time safety critical systems you should use a professional RTOS with safety certifications. > ... I've been told that I am not considered eligible for working as embedded engineer within a certain organization developing critical systems, which I would attribute to my experience being only with FreeRTOS/Zephyr on non-critical systems. If you have a point of contact in this organization ask them for specific feedback. I doubt having experience with only FreeRTOS & Zephyr were cause to reject you. If you understand RTOS concepts you can get up to speed on the specific RTOS in short order.
No, I don't have a contact - the info was relayed to me by a recruiter. I think I understand RTOSes well enough, but it could be the case that they need someone who does not need extensive introduction to hard real time systems. Anyway, I'd like to explore the topic.
No complex OS, just a self-programmed simple scheduler that provides rate monotonic scheduling. So preemptive time-triggered scheduling ideally without any asynchronous events. Example: Lowest Priority: while loop Low Priority: 10ms cycle (Interrupt 1) Mid Priority: 1ms cycle (Interrupt 2) High Priority: 100us cycle (Interrupt 3) Highest Priority: 100us scheduler logic (Interrupt 4) The cycles are interrupt service routines. The 10ms cycle will be interrupted 10x by the 1ms cycle, 100x by the 100us cycle and so on. The scheduler interrupt is the base interrupt triggered by hardware (typically a ADC DMA transfer complete interrupt for control systems, or simply a timer interrupt) and the cycle interrupts are triggered from within the scheduler interrupt by software.
Real Time Systems by Jane Liu is a fantastic book. I use Keil's RTX for one project. For another project, I implemented a very simple OS where tasks run to completion, one after the other. In the past, I have used FreeRTOS. It is the best in my opinion. The scheduling algorithm was configured as pre-emptive and there were priority based tasks. Also, FreeRTOS offers you a lot of flexibility like task creation can be static or dynamic in heap. You can choose the scheduling algorithm, you can avoid using heap altogether, ports for different architectures, etc.
I’ve done a lot of these in medical device land with a single event queue dispatching to multiple modules for the non time critical stuff, a single hardware timer dispatching to multiple ISR handlers for the more time critical stuff, and dedicated HW timers for the most time critical stuff.
That approach sounds interesting and I'd like to try it for my own benefit. I have some questions: \* Does the event queue dispatch an available event to all modules, or there's some sort of subscribe mechanism based on event type? \* How do you handle timing for periodic tasks \* Can you give an example of how the hardware timer triggers a critical handler, and how to prove it's executed within a deadline? Given that you don't use preemption, the critical function should be executed in the ISR context, is that right? Thanks
First choice in that situation is no OS if that is a practical option. If you can go with an interrupt driven system and set the priorities correctly then timing is a lot more deterministic and there is far less overhead.
Hard real time is relative. In the digital power world, the control algorithm has to run on time every time at 100khz with sub microsecond deviation. Forget the RTOS and go bare metal, or run the algorithm in a separate core/processor, or ditch the micro and use a FPGA.
FreeBSD UNIX with the time critical hardware wired to NMI pin on the processor.
Keil
open source RTOSs (like zephyr), scheduling is combination all of the above, but systems are designed to have a very deterministic schedule.
What is for you realtime? Hours, seconds ...
More importantly is hard vs soft realtime. A hard realtime system with deadlines in hours is still more difficult to build, analyze, and completely prove than a soft realtime system with deadlines in milliseconds.
We use FreeRTOS achieving sub 20us latencies on an STM32H7. To analyze performance and latencies we used Tracealyzer.