Home > On-Demand Archives > Q&A Sessions >
Live Q&A - Demystifying Embedded: Techniques for Low-level Testing and Debugging
Gillian Minnehan - Watch Now - EOC 2023 - Duration: 30:48
Great presentation, Gillian! One interesting bug I had recently encountered was a stack corruption issue. The issue was caused by the bootloader copying the application image and over-writing the bootloader stack. I spent many hours breaking my head to figure out the issue was in the linker scripts. Do you have any thoughts/tips to catch such issues quicker?
incredibly useful and thorough presentation Thanks
Thank you for the comment! Glad you enjoyed the talk.
Great talk Gillian!
The section about the fault handlers was particularly fascinating..
Can I trouble you to point where to get the segger fault handler code you have shown ..
I d like to experiment more with this approach.. :-)
Fantastic! :-)
Thank you very much Gillian!
I appreciate the example project very much.. It gives a nice starting point.
Great rundown! In the section about toggling GPIO for execution information, you mentioned that one advantage is having finer granularity than that provided by the cycle count register. I understand that this is a result of the fact that, in using the cycle count register, the timing information is only represented as an integer multiple of the clock period. Don't both techniques suffer from the (possibly unknown) amount of time it takes the MCU to execute the instructions to either read from the cycle count register or to set the GPIO register, though? I had thought that it can takes hundreds of ns to update a GPIO output, which might swamp out any improvements in precision.
Nathan, good point. I think an improved statement is that both are reasonably accurate; while logic analyzers and oscilloscopes sample at a higher resolution than the cycle counter can count, there are multiple factors such that we can't definitively state one is more accurate than the other -- it depends on your setup. As you stated, without considering anything else, the logic analyzer should be more accurate; the Saleae is sampling at 500 MHz (2 ns per sample) vs the cycle counter counting at 48 MHz (20.8 ns per clock period). There is a small delay between the execution of the instruction that performs the GPIO toggle and the actual GPIO changing, since the output driver has a slew rate. That is a potential source of delay. There is also some load capacitance on the pin for the Logic Pro 8 (specs here) that I'm using (input impedance 2M Ohm with 10 pF) so the cap takes time to charge/discharge. And then yes, there is the time difference between the end of the execution of the last instruction in the code block under test and the end of the execution of the instruction that either toggles the GPIO or reads the value of the cycle count register. Plus, if we account for the Saleae and MCU clocks not being synced, then it is reasonable that the Saleae might take up to 1 sample clock before it sees the GPIO change (so 2 ns). I believe all of these factors are potentially contributing to the discrepancy shown in the demo where the toggle records a slightly longer execution time than the counter demo. Shoutout to my colleague Joe Trainor for help thinking through this one.
Thanks for that clarification!
Certainly. Thanks for the question!
That's very impressive. A simple technique I use often during a debugging session is to add some temporary global bools or ints so that different code branches can be executed and values changed easily, without having to waste time rebuilding and reflashing. Another technique I've found useful when reading and writing data files that are corrupted (and uncertain if it's the reading, and/or the writing), is to compare memory contents with the file in a hex editor, such as HxD (freeware!) or even write the file contents manually using it to check the reading. HxD has also been useful in locating corruption in the disk/partition structure, e.g. if using FatFS. I'd be interested to know if anyone knows of a tool for reading flash memory directly, as I had a similar problem once when developing the driver code, but no 3rd party way of seeing what's actually written to the device, until I got all the settings right and it just worked.
Hi Simon,
Could you please elaborate on the first technique you mentioned? It sounds interesting, but I'm not quite understand the lifetime of those variables and how (if any) they mutated (via a debugger?).
Sure, something as simple as adding some globals (so they don't get optimised out and can be modified at any time in the watch window of the debugger), then deleted later on. This is totally made-up (sorry it removed all the indentation on submitting):-
bool doSomething = false;
int overrideSleepTime=0;
void fn(int sleepTime)
{
if (doSomething)
{
// run or don't run some extra code that wouldn't normally get called
}
if (overrideSleepTime > 0)
{
sleepTime = overrideSleepTime;
}
HAL_Delay((uint32_t)sleepTime);}
Oh I see, nifty! Thank you for the clarification.
I answered a question at my Q&A about unit testing, and the challenges that embedded developers face with unit testing. I mentioned that there is a great article from Memfault on getting started here: Embedded C/C++ Unit Testing Basics. Embedded Artistry also has a great collection here: Articles with Unit Testing Tag. My 2 cents is that off-target unit testing is very important, but it is important to also know when it makes more sense to test on hardware. It should be rare, especially if you are mostly doing standard, application-level embedded software, but it happens. There are a lot of factors. I've worked on a handful of teams, and they all struck different balances with off-target unit tests, on-target unit tests, and integration tests. I'm always striving to make more of my testing off-target, but a variety of factors pull me to sticking to hardware sometimes. As always, it depends. Maybe a topic for another talk.
There was a question at my Q&A about using VS Code for an IAR project. Someone mentioned there is support for IAR in VS Code via an extension, which I am now remembering I have used before. There is a basic guide from IAR here: Using VS Code for IAR Embedded Workbench. I don't use IAR anymore, but when I first started working, I dabbled with it a little bit and got it working with VS Code. I have some instructions written up on it, but I haven't tried them in a while and I don't think I have an IAR Server License anymore. If you're interested in those instructions anyways, shoot me a message (I'm on LinkedIn or you can get in touch at gillian.minnehan@jhuapl.edu).
Great talk. I had an interesting debugging experience recently.
For context, I'm using an 8-bit, 24MHz AVR64DA64 micro from Microchip. First time working with this family of micro, so maybe this is an obvious feature of the chip to others... I set the chip to be a simple bare-metal round-robin-with-interrupts running at 1kHz and wanted to measure the time band utilization, so I measured a timer register and with a scope. What was strange was that the interrupts were firing normally, but the code was running much slower than it should. After much digging, it turned out to be that the peripheral clock (which was running something like 4MHz) not only controls the timers, ADC, USART, etc, but it also controls the RAM! So even though the main clock was running at 24MHz, the code could only execute as fast as the RAM was running. After I adjusted the peripheral clock to also be 24MHz and rescaled everything else, my time band utilization dropped accordingly.
The tools and techniques described by the presenter really are the gold standard for making sure the processor is doing what it's supposed to be doing!
Matt, that's a fun find. I looked up that data sheet (https://www.microchip.com/en-us/product/AVR64DA64) and yep on page 89 sure enough, RAM uses CLK_PER.
Thanks for the comment! Glad you enjoyed it.
Thank you for a well organized and practical talk.
I tend to work more often on the hardware (fpga) side but many of the same debugging ideas apply there. I learned some new things about registers available in modern embedded processors for help with debug. This is a great slide deck and video link to keep in my back pocket for next time I'm working in embedded SW again.
Dan, thanks for tuning in. I'm glad you learned a few things to help when you get back to embedded software. Feel free to reach out (my LinkedIn is in my speaker profile) if you have additional questions when you are re-reviewing the material.
Nice summary of debugging techniques that new engineers should find useful. Even experienced engineers may not have used all of them. I particularly enjoyed the clock counting one. This could be useful when you don't have a scope or logic analyzer at your disposal.
Glad you enjoyed it Dave. Agreed, clock cycle counting is good option too if you don't have a gpio available to do the toggling. On a custom board, you might have every gpio called for.
One of my fsavorite go-to techniques is the HW breakoint on data access when you know what data is being corrupted (i.e. you have a gremlin that is consistent) but you don't know when it's being modified. This technique can make short work of an otherwise hard-to-discover bug.
Thank you for this survey of techniques -- they'd make an excellent training for incoming junior engineers to study.
-dan'l
Dan, agreed, hardware watchpoints are the best. For anyone reading who hasn't used them before, you can have gdb break whenever the data at a particular address is modified. I have also used that to find a corruption issue (My favorite was a buffer overflow that initially manifested as a semaphore take failing. The sem id was the piece of data getting overwritten accidentally.). You execute them with watch <expression> which could be watch my_struct.item if my_struct.item was getting corrupted. Or, directly use the memory address with watch *0x0FE00000 if data at the address 0x0FE00000 was getting corrupted.
10:31:32 From Keith J : Thanks for the presentation Gillian. It's great to see young engineers like yourself getting involved with the community! 10:31:41 From Jacob Beningo : Reacted to "Thanks for the prese..." with 👍 10:31:45 From Joe Trainor : Reacted to "Thanks for the prese..." with 👍 10:31:46 From Gillian Minnehan (she/her) : Reacted to "Thanks for the prese..." with ❤️ 10:32:35 From Thomas Schaertel : Reacted to "Thanks for the prese..." with 👍 10:32:45 From Naishadh K : Reacted to "Thanks for the prese..." with 👍 10:35:32 From Adam P : In the 'hard fault' section, you mentioned you're using remote GDB debugging with some libraries Segger provided. Does this technique work while debugging locally with JTAG? For context, i'm running locally with a Microchip SAME70 with a Segger debugger. 10:35:42 From maria : Your favorite logic analyzer? 10:35:56 From William Heimsoth : Reacted to "Thanks for the prese..." with 👍 10:36:37 From Isha Bhatt : What are some best practices for unit testing embedded systems projects that may not apply to general software projects (additional things to keep in mind)? 10:37:01 From Nathan O. : In your demos you used VS Code. Do you always go to getting a project setup with cmake / make and use VS Code instead of vendor provided IDEs or professional paid IDEs like Keil / IAR ? 10:37:21 From Levy Gabriel : I've noticed that you love to use vscode extensions (like myself). Apart the ones you used in the talk, is there any other extension you like to use when debugging embedded systems? 10:40:47 From Keith J : what unit test framework do you prefer? 10:40:56 From Julian Krämer : Enjoyed your talk, thank you! Do you have any tips on how to record hard faults during testing for later inspection, e.g. in a remote CI environment? 10:43:07 From Puru Patil : cmake or make which one do you prefer? 10:44:43 From BobF : Any thoughts as to what would be superior to those covered but simply does not yet exist, or known about generally? 10:45:05 From Simon Smith : I still use IAR (which often crashes at the worst possible time). Do you know if it's possible to import an IAR project into VSCode, so that it isn't necessary to create a makefile by hand? 10:46:16 From Nathan O. : Répondre à "I still use IAR (w..." I think IAR has released a VS Code extension recently. You might want to look at it. 10:46:50 From Simon Smith : Reacted to "I think IAR has re..." with 👍 10:47:28 From BobF : Yes, In the slides/talk covered ... beyond these. 10:48:52 From Simon Smith : thanks 10:50:52 From BobF : Admittedly, defect dependent so no one universal solution just a comprehensive suite of tests. 10:52:14 From Adam P : To add to the GPIO idea, it's a great visual when you toggle separate IO pins within various ISR of different priorities. It can be shocking to see how long it takes for lower level ISRs to execute when you have higher priority (and higher frequency) ISRs continuously interrupting the lowers. 10:52:16 From Nathan O. : Have you had experience using tracing tools (like Segger Systemview or Percepio Tracealyzer) and if so which did you prefer ? 10:54:14 From Simon Smith : I've used Systemview, but find it hard to see the woods for the trees in a complex system with many tasks and interrupts. The filtering is all or nothing. 10:54:43 From maria : You said you work mostly with rtos. Do you have any preferences on that? 10:57:25 From Al Anway : Great talk! Any chance of a 2-3-page writeup? 10:57:44 From Leroy : Thanks for the presentation 10:58:15 From Gillian Minnehan (she/her) : Gillian.minnehan@jhuapl.edu 10:58:40 From Charles Miller : Back in the Jurassic,, my CS degree program only included a digital logic course--just one. Did your program include EE courses (and assembly, lower-level material)? Analog scopes are useful! How important is EE in our learning? 10:59:59 From BobF : Laplace ? 11:00:14 From Gillian Minnehan (she/her) : Yes Laplace for sure haha 11:02:23 From Simon Smith : Thanks, appreciated!
Great presentation with ton if useful examples