The Real-Time Embedded web site is an informative blog, dedicated to Embedded Linux software engineering. Here you can find many tutorials, how-to's, reviews, training lectures and articles about interesting stuff which I deal with as part of my professional work. The blog is updated from time to time. More information about me and about the blog can be found in the about page.
You can roll down to see the recent posts, browse the categories, archives and links on the left, or use the search box on the right if you're looking for something specific. Click on the RT-Embedded logo to jump back to the home page from any page.
Please support the blog by clicking on the ads, even if you are using ADP 
The ads help me to pay for the domain and storage.
Thanks, and enjoy reading!
Macros are very common and often used in C programming to declare constants, such as strings, addresses, or any other values such as maximum number of elements in an array, and so on. For those who are not familiar with them, Macros are declared by the #define keyword. Macros are also used to define some basic functionality given a set of 1 or more typeless parameters, similarly to an inline function.
There are some benefits by using Macros:
- Code maintainability: Using a macro to define a constant provides an easy and correct method to ensure that if this value needs to be updated; all instances of this value will be automatically updated. Otherwise, the program may behave incorrectly, and even crash.
- Code readability: Specifying a macro name is much more readable and understandable to someone who needs to read the code, than a plain number or address.
There are also some disadvantages:
- Hard to extend: If the macros become complex and a change is required, any errors you introduce may yield vague compilation errors by the compiler, and always, the error line number points to the start of the macro.
- Hard to debug: Debuggers often do not provide clear access and step ability to a code inside macros.
Memory corruption is a scenario where a given buffer (or memory area) is unintentionally modified by an unknown piece of code. This data change causes the rightful users of the buffer to receive bad information which could either modify their behavior or even crash the program. It is usually very hard to find the root cause of memory corruptions because the corruption itself may take place asynchronously to the actual use of the buffer, thus when it actually happens, the crash may occur seconds or minutes later.
In this post I will present one of the ways that may help you catch such corruptions using memory protections.
U-boot is one of the most popular boot-loaders that are used in the industry. It supports plenty of platforms and it's open source. There are many other advantages of using u-boot, however, in this post I will present the concept of u-boot scripts (or macros) which could improve your productivity (and your life…) if you are a programmer or a user of u-boot and do many repeating operations.
We all know that when we work on a project, we need to perform various repeating operations, such as erasing a partition in the flash, loading an image, loading parameters block, or any other repeating operation. These operations usually are composed of several commands which also require us to type addresses in the memory. This operation is inconvenient and error prone. There are some cases where the memory map changes, and the commands which were used are invalid and need to be updated.
The NFS (Network Filesystem) is a very common Linux feature. It enables any Linux machine to mount directories which are not physically available in the machine's hardware, but located somewhere else, and reachable only via a network interface. Mounting an NFS allows us to extend the storage capabilities of the mounting machine beyond its physical storage limitations. The NFS mount can be used to store additional software or data. For embedded devices, NFS is usually used for debugging purposes, mostly because these products must operate without network availability and must contain all required software in their storage device. With limited storage capabilities, NFS is a great way to extend the machine's storage capabilities without changing any hardware. Make sure that you grant access to the IP address of your target to the requested directory.
Continue reading “How to setup an NFS client on the target” '
Today's consumer electronics, networking and communication devices are small and highly sophisticated computers (or Systems-on-Chips; SoCs). Each chip contains numerous peripherals and special features that require the right software and drivers to drive them correctly and efficiently. In this article, I am going to describe in high level the system's hardware components and the system's software components. I will also describe what they do in high level. This article is meant for people who want to get a high level introduction to the Real-Time Embedded systems.
Bitwise operations are widely used in embedded systems, both in assembly code and in C code. Bitwise operations are used for reading and writing hardware registers, enabling or disabling hardware features, setting or masking interrupts, writing values to GPIO pins and many other uses. Bitwise operations are different than the corresponding logical operations. In this article I will cover the basic bitwise operations and provide some macros for the most common bitwise operations and some common practices when using them.
Core dumps are the standard Linux way for post-mortem analysis of crashed applications. Given some preconditions, the core dumps can provide a detailed back trace and shed some light about the last whereabouts of the crashed application.
The core dump itself is an executable file format, and it is generated by the Kernel. The core dump contains a list of memory sections which were accessible by the process, and their memory image. This allows you to analyze the core dump offline, on a host machine, using gdb which was compiled for the appropriate target.
Core dump is not enabled by default in embedded systems mainly due to memory limitations. In this article I will explain what is required in order to enable core dumps support on Embedded Linux/uClibc/Busybox platforms for a specific debug task.
Continue reading “Enabling Core Dumps in Embedded Systems” '
The Open Source concept is not new, and has been studied and researched by professionals and by the academia for quite a while. Open Source software products are the outcome of collaboration and cooperation of individuals programmers from all over the world, who gather in communities. The communities are non-profit organizations that distribute and support their software without any charge. The term Open Source, also referred as Free Software, constitutes the main concept of the Open Source movement, which believes that the user must have the rights to use, modify or redistribute the software as he wishes, while granting the same rights to the users of his derivative work or redistribution. The term “Free” is referred as “Free Speech” and not “Free Beer”.
Nowadays, Open Source software is widely used by for-profit organizations. In this article, I will provide an overview of the most used Open Source Licenses, the rights that they grant, and the obligations they require.
Ever wanted to see (in run-time) which function a program is currently executing, or what is the current calling stack of a program? I'm sure you do. In many cases, it is very helpful to see a back trace of all the calling functions in a program, at any given time. Normally, this is a basic feature when you are using a debugger (either a JTAG based hardware debugger or a software based debugger like gdb). However, in this post I would like to show you that this is also possible, with some preconditions, to be done without any attached debugger. It could be useful to remotely debug units in the field or in the lab, where any extra debug equipment may be unavailable.
Continue reading “Getting a process call-stack using pstack” '
The traditional definition of efficiency has two aspects: speed and size. In most cases, optimizing the first aspect causes a degradation in the other, and it's a matter of balancing between the two, according to the specific needs. Per each embedded system, or even a software module, the appropriate strategy must be balanced between the two. Nowadays, there is a new dimension to this definition; power. In this article, I am going to discuss about the traditional aspects. In the years I've been working as a software engineer, I gained a lot of experience about C code efficiency. I saw how changing a few lines of code makes the difference, either in performance, final size or memory consumption. The examples I'll show here were written in C and tested on an ARM platform. You should expect similar behavior by other processors. I might update this article from time to time with some more tips, so it is recommended to bookmark it and catch up.
Continue reading “Writing Efficient C Code for Embedded Systems” '
The proc files implement the simplest method of communication and data exchange between the Linux kernel or its drivers, and the user space applications or human users. The kernel provides many proc files for setting various settings, and getting plenty of information (see the article about process procs). Each driver or loadable kernel module can add more proc files in the /proc directory, in order to set its specific parameters or to get its specific information. The proc files are not really files, but referred as “pseudo-files”, where all the /proc directory is a “pseudo-filesystem”. The reason for this name convention is the fact that unlike other filesystems, these files do not really exist. Instead, the kernel (or each driver which needs a proc file) registers a file, defines its permissions and implements a read and/or write functions. The read function will be invoked whenever a user space application wishes to read information, so the information that is read is actually generated by the kernel upon request. The same applies for the write function.
PCD (Process Control Daemon), is an open source, light-weight, system level process manager for Embedded Linux based products (Consumer electronics, network devices, and more). With PCD in control, it is guaranteed that the system's boot up time will be reduced, and its reliability, availability and robustness will be enhanced.
Why do you need PCD in your Embedded Linux based product?
With PCD driving your product, you will gain:
- Enhanced control and monitoring on all the processes in the system.
- Simple, yet powerfull means to configure each process and service in the system:
- When to start each process, configure its dependencies.
- What resource or condition to check, per each process, in order to verify its successful startup.
- What recovery action to take when a process fails to start or crashes during its work.
- Reduced system startup time
- Rules are started as soon as their start condition is satisfied.
- No need for non-deterministic delays. Dependencies are well defined and enforced.
- Rules without inter-dependency are started in parallel.
- Enhanced debug capabilities of faulty processes
- Exception handlers and logs provide useful and detailed information about process crashes, such as the process name and id, exception description, date and time, fault address, register dump, map file and more.
- Crash log storage in non-volatile memory for post-mortem/offline analysis.
- Improvement in system stability and robustness.
- Generation of a graphic representation of the system startup sequence for further analysis.
Now that you've optimized your applications (programs) and archives (static libraries), we'll discuss how to optimize your shared libraries. Unlike archives which are used only during link time on the host machine, shared libraries reside on the target's file system, and cannot be reduced using the the same techniques. Furthermore, when you create a shared library, you can not know which functions will be used by the applications and which functions are not used. It is also not trivial to figure out the dependency between the library's functions (which function requires another one inside the library). Therefore, shared libraries always contain the full set of functions. The question that we ask is; how much storage space is wasted for unused code of a shared library?
The tips and information in part 1 are too general and common, and it is highly likely that you've already implemented them. In this article, I will show how we can reduce the size of static libraries (archives) and applications (programs) by specifying advanced compilation flags which utilize the special properties of archives and applications.
In embedded systems, size does matter. Embedded products are usually limited in resources of RAM and storage (usually Flash) and the cost pressure forces you to think about creative ways to reduce the overall size of the binary applications and libraries, without reducing the features and functionality. In the years I've been working in the embedded systems business, I often deal with requirements to reduce the overall size of the application due to system limitation. Therefore, I have a lot of experience in this field which I am going to share with you in this “Size optimization” series. The series will include information about size optimization and reduction, general tips, optimizing applications, static libraries, shared libraries, file systms and the Linux kernel.
One measurement of the quality of your project's code is the amount of outstanding compilation warnings. In my opinion, these compilation warnings could potentially be responsible for a system crash and unexplained behaviour, and are one of the top 5 destructive bugs. If you are a project manager, it is your interest to eliminate them, especially if your deliverable is the actual source code. As a customer, a software deliverable with plenty of compilation warnings appears to be non-professional just by looking at the compilation process. If you are a developer, it is your aim to deliver warning-less code.
Last week, one of the company's customers had a major issue with a mysterious process that is periodically spawned, and when it runs, it allocates a resource and terminates without freeing it. Unfortunately, the allocated resource is a shared memory segment, which is not controlled by the kernel. Unlike dynamic memory allocation which gets cleaned up when a process terminates, this resource kept on leaking until it was completely exhausted. Once all the shared memory was consumed, the system was unable to operate correctly, although the kernel was still alive, and there was plenty of free RAM. This is just a single scenario where you might need to get a clear picture about the running processes in the system. How can we log and monitor the creation, execution and termination of processes in the system? Just continue reading.
Continue reading “Monitoring process creation, execution and termination” '
The Linux kernel, like any other piece of software, can suffer from memory leaks. However, if you are using a recent version of the kernel, or a “commercial version” of it (such as Red-Hat, or MontaVista), the probability for memory leaks which originate from the kernel itself is relatively low (yes, I have high confidence in it). So what else could eat up the memory in the kernel space? There are a few possibilities for this:
- Misuse of standard kernel API.
- Local modifications which were added to the kernel in your project.
- A loadable kernel module.
Continue reading “Resolving memory leaks in the Linux kernel” '
Nowadays, many software products support a mechanism that allows programmers and 3rd party vendors to enhance their features and capabilities without the need to recompile them, or actually, without the need to change any of their existing executables or libraries. This mechanism is usually called “Plugins”. Plugins are very popular today and can be also used to customize a program according to the requirements of your sysetm. In this article, I will present the Linux version of Plugins, how to support them in your application and how to write plugins. I will also provide examples of an application and a matching Plugin.
Continue reading “Extending your application with Plugins” '
Memory leaks is one of the RT Embedded major bugs, as I described in the article about the 5 most destructive bugs. Memory leak bugs usually kill your system slowly and painfully. Depending on the total amount of RAM and the extent of the memory leak, the system will continue to run and perform well for a certain amount of time. It could be hours, days, or weeks. During this time, the amount of free memory is constantly decreased. In Linux based systems, when memory is required and there is no free memory, the kernel will start paging out programs and clearing the page cache. Further memory requests will cause performance impact due to paging out living tasks. Eventually, the kernel will trigger the Out-Of-Memory killer and start killing processes. In this stage, the system may be unusable already. The last stage will cause the kernel to panic and hang/reboot. Many consumer electronics products in the field remain active their whole life, therefore you can't allow any memory leaks to happen, or else the customer will have to power cycle the unit from time to time (this is annoying, isn't it?). Memory leaks are also hard to notice during unit-tests or lab tests because in these scenarios, the unit is rebooted constantly, as part of the development or testing work.
This article covers a few techniques that will help you isolate and resolve the memory leak in your user-space application. An article about kernel memory leaks will be provided as well.
Continue reading “Resolving memory leaks in user-space applications” '
The CPU in embedded systems is an important resource. The CPU type, model and processing power is carefully selected during the project's first steps. A system that its CPU is busy most of the time is not healthy, and a system that its CPU is idle most of the time is wasteful. Assuming we're past the design stage and the system is running, you and your managers will probably want to know how busy is the CPU while performing various tasks. Operations that require extensive CPU processings include cryptographic calculations, handling and routing network traffic, graphic calculations and many other product specific calculations.
This article covers 3 different utilities which will help you measure the CPU load in Linux environment. Their output defers from each other by the levels of report details.
We all make mistakes, and it is only natural. However, if the product is already running in the field, mistakes we do in the code which are not detected on time, can cause:
The error types I am going to present are generic errors, which are not tied to specific architectures. A bug which belongs to these error types may be hidden, hard to find and hard to reproduce. The system may fail randomly or unexpectedly (Referred as the “Voodoo effect”).
We can never guarantee that the software we provide is error free. What can we do in order to minimize the chance for these errors to pop up? Well, just continue to read.
Continue reading “Embedded software 5 most destructive bugs” '
You've probably heard the managers asking questions like; how much free memory does the system have? Why is the amount of free memory so low? Where did the entire RAM go?
This article will provide you the right answers that will make the managers and customers smile, and understand the Linux memory consumption concept.
The standard Linux utilities for memory consumption report are “free” and “top”. The “free” utility displays the amount of free and used RAM, and “top” utility also provides a per-process memory usage breakdown. These utilities may often report that the free amount of memory left in the system is rather lower than you would expect, especially after the system was up and running for a while. You may conclude that there are problems related to memory leaks, or that your available RAM is being used inefficiently or even that your system does not have enough RAM to run correctly. Whilst these issues could be real problems with your system, it doesn't always have to be case.
Linux loadable Kernel Modules (also referred as LKMs, DLMs (dynamically loaded modules), or just Kernel modules) can be viewed as “kernel extensions”. They are programs that run in the Kernel's execution environment and extend its basic functionality.
Some more facts about LKMs:
- Most modules in the Kernel tree provide: device drivers, networking support, filesystem support, and more.
- Modules can be loaded and removed in runtime, without needing to neither recompile nor reboot the system.
- Modules are not user space programs. They can access the Kernel's API directly, but they can cause a system crash as well (A kernel panic).
- A module can overwrite kernel system calls (replace system calls with its own implementation).
- A module can export API's as well.
- Some drivers in the Kernel can be compiled as an Integral mode or module mode.
- A module source code does not have to reside in the Kernel tree. It could reside outside of it as well.
- The Loadable Kernel module extension name is “.ko”.
- The compiled modules reside in the filesystem under the /lib/modules directory.
What is a segmentation fault?
Segmentation fault is the most common error condition where your program tries to access either an invalid memory location, or a memory location it is not allowed to access.
A few examples for this could be:
- Dereferencing a NULL pointer.
- Dereferencing an uninitialized pointer.
- Accessing memory with a wrong alignment.
- Writing to a read only area.
- Writing or reading beyond program allocated resources (buffer overflow).
- Memory corruption/overrun.
Continue reading “Resolving/Debugging user space crashes and segmentation faults” '
What is an Alignment Trap?
The memory of the system is used for two main purposes: Store code, and store data. Regarding code, each 32-bit opcode must be stored in a 32-bit aligned address, meaning, the address of each opcode must divide by 4. A 16-bit processor (or ARM in Thumb mode), can access opcodes only in 16-bit aligned addresses, meaning the address must divide by 2. Storing the code differently in the memory will cause an unpredictable behavior because the CPU will read the opcodes incorrectly.
An alignment trap occurs when the CPU is required to access a 32-bit variable which is not 32-bit aligned (same for 16-bit).
Some examples for alignment of variables
When you define a variable globally, on the stack, or allocate memory for it on the heap, it will be positioned in a memory location according to its size. A 32-bit variable address must divide by 4, a 16-bit variable address must divide by 2 and an 8-bit variable address could be anywhere. The CPU has opcodes to access words, half-words and bytes, accordingly. Data structures which contain variables with various sizes will be padded to make their variables aligned according to their sizes, unless forcing it to skip padding with packed attribute. In this case, the compiler generates code that reads the nearest aligned address and isolates the required value using register shift, logical and and multiple byte access functions. This of course makes the code larger, and therefore, it is always recommended to work with the native-size variables of the CPU (in this case; int or unsigned int) to get the optimal performance, even if we don't need their full range, and avoid packed structures as much as possible (note that there are cases that packed structures must be used, for example, hardware address or network packets). For conclusion, it looks like we are ok, and all kinds of memory accesses should be covered. Alignment traps occur when the CPU tries to perform a memory access on an unaligned address. But if we are covered, why do we still get alignment traps?
The proc filesystem is a pseudo/virtual filesystem which is used as a communication interface with the kernel. It is commonly mounted at /proc and the files and directories which reside there are created in run-time by the kernel, drivers and loadable modules. The directories usually mark an area or an item, and the files, which are usually read-only, provide information about this area or item. Files that allow writing can change the behavior of the kernel or the driver which provided the file.
In this article, I will present most of the files which you can find for each process in the proc filesystem. It might be slightly different in each and every Linux distribution, but most of it is the same. The following text was taken from the proc manual, and I added some more information when I felt necessary. Wherever you see a [pid], you need to replace it with an actual process ID number. To see the information, use the cat command (e.g. cat /proc/225/cmdline).
The list is long because there is a lot of information. Remember that this is only a partial list, which covers only the Process related files. There are more interesting files in the proc filesystem which will be covered in a different article.
Continue reading “Process information in the proc filesystem” '
This article is the first amongst a series of articles about processes in Linux. The first article provides some background about processes, and covers process priority and scheduling policies.
What is a process?
A process is a program, an independent entity, which performs a dedicated task in the system. A process is composed of one or more binary objects which contain function implementations and an entry point function called main( ). Each process has a father, which created it, and can have many children processes which it can create. Each process can belong to a certain group (similar to a family).
Suppose you have a very efficient algorithm which performs a certain action (say, division of floating point number in O(n)), and you want to use this algorithm in 10 different programs in your RT Embedded project. What is the best approach to apply here, in terms of performance and size? Suppose I want to use an algorithm from open source which is packed as a library on a specific program in my project. What is the best approach here?
In this article, I will explain what libraries are in high level, and which approach to take in common usage scenarios.