ESP32 Integration Case Study
This is a case study of the integration code that enables Nabto Edge to be used on ESP32 boards. The Nabto Edge Embedded SDK gives a suite of functions and data structures to enable using Nabto Edge for communication between embedded devices and client devices such as Android or iOS phones.
This article is a dive into integrating Nabto Edge in an embedded device by using ESP32 as an example. It is recommended you read the Nabto Edge integration full guide before proceeding.
Nabto Edge Embedded SDK is by default platform-agnostic. This means that the SDK is not designed to work with any one specific platform. Instead every platform that is to be supported must provide the integration code (commonly called an integration or a platform layer) to enable the embedded SDK to use platform facilities such as DNS, TCP, UDP and so on.
The project setup
All the ESP32 integration code, plus a few example programs, can be found on the following github link
Let’s first take note of how the project is set up. To this end it may be a good idea to familiarize yourself with how ESP32 projects and their build systems are typically set up, and have some knowhow with cmake. The directory tree is set up in the following way.
edge-esp32/
├─ components/
│ ├─ nabto_device/
│ │ ├─ include/
│ │ ├─ src/
│ │ ├─ CMakeLists.txt
│ │ ├─ Kconfig
│ │ ├─ idf_component.yml
├─ examples/
| ├─ components/
│ ├─ simple_coap/
│ ├─ tcptunnel/
│ ├─ thermostat/
├─ nabto-embedded-sdk/
nabto-embedded-sdk is the embedded SDK mentioned earlier, added to the project as a git submodule. On some platforms you may have the option to use a prebuilt static or shared library of the embedded SDK (or build your own), but on ESP32 we will include the source code directly into the binary that is to be flashed onto the board. As such we have the full source code of the embedded SDK as a dependency.
examples contains a few example projects that show how to use CoAP or tcp tunnelling. It also carries a more feature-rich example in thermostat that can be connected to with one of our mobile apps.
components is the folder we are most interested in. In this folder we can find the nabto_device component, which is where the platform integration code needed to run Nabto Edge on ESP32 lives. Within components/nabto_device you may notice that the setup is actually extremely simple. There are a couple of source code and header files in src and include. Then we have a few loose project files.
Looking at idf_component.yml shows that we use the IDF component manager to pull in dependencies. At the moment we only need the espressif/mdns component to enable using mDNS functionality.
Kconfig is another file commonly seen in ESP32 projects. ESP32 permits compile-time project configuration, and idf.py menuconfig can be used to control and edit this configuration. Kconfig is used to add new configuration options. Opening the file reveals a few things we’ve exposed for configuration, primarily stack sizes for various operations.
Finally CMakeLists.txt is the well known cmake configuration file. ESP32 primarily uses cmake files for specifying sources. The embedded SDK has first-class support for cmake and looking in the file reveals that for ESP32 we mostly need to use cmake variables that are set in nabto-embedded-sdk/nabto_primary_files.cmake (specifies the required core SDK files to be included) and nabto-embedded-sdk/nabto_modules_files.cmake (specifies some optional modules that can make development easier). On top of using files from the embedded SDK, we of course also need this cmake file to include the sources that are in components/nabto_device/src.
Integration
Now we get to the meat of the integration, the source code within the components/nabto_device/src folder. Take note of the following diagram from the integration guide.
By looking at each part that is needed for the integration such as logging, DNS, UDP, TCP etc you can link each part to a corresponding set of files. For example the code related to implementing DNS support are contained in esp32_dns.c and esp32_dns.h. We have a step-by-step walkthrough that goes through showing how one can implement each part on a Linux machine. We will go through the parts in the same order as this walkthrough.
Threads
The first step is to implement the threads interface. At the moment the embedded SDK requires multithreading support through this interface, in the future it is planned to support single-threaded platforms.
Surprisingly in the src folder there is no files related to this thread interface. The ESP-IDF toolchain uses FreeRTOS, so we could in principle have implemented this thread interface using FreeRTOS threads. However, ESP-IDF also supports a POSIX-compliant pthreads API. So we defer the threads interface to an embedded SDK module in nabto-embedded-sdk/src/modules/threads/unix.
If we take a look at nabto_device_threads_unix.c in that module folder, the structure of the integration is simply to include nabto-embedded-sdk/src/api/nabto_device_threads.h and implement the functions specified in that header file. Since the embedded SDK thread interface is already very similar to the POSIX API, the implementation is nothing more than a thin wrapper around the POSIX API.
Platform adapter
The next part is the platform adapter which can be found in platform_modules_esp32.c. This file implements nabto_device_platform_init() and nabto_device_platform_deinit(). These two functions are in charge of initializing and deinitializing the integration layer. This includes things such as running initializing code for the other modules, for example you may see esp32_mdns_start() inside the init function, which comes from esp32_mdns.c in the same folder. Other things you may spot in the code is the initialization of UDP, TCP, DNS and event queue.
Timestamps
Now we get to a ‘real’ and simplest part of the integration, timestamp support. The remaining parts will be similar to implementing timestamps in theory, but with more complexity for the other parts. Timestamps act as a good starting point for understanding how platform integration with the embedded SDK works.
Note that there is no file for timestamps in the ESP32 project. This is because we once again can get away with using standard POSIX facilities. that ESP-IDF supports. You may find the timestamp module under nabto-embedded-sdk/src/modules/timestamp/unix/nm_unix_timestamp.c.
In here we find a static module struct. This struct is defined in nabto-embedded-sdk/src/platform/interfaces/np_timestamp.h. A module struct is meant to be an opaque interface that presents access to a few functions, in the case of timestamps there is only now_ms().
nm_unix_timestamp.c contains the nm_unix_ts_get_impl() function which returns the struct in the same file. That struct contains a function pointer that points to ts_now_ms() in the file. ts_now_ms() uses clock_gettime() to return a timestamp.
If you take a look in platform_modules_esp32.c again, you will see that inside nabto_device_platform_init() is a call to nm_unix_ts_get_impl() to get a pointer to this module struct. That pointer is then passed to nabto_device_integration_set_timestamp_impl() which is the actual function that saves the pointer to the embedded SDK core and uses it to interact with the timestamp interface.
Note that this setup with a struct carrying function pointers as an interface for the embedded SDK to use is prevalent throughout the remaining steps. Once you’ve understood how the timestamp module works, you’re in good position to understand the rest of the interface implementations. For more details you may again consider reading the integration guide.
Event queue
An event queue implementation is required by the embedded SDK. To understand what the event queue is in more detail, consider reading the event queue article under Nabto Edge integration guide. In short it is primarily used to minimize call stack usage, instead prefering to queue up events in the event queue that are then executed with their provided context.
In the esp32_event_queue.c file you can see the module struct and the accompanying functions that initialize and deinitialize the event queue as well as creating and posting events to the queue. This event queue is adapted from nabto-embedded-sdk/src/modules/event_queue module to support configuring stack sizes. You may recall the Kconfig file that exposed options for controlling stack usage in the embedded SDK.
DNS
The DNS interface is in charge of implementing functionality for DNS lookups in the embedded SDK. That is, resolving hostnames to their corresponding IPv4 or IPv6 addresses.
Taking a look in esp32_dns.c we once again see the same setup as the event queue and timestamps. The ESP-IDF toolchain uses the lwIP TCP/IP stack, as such the DNS interface is implemented using lwIP functionality.
lwIP commonly uses a callback-style API for its facilities. In the DNS interface we see this with the use of dns_gethostbyname_addrtype(), a callback-style function that implements a hostname to IP address DNS resolver.
Networking (TCP and UDP)
The “Internet of Things” wouldn’t really be an internet without networking support. As such the embedded SDK requires implementations of a TCP and UDP interface. These interfaces are implemented in the nm_select_* files in the nabto_device/src folder of the ESP32 repository.
This is actually a module from the embedded SDK that has been slightly adapted for ESP32. You may want to compare with the implementation found in nabto-embedded-sdk/src/modules/select_unix. The select_unix network usually uses a thread together with the POSIX select interface. The primary difference is that the thread initialization is changed to support setting the stack size for the thread using ESP32 configuration.
As mentioned earlier, ESP-IDF uses lwIP so this select function is actually implemented through lwIP. In lwIP has an extensive implementation of BSD sockets which is the TCP/IP sockets API used in POSIX, ESP-IDF supports this part of lwIP as well. Other than these differences the TCP and UDP interfaces are implemented largely the same as the select_unix module.
Local IP
This interface has the rather unspectacular job of allowing the embedded SDK to retrieve the board’s local IP addresses on the various networks it may be connected to. Once again since ESP32 supports a large part of the POSIX API, we defer the implementation to nabto-embedded-sdk/src/modules/unix/nm_unix_local_ip.c.
mDNS
Multicast DNS, or mDNS, is required for local network discovery of devices. You may imagine having a remote controller such as a mobile phone connected to the same local network as your ESP32 board, it would then be useful for the phone to be able to easily find the ESP32 and connect to it. mDNS is what allows this to be the case.
You may remember the idf_component.yml file specifying a dependency on espressif/mdns. This dependency is exactly used for the implementation of the mDNS interface. The interface is quite simple and is implemented in esp32_mdns.c, all it really needs to do is to initialize mDNS with esp32_mdns_start() which is called from the platform_modules_esp32.c file as mentioned earlier.
After that it presents a function that allows the embedded SDK to create mDNS services to allow the board to be discovered on local networks. The documentation for the ESP-IDF mDNS dependency explains the API used by the implementation to use mDNS.
Review
As you might’ve seen, a lot of the ESP32 platform integration for Nabto Edge relies on the POSIX API that ESP-IDF gives access to. If you were to implement it on your own, you may have made more considerations such as using lower level facilities like FreeRTOS threads and lwIP Netconn (which is what the BSD sockets use in lwIP) to minimize resource usage.
However it is hopefully clear by now that the daunting task of porting Nabto Edge to an embedded platform does not have to be particularly scary. It is also worth considering that even if you have a target which is missing some of the facilities required for implementing these interfaces, some of the interfaces like mDNS are optional.
Flash and memory usage by Nabto Edge
Now that we’ve studied the interface implementations, we present in this section some measurements to see resource usage by Nabto Edge on the ESP32. We measure the difference with the Wi-Fi Station example from ESP-IDF against our thermostat example found in the examples folder.
Wi-Fi Station measurements
The Wi-Fi Station example has been modified to print out heap information in app_main() the following snippet has been added:
void app_main(void) {
...
...
for (;;) {
vTaskDelay(10000/portTICK_PERIOD_MS);
heap_caps_print_heap_info(MALLOC_CAP_8BIT);
fflush(stdout);
}
}
Flash usage (idf.py size):
Total sizes:
Used static DRAM: 31072 bytes ( 149664 remain, 17.2% used)
.data size: 15088 bytes
.bss size: 15984 bytes
Used static IRAM: 85166 bytes ( 45906 remain, 65.0% used)
.text size: 84139 bytes
.vectors size: 1027 bytes
Used Flash size : 615825 bytes
.text: 488759 bytes
.rodata: 126810 bytes
Total image size: 716079 bytes (.bin may be padded larger)
Heap usage (heap_caps_print_heap_info(MALLOC_CAP_8BIT)):
Heap summary for capabilities 0x00000004:
At 0x3ffae6e0 len 6432 free 4 allocated 4648 min_free 4
largest_free_block 0 alloc_blocks 39 free_blocks 0 total_blocks 39
At 0x3ffb7960 len 165536 free 107844 allocated 55448 min_free 103976
largest_free_block 104448 alloc_blocks 155 free_blocks 5 total_blocks 160
At 0x3ffe0440 len 15072 free 13448 allocated 0 min_free 13448
largest_free_block 13312 alloc_blocks 0 free_blocks 1 total_blocks 1
At 0x3ffe4350 len 113840 free 112216 allocated 0 min_free 112216
largest_free_block 110592 alloc_blocks 0 free_blocks 1 total_blocks 1
Totals:
free 233512 allocated 60096 min_free 229644 largest_free_block 110592
Nabto Edge measurements
Flash usage (idf.py size):
Total sizes:
Used static DRAM: 34016 bytes ( 146720 remain, 18.8% used)
.data size: 15592 bytes
.bss size: 18424 bytes
Used static IRAM: 85878 bytes ( 45194 remain, 65.5% used)
.text size: 84851 bytes
.vectors size: 1027 bytes
Used Flash size : 926669 bytes
.text: 751227 bytes
.rodata: 175186 bytes
Total image size: 1028139 bytes (.bin may be padded larger)
Heap usage (heap_caps_print_heap_info(MALLOC_CAP_8BIT)) when attached to basetation.
Heap summary for capabilities 0x00000004:
At 0x3ffae6e0 len 6432 free 4 allocated 4648 min_free 4
largest_free_block 0 alloc_blocks 39 free_blocks 0 total_blocks 39
At 0x3ffb84e0 len 162592 free 348 allocated 157032 min_free 4
largest_free_block 156 alloc_blocks 897 free_blocks 6 total_blocks 903
At 0x3ffe0440 len 15072 free 10468 allocated 2908 min_free 64
largest_free_block 6400 alloc_blocks 18 free_blocks 5 total_blocks 23
At 0x3ffe4350 len 113840 free 112216 allocated 0 min_free 110516
largest_free_block 110592 alloc_blocks 0 free_blocks 1 total_blocks 1
Totals:
free 123036 allocated 164588 min_free 110588 largest_free_block 110592
Heap usage when attached to basestation and one client is connected.
Heap summary for capabilities 0x00000004:
At 0x3ffae6e0 len 6432 free 4 allocated 4648 min_free 4
largest_free_block 0 alloc_blocks 39 free_blocks 0 total_blocks 39
At 0x3ffb84e0 len 162592 free 88 allocated 157264 min_free 4
largest_free_block 48 alloc_blocks 904 free_blocks 3 total_blocks 907
At 0x3ffe0440 len 15072 free 3432 allocated 9888 min_free 64
largest_free_block 1472 alloc_blocks 32 free_blocks 9 total_blocks 41
At 0x3ffe4350 len 113840 free 95492 allocated 16720 min_free 90392
largest_free_block 94208 alloc_blocks 1 free_blocks 1 total_blocks 2
Totals:
free 99016 allocated 188520 min_free 90464 largest_free_block 94208
Heap usage when attached to basestation and two clients are connected.
Heap summary for capabilities 0x00000004:
At 0x3ffae6e0 len 6432 free 4 allocated 4648 min_free 4
largest_free_block 0 alloc_blocks 39 free_blocks 0 total_blocks 39
At 0x3ffb84e0 len 162592 free 72 allocated 157264 min_free 4
largest_free_block 48 alloc_blocks 908 free_blocks 2 total_blocks 910
At 0x3ffe0440 len 15072 free 1408 allocated 11872 min_free 4
largest_free_block 1120 alloc_blocks 42 free_blocks 8 total_blocks 50
At 0x3ffe4350 len 113840 free 73516 allocated 38664 min_free 65948
largest_free_block 69632 alloc_blocks 9 free_blocks 4 total_blocks 13
Totals:
free 75000 allocated 212448 min_free 65960 largest_free_block 69632