ESP8266: Closed source SDK with a wide open community

The Internet of Things (IoT) world is filled with countless microprocessors. Many readers may already be familiar with the Arduino ecosystem. In the same vein, we now will look at another interesting segment of that community: The WiFi-enabled Espressif ESP8266 chip.

The ESP8266 chip is one of the more ideal chips for home-grown IoT development. It is cheap, has built-in 2.4GHz WiFi capabilities, has up to 17 different general purpose I/O (GPIO) pins available, and can take advantage of the extensive libraries available in the larger Arduino community. Many different commercial IoT devices use the ESP8266 chip; some devices, like those provided by Sonoff are known to be reprogrammable with custom firmware, while other off-brand devices are also known to be unlocked as well. The downside to the ESP8266 is that the SDK is closed-source; consisting of headers and binary libraries distributed under a non-free license that forbids use on anything but the ESP8266.

The lack of source code for the SDK is regrettable, but there still is an open-source community around the chip worth understanding and exploring. For readers unfamiliar with the ESP8266 chip, we will start with how it fits into the Arduino ecosystem.

Understanding the layers

When getting started with ESP8266 development, it is helpful to understand the various approaches that can be taken to use the chip, and how it fits into the development landscape of the Arduino project.

Many (if not most) ESP8266 chips come standard with the Espressif AT firmware that provides an AT-command interface over serial to make various TCP/IP requests (some documentation of available AT commands can be found here). In fact, it is not uncommon to use both an Arduino device and an ESP8266 chip together. One example would be an Arduino device controlling sensors and motors, while communicating with the ESP8266 over serial to provide network connectivity. For circumstances where the primary firmware is written for an Arduino device, and the ESP8266 simply provides networking capabilities, there may not be any need to re-program the stock ESP8266 at all.

That said, the Arduino board can often be cut out entirely and replaced using only the ESP8266 chip and its own I/O functionality. In these cases, the stock AT firmware on the ESP8266 is replaced with a custom one that accomplishes the same tasks normally done using an Arduino board. To begin we have the Espressif ESP8266 SDK which, as mentioned, is distributed closed-source. The SDK provides the C APIs (PDF) for the building of custom firmware that can replace the default firmware on the chip.

This SDK, as its name implies, is all that is needed to write and deploy code to an ESP8266 using the lowest-level APIs available. However, most developers do not use the Espressif SDK directly in favor of the Arduino Core for ESP8266 (Arduino Core). This LGPL-licensed code base provides the Espressif SDK in a way that is compatible with the Arduino ecosystem, allowing developers to write Arduino-style code for their ESP8266 devices. In this approach, one would add the ESP8266 board definitions to the Arduino IDE, and from there it can be treated like any other Arduino board. It’s still using bits from the closed SDK, such as TCP/IP and WiFi stacks, but a lot more of the code base is open and when possible kept in step with other boards in the Arduino ecosystem.

What you get with the Arduino Core for ESP8266

Many efforts around the ESP8266 Arduino Core make it worth the effort to use it over the stock SDK, besides simply being as open source as possible. Arduino Core includes C++ classes for a web server (SSL/TLS available), a filesystem implementation, and both a DNS and mDNS server. Client tools for communicating over TCP/IP, including an implementation for making HTTP requests, are also provided.

The Arduino Core enables access to most of the Arduino project’s library of modules. According to the Arduino Core project documentation, most Arduino libraries should work without modification. This includes a VNC client, libraries to implement REST APIs, and MQTT clients. Since the ESP8266 has multiple GPIO pins available, most libraries that interact with other hardware such as temperature sensors or individually-addressable LEDs should also work out of box. These libraries provide the bit-level protocol implementations to communicate with, for example, a temperature sensor — providing the developer with a simple API to access the temperature reading.

Other open-source efforts around ESP8266

Cost and availability of the ESP8266 have made it a favorite in the Arduino community when network connectivity is needed, despite the nature of the chip’s SDK. Those same reasons have also driven efforts to reverse-engineer the closed-source SDK and replace it with open-source solutions.

The ESP Open SDK is the most comprehensive attempt at an open-source solution for the ESP8266. This project combines the closed-source SDK with a variety of open-source projects to provide a complete stand-alone SDK including a toolchain for development. While the project appears to have stagnated in recent years, it is still a viable way to build firmware for those who would prefer as much open source as possible. It is also noteworthy that, due to the stagnation, it seems to still rely on older versions of the closed source SDK for the binary components it needs.

Another interesting open source project trying to demystify the software stack of the ESP8266 is an open-source bootloader named rBoot developed by Richard Burton. The bootloader in the ESP8266 is the first code executed when the device is powered on, and is responsible for a number of pre-application tasks, like switching to a new firmware version after an over-the-air (OTA) update.

rBoot is enabled by some linker-hacking at compile time to replace a function in the SDK. By replacing this function, rBoot is able to trick the SDK into loading rBoot instead of eBoot. As is typically the case, rBoot provides not only confidence in the code executing on the chip, but more features when compared to the stock eBoot bootloader. Key features such as multiple firmware images, GPIO-based firmware selection, seamless fallback if a firmware is corrupt, and more robust firmware checksum-validation are all welcome additions. To use this validation feature of rBoot, Burton has also written his own version of an ESP8266 ROM packaging tool called esptool2. This ROM-creation tool enables the checksum-validation found in rBoot by including the .irom0.text section of the ROM in checksum calculations. For those who would like to learn more, Burton’s blog is a plentiful source of information on both projects.

The linker-hacking aspect of rBoot can make it challenging to use, however. The biggest barrier is that the most common way to build an ESP8266 firmware is to use the Arduino IDE, which doesn’t support rBoot. To use rBoot, a level of comfort with more traditional build tooling such as ld and make is necessary to produce an rBoot-enabled ROM. Once built, you will also need to be familiar with esptool2 to package and then flash that ROM onto the ESP8266 itself.

Getting away from the Arduino IDE is hardly an unwelcome idea in the community, as the shortcomings of the Arduino IDE have not gone unnoticed by seasoned developers. Those who prefer not to be forced into using the Arduino IDE for ESP8266 development will be happy to know about the makeEspArduino project. This project’s goal is to remove the dependency on the Arduino IDE entirely for ESP8266 development, without losing access to the vast array of Arduino-compatible libraries available. Essentially, it is a complicated makefile system that does all the heavy lifting when it comes to making sure everything is compiled and built correctly. It parses the same board definition files used by Arduino IDE when building the project, which is helpful as they are updated frequently. With a little understanding of build tools and makefiles, it is reasonable to modify makeEspArduino to use rBoot instead of the stock SDK bootloader.

Keeping in line with getting away from the Arduino IDE is the Arduino CLI project. This Go-based command-line tool effectively replaces other important bits of the Arduino IDE, such as the library and board managers, entirely allowing the developer to install and use Arduino library dependencies for their projects directly from the shell. When coupled with a project like makeEspArduino, it completely removes the dependency on the Arduino IDE when building ESP8266 projects.

Other considerations and wrapping up

For developers who are looking to use something other than the C++ development framework used in Arduino environments, there are multiple other potential language options available. There is the MicroPython project, Lua provided by NodeMcu, and even, JavaScript for the ESP8266 provided by the Espurino project. All these projects by necessity still require the binary blobs that ship with the official SDK, but all may be useful if there is a desire to use a different language than C++.

With so many language choices and reasonable GPIO abilities, it is unsurprising there are many ESP8266-based projects to be found. For users of Home Assistant smart hub, ESPHome integrates complete with Home Assistant and claims to enable people to build sensors and IoT devices for their hub without touching a single line of code. There are also projects like Tasmota designed to take the difficulty out of designing an ESP8266 firmware for many common devices types.

The bottom line when it comes to building network-aware devices using the ESP8266 chip is that there are a lot of options. From straight use of the official SDK, to the Arduino Core, to another language entirely like Python or JavaScript, the chip is remarkably accessible — especially considering it is fundamentally closed-source. Moreover, the chip comes in many different form factors, each with a different price point and set of capabilities, allowing it to fit into almost any project. These qualities make it a prime candidate for any hacker interested in implementing their own IoT devices. Coupled with an open-source smart hub like Home Assistant, it makes the idea of a (largely) open-source smart home more attainable.