The ESP-IDF framework I’m using for the OpenHelmsman firmware provides a standard sockets interface using LWIP. Originally I was doing basic HTTP parsing myself secure in the knowledge that I only wanted to serve up simple responses to simple requests. After advice that I’d probably end up consumed by edge cases and maintenance for this bare bones server I turned to Mongoose. Mongoose is a lovely networking library that’s small and efficient enough to work in an embedded environment. Especially when your embedded environment is pretty powerful like an ESP32.

Now armed with a working web server the question became how to get the files for the dashboard onto the ESP32 for it to serve. Mongoose has a lot of functions that make it work best with a file system, but for me a file system seemed like overkill. I don’t need it for anything else (logging is on the serial port for example) and the framework doesn’t provide any tools for writing files to such a file system. That means that after programming you’d need to write your own utility to download the files which seems like a pain.

One could also use XXD, though I’m more comfortable with that hack when it’s fewer smaller files and you need more information included anyway such as the resource’s MIME-type. Happily, the framework provides a GCC way of embedding binary data using the linker. Unhappily, the symbol names produced by the linker are based on the file names which change every time the dash is built and even were I to manually change those names in the firmware source after every build, it’s pretty inflexible when I want to add more files, such as an extra picture.

Ultimately, I wrote my own python program that takes a folder, creates the embedding file for the linker, creates a c file where a function receives a URI and returns a pointer to the resource’s start and end in flash. I can then dump the dash build folder in the source folder of the firmware and everything works great. The automatically generated c file emulates a file system structure, automatically fills in the correct symbols for each file, and even adds the MIME-type field the HTTP server will need to return. I loaded the new firmware onto the board excited to see the dash instead of a simple echo of the request I was currently serving.

Unfortunately, that didn’t happen. At first the index page didn’t even load. The bugs were relatively minor and easy to solve (as an example I was using mg_printf to write data to a mongoose buffer, but this obviously caused issues when treating an image as a string. Those things are full of 0 bytes, which would end the data transfer. I felt dumb as soon as I realised that mistake). Pretty soon all the files were being served up correctly, except for the react.js file. It wasn’t that the served file was garbled or there was some issue in the header. Literally nothing was being sent onto the network. I’m sure everyone’s had the problem where you’re hunting a bug that you don’t understand and you’re so focused on making sure your code is correct that you don’t think about other possible causes.

The issue was that react.js is big. Not big as far as js framework files go, but at 300kB way too big for the esp32 with its free heap of 200kB. Mongoose didn’t crash when the malloc call returned null because unlike my terrible parser Mongoose seems to properly handle its errors. The alternatives were to start using HTTP chunking but because of some ambiguities in Mongoose’s documentation on its connection data structure I wasn’t exactly sure how that’d work. And anyway, with only 4MB of flash, it wasn’t like I was going to ever be serving up huge files anyway. As such, I decided that I should see what I could do to cut down the dash size before having to experiment with Mongoose too much.

First of all, none of the files were gzipped, which was an easy gain. The python program now gzips all the files and strips the gz extension. React.js is still 100kB though which can fit in RAM, but probably not that well. Plus, the Autopilot dash is only going to grow over time. Happily, rather than having to switch away from React, I happened upon Preact. Preact is almost a drop in replacement for React which produces far smaller files. I’ve been told that the trade-off for saving that space is a less sophisticated diff algorithm, but the poor little ESP32 probably doesn’t care if a PC processor has to render a bit more than otherwise. I don’t know, I can’t really speak for it I guess. In the end, the gzipped js file was only 10kB. Plenty of space to grow there.

After loading the resulting firmware onto the board I was excited to see the page pop up into my browser straight away. I was also interested in seeing how it looked on my phone, but for some reason there was nothing happening. Turns out that Android doesn’t have support for MDNS. As an aside, I don’t get the lack of interest in MDNS. I think it’s great. Anyway, worked fine as soon as I realised I needed to use the board’s IP instead.

Now that I’ve got this nice dash for viewing information, it’s probably time to get the sensors working on I2C instead of SPI so I have something to look at (SPI issues are a whole separate shit show I’ll go into next time).