Whoops

So I learned something recently. There is a big difference between HTTP and HTTPS, and the ESP8266 Wi-Fi module is not made for HTTPS communication. There is no simple or reasonable way for for this to be done as far as I've seen. Apparently it can be done, but the knowledge required to make it work is far beyond me.

In my last I mentioned I would be using The Dark Sky weather API for my forecast data. But on my first try making the API call, the call was unsuccessful. This website uses the secure HTTPS pages unlike OpenWeatherMap. I decided to use Wunderground.com which is a non-secure weather API with all of the weather forecast data I was looking for.

Parsing the data

Before this I only had two points of data to gather, current temperature and wind speed. This revision has 20 different fields of data that I want to parse from the data from wunderground.

The API call to get the current weather and forecast returns nearly 10,000 bytes of data to parse through, way more than the old weather and word APIs.

I set up the code to pick out the data at each point based on the format seen by pasting the API URL into the browser. I was confident I had everything lined up correctly, but I was getting strange results. Sometimes it was perfect, sometimes it was taking incorrect points of data. I took a look at the serial monitor to see what could be causing this.

TCP Packet Size

I never gave it any thought, but with an API call with the ESP8266 would always begin with +IPD,XXXX:. This string was appearing in the middle of the data being sent at different intervals. And since JSON data is dynamic, sometimes the location of this +IPD,1460: would cause problems in the parsing patterns I would follow.

After some research I learned that this +IPD was signifying the start of a new packet of data, with the number being the number of bytes sent in that packet. And the maximum number of bytes that can be sent on TCP in a packet is 1460. So each packet besides the last one sent would begin with +IPD,1460:.

To get around this I would need to keep track of when each packet started, which byte was in what location in the packet, as well as ignoring the +IPD string indicating the new packet. The method of ignoring this interrupting string was to increment a counter on every character read, and once the count reached 1460, I would skip the next 12 characters and reset the counter. After doing this it worked like a charm.

The reason this was never a problem for me before was that I would only ever be receiving one packet of data, and I didn't know the ESP8266 interrupted the string of data with the start of each new packet. Just when I start to think I understand how this thing works, I realize there is so much more that I didn't know or didn't consider.

All that's left now is to make some finishing touches and make another video showing the updated device.