Close
0%
0%

Web Smart (Phone) Screen Blink Bot

Secure web page on smartphone safely drives cheap bot with flashes of light using internal sensors and web services.

Similar projects worth following
"...Let me sum up... No, there is too much" see the detail below. Basically this is about how to avoid all problems with our prior bots and safely use your expensive smartphone, with free online providers to host secure web pages that can gain access to the phones sensors, to control a very low cost robot. The phone is then also able to connect to extensive web services for additional processing.

What if I told you it was possible to SAFELY use ANY SmartPhone as a robot brain?

On any smartphone with a browser, load a freely hosted github.com webpage with Javascript that blinks areas on the screen to send motor commands to an e.g. Arduino. The page can also access web services API's for voice, object recognition, teleoperation, etc... via WiFI or Cellular data.

- Safe: No electrical connections to the phone, so you can't possibly damage it.
- Any SmartPhone: Not dependant on any hardware support; doesn't need USB, audio jack, BLE, anything.
- Secure: No app needed, so no security concerns, no development software required, nothing to learn but Javascript.
100% reliable, no configuration, easy debugging because everything is visible.
- Sensors: You still get access to all the sensors on the phone; camera, accelerometer, compass, GPS, and to the web via WiFI or cellular connection.
- Internet: You can remote control it, or have it log data, pics, etc... It can access online services for object reco, voice APIs, etc... but no for $ service provider is required; it's free.
- Stunningly Cheap: Total additional cost, beyond a tiny circuit or a cheap mcu (Arduino, Pi Pico, etc...), dc motors or servos, battery, whatever else you use for the robot, is pennies. All open source, no required services, no BS.

History: ESP's web servers provide great UI's on SmartPhones

The San Diego Makers Guild has provided literally hundreds of low cost robots to kids at events. These are based on the NodeMCU (ESP32 or 8266) devices, with a couple continuous rotation servos and a battery pack, programmed with a web server accessible via direct WiFi connections, which allowed them to be controlled by a web page on a cell phone or PC. There are many advantages:

- $5 chip, a pair of $5 continuous rotation servos, and a $2 battery pack. All the display, controls, user interface, etc... are on the SmartPhone
- Very clean, responsive, and fun remote control interface, without any app to install; just uses the browser on the phone. This video explains it pretty well. It's from a time when we were using the bots in a combat league. 
- You can serve any static web page, so not only the basic remote control interface, but also your own custom version, settings, and even a simple web based IDE to edit the files on the ESP; it becomes a mobile webdev platform!

The use of the ESP chips to host web pages for smartphones is a great trick but... it's a bit of a trap, in that it's limiting. 

Historical problems: No web services, no access to sensors:

It doesn't allow one to use the phone to contact the internet in general, and worse than that, you can't put the phone ON the robot /and/ use the sensors in the phone from a web page unless the web page is https with a valid certificate. The browsers security won't allow it. 

Which means you either have to write an app, or install a cert on the ESP (nightmare) or connect the ESP to your house WiFI (which is VERY insecure and just a general bad idea) or... you have to just live without the camera, mic, accelerometers, GPS, compass, etc... all the wonderful sensors that the phone can give us. Which is a huge loss, because they can tell the robot what is around it, that it bumped into something, what direction its moving, where it is, etc.. Not to mention sending pictures to google for classification of objects or sending sound to recognize verbal commands, etc...

If not WiFi...
Instead, let's connect the smartphone to the robot some other way so we can still just use a web page, avoid an app, and run on the net. But what ways can we connect? 

- Bluetooth Low Energy is horribly unreliable. Getting every model of phone to pair correctly is a nightmare, and... sending data from a web page to a BLE device isn't supported anyway. And you need a microcontroller that supports it, so it really limits your choices. 
- We could use Bluetooth...

Read more »

Two Photodiodes Detectors.stl

3D Model for bracket to hold the 2 photodiodes.

Standard Tesselated Geometry - 298.52 kB - 10/16/2022 at 14:45

Download

Web2Wire.zip

The test HTML, Javascript and CSS files.

x-zip-compressed - 1.61 kB - 10/16/2022 at 14:33

Download

TestWeb2Wire.ino

Test Nano program to read the sensors.

ino - 1.46 kB - 10/16/2022 at 14:32

Download

TestWeb2Wire.fzz

Fritzing showing connections.

fzz - 32.42 kB - 10/16/2022 at 14:31

Download

  • 1 × Arduino Nano The main microcontroller. Although a Nano is being used, any microcontroller should be able to be used.
  • 1 × Nano IO Expansion Shield This shield makes it easy to connect sensors/actuators to the Nano. Here's a link to one on Amazon: https://www.amazon.com/gp/product/B087ZRMG2B.
  • 2 × Photodiode module This detects the light levels on the screen. Although the original project used photo resistors, photodiodes are faster. You need one for Clock and another one for Data. Here's a link to ones on Amazon: https://www.amazon.com/gp/product/B0B389WHM6. In addition, a 3D printed holder is required to hold the photo diodes in place, the STL model is in the Files section.
  • 6 × Dupont female to female jumper wires Wires to connect photodiodes to Nano shield. See the Fritzing file for connection details.

  • H-Bridge Control with 2 lightblocks

    James Newton02/10/2024 at 20:14 0 comments

    So the prior electronics only allow forward motion. Here is a super quick design for a circuit that uses two blocks, one for "Enable" and another for "Forward / Reverse". The Enable line can't really be analog, but it can be PWMd through a few different levels at least. 

    You can play with it at:
    https://www.tinkercad.com/things/9NsnYvF5ILO-light-block-h-bridge

    by clicking Run Simulation in the upper right corner and then clicking on the opto sensors and dragging the sliders around. The position of the sliders is a bit... tricky... but you can figure it out. 

    One of these days I might actually build it in the real world. I need something like this the play with my twisted string actuators

  • Another Fine Mess...

    James Newton08/10/2023 at 00:53 0 comments

    Here is another example of why I turned to light blocks on a screen to enable IOT control via web browser: 

    "The Goal is a web hosted app (IDE, control panel, configuration manager, etc…), served via a certificate (https) from a public domain, that talks to local (e.g. 192.168.x.x) devices via http(s), ws(s), etc… requests from the javascript in the browser. Turns out, as obvious and simple that sounds, it might not be possible." Full write up at:
    https://docs.google.com/document/d/1BCrlBoenSeVpMbrXfw0HMXn-aOlhhMvPs0W7Pg-RUfo/edit

  • Speeds and Feeds

    James Newton08/07/2023 at 18:29 0 comments

    Based on my own experiments and those of Aram and another friend, it looks like the speed limit on this blinking screen block to photo detector idea is about 200 bits per second. That is using an I2C type system were there are 3 transitions per bit over two block-sensor "wires". 

    Speed demon it aint. 

    But... there are several ways to work around this:

    1. Parallel: Use multiple channels. There is a lot of room on a screen and photo sensors are tiny. With good alignment, it would be easy to run 8 data blocks and one clock over the bottom edge of a phone screen and so xfer byte at a time or 1600 bps. Still not fast, but, not horrible. 

    2. Analog: e.g. shades of gray. Probably more than 50 even. We see this in the super simple robot linked in prior log entries. This is probably what I'll use for my robot design for left and right wheel control, combined with a digital link to control reverse and other devices like a laser line or gripper. So that is just 4 blocks-sensors and should provide very responsive control of the robot. 

    I've never had any illusions that this thing would be fast. I did expect it would be faster than 200 bps, but even with that slow speed, given parallel and analog expansion, and the relatively slow speed of motors and physical devices, for the application of driving a robot, it's perfectly usable. The speeds of data processing and control between the phone sensors and code running on the phone are unaffected. And the data link is still there and is wide and fast. 

  • Simple Simulation of Simple Circuit

    James Newton02/09/2023 at 06:00 0 comments

    A very very simple simulation of the very simplest version of the light controlled motor circuit. One of these on the left and one on the right. The photoresistor, with tape over it to keep it from activating the touch sensors in the screen, is placed facing the block displayed on the cell phone screen. When the block is white, the motor moves. When it's black, the motor stops. 

    You can play with it here:

    https://www.tinkercad.com/things/aQwsvZ6Vvcn-spectacular-fulffy/editel?sharecode=O67i2i5PQV1QD5hZTAP_-gB1w6oQcbMkimeN0p2bH6E


    Limitations:
    1. Can't go backward. (A more complex circuit might do that)
    2. No speed regulation. (But the phone can use the compass / IMU to stay on course)
    3. Very limited power (resistive drop through transistor, meh)

    Note that the batteries shown here are just a sample. AA's or a USB battery are probably a less costly option. 

    KEY POINT: The cell phone controls the motor from a web page (or whatever) which has access to the phone sensors, the internet, javascript, etc... The web page can be served from any free host, like github. The javascript on that page can send sensor data to services, get back complex analysis, directions, etc... You can do voice, image, teleoperation, GPS, etc... With an old smartphone and a few dollars worth of electronics. 

  • More Simple Friction Drive idea

    James Newton02/08/2023 at 05:30 0 comments

    Clarifying the prior post on this idea. The key is that the motor shaft is pressed own against the outside of the wheel by the weight of the cell, motor, frame, etc... The wheel is something with a rubber outside, like a fat rubber band over cardboard or an o-ring between laser cut acrylic disks. The wheel shaft fits in a vertical slot not a hole, so the shaft slides up until the top of the wheel hits the motor shaft. Motor is driven by a circuit that senses light from blocks displayed on the cell phone screen. 

  • Easier with Analog?

    James Newton02/07/2023 at 05:41 0 comments

    A friend sent me a link to a very minimal version of this same idea. An analog DC motor driver looking at the light level on the screen of a smartphone and using that to drive a robot. Around. Honestly, I'm hard pressed to see why that isn't better? 

    https://www.voltpaperscissors.com/diy-smartphone-robot

    The only advantage of using servos (which require a controller) is that the speed regulation is better.

    I'm not crazy about the fact that he is using an app / photos / etc... instead of just using a web page. Also, his remote control depends on his server, but that could easily be changed. I think a local server is much better to start with especially as a gateway to the joy of simple e.g. node.js web servers or ESPs or whatever. Many security issues with an external server and apps. 

    Of course, the analog route doesn't really support controlling other devices, but... shrug... each new device is just a square on the screen and a simple circuit. 

    Also, speed regulation is really a non-issue if the program running in the phone can "learn" and watch it's accelerometers, and compass. 

  • Simple Friction Drive idea

    James Newton02/07/2023 at 05:37 0 comments

    (edit: picture didn't show up last time)

    The idea here is that the motor shaft presses against the outside of the wheel. The wheel is something with a rubber outside, like a fat rubber band over cardboard or an o-ring between laser cut acrylic disks. The shaft actually goes all the way through to the other side (only showing half here). The red block has a slot not a hole, so the shaft slides up until the top of the wheel hits the motor shaft. The motor is connected to the red block, or to a sheet or PCB connected to that block. A PCB behind the motor also holds the light sensor, pointed at the phone screen. 

    https://www.tinkercad.com/things/2JTX6x4Dj0k-dazzling-lappi/edit

  • Update to Setup and Software

    Aram Perez10/16/2022 at 14:44 1 comment

    Hi Folks,

    I've added some components and files to this project that I used to speed up the whole Screen Blink to Microcontroller communications. There might be some remnants from all my testing that are no longer required.

    I changed the "protocol" from I2C to what I'm calling TwoWire. I'm sending 8 bits at a time, MSB first (makes it easier to see what data was send on a scope or logic analyzer), with a small delay between bytes.

    Any comments/suggestions/corrections/money will be appreciated ;-)

    /Aram

  • Swap logic levels for circuit with pullup.

    James Newton09/08/2022 at 18:48 0 comments

    Assuming the use of a photo sensitive resistor between the analog pin and ground, and an internal or external pullup resistor to the power rail, a white screen will cause a low voltage, and a black screen will allow the voltage to rise. So, to make this the default (as it's the simplest circuit possible, consisting of only the sensors plugged into the Arduino header) I've swapped the black and white blocks on the test web page, but I also made it easy to switch back with a couple of "defines" at the start:

      var logic1 = black
      var logic0 = white

    https://github.com/JamesNewton/JamesNewton.github.io/commit/f36116a8cec1beee6d23ae3cc618ed002eca941c

    Next steps (please help if you can?): Update the Arduino code below to decode the I2C start, data, stop and check for reliability of communications. The time required for the serial output to dump all the data into the plotter far exceeds the timing of the actual binks, so it's good to set the analog levels, turn that print off, and only output the digital signal and decoding. 

  • A start on the Arduino code

    James Newton08/27/2022 at 21:43 0 comments

    This code reads the A2D pins from the sensors, tracks the middle of the signal range, chops that into a high or low signal, and displays all that for the Serial Plotter so you can debug the setup.

    The sensors in this case are nothing more than photoresistors using the internal pull up in the Arduino. (Yes, the analog pins in the Arduino CAN have a pull up enabled). The total hardware cost outside the Arduino are a pair of light dependant resistors. (!)

    Here is a sample: The red is the data, with the green showing the cutoff point / AGC (Automatic Gain Control). You can see the slight increase on agc when the signal goes low, and the decrease when it goes high. That provides some hysteresis to avoid "bounce" or multiple edges. The blue is the clock, which also has an AGC, but it isn't shown here. Note the difference in signal levels caused by different positioning of the sensors to the screen. The yellow and purple are the resulting SCL and SDA signals. I've not yet decoded those to retrieve the data. The data is coming from this page:
    https://jamesnewton.github.io/webbot.html
    and the source for that page is
    https://github.com/JamesNewton/JamesNewton.github.io/blob/master/webbot.html
    (not the free secure web page hosting from github and the ability to access the phone camera inside the browser)

    Here is the code, such as it is. Much work remains to be completed, and I will NOT have time for it. I really hope others will pickup and run with this. 

    /*
      ReadAnalogVoltage
    
      Reads an analog input on pin 0, converts it to voltage, and prints the result to the Serial Monitor.
      Graphical representation is available using Serial Plotter (Tools > Serial Plotter menu).
      Attach the center pin of a potentiometer to pin A0, and the outside pins to +5V and ground.
    
      This example code is in the public domain.
    
      https://www.arduino.cc/en/Tutorial/BuiltInExamples/ReadAnalogVoltage
    */
    
    #define SDA_CHANNEL A0
    #define SCL_CHANNEL A1
    #define FILTER 256
    
      float agc0, agc1, hi1, lo1;
      float hysteresis;
      bool scl, oscl, sda, osda;
      bool pause;
      
    
    // the setup routine runs once when you press reset:
    void setup() {
      // initialize serial communication at 9600 bits per second:
      Serial.begin(115200);
      pinMode(A0, INPUT_PULLUP);
      pinMode(A1, INPUT_PULLUP);
      //good starting values for midpoints, highs and lows.
      agc0 = 2.0;
      agc1 = 2.0;
      hi1 = 3.8;
      lo1 = 1.1;
      hysteresis = -0.1;
      scl=sda=oscl=osda=0;
      pause = 1;
    }
    
    // the loop routine runs over and over again forever:
    void loop() {
      int sensorValue0 = analogRead(SCL_CHANNEL); //clock on A1
      float voltage0 = sensorValue0 * (5.0 / 1023.0);
      int sensorValue1 = analogRead(SDA_CHANNEL); //data on A0
      float voltage1 = sensorValue1 * (5.0 / 1023.0);
      scl = (voltage0 > agc0)?1:0;
      if (scl != oscl) { //clock line changed
        agc0 += scl?hysteresis:-hysteresis; //noise suppression
        oscl = scl; //update
      }
      agc0 = ((agc0*FILTER) - agc0 + voltage0)/FILTER;
      //sda = (voltage1 > agc1)?1:0;
      if (voltage1 > agc1) { //data high
        hi1 = ((hi1*FILTER) - hi1 + voltage1)/FILTER;
        sda = 1;
      } else { //data low
        lo1 = ((lo1*FILTER) - lo1 + voltage1)/FILTER;
        sda = 0;
      }
      if (sda != osda) { //data line changed
        //agc1 += sda?hysteresis:-hysteresis; //noise suppression
        hi1 += sda?hysteresis*2:-hysteresis*2; //noise suppression
        osda = sda; //update
      }
      //agc1 = ((agc1*FILTER) - agc1 + voltage1)/FILTER;
      agc1 = (hi1 + lo1)/2;
    
      if (Serial.available() > 0) {
        // read the incoming byte:
        Serial.read(); //clear it but toss it
        pause = pause?0:1;
      }
      if (!pause) {
        Serial.print("sclv:"); 
        Serial.print(voltage0);
        Serial.print(", sdav:"); 
        Serial.print(voltage1);
        Serial.print(", agc1:"); 
        Serial.print(agc1);
        Serial.print(", scl:"); 
        Serial.print(scl);
        Serial.print(", sda:"); 
        Serial.print(sda);
        Serial.print("\n"); 
      }
    }

View all 13 project logs

Enjoy this project?

Share

Discussions

James Newton wrote 07/16/2023 at 04:35 point

I've been looking into twisted string actuators for use in any setup where the cell phone can be used to track an APRIL tag so that force type actuators are being used. Visual servoing.
https://patents.google.com/patent/US9272425B2/en
Given a material for the "string" with just the right properties, it will actually unwind on it's own, which means you don't even need an H-bridge for the actuator; a simple single transistor driver will do it. 
https://docs.google.com/spreadsheets/d/1n7-rFFa2A_Ioa8paRK7CPpTLl38DlTkTA7gaNKXrrYw/edit#gid=0

But... the other option is good old h-bridges and gear drives. I like worm gears. Really good ones can be 3D printed, but the plastic isn't that strong. 
https://www.thingiverse.com/thing:2776688
https://github.com/chrisspen/gears
Metal worms don't cost all that much anyway. 

I have this crazy idea that a metal worm could have a notch cut in it, and then be used to cut the driven gear from a disk of delrin, fiberboard, phenolic or carbon fiber. Something that will chip away but otherwise be very strong. But that isn't needed for a small version. 

  Are you sure? yes | no

James Newton wrote 01/26/2023 at 16:37 point

A friend sent me a link to a very minimal version of this same idea. An analog DC motor driver looking at the light level on the screen of a smartphone and using that to drive a robot. Around. Honestly, I'm hard pressed to see why that isn't better? 
https://www.voltpaperscissors.com/diy-smartphone-robot
The only advantage of using servos (which require a controller) is that the speed regulation is better.

I'm not crazy about the fact that he is using an app / photos / etc... instead of just using a web page. Also, his remote control depends on his server, but that could easily be changed. I think a local server is much better to start with especially as a gateway to the joy of simple e.g. node.js web servers or ESPs or whatever. Many security issues with an external server and apps. 

  Are you sure? yes | no

James Newton wrote 01/26/2023 at 23:56 point

Of course, the analog route doesn't really support controlling other devices, but... shrug... each new device is just a square on the screen and a simple circuit. 

Also, speed regulation is really a non-issue if the program running in the phone can "learn" and watch it's accelerometers, and compass. 

  Are you sure? yes | no

Aram Perez wrote 10/16/2022 at 14:47 point

I've added some files and components to the projects.

  Are you sure? yes | no

Aram Perez wrote 09/12/2022 at 01:00 point

James, I've modified your Web code to use CSS instead of images and hopefully made it faster. I tried to create a new project on stackblitz but I didn't have the time to figure out how. Send me your email so I can send you the zipped files.

  Are you sure? yes | no

James Newton wrote 09/13/2022 at 04:57 point

Sorry, just now seeing this. JamesNewton@MassMind.org

  Are you sure? yes | no

Aram Perez wrote 09/02/2022 at 19:48 point

Hi James,

Here are two options you might consider:

1) If you're set on using full rotation servos, here's a board to which you can connect servos and sensors:  https://www.diymore.cc/collections/esp8266/products/esp8285-esp-m2-wireless-wifi-development-board-micro-usb-ch340-microcontroller-module-dm-strong-for-arduino-esp-m3-nodemcu-lua. It has an ESP8266 along with 2M flash. All the GPIO's have GND, Vin and signal pins. Vin can be 5 - 10 V.

2) If you want to consider using small DC motors, search for "nodemcu motor shield" on Aliexpress. The motor shield also breaks out the GPIO pins with GND, Vin and signal (however, 4 GPIOs are used for driving the motor controller). This is the combo I use for my STEAMbot Robot Kit and would give you 4M flash.

/Aram

  Are you sure? yes | no

James Newton wrote 09/02/2022 at 21:08 point

Thanks. That's a cute board. I'm moving /away/ from the ESP's though. Since I don't need wifi on the robot, or sensors other than the light sensors I'll use an Arduino, and my own servo adapter board. I'll eventually make a little PCB to hold the light sensors in place with cables back to the Arduino so they can be held to the phone. 

  Are you sure? yes | no

Aram Perez wrote 09/04/2022 at 01:03 point

Here’s a shield for a Nano: HiLetgo 5pcs Nano I/O Expansion Sensor Shield for Arduino UNO R1 Nano 3.0 Duemilanove 2009 https://a.co/fQXgGvb. 

  Are you sure? yes | no

James Newton wrote 09/08/2022 at 19:47 point

Again, thanks for listing a nice board for the nano, provides connection to a servo which is of value. It will make it a bit harder to connect the sensors, but it can still be done. e.g. the female headers are easier for just sticking component wires into. 
All in all, this is nice, but what we NEED is development on the decoding of the signals and a standard Arduino is fine for that. And easy to use. And we can add the servos with the existing adapter board or by just taking off the holder on a servo header and plugging in the wires individually. If you want to help, please see the latest log entry and the call for help at the end. 

  Are you sure? yes | no

Aram Perez wrote 09/08/2022 at 20:21 point

Ok, I’ll take a look at the log. However, I have to take care of some business concerning my 88 year old mom in Puerto Rico so it will be a couple of weeks before I can actually do anything. 

  Are you sure? yes | no

Ken Yap wrote 05/19/2022 at 01:15 point

Haha, very cool hack. I'll have to start thinking of other places where a phone can provide data contactless.

  Are you sure? yes | no

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates