Scanning has reached 900K and the last steps are excruciatingly slow and far from practical.

I have scripted as much as I could, thanks to basic UNIX/Linux tools, but this barely qualifies as a band-aid so of course, I'm thinking of reviving the idea of #CHOMP and finally giving it some substance. This means I need to refresh the #micro HTTP server in C.

The scanning currently mobilises 3 computers and has been plagued by several interruptions, with painful manual restarts, and constant babysitting to keep all the cores busy. Without the previous optimisations, the scan would have lasted a whole month, I believe. But then, manually checking all the fragments of log files gets ridiculous. So it's time to get #CHOMP into reality.

The idea has been around for a while now (Jan. 2022) so some ideas have already percolated.

• It uses the #micro HTTP server in C without the #HTTaP protocol because it must deal with several "clients" in fair (round-robin) fashion.
• it should work in JSON as well as binary format for convenience, and I must develop a client code
• each client can deal with one to thousands of "workers", and it could be a HTML/JS page or a C program with pthreads. This way, any type of computer or resource can participate in the computation effort. After all the goal is to use SIMD and CUDA for example, so a future client could provide 100K workers one day.
• The protocol is simple, graciously fails, with timeouts for the clients, in case one loses power/connection.
• The server manages the logs, gathers the data, allocates the ranges, makes sure the workers are not cheating, reallocates jobs when a client vanishes...
• This will be useful not only for scanning gPEAC smoothly, but also to scan PEACs exhaustively (among other trivially parallel tasks)

Meanwhile, I'll have reached 1 million soon, and might continue scanning in the background, just for the sake of it and because it's free time, and the computer will not be used for other purposes anyway.