IPv6 adoption in exponential growth stage

This is the third year in a row that sees IPv6 adoption growing exponentially, with a factor a bit over 2:1 year-on-year (2011 -> 2012 -> 2013). Since we're in the early deployment stage, this trend can be expected to continue for at least several years (and maybe with an even larger factor), such that by the end of 2015 a 10% IPv6 penetration is most likely a conservative estimation.

 

(source: http://www.google.com/ipv6/statistics.html)

The IPv6 penetration, even in low numbers, is important for this project because IPv6 can be used to do the heavy lifting in the routing ring, such that the more IPv6 connections are deployed, the less the bandwith strain on the routing ring peers.

Merry X-mas!

Cross-platform multi-threaded foundation library

After trying to probe the future for over a year for potential show-stopper problems, a few months ago i basically decided that i gathered enough information to effectively start coding on P2P OS with [what i think it is] a pretty good chance of having covered all the major issues that might stop me dead in my tracks, and today is the day i can proudly announce the first working version of a foundation library that i'll be using for all P2P OS development: say hello to my new shiny libposif-0.9

So why another library? Well, to make the long story short, i had three main reasons for this:

1. i wanted an independent (and free, at least LGPLed) library. Sure enough, there are quite a number of LGPLed libraries out there, but none of them is modular- and minimalistic enough for my taste: what i wanted was a library that has no dependencies on anything else than its host OS' kernel API (and even these dependencies must be fully POSIX-compatible) - or, if it does have some dependencies, those dependencies should be very easy to eliminate
2. i wanted a cross-platform library. Again, there are many cross-platform libraries in the wild, but they come in bloated bundles from which it's very hard to extract only the modules one actually needs for a specific application
3. finally, i wanted a standards-based library (except only for a minimalistic POSIX-compatible OS API, which should itself be encapsulated into a cross-platform wrapper): this means e.g. i can use C++11's std::threads but not pthreads, i can use a [platform-abstraction wrapper for] an "mkdir" command but i cannot use a Recycle Bin API, i can use HTML5 for a GUI (or text consoles for text-only UIs) but not graphical widgets or non-standard multi-media components (whether they are wrapped into a cross-plaform library or not), etc

So now with listing the goals out of our way, here's a brief description of what libposif is all about:

• Messaging-based multi-threaded processing: i called this library module "libposif-mt", and what it does is it allows a program to:
• organize processing into multiple "Tasks", where each task is a collection of one or more execution "Threads". Each task can start any number of threads, where a libposif "Thread" corresponds to an OS thread (i.e. it's a C++11 std::thread)
• each "Thread" groups any number of "Automata", where each automaton is an independent state machine that can send messages ("SendMessage()") to other automata (may the destination automaton be part of the same thread, or in a different thread in the same task, or in another task altogether) and is notified via a callback function ("onMessageReceived()") of incoming messages sent by other automata
And here's a picture of it all:





To explain why i need this kind of functionality, i'll quote from an interminable doc in which i gathered all the various things that need to be implemented:

• each router in a node pings all the other routers in its node every ~1 minute, and it updates its own node image and its live routers list
• if a pinged router does not respond to a ping then another 2 pings are immediately retried at 10s interval, and if 5 successive ping “sequences” (i.e. 1 ping+2 retries) fail on a router, or if a router informs the ping sender that it has left the node, then said router is marked as offline in the ping sender's live router list and it will no further be pinged
• when a router detects 5+/10 routers in its own node as offline, it sends to the server its list of offline routers with a rate limiting scheme; multiple such messages coming from different routers in a node will eventually trigger a node image update on the server
The quote above is by no means intended to shed any light on the inner workings of an algorithm, but rather it's meant to show that the algorithm is completely asynchronous (when/then-based instead of if/then), i.e. it begs for independent inter-connected state machines that simply track some state conditions, exchange messages with one another, and change their state when a given situation occurs - and this is exactly what libposif's "Automaton" does.

• note that the above algorithm's mechanics are very similar in nature to how the data chunks are handled in torrents, i.e. each file in a torrent is processed independently, a file is defined as consisting of blocks which are themselves processed asynchronously, etc, and depending on various [asynchronous] conditions associated with a file, or a block, etc, the torrent client takes a specific course of action

• Portable file system interface: i called this module "libposif-fs", and it's basically implemented using POSIX functions plus several OS-specific commands which are not defined in POSIX (i didn't want to use boost, it's way too bloated a library for my taste, so i'll just wait for the file system functions to be included into std:: before using them). The big deal about this library module is that i tried to make it [reasonably] safe for program development, i.e. i tried to minimize the risks of seeing my "windows" directory vanish because i'm sending an empty path to a "rmdir" function and the likes
  
  • in brief, libposif-fs declares a "Sandbox" class which has to be initialized with a "base path", and any and all file operations are methods of a Sandbox object and are confined to the base path (and its sub-directories) of the Sandbox object that they use; equally important, the base path is thoroughly tested against critical system paths and against a user-definable set of "pathNotAllowed()" set of rules when a Sandbox object is created, such that with a little bit of care (when initializing a Sandbox object's base path) the potential for damaging other applications' files is really slim

• UDP sockets: this one is unsurprisingly called "libposif-udp", and i used Qt's QUdpSocket library for its cross-platform implementation (this is a good example of using a thrid-party cross-platform library without critically relying on it because i'm not using Qt's event loop-based signal/slot mechanism or any other fancy stuff - just calling QUdpSocket's plain-vanilla read/write methods) 

• HTTPQueryServer: part of the "libposif-tcp" module, this component is a minimalist HTTP server intended to be run on the localhost and serve as the backend for HTML5-based GUIs (see the HTML browser-based client/server GUI model description here). In order to allow multiple simultaneous Ajax connections from the browser to multiple server sockets on the localhost, the HTTPQueryServer object implements the CORS specification

• Miscellaneous networking functions: this "libposif-netmisc" module contains a collection of networking functions such as enumerating the local host's IP addresses (IPv4 & IPv6), performing DNS lookup and reverse DNS, etc, and it's implemented using the Qt library (it's just a wrapper over the corresponding QtNetwork functions)

• UPnP client: "libposif-upnp": i just grabbed miniupnp for this, so not much to say about this one since it's as cross-platform a library as it can get

• Firewall controller: part of the "libposif-fwl" module, this is a platform-dependent component implemented as a wrapper object over whatever firewall is installed on the system; for the time being i only wrote a netsh wrapper for plain vanilla windows, but it's all dead simple to extend (just add some extra files and configure the library to point to them)
• just for the purpose of illustration, here's how the library is configured to compile for windows with [a wapper over] windows' netsh.exe's firewall controller:
  

  #define libposif_fwlctrl_h\
    "libposif-fwl.src/netsh-exe/netsh_exe.h"

  #define libposif_fwlctrl_cpp\

    "libposif-fwl.src/netsh-exe/netsh_exe_win.cpp" 

  Now suppose i want to add support for a linux firewall controller to libposif: this will involve writing/grabbing a linux firewall controller, packing the source code in a sub-folder e.g. "iptables-ctrl" of the firewall controller's source modules folder "libposif-fwlctrl.src", and then when i need to compile the library for linux (and use this firewall controller implementation) i'll just need to set the firewall #defines in the library configuration file point to this implementation:
  

  #define libposif_fwlctrl_h\
    "libposif-fwl.src/iptables-ctrl/iptables_ctrl.h"

  #define libposif_fwlctrl_cpp\

    "libposif-fwl.src/iptables-ctrl/iptables_ctrl_lin.cpp"

So, to rise, this is where i stand right now: i have a 20-something-page document where i gathered all the most minute details of the algorithms involved in P2P OS (network policies, client, distributed server, software protection, etc), i now have this libposif foundation library to build upon, so i guess the next big thing should be a glorious post about the first piece of code that will actually do something :)

PS
I'll most likely be expanding this library with new functionality over time, but it's probably not worth it writing a new post each time i'll do this (well, unless it's something that i'll deem spectacular enough to warrant a separate post), so i'll just keep silently updating this post in the background as i'll add new modules and/or features.

PPS
In other news, i completely switched to Qt Creator, and after playing around with it for the last several months (and after quite a number of bugs and idiosyncrasies have been fixed during this time) i can now recommend it for any serious cross-platform standards-based development (there still are a few rough edges to be polished here and there, but it's already usable as it is). Here's a glimpse of my new Qt Creator 3.0 desktop in all its glory:

So good bye Borland Builder, you served me well for over 15 years, but the days of closed source software extortion are pretty much over. Nice to meet you Qt, and have a nice life!