libagents-1.0.0 released

It's been about a year and a half since my last post on the 'libposif' library, and at that time my original intention was to bring together all of the various functions that I will need for the P2P OS algorithms inside a single library; however, after a while i realized that a better option would be to have two separate libraries: namely, the 'libposif' library should be dealing only with the cross-platform portability issues, and a separate library, 'libagents', should deal with the agents-based programming model.

So the 'libagents' library project was born: I wanted a pure C++ implementation of the actor model, and I wanted it released as a stand-alone open source project, with proper documentation and all. Now, after over a year of tweaks and hundreds of pages written and then deleted in the doc, today is the day when libagents-1.0.0 is finally available for download. Grab it, use it, and spice up your programs with some elegant parallel processing :)

PS
Don't be discouraged by the sheer size of the documentation, just take it one page at a time. I spent countless hours trying to make it easy to read, so don't try to cut corners, that ain't gonna cut it: the doc is written with the assumption that it will be read and understood incrementally, from start to finish.

Critical milestone reached: framework-independent cross-platform app development got the green light

It's been about two months since my libposif-0.9 library started to show its first signs of life, and now i have my first libposif-1.0 with [what i believe it is] a stable API. When i first defined the libposif API it wasn't totally clear to me how it will integrate in a message loop-based environment (such as is the case with all modern GUI frameworks, e.g. GTK, Borland, MS, Qt), so during the past couple of months i had to re[de]fine the API specifications and make the required implementation changes.

Well, libposif-1.0 is now here, together with my first fully-functioning libposif-1.0-based GUI application. By far the #1 candidate for a host environment is the Qt framework because it's both LGPL-ed and multi-platform, but by no means does a libposif-based application rely on any of the Qt-specific mechanisms (no signal/slot, event loop, etc dependencies whatsoever): in fact, all the fundamental multi-threading and messaging-related mechanisms available in Qt have have a [somewhat equivalent, or more sophisticated] native C++11 implementation in libposif, and all that is required for having a libposif-based application integrated in any host environment, may it be event loop-based or not, is a sequence of three next-to-trivial steps:

1. Derive a host environment-specific class from a libposif-defined "MessagingInterface" abstract class - e.g. "MyMessagingInterface": this class is the host environment's messaging interface with the application, and it implements [pre-processing and] relaying of application-defined, environment-agnostic messages sent by back and forth between the libposif application's I/O buffer and the native environment's components (windows/widgets/etc)
2. Instantiate a MyMessagingInterface object inside the host environment, e.g. by including it in a GUI application's main window or inside an invisible form, etc
3. Cross-link the native environment's objects (i.e. windows, forms, dialogs, etc) with the MyMessagingInterface object (nothing weird or hi-thech here, just some plain old pointer cross-referencing between the host environment's native objects and the MyMessagingInterface object, say 10 minutes of typing some pointer variable declarations and some pointer assignments).

Sure enough, the list above is just a power point list and nothing more, but the point is that that's really all there is to integrating a libposif-based application into any host environment; once the above procedure has been walked though, the libposif application's output messages will trigger methods in the host environment, and host environment's events will be pushing messages to the libposif application's input buffer.

And because a picture's worth a thousand words, here's how a libposif-based application integrates in a host environment...

...and here's a screenshot of my test application, integrated in a Qt-based GUI:

A few words about the test application above and how it is internally implemented by making use of libposif's features:

1. first of all, it is a proof of concept for an event loop-based environment integration:
• the two "Send to" buttons in the main window actually send a message to the application, and the application sends back the message to the GUI which directs it to the proper window (as specified in the message)
• all the other buttons send messages to the application, which then relays them to the appropriate Automaton for processing
2. tests the threading/automata model, namely:
• each counter is implemented as an Automaton in a separate Tread
  
• each counter Automaton is implemented by having it send a scheduled message targeted to itself, with the schedule specifying that the message is to be actually sent with a specified delay after the SendMessage() method invocation (scheduled messages and targeted messages are examples of features that do not have a Qt equivalent)
• counter 1 in the Main window sends a message to itself with precisely one-second delay, thus making the counter increment once/second
  
• counter 2 in the Main window sends a message to itself with a three second average delay, and with a specified delay dispersion (in %), i.e. the messages will actually be sent with an average, but not precise, 3 second delay (e.g. for a 40% dispersion the successive delays might be e.g. 3, 2, 2, 4, 3, 4... seconds ), which will cause the counter to be incremented on average, but not precisely, every 3 seconds
• the two counters can be started/stopped from their corresponding Start/Stop buttons: each button sends a message to the application, which then relays the message to the corresponding counter Automaton
  
• counter 1 broadcasts a message to all the Automata in the application every time it increments; the libposif method for having an Automaton broadcast a message is BroadcastMessage(), and it is roughly equivalent to Qt's signal()
• the two Reg/Unreg buttons register/un-register a connection between the "tick" messages broadcasted by the counter 1 Automaton and a corresponding "divider Automaton" represented by the yellow counters to the right of said buttons, where each "divider Automaton" is dividing the incoming "tick"s (broadcasted by counter 1) by the number specified in its corresponding drowp-down list (i.e. by 1 and by 2 in the screenshot above); the libposif methods for registering/un-registering an Automaton as a listener to a broadcasting source are AddMessagingRoute(src, dest) / RemoveMessagingRoute(src, dest), and are roughly equivalent to Qt's connect(src, dest) / disconnect(src, dest) methods
3. finally, the Main window's "Show Clock Form" menu item illustrates how a message can be sent by a window (namely the Main window) to another window (namely the Form window) without going through the application's Messaging Interface, i.e. this is an internal Qt message which never leaves the Qt-based GUI

And because talk is cheap, here's a code snippet that illustrates how a counter automaton is actually implemented by having it send a scheduled message back to itself:

• onMessageReceived() is a predefined virtual function of libposif's Automaton abstract class, and it is triggered (by libposif's internals) each time an Automaton object receives a message (this method has to be implemented by each specific automaton that is derived from libposif's Automaton abstract class)
• SendIntercomMessage() sends a scheduled message to a specified Automaton (which can be - and in the code snippet below is - the sender Automaton itself)
• BroadcastMessage() broadcasts a scheduled message to all the automata in the application, and the message will be actually received (and processed) by any/all other Automaton objects that have been set up to listen to the sender Automaton (via AddMessagingRoute())
• SendIOMessage() sends a message to the application's Messaging Interface
• State is the Automaton's [integer] state variable: it is a mandatory component
  of any automaton, declared in the Automaton base class

int MyClockAutomaton::onMessageReceived(

   const message_t& msg, 

   const alphanum_t& sourceThreadId, 

   const alphanum_t& sourceAutomatonId) 

{

   switch (State) {

   case counter_on:

      if (msg==CLOCK_TICK) {

         assert(sourceThreadId==parentThread()->threadId() &&

                sourceAutomatonId==automatonId());

         counter++;

         SendIOMessage((message_t)PRINT_COUNTER<<counter);

         BroadcastMessage(DIVIDER_TICK);

         SendIntercomMessage(CLOCK_TICK,"","",rate); //tick

      }

      if (msg==STARTSTOP_COUNTER) {

         assert(sourceAutomatonId=="" && sourceThreadId=="");

         State=counter_off;

      }

      break;

   case counter_off:

      if (msg==STARTSTOP_COUNTER) {

         assert(sourceAutomatonId=="" && sourceThreadId=="");

         State=counter_on;

         SendIntercomMessage(CLOCK_TICK,"","",rate); //start

      }

      break;

   }

   return 1;

}

Well, all in all it's been a bit long (and occasionally bumpy) a road to reaching this point, but given what i know is needed to implement the P2P OS algorithms, there was simply no way of cutting corners: i needed an asynchronous computing framework to implement autonomous agents that talk to each other, i needed this framework to be exclusively standards-based in order to be truly cross-platform, and i wanted complete control over how my applications will integrate in any host environment in order not to rely on any particular host framework (may it be open-source or not). And libposif-1.0 does just that. So, finally, i'm now ready to start working on P2P OS itself.

PS
It is my intention to release libposif as a stand-alone open-source module, but before doing that i'll have to write at least a brief documentation for its API, and i have no clue when/if i'll find enough energy to do that. Until then, as always, anyone interested just drop me a line.

Cross-platform multi-threaded foundation library

After trying to probe the future for over a year for potential show-stopper problems, a few months ago i basically decided that i gathered enough information to effectively start coding on P2P OS with [what i think it is] a pretty good chance of having covered all the major issues that might stop me dead in my tracks, and today is the day i can proudly announce the first working version of a foundation library that i'll be using for all P2P OS development: say hello to my new shiny libposif-0.9

So why another library? Well, to make the long story short, i had three main reasons for this:

1. i wanted an independent (and free, at least LGPLed) library. Sure enough, there are quite a number of LGPLed libraries out there, but none of them is modular- and minimalistic enough for my taste: what i wanted was a library that has no dependencies on anything else than its host OS' kernel API (and even these dependencies must be fully POSIX-compatible) - or, if it does have some dependencies, those dependencies should be very easy to eliminate
2. i wanted a cross-platform library. Again, there are many cross-platform libraries in the wild, but they come in bloated bundles from which it's very hard to extract only the modules one actually needs for a specific application
3. finally, i wanted a standards-based library (except only for a minimalistic POSIX-compatible OS API, which should itself be encapsulated into a cross-platform wrapper): this means e.g. i can use C++11's std::threads but not pthreads, i can use a [platform-abstraction wrapper for] an "mkdir" command but i cannot use a Recycle Bin API, i can use HTML5 for a GUI (or text consoles for text-only UIs) but not graphical widgets or non-standard multi-media components (whether they are wrapped into a cross-plaform library or not), etc

So now with listing the goals out of our way, here's a brief description of what libposif is all about:

• Messaging-based multi-threaded processing: i called this library module "libposif-mt", and what it does is it allows a program to:
• organize processing into multiple "Tasks", where each task is a collection of one or more execution "Threads". Each task can start any number of threads, where a libposif "Thread" corresponds to an OS thread (i.e. it's a C++11 std::thread)
• each "Thread" groups any number of "Automata", where each automaton is an independent state machine that can send messages ("SendMessage()") to other automata (may the destination automaton be part of the same thread, or in a different thread in the same task, or in another task altogether) and is notified via a callback function ("onMessageReceived()") of incoming messages sent by other automata
And here's a picture of it all:





To explain why i need this kind of functionality, i'll quote from an interminable doc in which i gathered all the various things that need to be implemented:

• each router in a node pings all the other routers in its node every ~1 minute, and it updates its own node image and its live routers list
• if a pinged router does not respond to a ping then another 2 pings are immediately retried at 10s interval, and if 5 successive ping “sequences” (i.e. 1 ping+2 retries) fail on a router, or if a router informs the ping sender that it has left the node, then said router is marked as offline in the ping sender's live router list and it will no further be pinged
• when a router detects 5+/10 routers in its own node as offline, it sends to the server its list of offline routers with a rate limiting scheme; multiple such messages coming from different routers in a node will eventually trigger a node image update on the server
The quote above is by no means intended to shed any light on the inner workings of an algorithm, but rather it's meant to show that the algorithm is completely asynchronous (when/then-based instead of if/then), i.e. it begs for independent inter-connected state machines that simply track some state conditions, exchange messages with one another, and change their state when a given situation occurs - and this is exactly what libposif's "Automaton" does.

• note that the above algorithm's mechanics are very similar in nature to how the data chunks are handled in torrents, i.e. each file in a torrent is processed independently, a file is defined as consisting of blocks which are themselves processed asynchronously, etc, and depending on various [asynchronous] conditions associated with a file, or a block, etc, the torrent client takes a specific course of action

• Portable file system interface: i called this module "libposif-fs", and it's basically implemented using POSIX functions plus several OS-specific commands which are not defined in POSIX (i didn't want to use boost, it's way too bloated a library for my taste, so i'll just wait for the file system functions to be included into std:: before using them). The big deal about this library module is that i tried to make it [reasonably] safe for program development, i.e. i tried to minimize the risks of seeing my "windows" directory vanish because i'm sending an empty path to a "rmdir" function and the likes
  
  • in brief, libposif-fs declares a "Sandbox" class which has to be initialized with a "base path", and any and all file operations are methods of a Sandbox object and are confined to the base path (and its sub-directories) of the Sandbox object that they use; equally important, the base path is thoroughly tested against critical system paths and against a user-definable set of "pathNotAllowed()" set of rules when a Sandbox object is created, such that with a little bit of care (when initializing a Sandbox object's base path) the potential for damaging other applications' files is really slim

• UDP sockets: this one is unsurprisingly called "libposif-udp", and i used Qt's QUdpSocket library for its cross-platform implementation (this is a good example of using a thrid-party cross-platform library without critically relying on it because i'm not using Qt's event loop-based signal/slot mechanism or any other fancy stuff - just calling QUdpSocket's plain-vanilla read/write methods) 

• HTTPQueryServer: part of the "libposif-tcp" module, this component is a minimalist HTTP server intended to be run on the localhost and serve as the backend for HTML5-based GUIs (see the HTML browser-based client/server GUI model description here). In order to allow multiple simultaneous Ajax connections from the browser to multiple server sockets on the localhost, the HTTPQueryServer object implements the CORS specification

• Miscellaneous networking functions: this "libposif-netmisc" module contains a collection of networking functions such as enumerating the local host's IP addresses (IPv4 & IPv6), performing DNS lookup and reverse DNS, etc, and it's implemented using the Qt library (it's just a wrapper over the corresponding QtNetwork functions)

• UPnP client: "libposif-upnp": i just grabbed miniupnp for this, so not much to say about this one since it's as cross-platform a library as it can get

• Firewall controller: part of the "libposif-fwl" module, this is a platform-dependent component implemented as a wrapper object over whatever firewall is installed on the system; for the time being i only wrote a netsh wrapper for plain vanilla windows, but it's all dead simple to extend (just add some extra files and configure the library to point to them)
• just for the purpose of illustration, here's how the library is configured to compile for windows with [a wapper over] windows' netsh.exe's firewall controller:
  

  #define libposif_fwlctrl_h\
    "libposif-fwl.src/netsh-exe/netsh_exe.h"

  #define libposif_fwlctrl_cpp\

    "libposif-fwl.src/netsh-exe/netsh_exe_win.cpp" 

  Now suppose i want to add support for a linux firewall controller to libposif: this will involve writing/grabbing a linux firewall controller, packing the source code in a sub-folder e.g. "iptables-ctrl" of the firewall controller's source modules folder "libposif-fwlctrl.src", and then when i need to compile the library for linux (and use this firewall controller implementation) i'll just need to set the firewall #defines in the library configuration file point to this implementation:
  

  #define libposif_fwlctrl_h\
    "libposif-fwl.src/iptables-ctrl/iptables_ctrl.h"

  #define libposif_fwlctrl_cpp\

    "libposif-fwl.src/iptables-ctrl/iptables_ctrl_lin.cpp"

So, to rise, this is where i stand right now: i have a 20-something-page document where i gathered all the most minute details of the algorithms involved in P2P OS (network policies, client, distributed server, software protection, etc), i now have this libposif foundation library to build upon, so i guess the next big thing should be a glorious post about the first piece of code that will actually do something :)

PS
I'll most likely be expanding this library with new functionality over time, but it's probably not worth it writing a new post each time i'll do this (well, unless it's something that i'll deem spectacular enough to warrant a separate post), so i'll just keep silently updating this post in the background as i'll add new modules and/or features.

PPS
In other news, i completely switched to Qt Creator, and after playing around with it for the last several months (and after quite a number of bugs and idiosyncrasies have been fixed during this time) i can now recommend it for any serious cross-platform standards-based development (there still are a few rough edges to be polished here and there, but it's already usable as it is). Here's a glimpse of my new Qt Creator 3.0 desktop in all its glory:

So good bye Borland Builder, you served me well for over 15 years, but the days of closed source software extortion are pretty much over. Nice to meet you Qt, and have a nice life!

A milestone year in the standardization of computing

The time for staring work on the production-quality P2P OS is getting nearer by the day, so i thought it's high-time to look around a bit and try to get a clear picture of what tools and technologies are out there for building true cross-platform applications nowadays, and much to my delight here's what i found:

1. we now have native threads built right into C++11, so there's no need for third-party libs (with God knows what kinds of portability problems each) any more
2. since the days of HTML4 already, one could build a full-featured GUI right onto a web page and have it rendered in any compliant browser (Wt is a pretty neat showcase for what can be done this way), except only for the multimedia features which were not included in the HTML4 standard; now the W3C is making really quick progress towards baking exactly these capabilities into the HTML5 standard (via the free VP8 format), plus device I/O and persistent storage (all of theses especially driven by WebRTC's needs), such that once work will have been completed on these issues a standard HTML5 web page will be able to deliver a full desktop-like experience right inside a web browser (including smooth 2D graphics editing and animation via SVG and high-quality 3D effects via WebGL)
3. last but not least, Qt is making steady progress towards becoming a usable C++ cross-platform development tool not only for all the major desktop OSes Win/Mac/Lin, but also for iOS, Android, and BB10
   

So at the end of the day, the world we live in looks a bit like this to me:

Back to the future: the good old tried and tested X client/server model anyone?

Now, while things look pretty neat the way they are already, how about going one step further? Namely, consider you decide to write your apps' UI based on a minimalist restriction of what HTML5 has to offer, e.g. you'll only use a limited set of widgets (say buttons, drop-down lists, and check-boxes), you'll use only an [editable] text area for all your text interactions, a canvas and an image renderer for graphics, and a simple file system API (well, this is prolly "too minimalistic" a set of UI functions, but i'm picking it up here just to get my message across); in this case, once your app is written based solely on (such) a subset of functions, what you'll have at the end of the day is a native app which will require a UI renderer with a very limited set of capabilities, such that not only you can use a standard HTML5 renderer on systems where it is available, but you can also build a minimalist UI renderer of your own (in native code) on systems that do not provide an HTML renderer: more specifically, where ever you'll find a POSIX(-like) system it's very likely you'll be able to get a port of WebKit (or even a full-fledged browser) for a pretty decent price (if not for free altogether), but if you are about to build an entirely new system of your own, then porting the Linux kernel (or writing a new fully POSIX-compliant kernel of your own) just in order to get WebKit running would be a titanic work that no ordinary small(ish) company can put up with all by itself.

And this is where the minimalist UI model comes into play: if you can have the GNU toolchain ported on your system (which should be pretty easy stuff, especially if you use an existing processor architecture - e.g. there's the OpenRISC out there for grabbing, and it comes with GNU toolchain support and all right out of the box), then all you'll have to do is to implement your small(ish) UI spec (e.g. in C++) and compile it with your gcc, and there you go, you'll have your apps truly cross-platform, ready to be deployed as-is both on any (minimalist) systems that implement your UI spec, and also on any systems that have an HTML5 browser:

The no-brainer solution: have your apps truly cross-platform-ready with a minimalist UI spec

Well, you tell me if i am wrong, but in my view (and given all of the above) this year may well become the most significant milestone in the evolution of standards-based programming after the standardization of ANSI C back in 1969.

PS
Apparently we'll still have to cope with writing BSD sockets wrappers for a while until/if they'll eventually be included in the C++ stdlib, but quite frankly that's a pretty trivial piece of code (given that we have standardized multi-threading baked inside C++) and not much more then a residual problem nowadays.

WebRTC starts flexing its muscles

During the past two years since i started working on P2P OS there has been some significant progress on the WebRTC project which two years ago looked more like a statement of intent than anything else; and since WebRTC is backed by big players like Google, Mozilla, and Opera (with Microsoft notably missing from this lineup after it flushed $8 billion down the drain last year on Skype and it's now pitifully crying foul and trying to screw things up again), it might eventually turn into a viable P2P solution that could make P2P OS rather redundant. However, while a side-by-side comparison between P2P OS and WebRTC does have some merit, i think the case for which of the two will come on top (in terms of quintessential project goals) is far from being settled at this point in time, mainly because WebRTC pays little to no attention to properly dealing with the very real (and critical) problems that the real-world internet topology of today and tomorrow pose to P2P connectivity: in a nutshell, WebRTC opted for a conservative SIP-like technology wherein it falls back to using a network of dedicated TURN relay servers whenever a direct P2P connection between two nodes cannot be made, with little consideration to the fact that such a relay server network requires some big-pockets "sponsors" that can throw in enough ca$h to keep it up and running (e.g. Viber pumps some $2.5 million a year to keep its rather small-ish network up and running), and i think it's very likely that users will be forced to have/get a Google/Mozilla/Opera/M$/whatever account in order to use the service.

Alternatively, P2P OS aims at creating a self-reliant P2P network which is meant by design to gracefully navigate both the rough waters of the current IPv4 exhaustion and IPv4-to-IPv6 transition, and the promised shiny days of a P2P-friendly IPv6-only world of the decades ahead. Also, the scope of P2P OS is not restricted to point-to-point communication between nodes; instead, its design goal is to provide a generic foundation for content-centric networking (a.k.a. named data) where point-to-point communication is only one of many use case scenarios.

To rise, while i can see WebRTC as a serious potential competitor to P2P OS, i think abandoning P2P OS because of WebRTC would be premature at this point in time.

Qt resurrected, support planed for Android and iOS

This might be really big: http://qt.digia.com/About-us/News/Digias-Completion-Of-Acquisition-Paves-Way-For-Qt-To-Become-Worlds-Leading-Cross-Platform-Development-Framework/

Helsinki, Finland and Santa Clara, US - September 18 2012, Digia (NASDAQ OMX Helsinki: DIG1V) today announced that it has completed the acquisition of the Qt software technologies and Qt business.

Qt already runs on leading desktop, embedded and real-time operating systems, as well as on a number of mobile platforms. Digia has initiated projects to deliver full support for Android and iOS mobile and tablet operating systems within Qt and will present the product roadmap and strategy later in the fall.

All significant existing online Qt information has been transferred to qt.digia.com, which, along with qt-project.org, is now the main information-sharing site for Qt.

Now add to the above the cut-them-chains-and-break-free move of Qt5 to QPA (a nice tour at Dr. Dobb's here), and Qt might just have set itself on course for becoming a reliable and future-proof platform. But then again, there's still allot of work to do to make Qt truly platform agnostic - namely, QPA must become a full platform abstraction layer instead of focusing only on the display server, and there's no clearly stated plan in this direction on Qt's website, so only time will tell...

PS
I can't help wondering if Digia's management really understands what QPA's potential implications are for their business (especially if it is to grow into a complete abstraction layer, multimedia included and all), so until i see QPA fully integrated (and properly documented) in the LGPL-ed version of Qt5 i'll just call myself "cautiously optimistic" on this rather extreme openness move...

Multimedia for world+dog

Google might have its issues and there are definitely quite a few haters out there, but it's still one of the very few large-scale corporations with a fair share of good deeds, the latest of which being the open-sourcing of the VP8 video codec as part of the WebM project. What this means for the garage-based developer (like me) is that anyone can now build top-notch multi-media applications "that just work", without needing to dive into that funky world of cosine transforms and the like. As far as the P2P OS project is concerned, this is heavenly-sent.

A web browser for world+dog

Just found out that WebKit is an integral part of Qt, which essentially means that your average Joe Programmer can embed an HTML5-based GUI in his applications as easy as pie; to me this was huge news. But who in their right minds will bet on Qt after Nokia shook hands with MS and all but abandoned Qt, and there's only one big name left out there - SuSe with the KDE Desktop - actually relying on Qt (but with an option to abandon ship at any time and jump on GTK, wxWidgets, or any of a few others to chose from). Unless a big name (Google?) comes to the rescue...

Another one bytes the dust

I really didn't see this coming: www.itworld.com/open-source/136873/qts-future-bigger-nokia
Qt was the only professional free-software cross-platform C++ development kit out there, supported by a big name with a big promise. So now what?