Signing off, very likely permanently

There is some good news, some bad news, and a killer (or maybe just artificially-induced coma - this remains to be seen) conclusion.

The good news:
There seems to be a way of shouldering one billion users on a P2P network with just some $1,000/mo central server traffic: it's called Kademlia (seminal article, wikipedia, search). Kademlia essentially creates an overlay network of micro-servers which, in turn, sustain the rest of the network users, while a central server is only used to log on to the network. But there's a catch: in order to distribute micro-server services over a P2P network there have to be at least ~1% of the participating peers that can act as micro-servers, i.e. they must:

1. be directly connected to the internet (i.e. they are not behind a NAT or firewall)
2. have relatively stable connections, i.e. they should stay connected tens of minutes after they completed a network operation (e.g. a chat session, a file transfer, etc)

Both of the above conditions are easily met in today's internet topology: (1) is achieved by any peer that has maximum one router and the router is UPnP-enabled, and (2) is achievable by having the P2P application remain running in the background for a certain amount of time after a specific p2p session ends, or by making the sessions themselves last a relatively long time. BTW, the most successful p2p applications currently available - Skype and BitTorrent - both do their best to enforce the above two conditions upon all the nodes on which it they are installed (by remaining online even after you click their 'close' button, and by turning on UPnP by default).

The bad news:
In brief, the bad news is that the good news don't do me much good. Because:

• a) i no longer trust that (1) will be maintained in the future. People increasingly install cascaded routers in their homes, and UPnP does not give any signs whatsoever that it is willing to address this issue (i.e. make a computer directly accessible on the internet when connected through cascaded routers). Furthermore, ISPs can deploy new methods of preventing p2p applications from running at any time (and i don't mean filters, but rather generic methods such as CGNs and firewalls, and i really don't think this is very far fetched)
• b) the new mobile communications paradigm will become an increasingly significant burden that a depleting pool of directly-connected peers will have to deal with (because mobile data plans are always offered through operator NATs - but, even if they weren't, a mobile peer can't be used a a server because of mobile traffic costs). And BTW, i have this feeling that one of the main reasons for which skype was sold was the realization that the pool of "supernodes" (skype's terminology for shamelessly using people's bandwidth and making $8 billions out of this scheme) is depleting and some big-pockets company will eventually need to step in and shoulder the network with dedicated servers of their own (sure, that's just a hunch, but it will be interesting to see if/when that "use UPnP" checkbox will go away from skype's connection settings, cause that'd pretty much say that skype fully transitioned to in-house supernodes)
  
  Update
  It happened: arstechnica.com/business/news/2012/05/skype-replaces-p2p-supernodes-with-linux-boxes-hosted-by-microsoft.ars (and never mind the M$ lady's reply, it's damage control nonsense)

My killer conclusion:
At this point in time i'm incapable of evaluating just how serious the above challenges (1) & (2) are, let alone other unforeseen issues that might creep in; and since implementing Kademlia would require anywhere from 6 moths to one year of hard work, i'm just not ready to plunge into such an effort alone and empty-handed.

So i stop. Very likely for good.

Roadblock

Mea culpa for not updating this blog in quite a while, but it's not because i slowed down work on it, but rather, much worse, i've hit a roadblock which is pretty darn serious. In brief, when trying to write the algorithm that would enable a p2pOS client to act as a relay for its plugins (i.e. to "connect the crossed red lines" that i talked about in a previous post), i also had to try to define the low-level API that the plugins will be using to connect to one another (by means of their associated p2pOS clients that will be acting as the relays, one relay at each end of a P2P connection). And as one thing led to another, i eventually reached the conclusion that what i need to do first is to establish exactly how a peer joins the network and connects to another peer. And this is where things got really, really ugly.

Without entering into too many technical details, the important point here is that in order to have two peers connect to each other, they have to go through an initial "handshaking phase" during which the two peers learn some essential things about one another (e.g. their IP addresses, what kind of router(s)/firewall(s) they are behind, etc), and this handshaking phase has to be negotiated through a dedicated handshaking server. Well,  i can try to hide behind all sorts of technical arguments, but that fact of the matter is that ever since i started this project i never tried to calculate exactly how much traffic such a central handshaking server would require for a large P2P network (i'm talking about 100,000,000...1,000,000,000 users being online), only to find out now that the numbers are astronomical: namely, we're talking about thousands, or even tens of thousands, of terrabytes/month, which translates into a handshaking server operating cost somewhere in the hundreds of thousand, maybe millions, of dollars a month. This in turn means that hosting such a server is not something that just about any punk can do in his basement, which in turn means a large company would be required to finance the network operation. Or, in layman's terms: the network can never be truly open, no matter what license will be covering this project, no matter what verbal commitments a company would make, etc. And since the (maybe only) non-negotiable objective of this project is to create a truly open P2P platform, well... you guessed it: i'm stuck.

But, as i said at the beginning of this post, all this mess doesn't mean i gave up on the project; in fact, because it seems increasingly likely that some sort of distributed handshaking algorithm will be necessary, i made quite a few tweaks in the program in order to reduce the traffic between connected peers (e.g. i managed to reduce the P2P keep-alive traffic by about an order of magnitude by detecting the peers' routers' port timeouts and only send keep-alive messages at the required rate), i refined the router classes such that over 90% of the routers models can now act as relays, and i introduced an algorithm that detects if a peer is directly connected to the internet (i.e. public IP or UPnP) such that it can serve as handshaking server in the network. This is how the new "Settings" panel looks like now:

So what i'm doing right now is study what other smart-@$$ P2P projects have done (e.g. Gnutella, Freenet, etc), i'm trying to learn about various DHT approaches (there's a very nice tutorial talking about the basics here), etc, and i'll see if i'll be able to come up with a solution. Keep ya fingers crossed for me, it's in the world's best interest and sh*t :)

PS
Here's how a p2pOS-based P2P session is established through NAT routers with the help of a handshaking server: the blue messages are P2P messages, while the green messages are relayed via the handshaking server (once the handshaking phase is completed, all messages from one peer go directly to the other, i.e. they are P2P messages):