Request for Requests

It's kind of early, as the extension isn't useful or usable to anyone at the moment (next week!), however if you want the extension as either an end in itself, or as a component to enable torrenting for your extension, please offer up some request on what you want.

At the moment, it's pretty pointless installing the add-on, but someone might have an idea of how or why they'd use it, so some UI or preferences requests would be applicable, or some IDL request.

I figure if people use the extension, they'll find bugs and that helps me with my own purposes, so I'm giving a bit of my time to fulfill requests to get a bit of help. :-)

There is some possibility of making an IDL for ashughes, and I'd like to work on that with you ASAP, if you're interested. Asaik the IDL would abstract what is needed from a torrent handler. I'm thinking it'd provide a connection state (IE. whether or not satisfactory connection with other peers), torrent state (IE. whether the swarm has all pieces), start/pause/stop methods, open method that takes the contents of a .torrent file, and a piece listener (analogous to nsIStreamlistener, except data comes in out of order).

Upcoming Week

1. Add saving to folder, when used as a content type handler (IE. When you click a .torrent file, you'll be given an option of a folder to save to). This will probably be an inefficient save, made when the torrent has completed downloading. That means, all the download is still in memory, and the storage used by JS will mean that's probably more than twice the size of the torrent itself.
   
2. Separate log and reportError commands, and enable each to be turned off with a global const.
3. Change the current protocol handler (x-http2p://) to x-bt:// (it's shorter)
   
4. Make a map of file types to content types, to allow the x-bt protocol handler to return a content-type when streaming.
   
5. Correctly inform the model of current uploaded bytes and downloaded bytes (these are pieces sent and received), and current sent bytes/received bytes (these are total bytes sent and received).
6. Add to piece manager to make an easier calculation of bytes left (of the pieces left).
7. Use uploaded/downloaded/left to update the tracker.
   
8. Create a torrent health check event, firing on a fixed interval.
   
9. For leeching/seeding behaviour: Will check if average bytes received in the last X seconds is less than Y% of the experienced max average in the last Z second period. If it is, try to resolve by dropping or choking selfish peers (that we are seeding to, and we're interested in, and we've sent more than received, but haven't unchoked us in B seconds), connecting to new peers, or even upping the allowed connections on the server port to encourage fresh connections.
   
10. For seeding behaviour: Will check if we have reached our seeding limit, if not infinite. Check if we're maxed out on peers. If we are, see if there are any lazy peers: connected but not interested; connected, interested, unchoked, but not downloading. Drop them.
11. Add a shutdown to the model, the piece manager, the peer manager, the peer, the server, and the event pump. For the moment, it should disconnect if required, then clear all references except those that would be needed to indicate shutdown state (IE. probably want a _shutdown attribute, if a public function is called and this is set to true, throw an exception). The event pump should refuse any new events - if it's found an event would be useful during shutdown, allow only shutdown events to be added.
12. Fix so all torrents use the same server port. This will likely be very difficult.
13. Send correct IP address to tracker. This will likely be very easy.
    

Second Release

Again, code that is pretty much unusable as is, except for creating a Seed and '.torrent' file.

20090128

• The component source
• The XPI

What You Get
Clicking a torrent will give a choice whether to 'download' the torrent. The download will not be saved to disk. Next time :-)
Clicking Tools > Create Seed allows the creation of a single file torrent, including setting of the tracker. It also provides the choice of saving the '.torrent' file and/or starting to seed immediately. A server socket will be opened, but unfortunately the IP address reported to the tracker will be incorrect - this doesn't matter so much, as trackers like OpenTracker ignore the IP address reported, and use the IP address the report came from.
Still only bothers to open one peer connection, and when that connection dies, doesn't recover. (This was pretty much left as-is, because it is easier to debug one peer connection than many).

Fixes

• Bencoding dictionaries has been corrected. Previously, the keys weren't sorted; this is incorrect behaviour, they needed to be or bdecoding fails in Mainline BT (and probably any other bencoding library)
• Bitfield messages from peers has been corrected. Was reading from the wrong end of the byte, eg. Given a bitfield 00000001, it was interpretted that the peer had piece 0, not piece 7.
• If a peer had only one piece of interest, receiving the unchoked message did not start the download of that piece. This has been corrected, but I think needs some more work.
  

Progress - Seeding Working

Yay. Seeding has been tested, and we can successfully operate as a server.
Additionally, the http-replacement protocol (see tasks from last week, Friday item 11/12) worked, with a bit of hardcoding, and successfully returned an image. I'll put together the XPI and source package for downloading later today.
In summary of progress on last week's tasks, 1-11 and 14 were worked on. I'll continue on the skipped items this week, and draw up new tasks. Expect downloading to file, and storage of seeding data on disk instead of in memory.

Progress - Setup

Testing the seeding action is a bugger. In order to test a seed, the client must already have loaded the torrent (at minimum a single piece, at best random pieces), then a leech must connect and download.
So, in order to make life somewhat easy (funny, it doesn't sound easy), I'm going to make a tool to create torrents.
When it comes to test a seed, I can open a dialog, select a file and specify a tracker. If I want, I will then be given the option of saving the .torrent to disk. Now I will have a seed, and have a .torrent file I can pass to a leech to test with.
I can't think of an easier way, without introducing another bittorrent client into the picture, which is something I'm mostly trying to avoid.
EDIT (Jan 26): To clarify, I'm happily connecting to other clients, I just don't want to run another bittorrent client on the same machine at this time.

Progress - Timeouts

In order to add timed tasks, such as keepalives to maintain connections with peers, I thought I might add timeouts; XPCOM provides nsITimer, and through nsIAppShellService access to the well known setInterval/setTimeout functions via the hidden DOM window.
However, I disliked the idea of running 30 or so timeouts for each torrent, instead I thought I'd re-use the existing event queue by introducing a new method called 'addDelayedEvent(event, delay)' which adds events to an ordered-by-event-time queue. The idea being, at each step this queue is checked, and any items at the front of the queue that should have fired will be added to the normal event queue.
So, now we handle timed events not by many extra timeouts, but by adding to a new ordered queue which feeds into the normal event queue. Somewhat satisfying.

Initial Release

So, here's some code that doesn't do much.

20090122

• The component source
  
• The XPI

What You Get
On clicking a torrent link eg. Ubuntu 8.10 386 iso a prompt is fired. If responding with a Yes, the torrent will start 'downloading'. Downloading at this stage means connecting to the first peer and downloading sequentially until A) there is no more to download B) an error occurs C) peer disconnects.
There is not seeding, there is not error handling, there is not tracker updating, there is not saving or delivery of the download - it stays in memory.
Plans for the next week are here.

Upcoming Week

Sunday Jan 18th - Wednesday Jan 21st : Actual, real holiday.
Thursday Jan 22nd - Saturday 24th: Maybe holiday. If it isn't:

Thursday

1. Analysis of code produced to achieve these goals, in mind to get feedback and to ease maintenance: Logical order; Structure; Commenting (just want to remove anything too embarrassing, there's been too many late nights recently).
2. Release code to here.

Friday

1. Add functions relying on timed event, specifically:- updating the tracker; keep-alive with peer.
2. Buffer on write fails: a write fail should result in a new rewrite attempt in the event queue.
3. Update peers with HAVE messages on successful download of a piece. (Fire an event, let the event handler deal with the distribution of HAVE to peers)
   
4. Start responding to INTERESTED. Fire an event, the handler should check how many are currently unchoked, and if a threshold is not reached, unchoke. Maintain list of INTERESTED & unchoked, and INTERESTED & choked.
   
5. If a REQUEST msg is fired on an unchoked peer, write the request back. After writing it back, examine history. If this is a peer we are actively exchanging with, keep unchoked. If this is a peer we are not actively exchanging with, increment _sent. If _sent is greater than threshold, write choke back, remove from INTERESTED & unchoked, add to end of INTERESTED & choked. Fire event that will allow INTERESTED & choked to be re-examined, and a new peer to become unchoked. Fire this event on peer disconnected too, after cleanup (ie. after peer is removed from INTERESTED & unchoked or INTERESTED & choked, if they're in either)
   
6. If a NOT INTERESTED msg is fired on an unchoked peer, remove from any INTERESTED list it is part of, and then choke.
7. Open the Mozilla server socket.
8. Accept connections. Add timeout to drop connections that have timedout.
9. Listen for handshakes. On Handshake, send Handshake. Send BITFIELD. Then act as normal.
10. Set up tracker on test server.
11. Resolve x-htp2p URLs to multiple .torrents. Treat file in torrent as the data from that URL.
12. Resolve x-htp2p URLs to single .torrent, then try to resolve URL to file in that torrent. Treat that file as the data from that URL.
13. Make limit on PieceManager so that only pieces related to target URLs are downloaded.
14. Make PieceManager talk to return channel, so that as Pieces are verified, they are written out.
    

Saturday

1. Do what wasn't achieved Friday. If it all was, have a beer. Maybe even have a TSD. Congratulations, you're ready for the next supervision meeting.
   

Purpose

This is the obligatory statement of why a blog exists.

I'm writing code, and have a compulsion to explain my pain. I'm doing that here, so as to relieve the desire of doing it on IRC. This may also benefit others who are experiencing the same pain, and happen to phrase it in a Google query that gets such a blog entry.

Secondly, it provides a focal point in my software development that is more communication and exposition friendly. Expect version announcements, and thoughts that may become rants.