Info - HowTo write to specific location in a file using XPCOM

I'm writing torrents to disk, so a lot data is written to files out-of-order. During setup, I pre-allocate the file space by opening a file and moving to the position the last byte would be, using the nsISeekableStream interface. This technique I picked up from DownThemAll's manager class.
When I receive a piece of a file, I need to put it in the right place. What I was doing was moving to the right place, then writing:

seekstream.seek(0x00, offset);
stream.write(data, data.length);

This failed, somehow I was overwriting/truncating what was in the file. The solution was to first move to the end of the file, then move to the offset.

function insertInFile(filename, offset, data) {
var file = Components.classes["@mozilla.org/file/local;1"]
           .createInstance(Components.interfaces.nsILocalFile);
file.initWithPath( filename );
var stream = Components.classes["@mozilla.org/network/file-output-stream;1"]
            .createInstance(Components.interfaces.nsIFileOutputStream);
stream.init(file, 0x04 | 0x08, 0664, 0); // readwrite | create
var seekstream = stream.QueryInterface(Components.interfaces.nsISeekableStream);
seekstream.seek(0x02, 0); // move to end of file
seekstream.seek(0x01, offset - seekstream.tell() ); // move to where we actually want to write
stream.write(data, data.length);
stream.close();
}

I'll leave with a note that in my tests I've been seeing a byte with the value 0x0A at the end of my files. If I find that's to do with something I've done here, I'll try to remember to update this post. Just watch out for it if you're using the above method.

Upcoming later today - 5th Release of Bittorrent Firefox Extension with fixes Not happening. Latest version has a critical bug with reconnecting to it's own server.

Upcoming Week

This is highly optimistic, as I can't get this much done. The issues that relate/contribute to my research have priority, which is pretty much the first point, maybe the last point, and a number of other items that no-one would be interested in, so I don't blog. Feedback welcome.

• Log Socket Layer to Disk: need to record communications with peers for later scrutiny. Particularly when there's many peers, it's hard to decipher what problems have occurred.
  
• Implement Pause: pause should put the torrent into a holding pattern: no more connections should be accepted or sought. Tracker should be informed downloading has stopped. Existing connections should be choked. Should pass "not interested" to all peers for the moment. If disconnects occur, that's okay. Unsure about keep-alives.
• Implement Memory on Disk: create an abstraction layer, so that a block can be resolved to part (or parts) of the destination files for saving and uploading. This is so we don't need to store the whole torrent in memory at one time, instead we save what we receive as soon as it's confirmed.
  
• Implement Memorising Torrents between Sessions: so we can close Firefox and resume later.
• Implement Resuming from File Set: if we find the target files already in the destination folder, let's see if we can complete them against the torrent's description.
• Implement Limiting Which Files Downloaded: this is partially done already, due to my other goals. Just needs a user interface.
  
• One Server handles all incoming connections: Still not done.
  

Info - History for Firefox and Bittorrent

So, if you want to see integration between Firefox or Songbird and torrents, look in fear. Here's the history that current development can look back on.
Plug for current development efforts: Ashughes on a Songbird bittorrent extension leveraging libtorrent. I should plug my own efforts, but really, has anyone installed it for testing? I'm looking at enabling early writing to disk this week, which should allow large ISO size torrents without a significant memory drain, but there really needs to be more testing or such a size may never download due to other issues. To that end, the next release (Friday?) will have the ability to log the history with each peer, so we can examine what went okay, and what went bad. For now, tests on torrents at mininova.org's featured content have worked for myself up to 20 or so MB (tested on OS X & Ubuntu 8.10, FF 3); I haven't attempted higher, as I don't actually have any need of the content.

Moztorrent
http://moztorrent.mozdev.org/
Last update: was there a first update? Probably the first Bittorrent/Mozilla attempt. Had screenshots, and mockups appear to use download manager. Had a Mailing list (I can't see it ever being used by the authors). No source, so development dead-end, and no updates since archive.org started watching it in December 2004.

FirePuddle
http://firepuddle.mozdev.org/
Thanks to humph I've recently found this one from 2005. Result of a Summer of Code project, this extension worked with Firefox 1.5, but has a list of bugs that make it quite brittle. Lack of support for multiple file torrents would be the killer bug, as that's too common to ignore. (For bugs, see http://firepuddle.mozdev.org/installation.html)

Torrential
http://torrential.mozdev.org/
There is nothing here. The source committed is the source of the website.

Firestorm
http://firestorm.mozdev.org/
Last update: 2007 May 22 (website) May 23 (mailing list). Jump started by leveraging Mainline (Bram Cohen's original client) via PyXPCOM. Has Screenshots, and looks like seeding support. Looks like fair information feedback, and appears to have had right click support on torrent links. Looks like preferences exist (though probably not enabled). Source available (browse, download instructions) And a mailing list: Mailing list. So what's bad? No XPI (which means it can't be installed) and no updates in over year. See the last words on the mailing list - which, thankfully, were about the release of the source code.

Bitfox
http://code.google.com/p/bitfox
I wonder if this will be picked up, or if there is enough to pick up. Some C source files.

Foxtorrent
http://www.foxtorrent.com/ (Dead, blog still up)
The standard by which all other FireFox Bittorrent extensions should be measured. Reached maturity, and survived for some time. The killer is the disappearance of the Red Swoosh app which this extension relied on, which as far as I can tell, was never open source software. So this is probably a development dead end, as the only source is javascript that links to the Red Swoosh backend via http.
Hit 1.0 on 26th April 2007.
You can install a reviewed addon v1.13: https://addons.mozilla.org/en-US/firefox/addon/4844 (5th June, 2007)
Install a not-reviewed 2.0: http://foxtorrent.googlecode.com/svn-history/r236/trunk/foxtorrent/ (6 August, 2008 maybe)
Info on API for Red Swoosh: http://code.google.com/p/foxtorrent/wiki/RedSwooshTorrentAPIDocumentation

FireTorrent
http://www.wyzo.com/firetorrent/
According to TechCrunch and TorrentFreak, this Wyzo/Firefox extension is developed using P2P Innovative tech. Well, okay, that was in 2007, in 2008 TorrentFreak also reports it's using libtorrent. How can we tell? It has the appearance of open source, but damned if I can find the bittorrent source - and the only source code is a single snapshot. The source provided is either dated 26 March 2007 (based on the contents of the source zip) or 30 March 2007 (the log date on the subversion repository).

AllPeers
http://developer.allpeers.com/
Honestly, I know too little about AllPeers, but it looks the most promising for open source software. The company is gone, but the developer site seems to still exist. The site reports the server is closed source, but the client is open, though I don't know what that means for ordinary torrents. I think digging through this code is where I'll be spending some spare time.

Early Fourth Release - Bittorrent for Firefox

Yay. Finally, a user interface for managing torrents: you can list torrents, view progress & file manifests. Feedback on using the UI would be handy. I'm looking at improving the ability of logging interactions with peers for future debugging, so next week's release I'm hoping for feedback on peer errors.

20090207

• The component source
• The XPI

What You Get

• Clicking a torrent gives you the option of saving it, passing on to the default download manager dialog, or canceling.
• Seeding/creating a single file torrent from the Tools menu. Provides option of saving .torrent files, and adding a web seed (Info on web seeds).
  
• Default seeding until reaching DL/UL ratio of 1/1.5, or no limit if creating a seed.
• Tools > List Torrents lists torrents.
  
• Have the option of pausing - but it's only cosmetic.
• Have the option of cancelling a torrent. Should work, testing desired.
  
• The first torrent started launches a server on port 6881. The others attempt to, but will fail.
• See communications via the error console. It gets very noisy.
  

Fixes

• Supports requests over piece boundaries.
• If there is no need to choke, it won't. Previously, any leacher was choked after a certain amount of downloading, and then unchoked almost immediately if no other peers were pending.
  

Early Third Release - Bittorrent for Firefox

Yay. The Bittorrent extension for Firefox is almost suitable for users. You can use it! Though you'll have to quit Firefox to cancel/pause/stop torrents. Thus far this hasn't caused problems for me, as seeding has been pretty quiet. Probably don't want to do it on huge torrents, as the torrent is still kept in memory. This release is a day earlier than planned.

20090203

• The component source
• The XPI

What You Get

• Clicking a torrent gives you the option of saving it, passing on to the default downloader manager dialog, or canceling.
• Seeding/creating a single file torrent from the Tools menu. Provide option of saving .torrent files.
• Default seeding until reaching DL/UL ratio of 1/1.5, or no limit if creating a seed.
  
• No way to manage (eg. stop, pause) the torrents other than closing Firefox.
• The first torrent started launches a server on port 6881. The others attempt to, but will fail.
• See communications via the error console. It gets very noisy.
  

Fixes

• Corrected call to XPCOMUtils.categoryManager.addCategoryEntry. Since components are NOT registered every application opening, the 4th parameter needs to be true if we want to handle the bittorrent content type (mimetype) when the browser restarts.
• All this time, the XPI has been badly structured. The components folder was being placed in the chrome jar, when it should be in the root of the xpi. A discussion with ashughes forced me to check the installation process and fix this issue. Everyone benefits.
• Compliance with tracker's minimum update time.
• Keepalive handler is being fired correctly.
  

Progress - Torrents to Disk

New - Choosing where to save
I've taken the path of least resistance in choosing how to save to disk. After opening a torrent file, I've used the nsIFilePicker for selecting a folder (use the constant Components.interfaces.nsIFilePicker.modeGetFolder when calling init), and then stored the returned nsILocalFile in a central location. Initially, only the PieceManager was informed of this nsILocalFile, however it's of greater value if the folder is known to all the components: the original .torrent is saved into the folder, and any tracker responses are also saved into the folder. This has made debugging easier, with the help of GHex, the Gnome Hex Editor.

Changes to Tracker
The tracker is the first port of call after a .torrent file is opened (hohoho). Trackers communicate via HTTP, responding to a well formed URI with a bencoded (Bittorrent specific encoding) data structure detailing various things about the tracker and other peers.
When I first started testing using the latest Ubuntu ISO, I ran into my first compatibility problem. There are a couple of ways of encoding the information on other peers, let's call them 'original' and 'compact'. Compact is specified by an optional query string parameter in the request to the tracker, and I wasn't providing that setting. As a result, the tracker didn't fall back to original, it instead rejected the request with a plain text reply, along the lines of get a new client. Unexpected, particularly when users won't see that response, and also being terribly useless for developers. Fixed by adding compact=1 and handling the compact encoded peers.
The latest change was for friendliness. The tracker provides minimum timeouts on tracker requests, which were being ignored.

Little Things
Saving files sucks. There's all the checking for paths existing, checking if a file can be created, etc. Saving with nsILocalFile rocks. The create method for nsIFile (the interface nsILocalFile extends) automatically creates paths. What if you've been told to save in a folder, but it already has a file with the same name? The createUnique method means you don't have to write the associated conflict resolution logic (unless the intention was to replace that file - the nsIFilePicker modeSave automates the asking of that question if you need it). Additionally, a .torrent file lists the download files with their paths. Recreating that folder structure can be error prone - if there were ".." components, it's possible the saving could end up somewhere it shouldn't (eg. writing into a folder mentioned in PATH environment var in Windows). Though I need to test to confirm, I use appendWithRelativePath on nsILocalFile to build the path from the initial save directory to the specified file . This should, I need to test, remove the unintended path exploitation. If it doesn't, oh well, we'll fix that.

Next: Early 3rd Release - with Saving goodness!

Progress - Testing Torrents

I'm updating because of the progress on downloading of a torrent to disk, the first complete results being saved a few hours ago (Woo hoo!). There's been mountains of changes in order to get this to happen, so the next couple of posts will be the build up to how the first torrent saved to disk.

Changes to Test Torrents
First, the testing grounds have greatly changed. Previously my tests have been on the latest Ubuntu ISO and on MIT's The Structure and Interpretation of Computer Programs Lectures. This last week I've changed specifically to small multiple file torrents, to which end shaver recommended a couple of the confidently legal downloads on http://www.mininova.org/.
These torrents are noticably different, particularly the treatment by peers; connections to peers are rarely accepted (presumably because each client is already well connected to the swarm); unchoking is comparatively rare. Unchoking is where the peer you're connecting to says it's okay to request blocks to download.

Changes to Peer Manager
Someone may recall I was only opening one connection to a peer and shutdowning(-ish) when that connection died. With the more 'pop' torrents mentioned above, this isn't enough for testing. The probability of hitting the one peer that accepts connections is pretty low. (How low? Not sure, I can see how those graphs and charts in Azureus could have come out of testing questions)
So, the Peer Manager is finally doing more than holding Peer objects that do nothing. Now, it attempts to connect to all the initial peers provided by the tracker, and performs a health check every 30 seconds to see if we need to attempt more connections. There's a timeout before re-trying peers, currently 3 minutes though I need to find the standard for this.

Changes to Piece downloading
Previously, I'd written so a Peer object (representing a conversation with a Peer) had a list of pieces we 'wanted' that we knew the peer had. The first 'want' would attempt to be downloaded, and when finished the data would be passed to the Piece Manager. This isn't good enough. Too often a disconnect or similar will happen before the full piece is downloaded, meaning many peers have partial pieces they can't release to the Piece Manager.
To correct this, partial pieces are now managed by the Piece Manager. This means any work on downloading a piece will be immediately useful, repeated downloads will be reduced thus the total download reduced, and seeding will occur earlier (this I want to enhance further - if a single piece is targetted first, we can start sharing earlier, which improves trade negotiations ;-) ).

Next: Saving Folder, Tracker Changes, Little Things