Info - Bencoding (and XPCOM component)

Bencoding is used in the Bittorrent protocol to encode simple structures containing dictionaries, strings, integers, booleans, and lists in .torrent files and tracker responses.

.torrent (Metainfo file)
The contents of a basic .torrent file will identify the tracker server, that is later used to find peers, and also describes the info dictionary (or hash table, or associative array). The info dictionary plays a couple of roles.
It lists the files in the torrent, and their lengths. Secondly, it describes the concatenated binary data as specific sized pieces, and provides a checksum (in the form of a SHA1 hash) for each piece. Thirdly, the dictionary is re-bencoded and a SHA1 hash of the resulting byte array is used to identify the torrent.

Tracker response
The tracker response should either identify some error, such as the tracker not handling the SHA1 torrent identifier, or specifies information about the tracker and a list of peers who are also part of the identified torrent's swarm.

Implementation of Bencoding XPCOM component
IDL for Bencoding XPCOM component

Example:

var bencoding = Components.
classes["@wikiscraps.com/encoding/bencoding;1"].
getService(Components.interfaces.Bencoding);
// Encode an object into a string
var data = bencoding.bencode({
announce:"http://127.0.0.1/announce"
,info: {}
});
// Decode a string into an object
var obj = bencoding.bdecode(data);

More information on bencoding (Theory.org) Bittorrent Protocol Spec (BEP0003)
More information on the .torrent structure (Theory.org) Bittorrent Protocol Specification (BEP0003)
More information on the tracker response (Theory.org) Bittorrent Protocol Spec (BEP0003)

Fifth Release - Bittorrent for Firefox

This release may be a tad broken, as there was a push to get new features in, but then that was left to focus on reducing memory usage. If you want to check it out, do so and please provide feedback.
If you're looking for a user friendly or stable bittorrent handler: try this out once before uninstalling, give feedback on this post, and come back to the blog (bookmark it!) later this week. If I get feedback, I'll make a nice tidy release based on the feedback, just for you (Awwh). After that, I'm on holiday. Woo! Adelaide Fringe Festival!

20090223

• The component source
• The XPI - Click to install or save the extension.
  

What You Get

• The opportunity to test.
• Torrent data is now saved directly to disk. Probably want some middle ground there, but whatever.
  
• Clicking a torrent gives you the option of saving it, passing on to the default download manager dialog, or canceling.
• Tools > List Torrents lists torrents. It's in the middle of cosmetic changes, feel free to comment. There's a 20 second timeout on updating stats: I found updating XUL listboxes on torrents with hundreds of pieces gave an unpleasant UI experience.
  
• Have the option of pausing - but it's only cosmetic.
• Have the option of cancelling - but cancelling code is incomplete.
  
• The first torrent started launches a server on port 6881. The next torrent launches a server on 6882, this continues with an increment of 1 for each new torrent.
  
• Default to hide log messages, but show error messages. (Needs a code edit to change, not in preferences yet)
• Pre-allocation of disk space.
• Less memory usage (so it would seem).
• When there's a single seed, the speed has been improved up to 4 times from the last release (results will vary wildly). This is likely to do with having implemented pipelined requests; now we keep 2-4 requests pending with the peer. Previously, we waited until receiving a response before sending a new request.
  

Fixes

• Correct reporting of remaining bytes to Tracker. Previously, treated last piece as equal length to normal pieces.
• Temporary fix for servers. Now we create a server for each torrent, starting at port 6881 and going up by 1 for each torrent managed.
• If we find the peer that connected has the same peerid as our model, we close the connection. It's ourself! (Or a very unlikely occurrence) This is actually kinda hard to know without connecting.
• Probably some other things.

Check Out
Here's some things to check out, that I've liked in the past. Test these with the extension!

• Structure and Interpretation of Computer Programs Lectures by Hal Abelson and Gerald Jay Sussman, release under Creative Commons Attribution-ShareAlike
• Interviews with various ranging from Doug Engelbart to Dave Winer (Dead, see Update at bottom of post) by Robert X. Cringley / PBS. (Torrent links appear in popup windows). Released under Creative Commons Attribution-NonCommercial-ShareAlike 2.5 License
  
• Try Bittorrent.com / Ask.com's search. I'd advise caution, as there appears to be some amount of content that doesn't have approval by their creators.

Update 2009-02-24: The PBS interviews have 2 problems: 1) the content type is incorrect 2) the trackers don't know the torrents. Soooo, they won't work anymore for any bittorrent client (except maybe DHT clients, ooh, next feature?)

Progress? Firefox Debug Builds on Ubuntu 8.10

I expected to have released a new version, but on finally using the Bittorrent extension on torrents with data greater than 300Mb, I can see that the memory usage is quite horrid, and memory isn't released when downloading is finished.
So now I'm stuck in trying to debug to find why memory isn't released, which brings me to getting a debug build on Ubuntu 8.10.

Building on Ubuntu 8.10

DO NOT follow this guide. You'll be building an OLD Firefox. Check out jboston's user page; you MUST use Mercurial (hg) to get the latest source. I wish this had been made more clear on MDC.

It appears there is a new hoop to jump through to build on Ubuntu 8.10. A better guide is on jboston's user page; consider this another way to skin a cat. I initially followed dbranski's guide to building Firefox on Ubuntu, though this doesn't appear to work on Ubuntu 8.10.
I grab some packages:

sudo apt-get install gcc g++ perl make cvs libxt-dev libidl0 libidl-dev libfreetype6 fontconfig libcurl4-openssl-dev libgtk2.0-dev libdbus-1-dev libdbus-glib-1-dev 

I grab a make file that lets me grab the source I'll need:

cvs -d :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot co mozilla/client.mk

I've also grabbed trace-malloc for this debug build, not sure if this is the right way, but it's what I've done. Skipped when not making a debug build.

cvs -d :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot co mozilla/tools/trace-malloc

I get into the mozilla folder, and open up the .mozconfig file to configure this build.

cd mozilla
nano .mozconfig

I paste the mozconfig for a debug build into the file (found the text from Mozilla QA's Howto Debug Builds). I comment out the CPU line and the let line beneath it, and the line marked MOZ_MAKE_FLAGS (the let line caused a complaint, and I don't care about the 'make' tool running jobs in parrallel). I then add the lines:

export CFLAGS="-U_FORTIFY_SOURCE"
export CXXFLAGS="-U_FORTIFY_SOURCE"

Then save the file. The above addition is because Ubuntu has changed the default gcc flags, and the fix is due to Ori Peleg. Unoptimised builds are not effected. If you're using an existing mozconfig that doesn't explicitly turn off optimisation, add the above U_FORTIFY_SOURCE statements.
I then get the source I need:

make -f client.mk checkout MOZ_CO_PROJECT=browser

And then start the build:

time make -f client.mk build

To find out if I'm leaking, I set an environment variable to a file I want to log to then run firefox (learnt from Mozilla QA leak testing):

export XPCOM_MEM_LEAK_LOG=~/leaklog0001.log
cd firefox-debug/dist/bin/
./firefox -P

When I've done whatever I want to test, I then quit Firefox and change the environment variable so I don't overwrite the log file.

export XPCOM_MEM_LEAK_LOG=~/leaklog0002.log

If I have leaks, the file tells me what objects weren't freed and in what quantity. To understand the output, I use MDC's debugging memory leaks page. It doesn't tell me why though, and that's my next problem.

Disliking First Steps in Debugging with Tools

I'm dealing with a component grabbing way, way too much memory. In the downloading of a Torrent of size 341Mb, the extension will cause Firefox to consume 2.1Gb of memory. Uncool. The most likely cause of the growth, in my opinion, would be the actual data being received.

I think it should be the case if nothing happens, not much memory will be eaten (though this isn't exactly true, I see a jump of 38Mb when my content type handler component is fired for the first time - this seems a bit much).

So, I'm trying to learn the ways of debugging memory issues in Firefox. I will admit, this is frustrating me more than I've been frustrated in some time. There are several options for finding memory leaks. So far, I've started down the route of gdb. I can start gdb, and that's about it. I was meant to set a breakpoint to watch for garbage collection, but that didn't quite work - perhaps I need a debug build.

I thought I'd try the leaksoup tool. To use that, I've tried a build with trace-malloc enabled. Failed, I get buffer overflows when running the build.

I've tried creating a debug build. That's failed, I get buffer overflows when running the build. Ugh. Perhaps I've installed the wrong pre-requisite?

I've tried just building. That's failed, again with a buffer overflow. Could be bad pre-requisite, or current trunk is bad? I've done another checkout, and building again.

I've tried setting the environment variable XPC_SHUTDOWN_HEAP_DUMP to a filename before running firefox from terminal, but that does nothing. (I shouldn't need a debug build for using this variable, according to https://wiki.mozilla.org/Performance:Leak_Tools#JavaScript_heap_dump )

I found something that looks pretty good - DumpHeap ( https://bugzilla.mozilla.org/show_bug.cgi?id=378261 ), but darned if I know how to call it. It seems to be concerned with finding objects in a particular scope though, and I doubt my problems are that easy.

I think any debugging route with a debugger (My current lists to try are valgrind and continue with gdb) would require the debug build to make things anywhere near easy, which requires me to resolve why I can't build anything. So I guess that's my current pain. I mean, plan.

Info - HowTo write to specific location in a file using XPCOM

I'm writing torrents to disk, so a lot data is written to files out-of-order. During setup, I pre-allocate the file space by opening a file and moving to the position the last byte would be, using the nsISeekableStream interface. This technique I picked up from DownThemAll's manager class.
When I receive a piece of a file, I need to put it in the right place. What I was doing was moving to the right place, then writing:

seekstream.seek(0x00, offset);
stream.write(data, data.length);

This failed, somehow I was overwriting/truncating what was in the file. The solution was to first move to the end of the file, then move to the offset.

function insertInFile(filename, offset, data) {
var file = Components.classes["@mozilla.org/file/local;1"]
           .createInstance(Components.interfaces.nsILocalFile);
file.initWithPath( filename );
var stream = Components.classes["@mozilla.org/network/file-output-stream;1"]
            .createInstance(Components.interfaces.nsIFileOutputStream);
stream.init(file, 0x04 | 0x08, 0664, 0); // readwrite | create
var seekstream = stream.QueryInterface(Components.interfaces.nsISeekableStream);
seekstream.seek(0x02, 0); // move to end of file
seekstream.seek(0x01, offset - seekstream.tell() ); // move to where we actually want to write
stream.write(data, data.length);
stream.close();
}

I'll leave with a note that in my tests I've been seeing a byte with the value 0x0A at the end of my files. If I find that's to do with something I've done here, I'll try to remember to update this post. Just watch out for it if you're using the above method.

Upcoming later today - 5th Release of Bittorrent Firefox Extension with fixes Not happening. Latest version has a critical bug with reconnecting to it's own server.

Upcoming Week

This is highly optimistic, as I can't get this much done. The issues that relate/contribute to my research have priority, which is pretty much the first point, maybe the last point, and a number of other items that no-one would be interested in, so I don't blog. Feedback welcome.

• Log Socket Layer to Disk: need to record communications with peers for later scrutiny. Particularly when there's many peers, it's hard to decipher what problems have occurred.
  
• Implement Pause: pause should put the torrent into a holding pattern: no more connections should be accepted or sought. Tracker should be informed downloading has stopped. Existing connections should be choked. Should pass "not interested" to all peers for the moment. If disconnects occur, that's okay. Unsure about keep-alives.
• Implement Memory on Disk: create an abstraction layer, so that a block can be resolved to part (or parts) of the destination files for saving and uploading. This is so we don't need to store the whole torrent in memory at one time, instead we save what we receive as soon as it's confirmed.
  
• Implement Memorising Torrents between Sessions: so we can close Firefox and resume later.
• Implement Resuming from File Set: if we find the target files already in the destination folder, let's see if we can complete them against the torrent's description.
• Implement Limiting Which Files Downloaded: this is partially done already, due to my other goals. Just needs a user interface.
  
• One Server handles all incoming connections: Still not done.
  

Info - History for Firefox and Bittorrent

So, if you want to see integration between Firefox or Songbird and torrents, look in fear. Here's the history that current development can look back on.
Plug for current development efforts: Ashughes on a Songbird bittorrent extension leveraging libtorrent. I should plug my own efforts, but really, has anyone installed it for testing? I'm looking at enabling early writing to disk this week, which should allow large ISO size torrents without a significant memory drain, but there really needs to be more testing or such a size may never download due to other issues. To that end, the next release (Friday?) will have the ability to log the history with each peer, so we can examine what went okay, and what went bad. For now, tests on torrents at mininova.org's featured content have worked for myself up to 20 or so MB (tested on OS X & Ubuntu 8.10, FF 3); I haven't attempted higher, as I don't actually have any need of the content.

Moztorrent
http://moztorrent.mozdev.org/
Last update: was there a first update? Probably the first Bittorrent/Mozilla attempt. Had screenshots, and mockups appear to use download manager. Had a Mailing list (I can't see it ever being used by the authors). No source, so development dead-end, and no updates since archive.org started watching it in December 2004.

FirePuddle
http://firepuddle.mozdev.org/
Thanks to humph I've recently found this one from 2005. Result of a Summer of Code project, this extension worked with Firefox 1.5, but has a list of bugs that make it quite brittle. Lack of support for multiple file torrents would be the killer bug, as that's too common to ignore. (For bugs, see http://firepuddle.mozdev.org/installation.html)

Torrential
http://torrential.mozdev.org/
There is nothing here. The source committed is the source of the website.

Firestorm
http://firestorm.mozdev.org/
Last update: 2007 May 22 (website) May 23 (mailing list). Jump started by leveraging Mainline (Bram Cohen's original client) via PyXPCOM. Has Screenshots, and looks like seeding support. Looks like fair information feedback, and appears to have had right click support on torrent links. Looks like preferences exist (though probably not enabled). Source available (browse, download instructions) And a mailing list: Mailing list. So what's bad? No XPI (which means it can't be installed) and no updates in over year. See the last words on the mailing list - which, thankfully, were about the release of the source code.

Bitfox
http://code.google.com/p/bitfox
I wonder if this will be picked up, or if there is enough to pick up. Some C source files.

Foxtorrent
http://www.foxtorrent.com/ (Dead, blog still up)
The standard by which all other FireFox Bittorrent extensions should be measured. Reached maturity, and survived for some time. The killer is the disappearance of the Red Swoosh app which this extension relied on, which as far as I can tell, was never open source software. So this is probably a development dead end, as the only source is javascript that links to the Red Swoosh backend via http.
Hit 1.0 on 26th April 2007.
You can install a reviewed addon v1.13: https://addons.mozilla.org/en-US/firefox/addon/4844 (5th June, 2007)
Install a not-reviewed 2.0: http://foxtorrent.googlecode.com/svn-history/r236/trunk/foxtorrent/ (6 August, 2008 maybe)
Info on API for Red Swoosh: http://code.google.com/p/foxtorrent/wiki/RedSwooshTorrentAPIDocumentation

FireTorrent
http://www.wyzo.com/firetorrent/
According to TechCrunch and TorrentFreak, this Wyzo/Firefox extension is developed using P2P Innovative tech. Well, okay, that was in 2007, in 2008 TorrentFreak also reports it's using libtorrent. How can we tell? It has the appearance of open source, but damned if I can find the bittorrent source - and the only source code is a single snapshot. The source provided is either dated 26 March 2007 (based on the contents of the source zip) or 30 March 2007 (the log date on the subversion repository).

AllPeers
http://developer.allpeers.com/
Honestly, I know too little about AllPeers, but it looks the most promising for open source software. The company is gone, but the developer site seems to still exist. The site reports the server is closed source, but the client is open, though I don't know what that means for ordinary torrents. I think digging through this code is where I'll be spending some spare time.

Early Fourth Release - Bittorrent for Firefox

Yay. Finally, a user interface for managing torrents: you can list torrents, view progress & file manifests. Feedback on using the UI would be handy. I'm looking at improving the ability of logging interactions with peers for future debugging, so next week's release I'm hoping for feedback on peer errors.

20090207

• The component source
• The XPI

What You Get

• Clicking a torrent gives you the option of saving it, passing on to the default download manager dialog, or canceling.
• Seeding/creating a single file torrent from the Tools menu. Provides option of saving .torrent files, and adding a web seed (Info on web seeds).
  
• Default seeding until reaching DL/UL ratio of 1/1.5, or no limit if creating a seed.
• Tools > List Torrents lists torrents.
  
• Have the option of pausing - but it's only cosmetic.
• Have the option of cancelling a torrent. Should work, testing desired.
  
• The first torrent started launches a server on port 6881. The others attempt to, but will fail.
• See communications via the error console. It gets very noisy.
  

Fixes

• Supports requests over piece boundaries.
• If there is no need to choke, it won't. Previously, any leacher was choked after a certain amount of downloading, and then unchoked almost immediately if no other peers were pending.
  

Early Third Release - Bittorrent for Firefox

Yay. The Bittorrent extension for Firefox is almost suitable for users. You can use it! Though you'll have to quit Firefox to cancel/pause/stop torrents. Thus far this hasn't caused problems for me, as seeding has been pretty quiet. Probably don't want to do it on huge torrents, as the torrent is still kept in memory. This release is a day earlier than planned.

20090203

• The component source
• The XPI

What You Get

• Clicking a torrent gives you the option of saving it, passing on to the default downloader manager dialog, or canceling.
• Seeding/creating a single file torrent from the Tools menu. Provide option of saving .torrent files.
• Default seeding until reaching DL/UL ratio of 1/1.5, or no limit if creating a seed.
  
• No way to manage (eg. stop, pause) the torrents other than closing Firefox.
• The first torrent started launches a server on port 6881. The others attempt to, but will fail.
• See communications via the error console. It gets very noisy.
  

Fixes

• Corrected call to XPCOMUtils.categoryManager.addCategoryEntry. Since components are NOT registered every application opening, the 4th parameter needs to be true if we want to handle the bittorrent content type (mimetype) when the browser restarts.
• All this time, the XPI has been badly structured. The components folder was being placed in the chrome jar, when it should be in the root of the xpi. A discussion with ashughes forced me to check the installation process and fix this issue. Everyone benefits.
• Compliance with tracker's minimum update time.
• Keepalive handler is being fired correctly.
  

Progress - Torrents to Disk

New - Choosing where to save
I've taken the path of least resistance in choosing how to save to disk. After opening a torrent file, I've used the nsIFilePicker for selecting a folder (use the constant Components.interfaces.nsIFilePicker.modeGetFolder when calling init), and then stored the returned nsILocalFile in a central location. Initially, only the PieceManager was informed of this nsILocalFile, however it's of greater value if the folder is known to all the components: the original .torrent is saved into the folder, and any tracker responses are also saved into the folder. This has made debugging easier, with the help of GHex, the Gnome Hex Editor.

Changes to Tracker
The tracker is the first port of call after a .torrent file is opened (hohoho). Trackers communicate via HTTP, responding to a well formed URI with a bencoded (Bittorrent specific encoding) data structure detailing various things about the tracker and other peers.
When I first started testing using the latest Ubuntu ISO, I ran into my first compatibility problem. There are a couple of ways of encoding the information on other peers, let's call them 'original' and 'compact'. Compact is specified by an optional query string parameter in the request to the tracker, and I wasn't providing that setting. As a result, the tracker didn't fall back to original, it instead rejected the request with a plain text reply, along the lines of get a new client. Unexpected, particularly when users won't see that response, and also being terribly useless for developers. Fixed by adding compact=1 and handling the compact encoded peers.
The latest change was for friendliness. The tracker provides minimum timeouts on tracker requests, which were being ignored.

Little Things
Saving files sucks. There's all the checking for paths existing, checking if a file can be created, etc. Saving with nsILocalFile rocks. The create method for nsIFile (the interface nsILocalFile extends) automatically creates paths. What if you've been told to save in a folder, but it already has a file with the same name? The createUnique method means you don't have to write the associated conflict resolution logic (unless the intention was to replace that file - the nsIFilePicker modeSave automates the asking of that question if you need it). Additionally, a .torrent file lists the download files with their paths. Recreating that folder structure can be error prone - if there were ".." components, it's possible the saving could end up somewhere it shouldn't (eg. writing into a folder mentioned in PATH environment var in Windows). Though I need to test to confirm, I use appendWithRelativePath on nsILocalFile to build the path from the initial save directory to the specified file . This should, I need to test, remove the unintended path exploitation. If it doesn't, oh well, we'll fix that.

Next: Early 3rd Release - with Saving goodness!

Progress - Testing Torrents

I'm updating because of the progress on downloading of a torrent to disk, the first complete results being saved a few hours ago (Woo hoo!). There's been mountains of changes in order to get this to happen, so the next couple of posts will be the build up to how the first torrent saved to disk.

Changes to Test Torrents
First, the testing grounds have greatly changed. Previously my tests have been on the latest Ubuntu ISO and on MIT's The Structure and Interpretation of Computer Programs Lectures. This last week I've changed specifically to small multiple file torrents, to which end shaver recommended a couple of the confidently legal downloads on http://www.mininova.org/.
These torrents are noticably different, particularly the treatment by peers; connections to peers are rarely accepted (presumably because each client is already well connected to the swarm); unchoking is comparatively rare. Unchoking is where the peer you're connecting to says it's okay to request blocks to download.

Changes to Peer Manager
Someone may recall I was only opening one connection to a peer and shutdowning(-ish) when that connection died. With the more 'pop' torrents mentioned above, this isn't enough for testing. The probability of hitting the one peer that accepts connections is pretty low. (How low? Not sure, I can see how those graphs and charts in Azureus could have come out of testing questions)
So, the Peer Manager is finally doing more than holding Peer objects that do nothing. Now, it attempts to connect to all the initial peers provided by the tracker, and performs a health check every 30 seconds to see if we need to attempt more connections. There's a timeout before re-trying peers, currently 3 minutes though I need to find the standard for this.

Changes to Piece downloading
Previously, I'd written so a Peer object (representing a conversation with a Peer) had a list of pieces we 'wanted' that we knew the peer had. The first 'want' would attempt to be downloaded, and when finished the data would be passed to the Piece Manager. This isn't good enough. Too often a disconnect or similar will happen before the full piece is downloaded, meaning many peers have partial pieces they can't release to the Piece Manager.
To correct this, partial pieces are now managed by the Piece Manager. This means any work on downloading a piece will be immediately useful, repeated downloads will be reduced thus the total download reduced, and seeding will occur earlier (this I want to enhance further - if a single piece is targetted first, we can start sharing earlier, which improves trade negotiations ;-) ).

Next: Saving Folder, Tracker Changes, Little Things