Info - Bencoding (and XPCOM component)

Bencoding is used in the Bittorrent protocol to encode simple structures containing dictionaries, strings, integers, booleans, and lists in .torrent files and tracker responses.

.torrent (Metainfo file)
The contents of a basic .torrent file will identify the tracker server, that is later used to find peers, and also describes the info dictionary (or hash table, or associative array). The info dictionary plays a couple of roles.
It lists the files in the torrent, and their lengths. Secondly, it describes the concatenated binary data as specific sized pieces, and provides a checksum (in the form of a SHA1 hash) for each piece. Thirdly, the dictionary is re-bencoded and a SHA1 hash of the resulting byte array is used to identify the torrent.

Tracker response
The tracker response should either identify some error, such as the tracker not handling the SHA1 torrent identifier, or specifies information about the tracker and a list of peers who are also part of the identified torrent's swarm.

Implementation of Bencoding XPCOM component
IDL for Bencoding XPCOM component

Example:

var bencoding = Components.
classes["@wikiscraps.com/encoding/bencoding;1"].
getService(Components.interfaces.Bencoding);
// Encode an object into a string
var data = bencoding.bencode({
announce:"http://127.0.0.1/announce"
,info: {}
});
// Decode a string into an object
var obj = bencoding.bdecode(data);

More information on bencoding (Theory.org) Bittorrent Protocol Spec (BEP0003)
More information on the .torrent structure (Theory.org) Bittorrent Protocol Specification (BEP0003)
More information on the tracker response (Theory.org) Bittorrent Protocol Spec (BEP0003)

Fifth Release - Bittorrent for Firefox

This release may be a tad broken, as there was a push to get new features in, but then that was left to focus on reducing memory usage. If you want to check it out, do so and please provide feedback.
If you're looking for a user friendly or stable bittorrent handler: try this out once before uninstalling, give feedback on this post, and come back to the blog (bookmark it!) later this week. If I get feedback, I'll make a nice tidy release based on the feedback, just for you (Awwh). After that, I'm on holiday. Woo! Adelaide Fringe Festival!

20090223

• The component source
• The XPI - Click to install or save the extension.
  

What You Get

• The opportunity to test.
• Torrent data is now saved directly to disk. Probably want some middle ground there, but whatever.
  
• Clicking a torrent gives you the option of saving it, passing on to the default download manager dialog, or canceling.
• Tools > List Torrents lists torrents. It's in the middle of cosmetic changes, feel free to comment. There's a 20 second timeout on updating stats: I found updating XUL listboxes on torrents with hundreds of pieces gave an unpleasant UI experience.
  
• Have the option of pausing - but it's only cosmetic.
• Have the option of cancelling - but cancelling code is incomplete.
  
• The first torrent started launches a server on port 6881. The next torrent launches a server on 6882, this continues with an increment of 1 for each new torrent.
  
• Default to hide log messages, but show error messages. (Needs a code edit to change, not in preferences yet)
• Pre-allocation of disk space.
• Less memory usage (so it would seem).
• When there's a single seed, the speed has been improved up to 4 times from the last release (results will vary wildly). This is likely to do with having implemented pipelined requests; now we keep 2-4 requests pending with the peer. Previously, we waited until receiving a response before sending a new request.
  

Fixes

• Correct reporting of remaining bytes to Tracker. Previously, treated last piece as equal length to normal pieces.
• Temporary fix for servers. Now we create a server for each torrent, starting at port 6881 and going up by 1 for each torrent managed.
• If we find the peer that connected has the same peerid as our model, we close the connection. It's ourself! (Or a very unlikely occurrence) This is actually kinda hard to know without connecting.
• Probably some other things.

Check Out
Here's some things to check out, that I've liked in the past. Test these with the extension!

• Structure and Interpretation of Computer Programs Lectures by Hal Abelson and Gerald Jay Sussman, release under Creative Commons Attribution-ShareAlike
• Interviews with various ranging from Doug Engelbart to Dave Winer (Dead, see Update at bottom of post) by Robert X. Cringley / PBS. (Torrent links appear in popup windows). Released under Creative Commons Attribution-NonCommercial-ShareAlike 2.5 License
  
• Try Bittorrent.com / Ask.com's search. I'd advise caution, as there appears to be some amount of content that doesn't have approval by their creators.

Update 2009-02-24: The PBS interviews have 2 problems: 1) the content type is incorrect 2) the trackers don't know the torrents. Soooo, they won't work anymore for any bittorrent client (except maybe DHT clients, ooh, next feature?)

Progress? Firefox Debug Builds on Ubuntu 8.10

I expected to have released a new version, but on finally using the Bittorrent extension on torrents with data greater than 300Mb, I can see that the memory usage is quite horrid, and memory isn't released when downloading is finished.
So now I'm stuck in trying to debug to find why memory isn't released, which brings me to getting a debug build on Ubuntu 8.10.

Building on Ubuntu 8.10

DO NOT follow this guide. You'll be building an OLD Firefox. Check out jboston's user page; you MUST use Mercurial (hg) to get the latest source. I wish this had been made more clear on MDC.

It appears there is a new hoop to jump through to build on Ubuntu 8.10. A better guide is on jboston's user page; consider this another way to skin a cat. I initially followed dbranski's guide to building Firefox on Ubuntu, though this doesn't appear to work on Ubuntu 8.10.
I grab some packages:

sudo apt-get install gcc g++ perl make cvs libxt-dev libidl0 libidl-dev libfreetype6 fontconfig libcurl4-openssl-dev libgtk2.0-dev libdbus-1-dev libdbus-glib-1-dev 

I grab a make file that lets me grab the source I'll need:

cvs -d :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot co mozilla/client.mk

I've also grabbed trace-malloc for this debug build, not sure if this is the right way, but it's what I've done. Skipped when not making a debug build.

cvs -d :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot co mozilla/tools/trace-malloc

I get into the mozilla folder, and open up the .mozconfig file to configure this build.

cd mozilla
nano .mozconfig

I paste the mozconfig for a debug build into the file (found the text from Mozilla QA's Howto Debug Builds). I comment out the CPU line and the let line beneath it, and the line marked MOZ_MAKE_FLAGS (the let line caused a complaint, and I don't care about the 'make' tool running jobs in parrallel). I then add the lines:

export CFLAGS="-U_FORTIFY_SOURCE"
export CXXFLAGS="-U_FORTIFY_SOURCE"

Then save the file. The above addition is because Ubuntu has changed the default gcc flags, and the fix is due to Ori Peleg. Unoptimised builds are not effected. If you're using an existing mozconfig that doesn't explicitly turn off optimisation, add the above U_FORTIFY_SOURCE statements.
I then get the source I need:

make -f client.mk checkout MOZ_CO_PROJECT=browser

And then start the build:

time make -f client.mk build

To find out if I'm leaking, I set an environment variable to a file I want to log to then run firefox (learnt from Mozilla QA leak testing):

export XPCOM_MEM_LEAK_LOG=~/leaklog0001.log
cd firefox-debug/dist/bin/
./firefox -P

When I've done whatever I want to test, I then quit Firefox and change the environment variable so I don't overwrite the log file.

export XPCOM_MEM_LEAK_LOG=~/leaklog0002.log

If I have leaks, the file tells me what objects weren't freed and in what quantity. To understand the output, I use MDC's debugging memory leaks page. It doesn't tell me why though, and that's my next problem.

Disliking First Steps in Debugging with Tools

I'm dealing with a component grabbing way, way too much memory. In the downloading of a Torrent of size 341Mb, the extension will cause Firefox to consume 2.1Gb of memory. Uncool. The most likely cause of the growth, in my opinion, would be the actual data being received.

I think it should be the case if nothing happens, not much memory will be eaten (though this isn't exactly true, I see a jump of 38Mb when my content type handler component is fired for the first time - this seems a bit much).

So, I'm trying to learn the ways of debugging memory issues in Firefox. I will admit, this is frustrating me more than I've been frustrated in some time. There are several options for finding memory leaks. So far, I've started down the route of gdb. I can start gdb, and that's about it. I was meant to set a breakpoint to watch for garbage collection, but that didn't quite work - perhaps I need a debug build.

I thought I'd try the leaksoup tool. To use that, I've tried a build with trace-malloc enabled. Failed, I get buffer overflows when running the build.

I've tried creating a debug build. That's failed, I get buffer overflows when running the build. Ugh. Perhaps I've installed the wrong pre-requisite?

I've tried just building. That's failed, again with a buffer overflow. Could be bad pre-requisite, or current trunk is bad? I've done another checkout, and building again.

I've tried setting the environment variable XPC_SHUTDOWN_HEAP_DUMP to a filename before running firefox from terminal, but that does nothing. (I shouldn't need a debug build for using this variable, according to https://wiki.mozilla.org/Performance:Leak_Tools#JavaScript_heap_dump )

I found something that looks pretty good - DumpHeap ( https://bugzilla.mozilla.org/show_bug.cgi?id=378261 ), but darned if I know how to call it. It seems to be concerned with finding objects in a particular scope though, and I doubt my problems are that easy.

I think any debugging route with a debugger (My current lists to try are valgrind and continue with gdb) would require the debug build to make things anywhere near easy, which requires me to resolve why I can't build anything. So I guess that's my current pain. I mean, plan.

Info - HowTo write to specific location in a file using XPCOM

I'm writing torrents to disk, so a lot data is written to files out-of-order. During setup, I pre-allocate the file space by opening a file and moving to the position the last byte would be, using the nsISeekableStream interface. This technique I picked up from DownThemAll's manager class.
When I receive a piece of a file, I need to put it in the right place. What I was doing was moving to the right place, then writing:

seekstream.seek(0x00, offset);
stream.write(data, data.length);

This failed, somehow I was overwriting/truncating what was in the file. The solution was to first move to the end of the file, then move to the offset.

function insertInFile(filename, offset, data) {
var file = Components.classes["@mozilla.org/file/local;1"]
           .createInstance(Components.interfaces.nsILocalFile);
file.initWithPath( filename );
var stream = Components.classes["@mozilla.org/network/file-output-stream;1"]
            .createInstance(Components.interfaces.nsIFileOutputStream);
stream.init(file, 0x04 | 0x08, 0664, 0); // readwrite | create
var seekstream = stream.QueryInterface(Components.interfaces.nsISeekableStream);
seekstream.seek(0x02, 0); // move to end of file
seekstream.seek(0x01, offset - seekstream.tell() ); // move to where we actually want to write
stream.write(data, data.length);
stream.close();
}

I'll leave with a note that in my tests I've been seeing a byte with the value 0x0A at the end of my files. If I find that's to do with something I've done here, I'll try to remember to update this post. Just watch out for it if you're using the above method.

Upcoming later today - 5th Release of Bittorrent Firefox Extension with fixes Not happening. Latest version has a critical bug with reconnecting to it's own server.

Upcoming Week

This is highly optimistic, as I can't get this much done. The issues that relate/contribute to my research have priority, which is pretty much the first point, maybe the last point, and a number of other items that no-one would be interested in, so I don't blog. Feedback welcome.

• Log Socket Layer to Disk: need to record communications with peers for later scrutiny. Particularly when there's many peers, it's hard to decipher what problems have occurred.
  
• Implement Pause: pause should put the torrent into a holding pattern: no more connections should be accepted or sought. Tracker should be informed downloading has stopped. Existing connections should be choked. Should pass "not interested" to all peers for the moment. If disconnects occur, that's okay. Unsure about keep-alives.
• Implement Memory on Disk: create an abstraction layer, so that a block can be resolved to part (or parts) of the destination files for saving and uploading. This is so we don't need to store the whole torrent in memory at one time, instead we save what we receive as soon as it's confirmed.
  
• Implement Memorising Torrents between Sessions: so we can close Firefox and resume later.
• Implement Resuming from File Set: if we find the target files already in the destination folder, let's see if we can complete them against the torrent's description.
• Implement Limiting Which Files Downloaded: this is partially done already, due to my other goals. Just needs a user interface.
  
• One Server handles all incoming connections: Still not done.
  

Info - History for Firefox and Bittorrent

So, if you want to see integration between Firefox or Songbird and torrents, look in fear. Here's the history that current development can look back on.
Plug for current development efforts: Ashughes on a Songbird bittorrent extension leveraging libtorrent. I should plug my own efforts, but really, has anyone installed it for testing? I'm looking at enabling early writing to disk this week, which should allow large ISO size torrents without a significant memory drain, but there really needs to be more testing or such a size may never download due to other issues. To that end, the next release (Friday?) will have the ability to log the history with each peer, so we can examine what went okay, and what went bad. For now, tests on torrents at mininova.org's featured content have worked for myself up to 20 or so MB (tested on OS X & Ubuntu 8.10, FF 3); I haven't attempted higher, as I don't actually have any need of the content.

Moztorrent
http://moztorrent.mozdev.org/
Last update: was there a first update? Probably the first Bittorrent/Mozilla attempt. Had screenshots, and mockups appear to use download manager. Had a Mailing list (I can't see it ever being used by the authors). No source, so development dead-end, and no updates since archive.org started watching it in December 2004.

FirePuddle
http://firepuddle.mozdev.org/
Thanks to humph I've recently found this one from 2005. Result of a Summer of Code project, this extension worked with Firefox 1.5, but has a list of bugs that make it quite brittle. Lack of support for multiple file torrents would be the killer bug, as that's too common to ignore. (For bugs, see http://firepuddle.mozdev.org/installation.html)

Torrential
http://torrential.mozdev.org/
There is nothing here. The source committed is the source of the website.

Firestorm
http://firestorm.mozdev.org/
Last update: 2007 May 22 (website) May 23 (mailing list). Jump started by leveraging Mainline (Bram Cohen's original client) via PyXPCOM. Has Screenshots, and looks like seeding support. Looks like fair information feedback, and appears to have had right click support on torrent links. Looks like preferences exist (though probably not enabled). Source available (browse, download instructions) And a mailing list: Mailing list. So what's bad? No XPI (which means it can't be installed) and no updates in over year. See the last words on the mailing list - which, thankfully, were about the release of the source code.

Bitfox
http://code.google.com/p/bitfox
I wonder if this will be picked up, or if there is enough to pick up. Some C source files.

Foxtorrent
http://www.foxtorrent.com/ (Dead, blog still up)
The standard by which all other FireFox Bittorrent extensions should be measured. Reached maturity, and survived for some time. The killer is the disappearance of the Red Swoosh app which this extension relied on, which as far as I can tell, was never open source software. So this is probably a development dead end, as the only source is javascript that links to the Red Swoosh backend via http.
Hit 1.0 on 26th April 2007.
You can install a reviewed addon v1.13: https://addons.mozilla.org/en-US/firefox/addon/4844 (5th June, 2007)
Install a not-reviewed 2.0: http://foxtorrent.googlecode.com/svn-history/r236/trunk/foxtorrent/ (6 August, 2008 maybe)
Info on API for Red Swoosh: http://code.google.com/p/foxtorrent/wiki/RedSwooshTorrentAPIDocumentation

FireTorrent
http://www.wyzo.com/firetorrent/
According to TechCrunch and TorrentFreak, this Wyzo/Firefox extension is developed using P2P Innovative tech. Well, okay, that was in 2007, in 2008 TorrentFreak also reports it's using libtorrent. How can we tell? It has the appearance of open source, but damned if I can find the bittorrent source - and the only source code is a single snapshot. The source provided is either dated 26 March 2007 (based on the contents of the source zip) or 30 March 2007 (the log date on the subversion repository).

AllPeers
http://developer.allpeers.com/
Honestly, I know too little about AllPeers, but it looks the most promising for open source software. The company is gone, but the developer site seems to still exist. The site reports the server is closed source, but the client is open, though I don't know what that means for ordinary torrents. I think digging through this code is where I'll be spending some spare time.