Update - Refactoring Bittorrent for Firefox

Just a small update.

So, what's the state of the Bittorrent for Firefox extension? The last released version works, is fine for use on smaller torrents but needs thorough debugging to find some leaks that become quite bad on larger torrents.

So, what am I doing? Well, I've been caught up in the academic year; I've been tutoring, and will be lecturing in a few weeks. It's taken time away from some research (particularly some experiments) I'm doing using P2P for content delivery. Ooh! Fancy! Disregarding that, I've been refactoring the existing code so that components can be used independently of one another as complete XPCOM components. A previous post provided a Bencoding component for encoding/decoding data structures, and more recently I've been refactoring the Bittorrent Tracker client code. It's painfully slow, but should be nice. If you can see yourself using bittorrent XPCOM components, I'd love to hear the interface you'd like.

So, what to look forward to? The code for shutdown of torrent has been corrected, and will be released on request or once refactoring complete. If anyone wants to build a different Bittorrent client, they can look forward to using the refactored and separated components. In the future, I'd like to see the downloads appear in the Download Manager, instead of in the current version's Tools > Torrents menu (this way I can piggy-back on the status bar download progress without taking up more space, and it also makes managing torrents easier for the user), though if anyone else would like to write that, that'd be wonderful. ;-)

Question: Preferred Behaviour on Torrent Problems

What kind of response would you like if:

1. The torrent file was corrupt
2. The tracker doesn't exist (therefore can't download)
3. The tracker may exist, but isn't responding (maybe can't download)
4. There are no peers (therefore can't download at the moment)
5. There are no peers accepting connections (therefore can't download at the moment)
6. Peers won't share after, for example, a minute of waiting (therefore can't download at the moment)
7. Given the peers we have seen in the last, for example, minute, no-one has the complete file we're interested in (therefore can't complete download at the moment)
8. There is a problem saving to the specified destination
9. Firefox is closing (there is the possibility of resuming when re-opening Firefox)
   
10. The download is complete
    

Info - Bencoding (and XPCOM component)

Bencoding is used in the Bittorrent protocol to encode simple structures containing dictionaries, strings, integers, booleans, and lists in .torrent files and tracker responses.

.torrent (Metainfo file)
The contents of a basic .torrent file will identify the tracker server, that is later used to find peers, and also describes the info dictionary (or hash table, or associative array). The info dictionary plays a couple of roles.
It lists the files in the torrent, and their lengths. Secondly, it describes the concatenated binary data as specific sized pieces, and provides a checksum (in the form of a SHA1 hash) for each piece. Thirdly, the dictionary is re-bencoded and a SHA1 hash of the resulting byte array is used to identify the torrent.

Tracker response
The tracker response should either identify some error, such as the tracker not handling the SHA1 torrent identifier, or specifies information about the tracker and a list of peers who are also part of the identified torrent's swarm.

Implementation of Bencoding XPCOM component
IDL for Bencoding XPCOM component

Example:

var bencoding = Components.
classes["@wikiscraps.com/encoding/bencoding;1"].
getService(Components.interfaces.Bencoding);
// Encode an object into a string
var data = bencoding.bencode({
announce:"http://127.0.0.1/announce"
,info: {}
});
// Decode a string into an object
var obj = bencoding.bdecode(data);

More information on bencoding (Theory.org) Bittorrent Protocol Spec (BEP0003)
More information on the .torrent structure (Theory.org) Bittorrent Protocol Specification (BEP0003)
More information on the tracker response (Theory.org) Bittorrent Protocol Spec (BEP0003)

Fifth Release - Bittorrent for Firefox

This release may be a tad broken, as there was a push to get new features in, but then that was left to focus on reducing memory usage. If you want to check it out, do so and please provide feedback.
If you're looking for a user friendly or stable bittorrent handler: try this out once before uninstalling, give feedback on this post, and come back to the blog (bookmark it!) later this week. If I get feedback, I'll make a nice tidy release based on the feedback, just for you (Awwh). After that, I'm on holiday. Woo! Adelaide Fringe Festival!

20090223

• The component source
• The XPI - Click to install or save the extension.
  

What You Get

• The opportunity to test.
• Torrent data is now saved directly to disk. Probably want some middle ground there, but whatever.
  
• Clicking a torrent gives you the option of saving it, passing on to the default download manager dialog, or canceling.
• Tools > List Torrents lists torrents. It's in the middle of cosmetic changes, feel free to comment. There's a 20 second timeout on updating stats: I found updating XUL listboxes on torrents with hundreds of pieces gave an unpleasant UI experience.
  
• Have the option of pausing - but it's only cosmetic.
• Have the option of cancelling - but cancelling code is incomplete.
  
• The first torrent started launches a server on port 6881. The next torrent launches a server on 6882, this continues with an increment of 1 for each new torrent.
  
• Default to hide log messages, but show error messages. (Needs a code edit to change, not in preferences yet)
• Pre-allocation of disk space.
• Less memory usage (so it would seem).
• When there's a single seed, the speed has been improved up to 4 times from the last release (results will vary wildly). This is likely to do with having implemented pipelined requests; now we keep 2-4 requests pending with the peer. Previously, we waited until receiving a response before sending a new request.
  

Fixes

• Correct reporting of remaining bytes to Tracker. Previously, treated last piece as equal length to normal pieces.
• Temporary fix for servers. Now we create a server for each torrent, starting at port 6881 and going up by 1 for each torrent managed.
• If we find the peer that connected has the same peerid as our model, we close the connection. It's ourself! (Or a very unlikely occurrence) This is actually kinda hard to know without connecting.
• Probably some other things.

Check Out
Here's some things to check out, that I've liked in the past. Test these with the extension!

• Structure and Interpretation of Computer Programs Lectures by Hal Abelson and Gerald Jay Sussman, release under Creative Commons Attribution-ShareAlike
• Interviews with various ranging from Doug Engelbart to Dave Winer (Dead, see Update at bottom of post) by Robert X. Cringley / PBS. (Torrent links appear in popup windows). Released under Creative Commons Attribution-NonCommercial-ShareAlike 2.5 License
  
• Try Bittorrent.com / Ask.com's search. I'd advise caution, as there appears to be some amount of content that doesn't have approval by their creators.

Update 2009-02-24: The PBS interviews have 2 problems: 1) the content type is incorrect 2) the trackers don't know the torrents. Soooo, they won't work anymore for any bittorrent client (except maybe DHT clients, ooh, next feature?)

Progress? Firefox Debug Builds on Ubuntu 8.10

I expected to have released a new version, but on finally using the Bittorrent extension on torrents with data greater than 300Mb, I can see that the memory usage is quite horrid, and memory isn't released when downloading is finished.
So now I'm stuck in trying to debug to find why memory isn't released, which brings me to getting a debug build on Ubuntu 8.10.

Building on Ubuntu 8.10

DO NOT follow this guide. You'll be building an OLD Firefox. Check out jboston's user page; you MUST use Mercurial (hg) to get the latest source. I wish this had been made more clear on MDC.

It appears there is a new hoop to jump through to build on Ubuntu 8.10. A better guide is on jboston's user page; consider this another way to skin a cat. I initially followed dbranski's guide to building Firefox on Ubuntu, though this doesn't appear to work on Ubuntu 8.10.
I grab some packages:

sudo apt-get install gcc g++ perl make cvs libxt-dev libidl0 libidl-dev libfreetype6 fontconfig libcurl4-openssl-dev libgtk2.0-dev libdbus-1-dev libdbus-glib-1-dev 

I grab a make file that lets me grab the source I'll need:

cvs -d :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot co mozilla/client.mk

I've also grabbed trace-malloc for this debug build, not sure if this is the right way, but it's what I've done. Skipped when not making a debug build.

cvs -d :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot co mozilla/tools/trace-malloc

I get into the mozilla folder, and open up the .mozconfig file to configure this build.

cd mozilla
nano .mozconfig

I paste the mozconfig for a debug build into the file (found the text from Mozilla QA's Howto Debug Builds). I comment out the CPU line and the let line beneath it, and the line marked MOZ_MAKE_FLAGS (the let line caused a complaint, and I don't care about the 'make' tool running jobs in parrallel). I then add the lines:

export CFLAGS="-U_FORTIFY_SOURCE"
export CXXFLAGS="-U_FORTIFY_SOURCE"

Then save the file. The above addition is because Ubuntu has changed the default gcc flags, and the fix is due to Ori Peleg. Unoptimised builds are not effected. If you're using an existing mozconfig that doesn't explicitly turn off optimisation, add the above U_FORTIFY_SOURCE statements.
I then get the source I need:

make -f client.mk checkout MOZ_CO_PROJECT=browser

And then start the build:

time make -f client.mk build

To find out if I'm leaking, I set an environment variable to a file I want to log to then run firefox (learnt from Mozilla QA leak testing):

export XPCOM_MEM_LEAK_LOG=~/leaklog0001.log
cd firefox-debug/dist/bin/
./firefox -P

When I've done whatever I want to test, I then quit Firefox and change the environment variable so I don't overwrite the log file.

export XPCOM_MEM_LEAK_LOG=~/leaklog0002.log

If I have leaks, the file tells me what objects weren't freed and in what quantity. To understand the output, I use MDC's debugging memory leaks page. It doesn't tell me why though, and that's my next problem.

Disliking First Steps in Debugging with Tools

I'm dealing with a component grabbing way, way too much memory. In the downloading of a Torrent of size 341Mb, the extension will cause Firefox to consume 2.1Gb of memory. Uncool. The most likely cause of the growth, in my opinion, would be the actual data being received.

I think it should be the case if nothing happens, not much memory will be eaten (though this isn't exactly true, I see a jump of 38Mb when my content type handler component is fired for the first time - this seems a bit much).

So, I'm trying to learn the ways of debugging memory issues in Firefox. I will admit, this is frustrating me more than I've been frustrated in some time. There are several options for finding memory leaks. So far, I've started down the route of gdb. I can start gdb, and that's about it. I was meant to set a breakpoint to watch for garbage collection, but that didn't quite work - perhaps I need a debug build.

I thought I'd try the leaksoup tool. To use that, I've tried a build with trace-malloc enabled. Failed, I get buffer overflows when running the build.

I've tried creating a debug build. That's failed, I get buffer overflows when running the build. Ugh. Perhaps I've installed the wrong pre-requisite?

I've tried just building. That's failed, again with a buffer overflow. Could be bad pre-requisite, or current trunk is bad? I've done another checkout, and building again.

I've tried setting the environment variable XPC_SHUTDOWN_HEAP_DUMP to a filename before running firefox from terminal, but that does nothing. (I shouldn't need a debug build for using this variable, according to https://wiki.mozilla.org/Performance:Leak_Tools#JavaScript_heap_dump )

I found something that looks pretty good - DumpHeap ( https://bugzilla.mozilla.org/show_bug.cgi?id=378261 ), but darned if I know how to call it. It seems to be concerned with finding objects in a particular scope though, and I doubt my problems are that easy.

I think any debugging route with a debugger (My current lists to try are valgrind and continue with gdb) would require the debug build to make things anywhere near easy, which requires me to resolve why I can't build anything. So I guess that's my current pain. I mean, plan.

Info - HowTo write to specific location in a file using XPCOM

I'm writing torrents to disk, so a lot data is written to files out-of-order. During setup, I pre-allocate the file space by opening a file and moving to the position the last byte would be, using the nsISeekableStream interface. This technique I picked up from DownThemAll's manager class.
When I receive a piece of a file, I need to put it in the right place. What I was doing was moving to the right place, then writing:

seekstream.seek(0x00, offset);
stream.write(data, data.length);

This failed, somehow I was overwriting/truncating what was in the file. The solution was to first move to the end of the file, then move to the offset.

function insertInFile(filename, offset, data) {
var file = Components.classes["@mozilla.org/file/local;1"]
           .createInstance(Components.interfaces.nsILocalFile);
file.initWithPath( filename );
var stream = Components.classes["@mozilla.org/network/file-output-stream;1"]
            .createInstance(Components.interfaces.nsIFileOutputStream);
stream.init(file, 0x04 | 0x08, 0664, 0); // readwrite | create
var seekstream = stream.QueryInterface(Components.interfaces.nsISeekableStream);
seekstream.seek(0x02, 0); // move to end of file
seekstream.seek(0x01, offset - seekstream.tell() ); // move to where we actually want to write
stream.write(data, data.length);
stream.close();
}

I'll leave with a note that in my tests I've been seeing a byte with the value 0x0A at the end of my files. If I find that's to do with something I've done here, I'll try to remember to update this post. Just watch out for it if you're using the above method.

Upcoming later today - 5th Release of Bittorrent Firefox Extension with fixes Not happening. Latest version has a critical bug with reconnecting to it's own server.