Saturday, December 19, 2009

XPCShell Testing for Firefox Addons (Extensions)

A quick bit of info on what I've done to get xpcshell unit tests working for my add-on.
Here's my folder structure:To the .mozconfig (not in the above diagram, as mine is located in my home folder), I add the following to enable tests:
ac_add_options --enable-tests

To the Makefile.in in the bittorrent folder, I take the existing lines:
# list of subfolders to process
DIRS = public src

And add the following (based on http://zenit.senecac.on.ca/wiki/index.php/Dive_into_Mozilla_Unit_Testing_Lab)
# Adding xpcshell unit tests
ifdef ENABLE_TESTS
DIRS += test
endif

This means in addition to Make processing the public & src subfolders, it should also process the test folder, if tests are enabled.
The Makefile.in in the test folder is terribly boring. My Add-ons module name is bittorrentwikiscraps, I can perhaps specify the module for this Makefile as that also, however the pattern seems to be test_modulename. (My notes say I was basing this on this Makefile.in, take a look if you need platform specific tests)
DEPTH        = ../../..
topsrcdir = @top_srcdir@
srcdir = @srcdir@
VPATH = @srcdir@

include $(DEPTH)/config/autoconf.mk

MODULE = test_bittorrentwikiscraps

XPCSHELL_TESTS = \
unit \
$(NULL)

include $(topsrcdir)/config/rules.mk

The XPCSHELL_TESTS defines the folders where the tests are stored, and MODULE defines where the tests will be copied, and run from. The tests will be copied to OBJDIR/_tests/xpcshell/test_bittorrentwikiscraps/unit/ when running 'make', and when running 'make xpcshell-tests' the tests in these folders will be executed.
The above is pretty normal for xpcshell tests. The next bit is what's been described to me by ted a couple of times, it's what crashreporter does to circumvent an issue with xpcshell (with a little hack of it's own^).
xpcshell doesn't know about add-ons, so the components I want to test won't be available when the tests run. So, I copy the components (JS files, in bittorrent/src/) to the testing area, by adding the following to bittorrent/src/Makefile.in (at the end of the file):
libs:: 
$(NSINSTALL) -D $(DEPTH)/_tests/xpcshell/test_$(MODULE)/unit/components

libs:: $(EXTRA_COMPONENTS)
$(INSTALL) $(foreach f,$^,"$f") $(DEPTH)/_tests/xpcshell/test_$(MODULE)/unit/components

With JavaScript components, the files are listed in EXTRA_COMPONENTS (you list them! It's not automatic!). What we're doing here is first creating the components subfolder in the testing area (the first use of libs::) and then copying every item in EXTRA_COMPONENTS to this new folder (second use of libs::). Yay! Now the component files are there. But how will xpcshell know about them when the tests are run?
This is acheived by using nsIComponentRegistrar to register the components during the test. Yes, every time a test is run the components are registered first. We do this by creating head_regComps.js in the unit folder. When tests are run, for example test_sample.js, all the head_*.js files are executed before the contents of test_sample.js to do any setup. So, head_regComps.js will be run first, which ensures the components are loaded before the testing code. The contents of head_regComps.js are from the crashreporter code:
// load components from test subfolder
// Get current working directory
let cwd = Components.classes["@mozilla.org/file/directory_service;1"]
.getService(Components.interfaces.nsIProperties)
.get("CurWorkD", Components.interfaces.nsILocalFile);
// load subfolder
let cd = cwd.clone();
cd.append("components");
Components.manager instanceof Components.interfaces.nsIComponentRegistrar;
Components.manager.autoRegister(cd);

The code gets the current working directory (the unit folder in the testing area) and then registers all the components in the components subfolder.
For tests that just involve new components, this is enough. My tests can now see the components (in test_sample.js):
function run_test() {
var btPresent = ("@wikiscraps.com/transport/wireInStream;1" in Components.classes);
do_check_true(btPresent);
}

However, my Add-on also has new component interfaces (see the IDL files in the public folder). This issue isn't tackled in the current Mozilla build setup, and the above approach based on crashreporter can't handle interfaces. So, to recap: the issue of xpcshell not being aware of Add-ons prevents the interface being loaded; and unlike components, interfaces can't be loaded dynamically. My hack is just to copy the interfaces to OBJDIR/dist/bin/components/. I'm fine with this, my Firefox build has an interface that belongs in the extension folder. This will probably cause an issue when the Add-on interface in a profile and this components folder are out of sync, but oh well. This build is fine for testing and generating the Add-on's XPI file.
Anyway, the copying of the XPT file is achieved by adding the following in the bittorrent/public/Makefile.in (at the end)
libs:: 
$(INSTALL) $(XPIDL_GEN_DIR)/$(XPIDL_MODULE).xpt $(DIST)/bin/components

Usually the xpt files are copied in a similar manner to the folder '${FINAL_TARGET}/components', however because Add-ons are built for an XPI that will mean FINAL_TARGET will point to the xpi-staging area, so instead '$(DIST)/bin/components' must be used. Now, when xpcshell is run the interface will be ready to be loaded.
Just a note, I had removed OBJDIR/dist/bin/components/components.list, OBJDIR/dist/bin/components/compreg.dat & OBJDIR/dist/bin/components/xpti.dat earlier during my development; I'm unsure if I hadn't done this if the interfaces would be loaded.
Anyway, we can now test and see the interface has been loaded, by adjusting test_sample.js:
function run_test() {
var btPresent = ("@wikiscraps.com/transport/wireInStream;1" in Components.classes);
do_check_true(btPresent);
var btiPresent = ("btIWireInputStream" in Components.interfaces);
do_check_true(btiPresent);
}

Yay. I can test my Add-on components using xpcshell unit tests.

Friday, November 27, 2009

Bah, 4 year old fixed chrome bug still an issue for TB developers

I idle in many Mozilla's IRC channels, and happened to see some chrome registration lines pass by on #extdev. The developer was having an issue with their chrome files not being found, with an error similar to:
no chrome package registered for chrome://extensionname/content/somefile.xul
The immediate thing I noticed was their chrome registration lines were describing extensionName rather than extensionname. Having been in the process of rewriting my extension to do things Mozilla Build way, I had been reading Chrome Registration a day or so earlier, and saw that it listed FireFox 2 as not handling mixed case package names as part of registration. Bug 132183 (thankfully listed on MDC article) seems to describe the issue, first reported in 2002 and resolved in 2005. A suggestion to use lower case for the package name fixed the bug for the #extdev developer, and a few hours later I've tried figuring out if this really was the cause.

I needed to find what code the latest release of ThunderBird was using. The version is 2.0.0.23 at this time, found here http://www.mozillamessaging.com/en-US/thunderbird/ . According to MDC, ThunderBird 2 uses the Gecko 1.8.1 branch. The bug report says the it was fixed on trunk in December 2005, somewhere around the 13th or 14th. After spending some time trying to work out what version was trunk at that time (I don't figure it out), I try to find the commit related to the bug. There isn't a link to the commit from the bug report. I look at the patches submitted on the bug report to find out the file most likely to change by the fix is nsChromeRegistry.cpp. Using humph's advice from the IRC channel #Education I use bonsai to find commits affecting this file between 2005-12-13 and 2005-12-15 (just in case the timezones are different between cvs commits and bugzilla).
Once the differences are found, and I still have no idea what gecko version this is, I compare the changes made to the source tar.bz2 - why not just check with the source repository for ThunderBird 2 releases? Well, that would assume it's documented somewhere, and in a place that could be found. Under the release notes for TB 2.0.0.23, the FAQ says the release code is in CVS, and to follow the build intructions. The 'TB simple build' on this page is for the coming-soon TB 3, and TB 2 just isn't mentioned. Since the release notes mention it's CVS, maybe the only cvs titled link works: 'Getting Mozilla Source Code Using CVS'. I can see a mention of how to get the TB 2 development branch, but no mention of how to get TB 2 releases. I follow over to CVS Tags and find that TB 2 was indeed using 1.8 branch, and it was made in September 2005 so a trunk commit in December 2005 won't help ThunderBird 2.
The checking of the source.tar.bz2 of the 2.0.0.23 confirms, this bug fix never came to ThunderBird 2. So what to do? The bug was fixed 4 years ago on a later code base; it's not a show stopper but extension developers involved or introduced to ThunderBird 2 have a fair possibility of running into it. So, I've done the only thing I can think of, and documented it on MDC that chrome registrations with mixed case won't work for ThunderBird 2 or SeaMonkey 1.1, in addition to the existing comment about FireFox 2, and used the warning template so people won't miss it. Unfortunately, the templates strip any formatting or links, which makes a link to the bug impossible, but at least there's a place to point for the next ThunderBird extension developer butting their heads against this.

Sunday, November 8, 2009

Forced fsck on Ubuntu 8.10 boot

During the boot I got one of those "filesystem has been mounted X times, time for a checkup" messages. I didn't particularly want that, I had other things to do, so pressed Esc to skip. It kinda hanged, and I may have pressed Esc again. Messages spewed across the screen, and Iwas forced to a shell prompt with an instruction to run fsck. Didn't want to do that, so rebooted. Dumped to shell prompt again with an instruction to run fsck. On running fsck, I get many screens of not-very-nice looking messages, and then this:
Buffer I/O error on device sda1, logical block 16777358
Error reading block 16777358 (Attempt to read block from filesystem resulted in short read) while getting next inode from scan. Ignore error? yes
Force rewrite?

Scary. Similar messages have been seen by others, and they've succeeded to get their system back by answering 'yes' to the above questions. Personally, this looks like the point where you will likely lose everything, so if there's anything you value, seek professional help. Answering 'yes' worked for me, but I have no idea what damage has been done. Time for a new hard drive.

Saturday, November 7, 2009

Building Firefox Round 2

Things are better the second time around? Rewind 8 months, I was struggling with a debug build of Firefox, nothing was working right. Well, the hint may have been in what I was compiling. I eventually got a build working, but looking back at that, I was building the wrong thing, I was building from cvs when it seems I should have been using Mercurial. Ugh. Makes some amount of sense, but looking at MDC I can see why I thought I was doing the right thing:
"Those doing active development can check out the latest source using CVS"
Which seems to contradict other information on the page. I'm always hesitant with editing this stuff; there are smarter people than me working on this, maybe it's right. I'm pretty darn sure it's wrong.

So, I've bought a new hard drive, installed the latest OS I favour, and starting fresh. I'm building from Mercurial looking at jbostons' wiki content on the build process for Ubuntu. and checking against MDC's build options info. It's been lovely. Here's some feedback I certainly didn't get last time:
checking for stdint.h... yes
checking for inttypes.h... yes
checking for sys/int_types.h... no
checking for GL/glx.h... no
configure: error: Can't find header GL/glx.h for WebGL (install mesa-common-dev (Ubuntu), mesa-libGL-devel (Fedora), or Mesa (SuSE))
*** Fix above errors and then restart with "make -f client.mk build"
make[1]: *** [configure] Error 1
make[1]: Leaving directory `/home/matt/Projects/mozilla/src'
make: *** [/home/matt/Projects/mozilla/src/ff-debug/Makefile] Error 2
What? It couldn't find a header and told me what to install? We are certainly living in the future.
sudo apt-get install mesa-common-dev
Running the debug build has also been a joy. I don't remember getting this much feedback at the console last time, I'm not entirely sure it's all good news:
++DOCSHELL 0xb11bbac0 == 5
WARNING: NS_ENSURE_TRUE(browserChrome) failed: file /home/matt/Projects/mozilla/src/docshell/base/nsDocShell.cpp, line 9888
WARNING: Something wrong when creating the docshell for a frameloader!: file /home/matt/Projects/mozilla/src/content/base/src/nsFrameLoader.cpp, line 1017
WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80004005: file /home/matt/Projects/mozilla/src/content/base/src/nsFrameLoader.cpp, line 1041
WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80004005: file /home/matt/Projects/mozilla/src/content/base/src/nsFrameLoader.cpp, line 197
pldhash: for the table at address 0xb11bbee8, the given entrySize of 48 probably favors chaining over double hashing.
++DOCSHELL 0xb11bbe80 == 6
But it's better the second time around.

Friday, November 6, 2009

Burying The Dead

Sigh. The time has come. I'll try to develop a more detailed post mortem, might create some content for a small bit of a thesis.
Partly in an effort to find a memory leak, way back in March I went about splitting the Bittorrent XPCOM work into separate reusable components. With hope, things like BirdTorrent could make use of these components, as the code is reliant on Mozilla's platform rather than any direct system calls.
Things attempted:
  • Splitting the work into parts eg. Server, PeerFactories, Torrents, Bencoding library, BitTorrent Wire library, BitTorrent Utility library
  • One step build
  • Testing of individual components
These didn't go well. Things didn't go well at all. The fact that the last release of something usable was February, and it's now November, isn't a particularly thrilling fact. Pretty much each task I've gone along with in that time has stalled at some point, so now my project has more loose ends than functioning ones. There's been times when I've thought it might get better, but I don't really have the support needed for this development (or not willing to ask for it, because it needs too much). It's damn irritating, but at this point I think fixing what's broken would take more time than just starting again.
My aim is to let the current work die. Start afresh, and as it goes I hope to pick through the bones for what may be used, without inheriting the bits that caused this collapse.

Wednesday, September 23, 2009

Wasted Time

I have no sympathy for plagiarists in education, they have literally wasted weeks of my life this year.

If you haven't learnt any damn thing well enough to do the work yourself, for what work will the degree aid you?

Wednesday, June 10, 2009

XPCOM Bittorrent Tracker IDL

Here's the working copy of the IDL I'm working on for the Bittorrent tracker component.


#include "nsISupports.idl"
#include "nsIVariant.idl"

[scriptable, function, uuid(a2a97e13-ddab-4a8c-a72e-45f3c26e5dce)]
interface nsIBittorrentInformation : nsISupports
{
nsIVariant get(in ACString key);
};

[scriptable, function, uuid(8bd92751-6566-4e8c-9d65-9a84195876d9)]
interface nsIBittorrentTrackerListener : nsISupports
{
void update(in nsIBittorrentTracker tracker, in nsIVariant information);
void onError(in nsIBittorrentTracker tracker, in nsIVariant information);
};



[scriptable, uuid(eb4f9895-e19a-4655-95d6-de801075e303)]
interface nsIBittorrentTracker : nsISupports
{
void init(in ACString announceURI, in nsIBittorrentInformation infoSource);
void start();
void stop();
// boolean forceUpdate(in boolean ignoreInterval);
void addTrackerListener(in nsIBittorrentTrackerListener listener);
void removeTrackerListener(in nsIBittorrentTrackerListener listener);
};


BittorrentInformation is just a simple map, the source for finding basic information about a torrent: info_hash, peer_id, port, uploaded bytes, etc. Perhaps this should be an interface with those attributes? At the moment, I find the thought of adding more and more interfaces for simple information a waste, but would be glad to hear differing opinions. I expect if the key isn't found, then an exception would be thrown.

The listener is for receiving the tracker updates, or errors if the tracker has issues. For example: a PeerManager may listen for tracker updates to record new peers.

I've thought about adding fancy extras to the Tracker, (eg. force updating, setting auto-retries) but this seems like predicting usage of the component before there's actually usage. Hoping to keep it simple.

I expect that the components implementing these interfaces will be completed this week. I'm also expecting to migrate new code from the current firewalled SVN repository to Google Code (Bittorrent4XPCOM).

Nice to be back at work.

Tuesday, April 7, 2009

Update - Refactoring Bittorrent for Firefox

Just a small update.

So, what's the state of the Bittorrent for Firefox extension? The last released version works, is fine for use on smaller torrents but needs thorough debugging to find some leaks that become quite bad on larger torrents.

So, what am I doing? Well, I've been caught up in the academic year; I've been tutoring, and will be lecturing in a few weeks. It's taken time away from some research (particularly some experiments) I'm doing using P2P for content delivery. Ooh! Fancy! Disregarding that, I've been refactoring the existing code so that components can be used independently of one another as complete XPCOM components. A previous post provided a Bencoding component for encoding/decoding data structures, and more recently I've been refactoring the Bittorrent Tracker client code. It's painfully slow, but should be nice. If you can see yourself using bittorrent XPCOM components, I'd love to hear the interface you'd like.

So, what to look forward to? The code for shutdown of torrent has been corrected, and will be released on request or once refactoring complete. If anyone wants to build a different Bittorrent client, they can look forward to using the refactored and separated components. In the future, I'd like to see the downloads appear in the Download Manager, instead of in the current version's Tools > Torrents menu (this way I can piggy-back on the status bar download progress without taking up more space, and it also makes managing torrents easier for the user), though if anyone else would like to write that, that'd be wonderful. ;-)

Friday, March 27, 2009

Question: Preferred Behaviour on Torrent Problems

What kind of response would you like if:
  1. The torrent file was corrupt
  2. The tracker doesn't exist (therefore can't download)
  3. The tracker may exist, but isn't responding (maybe can't download)
  4. There are no peers (therefore can't download at the moment)
  5. There are no peers accepting connections (therefore can't download at the moment)
  6. Peers won't share after, for example, a minute of waiting (therefore can't download at the moment)
  7. Given the peers we have seen in the last, for example, minute, no-one has the complete file we're interested in (therefore can't complete download at the moment)
  8. There is a problem saving to the specified destination
  9. Firefox is closing (there is the possibility of resuming when re-opening Firefox)
  10. The download is complete

Friday, February 27, 2009

Info - Bencoding (and XPCOM component)

Bencoding is used in the Bittorrent protocol to encode simple structures containing dictionaries, strings, integers, booleans, and lists in .torrent files and tracker responses.

.torrent (Metainfo file)
The contents of a basic .torrent file will identify the tracker server, that is later used to find peers, and also describes the info dictionary (or hash table, or associative array). The info dictionary plays a couple of roles.
It lists the files in the torrent, and their lengths. Secondly, it describes the concatenated binary data as specific sized pieces, and provides a checksum (in the form of a SHA1 hash) for each piece. Thirdly, the dictionary is re-bencoded and a SHA1 hash of the resulting byte array is used to identify the torrent.

Tracker response
The tracker response should either identify some error, such as the tracker not handling the SHA1 torrent identifier, or specifies information about the tracker and a list of peers who are also part of the identified torrent's swarm.

Implementation of Bencoding XPCOM component
IDL for Bencoding XPCOM component

Example:
var bencoding = Components.
classes["@wikiscraps.com/encoding/bencoding;1"].
getService(Components.interfaces.Bencoding);
// Encode an object into a string
var data = bencoding.bencode({
announce:"http://127.0.0.1/announce"
,info: {}
});
// Decode a string into an object
var obj = bencoding.bdecode(data);
More information on bencoding (Theory.org) Bittorrent Protocol Spec (BEP0003)
More information on the .torrent structure (Theory.org) Bittorrent Protocol Specification (BEP0003)
More information on the tracker response (Theory.org) Bittorrent Protocol Spec (BEP0003)

Monday, February 23, 2009

Fifth Release - Bittorrent for Firefox

This release may be a tad broken, as there was a push to get new features in, but then that was left to focus on reducing memory usage. If you want to check it out, do so and please provide feedback.
If you're looking for a user friendly or stable bittorrent handler: try this out once before uninstalling, give feedback on this post, and come back to the blog (bookmark it!) later this week. If I get feedback, I'll make a nice tidy release based on the feedback, just for you (Awwh). After that, I'm on holiday. Woo! Adelaide Fringe Festival!

20090223
What You Get
  • The opportunity to test.
  • Torrent data is now saved directly to disk. Probably want some middle ground there, but whatever.
  • Clicking a torrent gives you the option of saving it, passing on to the default download manager dialog, or canceling.
  • Tools > List Torrents lists torrents. It's in the middle of cosmetic changes, feel free to comment. There's a 20 second timeout on updating stats: I found updating XUL listboxes on torrents with hundreds of pieces gave an unpleasant UI experience.
  • Have the option of pausing - but it's only cosmetic.
  • Have the option of cancelling - but cancelling code is incomplete.
  • The first torrent started launches a server on port 6881. The next torrent launches a server on 6882, this continues with an increment of 1 for each new torrent.
  • Default to hide log messages, but show error messages. (Needs a code edit to change, not in preferences yet)
  • Pre-allocation of disk space.
  • Less memory usage (so it would seem).
  • When there's a single seed, the speed has been improved up to 4 times from the last release (results will vary wildly). This is likely to do with having implemented pipelined requests; now we keep 2-4 requests pending with the peer. Previously, we waited until receiving a response before sending a new request.

Fixes
  • Correct reporting of remaining bytes to Tracker. Previously, treated last piece as equal length to normal pieces.
  • Temporary fix for servers. Now we create a server for each torrent, starting at port 6881 and going up by 1 for each torrent managed.
  • If we find the peer that connected has the same peerid as our model, we close the connection. It's ourself! (Or a very unlikely occurrence) This is actually kinda hard to know without connecting.
  • Probably some other things.

Check Out
Here's some things to check out, that I've liked in the past. Test these with the extension!
Update 2009-02-24: The PBS interviews have 2 problems: 1) the content type is incorrect 2) the trackers don't know the torrents. Soooo, they won't work anymore for any bittorrent client (except maybe DHT clients, ooh, next feature?)

Wednesday, February 18, 2009

Progress? Firefox Debug Builds on Ubuntu 8.10

I expected to have released a new version, but on finally using the Bittorrent extension on torrents with data greater than 300Mb, I can see that the memory usage is quite horrid, and memory isn't released when downloading is finished.
So now I'm stuck in trying to debug to find why memory isn't released, which brings me to getting a debug build on Ubuntu 8.10.

Building on Ubuntu 8.10
DO NOT follow this guide. You'll be building an OLD Firefox. Check out jboston's user page; you MUST use Mercurial (hg) to get the latest source. I wish this had been made more clear on MDC.

It appears there is a new hoop to jump through to build on Ubuntu 8.10. A better guide is on jboston's user page; consider this another way to skin a cat. I initially followed dbranski's guide to building Firefox on Ubuntu, though this doesn't appear to work on Ubuntu 8.10.
I grab some packages:

sudo apt-get install gcc g++ perl make cvs libxt-dev libidl0 libidl-dev libfreetype6 fontconfig libcurl4-openssl-dev libgtk2.0-dev libdbus-1-dev libdbus-glib-1-dev 
I grab a make file that lets me grab the source I'll need:

cvs -d :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot co mozilla/client.mk

I've also grabbed trace-malloc for this debug build, not sure if this is the right way, but it's what I've done. Skipped when not making a debug build.

cvs -d :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot co mozilla/tools/trace-malloc

I get into the mozilla folder, and open up the .mozconfig file to configure this build.

cd mozilla
nano .mozconfig

I paste the mozconfig for a debug build into the file (found the text from Mozilla QA's Howto Debug Builds). I comment out the CPU line and the let line beneath it, and the line marked MOZ_MAKE_FLAGS (the let line caused a complaint, and I don't care about the 'make' tool running jobs in parrallel). I then add the lines:

export CFLAGS="-U_FORTIFY_SOURCE"
export CXXFLAGS="-U_FORTIFY_SOURCE"

Then save the file. The above addition is because Ubuntu has changed the default gcc flags, and the fix is due to Ori Peleg. Unoptimised builds are not effected. If you're using an existing mozconfig that doesn't explicitly turn off optimisation, add the above U_FORTIFY_SOURCE statements.
I then get the source I need:

make -f client.mk checkout MOZ_CO_PROJECT=browser

And then start the build:

time make -f client.mk build

To find out if I'm leaking, I set an environment variable to a file I want to log to then run firefox (learnt from Mozilla QA leak testing):

export XPCOM_MEM_LEAK_LOG=~/leaklog0001.log
cd firefox-debug/dist/bin/
./firefox -P

When I've done whatever I want to test, I then quit Firefox and change the environment variable so I don't overwrite the log file.

export XPCOM_MEM_LEAK_LOG=~/leaklog0002.log

If I have leaks, the file tells me what objects weren't freed and in what quantity. To understand the output, I use MDC's debugging memory leaks page. It doesn't tell me why though, and that's my next problem.

Sunday, February 15, 2009

Disliking First Steps in Debugging with Tools

I'm dealing with a component grabbing way, way too much memory. In the downloading of a Torrent of size 341Mb, the extension will cause Firefox to consume 2.1Gb of memory. Uncool. The most likely cause of the growth, in my opinion, would be the actual data being received.

I think it should be the case if nothing happens, not much memory will be eaten (though this isn't exactly true, I see a jump of 38Mb when my content type handler component is fired for the first time - this seems a bit much).

So, I'm trying to learn the ways of debugging memory issues in Firefox. I will admit, this is frustrating me more than I've been frustrated in some time. There are several options for finding memory leaks. So far, I've started down the route of gdb. I can start gdb, and that's about it. I was meant to set a breakpoint to watch for garbage collection, but that didn't quite work - perhaps I need a debug build.

I thought I'd try the leaksoup tool. To use that, I've tried a build with trace-malloc enabled. Failed, I get buffer overflows when running the build.

I've tried creating a debug build. That's failed, I get buffer overflows when running the build. Ugh. Perhaps I've installed the wrong pre-requisite?

I've tried just building. That's failed, again with a buffer overflow. Could be bad pre-requisite, or current trunk is bad? I've done another checkout, and building again.

I've tried setting the environment variable XPC_SHUTDOWN_HEAP_DUMP to a filename before running firefox from terminal, but that does nothing. (I shouldn't need a debug build for using this variable, according to https://wiki.mozilla.org/Performance:Leak_Tools#JavaScript_heap_dump )

I found something that looks pretty good - DumpHeap ( https://bugzilla.mozilla.org/show_bug.cgi?id=378261 ), but darned if I know how to call it. It seems to be concerned with finding objects in a particular scope though, and I doubt my problems are that easy.

I think any debugging route with a debugger (My current lists to try are valgrind and continue with gdb) would require the debug build to make things anywhere near easy, which requires me to resolve why I can't build anything. So I guess that's my current pain. I mean, plan.

Saturday, February 14, 2009

Info - HowTo write to specific location in a file using XPCOM

I'm writing torrents to disk, so a lot data is written to files out-of-order. During setup, I pre-allocate the file space by opening a file and moving to the position the last byte would be, using the nsISeekableStream interface. This technique I picked up from DownThemAll's manager class.
When I receive a piece of a file, I need to put it in the right place. What I was doing was moving to the right place, then writing:

seekstream.seek(0x00, offset);
stream.write(data, data.length);

This failed, somehow I was overwriting/truncating what was in the file. The solution was to first move to the end of the file, then move to the offset.
function insertInFile(filename, offset, data) {
var file = Components.classes["@mozilla.org/file/local;1"]
.createInstance(Components.interfaces.nsILocalFile);
file.initWithPath( filename );
var stream = Components.classes["@mozilla.org/network/file-output-stream;1"]
.createInstance(Components.interfaces.nsIFileOutputStream);
stream.init(file, 0x04 | 0x08, 0664, 0); // readwrite | create
var seekstream = stream.QueryInterface(Components.interfaces.nsISeekableStream);
seekstream.seek(0x02, 0); // move to end of file
seekstream.seek(0x01, offset - seekstream.tell() ); // move to where we actually want to write
stream.write(data, data.length);
stream.close();
}

I'll leave with a note that in my tests I've been seeing a byte with the value 0x0A at the end of my files. If I find that's to do with something I've done here, I'll try to remember to update this post. Just watch out for it if you're using the above method.

Upcoming later today - 5th Release of Bittorrent Firefox Extension with fixes Not happening. Latest version has a critical bug with reconnecting to it's own server.

Thursday, February 12, 2009

Upcoming Week

This is highly optimistic, as I can't get this much done. The issues that relate/contribute to my research have priority, which is pretty much the first point, maybe the last point, and a number of other items that no-one would be interested in, so I don't blog. Feedback welcome.
  • Log Socket Layer to Disk: need to record communications with peers for later scrutiny. Particularly when there's many peers, it's hard to decipher what problems have occurred.
  • Implement Pause: pause should put the torrent into a holding pattern: no more connections should be accepted or sought. Tracker should be informed downloading has stopped. Existing connections should be choked. Should pass "not interested" to all peers for the moment. If disconnects occur, that's okay. Unsure about keep-alives.
  • Implement Memory on Disk: create an abstraction layer, so that a block can be resolved to part (or parts) of the destination files for saving and uploading. This is so we don't need to store the whole torrent in memory at one time, instead we save what we receive as soon as it's confirmed.
  • Implement Memorising Torrents between Sessions: so we can close Firefox and resume later.
  • Implement Resuming from File Set: if we find the target files already in the destination folder, let's see if we can complete them against the torrent's description.
  • Implement Limiting Which Files Downloaded: this is partially done already, due to my other goals. Just needs a user interface.
  • One Server handles all incoming connections: Still not done.

Monday, February 9, 2009

Info - History for Firefox and Bittorrent

So, if you want to see integration between Firefox or Songbird and torrents, look in fear. Here's the history that current development can look back on.
Plug for current development efforts: Ashughes on a Songbird bittorrent extension leveraging libtorrent. I should plug my own efforts, but really, has anyone installed it for testing? I'm looking at enabling early writing to disk this week, which should allow large ISO size torrents without a significant memory drain, but there really needs to be more testing or such a size may never download due to other issues. To that end, the next release (Friday?) will have the ability to log the history with each peer, so we can examine what went okay, and what went bad. For now, tests on torrents at mininova.org's featured content have worked for myself up to 20 or so MB (tested on OS X & Ubuntu 8.10, FF 3); I haven't attempted higher, as I don't actually have any need of the content.

Moztorrent
http://moztorrent.mozdev.org/
Last update: was there a first update? Probably the first Bittorrent/Mozilla attempt. Had screenshots, and mockups appear to use download manager. Had a Mailing list (I can't see it ever being used by the authors). No source, so development dead-end, and no updates since archive.org started watching it in December 2004.

FirePuddle
http://firepuddle.mozdev.org/
Thanks to humph I've recently found this one from 2005. Result of a Summer of Code project, this extension worked with Firefox 1.5, but has a list of bugs that make it quite brittle. Lack of support for multiple file torrents would be the killer bug, as that's too common to ignore. (For bugs, see http://firepuddle.mozdev.org/installation.html)

Torrential
http://torrential.mozdev.org/
There is nothing here. The source committed is the source of the website.

Firestorm
http://firestorm.mozdev.org/
Last update: 2007 May 22 (website) May 23 (mailing list). Jump started by leveraging Mainline (Bram Cohen's original client) via PyXPCOM. Has Screenshots, and looks like seeding support. Looks like fair information feedback, and appears to have had right click support on torrent links. Looks like preferences exist (though probably not enabled). Source available (browse, download instructions) And a mailing list: Mailing list. So what's bad? No XPI (which means it can't be installed) and no updates in over year. See the last words on the mailing list - which, thankfully, were about the release of the source code.

Bitfox
http://code.google.com/p/bitfox
I wonder if this will be picked up, or if there is enough to pick up. Some C source files.

Foxtorrent
http://www.foxtorrent.com/ (Dead, blog still up)
The standard by which all other FireFox Bittorrent extensions should be measured. Reached maturity, and survived for some time. The killer is the disappearance of the Red Swoosh app which this extension relied on, which as far as I can tell, was never open source software. So this is probably a development dead end, as the only source is javascript that links to the Red Swoosh backend via http.
Hit 1.0 on 26th April 2007.
You can install a reviewed addon v1.13: https://addons.mozilla.org/en-US/firefox/addon/4844 (5th June, 2007)
Install a not-reviewed 2.0: http://foxtorrent.googlecode.com/svn-history/r236/trunk/foxtorrent/ (6 August, 2008 maybe)
Info on API for Red Swoosh: http://code.google.com/p/foxtorrent/wiki/RedSwooshTorrentAPIDocumentation

FireTorrent
http://www.wyzo.com/firetorrent/
According to TechCrunch and TorrentFreak, this Wyzo/Firefox extension is developed using P2P Innovative tech. Well, okay, that was in 2007, in 2008 TorrentFreak also reports it's using libtorrent. How can we tell? It has the appearance of open source, but damned if I can find the bittorrent source - and the only source code is a single snapshot. The source provided is either dated 26 March 2007 (based on the contents of the source zip) or 30 March 2007 (the log date on the subversion repository).

AllPeers
http://developer.allpeers.com/
Honestly, I know too little about AllPeers, but it looks the most promising for open source software. The company is gone, but the developer site seems to still exist. The site reports the server is closed source, but the client is open, though I don't know what that means for ordinary torrents. I think digging through this code is where I'll be spending some spare time.

Saturday, February 7, 2009

Early Fourth Release - Bittorrent for Firefox

Yay. Finally, a user interface for managing torrents: you can list torrents, view progress & file manifests. Feedback on using the UI would be handy. I'm looking at improving the ability of logging interactions with peers for future debugging, so next week's release I'm hoping for feedback on peer errors.

20090207
What You Get
  • Clicking a torrent gives you the option of saving it, passing on to the default download manager dialog, or canceling.
  • Seeding/creating a single file torrent from the Tools menu. Provides option of saving .torrent files, and adding a web seed (Info on web seeds).
  • Default seeding until reaching DL/UL ratio of 1/1.5, or no limit if creating a seed.
  • Tools > List Torrents lists torrents.
  • Have the option of pausing - but it's only cosmetic.
  • Have the option of cancelling a torrent. Should work, testing desired.
  • The first torrent started launches a server on port 6881. The others attempt to, but will fail.
  • See communications via the error console. It gets very noisy.

Fixes
  • Supports requests over piece boundaries.
  • If there is no need to choke, it won't. Previously, any leacher was choked after a certain amount of downloading, and then unchoked almost immediately if no other peers were pending.

Monday, February 2, 2009

Early Third Release - Bittorrent for Firefox

Yay. The Bittorrent extension for Firefox is almost suitable for users. You can use it! Though you'll have to quit Firefox to cancel/pause/stop torrents. Thus far this hasn't caused problems for me, as seeding has been pretty quiet. Probably don't want to do it on huge torrents, as the torrent is still kept in memory. This release is a day earlier than planned.

20090203
What You Get
  • Clicking a torrent gives you the option of saving it, passing on to the default downloader manager dialog, or canceling.
  • Seeding/creating a single file torrent from the Tools menu. Provide option of saving .torrent files.
  • Default seeding until reaching DL/UL ratio of 1/1.5, or no limit if creating a seed.
  • No way to manage (eg. stop, pause) the torrents other than closing Firefox.
  • The first torrent started launches a server on port 6881. The others attempt to, but will fail.
  • See communications via the error console. It gets very noisy.

Fixes
  • Corrected call to XPCOMUtils.categoryManager.addCategoryEntry. Since components are NOT registered every application opening, the 4th parameter needs to be true if we want to handle the bittorrent content type (mimetype) when the browser restarts.
  • All this time, the XPI has been badly structured. The components folder was being placed in the chrome jar, when it should be in the root of the xpi. A discussion with ashughes forced me to check the installation process and fix this issue. Everyone benefits.
  • Compliance with tracker's minimum update time.
  • Keepalive handler is being fired correctly.

Progress - Torrents to Disk

New - Choosing where to save
I've taken the path of least resistance in choosing how to save to disk. After opening a torrent file, I've used the nsIFilePicker for selecting a folder (use the constant Components.interfaces.nsIFilePicker.modeGetFolder when calling init), and then stored the returned nsILocalFile in a central location. Initially, only the PieceManager was informed of this nsILocalFile, however it's of greater value if the folder is known to all the components: the original .torrent is saved into the folder, and any tracker responses are also saved into the folder. This has made debugging easier, with the help of GHex, the Gnome Hex Editor.

Changes to Tracker
The tracker is the first port of call after a .torrent file is opened (hohoho). Trackers communicate via HTTP, responding to a well formed URI with a bencoded (Bittorrent specific encoding) data structure detailing various things about the tracker and other peers.
When I first started testing using the latest Ubuntu ISO, I ran into my first compatibility problem. There are a couple of ways of encoding the information on other peers, let's call them 'original' and 'compact'. Compact is specified by an optional query string parameter in the request to the tracker, and I wasn't providing that setting. As a result, the tracker didn't fall back to original, it instead rejected the request with a plain text reply, along the lines of get a new client. Unexpected, particularly when users won't see that response, and also being terribly useless for developers. Fixed by adding compact=1 and handling the compact encoded peers.
The latest change was for friendliness. The tracker provides minimum timeouts on tracker requests, which were being ignored.

Little Things
Saving files sucks. There's all the checking for paths existing, checking if a file can be created, etc. Saving with nsILocalFile rocks. The create method for nsIFile (the interface nsILocalFile extends) automatically creates paths. What if you've been told to save in a folder, but it already has a file with the same name? The createUnique method means you don't have to write the associated conflict resolution logic (unless the intention was to replace that file - the nsIFilePicker modeSave automates the asking of that question if you need it). Additionally, a .torrent file lists the download files with their paths. Recreating that folder structure can be error prone - if there were ".." components, it's possible the saving could end up somewhere it shouldn't (eg. writing into a folder mentioned in PATH environment var in Windows). Though I need to test to confirm, I use appendWithRelativePath on nsILocalFile to build the path from the initial save directory to the specified file . This should, I need to test, remove the unintended path exploitation. If it doesn't, oh well, we'll fix that.

Next: Early 3rd Release - with Saving goodness!

Progress - Testing Torrents

I'm updating because of the progress on downloading of a torrent to disk, the first complete results being saved a few hours ago (Woo hoo!). There's been mountains of changes in order to get this to happen, so the next couple of posts will be the build up to how the first torrent saved to disk.

Changes to Test Torrents
First, the testing grounds have greatly changed. Previously my tests have been on the latest Ubuntu ISO and on MIT's The Structure and Interpretation of Computer Programs Lectures. This last week I've changed specifically to small multiple file torrents, to which end shaver recommended a couple of the confidently legal downloads on http://www.mininova.org/.
These torrents are noticably different, particularly the treatment by peers; connections to peers are rarely accepted (presumably because each client is already well connected to the swarm); unchoking is comparatively rare. Unchoking is where the peer you're connecting to says it's okay to request blocks to download.

Changes to Peer Manager
Someone may recall I was only opening one connection to a peer and shutdowning(-ish) when that connection died. With the more 'pop' torrents mentioned above, this isn't enough for testing. The probability of hitting the one peer that accepts connections is pretty low. (How low? Not sure, I can see how those graphs and charts in Azureus could have come out of testing questions)
So, the Peer Manager is finally doing more than holding Peer objects that do nothing. Now, it attempts to connect to all the initial peers provided by the tracker, and performs a health check every 30 seconds to see if we need to attempt more connections. There's a timeout before re-trying peers, currently 3 minutes though I need to find the standard for this.

Changes to Piece downloading
Previously, I'd written so a Peer object (representing a conversation with a Peer) had a list of pieces we 'wanted' that we knew the peer had. The first 'want' would attempt to be downloaded, and when finished the data would be passed to the Piece Manager. This isn't good enough. Too often a disconnect or similar will happen before the full piece is downloaded, meaning many peers have partial pieces they can't release to the Piece Manager.
To correct this, partial pieces are now managed by the Piece Manager. This means any work on downloading a piece will be immediately useful, repeated downloads will be reduced thus the total download reduced, and seeding will occur earlier (this I want to enhance further - if a single piece is targetted first, we can start sharing earlier, which improves trade negotiations ;-) ).

Next: Saving Folder, Tracker Changes, Little Things

Thursday, January 29, 2009

Request for Requests

It's kind of early, as the extension isn't useful or usable to anyone at the moment (next week!), however if you want the extension as either an end in itself, or as a component to enable torrenting for your extension, please offer up some request on what you want.

At the moment, it's pretty pointless installing the add-on, but someone might have an idea of how or why they'd use it, so some UI or preferences requests would be applicable, or some IDL request.

I figure if people use the extension, they'll find bugs and that helps me with my own purposes, so I'm giving a bit of my time to fulfill requests to get a bit of help. :-)

There is some possibility of making an IDL for ashughes, and I'd like to work on that with you ASAP, if you're interested. Asaik the IDL would abstract what is needed from a torrent handler. I'm thinking it'd provide a connection state (IE. whether or not satisfactory connection with other peers), torrent state (IE. whether the swarm has all pieces), start/pause/stop methods, open method that takes the contents of a .torrent file, and a piece listener (analogous to nsIStreamlistener, except data comes in out of order).

Upcoming Week

  1. Add saving to folder, when used as a content type handler (IE. When you click a .torrent file, you'll be given an option of a folder to save to). This will probably be an inefficient save, made when the torrent has completed downloading. That means, all the download is still in memory, and the storage used by JS will mean that's probably more than twice the size of the torrent itself.
  2. Separate log and reportError commands, and enable each to be turned off with a global const.
  3. Change the current protocol handler (x-http2p://) to x-bt:// (it's shorter)
  4. Make a map of file types to content types, to allow the x-bt protocol handler to return a content-type when streaming.
  5. Correctly inform the model of current uploaded bytes and downloaded bytes (these are pieces sent and received), and current sent bytes/received bytes (these are total bytes sent and received).
  6. Add to piece manager to make an easier calculation of bytes left (of the pieces left).
  7. Use uploaded/downloaded/left to update the tracker.
  8. Create a torrent health check event, firing on a fixed interval.
  9. For leeching/seeding behaviour: Will check if average bytes received in the last X seconds is less than Y% of the experienced max average in the last Z second period. If it is, try to resolve by dropping or choking selfish peers (that we are seeding to, and we're interested in, and we've sent more than received, but haven't unchoked us in B seconds), connecting to new peers, or even upping the allowed connections on the server port to encourage fresh connections.
  10. For seeding behaviour: Will check if we have reached our seeding limit, if not infinite. Check if we're maxed out on peers. If we are, see if there are any lazy peers: connected but not interested; connected, interested, unchoked, but not downloading. Drop them.
  11. Add a shutdown to the model, the piece manager, the peer manager, the peer, the server, and the event pump. For the moment, it should disconnect if required, then clear all references except those that would be needed to indicate shutdown state (IE. probably want a _shutdown attribute, if a public function is called and this is set to true, throw an exception). The event pump should refuse any new events - if it's found an event would be useful during shutdown, allow only shutdown events to be added.
  12. Fix so all torrents use the same server port. This will likely be very difficult.
  13. Send correct IP address to tracker. This will likely be very easy.

Wednesday, January 28, 2009

Second Release

Again, code that is pretty much unusable as is, except for creating a Seed and '.torrent' file.

20090128
What You Get
Clicking a torrent will give a choice whether to 'download' the torrent. The download will not be saved to disk. Next time :-)
Clicking Tools > Create Seed allows the creation of a single file torrent, including setting of the tracker. It also provides the choice of saving the '.torrent' file and/or starting to seed immediately. A server socket will be opened, but unfortunately the IP address reported to the tracker will be incorrect - this doesn't matter so much, as trackers like OpenTracker ignore the IP address reported, and use the IP address the report came from.
Still only bothers to open one peer connection, and when that connection dies, doesn't recover. (This was pretty much left as-is, because it is easier to debug one peer connection than many).

Fixes
  • Bencoding dictionaries has been corrected. Previously, the keys weren't sorted; this is incorrect behaviour, they needed to be or bdecoding fails in Mainline BT (and probably any other bencoding library)
  • Bitfield messages from peers has been corrected. Was reading from the wrong end of the byte, eg. Given a bitfield 00000001, it was interpretted that the peer had piece 0, not piece 7.
  • If a peer had only one piece of interest, receiving the unchoked message did not start the download of that piece. This has been corrected, but I think needs some more work.

Progress - Seeding Working

Yay. Seeding has been tested, and we can successfully operate as a server.
Additionally, the http-replacement protocol (see tasks from last week, Friday item 11/12) worked, with a bit of hardcoding, and successfully returned an image. I'll put together the XPI and source package for downloading later today.
In summary of progress on last week's tasks, 1-11 and 14 were worked on. I'll continue on the skipped items this week, and draw up new tasks. Expect downloading to file, and storage of seeding data on disk instead of in memory.

Saturday, January 24, 2009

Progress - Setup

Testing the seeding action is a bugger. In order to test a seed, the client must already have loaded the torrent (at minimum a single piece, at best random pieces), then a leech must connect and download.
So, in order to make life somewhat easy (funny, it doesn't sound easy), I'm going to make a tool to create torrents.
When it comes to test a seed, I can open a dialog, select a file and specify a tracker. If I want, I will then be given the option of saving the .torrent to disk. Now I will have a seed, and have a .torrent file I can pass to a leech to test with.
I can't think of an easier way, without introducing another bittorrent client into the picture, which is something I'm mostly trying to avoid.
EDIT (Jan 26): To clarify, I'm happily connecting to other clients, I just don't want to run another bittorrent client on the same machine at this time.

Thursday, January 22, 2009

Progress - Timeouts

In order to add timed tasks, such as keepalives to maintain connections with peers, I thought I might add timeouts; XPCOM provides nsITimer, and through nsIAppShellService access to the well known setInterval/setTimeout functions via the hidden DOM window.
However, I disliked the idea of running 30 or so timeouts for each torrent, instead I thought I'd re-use the existing event queue by introducing a new method called 'addDelayedEvent(event, delay)' which adds events to an ordered-by-event-time queue. The idea being, at each step this queue is checked, and any items at the front of the queue that should have fired will be added to the normal event queue.
So, now we handle timed events not by many extra timeouts, but by adding to a new ordered queue which feeds into the normal event queue. Somewhat satisfying.

Initial Release

So, here's some code that doesn't do much.

20090122

What You Get
On clicking a torrent link eg. Ubuntu 8.10 386 iso a prompt is fired. If responding with a Yes, the torrent will start 'downloading'. Downloading at this stage means connecting to the first peer and downloading sequentially until A) there is no more to download B) an error occurs C) peer disconnects.
There is not seeding, there is not error handling, there is not tracker updating, there is not saving or delivery of the download - it stays in memory.
Plans for the next week are here.

Sunday, January 18, 2009

Upcoming Week

Sunday Jan 18th - Wednesday Jan 21st : Actual, real holiday.
Thursday Jan 22nd - Saturday 24th: Maybe holiday. If it isn't:

Thursday
  1. Analysis of code produced to achieve these goals, in mind to get feedback and to ease maintenance: Logical order; Structure; Commenting (just want to remove anything too embarrassing, there's been too many late nights recently).
  2. Release code to here.

Friday
  1. Add functions relying on timed event, specifically:- updating the tracker; keep-alive with peer.
  2. Buffer on write fails: a write fail should result in a new rewrite attempt in the event queue.
  3. Update peers with HAVE messages on successful download of a piece. (Fire an event, let the event handler deal with the distribution of HAVE to peers)
  4. Start responding to INTERESTED. Fire an event, the handler should check how many are currently unchoked, and if a threshold is not reached, unchoke. Maintain list of INTERESTED & unchoked, and INTERESTED & choked.
  5. If a REQUEST msg is fired on an unchoked peer, write the request back. After writing it back, examine history. If this is a peer we are actively exchanging with, keep unchoked. If this is a peer we are not actively exchanging with, increment _sent. If _sent is greater than threshold, write choke back, remove from INTERESTED & unchoked, add to end of INTERESTED & choked. Fire event that will allow INTERESTED & choked to be re-examined, and a new peer to become unchoked. Fire this event on peer disconnected too, after cleanup (ie. after peer is removed from INTERESTED & unchoked or INTERESTED & choked, if they're in either)
  6. If a NOT INTERESTED msg is fired on an unchoked peer, remove from any INTERESTED list it is part of, and then choke.
  7. Open the Mozilla server socket.
  8. Accept connections. Add timeout to drop connections that have timedout.
  9. Listen for handshakes. On Handshake, send Handshake. Send BITFIELD. Then act as normal.
  10. Set up tracker on test server.
  11. Resolve x-htp2p URLs to multiple .torrents. Treat file in torrent as the data from that URL.
  12. Resolve x-htp2p URLs to single .torrent, then try to resolve URL to file in that torrent. Treat that file as the data from that URL.
  13. Make limit on PieceManager so that only pieces related to target URLs are downloaded.
  14. Make PieceManager talk to return channel, so that as Pieces are verified, they are written out.
Saturday
  1. Do what wasn't achieved Friday. If it all was, have a beer. Maybe even have a TSD. Congratulations, you're ready for the next supervision meeting.

Friday, January 16, 2009

Purpose

This is the obligatory statement of why a blog exists.

I'm writing code, and have a compulsion to explain my pain. I'm doing that here, so as to relieve the desire of doing it on IRC. This may also benefit others who are experiencing the same pain, and happen to phrase it in a Google query that gets such a blog entry.

Secondly, it provides a focal point in my software development that is more communication and exposition friendly. Expect version announcements, and thoughts that may become rants.