Metrics Server-Collection 0.3 Release

Well, that was quite a project. I went in knowing nothing so, there was a lot of research effort needed to get even the littlest things done. My project basically required me to learn Python, the Django Framework, http request/response, MySQL, and database design. Although I knew some SQL and database design, the project really required me to think and re-learn a lot of the material I never really thought was that important.

Django is an awesome framework to get things done quickly, so I strongly recommend it to anyone who wants to do average database work and page implementation. Python is an odd language, it is nothing like the other languages I've learned; I mean how was I suppose to know what a "tuple" was; its basically a list for those who want to know. Hm... what else? Http was definitely the hardest for me to grasp; even though the concept is so simple. There's just not enough of this being taught in my courses and I feel that a lot of the hardships other students are going through comes from the same problems; not enough concepts being taught vs code.

Anyways, for anyone interested in getting some cool user data and putting it into a database, check out my project wiki and keep me posted on what you think. As far as project contribs go, I did that canvas3d test, got some useful favicons for a colleague, help brainstorm on solutions for the OpenID http probs, and sent some C# code to the distCC project (although I got no confirmation back on that). It's getting real late, so goodnite.

Favicons for Firefox

Well, I just finished contributing to the Favicon Project. Favicons for Firefox are basically icons you can use to represent sites; very similar to the ones you see next to the URL in the address bar. I think the idea is kind of neat since it allows users to customize their browser even more. I gave the project owner some nice icons, some of which I think will be useful - like Wikipedia, MySpace, YouTube, and other famous site icons. Hopefully, they all work properly and can be of some use. Anyone interested in the project or want to contribute some more can check it out at the projects wiki; My Favicons Extension.

Canvas 3d

Well, I just got around to testing the canvas3d extension on my home computer. There were some complications since I had like 5000000 different installations of Firefox on my PC that I lost track of the one I installed it on. However, in the end I got it displaying and I'm glad. There was one thing I did find a bit odd, in class when Cathy was testing the installation her teacup was upside down; but, on my computer the teacup is right side up. I wonder if there are conflicts with the way canvas3d is displaying 3d content on the monitor. I know that sometimes this happens when my driver set-up goes wrong for my games. Anyways, here's an image of what the teacup looks like.
If anyone else wants to test it out and haven't found a chance to, download the canvas3d extension and try one of the html page tests.

First XPCOM chrome

Well, I just finished the xul overlay for the firstxpcom extension we created in the previous lab. It wasn't actually that difficult; however, for those of you who regularly download a nightly; like I do. You need to make a few modifications to the lab. Here are some quick fixes I needed to make to get everything working properly.

1. When using Ted's extension creator, make sure you have the right max version. The lab says, "Maximum Version=3.0a3pre" this for me is not correct; I build version 3.0b2pre.

2. If you followed the labs instructions, the chrome extension doesn't seem to appear in the addons list (after extracting the needed files into the ../extensions/firstxpcomchrome@senecacac.on.ca), try renaming the .zip into a ".xpi"; which will allow you to drag and drop the extension into your browser for installation.

Those are basically all the needed steps I took to get the extensions working. Here are some screenshots of the results.

First XPCOM Extension

Well, I finally did it. Here are the screenshots to my new firstxpcom extension which doesn't do much. However, I would like to start editing it so that one day it can actually do something; even something small.

The lab was quite interesting, although I'm curious to know whether or not binary extensions be installed on other peoples machines without building. I know some are probably not able to without some work, but can this one?

Oh! and here's the zip with the extension.

Metrics 0.2 Released

Well, my o.2 was finally released. It was a very painful process getting some of the Django framework to work properly with my database layout. I don't know why they haven't fixed a lot of the issues with database relationships; you would think that those bugs would be the first ones they'd jump on. However, I guess there are other issues to resolve?

For those who don't know what my 0.2 is about, the release basically was to create the database infrastructure for the metrics data that I would be receiving from the metrics extension (firefox browser). The database structure was created using the Django framework; which as I already posted makes Web and database work easier. The framework basically helped me create all the MySql tables, an administrator page w/ logins, search functionality, and methods to help with the CRUD.

My 0.3 release will now be creating a python script that uses the database models I created in my 0.2 to parse all the data from client into the database. Hopefully, this will be a semi easy task. I don't think parsing will be a problem or even getting the data; however, I'm not 100% sure if I can just use a python scripts to do all these tasks ( I'm a completely new to python). Although, some people of the Django community told me it would be possible and I will try to take their advice and get the ball rolling.

For anyone who wants more information on the project or try it out for themselves, just install all the necessary apps for Django Installation and do the setups for your platform then head on over to my wiki on zenit and get the source code; please use the nightly. Then unzip it into any directory, load up a shell, and use "python manage.py runserver" in the root directory of my project ("you_directory/metrics/").

Django - the "D" is silent

"Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design".

I am quite new to different frameworks and this is probably the first Open Source framework I've actually used for anything. I've heard that there are other frameworks that offer similar things - like Ruby on Rails; however, I've never actually used any of them, so Please keep in mind that my opinion may be a bit biased.

Django tries to keep everything automated and adheres to something called the Dry Principle. It makes life so much easier when creating database designs that I'm pretty impressed. From what I've experienced, it can create full database structures, administrative logins/web pages given object specifications (similar to normal OO Programming Syntax); on top of that you can design urls with minimal code.

Here is an example of an object model (taken from the Django tutorials) used to create the database:

from django.db import models

class Poll(models.Model):
question = models.CharField(max_length=200)
pub_date = models.DateTimeField('date published')
class Admin:
pass

As you can see from the example, that is how you would create a table called "Poll" with two fields - question and pub_date. Now you can login to the administrator page (already automated) and add as many new polls as you like; simple right? You could even use python shell or a script, and create it like you would a class object. Like this:

p = Poll(question="What's up?", pub_date=datetime.datetime.now())

If you ever wanted to create a simple website or blog, you should check Django out. Try a few tutorials and see what it has to offer for yourself; if you have any trouble login their irc #django IRC channel on irc.freenode.net. There's a lot of nice people there who are more than willing to help out.

Metrics Project Contributions

As I have explained in class, my project requires some contributions. Currently, I require some people to test the patch I created on different platforms (linux, mac) and install the extension to test the server script.

Testing the patch only requires someone who uses or has one of the platforms listed above to build firefox with the metrics extension. All details on how to do this is on this blog (click here). I only require one of each, so that would be two people. Please leave a comment below if you're interested; or email me at my student email located on the school wiki (here).

Testing the server requires someone to build the extension with firefox and change some preference settings so that the extension will automatically start and send data to my server (click here). Once the extension has started (you can check by typing about:metrics in the address bar), you should see some xml style data, you just need to use it for about 5 mins. After around 5mins, the data in about:metrics should be reset and that means the data was sent; this is where you can disable the extesnion in the preferences.

Thanks to those who are willing to contribute. Please leave any comments or questions for the above.

My Experience at FSOSS 2007

FSOSS 2007 was a great learning experience! Before I attended the event, I dreaded the idea of coming to school on my week off; however, the knowledge I gained was irreplaceable. Although I'm still a student, I create a lot of little applications in many different languages for class, and I have always loved the design aspects. I mean, why do we develop applications? I think that many people get so involved is the code or the next best feature they thought of that they forget who the application is for. The average user doesn't care about what happens behind the scenes; they care about how easy and suited the application is for them; which is why I enjoyed the presentations that Jay Goldman/David Crow and Mike Beltzner gave.

Mike Beltzner presentated "Product and User Experience Design in Open Communities", and Jay Goldman and David Crow presented "Usability Anonymous: A 12 Step Program for Better User Experiences", both of which I thought was quite interesting. They kept the audience a bit more lively and made the experience fun. Both presentations were about the user, and tips in creating better software for them; I think they both mentioned the same book, "Design Patterns" by Jenifer Tidwell. However, each presentation took a slightly different focus. Mike Beltzner discussed much of building a community and letting the users get involved, while Jay Golman and David Crow talked more about understanding the user in design decisions. Although I liked the slides that Mike Beltzner gave (very colorful), I thought I gained a little more from Usabiltiy Anonymous. Sorry Mike!

I was even registered for the Drupal workshop which I didn't attend since I wanted to go see the presentation on Facebook, but it wasn't what I thought it would be about. The presentation was great and they presented a lot of interesting concepts provided in Thrift, an open source cross-language development framework, but my brain must have been on off because it got a little too technical for me. On another note, facebook does provide cool little programming challenges every once in awhile and those can be fun to solve.

In the end, I think I liked my time at FSOSS. The food was okay, the presentations were fun, but the little things that were gained were more than worth the loss of two days of my break. Thanks again to all those who participated in making the event go as good as it did!

My O.1 Experience

What is the metrics extension you ask? Well, the general idea of the metrics extension is to collect browser usage data. Although the metrics extension is not yet complete, they want to keep track of UI interaction, Usage of Features (bookmarks, extensions, themes.....), Memory usage/effectiveness, system configurations, and other related subjects. It will not however, collect any personal data that is identifiable to the user; everything will just be appended to a non-identifiable unique ID to distinguish different browsers.

For most of my 0.1 release, I have to created a small server script to the metrics extension. This basically involved my server script to recieve requests from the metrics extension and do the following - recieve uploaded data, or send back a config file. Currently, the data sent to the server is just uncompressed and saved into a file to read manually; however, in the future, it will be stored in a database infrastructure and hopefully used to display the data in many ways. For a better understanding of my o.1 release please check out it's wiki - Metrics Server Collection Wiki and to get a working version of the extension please follow these instructions How to enable Firefox Metrics extension.

The experience I had working on my project was quite different from most of my other classes at Seneca. The project required me to work with a massive existing code base, creating patches, debugging, server scripting, and http response headers. The experience was something else and I believe I have gained a better understanding of how my future career experience will be like. In addition, the project has also given me the opportunity to learn about working in an open source environment and the real idea behind community. I would suggest that all student at least try to do something in the open source community once.

How to Enable Firefox Metrics Extension

Before reading this guide, please make sure that you understand how to Patch and build Firefox. The steps will basically assume that you have knowledge of building Firefox and will skip basic information. If you don't understand how to Patch or build Firefox please refer to the "Resources" below.

Basic steps to install & enable the Firefox Metrics Extension

1. Checkout a nightly build from the Mozilla CVS.

2. Patch the metrics extension with this patch
--Metrics Patch--

3. Add the following to the .mozconfig

ac_add_options --enable-extensions=metrics

4. Build Firefox.

5. Go into the "dist/bin/extensions/metrics@mozilla.org/defaults/preferences" folder and edit the metrics.js. It should look like this.

pref("metrics.upload.enable", true);
pref("metrics.upload.uri", "http://matrix.senecac.on.ca/~sljung/metrics/metrics.cgi");
pref("metrics.event-count", 0);

6. Start the Firefox build.

Resources
MDC Build Guide
Guide to Build Firefox in Windows (Blog)
Seneca Lab in Patching
Guide to Patching & Building (Blog)

Watching a Bugzilla User

For my weekly assignment, I decided to watch Vlad for a week. It was quite interesting to see how many e-mails I would get on a daily basis. In fact, I recieved emails faster than and in more quantity then the spam that goes to my mail. From it, I believe that my user, more or less, watches specific types of bugs. Most of the e-mails I recieved about bugs were on things related to graphics. Not sure if that's the right term; however, most of the subjects were on things like SVG, APNG, CGBITMAPS, canvas, etc....

I actually found most of the bugs interesting and hope that probably in the future I may be more involved in trying to fix them. For now, I will basically do research and learn more there is to all things Moz.

Firefox Extension - File I/O

For my blog post today, I will briefly explain how to write data to a file in a Firefox extension. This can be used for error logging or whatever your purpose. One of my projects has required me to produce XML documents on the local machine; therefore, I have used it for that. Please remember that this is all done using javascript.

The first part of the code basically needs to find the extensions file path. To most, this is within the mozilla profile folder. This is needed if you want to keep the file in the same folder as your extension. Otherwise, you could leave it out or give it a different file path. Id is used to hold the id of my extension.

//get extensions file path
const id = "sample@foo.net";
var extension = Components.classes["@mozilla.org/extensions/manager;1"]
.getService(Components.interfaces.nsIExtensionManager)
.getInstallLocation(id)
.getItemLocation(id);

The second part of the code is used to check whether or not you have the proper permissions to write a file to the local machine. I have not thoroughly tested this though.

try
{
netscape.security.PrivilegeManager.enablePrivilege("UniversalXPConnect");
}
catch (e)
{
alert("Permission to write to file was denied.");
}

Next, I create a variable to hold my file path (just easier for me); this is done using the extension path that we found earlier. Then create the proper components needed to write the file. First we need nsILocalFile, which to my understanding is like creating an instance of a file object. This needs a path which is basically the filepath I created. Then you need to create it.

//create proper path for xml file
var theFile = extension.path + "\\" + "metrics.xml";
//create component for file writing
var file = Components.classes["@mozilla.org/file/local;1"]
.createInstance(Components.interfaces.nsILocalFile);
file.initWithPath( theFile );
if(file.exists() == false) //check to see if file exists
{
alert("creating file...");
file.create( Components.interfaces.nsIFile.NORMAL_FILE_TYPE, 420);
}

Now, we create an output stream; nsIFileOutputStream, and give it the proper modes. As shown in my comments, I have chosen to do only write, create file if not exists, and truncate which basically deletes the content (leave this out and add 0x10 if you want to append to a file).
Then, you write to the file; giving the content and the length of the content, and close close the stream.

//create file output stream and use write/create/truncate mode
//0x02 writing, 0x08 create file, 0x20 truncate length if exist
var stream = Components.classes["@mozilla.org/network/file-output-stream;1"]
.createInstance(Components.interfaces.nsIFileOutputStream);

stream.init(file, 0x02 | 0x08 | 0x20, 0666, 0);

//write data to file then close output stream
var content = this.xmldoc;
stream.write(content, content.length);
stream.close();

For more information on File I/O you can find it at
Mozilla Developer Center


Also check out the XPCOM reference at XUL Planet
. This is basically where I go to research each component used.

Reading Mozilla Source Code

What can I say? Reading the Mozilla source code seemed to pose as a large scale challenge. Before I ever looked at the source code I was pretty confident with my programming knowledge. However, after looking at the source code (cpp files in particular) I was in awe. All the use of defines, typedef pointers (i think?), and extern C? made me run around in circles to understand just one method which to me seemed would be easy. The javascript and XUL was pretty straight foward and easily understandable so I was able to grasp some of what was happening. For those of you who really want to get involved with the code I would suggest reading the Mozilla structure and Styles in our weekly course schedule.

All in all, the basic idea that I could acquire after checking out the source code is that Mozilla works on 3 parts. The C++ code does all the actual work for most of the part, the javascript is called by different interactions with the UI, and the XUL files are basically used to display what you see on the screen. Please don't misunderstand me, not every process requires all 3 sections to do something however some actions cannot be done by javascript alone.

Here are the links to some of the information listed there:

Mozilla Source Code Directory Structure
A good way to understand the directory structure, however I like to use it as an index to navigate around MDC

Mozilla Style Guide
Portability Rules for C++
These two links were probably the most useful for me by giving me sample code to enforce the information

What Linux distro are you?

Have you ever wanted to try a new OS? Ever since I started getting into programming I've always wanted to try using Linux but never got around to it or have been a bit intimidated by the idea of not using Windows because it ran all the programs that I use. The only real contact that I've had with linux was probably using the school's servers for my assignments or playing with a thing or two on my own webhost. However, recently I've decided that I will never be able to experience what Linux is like until I install it and use it as my own platform. After doing some research, I realized that the choice wasn't so easy and that so many different distributions of Linux existed. So when I finally sat down I asked myself a question, "Which distribution is for me?" Seeking an answer I went to my good old friend, the Internet, and came up with this interesting site

>> Linux distribution chooser <<

Firefox 3

For those looking to work on Firefox 1.9 Trunk, you can use CVS to download the source code in the same fashion as my post on "Building Firefox in Windows".
Basically, you just run the start-msvc8.bat located in the c:\mozilla-build directory (this directory is installed when you run the Mozilla Build setup package 1.1).
Then when the MSYS console window opens you just need to:

Create a new directory in c:\

mkdir ff3project

Navigate into the directory

cd ff3project

Login to the Mozilla CVS mirror and get the make file

cvs -d :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot co -r HEAD mozilla/client.mk

Download the source code using the make file

make -f client.mk checkout MOZ_CO_PROJECT=browser

I personally chose to keep the project in a seperate directory from my project. However, the choice is really yours.

Fore more information on CVS tags check out CVS Tags at developer.mozilla.org.

Building Firefox in Windows

I don't know if I was doing anything wrong or maybe if I had chosen the wrong set of build options, but building Firefox from source code does take pretty long on my pc. The whole build process took about 90 minutes of my life and 1.2GBs of hard drive space. I never thought that the files would be so large, so I had to start deleting stuff while it was compiling. Well, here is what I did to get it built. I was able to do this with cygwin and msys.

**NOTE: You will need about 2GBs of HD space before you begin. 250MBs for the cvs source, about 100MB for the tools (could be less but I had cygwin pre-installed), and 1.2GBs for the code?

STEP 1.
Download and install Visual Studios 2005 (referred to as VC8 on Mozilla sites).
Seneca students should be able to download and install this from our ACS site.
Click here for ACS
Then download and install the Mozilla Build Package 1.1
Download Mozilla Build Package 1.1
The package should install everything into c:\mozilla-build directory by default.
Once these two things are done, run the start-msvc8.bat in the c:\mozilla-build directory which will open up an application window with MINGW32 (referred to as MSYS).

STEP 2.
Now that the MSYS window is open (it will look something like command prompt), navigate your way back to the root c:\ - by default the application root for me was somewhere in my c:\document and settings\user\ which isn't allowed for the build procedure because of the spaces between the directory name.
Once you are in c:\ do the following in the MSYS console:

Make a new directory called proj.

mkdir proj

Navigate into proj directory.

cd proj

Open a connection with mozilla cvs server.

cvs -d :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot co mozilla/client.mk

Navigate into the new mozilla directory that was created by the cvs.

cd mozilla

Download the source code from CVS. This might take a little while so be patient. It took me about 5-10 minutes.

make -f client.mk checkout MOZ_CO_PROJECT=browser

*** You can easily change the MOZ_CO_PROJECT tag to download different projects or versions. Check out the mozilla dev site for more info or Click Here

STEP 3.
Once all the source code is downloaded. Create a ".mozconfig" file in the c:\proj\mozilla directory and fill it with your build options. Mine looked something like this:

# Options for client.mk.
mk_add_options MOZ_CO_PROJECT=browser
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/obj-@CONFIG_GUESS@

# Options for 'configure' (same as command-line options).
ac_add_options --enable-application=browser
ac_add_options --enable-debug
ac_add_options --disable-static
ac_add_options --disable-optimize

Once that is done type:
make -f client.mk build MOZ_CO_PROJECT=browser
This will build the source code. I would suggest you go do something else unless you have one of them top of the line PCs - this process took my about 90mins.

STEP 5.
Once the source code is built, hopefully with no errors, you should have a directory called "obj-i686-pc-mingw32" or something similar. Make sure any Firefox browsers are closed before continuing.
To run Minefield (Firefox 2 or 3?), run firefox located in c:\proj\mozilla\obj-i686-pc-mingw32\dist\bin

After a minute or two you should have a working build of Minefield.



If you would like to make an installer, you can type make installer in the c:\proj\mozilla\obj-i686-pc-mingw32 directory.

**NOTE: If Firefox is already running on your desktop running Minefield will only open a new window, so please close Firefox before running the application.

You are now COMPLETE!

The Cathedral and the Bazaar by Eric S. Raymond

I am impressed! The ideology behind open source (referred to as a "babbling bazaar") allows users around the world to take part in the development of software. Why does this method work so well? Just as Eric S. Raymond suggests, I believe that allowing others to contribute in more ways than filling a comment card gives them a sense of ownership; not in a literal sense that they own the software but that they are co-developers in the process. This model makes it more personal on the users and in return gets them more committed to seeing its success. The power of community also allows people from all around the world with many different expertise to work together and solve bugs; Eric S. Raymond's quote could not have said it any better - "Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix obvious to someone."

Furthermore, giving the users the ability to control where the software is headed by releasing it bit-by-bit is a great way to give the users what they want. Releasing software only at its final stage like the Cathedral model that Eric S. Raymond describes does not allow users to have much say in anything that they will be using. I am not saying that the Cathedral model is bad, however open source allows me to add some flavour.