Monday, March 21, 2011

Minor Adjustments to the Scoring Algorithm

I was analyzing the data collected and I came across some profiles that were correct but had given a rather low score by my algorithm, mainly because the 'Education' field in the profile wasn't set.

So I modified the scoring algorithm as follow:
  • if 'Davis' is in the profile, gain 1 point
  • if 'engineer' is in profile, gain 2 points
  • if 'UC Davis' is in profile, gain 3 points
  • if 'University of California, Davis' in profile, gain 4 points
then,
  • if 'University of California, Davis' is in the education field, gain 3 additional points,
  • if then 'engineer' is in the profile, given that the previous statement is true, gain 3 additional points,
  • if the first name matches what is in my input database, gain 1 point
  • but if the first name contains the word 'Davis' then lose 2 points (Davis can be a first name as well as last name)
  • similar scoring for the last name.
This scoring convention should spread out even more the profiles narrowing down the number of false positive and especially false negatives.

Sunday, March 13, 2011

Some Interesting Statistics

Of all people searched:

~ 17% returned no result
~ 64% returned multiple result
~ 19% returned a link to a single profile [EDIT: previously: had profiles that were not publicly available]

I suspect there might be an error though because 0% of searches returned a direct link to 1 profile, while an earlier search suggested that at least some searches returned that kind of webpage.

It might have been linkedIn removing that kind of webpage, but I will need to look into it.

Edit: Ok I found what the problem is. Linkedin.com moved completely that public profile category of searches to the 'no-public-profile available' category of searches.

I think this actually plays to my advantage because now I can still assign a likelihood score to that entire category and write it to the database.

The Newline

The only cross-platform problem I have encounter is the newline char.

In Windows it is '\n' and on Unix it is '\r\n', which caused my database to have too many lines with not aligned correctly.

It was an easy fix: on opening the files I remove the last char from each line plus I replace each '\r' with '', so that I can use that line on each operating system the same. I also check on exit that the final database does indeed not have any '\r' in the middle of a line, and if it does, I replace them all with '' and rewrite the lines in the file.

On the Unix machine the program runs under 4 hours (3 hrs and 22 minutes) when the output is redirected to a file (called LOG.txt), thus it is much faster than on my machine.

Now I just need a driver for my program that takes the arguments and flags from the command line and modifies the internal variables of my program. That shouldn't be very hard, I'll do it during finals week or so.

Saturday, March 12, 2011

The Subprocess Module

I think I'll use the subprocess module to automatically verify those 404s errors. I'll open each of the link in a new tab in chrome, possibly from the computer lab, and manually verify most of them. If none of them actually contain data, I wouldn't think there's need to check all of the others.

A weird thing happened now. The program kept stopping saying it couldn't access one of the files it was supposed to access regularly. I think it's an antivirus feature that it is causing my program not to have repetitive access to my files. Alternatively it might be some other program running on my computer trying to access the same file. Weird anyway. It didn't bother me for about 5 hours and then all of a sudden I can't open a file a few times in a row.

I moved the execution of the program to a linux machine, so I can also test the portability my program. I might choose to let the program run there all night and check on it tomorrow morning if I don't have to have my computer on as well.

anyway, I hope it doesn't freeze over there neither cause that would mean there's some other kind of bug in my program, but I really don't think it will happen.

The Latest Project is Running Fine

My Latest Project is running fine.

I am constantly checking for bugs and checking the other parameters to make sure the program runs well.

I also checked the database that the program is building. It seems that a bit less than a half of the people have LinkedIn accounts. Some people have more than one too, and for now I am saving all links to the profiles that correspond to the same person.

I am trying to refine the output now so that I can collect more statistical data on the final database. All of this data seems very interesting to me. I want to know exactly how many people have these accounts and how well I could identify them with my scoring technique. I could then plot the score distribution and see if it is anything better than a Gaussian Distribution. Or at least see where the mean is.

With that data I could then make a better scoring method, or higher the minimum score for a profile in order for it to be considered useful.

Also, storing the score in the database would help whoever needs to use the database after me to do more filtering in case they don't want to send out as many emails.

Performance in Computer Architecture

An off-topic thought:

In my computer architecture 2 class we are talking about performance in using caches, data prediction, branch prediction, and so on. Two main types of prediction are used, static, and dynamic. These kinds of prediction are very important as bad predictions could ruin the performance on any CPU very very significantly (as much as 80-90% less performance).

The static kind of prediction is done by the compiler and basically the compiler tries to identify how to resolve hazards, how to fill up the caches the best way possible and how to predict branches.

The dynamic kind is done by the hardware. The hardware uses the current information it has about the state of the CPU and the past information about running the program to more accurately predict branches and fill up the cache with the right things.
If done right the dynamic prediction is much more accurate than the static one because the hardware at run-time has access to registers, caches, etc that the compiler does not have access to.

My thought is: Python programs are both compiled and then interpreted. Couldn't Python use some static prediction during compile, and software-based dynamic prediction when it actually run the program?
More importantly the interpreter could store in the python-object-code the prediction information that arise from running the program the past times. This way the future times the program is run, the program will use the past info to run faster and to fill up the cache better.


I can see a time when interpreted languages as Python will run faster on simpler CPUs than compiled languages on the same CPU (or more complicated). The limitation of the compiler is inherent to its nature, and the difficulty of dynamically predicting things in hardware is also inherently difficult.

Python could use the research done in compiler prediction optimization, while also using the research done in dynamic prediction optimization.
It would take compiled languages a shift in paradigm to achieve the same benefits.

LinkedIn Started Sending 404s to me

Which I wasn't handling at first. I suspect they found out that somebody using python was getting scraping information on their websites and so they decided to save bandwidth by just sending me 404 errors whenever the search did not yield any result.

I tested it on my browser and the same link/queries yield a page saying no results, so I'm quite confident they are only sending me 404s when appropriate. I should find a way to test this for all links that yield 404s though, it shouldn't be too hard, I have a log after all.