Saturday, March 12, 2011

DFS done! The program is running and the database is being built!

Ever since I found out BFS wasn't a viable option I have been wondering how should DFS be carried out. Turns out that the DFS I have to run for this project is not very deep and it will actually save me a lot of resources as I don't have to store the webpages on my hard drive.

I thought that BFS would be more parallelizable, because each level in the tree has to be managed by methods with different methodologies, but until I get to that DFS is the way.

All basic features are down, and I'm building the database now.
Because of the exponential distribution with which I query the website that has mean 4 seconds, I'll have to run the program for at least 9 hours.

It was fun doing this project. I learned a lot of things, and I gained insight on how a project should be developed.

Looking back I think I have some more refinements that I should implement before releasing it to the open source world.

Overall the program is flexible and it wont miss a record or it wont search a record twice, not even if the program is terminated and the restarted. I save in a file all records fully searched so that next time the program is resumed it can skip what has already been searched. This is also good to avoid duplicates in the final database.

No comments:

Post a Comment