I thought that BFS would be more parallelizable, because each level in the tree has to be managed by methods with different methodologies, but until I get to that DFS is the way.
All basic features are down, and I'm building the database now.
Because of the exponential distribution with which I query the website that has mean 4 seconds, I'll have to run the program for at least 9 hours.
It was fun doing this project. I learned a lot of things, and I gained insight on how a project should be developed.
Looking back I think I have some more refinements that I should implement before releasing it to the open source world.
Overall the program is flexible and it wont miss a record or it wont search a record twice, not even if the program is terminated and the restarted. I save in a file all records fully searched so that next time the program is resumed it can skip what has already been searched. This is also good to avoid duplicates in the final database.
No comments:
Post a Comment