|All requests||>||What is the best PHP url crawler class?||>||Request new recommendation||>||Featured requests||>||No recommendations|
by Dan Beck - 7 years ago (2016-02-21)
I need a crawler which can extract all pages metadata and URL into a database.
This class does exactly what you ask. It can crawl Web site pages and extract links and metadata.
It does not crawl the whole site automatically but the example script demonstrates how to do it calling the class recursively.
There are other classes that crawl the site recursively but they do not extract the pages meta data.