The News feed from ATN…

The German National Library on the way to experience the harvesting of its TLD .de

In parallel to its selective crawls, the German National Library plans, in collaboration with Internet Memory, to collect .de.

For a first snapshot (before Summer 2014), the German National Library aims at harvesting 100 Tb of resources from its TLD .de.
This project has a experimental character and depending on the results, the National Library will go further (extend and/or put in place regular global crawls).

At Internet Memory, we are very excited about this new project.
To operate it, we will use our crawler specifically designed and developed for web scale crawls: this proprietary crawler, MemoryBot, collectes hundreds of millions of resources per day.

Stay tuned to be informed about the results.