I'm working with [[https://gofishdigital.com/|Go Fish Digital]] to help me understand more about what we can do to improve our SEO. As part of their work, they need some of our data.
They need crawl logs from the following bots:
* Googlebot
* Googlebot mobile
* Googlebot smartphone
* Bingbot
* Baiduspider
* Yandexbot
Requirements:
* They want about four days worth of raw data of the activity of these bots on our sites.
* The data shouldn't be older than, say, one month, but it doesn't necessarily need to be the four days immediately preceding today.
* Data format should be gzipped TSV.
Notes:
* I'll get a public key using 2048-bit RSA to transmit the data to them securely.
* It's possible that they'll come back to us and ask for more or less data, so it'd be good if you could be prepared for that.
* I believe these bots use specific user agents which should make your job a bit easier.
They have signed a master service agreement which fully covers our privacy policy, data retention, and data security requirements, and the agreement received signoff from Jim Buatti (in Legal) and Toby (the Chief Product Officer), amongst others.