I'm working with Go Fish Digital to help me understand more about what we can do to improve our SEO. As part of their work, they need some of our data.
They need crawl logs from the following bots:
- Googlebot mobile
- Googlebot smartphone
- They want about four days worth of raw data of the activity of these bots on our sites.
- The data shouldn't be older than, say, one month, but it doesn't necessarily need to be the four days immediately preceding today.
- Data format should be gzipped TSV.
- I'll get a public key using 2048-bit RSA to transmit the data to them securely.
- It's possible that they'll come back to us and ask for more or less data, so it'd be good if you could be prepared for that.
- I believe these bots use specific user agents which should make your job a bit easier.