What we learned from Google crawling (and indexing) 1.14-billion accidental URLs
Dan Taylor
- Organisation:
- SALT.agency
- Region:
- Anywhere
- Notice Period:
- Short (maybe less than one month's notice)
- Type:
- Professional
- Fee:
- Expensed
- Category:
- Business
- Updated:
- 9th October 2024
We know from time to time that Google’s crawling and indexing behaviors can be problematic, and even contradictory to themselves. In 2023 we encountered a case were Google had managed to crawl two staging environments and crawl 1.14bn URLs caused parameters on a website with only ~6k URLs. It even managed to index a large percentage of them. Whilst troubleshooting and unpicking this mess, we uncovered weird Googlebot behavior and some learnings we’ve been able to apply to eCommerce sites with 000s of SKUs and high product turnover.
Views: 24 | Enquiries: 0About Dan Taylor
Dan Taylor is an SEO consultant and growth marketer with experience in improving the digital presence of companies, especially in eCommerce, SaaS, and travel sectors, including Cloudflare, Gitlab, TCGPlayer, Travel Supermarket, and Airbnb.
Based in the UK, Dan was recognized as one of the top SEOs by Search Engine Journal from 2018 to 2020 and named one of the top 20 technical SEOs by OnCrawl.
His work in edge SEO earned him the 2018 TechSEO Boost Research & Innovation Competition award.
Dan is known for his SEO consultancy and contributions to the industry as a speaker, writer, and knowledge sharer.
He has spoken at conferences like BrightonSEO and TechSEO Boost and conducted seminars with the Department for International Trade and Google Digital Garage.
Dan also writes for Search Engine Journal and Search Engine Land and engages with the SEO community through webinars and organizing events like the New York SEO Meetup.
Other Talks on SpeakerNet by Dan Taylor
Send a message to the speaker
If you are interested in this talk and wish to contact the speaker, please complete the following form: