There’s a couple things that needs to be addressed here with Amazon major changes lately.
First of all, storefront scraping is greatly reduced from the beginning which you can get 1 million products to only 100,000. Now I believe it is only 4000 products. You can find out yourself by going through 200-ish page and see that Amazon doesn’t show any more products:
I know this is frustrating, but nothing we can do to get around of this updates.
Second of all, Amazon looks like enforcing updates to even newer storefronts, but only gives 200 pages for regular users to scrape.
So I suggest you use regular storefront URLs to get more data, not the ajaxified URLs. We also have improved the Dig Mode algorithm so it will cope with the new updates, not only getting subcategories but also the list of brands on very large lists. However, Amazon also limits this as you will getting numerous duplicates over the pages.
These updates make it harder for you to get very large list, so in order to handle this situation I suggest you use filters and reduce the number down to less than 10,000. It’s safer and you will not get as many duplicates.
Another update which I noticed that Amazon limits storefront displaying to a certain pages. Here’s an illustration of how it does:
Here’s an example of a storefront with 170 pages. However, if you browse through 69 pages, you will get to the end of the results.
However, despite this limitation you can squeeze more by enabling Dig Mode. I’ve checked and able to get 13K out of 45K on this storefront.
If you need more assistance, please submit a ticket to our support team.
