API Rate Limit Changes

TLDR

SMAT is introducing new rate limits to the public API to counter abuse. These include limits on the number of free public queries a single IP can make per day, as well as the window of data non-authorized users are able to see. We are now only allowing free users on public access to see collections that are at least 6 months old.

If you’d like to have unlimited access to our API or the ability to run queries over all of our entire live collections, please get in touch with us at info@smat-app.com.

Background

As SMAT has continued to grow, we’ve wanted to get a better understanding of our use base and a sense for our daily active users (DAU). Since introducing mixpanel to our public API, we’ve been able to monitor our tools’ anonymized usage. What we found was somewhat unsurprising: a handful of users are taking advantage of our free public services in a few ways, namely by:

• Using SMAT in their own products without citing, crediting or supporting us

• Continually running the same search terms and rotating IP addresses to avoid detection

• Spoofing headers to inauthentically appear as if origin traffic is coming from our resources

Beyond the fact that resource intensive queries reduce load capacity for authentic users, misuse of our tools also presents SMAT with a few concerns:

• We want to be appropriately credited for what we have built and continue to create. We are proud of our work!

• We’d like to be able to accurately measure DAU, but doing this we need to filter out inauthentic traffic, which unfortunately comes at the cost of reducing our tools’ investigative capacity for authentic users. 

• We want to prioritize authentic user requests, whereas the scale of abusive traffic we face tends to deprioritize them. 

• More funds spent on covering the hosting costs of malicious traffic means less funds for expanding our collections, innovating, and developing new tooling. Running SMAT costs money!

To give readers a sense of scale, we are estimating that ~80% of our traffic is coming from the same user(s) attempting to bypass our rate limits.

New Limits

To counter abusive traffic, SMAT is introducing some new limits to our API:

• First, we are introducing a cap of 39 requests a day per IP address. 

• Second, we are limiting the visibility of our more recent collections. Any requests from unknown users will only be able to access data that is at least six months old.

Conclusion

Though these changes will affect overall functionality (even for legitimate users), they are unfortunately necessary to protect SMAT from malicious abuse and to ensure that our tools are available for authentic users who need them. 

If you’re an authentic user who will be negatively affected by these changes or if your use case requires more recent data, please drop us a line at info@smat-app.com. In alignment with our values, SMAT’s mission is to get our collected data in as many hands as possible.

Finally, SMAT is still collecting live data. Our team works hard to ensure that our crawlers stay live, and in the coming weeks, we hope to produce public visualizations and dashboards to reflect that. Stay tuned throughout the year for more updates and robust solutions to your API needs!