Making the most of SMAT's streaming public data API (Telegram, Parler, 8kun, etc.)
NOTE: We are a small org so we cannot handle aggressive crawling (unless you substantially back our compute!). For larger datasets please contact us and we can work something out!
Most know us for the open-source free online visual tool, SMAT for data analysis. However, we also have an incredible API full of stories waiting to be found for those with more technical needs. The API is ideal for not just researchers and activists, but also journalists, as covered here in our last Substack. Here’s an example Colab/Jupyer notebook to get started. Check out this SMAT SDK as well:
Some of the sites you can query with data services from our API are:
A recent update added a link to the API in the top menu bar of the main SMAT app page as follows:
The API can also be found at api.smat-app.com/docs directly.
The API has access to the raw JSON behind all of our front-end tools and can be useful for developers and analysts who want to dive deeper into the data or make more fine-grained queries. The API has three endpoints and has an optional boolean logic query for the Content endpoint. If all this sounds too complex don’t worry, we make it easy with the interactive documentation.
Example Content Endpoint Query
First, click the Content endpoint and then click “Try it out”.
Choose Telegram as the Site, and then click Execute. It will give you a regular URL link that will have the raw JSON content of your query. The interface will look like this:
If you Curl or go to the URL you will get a raw JSON that will look like the following (certain browsers like Firefox will automatically “prettify” the JSON to make it easier to read):
Example of the same request using curl from the command line and jq to pretty print:
Everything is nested in “hits” and “_source” for all of our data as you can see but the key fields for Telegram for most use cases are:
views (note: views count is dependent on when it was crawled)
NOTE: All datasets are structured slightly differently than each other. You have to analyze the structure of the JSON individually.
If you need more data or more advanced interfaces for analysis please reach out to us at email@example.com
As always, you can support our efforts (in large or small ways) to provide open access to important data services and tools through our Open Collective page.
Thanks for your interest and attention!
- SMAT Collective