I want to crawl reddit for a certain keywords and ...
# markops
s
I want to crawl reddit for a certain keywords and then push to slack alert if any post is found for the same. Any idea how to go about this? I tried PRAW, but it feels too much work 💀
👍 1
r
chatgpt suggestion: If you prefer to avoid using PRAW, you can set up the automation using a combination of other tools and services. Here’s a step-by-step guide: ### 1. Use Pushshift API for Reddit - Pushshift API: Pushshift is a free API that provides access to historical Reddit data, including comments and submissions. It’s a good alternative to PRAW for searching Reddit. - Fetch Data: You can use Pushshift to search for keywords across Reddit posts and comments. Example of a Pushshift API call:
Copy code
bash
   <https://api.pushshift.io/reddit/search/submission/?q=your_keyword&size=10>
### 2. Set Up a Slack Webhook - Incoming Webhooks: Slack allows you to create incoming webhooks, which are a simple way to send messages to a Slack channel. - Create a Webhook URL: Set up an incoming webhook in Slack by going to your Slack workspace’s App management page and creating a new app. Once set up, you’ll get a Webhook URL. ### 3. Write a Python Script - Fetch Data from Pushshift: Use Python’s
requests
library to fetch data from Pushshift. - Send Data to Slack: Use the Webhook URL to send the message to your Slack channel. Example Python script:
Copy code
python
   import requests

   # Pushshift API URL
   pushshift_url = "<https://api.pushshift.io/reddit/search/submission/?q={}&size=10>"

   # Slack Webhook URL
   slack_webhook_url = "<https://hooks.slack.com/services/YOUR/WEBHOOK/URL>"
   slack_channel = "#your_channel"

   # Keywords to search for
   keywords = ['keyword1', 'keyword2']

   def search_reddit_and_notify():
       for keyword in keywords:
           response = requests.get(pushshift_url.format(keyword))
           data = response.json()
           for submission in data['data']:
               message = f"Keyword '{keyword}' found: {submission['title']} - {submission['full_link']}"
               send_to_slack(message)

   def send_to_slack(message):
       payload = {"channel": slack_channel, "text": message}
       <http://requests.post|requests.post>(slack_webhook_url, json=payload)

   # Schedule the script to run periodically
   import schedule
   import time

   schedule.every(10).<http://minutes.do|minutes.do>(search_reddit_and_notify)

   while True:
       schedule.run_pending()
       time.sleep(1)
### 4. Deploy the Script - Run Locally: Test the script on your local machine. - Deploy on a Server: Use a VPS, cloud function (like AWS Lambda), or any other service that can run Python scripts at regular intervals. ### 5. Monitor and Maintain - Logging: Implement basic logging to capture errors or issues. - Scalability: If you need to scale, consider running the script in parallel or optimizing the search frequency. ### Tools Summary: - Pushshift API: For retrieving Reddit data without using PRAW. - Slack Webhook: For sending notifications to Slack. - Python + Requests Library: For making HTTP requests to Pushshift and Slack. - Scheduler: To run the script periodically. This setup allows you to monitor Reddit for specific keywords and receive notifications in Slack without relying on PRAW.
s
F5bot will do the job
👍 1
k
r
I've used a tool called Syften in the past for this
p
We used feedcomber that provides hosted bot that runs the script