Views:
107
Votes: 5
Tags:
support
data-explorer
data-dump
Link:
🔍 See Original Question on Meta Stack Exchange ⧉ 🔗
URL:
https://meta.stackexchange.com/q/372300
Title:
Use cron to get SE Data Explorer data when it's updated
ID:
/2021/11/27/Use-cron-to-get-SE-Data-Explorer-data-when-it_s-updated
Created:
November 27, 2021
Edited: November 27, 2021
Upload:
May 12, 2024
Layout: post
TOC:
false
Navigation: false
Copy to clipboard: false
According to this answer from 2012, the SE Data Explorer data is updated every Sunday at 3am. If true, is it possible to schedule a cron job to run every Monday and execute this query?
When the query finishes it would need to download the results in CSV format somewhere.
More specifically - is there a simple way I can run a query against SEDE directly from the terminal?
Concerns
I imagine it’s a little complicated getting cron authenticated by Stack Exchange Data Explorer. I’ve never run cron with my own User ID. It always runs as sudo either every 15 minutes to execute updatedb or via scripts in /etc/cron.daily, /etc/cron.weekly and /etc/cron.monthly. SE always wants authentication when I run a DE query as a “normal” user. I have never used cron for signing into a site, running a query, waiting for results, clicking a “download” button, and logging out. It sounds horribly complicated giving cron an X11 session, a GUI Desktop and a browser.