r/pushshift • u/Pushshift-Support • Jun 20 '23
Pushshift Live Again and How Moderators Can Request Pushshift Access
Dear Reddit community
Earlier this month we shared an update about our collaboration with Reddit to grant access to community-enabled moderation tools developed through the Pushshift API, which would be reinstated for approved Reddit moderators. Today we are updating you that Pushshift is live again and sharing how moderators can request Pushshift access.
Note the process outlined below will be contingent on moderators registering for Pushshift accounts if you don’t already have an account. Each moderator will also need explicit approval from Reddit and the use of Pushshift will be limited to moderation use cases only. This will enable moderators to effectively use these tools to enhance community moderation and enforce guidelines, while protecting the privacy and data security of Reddit's user base.
Eligibility Criteria
- Reddit will prioritize requests from mods of reasonably sizable communities with consistent, rule-abiding engagement.
- Moderators or communities with a history of Content Policy or Code of Conduct violations can impact eligibility.
Steps to request Pushshift access
- Submit modmail to r/pushshiftrequest using this link. Please include the following details in your request:
- Which communities do you intend to use Pushshift for?
- What types of moderation activities do you require Pushshift access for?
- You should receive a message in your inbox from r/pushshiftrequest within one week after your request has been submitted. The message will indicate whether your application has been approved or denied. If approved, your moderator username will be shared with Pushshift for verification.
- If your request has been approved, sign into Pushshift at https://api.pushshift.io/signup using your Reddit account to retrieve Pushshift API keys.
Announcing Pushshift Search
Pushshift has added a search page for authorized users to make it easier for mods to use pushshift. To use it:
- Log into your pushshift account at https://api.pushshift.io/signup
- If verified, you will be redirected to the search page
- Search away!
Data has been Backfilled
Data has been fully backfilled and up to date. No data should be missing.
Getting support
If you are experiencing issues with Pushshift or have any questions, please send a private message to u/pushshift-support.
To help direct members of the Pushshift community to gain API access, we have put together a guide for approved moderators.
We are excited about this partnership to support the Reddit community. Thank you again for your passion and continued support!
Sincerely,
Pushshift and the Network Contagion Research Institute
26
u/Btan21 Jun 20 '23
Are there any plans for allowing non-moderators access to the Pushshift API?
17
u/Pushshift-Support Jun 20 '23
We are working together with Reddit to explore ways for other audiences to access Pushshift. We will keep you posted!
→ More replies (1)
3
u/9-T-9 Jun 23 '23
Can you share any updates for allowing access for academic uses? It's urgent to my situation and would love to be able to use the Pushshift API for my research.
3
u/Pushshift-Support Jun 30 '23
It’s not available at the moment -- but we are actively working with Reddit on solutions to provide access to academic researchers. We will keep this community updated!
→ More replies (1)
8
u/bizude Jun 21 '23
This sounds like a nice step, but many of the tools which we used won't work "out of the box" without some changes. For example, I used browser extensions which incorporated pushshift.
Will Reddit assist porting some of these tools, if the source is available? Some of these devs are no longer maintaining these tools because pushshift is mod-only now.
7
u/shiruken Jun 21 '23
Could you provide clarity on whether data ingest practices have been changed as a result of the Reddit partnership? For example, if a user deletes their comment on Reddit, are you now mirroring that deletion on Pushshift?
2
u/exposecreepsandliars Jul 04 '23 edited Jul 04 '23
I was hopeful for a little bit, but after trying the official tool (https://search-tool.pushshift.io/), I'm kinda speechless.
-
you can't see what post comments were left on
-
you can't see the body text of any posts
-
you can't see which subreddit anything was posted to
-
nothing is linked back to the original content on Reddit
And to add insult to injury, the dash bug is still a thing months later, making it impossible to effectively search for any user with a dash in their name. How is this not one of the top things to fix?
How am I supposed to effectively moderate with this?
2
u/Pushshift-Support Jul 06 '23
Hey there! Thanks for letting us know about these issues. We've worked to resolve them and all but the dash bug have been fixed at this time. We'll be updating ASAP once this is fixed too. Sincerely appreciate your patience!
→ More replies (8)
5
u/danke-Empire Jun 20 '23
Will you still be providing the SSE Stream API using this new bearer token authentication? If so, what does this look like in practice — is the websocket connection valid for the lifetime for the API key (24 hours)?
→ More replies (1)
5
u/shiruken Jun 21 '23 edited Jun 22 '23
For anyone that's interested, I forked one of the search websites and added support for the new authentication token: https://shiruken.github.io/chearch
Edit: Chearch now supports the authentication token
→ More replies (3)4
u/randomthrow-away Jun 21 '23 edited Jun 21 '23
I do like how yours keeps the API token (I'm guessing stored in a cookie/browser storage) the other one you referenced that you forked seems to require putting it in every time the page is visited which is a bit of an annoyance so I have a preference on using yours (which I've made a quick bookmarklet for to pre-populate some of search fields.)
I like that you've also added the "Score" alongside the Sub and Username line.
The only hiccup/glitch I'm seeing with yours is the "thumbnail expander +" button is offset creating a new line with a bunch of extra whitespace (or rather grey space) so things are a bit more spread out because of that.
https://adhesivecheese.github.io/chearch/ on the left
https://shiruken.github.io/chearch/ on the right
as a comparison of a little bit of a potential tweak just to condense the data and not have that extra blank space :)
https://i.imgur.com/pzNlrLH.png
(screenshot is highly censored screenshot due to NSFW content and username.)
If you could tweak the layout a little more to prevent that extra blank space then it would be perfect! :)
I found the tweak required, I believe it's the
#results .button { margin-top: 25px; }bit within the custom.css file that's affecting it which isn't in the CSS on the original.
The original has a 2px margin-top from the
.button.is-danger.is-small { font-size: 8px; margin-top: 2px; margin-left: 5px; padding: 6px; background-color: #f56565; }But the extra #results .button class in yours is overriding that, so simply removing
#results .button { margin-top: 25px; }Altogether seems to correct the issue without changing anything else. :)
→ More replies (4)
10
u/HTC864 Jun 20 '23
What types of moderation activities do you require Pushshift access for?
Glad it's back in some form, but I was expecting a little more automation. They really want thousands of people sending messages for this?
14
u/Qudit314159 Jun 20 '23
It's probably an effort on Reddit's part to nominally provide it while making it as useless as possible.
1
u/luisadoamaral Jun 25 '23
can we consider a similar thing for researchers with institutional emails and official letters from their institutions
or anything else that could convince Reddit that I need access too
→ More replies (3)
1
u/Future-Trillionaire Aug 04 '23
just received access. is there currently any way to see the body text of a deleted post?
2
3
u/HerrX2000 Jun 21 '23 edited Jul 11 '23
As someone who used Pushshift for research in the past. I would greatly appreciate if the scientific community world also get access. At best the data dumps would be available on some uni server behind some research network, so only authorized researcher would have access.
→ More replies (4)
3
u/rebutv Jun 30 '23
so complicated, i miss the old day when i could easily search thing through a single website without all these hassle
2
u/Glad-Acanthaceae-467 Sep 13 '23
I am a university researcher who used to use pushift to get reddit data for my project. After changes, i duly submitted my academic application to reddit and got an autoresponse and…. Eternal oblivion from reddit… how can i still get access to pushift data? My research, my work -everything is in that data! And i am poor as a … academic …
25
u/Watchful1 Jun 20 '23
Is there no way to get a longer term api key or an automated way to get one? Any automated tool using the service for moderation tasks would need to be manually updated every day with a new key. That makes any bots that use it nearly worthless unless someone is willing to update them daily.