Uploading Stack Overflow dumps to archive.org

In my last post about the Stack Overflow data dumps, I mentioned that, while StackExchange Inc. was no longer uploading dumps to archive.org, that task had been taken up by a member of the community. This remained true through the March 2026 data dump, but I discovered last week that the June data dump hadnʼt made it there even though it was already mid-August, so I decided to take up the mantle.

The reason Iʼd been using the unofficial Internet Archive copy of the dumps rather than the official downloads is mentioned in my previous post: itʼs a six-step manual process that requires clicking around in a browser, and doing that four times a year for the 64 sites that Feep! uses would be very tedious. Doing that to archive all 362 sites in the Stack Exchange network was a non-starter; clearly I needed some automation.

Fortunately for me, Olivia has published se-data-dump-transformer, which does exactly that: using Selenium browser automation, it automates the process of logging into each site, clicking through the settings page, and downloading the dump. I cloned the repository and followed the “Using the downloader” instructions to set everything up and get it going. Thankfully, I didnʼt encounter any of the Cloudflare captcha issues the documentation mentions, but I did find that this program is (understandably) pretty hacky and had a lot of weird corner cases.

For one thing, it crashes when it canʼt click the things it needs to, and I soon found it did that frequently, seemingly at random. I quickly found that I needed to pass the --skip-loaded flag, which tells it to skip any sites which have already been downloaded: otherwise, every time I restarted it, the process would start over from the first site on its list and I would never make any progress.

I also found that it would intermittently crash with an exception:

selenium.common.exceptions.ElementClickInterceptedException: Message: Element <button id="datadump-download-button" class="s-btn mt16 s-btn__filled" type="button"> is not clickable at point (471,676) because another element <main id="mainbar" class="flex--item fl-grow1 user-show-new settings-page"> obscures it

This <main id="mainbar"> element is simply the form that the button is inside of, so I donʼt understand how it could be obscuring its own child, but apparently it somehow did, sometimes. It would succeed when I restarted the process, but this was annoying enough that I figured out a workaround instead of hammering retries:

diff --git a/sedd/main.py b/sedd/main.py
index 7b9f121..c4dad89 100644
--- a/sedd/main.py
+++ b/sedd/main.py
@@ -1,4 +1,5 @@
 import traceback
+from selenium.webdriver import ActionChains
 from selenium.webdriver.common.by import By
 from selenium.webdriver.firefox.webdriver import WebDriver
 from selenium.common.exceptions import NoSuchElementException
@@ -236,7 +237,7 @@ def download_data_dump(browser: WebDriver, site: str, meta_url: str | None, etag
 
         checkbox.click()
         sleep(1)
-        btn.click()
+        ActionChains(browser).click(btn).perform()
         check_cloudflare_intercept(browser)
         sleep(2)
         url = browser.execute_script("return window.extractedUrl;")

This switches to the WebDriver Actions API, which skips over the additional validations performed by the higher-level Selenium interactions interface, thus avoiding this failure mode.

The next annoying intermittent error was:

requests.exceptions.MissingSchema: Invalid URL 'None': No scheme supplied. Perhaps you meant https://None?

This one seemed to be caused by some kind of ETag handling: the script had a bunch of code to intercept the fetch() request for the download link, request the headers, and log the resulting ETag, and something in that process occasionally fails to do the interception properly. It turned out that the ETag wasnʼt actually necessary for anything, so I just ripped all of that code out entirely.

Finally, I also found that it would hang while repeatedly printing:

2026-08-13 15:16:58.476 | WARNING | sedd.main:try_recover_fucked_download:394 - photo.stackexchange.com.7z has 0 part files. Race condition?

Iʼm not really clear what this code is trying to do; it seemed to have something to do with getting confused by its own .part files. I concluded that I didnʼt need whatever feature this was for, and turned it off with --no-wipe-part-files, except that flag doesnʼt work (despite being in the documentation) so I passed the short form, -N, instead.

With all of those changes (which I have also submitted to Olivia), plus just retrying repeatedly, I eventually had all 362 .7z files downloaded. After verifying them against the official checksums with sha256sum --check --quiet, it was time to upload them to the Internet Archive. This was pretty straightforward, with their command-line tool:

ia upload stackexchange_20260630 --keep-directories \
    --metadata collection:opensource_media \
    stackexchange_20260630/*

I left this running overnight, and when I came back in the morning it had finished successfully but the page for my upload on archive.org now said, simply:

This item is no longer available.

Items may be taken down for various reasons, including by decision of the uploader or due to a violation of our Terms of Use.

“Violation of our Terms of Use”?? I immediately remembered some trouble the official uploads had a few years back, when the March 2022 dump was delayed due to being flagged as malware. Sure enough, I asked VirusTotal about the reverseengineering and monero dumps (it conveniently takes SHA256 values so I could just copy the checksums and didnʼt have to upload anything), and it reported that it had scanned them and found some viruses. Following the archive.org troubleshooting FAQ, I emailed the Internet Archive helpdesk with a brief summary of the situation and links to my upload and the 2022 discussion. A bit over a day later, they reviewed it and removed the automated flag.

After all that work, the archive is now available at archive.org/details/stackexchange_20260630. To be honest, Iʼm not sure I would bother again, and Iʼm certainly not going to mark my calendar to jump straight on it October 1. I still think this data is a valuable resource, but itʼs notable that for more than a month nobody else bothered to preserve it, even though even up through last year there were many people expressing concerns about its availability. I think this is a symptom of the continued decline of StackOverflow; it seems like even the most ardent supporters have given it up as a lost cause.