![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
brin_bellway(This post is a supplement to this Tumblr comment, but also stands alone.)
---
I wrote an extremely outdated guide in 2015, and a substantially outdated guide in 2018, so it would seem I’m due for another one.
I did not have a data plan (or, indeed, any cell plan) for over six years after getting my first smartphone. Currently I have a 250 MB/month plan, and I never max it out.
I also like to keep the amount of corp on my phone to a minimum where practical, so some of these may not be the most optimal app if prioritising *purely* offline-ness. Those two things often go together, though.
---
Choosing a Phone Model
(These are good to keep in mind next time you're in the market for a new phone, but don't despair if your phone doesn't fit these criteria: quite a bit of the stuff in later sections can be done regardless.)
My absolute highest priority when picking out a new smartphone is a microSD slot. If you're not relying on the Internet to hold all your stuff for you, you have to hold it yourself, which means you want as much space as possible. 256 GB microSD cards are cheap these days (CAD$29); 512 GB is fairly affordable (CAD$67); 1 TB is very doable if you have a good income (CAD$155), will tend to get more affordable over time, and is certainly far cheaper than getting a phone model with an extra terabyte of internal storage.
I recommend turning on adoptable storage, especially if you are storing any data that is even slightly sensitive, in order to make it both harder for a phone thief to get at the card's contents and easier for your own apps to get at the card's contents (modern versions of Android put up a lot of obstacles to things like setting file-sync apps to sync to external folders). The main downsides of adoptable storage are (1) it makes it hard to use the internal storage for anything except maybe apps (but with 512 GB of external, [64 GB minus system storage] internal isn't that big a loss to bite), and (2) you can't remove the card and read its contents on another device even *with* the owner's consent. It's more important than ever to keep backups. (We'll be talking a bit about how to automate that later.)
(If you're going to turn on adoptable storage, do it *before* loading a copy of your data onto the card: in order to adopt a card the phone has to erase all the data that's on it, so you'd only have to start over anyway.)
Buy your microSD cards directly from the manufacturer if feasible (I linked to Western Digital Canada above, but they sell in many countries): Amazon has a bad habit of sending counterfeits, "16 GB cards that have been repainted to say '512' on the label" and stuff like that.
I have a pretty low desire for novelty in my music as people go, but even I think an FM radio receiver is a nice-to-have, and people who want a lot of new and different music in their lives and don't want to be dependent on Spotify etc for that may consider it a high priority.
I personally have a Teracube 2e and am pretty happy with it, but there are other options, especially if you don't also care about having a removable battery (which I do).
iPhones do not have microSD slots. The rest of this guide will assume you have an Android, though some of it is doable on iOS and on more esoteric operating systems.
---
Apps
First and foremost, let's talk about Kiwix. Did you know you can store Wikipedia on your phone? Not in an abstract "theoretically if you compare the amount of data in Wikipedia and the amount of storage in your phone it would fit" way: I mean you can literally download Wikipedia onto your phone.
And while Wikipedia is their flagship usecase, the folks at Kiwix are not kidding when they say "any website". There's a whole bunch of pre-built website files, they will help you download smaller websites (and it looks like the cap on what constitutes a sufficiently small website is higher than it was the last time I checked), and down in the section on advanced techniques we will be talking about how to convert a website to Kiwix-compatible format yourself.
Also, if you have other people connect to your mobile hotspot (even if your mobile hotspot is not in turn connected to the Internet), you can turn on your Kiwix app's server mode and show the websites to others. (This is entirely optional, and if you do turn it on you can select which websites to share and which not to.)
(As for downloading individual webpages (and parts of webpages: I've found the "save selection" button to be pretty useful), check out the SingleFile browser extension.)
Speaking of sharing things with people around you, the open-source app store F-Droid has a "Nearby" option for swapping APKs with people. (You *can* just send extracted APKs over later Android versions' built-in Nearby Share, and if your trading partner doesn't already have F-Droid installed you'll need to bootstrap them that way, but F-Droid offers a nicer and more detailed interface.) Apps do *not* need to originally be from F-Droid to be shared via F-Droid Nearby.
In 2020 I wrote a post on the state of the field for mesh-based texting and a postscript on Bridgefy meshes becoming much easier to create. As far as I know things have not really changed since then except that Berty has an alpha release out; I haven't tested it.
Interfaces to access your local media files:
Audio: Odyssey. I chose this one because I strongly value being able to mass-add an entire file folder's contents to a playlist: if that's not something you care about, you may well find that a different music-player app fits your needs better, and I encourage you to poke around.
Text (read-write): Collabora Office, a LibreOffice fork with an Android version. There have been a couple other LibreOffice ports for Android over the years, but they were merely better-than-nothing: this one straight-up *works*. I set up access to it through F-Droid as described here, but if you don't care about that you can also just use Google Play.
Text (read-only): Last time I was poking around the options here I liked Librera best, though it's been a while and I'm not sure exactly which features led me to that decision. I think the primary thing I liked about it was the wide variety of file types it supports.
Video: I never watch videos on my phone and have no opinion on this. Apparently my image-gallery app has a built-in video player? It seems fine?
For GPS routing I use OSMAnd and I'm happy with it, though some people find the interface too busy. I've also heard praise for Organic Maps, and if you don't mind it being for-profit MapFactor works pretty well.
The core thing here is that you want an app with OpenStreetMap-based downloadable map packages. They won't try to "expire" and vanish out from under you (Google Maps is not *as* useless offline as it once was, but last I heard it was still not great), and if you run into a map error you can just go on OpenStreetMap and fix it and it'll be included in your app's future map updates. (Note that some apps do map updates separately from updates to the app software, so keep an eye out for that.)
Translation is a big exception to the way that ~indie/open-source/nonprofit/etc and offline often go together: if you want offline translation you *have* to get your translation app from a megacorp, and if you want your translation app to not be from a megacorp you *have* to get a cloud-dependent one. As such, I've been using Google Translate.
Make sure to download in advance the language packs for each language you think you might need: they're not included by default, to save on space.
Syncthing lets you synchronise your files between phone and laptop/desktop without going via a cloud server, with a variety of options regarding one-way vs two-way sync, syncing intervals, how long to keep old versions, and so on.
(Edit for clarity: this makes files more accessible without having to worry too much about which device you made them on, and *also* it makes it easy to put a spare copy of things like camera photos onto your PC (and a spare copy of things like stories you're writing onto your phone). When I was 15, I was able to shrug off an abrupt laptop failure by repopulating the new laptop with files from my MP3 player (it could only *play* music files, but it could *store* pretty much any file); having your PC and your handheld computer act as backup drives for each other is even easier to do now.)
(I *strongly* recommend that you also keep at least one offsite backup, ideally with version history, but that's beyond the scope of this post. The important thing for the purposes of *this* post is to treat your cloud copy as a backup, rather than as the primary.)
ICSx⁵ lets you subscribe to calendars (that will then appear on your calendar app). Notably, you can subscribe *to .ics files on your local phone storage*, leaving you with a local calendar that can then be shared with other computers over Syncthing.
I almost didn't include SMS laptop/desktop clients as being out of scope, but then I remembered I've seen people using cloud-based solutions for it, so do note that KDE Connect will let you remotely access your phone's SMS function over local Wi-Fi. You can also check your phone's battery level, make it ring (even in silent mode) to help you track down where in your house it is, and use your phone as a backup trackpad and keyboard for your computer.
My top three favourite unit converters are Smart Tools Unit Converter Pro > Equate > Smart Tools Unit Converter Free. In addition to keeping you from needing to google what 5.7L/100km is in miles per gallon (it's 41), all three offer currency conversion, and if you don't have Internet at that moment it'll use the exchange rates it had cached from last time you used it *with* Internet. I try to check in on it while in Wi-Fi zones every once in a while, to keep the cache fresh.
I prefer to reserve more involved video gaming for my laptop, but I do have a few casual games on my phone for waiting rooms and the like: 2048, Sudoku, and (with its Internet access forbidden: see the NetGuard section below) Hangman.
---
Other Techniques
Did you know you can send an SMS to an email address and your mobile provider will pass it along for you? You can also email a phone number, if you know which mobile provider to address it to!
Although in theory this should also be possible on non-stock SMS apps, in practice neither of the main F-Droid ones have interfaces set up to allow for it. I almost never use SMS-to-email now that I do have a bit of mobile data (I only just realised you can't do it on QKSMS while testing it for this guide, having been using QKSMS for years), so I leave the Google Messages app disabled knowing that I can re-enable the app if I need to.
Advanced Kiwix technique: you can turn pretty much any (reasonably static) website into a Kiwix file, even if it's too big for youzim.it, by scraping it with grab-site and then running the resulting WARC file through warc2zim.
I used to use the classic wget, but grab-site is better at downloading the things you want it to download (like embedded images) and not downloading the things you don't want it to download (like one zillion "replytocom" links on a WordPress blog, none of which contain any information that the post's main page didn't already have). (Some useful exclusions can be found in Appendix 3.)
Make sure to read the grab-site instructions: there's a lot of useful info in there.
Please be polite and rate-limit your scraper to one page every second or three when working with small-time websites: I *did* accidentally denial-of-service someone once.
Warc2zim outputs will work just fine on Android (on Linux you can't open them directly in Kiwix, but you can work around this by putting Kiwix in server mode and looking at it in a browser). There's some sort of bug where warc2zim files can't load Dreamwidth (nor Livejournal) homepages, so if you're compiling a Dreamwidth file I recommend setting the file's default opening page to, for example, https://brin-bellway.dreamwidth.org/archive instead of https://brin-bellway.dreamwidth.org/.
---
Appendix 1: Making the Most of Limited Data
NetGuard lets you deny Internet access (with separate toggles for Wi-Fi and mobile data) to individual apps, as well as more fine-grained blocking of individual addresses. I find that the in-app-adblocking function tends to be losing the arms race at any given time, but is better than nothing; denying *all* Internet access to an app works great, though.
For web-browsing specifically, UBlock Origin is available for Android Firefox and is having a much better time with the arms race. The "block large media elements" button can also be particularly helpful.
---
Appendix 2: Life Without a SIM Card
I'm not *satisfied* with Talkatone, as such: the ads are a bit intrusive (and it's very hard to successfully strangle them with NetGuard because the app needs Internet to perform its actual function), plus they don't let you port the number out. But it's better than nothing, and worth keeping on hand. If anyone knows better VoIP solutions, I'd be happy to hear them.
Know where long-distance communication is possible: keep an eye out for public Wi-Fi zones. Back in the day, I knew the location of every public Wi-Fi in town and how far away you could get from each one before the signal failed. (Fun fact: in 2017 I played Pokemon Go 1 - 2 hours/day on public Wi-Fi and a borrowed 100 MB/month mobile hotspot. I almost never ran out of hotspot data.)
OpenStreetMap's coverage of available amenities isn't super reliable, but it's something. If you need to send an email or VoIP call and you don't know where the nearest Wi-Fi is, try looking around your map for things like cafes, fast-food restaurants, and government services.
You don't need a SIM card to call 911†: any device with the right hardware to contact 911 may do so, and the cell infrastructure is required to let you through even if you haven't paid them and even if you have no phone number. The SIM-less phones I've had conveniently inform you of this on the lock screen if you're not in airplane mode. (The flip side is that you should be careful about letting small children play with hand-me-down phones.)
†I'm not familiar with how this works outside of Canada and America, so if you're elsewhere I recommend checking your local sources.
---
Appendix 3: Reference on Useful Grab-Site Exclusions
You might not entirely agree with me on which bits of a website are and aren't worth keeping, perhaps especially for wikis. I encourage you to remove from and add to these lists as necessary, checking in on active scrapes occasionally to see if they've gone down rabbit holes you don't want them going down.
I am not fluent in Python regex and I'm sure this list contains more any-length wildcards than strictly necessary. It works.
.*edit=.*
.*mode=reply.*
.*replyto=.*
.*style=.*
.*thread=.*
.*view=top-only.*
.*nc=.*
.*/data/.*
.*get.adobe.com/flashplayer.*
.*photo.siteroot.*
.*flymeango.*
.*arte.tv.*
.*banner.native_install_prompt.*
.*imgprx.livejournal.net.*
.*banner.android.text.*
.*spring.me.*
.*likus.users.sc.*
.*popup.suggestion.text.*
.*mobileAppBanners.footer.text.*
.*comment.form.hint.text.*
.*livejournal.com/..../../../gtm.js
By default it will scrape daily summaries for every day right up to the present (unlike Dreamwidth, for which it will only scrape daily summaries on days where they actually posted something), so if you're scraping a dormant blog I suggest excluding years and months from after they stopped posting.
.*replytocom.*
.*pfstyle.*
.*share=.*
.*like_comment.*
.*showComment.*
.*review=.*
.*/trackback/.*
.*/amp/.*
.*/feed/.*
.*utm_source.*
.*amp=.*
.*/author/.*
.*require_name_email.*
.*/search/label/.*
.*/feeds/.*
.*feedburner.*
.*with_password.*
.*change_user.*
.*infinity=scrolling.*
.*view=print.*
.*feed.php.*
.*f=14.*
.*hilit.*
.*memberlist.php.*
.*action=.*
.*oldid.*
.*printable=.*
.*redirect=.*
.*Special:.*
.*Template:.*
.*t=.*
.*Forum:.*
.*Forum_talk.*
.*useskin.*
.*User:.*
.*User_talk.*
.*User_blog.*
.*/ru/.*
.*/zh/.*
.*/de/.*
.*fandom.com/topics.*
.*kickstarter.*
.*linkedin.*
.*fandomatic.*
.*apps.apple.*
.*futhead.*
.*bandcamp.*
.*www.fandom.*
.*youtube.*
.*universeconquest.*
.*jwpsrv.*
---
I wrote an extremely outdated guide in 2015, and a substantially outdated guide in 2018, so it would seem I’m due for another one.
I did not have a data plan (or, indeed, any cell plan) for over six years after getting my first smartphone. Currently I have a 250 MB/month plan, and I never max it out.
I also like to keep the amount of corp on my phone to a minimum where practical, so some of these may not be the most optimal app if prioritising *purely* offline-ness. Those two things often go together, though.
---
Choosing a Phone Model
(These are good to keep in mind next time you're in the market for a new phone, but don't despair if your phone doesn't fit these criteria: quite a bit of the stuff in later sections can be done regardless.)
My absolute highest priority when picking out a new smartphone is a microSD slot. If you're not relying on the Internet to hold all your stuff for you, you have to hold it yourself, which means you want as much space as possible. 256 GB microSD cards are cheap these days (CAD$29); 512 GB is fairly affordable (CAD$67); 1 TB is very doable if you have a good income (CAD$155), will tend to get more affordable over time, and is certainly far cheaper than getting a phone model with an extra terabyte of internal storage.
I recommend turning on adoptable storage, especially if you are storing any data that is even slightly sensitive, in order to make it both harder for a phone thief to get at the card's contents and easier for your own apps to get at the card's contents (modern versions of Android put up a lot of obstacles to things like setting file-sync apps to sync to external folders). The main downsides of adoptable storage are (1) it makes it hard to use the internal storage for anything except maybe apps (but with 512 GB of external, [64 GB minus system storage] internal isn't that big a loss to bite), and (2) you can't remove the card and read its contents on another device even *with* the owner's consent. It's more important than ever to keep backups. (We'll be talking a bit about how to automate that later.)
(If you're going to turn on adoptable storage, do it *before* loading a copy of your data onto the card: in order to adopt a card the phone has to erase all the data that's on it, so you'd only have to start over anyway.)
Buy your microSD cards directly from the manufacturer if feasible (I linked to Western Digital Canada above, but they sell in many countries): Amazon has a bad habit of sending counterfeits, "16 GB cards that have been repainted to say '512' on the label" and stuff like that.
I have a pretty low desire for novelty in my music as people go, but even I think an FM radio receiver is a nice-to-have, and people who want a lot of new and different music in their lives and don't want to be dependent on Spotify etc for that may consider it a high priority.
I personally have a Teracube 2e and am pretty happy with it, but there are other options, especially if you don't also care about having a removable battery (which I do).
iPhones do not have microSD slots. The rest of this guide will assume you have an Android, though some of it is doable on iOS and on more esoteric operating systems.
---
Apps
First and foremost, let's talk about Kiwix. Did you know you can store Wikipedia on your phone? Not in an abstract "theoretically if you compare the amount of data in Wikipedia and the amount of storage in your phone it would fit" way: I mean you can literally download Wikipedia onto your phone.
And while Wikipedia is their flagship usecase, the folks at Kiwix are not kidding when they say "any website". There's a whole bunch of pre-built website files, they will help you download smaller websites (and it looks like the cap on what constitutes a sufficiently small website is higher than it was the last time I checked), and down in the section on advanced techniques we will be talking about how to convert a website to Kiwix-compatible format yourself.
Also, if you have other people connect to your mobile hotspot (even if your mobile hotspot is not in turn connected to the Internet), you can turn on your Kiwix app's server mode and show the websites to others. (This is entirely optional, and if you do turn it on you can select which websites to share and which not to.)
(As for downloading individual webpages (and parts of webpages: I've found the "save selection" button to be pretty useful), check out the SingleFile browser extension.)
Speaking of sharing things with people around you, the open-source app store F-Droid has a "Nearby" option for swapping APKs with people. (You *can* just send extracted APKs over later Android versions' built-in Nearby Share, and if your trading partner doesn't already have F-Droid installed you'll need to bootstrap them that way, but F-Droid offers a nicer and more detailed interface.) Apps do *not* need to originally be from F-Droid to be shared via F-Droid Nearby.
In 2020 I wrote a post on the state of the field for mesh-based texting and a postscript on Bridgefy meshes becoming much easier to create. As far as I know things have not really changed since then except that Berty has an alpha release out; I haven't tested it.
Interfaces to access your local media files:
Audio: Odyssey. I chose this one because I strongly value being able to mass-add an entire file folder's contents to a playlist: if that's not something you care about, you may well find that a different music-player app fits your needs better, and I encourage you to poke around.
Text (read-write): Collabora Office, a LibreOffice fork with an Android version. There have been a couple other LibreOffice ports for Android over the years, but they were merely better-than-nothing: this one straight-up *works*. I set up access to it through F-Droid as described here, but if you don't care about that you can also just use Google Play.
Text (read-only): Last time I was poking around the options here I liked Librera best, though it's been a while and I'm not sure exactly which features led me to that decision. I think the primary thing I liked about it was the wide variety of file types it supports.
Video: I never watch videos on my phone and have no opinion on this. Apparently my image-gallery app has a built-in video player? It seems fine?
For GPS routing I use OSMAnd and I'm happy with it, though some people find the interface too busy. I've also heard praise for Organic Maps, and if you don't mind it being for-profit MapFactor works pretty well.
The core thing here is that you want an app with OpenStreetMap-based downloadable map packages. They won't try to "expire" and vanish out from under you (Google Maps is not *as* useless offline as it once was, but last I heard it was still not great), and if you run into a map error you can just go on OpenStreetMap and fix it and it'll be included in your app's future map updates. (Note that some apps do map updates separately from updates to the app software, so keep an eye out for that.)
Translation is a big exception to the way that ~indie/open-source/nonprofit/etc and offline often go together: if you want offline translation you *have* to get your translation app from a megacorp, and if you want your translation app to not be from a megacorp you *have* to get a cloud-dependent one. As such, I've been using Google Translate.
Make sure to download in advance the language packs for each language you think you might need: they're not included by default, to save on space.
Syncthing lets you synchronise your files between phone and laptop/desktop without going via a cloud server, with a variety of options regarding one-way vs two-way sync, syncing intervals, how long to keep old versions, and so on.
(Edit for clarity: this makes files more accessible without having to worry too much about which device you made them on, and *also* it makes it easy to put a spare copy of things like camera photos onto your PC (and a spare copy of things like stories you're writing onto your phone). When I was 15, I was able to shrug off an abrupt laptop failure by repopulating the new laptop with files from my MP3 player (it could only *play* music files, but it could *store* pretty much any file); having your PC and your handheld computer act as backup drives for each other is even easier to do now.)
(I *strongly* recommend that you also keep at least one offsite backup, ideally with version history, but that's beyond the scope of this post. The important thing for the purposes of *this* post is to treat your cloud copy as a backup, rather than as the primary.)
ICSx⁵ lets you subscribe to calendars (that will then appear on your calendar app). Notably, you can subscribe *to .ics files on your local phone storage*, leaving you with a local calendar that can then be shared with other computers over Syncthing.
I almost didn't include SMS laptop/desktop clients as being out of scope, but then I remembered I've seen people using cloud-based solutions for it, so do note that KDE Connect will let you remotely access your phone's SMS function over local Wi-Fi. You can also check your phone's battery level, make it ring (even in silent mode) to help you track down where in your house it is, and use your phone as a backup trackpad and keyboard for your computer.
My top three favourite unit converters are Smart Tools Unit Converter Pro > Equate > Smart Tools Unit Converter Free. In addition to keeping you from needing to google what 5.7L/100km is in miles per gallon (it's 41), all three offer currency conversion, and if you don't have Internet at that moment it'll use the exchange rates it had cached from last time you used it *with* Internet. I try to check in on it while in Wi-Fi zones every once in a while, to keep the cache fresh.
I prefer to reserve more involved video gaming for my laptop, but I do have a few casual games on my phone for waiting rooms and the like: 2048, Sudoku, and (with its Internet access forbidden: see the NetGuard section below) Hangman.
---
Other Techniques
Did you know you can send an SMS to an email address and your mobile provider will pass it along for you? You can also email a phone number, if you know which mobile provider to address it to!
Although in theory this should also be possible on non-stock SMS apps, in practice neither of the main F-Droid ones have interfaces set up to allow for it. I almost never use SMS-to-email now that I do have a bit of mobile data (I only just realised you can't do it on QKSMS while testing it for this guide, having been using QKSMS for years), so I leave the Google Messages app disabled knowing that I can re-enable the app if I need to.
Advanced Kiwix technique: you can turn pretty much any (reasonably static) website into a Kiwix file, even if it's too big for youzim.it, by scraping it with grab-site and then running the resulting WARC file through warc2zim.
I used to use the classic wget, but grab-site is better at downloading the things you want it to download (like embedded images) and not downloading the things you don't want it to download (like one zillion "replytocom" links on a WordPress blog, none of which contain any information that the post's main page didn't already have). (Some useful exclusions can be found in Appendix 3.)
Make sure to read the grab-site instructions: there's a lot of useful info in there.
Please be polite and rate-limit your scraper to one page every second or three when working with small-time websites: I *did* accidentally denial-of-service someone once.
Warc2zim outputs will work just fine on Android (on Linux you can't open them directly in Kiwix, but you can work around this by putting Kiwix in server mode and looking at it in a browser). There's some sort of bug where warc2zim files can't load Dreamwidth (nor Livejournal) homepages, so if you're compiling a Dreamwidth file I recommend setting the file's default opening page to, for example, https://brin-bellway.dreamwidth.org/archive instead of https://brin-bellway.dreamwidth.org/.
---
Appendix 1: Making the Most of Limited Data
NetGuard lets you deny Internet access (with separate toggles for Wi-Fi and mobile data) to individual apps, as well as more fine-grained blocking of individual addresses. I find that the in-app-adblocking function tends to be losing the arms race at any given time, but is better than nothing; denying *all* Internet access to an app works great, though.
For web-browsing specifically, UBlock Origin is available for Android Firefox and is having a much better time with the arms race. The "block large media elements" button can also be particularly helpful.
---
Appendix 2: Life Without a SIM Card
I'm not *satisfied* with Talkatone, as such: the ads are a bit intrusive (and it's very hard to successfully strangle them with NetGuard because the app needs Internet to perform its actual function), plus they don't let you port the number out. But it's better than nothing, and worth keeping on hand. If anyone knows better VoIP solutions, I'd be happy to hear them.
Know where long-distance communication is possible: keep an eye out for public Wi-Fi zones. Back in the day, I knew the location of every public Wi-Fi in town and how far away you could get from each one before the signal failed. (Fun fact: in 2017 I played Pokemon Go 1 - 2 hours/day on public Wi-Fi and a borrowed 100 MB/month mobile hotspot. I almost never ran out of hotspot data.)
OpenStreetMap's coverage of available amenities isn't super reliable, but it's something. If you need to send an email or VoIP call and you don't know where the nearest Wi-Fi is, try looking around your map for things like cafes, fast-food restaurants, and government services.
You don't need a SIM card to call 911†: any device with the right hardware to contact 911 may do so, and the cell infrastructure is required to let you through even if you haven't paid them and even if you have no phone number. The SIM-less phones I've had conveniently inform you of this on the lock screen if you're not in airplane mode. (The flip side is that you should be careful about letting small children play with hand-me-down phones.)
†I'm not familiar with how this works outside of Canada and America, so if you're elsewhere I recommend checking your local sources.
---
Appendix 3: Reference on Useful Grab-Site Exclusions
You might not entirely agree with me on which bits of a website are and aren't worth keeping, perhaps especially for wikis. I encourage you to remove from and add to these lists as necessary, checking in on active scrapes occasionally to see if they've gone down rabbit holes you don't want them going down.
I am not fluent in Python regex and I'm sure this list contains more any-length wildcards than strictly necessary. It works.
Dreamwidth/Livejournal:
.*edit=.*
.*mode=reply.*
.*replyto=.*
.*style=.*
.*thread=.*
.*view=top-only.*
.*nc=.*
.*/data/.*
Livejournal-specific:
.*get.adobe.com/flashplayer.*
.*photo.siteroot.*
.*flymeango.*
.*arte.tv.*
.*banner.native_install_prompt.*
.*imgprx.livejournal.net.*
.*banner.android.text.*
.*spring.me.*
.*likus.users.sc.*
.*popup.suggestion.text.*
.*mobileAppBanners.footer.text.*
.*comment.form.hint.text.*
.*livejournal.com/..../../../gtm.js
By default it will scrape daily summaries for every day right up to the present (unlike Dreamwidth, for which it will only scrape daily summaries on days where they actually posted something), so if you're scraping a dormant blog I suggest excluding years and months from after they stopped posting.
Blogs (other):
.*replytocom.*
.*pfstyle.*
.*share=.*
.*like_comment.*
.*showComment.*
.*review=.*
.*/trackback/.*
.*/amp/.*
.*/feed/.*
.*utm_source.*
.*amp=.*
.*/author/.*
.*require_name_email.*
.*/search/label/.*
.*/feeds/.*
.*feedburner.*
.*with_password.*
.*change_user.*
.*infinity=scrolling.*
Forums:
.*view=print.*
.*feed.php.*
.*f=14.*
.*hilit.*
.*memberlist.php.*
Wikis:
.*action=.*
.*oldid.*
.*printable=.*
.*redirect=.*
.*Special:.*
.*Template:.*
.*t=.*
.*Forum:.*
.*Forum_talk.*
.*useskin.*
.*User:.*
.*User_talk.*
.*User_blog.*
.*/ru/.*
.*/zh/.*
.*/de/.*
.*fandom.com/topics.*
.*kickstarter.*
.*linkedin.*
.*fandomatic.*
.*apps.apple.*
.*futhead.*
.*bandcamp.*
.*www.fandom.*
.*youtube.*
.*universeconquest.*
.*jwpsrv.*
no subject
Date: 2023-03-20 06:41 pm (UTC)(I've got the additional selection-criterion of wanting my phone to be physically small, which makes things a bit more complicated; but on the other hand that's a lot less of an urgent criterion now than it was this time last year, thanks to my having upgraded to a utility belt with a larger pocket, so I've now got a lot more room to optimize for things like storage, relative to before.)
In the section on adoptable storage, you mentioned the intention to go more into backup-automation later. Was that just about the Syncthing paragraph, or are you planning a followup post on the subject? (If the former, it might be useful to add a telegraphing-note to that paragraph connecting it back to the backups-to-ameliorate-the-dangers-of-adoptable-storage point, because on my first readthrough of this post I entirely missed the connection there.)
no subject
Date: 2023-03-20 10:31 pm (UTC)---
>>In the section on adoptable storage, you mentioned the intention to go more into backup-automation later. Was that just about the Syncthing paragraph, or are you planning a followup post on the subject?
I meant the Syncthing paragraph. I've added another paragraph to the Syncthing subsection to hopefully make things more clear.
(As for offsite backups, I personally keep a BorgBackup repository on a Hetzner Storage Box, but while it works well for me it required too much messing around with Linux command-lines for most people's tastes. Duplicati on [insert any cloud-storage service here] would probably work similarly while being more user-friendly and compatible with a wider variety of operating systems? You'd know a lot more about Duplicati than I would: I mention it here for the benefit of other people reading.)