Ask me about my special interests

[brin_bellway]

By popular request (both of the people who were around when it came up), a post about how my routine backups work.

[cut for length; much infodumping awaits]


Continuously:

• Personal files on laptop




• Method: MEGA client for Linux, set to automatic two-way sync.


• I have the lowest paid tier of MEGA account: €50/year for 400GB. (This is more than enough for my current needs, though I do have eventual plans to start working with videos more and expect I will need to upgrade at that point.) I described the process of deciding on and moving to MEGA in this series of posts.

Daily:

• Lifelogger




• Method: Transfer the files from the lifelogger to my laptop (a non-synced folder labelled "Lifelogger Waystation") via USB cable. Create a folder named with the ISO-format date of the day in question (generally yesterday, though occasionally I miss a day of processing and have to catch up with two days). Check for and delete any files that are only a few seconds long and contain nothing remotely useful; if I know there's a continuous-mode file with a long stretch of silence before I remembered to turn it off, maybe go in with Audacity and clip that out. Use WinFF (Synaptic version) to convert the files to 64kbps MP3 and have it place the results in the dated folder. Batch-rename the files to place the "R"/"V" marker (whether the file was recorded in normal recording mode or voice-activated mode) at the *end* of the file rather than the default beginning, so that the files alphabetically sort by timestamp. Compress on 7-zip "ultra" mode, saving ~5% space and reducing the number of individual files in my archives for easier scanning and copying. (In particular, the time it takes to sync with my smartphone seems to scale with the number of individual files to be checked for updates.) Place the zip in a MEGA-synced folder. Delete the unzipped dated folder and the original WAV files.


• As a general rule, a non-workday is about 70 - 100 MB and a workday is about 180 - 210 MB. I started on January 22nd (a little over five months ago) and have collected 18.6 GB of compressed data in that time.





• Food log




• Method: Write down a short description of what I've eaten lately (and, if it was new or different, notes on what I thought of it) in an ODT file.


• I've been using the same ODT file since October of 2012, and it's now so big that it takes a while to finish saving (maybe 15 seconds?). I like being able to search the whole log with just an ordinary within-file text search, though, so for now I'm putting up with it rather than splitting off a new file.


• Sometimes I update this two or three times throughout the day; other times I miss a day or two and do my best to reconstruct what happened in the gap.





• Health log




• Method: Write down a short description of how I felt each day in an ODT file.


• Started doing this in mid-May after seeing Siderea advise it. Working out well so far, and I can see how it will become handier in the future.


• Like the food log, this is sometimes two or three times a day and other times every couple days.





• Tumblrs




• Method: A cron job set to execute "python tumblr_backup.py {{username1}} && python tumblr_backup.py {{username2}} && {{etc}}" for the half-dozen not-completely-dormant Tumblrs I value highly enough to preserve locally and in their entireties, including mine. This occurs at 9 PM: it sometimes skips days if my computer is not on at 9 PM, catching up on future occasions. I leave the resulting folders where they are: performing another backup when an older backup is already there causes the new backup to be semi-incremental (accommodating post edits, but not post or image deletions).


• These folders are *not* synced with MEGA (you think *lifelogger* file proliferation is bad, just look at tumblr-utils file proliferation): see the fortnightly section for how I deal with that.


• Make sure to get your own API key!

Weekly:

• Tumblr-to-WordPress




• Method: Manual copy-pasting of each of my new Tumblr posts from the tumblr-utils copy into a "classic" WordPress draft (classic mode handles nested blockquotes much better than the new drafter does), with manual tweaking of the results.


• I've tried the automatic import function, and I found it inadequate at best and actively counterproductive (creating extra work) at worst.

When I feel like it, especially (but not necessarily, in either direction) before leaving the house, but no less often than fortnightly:

• MEGA-to-smartphone




• Method: The paid version ($8 one-time) of MetaCtrl's third-party MEGA client for Android, set to "two-way sync when, and only when, I press this button". (I don't want it accidentally consuming background data while I'm working with a metered or capped connection, plus if something goes wrong with my cloud folder I want at least *some* opportunity to notice before syncing the corrupted data.)


• I loved MetaCtrl's Dropbox client (which has essentially the same interface), so when I switched from Dropbox to MEGA I stayed within the same family of Android cloud-sync clients.


• Starting last autumn, it began having trouble updating edited files: the first sync, it would remove the phone's copy but *not* replace it with the updated version, throwing a FileNotFoundException. If you then synced again, the phone's file--presumably having "gotten out of the way" now that it had been deleted?--would be replaced successfully. I contacted MetaCtrl and he says it's a problem with my phone's SD card subsystem. (Seems plausible, since the SD card subsystem is noticeably wonky elsewhere: it accepts microSDXC, but--and it is part of the official SDXC standard that devices accepting SDXC should not do this--it insists on only using cards that have been FAT32 formatted! I now have a 256GB microSD that can't take individual files above 4 GiB! (don't worry, I can fix the card easily once I get a phone willing to work with the result)) I'm not ready to replace my phone yet, so for now I'm just working around the problem by syncing multiple times on each occasion.


• Since I have the paid version of the client, I can have different folders sync to different places. Most of it goes to my SD card, but the tumblr-utils backups go to internal storage because the internal storage drive *can* accept files above 4 GiB like a proper modern-day system. My virtual-machine export files are above 4 GiB each and don't fit in the remaining internal space, so for now they're not on my phone at all.

Fortnightly:

• Update lifelogger's clock, to make sure timestamps remain accurate without having to find out the hard way what its exact rate of clock drift is



• Tumblrs-to-MEGA




• Method: Zip each tumblr-utils folder, label the zips with the ISO date, then place them in a MEGA-synced folder.


• I use regular zipping rather than 7-zipping because the space savings of 7-zipping on tumblr-utils folders is small and regular zipping is faster. For dormant Tumblrs I use 7-zipping because I won't be frequently making new copies, so speed is not really a concern and I might as well squeeze a bit more space out of it.





• Write a roundup post



• Dreamwidth




• Method: Described here.





• Google Calendar, Chromium settings, Google Sheets, list of installed Google Play apps, Gmail




• Method: https://takeout.google.com/, label the resulting zip with the ISO date, place in a MEGA-synced folder.


• I have plans to de-googlify my calendar and email, and maybe also my spreadsheets. For the moment, I'm still working on the tasks above this on the to-do list.


• I don't *primarily* use Chromium, but it does come in handy on occasion. Might as well throw this in.


• I also have plans to start keeping APK backups, and probably switch to F-Droid for apps available through both stores. In the meantime, I export the app list so if something happens between now and then I know which apps to try to salvage.


• Youtube favourites are prone to linkrot, so instead of backing it up through here I keep notes in an ODT: that way I know which videos to look for alternate copies of if that one disappears. (Wait, only one of my saved-video playlists has an ODT copy: the other doesn't. *fixes*)


• (Of course, the Youtube videos I *really* like I save locally.)





• Thunderbird




• Method: Zip the ".thunderbird" folder in my home directory, label the zip with the ISO date, place the zip in a MEGA-synced folder.


• (Zip and not 7-zip, again because it's going to be replaced soon enough anyway.)


• To restore from this style of backup, all you have to do is get a fresh installation of Thunderbird and stick this ".thunderbird" folder in the place it will think to look for its data (generally the home directory). (Presumably you could also override a pre-existing Thunderbird setup by replacing its ".thunderbird" folder with yours, but I haven't tried that myself.) I've done this when moving to a new laptop, and it really was just that easy!


• If you want to do this for the Windows version of Thunderbird, the process is slightly different. Here's the official guide.






• Yahoo Calendar




• Method: Right-click on the appropriate calendar in the "My Calendars" section of the left sidebar, then press "export". Label the resulting .ics with the ISO date and place in a MEGA-synced folder.


• I actually have to either talk Mom through this (she *always* has to be talked through the process no matter how many times she does it) or get permission to use her computer to do it, because the family calendar is on Mom's account and Yahoo Calendar doesn't allow granting export permissions to secondary users.


• Mind-bogglingly, Yahoo Calendar has no Android app. Oh sure, you could use the web interface on a mobile browser, but what if you're offline (as I very often am, on Android)? I used to use a cross-calendar syncing app, but Mom was having some login issues (and that app appears to have stopped working in late 2018 anyway). Now I just import the .ics into my newsletter email's associated Google Calendar every time I get a fresh Yahoo export, and set up my Google Calendar app to display that as one of its calendars. (Repeated importing of incrementally-different files does not cause duplicate entries: Google Calendar automatically corrects for that.) Having up to a fortnight of lag before I can access new entries on Android, and being unable to tell which entries were later deleted on the Yahoo version, is unfortunate, but right now I don't have any better ideas.


• Mom refuses to switch to anything else until and unless (let's face it, until) they pull a Geocities on her. Good thing I've built a safety net.






• Firefox open tabs, notepads




• Method: Copy-paste into an ODT (the same ODT for both, for historical reasons).


• Since I first started doing this I've obtained a Firefox Sync account and switched from Tomboy Notes (which has a file format incompatible with everything else and an extremely neglected and shitty Android app) to [ReText for Linux]/[Markor for Android] (which both use broadly-compatible .md files). As such, this is probably not *really* necessary anymore. It's nice to have a spare format, though.






• Podcasts




• Method: Go to the Podcasts section of Rhythmbox (which has the feeds loaded into it already). Hit "Update" (I think I deliberately set it to update only manually, so it wouldn't bug me with notifications when I didn't want them). Right-click on the resulting entries and download. After they've downloaded, right-click to delete episode, and hit "delete episode only" (*not* "delete episode and file"). (I don't listen to them *through* Rhythmbox, so the entries are just clutter once I've downloaded them.) Run GNOME SoundConverter on them to convert to 96kbps (I find 64 isn't quite enough). In a terminal, run each file through replaygain with a -r flag of 80 to standardise their volumes on something more reasonable. Put each file in the MEGA-synced folder for its podcast series.


• My use of GNOME SoundConverter pre-dates my use of WinFF. I had to start using WinFF for lifelogger files because SoundConverter is much less flexible in its options: in particular, it won't do 64kbps. I still use GNOME SoundConverter when possible because it runs in parallel, whereas WinFF only processes one file at a time.


• (...mind you, if I think back to how long lifelogger processing takes and how long podcast processing takes, I think GNOME SoundConverter might actually be slower on any given file, in a way that more than cancels out being parallel. Will need to experiment with that.)


• I actually ctrl-x the name of the file to be replaygained and replace it with "q", so that I can just re-use the exact same replaygain command for each file. I paste the name back in afterward.





• Clean out newsletter email




• Method: Look at Gmail web interface. Skim the new emails to see if anything useful (such as a freebie) has come up. Delete the rest.


• I don't care about this email enough to add it to Thunderbird or perform Google Takeouts on it. There were a *few* household-finance discussions circa 2017 that took place through it, but I've simply forwarded those to my primary email to be included in that one's backups.


• I used to work with newsletters a lot more, but I've burned out on it and also have better access to better money-making methods than I used to. I keep the newsletter email around just in case, and also to get free birthday food. And the Tor eBook of the Month, if it's a book I've heard good things about.





• General smartphone updates




• Method: On mobile, go into the "my apps" section of Google Play and hit "update" on the apps I am allowing to update, which is most but not all of them. Go into LastPass and log in while in online mode, in order to update the offline vault. Go into Unit Converter (I have the ad-free version) and update the currency exchange rates. Go into MapFactor and update the OpenStreetMap maps. Go into WiFi Map and update the regional offline maps. Check Google Translate for updated offline translation packages. Perform a couple MEGA syncs until the files affected by the abovementioned SD-card glitch have all come through.


• I plan to switch from LastPass to KeePass (XC variant for Linux, probably this intercompatible one for Android), at which point the vault will simply sync through MEGA like any other file, but it hasn't reached the top of my to-do list yet.

Monthly:

• List of followed Tumblrs




• Method: Open the ODT from last time. Go to https://www.tumblr.com/following: add any new followed blogs since last month to the top of the ODT list, and add that number to the "total number of blogs followed" written at the top of the ODT. Then, check the number of followed blogs listed on the Tumblr page and see if it matches. If not, go through the lists and figure out who's missing, and whether they've deactivated or been glitch-unfollowed.


• I started doing this during the NSFW purge and the spate of deletions and bannings then, though I'd been considering it for a while due to occasional glitch-unfollowings. Now that I'm keeping an eye on the situation I've actually found surprisingly few problems, but it's good to have the peace of mind.





• Nethack and Crawl game-guide wikis




• Method: In a terminal and having installed zimmer, run "wikizimmer https://nethackwiki.com/wiki/Potion_of_holy_water && zimmer --optimg nethackwiki.com && rm -r nethackwiki.com && wikizimmer http://crawl.chaosforge.org/Potion_of_blood && zimmer --optimg crawl.chaosforge.org && rm -r crawl.chaosforge.org". (The exact pages used are flexible, but cannot be the front page.) This results in very highly compressed files compatible with Kiwix. Label these files with the ISO date and place them in a MEGA-synced folder.


• For a while there, the wikizimmer command worked but the zimmer command didn't. I would end up with the intermediate product of a folder full of HTML files, go "well, close enough", and zip it. I don't know why it started working.


• I tried running the intermediate-product version of the command as a cron job set to run at noon on the 1st of each month, but it just did nothing instead. And yet, pasting exactly the same command into a terminal window and hitting "enter" went fine. I haven't bothered trying the final-product version as a cron job: pasting and hitting "enter" is only slightly more effort anyway, and only once a month.





• Dreamwidth-to-Wayback




• Method: Go to the monthly overview page for *two* calendar months ago (example: during the June backup, I did https://brin-bellway.dreamwidth.org/2020/04/ ). Open every public post in a new tab. For each tab (including the monthly overview), open the Internet Archive extension and click "Save Page Now".


• I built in a 1 - 2 month lag to give the posts a chance to settle: most comments and edits will have happened by then, so it's the "long-term" version of the post that gets saved. (And if for some reason I've deleted the post before then, well, clearly I must approve of it not being around or I wouldn't have deleted it. I know I wouldn't delete things lightly.) Of course, one can always save new versions if something happens with an older post.


• I am aware that Gwern has an automated page submitter, but I very much do not speak Haskell and I've never been able to get it working. Also, for a dozen or so posts a month it wasn't worth it.


• (Would be nice for these 4,800 WordPress URLs in the initial upload, though. I've managed to make that a *bit* more efficient than full manual by generating a list of all post URLs (the first method I found that worked involved doing it through my local WordPress server, which doesn't generalise to other sites at all but at least it worked for this one), then using mass find-and-replace to add https://web.archive.org/save/ to the beginning of each URL in the list, so one can just cut and paste into a browser URL bar. And it *does* at least make a nice excuse to listen to music.)





• Upon reflection, this blog I've been doing through Bloxp should probably be through either wget or SingleFile (neither of which I knew how to use when I first started doing this one). I'll have to think about whether I value the automation and internal linking of wget more than the [ability to filter out his discourse posts] and incremental nature of SingleFile. (indeed, I'll have to think about whether an incremental approach is a net positive or negative in this case)



• Slate Star Codex




• Method: Copy-pasted ODTs with a few manual formatting tweaks, plus pasting highlighted comments beneath the threads in which they were highlighted. (Note: the formatting requires much less tweaking if you start by copying from a Firefox tab rather than a Chrome tab.) They are labelled in the format "[ISO date of posting] - [title, with Windows-unfriendly filename characters censored] ([rounded wordcount, both before and after highlighted comments if applicable])": this lets me sort alphabetically by date and seek out a post with the amount of meatiness I am in the mood for. Normally has the same built-in lag as Dreamwidth-to-Wayback to catch edits, but when he took the site down I went into the Wayback Machine (where SSC is well covered) and patched the gap. It is currently fully up-to-date.


• The ODT method is a holdover from when I didn't know of anything better. I think wget would be a poor fit because I'm mostly uninterested in the comments, so I'd end up storing a lot of data I don't want. But upon reflection, running SingleFile on the link-without-comments pages would probably get *some* of the time savings of wget and *most* of the space savings of ODT (especially after compressing each non-current year together to reduce file proliferation), for overall improved spacetime-efficiency. The link-without-comments versions are *not* well covered in the Wayback Machine, but I'll have to try that out when they become available again (assuming they do). I could always continue doing the threads with highlighted comments in ODT to preserve that aspect.

Quarterly:

• Liferea




• Method: Open Liferea window. Under "Subscriptions" on the menu bar, click "Export Feed List". Label with ISO date and store.


• I used to use CommaFeed, but they quietly remove unread posts older than a month (I don't always get around to reading these quickly! I have that right!) and anyway fuck cloud dependence. (I *did* do quarterly CommaFeed exports back in the day, though.)


• I don't read RSS feeds on mobile and haven't looked into getting an RSS-reader app.





• XKit




• Method: Go to Tumblr. Click on XKit button. Click on XCloud tab. Click "Export". Label with ISO date and store.


• The main thing that changes from one quarter to the next is my blacklist.





• Kiwix




• Method: Go to the list of English-language Kiwix files. Do a page search for the date for three months ago (for example, the July instance begins with "2020-04"). Look at each listing with a last-updated date of 2020-04 and see if it's a new file I want OR an updated version of a file I already have (Wikipedia, Wiktionary, Ask Ubuntu). Repeat with the following three months (in this case: 2020-05, 2020-06, 2020-07).


• The idea of having each quarter overlap is to catch files made in, say, late 2020-04. Perhaps if I were careful to check at the very beginning of the month I could skip overlapping.


• You know, it might be easier to create an account at this wiki, add this page to my watchlist, ensure email notifications are enabled, and monitor changes as they come in. I'm going to try that.


• I don't have Kiwix files on my laptop because I don't quite have the space. They're just on my phone and on MEGA (in a folder exempted from desktop sync).


• Older versions of Kiwix accept files that have been split into 2 GB pieces in order to circumvent FAT32 restrictions. Kiwix has since stopped supporting this on the grounds that it is archaic. And it is, but some of us still need it for some godforsaken reason. As such, I use an archaic version of the Kiwix app to meet my archaic needs.





• LB Lee's fiction




• Method: Visit the writing masterpost and compare the "last updated" section with the last-updated section on the SingleFile version stored locally. If it's different, compare the local and remote storylists and pull the new stories with SingleFile. Repeat with the Infinity-Smashed-specific masterpost and the stand-alone story index.


• Unlike Kiwix, the thought of having these pages automatically monitored for changes *did* occur to me before today. But I did a bit of digging and couldn't figure out any way of doing that other than paid cloud services.


• I don't follow them on Dreamwidth anymore because I've grown tired of their discoursing, so I don't get story notifications that way.

Annually:

• Music to-listen list [September]




• Method: Open Rhythmbox. Look up [each artist for which I have more than just the occasional song] on Wikipedia and see if they've done anything new in the last year.


• I kept not finding out about new albums for years, so I have started actively seeking out this information. There's about 300 songs I mean to listen to but haven't yet, which is the main reason why the task isn't urgent enough to perform more often than annually. I do want to have *some* active monitoring for new songs rather than only checking after I've finished the current backlog, so that the list doesn't get *too* outdated.


• It's in September because I first did this at the end of August 2019.





• Glowfic [April]




• Method: In a virtual machine (I had enough trouble installing this software that I was worried I was going to damage my main computer attempting it there), install grab-site. Go to glowfic.com and accept the Terms and Conditions. Export the T&C cookie the same way you'd export a Dreamwidth login cookie. In a terminal, run "grab-site https://glowfic.com/ --no-offsite-links --concurrency=1 --delay=3000 --wpull-args=--load-cookies=/home/{{name of VM}}/Downloads/cookies.txt". In the resulting folder (as soon as possible after it begins, while the folder is still being actively updated with the scrape results), go into the "ignores" file and replace it with the ignores list compiled over the course of the previous glowfic scrape. After it finishes, use megawarc to combine the overflow file with the main one. Keep the entire folder (7-zipped, labelled with year and month, and stored) because I'm not entirely sure which of these files will come in handy for next year.


• One *must* use grab-site for this, even though it's more difficult to install than wget, because wget insists on downloading pages in the ignore list and *then* throwing them away. A scraper with *that* behaviour will *never* finish scraping the Glowfic Constellation (believe me, I know!): there are simply too many immaterially-different page variants.


• It's in April because, after over a year of on-and-off experimentation, the first scraper-bot iteration to successfully complete its task was in April 2020.


• (There were, uh, some failures. Sorry about that: I'd give you some financial compensation for the shit I accidentally put your servers through if Paypal weren't such an ass about international transfers.)


• (Fun fact: I literally *can't* obey the robots.txt they updated specifically in response to (malfunctioning versions of) my scraper bot, even though it's in my own interest to do so (since, as noted above, the scrape will never complete if it includes those things). AFAICT, there exist no personal-use scrapers capable of parsing wildcards in robot exclusions: only certain proprietary corporate scrapers know how to handle that. Grab-site doesn't even try to obey robots.txt as such, but I have incorporated their robot exclusions into the ignore list on my end.)


• I started throttling my scrape attempts to every 3 seconds after seeing I'd caused some problems. At one page per 3 seconds, the scrape took about three weeks of running 24/7 to complete.


• If you want a WARC of glowfic.com, you should probably just ask me for one rather than generating a copy independently. I will gladly help you out.


• For accessing the file, I've been using Webrecorder. (Unfortunately there don't seem to be any WARC readers for Android. If I were better at coding I would consider trying to do something about that.)


• (Hey wait a minute! The site maintainer has a Patreon! Last month I figured out how to use Patreon to make one-time donations by credit card in order to avoid Paypal's extortionate fees!)





• Local WordPress server [June]




• Method: Described here.


• It's in June because June 2020 is when I made the first local WordPress server to be based on a fully up-to-date version of my WordPress.

As needed:

• Chat logs




• Method: Generally copy-pasted ODTs, sometimes with minor manual tweaking depending on the chat service.


• Unlike Slate Star Codex, Discord copy-pasting works much better on *Chrome*, and not so well on Firefox. Another reason why it's always good to have a spare browser or two on hand!


• For private messages, each service/interlocutor combination gets one file. For TV-watching meetups, each conversation gets one file, labelled with ID number, date, and (if applicable: sometimes we just talk) episode. One-time-only group chats are integrated into the file for the user who invited me, since in practice I think of those chats as "a conversation with them, and also some other people". I have never hung out much in group servers: [my attempts at doing so that didn't stick] are in one file per channel.





• Comment logs




• Method: Copy-pasted ODTs, one file per blogging platform. Each comment is labelled with a timestamp: the timestamp is a link back to the associated OP.


• Under my linking policy, public posts will enter the Wayback Machine when I include them in the next comment roundup, but the file does not *locally* incorporate any context beyond what was already in the comment.





• Various AO3 fics




• Method: Sign up for email notifications when a new chapter or fic releases. When the notification comes in, go to the page and click on "Download"-->"EPUB". Label with [author's name, full story name if truncated, wordcount, most recent chapter number (to make it easier to distinguish outdated exports and know *how* outdated they are if so), ISO date] and store.


• I've had to do a couple of fics (notably Worth the Candle, which I've only read the first chapter of so far but I reserve the right to continue) by running SingleFile on the "entire work" variant of the page, because those fics had formatting that didn't translate to .epub well. I'm considering maybe doing that by default in order to catch fics I don't yet *realise* have untranslatable formatting.





• Various blogs




• Method: A mishmash of Bloxp, wget, print-to-PDF, and SingleFile, depending on how long ago I did them (many of these blogs are very dormant) and how the automation vs filtering-out-unwanted-posts vs incrementalness tradeoff shakes out in each particular case. For some of the print-to-PDFs I've gone back and replaced them with SingleFiles: others, while inferior to SingleFile, were close enough that it wasn't worth the effort to fix retroactively.





• Bank statements




• Method: Where possible, I have these set for email notifications so I know when to go to each website and download their latest PDF to stash away in my records. Last I checked Simplii doesn't offer email notifications for new statements, so I have a monthly calendar reminder to do that one.


• While I'm at it, I read them over to check for anything unexpected.





• MapFactor saved waypoints




• Method: Described here.





• Savegames




• Method: Go into the Android app-data folders for the Game Boy emulators and copy the .sav files. I don't play many local games, so desktop savegames haven't really come up lately (and mobile savegames barely have).





• andOTP




• Method: Go into the Backups section of the settings and export a password-protected file. Label with date and store.


• Finally, a TOTP app with a decent backup function! That *alone* was well worth the switch from Google Authenticator, even before the reduction of megacorp dependence. I only used Google Authenticator when absolutely necessary because I was concerned about losing access: I'm much more willing to sign up for 2FA now that I have andOTP.

---

I hope you found some of this interesting or useful. I found it surprisingly useful just to write out an explanation aimed at the public: it helped my mind correlate some more of its contents in a non-Lovecraftian way, noticing some gaps and inconsistencies in how I've been doing things.

(And if you've spotted any gaps or inconsistencies I haven't, let me know.)

If you liked this post, you might also be interested in gear I keep in my belly bag and (some) apps I keep on my phone.


(edit: addenda)

Comments

[lunartulip]

This was, indeed, helpful! I expect to consult this post semi-often over the next while as I improve my backup arrangement, and in particular am very glad you introduced me to Google Takeout. With that said, I've got a few questions about things related-to-but-not-discussed-in the post:

1. Do you know of any way to get the MEGA desktop client to default to not downloading things from the cloud except where prompted, and instead only uploading? I currently get occasional sync-related bugs where it ends up downloading something I'd rather it didn't (most notably, it occasionally fails to notice when I delete a file or folder and proceeds to undelete it, which it often takes me weeks to months to notice), and if there's a good way to prevent those without losing the core constant-backup-syncing functionality, I'd like to do so.

2. Do you currently take any precautions, or do you know of a way to take precautions, against ransomware encrypting your MEGA-based backups? This is currently one of my major concerns, and among the reasons I'm planning to get an airgapped backup drive once I can afford it; but if there are protective measures I'm missing for use in the interim then I'd like to start using those.

3. Do you have any way to keep track of updates to video games, for purposes of keeping up-to-date backups of the installers? That's a thing I've recently realized my current setup is pretty weak on, and don't currently have a good solution for.

4. Do you know of a way to do iterative, rather than full, gmail backups? I have nearly 3gb of messages archived, at present, such that doing full backups on any frequent basis will very quickly fill up my available storage.

[brin_bellway]

1. Not sure, haven't tried. Setting the maximum download bandwidth to 0 KB/s might work as a kludge?


2. I'm at relatively low risk for ransomware, in that I don't use Windows and don't currently lead a lifestyle (such as many office jobs) where people unexpectedly sending me files is normal and unsuspicious. Also, as I mentioned, part of the reason my phone only syncs with MEGA when explicitly commanded to do so is that I wanted a chance to retain an intact phone copy if anything went wrong elsewhere. Depending on the form of ransomware, one also might be able to recover one's files through MEGA's file versioning.

That being said, the above may not be entirely sufficient. I suppose this is a point in favour of increasing the priority of my "go dig an old hard drive out of the spare-parts shelf" idea, with an eye towards an eventual upgrade to something more robust.


3. Most of the games I play are online multiplayer and can't be locally backed up even in principle. I *have* been thinking of looking into a rescue operation for my Steam account, see whether I can break some stuff out of the walled garden. Now that I care more about avoiding walled gardens I likely won't buy games on Steam in the future (unless it turns out to be easy to strip the DRM, like with Amazon ebooks).

(It may also be somewhat relevant to note that I plan to do more with local software libraries in the future, including ripping some nostalgic childhood CD-ROM games (especially Age of Empires, which I would play even for non-nostalgia reasons) to .iso. I think ddrescue will probably work for that, though so far I've only tested it on DVDs (which I can't yet afford the space to rip en masse).)

Keeping track of updates likely depends on the specifics of where you're getting the updated installers from. What sort of notifications does the provider offer?


4. First of all, I should note that I don't keep every Gmail backup I make in MEGA, only the most recent. (Well, others made in the past month could presumably be recovered through version control.) But I do want to do more with making occasional permanent backups (which is the main motivator for the external hard drive in my case), so this is going to be more relevant to me soon.

Hmm. 7-zipping multiple analogous backups together does not seem to have caused as much data deduplication as I had hoped, although it *did* help a fair bit (the 7-zip containing both files is much smaller than the sum of the 7-zips of each individual file).

When you say 3 GB of messages archived, where are you getting that figure from? A bit of poking at Gmail suggests that it's easy to see a figure much higher than the true storage space required: the fine print at the bottom of mail.google.com shows the storage for *the entire Google Drive*, not just Gmail. Even if you look at the breakdown by service, that's the uncompressed figure. My Gmail is 0.77 GB uncompressed, 0.43 GB zipped, and 0.40 GB 7-zipped.

Still, even if the problem turns out not to be *as* bad as it looks, it would likely be good for us to look into better data-deduplication methods. I suspect r/DataHoarder would be a good place to start searching.

[lunartulip]

On Steam DRM: it varies heavily in how easy or hard it is to strip. There are a bunch of Steam games which don't actually have any DRM at all, and if their directories are copied to a non-Steam-containing computer will continue to run fine. There are a bunch more which, while they don't run quite that easily without Steam, are relatively trivial to hack or patch to allow them to do so. Both of these categories are documented on the Steam wiki's list of DRM-free Steam games; the information there isn't complete, but it's at least a very useful starting place.

On game updates: the main three places I'm concerned with keeping track of game updates from are GOG, Humble Bundle, and Itch, since those are the three sites on which I have largeish libraries of DRM-free games.

I'm not aware of any mechanism for tracking Humble Bundle updates short of manually scrolling through my library and seeing whether the game installers' associated checksums have changed. Itch in theory offers notification emails when new files are uploaded for a given game, but in practice I have that setting checked and don't recall having ever received such emails, even through some very unambiguous game updates; meanwhile, it offers definitely-working emails for update announcement posts, but those need to be made manually by the developers, so aren't reliably thorough. And then GOG does keep track of each game update in a relatively-practical-to-check way, although without any direct notifications beyond "check library and see what's been marked as updated since last check".

On email: when I say 3 GB archived, that's the figure given in my breakdown-by-service for Gmail, alongside 0 GB from Drive and 0 GB from Photos; I haven't tried zipping or 7-zipping my newly-exported archive, but I'd be surprised if it ended up smaller than 1 GB. I haven't encountered /r/DataHoarder before, though, so that definitely seems like a good next place to look, yup!

[brin_bellway]

>>I'm not aware of any mechanism for tracking Humble Bundle updates short of manually scrolling through my library and seeing whether the game installers' associated checksums have changed.<<

Are the checksums directly visible on a main library page, or do you have to look at each one individually?

If you can see them all lined up, I think the tactic I'd try there is to download a baseline of that page (probably with SingleFile), make a note to check back every so often (monthly? quarterly?), download a current version of the page, and then use a local file-comparison tool (maybe Meld?) to see where the differences are.

---

>>And then GOG does keep track of each game update in a relatively-practical-to-check way, although without any direct notifications beyond "check library and see what's been marked as updated since last check".<<

Yeah, I'd just make a note to check back at some reasonable frequency. Near the top of my notepad window is a set of lists of routine-backup tasks, one list per frequency.

---

>>I haven't tried zipping or 7-zipping my newly-exported archive<<

Huh, my takeout comes zipped. Maybe that's just because I request several things at once.

---

>>I haven't encountered /r/DataHoarder before, though, so that definitely seems like a good next place to look, yup!<<

Let me know if you find anything useful!

[lunartulip]

The Humble Bundle library's checksums require two layers of looking-at-each-one-individually: first I need to click into a given game's library entry (of which only one is displayed at a time), and then I need to press the button next to the installer link to display its associated checksum. So it would be pretty difficult to scrape them into an easily-diffable format, I think.

(There does seem to be an unofficial Humble Bundle API which can return checksums for requested games, which would potentially be a solution if I were very desperate; but, between the API's lack of official support (and consequently spotty documentation) and my lack of experience with API programming, trying to do anything with it seems like a pretty daunting prospect.)

[brin_bellway]

Update regarding backup deduplication: I've started using BorgBackup and so far it's going very well. I even copied the backup folder onto my offline test VM along with the .deb files for BorgBackup and its prerequisites, and after installing the .deb files I was able to successfully list the backup's files and extract a sample.

(The TV's new prosthetic brain has less storage space than I thought, so it doesn't look like I'll be able to keep it on there. I'll leave the backup on my main computer temporarily and ask for an external hard drive for birthday or Hanukkah or something.)

[contrarianarchon]

Finally got around to sitting down and reading this properly; it got shunted to "Articles-to-read" in my mind rather than social media.

Very interesting! A good reference for sensible archiving methods. Right now I don't plan to emulate a lot of this, since I don't think I have the spare memory to sustainably archive more than I already am; my laptop is at "delete unplayed video games to make room for more books" levels of memory management, if only barely (warframe being 38gb and all that).

That said, one thing that I *really* do need to do is a health log; a bunch of friends shouted at me for letting myself over-normalize chronic pain during the crisis I had on that front a few weeks ago, so I need to actually start generating data on that other than "IDK it hurts a bunch". I don't think I'm generally as diligent as you on that front (And thus have a mild expectation that your answer will boil down to "Well you remember to do the thing then do the thing, every day"), but still, do you have any tips for getting that kind of logging set up?

[brin_bellway]

>>Finally got around to sitting down and reading this properly; it got shunted to "Articles-to-read" in my mind rather than social media.

:)

Yeah, when I saw you commenting on later posts without first commenting on this one you expressed interest in, I figured that's what had happened. And reasonably so: it's over 5k words.

---

>>I don't think I have the spare memory to sustainably archive more than I already am

Just to check: have you been making much use of compression?

Collections of HTML files from the same website (such as from wget scrapes) are particularly amenable to compression, especially if they're not very image-heavy and especially especially if you use 7-zip (at least compared to regular zip; make sure to set it to "ultra" mode if you care more about storage space than about how long the compressing takes). I've gotten 100x space savings on some websites! Not everything turns out *that* well, but in most cases it's still helpful.


My new laptop has a 1 TB HDD *and* a 256 GB SSD, so I should be good for a while as long as I continue to refrain from backing up our DVDs. (Though I'll need *some* DVD files kicking around at any given time, since it doesn't have an internal DVD drive. My current plan is to watch Doctor Who by ripping discs on the TV computer and transferring the files over.)

I know you've complained in the past about low storage space from the great leap backward to SSDs. Some SSD laptops that don't *come* with a dual-storage-drive system have an empty HDD slot to allow for aftermarket upgrading (and vice versa with some HDD laptops): is yours one of them? If not, there's always external drives, though a separate unit can get unwieldy. (Of course, whether you can get more storage *within your budget* is potentially another matter, but it's good to know your options.)

---

>>I don't think I'm generally as diligent as you on that front (And thus have a mild expectation that your answer will boil down to "Well you remember to do the thing then do the thing, every day"), but still, do you have any tips for getting that kind of logging set up?

While personally I find leaving the LibreOffice document open at all times to be a sufficient reminder for the most part, this seems like the sort of thing where a lot of people would find recurring phone alarms very helpful.

The thread I got the idea from has some interesting variations and tips for how to make it a feasible habit to cultivate. I recommend reading the whole thread (it's fairly short) to see what you get out of it, but I'll give a particular shoutout to "labelling dates and symptoms on the columns and rows of a sheet of graph paper, then colouring in the appropriate squares". (That sounds like it could be pretty easily adapted to a digital spreadsheet if you'd prefer.)

(Good luck with the pain. *hug*)

---

Migrating to a new laptop, like writing a public-facing article, is a good time to take an extra-thorough stock of one's methods. I've been making a few notes on things to tweak as I settle into my new computer, and I expect I'll post them once I've finished.

[contrarianarchon]

>>have you been making much use of compression

Not in the slightest, I'm afraid - I just went and checked and "low-access HTML files" consist of <4gb of my ~450gb of currently-used memory (out of 500gb, taking into account that I recently deleted a 38gb game). So even if I got very good compression it'd be a small overall gain. Probably worth it anyway though. (esp since once I get more space I want to e.g. start making scrapes of webcomics and webnovels that I like, so having a process would be good.).

>> Some SSD laptops that don't *come* with a dual-storage-drive system have an empty HDD slot

Unfortunately, I have the vice of buying very light-weight 2-in-1 laptops (since I will be carrying them with me literally everywhere short of the grocery store); I don't think it has space for that (google is ... very unhelpful, since HP seems to name laptops in parallel across all models or something and there are similarly-named laptops in many styles.

>> there's always external drives, though a separate unit can get unwieldy

Yeah if I come up with a good plan for splitting up all my stuff into stuff that can be kept on external drives, I have two (one that I bought explicitly for backups; one that came with the process of extracting stuff from my damaged laptop when I realized I had like an ~~very sensible person~~ idiot, left the drive with the most recent backup (made just before I left by a matter of hours) in Australia two weeks after I got here, and brought a drive whose most recent backup was six months old. So I have the hardware; I don't have a schema for which book I can put up with having to go get that hardware to read. Maybe I should make a spreadsheet of what exactly is taking up how much memory.

>>While personally I find leaving the LibreOffice document open at all times to be a sufficient reminder for the most part, this seems like the sort of thing where a lot of people would find recurring phone alarms very helpful.

My related experience is with keeping a diary; this fails either way when my sleep cycle gets out of whack because the definition of "end of day" gets fuzzy. My sleep cycle is currently very out of whack, mostly because I keep trying to drag it to local time and it keeps getting dragged back to silly times by lack of IRL commitments and the desire to chat with friends who wake up at local 1am-3am.

>>The thread I got the idea from

*adds to to-read list*

>> (Good luck with the pain. *hug*)

... if I say it's not that bad you'll shout at me, no doubt, but I genuinely don't think it's that bad by chronic-pain standards. *hugs anyway* (The week I spent four (non-consecutive) days nearly unable to walk due to pain was, uh, not good, though, but that's literally the worst it's ever been)

>> Migrating to a new laptop, like writing a public-facing article, is a good time to take an extra-thorough stock of one's methods. I've been making a few notes on things to tweak as I settle into my new computer, and I expect I'll post them once I've finished.

I hope it goes well and look forward to hearing the results.

[brin_bellway]

>>So even if I got very good compression it'd be a small overall gain.

While HTML tends to have the biggest gains in my experience, a lot of things get *some* space benefit out of compression. Even sometimes things like MP3s that are *already* compressed, just not quite as highly: the ~5% savings I'm getting on lifelogger archives adds up over time.

---

>>Unfortunately, I have the vice of buying very light-weight 2-in-1 laptops (since I will be carrying them with me literally everywhere short of the grocery store)

My usual strategy is to rely on smartphones for portability-requiring uses, and buy 4- to 7-year-old high-end-at-the-time mobile workstations (= stuffing about as much into the laptop form factor as was possible back then) that I take out of the house only when travelling. I hear that tends to be less feasible for university, though.

New laptop is...I haven't weighed it myself, but judging from these spec sheets it's about 600g lighter than my previous one, and that's definitely taking some getting used to. I can pick it up with one hand! I don't have to be careful to lift with my legs!

---

>>since HP seems to name laptops in parallel across all models or something and there are similarly-named laptops in many styles.

Yeah, I ran into something similar myself. Does this laptop come with a webcam? Maybe™.

(Apparently the answer in this case is yes, but I honestly did not know until it arrived. The pictures on the listing cut off before showing that part of the monitor, so I couldn't get the information that way.)

---

I was thinking the practicality issues of an external drive would be mostly "having an extra USB device attached to your computer ~all the time", rather than "having to go and get it". But if you carry your laptop around a lot I suppose it *would* be more the latter.

[contrarianarchon]

Yeah "take my laptop to university and ttrpg-club and use it there for reading, writing, browsing, etc, plus plausibly video games/other use during down-time between those things in public places where I can't actually go home" is a central use-case for my life.

[lunartulip]

A difficulty I'm running into with health-logging, which that thread doesn't really address: in my experience, most of my most common symptoms from health issues are ones which differ from day-to-day life in degree, but not in kind. Sneezing more often, or being thirstier, or having less energy, or the like, where my baseline existence already involves nonzero amounts of sneezing and thirst and energy-limitation.

When it comes to things along those lines, how do you (a) notice when they're happening an abnormal amount (which often takes me a couple days, when I get sick), (b) decide when the abnormal amount in question is beyond the cutoff point between "normal unremarkable fluctuation" and "Something Happening", and (c) notice, after a bout of sickness, when the raised levels are gone and things are back in the "normal unremarkable fluctuation" zone? Currently, I struggle with all of those, and it's hindering my attempts at health-logging pretty heavily.

(Also hindering my attempts at health-logging: getting used to things. It's a lot easier to go "there are sores in my mouth being bothersome" two days into having them than two weeks in, because after a while, even if they continue to be bothersome, they're no longer attention-gettingly unusual.)

[brin_bellway]

>>(a) notice when they're happening an abnormal amount (which often takes me a couple days, when I get sick)

As long as you *do* tend to notice eventually, writing it down when it comes to your attention with a note of "come to think of it, this has been going on for [as good a retrospective estimate as you can make], but I hadn't paid it much mind before" is a lot better than nothing.

---

>>(b) decide when the abnormal amount in question is beyond the cutoff point between "normal unremarkable fluctuation" and "Something Happening"

I try to err on the side of noting bad-end-of-normal days just in case it comes in handy later.

With such a high overlap between the kind of symptoms you get from normal fluctuations and the kind of symptoms you get from illness, you might find it helpful to mark today's levels of [sneezing frequency, thirstiness, fatigue, etc] on 1 - 10 scales or something along those lines.

---

While not *everyone* can cultivate an unobtrusive health-monitoring background subroutine in their brain--just as not everyone can learn to balance on a bicycle--I think for the most part it does get easier as you get more practice. It's still fairly early days for me, but I think I'm beginning to see that happening.

---

>>Also hindering my attempts at health-logging: getting used to things.

I think that one *is* addressed in the thread, which recommends re-reading the previous day as a reminder to note down "oh yeah, that thing that was bothering me yesterday is still here" or "come to think of it, that one's actually gone now". A one-day buffer won't help against relapsing-remitting problems, but perhaps once you catch a problem doing that you can add it to a list of suspects.

I do have some days of missing data, neglecting to note down today's severity of [an issue I've been having most days, with fluctuating but very gradually worsening-on-average intensity, for ~1.5 years]: I try to move on from that loss and do what I can in the future. Especially on that one, because that's something where I wish I'd started making at least *some* notes on it sooner.