Brin (![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png)
brin_bellway) wrote2019-10-14 02:01 pm
brin_bellway) wrote2019-10-14 02:01 pm
PSA
If you mark a Dreamwidth post as "viewer discretion advised" (presumably 18+ would do this as well), the Internet Archive cannot crawl it. Whether you consider this a downside or an upside is up to you, but it's worth knowing either way.
(If one specifically *sets out* to scrape Dreamwidth one can edit one's scraper bot to include a "show me discretion-advised posts" cookie, but generalised scrapers like the Wayback Machine can't get past the "are you sure you want to continue" page.)
(If one specifically *sets out* to scrape Dreamwidth one can edit one's scraper bot to include a "show me discretion-advised posts" cookie, but generalised scrapers like the Wayback Machine can't get past the "are you sure you want to continue" page.)