default header

Development

The Trove and other resources

Moderator: JC Denton

The Trove and other resources

Unread postby Beakman » 29 Sep 2020 16:44

Found a useful utility to download contents from The Trove in bulk:

https://github.com/felipegiacomozzi/the ... r/releases

I tested it and downloaded everything from the Dungeons & Dragons, Pathfinder, and Cyberpunk folders. The downloads got stuck a couple of times, but I simply closed the app via task manager, deleted the last half-downloaded file, and ran the downloader again, which checks for already downloaded items.

This is complementary to a torrent I posted some months ago on Discord about a full backup of a defunct website that, as far as I understand, was the predecessor to The Trove. It can be downloaded in here:

https://the-eye.eu/public/Books/rpg.rem.uz/

https://the-eye.eu/public/Random/torren ... 18.torrent

I haven't done a thorough file comparison check between that repository and The Trove, but it bears to say that a lot of their files can be found in both repositories. The Trove has more data in terms of GB, but I found some stuff in that rpg.rem.uz backup not present in The Trove (e.g. a bunch of Pathfinder comics).
Image
User avatar
Beakman
 
Joined: 26 Apr 2009 17:30
Location: Mexico City

Unread postby icycalm » 30 Sep 2020 21:57

We also want to download this site: https://www.tsrarchive.com/

It has only front and back covers, but many of them at higher quality than what the Trove has, and I've already used several of them to populate the maps with adventures. They will be invaluable in the long run.

danjiro has downloaded both sites. I am not sure what technique he's used, maybe he can tell us so we can compare it with yours Beak. Apparently, he can set it up to check for updates weekly, or something like that. I will definitely be downloading everything very soon once I have set up my roughly 15 TB of storage. So between the three of us, the material will be secure, and then beyond that the more people download it the better.
User avatar
icycalm
Hyperborean
 
Joined: 28 Mar 2006 00:08
Location: Tenerife, Canary Islands

Unread postby shubn » 06 Oct 2020 18:53

I downloaded TSR Archive using HTTrack: https://www.httrack.com/

The archive is 5.7 GB, and it took two hours to download.

I tried downloading The Trove using the utility that Beak posted, but it kept stopping and ignoring folders and files for no apparent reason, so I just deleted the 50 GB I had already downloaded (because I don't feel like manually checking what was downloaded properly and what wasn't), and I'll try it again at a later time using HTTrack.

Note that, if you're going to use HTTrack, you have to make sure to include the types of linked files for it to download in the project settings. For The Trove, for instance, it's probably a good idea to include PDFs.
Image
User avatar
shubn
 
Joined: 10 Jan 2012 03:17
Location: France

Unread postby danjiro » 07 Oct 2020 02:38

HTTrack is what I used. I downloaded TSR Archive no problem, but there are a few issues I still need to resolve with the Trove.

Oh, and as Beakman mentioned, rpg.rem.uz was the original Trove site back in the day. I used to go there on occasion to reference books and one day it just disappeared. Then the Trove showed up around half a year later and they had almost the exact same directory setup. They either used the rem.uz torrent to start their own site or they're related to the original rem.uz.
User avatar
danjiro
 
Joined: 08 Mar 2011 19:24
Location: Florida, USA

Unread postby shubn » 18 Oct 2020 19:42

I downloaded the following Trove folders:

- Call of Cthulhu (33.5 GB)
- Cyberpunk (4.9 GB)
- Dungeons & Dragons (310 GB)
- Fallout (181 MB)
- Pathfinder (65.5 GB)
- World of Darkness (31 GB)

In my previous post, I mentioned having to specifically include PDF files in the HTTrack settings. I did a test, and it seems I was wrong. I suspect this option has to do with linked files located on other domains than the one specified. I skimmed the manual and couldn't find this information.

Another thing I learned: HTTrack seems to limit the max transfer rate and the number of connections at first, but it is possible to increase these values over the default limit during the download, via the Mirror > Modify options menu option. I'm not sure changing the max transfer rate actually works, but changing the number of connections definitely does (I used 14), making the overall download time shorter.
Image
User avatar
shubn
 
Joined: 10 Jan 2012 03:17
Location: France

Unread postby icycalm » 19 Oct 2020 05:31

The folder sizes provide essentially a ranking of the games from best to worst lol, as I explained in Role-playing Culture. Look at Fallout with 180 MB lol.
User avatar
icycalm
Hyperborean
 
Joined: 28 Mar 2006 00:08
Location: Tenerife, Canary Islands

Unread postby icycalm » 15 Jun 2021 09:47

https://thetrove.is/

Guys the Trove is down, and my quick Twitter search suggests it’s been down for months. I hope your backups work because otherwise we’re fucked. My Battlegrounds can’t really work without the Trove, even if we have infinite money, because most of the stuff is out of print and perhaps can’t be found anywhere else. Please check your backups and report back and let’s find a way to share them between ourselves at least. I can’t even work on restarting the game without these files.
User avatar
icycalm
Hyperborean
 
Joined: 28 Mar 2006 00:08
Location: Tenerife, Canary Islands

Unread postby shubn » 15 Jun 2021 11:44

I still have the backup I posted above. I last updated it when I made the post, so it doesn't contain anything they may have added since then.

I have a couple good ways to share it, though I'll leave the specifics for private conversation, so as not to get my accounts nuked.
Image
User avatar
shubn
 
Joined: 10 Jan 2012 03:17
Location: France

Unread postby Beakman » 15 Jun 2021 11:45

The D&D and Pathfinder content can still be found in that backup from rpg.rem.uz in The Eye:

https://the-eye.eu/public/Books/rpg.rem.uz/

I have my Trove backups at home, but I'm away at the moment.
Last edited by Beakman on 15 Jun 2021 12:23, edited 1 time in total.
Image
User avatar
Beakman
 
Joined: 26 Apr 2009 17:30
Location: Mexico City

Unread postby icycalm » 15 Jun 2021 12:07

The adventure paths in that archive stop at Ruins of Azlant. They don’t have the newer ones like War for the Crown or even the updated versions of Rise of the Runelords and Curse of the Crimson Throne that we are running. Even worse, they don’t have many of the hundreds of standalone Pathfinder and Organized Play adventures without which the sandbox nature of Battlegrounds is seriously compromised. And of course that archive has no PF2 stuff which would be the second season of our Pathfinder world.

It’s not nothing, that archive has a lot, including at least some stuff from Starfinder and Traveller that shubn didn’t backup (though not the newer of this stuff), so it’s definitely valuable, but we need more.
User avatar
icycalm
Hyperborean
 
Joined: 28 Mar 2006 00:08
Location: Tenerife, Canary Islands

Unread postby icycalm » 15 Jun 2021 14:11

Some good news. This woman on Twitter says it was working yesterday, so let’s hope it will be back soon: https://twitter.com/zetasole/status/1404780720661336067

Either way I am spooked now and will prioritize amassing all that stuff and safeguarding it with multiple backups. Maybe we can put it on the Internet Computer lol, test how really decentralized it is.
User avatar
icycalm
Hyperborean
 
Joined: 28 Mar 2006 00:08
Location: Tenerife, Canary Islands

Unread postby ChevRage » 15 Jun 2021 14:48

For what it's worth, 5e content can be found here: https://5e.tools/index.html

There aren't any PDFs though, it all seems to be republished on the website itself word for word, and the images link to high resolution JPEGs. It was very convenient for me when I was running games off this system. There even seems to be some kind of integration with Foundry.

You can find the adventures under the DM dropdown tab at the top of the page, and you can download the entire website's offline functionality here: https://get.5e.tools/
Image
User avatar
ChevRage
 
Joined: 23 Aug 2012 21:14
Location: Victoria, Australia

Unread postby danjiro » 15 Jun 2021 19:30

I went to the Trove to check something a few days ago and it was up. They go down like this pretty often and usually pop back up in a day or so. I noticed they were trying to do something fancy with PDF readers so maybe it was related to that.

I checked the Trove mirror I have and it's missing quite a bit. A bunch of files got corrupted because the mirroring was running at a time when they were changing their site structure and I didn't catch it. When it pops back up I'll immediately start backing it up again. I have a better method for capturing it this time. And if for some reason it doesn't come back I can still get the missing books individually, but that would take quite a while.

My TSRArchive backup seems complete, but I'll give it a check later today just in case. That stuff would be much harder to get if it disappears.
User avatar
danjiro
 
Joined: 08 Mar 2011 19:24
Location: Florida, USA

Unread postby icycalm » 15 Jun 2021 19:49

It’s worth expanding on what shubn said, that making a backup once a year is not good enough for us. Every month new material is released that we will need down the road, even if we don’t use all of it; it’s still needed to populate the overworld and give the players choices, even though they will only end up picking some of them. But the world would be much poorer and less interactive without the extra choices.

Paizo publishes new stuff every month, maybe half a dozen products including Starfinder, and Wizards publishes one or two new things every few months. Thankfully, since I plan to track all this stuff from now on as Insomnia starts reporting on TTRPGs, I can save the new stuff manually every month, once we have secured the backlog. And then a couple times a year we update our backlog backups for good measure.

The point is we need to get organized in this, we won’t be able to play it by ear any more once the world is in full swing.

For now, I’ll keep refreshing the Trove 20 times a day until it’s back. At that point I will post suggestions on how to proceed. In the meantime each of us should make plans according to your tech knowledge and hardware capacity to backup whatever you can. We need as much redundancy as possible.

Note that on top of the stuff shubn posted above that he has backed up, we need also at least Starfinder, Exalted and Traveller. I will make a more complete list soon, and feel free to drop suggestions. Sadly, it doesn’t sound like anyone of us can backup the ENTIRE Trove at this moment.
User avatar
icycalm
Hyperborean
 
Joined: 28 Mar 2006 00:08
Location: Tenerife, Canary Islands

Unread postby ExiledOne » 15 Jun 2021 21:53

I'm making a backup of the TSRArchive using HTTrack as well.
Image
User avatar
ExiledOne
 
Joined: 14 Jun 2008 10:43
Location: Maryland, USA

Unread postby icycalm » 19 Jun 2021 23:28

GREAT NEWS! PRAISE THE LORD ASMODEUS!

https://www.reddit.com/r/TheTrove/comme ... e_is_down/

NotAPoetButACriminal wrote:TheTrove is down

I have no clue why, I'm just a user that got annoyed about the fact there is no place to discuss whenever problems with the website occur (which has been increasingly frequent), so I made a subreddit. Anyone who has any idea please feel free to share.

EDIT: As per /u/feevian's comment, someone emailed the mods and they said they are reorganizing their folder structure.

EDIT: As per /u/pandacg's removed post, the website may be accessed using the wayback machine website and finding an older version. The post was removed cause it had a direct link and I dont wanna get us banned.


That said let’s not get complacent. Any of you guys that can afford to buy some hard drives for this, please do so. We will soon coordinate our efforts so that we have every file backed up by at least two people among us.
User avatar
icycalm
Hyperborean
 
Joined: 28 Mar 2006 00:08
Location: Tenerife, Canary Islands

Unread postby icycalm » 12 Sep 2021 10:18

Can you guys who are good at this stuff save a copy of this entire website?

https://www.d20pfsrd.com/

It's tremendously useful, as it combines rules from a variety of sourcebooks, not just the "Core Rulebook", and it's not maintained by Paizo but just by a guy with a Patreon, so it could go down at any time like the Trove.


EDIT: In fact, go ahead and backup all the similar sites he's running for a bunch of RPGs, listed in his Patreon bio: https://www.patreon.com/d20pfsrd

It'll be worth the effort in the long run.
User avatar
icycalm
Hyperborean
 
Joined: 28 Mar 2006 00:08
Location: Tenerife, Canary Islands

Unread postby icycalm » 22 Sep 2021 20:51

The Trove is back, sort of! It's back in torrent form. PARTIAL torrent form. But they are working to make it complete.

FIRST OFF, if you have the hard drive space, back up everything in the post above this. This post: https://culture.vg/forum/topic?p=34448#p34448

It's very important stuff!

Once you've done the above, go here and start downloading the new Trove torrent: https://www.reddit.com/r/TheTrove/comme ... rent_form/

Report back here with your progress. This stuff is very important to the Battlegrounds projects, guys, so pitch in to get this done.
User avatar
icycalm
Hyperborean
 
Joined: 28 Mar 2006 00:08
Location: Tenerife, Canary Islands

Unread postby ExiledOne » 23 Sep 2021 10:36

I don't know if anyone else had trouble grabbing the Patreon sites with the usual tools (like HTTrack) but a couple I tried did not work. There's a program in the Microsoft Store called Site Snatcher. I was able to grab the first two with it and I'm working on the third now so I feel safe in recommending it. It's a paid app but it's very cheap.
Image
User avatar
ExiledOne
 
Joined: 14 Jun 2008 10:43
Location: Maryland, USA

Unread postby shubn » 23 Sep 2021 17:45

I started backing up d20PFSRD using HTTrack. It's taking a long time because there are so many pages. I also encountered a couple of errors, so I'll tell you how I fixed them.

"Unauthorized" (or "Forbidden") when first starting the download: in the options, "Browser ID" tab, set "Browser Identity" to "(none)".

"Too many URLs, giving up": in the options, "Limits" tab, set "Maximum number of links" to the max (10000000).
Image
User avatar
shubn
 
Joined: 10 Jan 2012 03:17
Location: France

Unread postby icycalm » 30 Nov 2021 22:24

I've found some stuff we're missing.

Latest D&D: https://mega.nz/folder/QxdjBI4Z#5zlMSioceak5tFaflPN5KQ

Starfinder, Blackmoor, Shadowrun, Paranoia: https://cloudflare-ipfs.com/ipfs/Qmf4L3 ... y6rrcKtSNn

Also, Exalted 1E and 2E are inside the Storyteller folder, and 3E is outside.

I am grabbing as much as I can, and once I have these essentials I'll see what else is worth grabbing. Probably everything mentioned here: https://www.patreon.com/posts/56589089
User avatar
icycalm
Hyperborean
 
Joined: 28 Mar 2006 00:08
Location: Tenerife, Canary Islands

Re: The Trove and other resources

Unread postby ExiledOne » 21 Dec 2021 22:28

So far, I have everything except the stuff that was listed from IFPS.

I had trouble downloading it for whatever reason the first time I tried it and now the library there was removed.

But I have all the files from the torrents, back-ups of the websites (a few are no longer online so I had to use the Wayback Machine to copy them), and the Mega D&D 5E files. The Wayback Machine back-ups aren't as easy to navigate though. They don't let you just click on the links offline, but all the pages are there if you find and click on the index.html files manually in your local file system. The others work normally as you'd expect a normal website would. I separated the Wayback Machine back-ups from the others in a separate folder.
Image
User avatar
ExiledOne
 
Joined: 14 Jun 2008 10:43
Location: Maryland, USA

Re: The Trove and other resources

Unread postby ExiledOne » 15 Jan 2022 20:37

I found a couple of the one-shots that were listed in the Patreon post on this Telegram channel: https://t.me/TheAmberRoom

If you install the program on Windows, it's easier to search through the channel for particular books and download/organize them. So far I've found the Alien adventure, the Dune adventure, a bunch of the Conan stuff, and the Blackmoor adventures (which I had to write a request for) in the channel.
Image
User avatar
ExiledOne
 
Joined: 14 Jun 2008 10:43
Location: Maryland, USA




Return to Development