[META] Automatic downloading | Megalinks MegaDB

[-] ruralcricket | 3 points | Apr 11 2017 23:30:45

Probably not since posters are using various methods to prevent automation from working. Did you see the posting that used stenography to hide the links?

I'd also advise against auto downloading all of the content of open directories. Beside the need to have petabytes like /u/Dougygob mentioned, you also risking downloading much duplicate content, poor content (480p movies), content you really don't want in your possession, personal financial documents (e.g. tax returns). Finally, you get to weed through the crap to find the stuff you want to keep. Oh, and do you have usage limits on your ISP?

Example - I like e-books. Found a list of Calibre libraries. WGET'd just one | Megalinks MegaDB one of them. I now have over 360,000 files to go through!

[-] greytmeam | 2 points | Apr 12 2017 00:05:57

You bring up fair points. I have unlimited storage, so file amount shouldn't be an issue and sorting can be done essentially through filebot and similar scripts, but it would still require some sorting and removing of duplicated (and low quality) content. Ultimately, I might go back to using Sonarr, Headphones, and CouchPotato to auto-torrent, but I was hoping to avoid it. If you care to share the directories you used for books (and whatever else you might want to offer) though, I'd love to visit them and download (PM or comment). Thanks a ton for the input!

[-] Dougygob | 5 points | Apr 12 2017 00:20:28

I'd be happy to help with the sorting! (I have no life, I play video games all day and check /r/megalinks three times a hour.)

[-] PainDoflamiongo | 3 points | Apr 12 2017 14:42:06

bruh i need an income source like yours.

[-] Dougygob | 2 points | Apr 12 2017 17:33:45

I have none, I really live with my parents and I'm homeschooled, Moved too. No friends, Just video games. Pretty neato life tho, Excluding the loneliness and other shit.

[-] Cartmanishere | 2 points | Apr 12 2017 18:15:16

Can relate.

[-] greytmeam | 2 points | Apr 12 2017 18:35:57

Haha, that's kinda awesome. I'm pretty much in a similar situation -- high school is getting pretty boring, so I'm just dicking around with this type of stuff. Pro tip: try to get a girlfriend and/or find hobbies. For me: getting a girlfriend, rock climbing, sailing, etc. were the solution.

[-] greytmeam | 1 points | Apr 12 2017 18:34:08

Thanks a ton! For now, I'm probably gonna do most of it on my own, but I can send you links to it when it's in a somewhat stable state (and has a decent amount of content). Always appreciate help!

[-] ruralcricket | 3 points | Apr 12 2017 02:57:20

https://www.reddit.com/r/opendirectories/comments/5dw9gr/ebook_database_of_over_200000_ebooks/

[-] greytmeam | 1 points | Apr 12 2017 18:33:07

Awesome, thanks!

[-] ruralcricket | 1 points | Apr 12 2017 03:04:57

Also I have this list laying around of Calibre sites (probably outdated). You need to visit the site and then modify the num=xxx to be a value larger than the actual # of books.

then use wget

wget -rm -c --level=0 -np --accept "azw,azw3,azw4,cbz,cbr,chm,djvu,docx,epub,fb2,lit,lrf,mobi,odt,pdf,pdb,pml,rb,rtf,snb,tcr,txt,txtz" -i sitelist.txt

the -i sitelist.txt assumes the list is in a file, you could of course just do one at a time.

the contents of sitelist.txt https://pastebin.com/1kZx9BYX

[-] greytmeam | 1 points | Apr 12 2017 18:33:01

Awesome, thanks a bunch! I'll be sure to do that this weekend when I have a bit more time on my hands.

[-] Cartmanishere | 1 points | Apr 12 2017 18:18:08

This isn't the one you asked for. But this site has quite a lot of mobi ebooks. Probably in hundreds of thousands.

<a href="http://vanhoosier.com/books/">HERE</a> | Megalinks MegaDB HERE

[-] greytmeam | 1 points | Apr 12 2017 18:32:40

Awesome, thanks! I'll check it out this weekend. I typically use ebook.bike, but I'm not sure how to bulk download from there with scripts (tbh, I'm pretty bad at scripting).

[-] Cartmanishere | 1 points | Apr 12 2017 18:38:36

You can use wget to download this. Roughly 8.2GB.

wget -r --no-parent --reject "index.html*" "http://www.vanhoosier.com/books/"

Here's the relevant thread on /r/opendirectories.

[-] sneakpeekbot | 2 points | Apr 12 2017 18:38:41

Here's a sneak peek of <a href="https://np.reddit.com/r/opendirectories">/r/opendirectories</a> using the <a href="https://np.reddit.com/r/opendirectories/top/?sort=top&t=year">top posts</a> of the year! | Megalinks MegaDB Here's a sneak peek of /r/opendirectories using the top posts of the year!

#1: Every Season of Game of Thrones in 1080p - Updated Weekly | 22 comments
#2: Princeton.edu's FTP server, it contains thesis's submitted by every student from every year since the mid 1980's, among various other files hosted by the university. | 27 comments
#3: 2.7TB EN MSDN Dump [all microsoft products from its developer network] | 75 comments

^^I'm ^^a ^^bot, ^^beep ^^boop ^^| ^^Downvote ^^to ^^remove ^^| ^^Contact ^^me ^^| ^^Info ^^| ^^Opt-out

[-] greytmeam | 1 points | Apr 12 2017 20:55:11

Awesome, thanks! Do you by chance know if I can do something similar for sites like ebook.bike? I recall them having some token based download thing, but I'm not sure.