How to improve mod file hosting while reducing risk of loss

Discussion in 'Off Topic' started by Shovas, Jun 21, 2018.

  1. Shovas

    Shovas
    Premium

    Messages:
    452
    Ratings:
    +137
    Over at the Nogripracing.com gone? thread, many are reliving the pain of losing access to mod files when the sites hosting them have to shutdown.

    I've been considering an idea for a solution that utilizes bittorrent which, running a simple app on a home computer or hosted server, can provide potentially safer file hosting by distributing files across many stakeholders and decentralizing ownership so that if one goes down it's not such a big problem as if one whole website goes down.

    By building a small swarm of bittorrent peers among those dedicated to the mods, each hosting all or part of a given mod file collection, we reduce the risk of complete loss by having the data duplicated across many nodes as opposed to concentrated in one location such as is the case right now with NoGrip, RD, EEC, and many other sites.

    The question is, what kind of will exists in the community to move ahead with something like this? How many people would contribute by running a bittorrent client, on a home computer or server, and sharing mod files in the swarm?

    Additionally, are there any others like me who, perhaps, have a personal server they could dedicate some resources from to achieve at least a couple of 24/7 nodes, since presumably home connections wouldn't be seeding 24/7. Could RD, EEC, or other existing sites contribute server resources?

    It seems a real shame to lose accumulated years or resources on a site like NoGrip out of the blue, so I'm interested in gauging the community's interest and will to move on something like this.

    Problems:
    • Where are files uploaded to initially?
      • Modders might want to seed their own files to begin with
      • Or, certain nodes might provide upload capabilities
    • How would other nodes sync the newest files and start seeding them?
      • I'm not sure about an automated way to achieve this although bittorrent clients may allow for this in some way.
      • Otherwise, a manual process of request-to-seed notifications (ie. email, tweet, forum thread, etc.) sent to nodes might be employed.
    • Files are hosted but how to find them?
      • We'd still need easily searchable catalogs like RD, EEC, etc. provide
    • Too many files / too large for any one node to host
      • Each node can decide what they want to host. For example, I might host only GTR2 mod files, another might host only AC files, etc.
      • Each node can decide what sub-section of files to host. For example, one node might host only tracks and cars, while another might host patches and documents.
    • Home users don't have the bandwidth
      • Bittorrent clients can be bandwidth limited so the impact is minimal to you
      • Bittorrent is designed to aggregate bandwidth from many nodes so that many small nodes together can provide enough bandwidth for fast downloads
    • Home users computers aren't up 24/7
      • Hopefully, having many nodes around the world, a few would presumably be running at any given hour of the day.
      • Additionally, servers that are up 24/7 would be part of the swarm providing access at all times
      • Ideally, sites like RD, EEC, and others would contribute as it would lower their costs to offload their file hosting downloads.
    • Sites like RD wouldn't like taking away file hosting
      • Actually, it would take away the 'bad' kind of traffic (raw downloads) they don't like, leaving them with only the 'good' kind of traffic (ad-filled pages) used to link the mod file downloads. They'd save on bandwidth which is a significant cost for sites like RD.
      • Sites would still be used for the catalog and notifying of new versions. Discussions would still happen on the sites' forums.
    There's much more to think through and consider about organizing such an operation but I felt it was worth talking about given the fall of NoGrip and many other sites like it in the past.

    In the future, we'll be looking at the 'persistent web' where, by design, data loss like this won't happen, and peer-to-peer technology like bittorrent will underpin it. Until then, though, we do have the technology already available, with a little effort, and if the will is there to make it happen.
     
  2. Kjell Nielsen

    Kjell Nielsen

    Messages:
    276
    Ratings:
    +28
    many things has been used to spread files with friends or public it has been with ftp or http servers with pw or without pw
    with dc++ and torrent but someone must host the files or been online otherwise is the files "missing"
    like when megaupload site close down https://en.wikipedia.org/wiki/Megaupload

    When a site close down for some reason and not people have download all the files that a site have hosted
    then is the files "lost" like if you tube should close or facebook twitter instagram it is a small online problem
    it is not so important if a site close
    get over it and take a tour on google to search after the files
    if not Google close the service so people not can search online it can happend we never now .
     
  3. Bram Hengeveld

    Bram Hengeveld
    RaceDepartment Founder Staff Premium

    Messages:
    44,242
    Ratings:
    +17,590
    You forget a really important thing. We don't vanish without a trace @Shovas. W are properly backing up our files, have a top notch tech admin team looking after the site every day and if we ever go down in the future you will be notified up front. :)

    That other sites pull the plug out of the blue doesn't mean others sites like ourselves and other will do the same.

    RD, online since april 2006 and counting..... A little bit more trust in us and others contributing to the simracing community would be nice.
     
    • Beer Beer x 2
  4. Shovas

    Shovas
    Premium

    Messages:
    452
    Ratings:
    +137
    I do appreciate all that, really I do, but we all know things come and go. How many services has Google shuttered with huge numbers of active users. I ran a free web host with thousands of members from 1999 to 2010. Even I closed a service with lots of activity ;)

    Don't take it personally, I've just gone through enough of this type of thing that I think it's a good opportunity to take a step back and reconsider if what we're doing is the best approach given the change in technology.

    But, back to you guys at RD, and please take this as innocent conversation, I would think offloading your downloads would be very attractive to you guys? Have you guys had any chats about taking on an idea like this? RD could save money and you'd be providing insurance/assurance to modders worried about losing files if you were to close, or be hacked, or other issues.
     
  5. Benutzername

    Benutzername
    I breathe a lot. I guess i just really like air.

    Messages:
    8,082
    Ratings:
    +2,573
    A bit off-topic, but wasnt there something several years ago where many downloads ere lost?
    I barely remember, wasnt active during that time but only lurked a few times for mods.
    The site looked quite different back then.
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.