« previous next »
Pages: 1 ... 30 31 32 33 [34] 35 36 37 38 ... 42   Go Down

Author Topic: Any and all site related problems  (Read 223425 times)

0 Members and 1 Guest are viewing this topic.

Offline smokester

  • Administrator
  • Q
  • *
  • Posts: 15941
  • Gender: Male
  • Da mihi castitatem et continentiam, sed noli modo!
Re: Any and all site related problems
« Reply #495 on: February 23, 2016, 02:43:15 AM »
Anyone else getting the "connection problems" page? The server load looks ok so I'm not sure why that would be happening?
Logged
Don't put off until tomorrow, what you can put off until the day after.

There is an exception to every rule, apart from this one.

Offline goldshirt*9

  • Super Hero
  • *******
  • Posts: 7384
  • Gender: Male
  • Who yous looking ats
Re: Any and all site related problems
« Reply #496 on: February 23, 2016, 09:15:36 AM »
just did.
yesterday was offline also
Logged

Offline smokester

  • Administrator
  • Q
  • *
  • Posts: 15941
  • Gender: Male
  • Da mihi castitatem et continentiam, sed noli modo!
Re: Any and all site related problems
« Reply #497 on: February 23, 2016, 10:07:01 AM »
Quote from: goldshirt*9 on February 23, 2016, 09:15:36 AM
just did.
yesterday was offline also

Watching the server load it keeps exceeding 10 which would cause the site to fail.  Either they've oversold it or someone is hogging the resources.

I'll inform them that it's clipping.
Logged
Don't put off until tomorrow, what you can put off until the day after.

There is an exception to every rule, apart from this one.

Offline smokester

  • Administrator
  • Q
  • *
  • Posts: 15941
  • Gender: Male
  • Da mihi castitatem et continentiam, sed noli modo!
Re: Any and all site related problems
« Reply #498 on: April 15, 2016, 02:15:30 PM »
As some may have noticed, we've had very serious server issues.

Apparently things were so bad they had to install new drives and re-install from backup as things could not be recovered and the server, no matter what they tried, kept becoming unresponsive.

We're back now but I have no idea if we've lost anything.
Logged
Don't put off until tomorrow, what you can put off until the day after.

There is an exception to every rule, apart from this one.

Offline xtopave

  • Site Modette
  • Q
  • *
  • Posts: 28876
  • Gender: Female
Re: Any and all site related problems
« Reply #499 on: April 15, 2016, 08:22:56 PM »
I'm glad we're back!! I was worried for a while today.

Quote from: smokester on April 15, 2016, 02:15:30 PM
...but I have no idea if we've lost anything.

We've lost a very important post about my cousin setting up a pool table in his office.  :D
Logged

Offline dweez

  • Global Moderator
  • Q
  • *
  • Posts: 11622
  • Gender: Male
  • Rebel Mod
Re: Any and all site related problems
« Reply #500 on: April 15, 2016, 10:18:36 PM »
I was wondering what was going on.  I checked some of the old "hosting status" bookmarks I had but they ended up being pretty old (2011) and I guess we've moved since then.

xtopave, if only we knew someone who still new the story who could repost it.  Hmmm.
Logged
--dweez

Offline goldshirt*9

  • Super Hero
  • *******
  • Posts: 7384
  • Gender: Male
  • Who yous looking ats
Re: Any and all site related problems
« Reply #501 on: April 16, 2016, 12:48:02 AM »
I lost my lucky lottery numbers i kept here   ;D ;D ;D ;D

Logged

Offline smokester

  • Administrator
  • Q
  • *
  • Posts: 15941
  • Gender: Male
  • Da mihi castitatem et continentiam, sed noli modo!
Re: Any and all site related problems
« Reply #502 on: April 16, 2016, 04:50:10 AM »
Here's the legend of events (for dweez if no-one else) and all things considered, I think we didn't do too bad.  Mind you, isn't a RAID setup meant to avoid this kind of situation?

Quote

04-12-2016, 06:24 PM #1
Nasir

Maintenance -- Cedar.nocdirect.com [Completed]

    Reseller server cedar.nocdirect.com has gone unresponsive all of a sudden, our data center admins are checking it. More updates to follow.

    Thanks for your patience and understanding.


04-12-2016, 08:04 PM #2
Nasir

    Our data center admins are still working on the server, more updates to follow.



04-12-2016, 10:31 PM #3
Nasir   

    Please accept our since apologies, server is not booting due to drive issues, our admins are working hard to get it back online as soon as possible. More updates to follow.


04-12-2016, 10:46 PM #4
Nasir
   

    Server is back online now but RAID array is rebuilding at 13%, it is an I/O intensive process so it may push the load to go occasionally high. We will continue to monitor the server and keep you updated here.

    Thanks for your tremendous cooperation and patience.

04-13-2016, 12:02 AM #5
Vivek
   

    RAID rebuild has completed 20% successfully. More updates will follow.

04-13-2016, 02:16 AM #6
Vivek


    RAID rebuild has completed 64% successfully. More updates will follow.


04-13-2016, 02:55 AM #7
Vivek

    5,190   

    RAID rebuild has completed 67% successfully. More updates will follow.


04-13-2016, 03:26 AM #8
Vivek


    RAID rebuild has completed 72% successfully. More updates will follow.


04-13-2016, 04:03 AM #9
Vivek
   

    RAID rebuild has completed 82% successfully. More updates will follow.



04-13-2016, 04:58 AM #10
Anoop


    RAID rebuild has completed 91% successfully. More updates will follow.


04-13-2016, 06:09 AM #11
Vivek
   

    RAID rebuild has completed 99% successfully. More updates will follow.


04-13-2016, 08:27 AM #12
Vlad
   

    The rebuild got restarted and it is currently 34% complete. Updates will follow.

04-13-2016, 10:39 AM #13
Vlad
   

    RAID rebuild has completed 66% successfully. More updates will follow.

04-13-2016, 01:54 PM #14
Vlad
   

    The rebuild has completed but the raid array is still degraded. We've started an initialization process which should fix that. Updates will follow.

04-13-2016, 03:10 PM #15
Nasir


    RAID initialization 17% completed. More updates will follow.


04-13-2016, 05:56 PM #16
Nasir


    RAID initialization 53% completed. More updates will follow.


04-13-2016, 07:04 PM #17
Nasir


    RAID initialization 67% completed. More updates will follow.


04-13-2016, 09:30 PM #18
-Nasir


    RAID initialization was about to complete but server went unresponsive again. Our data center admins are checking it at this moment, we will post more details as soon as available.

    Thanks for your patience.


04-13-2016, 10:17 PM #19
-Nasir
   

    Our data center admins are still working on the server, more updates to follow.


04-13-2016, 11:06 PM #20
-Vivek
   

    We are still working with our cage techs to bring the server back online. More updates will follow.



04-13-2016, 11:43 PM #21
-Vivek


    We are still working with our cage techs to bring the server back online. More updates will follow.


04-14-2016, 12:24 AM #22
-Vivek


    It is an update that the server hangs during OS boot and we are troubleshooting why it happens to get the server loaded as quickly as possible. More updates will follow.


04-14-2016, 01:18 AM #23
-Vivek

    We are still working on this server to bring it back online. More updates will follow.


04-14-2016, 02:43 AM #24
-Vivek
   

    We are still working with our cage techs to bring the server back online. More updates will follow.


04-14-2016, 02:53 AM #25
-Vivek

    One of the recently replaced drives of the server is failing the rebuild due to which the server is not booting. We have booted the server on the rescue mode and are trying to restore the services on it and get it back up at earliest.



04-14-2016, 04:33 AM #26
-Vivek


    We are still working with our cage techs to bring the server back online. More updates will follow.


04-14-2016, 05:32 AM #27
-Vivek


    We are still working with our cage techs to booted the server on the rescue mode and are trying to restore the services on it and get it back up at earliest. . More updates will follow.


04-14-2016, 06:24 AM #28
-Vivek


    We are still working with our cage techs to booted the server on the rescue mode and are trying to restore the services on it and get it back up at earliest. . More updates will follow.



04-14-2016, 07:57 AM #29
-Vlad


    We are still working on the server. Updates will follow.

04-14-2016, 08:17 AM #30
-Vlad


    The server will have to be restored from backups. We are preparing a new server for that purpose. Updates will follow.

04-14-2016, 10:36 AM #31
-Vlad
   

    The server is ready and we are working on starting the restore.

04-14-2016, 12:00 PM #32
   

    The restore is currently in progress. Updates will follow.

04-14-2016, 01:31 PM #33
-Vlad

    110GB of data have been restored. Updates will follow.

04-14-2016, 02:56 PM #34
-Jacob


    The restore has reached 27% at this time. Thank you for your ongoing patience, and we'll continue to provide updates regarding this process.

04-14-2016, 05:12 PM #35
-Jacob
   

    The restore has reached 57% at this time. Thank you for your ongoing patience, and we'll continue to provide updates regarding this process.

04-14-2016, 08:57 PM #36
-Jacob

    The restore is over 90% at this time, and ongoing. Thank you for your ongoing patience, and we'll continue to provide updates regarding this process.

04-14-2016, 10:56 PM #37
-Vivek

    The restore has completed successfully and we will now attempt to boot it into OS. Thank you for your ongoing patience, and we'll continue to provide updates regarding this process.


04-15-2016, 12:19 AM #38
-Vivek


    We are working with our cage techs to restore the services on it and get it back up at earliest. More updates will follow.


04-15-2016, 01:22 AM #39
-Vivek


    We are still working with our cage techs to restore the services on it and get it back up at earliest. More updates will follow.



04-15-2016, 02:22 AM #40
-Vivek


    We are still working with our cage techs to restore the services on it and get it back up at earliest. More updates will follow.

04-15-2016, 03:25 AM #41
-Vivek


    We are still working with our cage techs to restore the services on it and get it back up at earliest. More updates will follow.



04-15-2016, 04:22 AM #42
-Anoop


    Our admins are still working with our cage techs to restore the services on it and get it back up at earliest. More updates will follow.

04-15-2016, 05:21 AM #43
-Vivek


    Our admins are still working with our cage techs to restore the services on it and get it back up at earliest. More updates will follow.

04-15-2016, 06:17 AM #44
-Vivek
   

    We are still working with our cage techs to restore the services on it and get it back up at earliest. More updates will follow.



04-15-2016, 08:36 AM #45
-Vlad


    The restore has been completed and the server is back online.
Logged
Don't put off until tomorrow, what you can put off until the day after.

There is an exception to every rule, apart from this one.

Offline dweez

  • Global Moderator
  • Q
  • *
  • Posts: 11622
  • Gender: Male
  • Rebel Mod
Re: Any and all site related problems
« Reply #503 on: April 16, 2016, 01:32:43 PM »
RAID is a way of configuring hard drives (hdd) so that in the case of a hdd failure data isn't lost and the server can still operate, albeit in a degraded state.  Once the failed hdd is replaced, the RAID rebuilds that hdd with the data that was on the failed one.

Some RAID configurations allow for being able to lose more than one hdd and still be functional but for most of the common configs, if more than 1 hdd is lost, the whole array and all data is lost.

It sounds like they lost an hdd, replaced it with what they thought was a good hdd but the rebuild process kept failing.  One of those failures cause the raid array data to become corrupted.  Thus, after sorting out and replacing any of the bad hdds, they would have had to restore from backups.

Sounds like they did what they were supposed to but kept having bad luck (bad hdd, bad replacement hdd, corrupted array).  All in all, it sounds like despite all the obstacles, they did a good and quick job of restoring back to the latest data they had.
Logged
--dweez

Offline smokester

  • Administrator
  • Q
  • *
  • Posts: 15941
  • Gender: Male
  • Da mihi castitatem et continentiam, sed noli modo!
Re: Any and all site related problems
« Reply #504 on: April 16, 2016, 04:04:07 PM »
The biggest problem for me was that I run a few different email accounts through various domains, most of which are for professional use.  I was really worried that during the downtime the mail would be lost, but thankfully that was not the case.
Logged
Don't put off until tomorrow, what you can put off until the day after.

There is an exception to every rule, apart from this one.

Offline smokester

  • Administrator
  • Q
  • *
  • Posts: 15941
  • Gender: Male
  • Da mihi castitatem et continentiam, sed noli modo!
Re: Any and all site related problems
« Reply #505 on: September 16, 2016, 02:31:07 AM »
Load averages are more than twice what they should be. I've alerted them but page loading will probably be shaky for a while.
Logged
Don't put off until tomorrow, what you can put off until the day after.

There is an exception to every rule, apart from this one.

Offline brickbatz

  • Cro-Magnon
  • ****
  • Posts: 803
  • Gender: Male
  • Politically Incorrect
Re: Any and all site related problems
« Reply #506 on: September 16, 2016, 05:09:06 AM »
Page loading is fast for me.
Logged

Offline smokester

  • Administrator
  • Q
  • *
  • Posts: 15941
  • Gender: Male
  • Da mihi castitatem et continentiam, sed noli modo!
Re: Any and all site related problems
« Reply #507 on: September 16, 2016, 03:50:02 PM »
Quote from: brickbatz on September 16, 2016, 05:09:06 AM
Page loading is fast for me.

Yes, I think they kicked the spammer or whoever it was hogging the resources.

Logged
Don't put off until tomorrow, what you can put off until the day after.

There is an exception to every rule, apart from this one.

Offline smokester

  • Administrator
  • Q
  • *
  • Posts: 15941
  • Gender: Male
  • Da mihi castitatem et continentiam, sed noli modo!
Re: Any and all site related problems
« Reply #508 on: September 17, 2016, 03:15:08 PM »
Logged
Don't put off until tomorrow, what you can put off until the day after.

There is an exception to every rule, apart from this one.

Offline dweez

  • Global Moderator
  • Q
  • *
  • Posts: 11622
  • Gender: Male
  • Rebel Mod
Re: Any and all site related problems
« Reply #509 on: September 17, 2016, 03:54:31 PM »
I thought that was New Hamsterdam?
Logged
--dweez
Pages: 1 ... 30 31 32 33 [34] 35 36 37 38 ... 42   Go Up
« previous next »