how to REbuild a database.

symptoms.  comic-con was dead to the server status page
I could log in with no problems.  the log file showed mysql restarting over and over on a read error.
I rebooted and i saw the same thing in the logs.
at that point i decided to rebuild the database.  that would take some time, so I temporarily removed any references to this database from the AX servers.

on karen 
modified ~/front/*/live_server/ax_alert_server/*txt to get rid of references to comic-con
updated datebase to remove references to comic-con
mysql -uroot -p -hwally-balls mydb
update monitor_alive set disabled="Y"  where name="comic-con" ;

on comic-con
killed mysql
started with /etc/init.d/mysqld stop to prevent automatic retries.  
it said it failed so i went to top and did a kill 9 on the process
From this point we're mostly following the procedure for building a database from scratch.  Three important changes.
  Remove the directory /var/lib/mysql/mydb/ and it's contents.
  After we can log into mysql, before we create mydb, be sure to say "stop slave".
  Don't forget the following rm statement.  
rm /var/lib/mysql/ib_logfile?
emacs /etc/my.cnf
change innodb_data_file_path = ../../../../../../dev/sdb1:140011Mraw to innodb_data_file_path = ../../../../../../dev/sdb1:140011Mnewraw
Make sure you delete the log files and change the my.cnf together or you will waste a lot of time before you see an inevitable error message. 
/etc/init.d/mysqld start
tail -f /var/log/mysqld.log
restore /etc/my.cnf before you forget!
You could copy the main part of the database at any time because it's done in a transaction, but you probably don't want to do it when things are too busy.  i.e. wait until after market hours.
You definitely don't want to copy the alerts table until after we've stopped sending new alerts.  It might be okay to do it before the maintenance cycle when we delete stuff, but it might be faster and easier if you wait until that's done.



This database was last rebuild on 6/13/2009.  Similar errors at that time.  Unable to read correctly.  It looks like it also crashed and restarted on 5/28, 6/8, 8/20, 9/15, 9/16, and 9/17.  Some of those times looked similar, a read error.  Most of those times it seemed to restart itself, but at least once I had to reboot it.  It is now 9/21/2009 as it is being rebuilt again.  This is very unusual.  I double checked, and two other machines initially built on the same date have had no problems.  I will order a replacement disk.