ubuntu - MySQL Replication was working, now broken -

so month+ ago setup replication 1 of web servers m local lan servers. had 2 local lan servers slaves (quad , hex) master web server (falcon). weeks replication worked without issue.

then noticed earlier week i/o thread wasn't running on either quad or hex. restarted them on , on , on again because every run normally, @ least 1 of slave, several minutes. i/o thread stop working again. have attempted reconfigure master on both lan machines via phpmyadmin , phpmyadmin reports slave connected master. phpmyadmin gives me option (that wasn't yesterday) of synchronizing slave dbs master's. run , in few seconds page stops loading no response (error or otherwise) given.

all of machines involved running lamp stack on ubuntu 12.04. web server 12.04 server while lan machines both 12.04 desktop. suspected had run apt-get update/upgrade on lan machines while neglecting on falcon. ran on 3 machines morning. still i/o thread won't run.

the sql thread running. occasionally, master shows 1 of slaves (hex) connected. doesn't stay connected.

like said worked weeks without issue. stopped. why suspect in recent update/upgrade broke all.

anyone know how ix this?

thanks

added per request - my.cnf falcon:

# # mysql database server configuration file. # # can copy 1 of: # - "/etc/mysql/my.cnf" set global options, # - "~/.my.cnf" set user-specific options. # # 1 can use long options program supports. # run program --help list of available options , # --print-defaults see understand , use. # # explanations see # http://dev.mysql.com/doc/mysql/en/server-system-variables.html  # passed mysql clients # has been reported passwords should enclosed ticks/quotes # escpecially if contain "#" chars... # remember edit /etc/mysql/debian.cnf when changing socket location. [client] port            = xxxx socket          = /var/run/mysqld/mysqld.sock  # here entries specific programs # following values assume have @ least 32m ram  # formally known [safe_mysqld]. both versions parsed. [mysqld_safe] socket          = /var/run/mysqld/mysqld.sock nice            = 0  [mysqld] # # * basic settings # user            = mysql pid-file        = /var/run/mysqld/mysqld.pid socket          = /var/run/mysqld/mysqld.sock port            = xxxx basedir         = /usr datadir         = /var/lib/mysql tmpdir          = /tmp lc-messages-dir = /usr/share/mysql skip-external-locking # # instead of skip-networking default listen on # localhost more compatible , not less secure. bind-address            = xxx.xxx.xxx.xxx # # * fine tuning # key_buffer              = 16m max_allowed_packet      = 16m thread_stack            = 192k thread_cache_size       = 8 # replaces startup script , checks myisam tables if needed # first time touched myisam-recover         = backup #max_connections        = 100 #table_cache            = 64 #thread_concurrency     = 10 # # * query cache configuration # query_cache_limit       = 1m query_cache_size        = 16m # # * logging , replication # # both location gets rotated cronjob. # aware log type performance killer. # of 5.1 can enable log @ runtime! #general_log_file        = /var/log/mysql/mysql.log #general_log             = 1 # # error log - should few entries. # log_error = /var/log/mysql/error.log # # here can see queries long duration #log_slow_queries       = /var/log/mysql/mysql-slow.log #long_query_time = 2 #log-queries-not-using-indexes # # following can used easy replay backup logs or replication. # note: if setting replication slave, see readme.debian #       other settings may need change. #server-id              = 1 #log_bin                        = /var/log/mysql/mysql-bin.log expire_logs_days        = 10 max_binlog_size         = 100m #binlog_do_db           = include_database_name #binlog_ignore_db       = include_database_name  # replication per phpmyadmin - falcon master server-id=8503668 log-bin=mysql-bin log-error=mysql-bin.err  # # * innodb # # innodb enabled default 10mb datafile in /var/lib/mysql/. # read manual more innodb related options. there many! #  # read manual more innodb related options. there many! # # * security features # # read manual, too, if want chroot! # chroot = /var/lib/mysql/ # # generating ssl certificates recommend openssl gui "tinyca". # # ssl-ca=/etc/mysql/cacert.pem # ssl-cert=/etc/mysql/server-cert.pem # ssl-key=/etc/mysql/server-key.pem    [mysqldump] quick quote-names max_allowed_packet      = 16m  [mysql] #no-auto-rehash # faster start of mysql no tab completition  [isamchk] key_buffer              = 16m  # # * important: additional settings can override file! #   files must end '.cnf', otherwise they'll ignored. # !includedir /etc/mysql/conf.d/ 

also, added per request, lines my.cnf on quad set replication

# replication per phpmyadmin server-id=1369932901 report-host=quad replicate-ignore-db=visits 

the rest of my.cnf default except bind address quad's lan ip 192.168.1.70. my.cnf on hex same way server-id different, report-host hex , ignores visits db. and, quad, bind-address local lan ip of 192.168.1.60.

weekend update: question floated around 4 days without answer how fix broken replication once worked. i've posted my.cnf (or @ least relevant sections of it) no 1 has had suggestions on fix. of 4 days question has been around @ point 2 weekend days didn't expect much.

oddly enough last night did full slave start on both slaves 100th time , took. both slaves ran night without issue , reporting caught master (0 seconds behind). created junk db on master test in fact working. , showed in both slaves immediately.

i still don't have idea why stopped working in first place. suggested mysql replications need monitored frequently. there tool out there purpose? (the slaves ubuntu , code in php.)

• if slave i/o thread no longer running on server, check slave's error log messages may indicate why thread may have stopped.
• check last 100 lines of error log see if gives clue. problem wrong position given in change master to command. easiest fix set slave again if new slave. last error can determined last error number , last error
• check whether able ping/telnet master slave server.