Monitoring share-based replication on Sun Fishworks (7000) Appliances

We use Sun/Oracle's Fishworks (7000) ZFS Storage Appliances to store our Oracle archive logs and to replicate them to our DR datacenter.

We generate more than 2TB of archive logs per day, and ZFS' compression helps knock that down to a somewhat more manageable 500GB a day. Initially we were using project-based replication which was easy to configure, but unfortunately there was not enough parallelism to keep up with our change rate.

Sun suggested setting up replications for each share (we have 16 shares per database cluster) to improve throughput. It's worked well, but the user interface didn't provide an overview of replication status.

Fortunately, the CLI can be scripted using JavaScript, so it was easy to loop over the projects and shares and extract the replication status.

To run the script, just ssh to the appliance and redirect stdin from the script:

 ldap1{jwasilko}64: ssh sun7310-1 < replication_status  
 Pseudo-terminal will not be allocated because stdin is not a terminal.  
 Password:   
 Current time: Sun Jan 02 2011 02:35:06 GMT+0000 (UTC)  
 Share             LastSync                 LastTry                 NextTry                   
 db/archivelogs_rman10     Sun Jan 02 2011 02:25:13 GMT+0000 (UTC) Sun Jan 02 2011 02:25:13 GMT+0000 (UTC) Sun Jan 02 2011 02:55:00 GMT+0000 (UTC)  
 db/archivelogs_rman12     Sun Jan 02 2011 02:26:13 GMT+0000 (UTC) Sun Jan 02 2011 02:26:13 GMT+0000 (UTC) Sun Jan 02 2011 02:56:00 GMT+0000 (UTC)  
 db/archivelogs_rman14     Sun Jan 02 2011 02:27:13 GMT+0000 (UTC) Sun Jan 02 2011 02:27:13 GMT+0000 (UTC) Sun Jan 02 2011 02:57:00 GMT+0000 (UTC)  
 db/archivelogs_rman16     Sun Jan 02 2011 02:28:13 GMT+0000 (UTC) Sun Jan 02 2011 02:28:13 GMT+0000 (UTC) Sun Jan 02 2011 02:58:00 GMT+0000 (UTC)  
 db/archivelogs_rman2      Sun Jan 02 2011 02:21:21 GMT+0000 (UTC) Sun Jan 02 2011 02:21:21 GMT+0000 (UTC) Sun Jan 02 2011 02:51:00 GMT+0000 (UTC)  
 db/archivelogs_rman4      Sun Jan 02 2011 02:22:13 GMT+0000 (UTC) Sun Jan 02 2011 02:22:13 GMT+0000 (UTC) Sun Jan 02 2011 02:52:00 GMT+0000 (UTC)  
 db/archivelogs_rman6      Sun Jan 02 2011 02:33:18 GMT+0000 (UTC) Sun Jan 02 2011 02:33:18 GMT+0000 (UTC) Sun Jan 02 2011 03:03:00 GMT+0000 (UTC)  
 db/archivelogs_rman8      Sun Jan 02 2011 02:24:13 GMT+0000 (UTC) Sun Jan 02 2011 02:24:13 GMT+0000 (UTC) Sun Jan 02 2011 02:54:00 GMT+0000 (UTC)  
   
   

The script is below. I hope it might be useful for you.

 script  
   
 //  
 // jwasilko@gmail.com  
 // fishworks' user interface doesn't provide a good way to monitor  
 // the health of share-based replication. this is an attempt to make  
 // up for that.  
 //  
   
   
 print("Current time: " + new Date());  
 printf('%-30s %-40s %-40s %-40s\n', "Share", "LastSync", "LastTry", "NextTry");  
   
   
 // Get the list of projects, to iterate over later  
 run('shares');  
 projects = list();  
   
 // for each project, list the shares  
 for (projectNum = 0; projectNum < projects.length; projectNum++) {  
  run('select ' + projects[projectNum]);  
  shares = list();  
   
  // Walk into the share and select replication, then actions  
  for (sharesNum = 0; sharesNum < shares.length; sharesNum++) {  
   try { run('select ' + shares[sharesNum]) } catch (err) { dump(err); }  
   share = projects[projectNum] + '/' + shares[sharesNum];  
   run('replication');  
   actions = list();  
   
   // Some shares may not have share-specific replication actions,  
   // so skip if needed. Otherwise, get the replication status  
   if ( actions.length > 0 ) {  
    for (actionsNum = 0; actionsNum < actions.length; actionsNum++) {  
     try { run('select ' + actions[actionsNum]) } catch (err) { dump(err); }  
     lastsync = run('get last_sync').split(/=/)[1];  
     lastsync = lastsync.replace(/\n/,"");  
     lasttry = run('get last_try').split(/=/)[1];  
     lasttry = lasttry.replace(/\n/,"");  
     nextupdate = run('get next_update').split(/=/)[1];  
     nextupdate = nextupdate.replace(/\n/,"");  
     printf('%-30s %-40s %-40s %-40s\n', share, lastsync, lasttry, nextupdate);  
    }  
   run('cd ../..');  
   }  
   else {  
    run('cd ..');  
   }  
   run('cd ..');  
   }  
 run('cd ..');  
 }