Monday, March 7, 2011

Celerra top talkers & suspicious ops defined

The EMC Celerra datamovers have the ability to log statistics about top talkers, which can be useful for tracking down problems. We run server_stats with these options to get top talker stats:

/nas/bin/server_stats server_2 -top nfs -i 5 -c 60

One thing worth noting is there's a column labeled "NFS Suspicious Ops". There's no documentation on this column, and it took EMC some time to dig up the answer. Here it is:

SUSPICIOUS EVENTS:
One of the TopTalker output columns lists Suspicious Ops/second.
"Suspicious" events are any of the following, which are typical of the patterns seen when viruses or other badly behaved software/users are attacking a system:

CIFS events:
  • ACCESS_DENIED returned for FindFirst
  • ACCESS_DENIED returned for Open/CreateFile
  • ACCESS_DENIED returned for DeleteFile
  • SUCCESS returned for DeleteFile
  • SUCCESS returned for TruncateFile (size=0)

NFSv2/v3/v4 events:
  • NFSERR_ACCES returned for NFS OPEN/LOOKUP/CREATE/DELETE
  • NFSERR_ACCES returned for READDIR/READDIRPLUS
  • NFS_OK for NFS REMOVE
  • NFS_OK for NFS SETATTR (size=0)

3 comments:

  1. Interesting, Jeff. I cannot find much information on the 'top talkers' feature for the Celerra. Do you know of any relevant documentation on PowerLink and which version of DART this feature was introduced?

    ReplyDelete
  2. I think it was in 5.6 first.

    In 6.0 it has a slightly different invocation (and more options).

    ReplyDelete
  3. Hi Jeff,

    I see you wrote a very interesting article on https://community.emc.com/message/574947#574947 where you took the output of server_stats and used this in graphite. Would you be so kind as to explain the process you took to get the data into graphite?

    Regards,
    Jason

    ReplyDelete