VMWare ESXi 5.5 NFS Disconnect Issues -- Notification on general vsphere events

Ever since we've been running vsphere 5.5 we have been experiencing NFS Datastore disconnects. We have delved into every facet of storage, networking, and even advanced configuration of our vsphere hosts.

So far little information is known about the root cause. VMWare finally put out a KB about the issue. The only problem with this guy is A. we're not running 5.5 update 1 and B. we're not running 5.5 update 1.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2004684

So for all the hardworking techies out there I provide at least a base consolation for you. Should you be living this dream, then you can use the following and save it as a .ps1 script then create a task that will kick it off every 5 minutes. This way you'll at least know where the pain is coming from without having to spend precious time finding the broken host(s). We've found the only option to recover is to power off the VM Guest, which is literally the only command you'll be able to effectively issue it, then power it on, which will move it to a working host. To simplify it further you can put the host into maintenance mode. By doing this you'll know of any VM Guests that are in a locked state.

##### Start Script #####

#Generated by JH

####################FUNCTIONS####################
Function sendMail($body){
set-content \\pathtoalogfile\nfs_logs_$(get-date -format M.d.yyyy).txt $body
Write-Host "Sending Email"
$smtpserver = "yourmailserver"
$recipient = "yourdistributionlist"
$sender = "yoursourceemailaccount"
$subject = "ESX NFS Issue Report"
send-MailMessage -SmtpServer $smtpserver -To $recipient -From $sender -Subject $subject -Body $body -Priority high
}

Function processoutput($bodyfile){
$body = [IO.File]::ReadAllText($bodyfile)
$endcount = $log.count + 1
$count = 0
Do {
$log[$count] | % {
If ($psitem.createdtime -ne $null){
If (!($body -imatch "$($psitem.createdtime) -- $($psitem.objectname)")){
add-Content $bodyfile @"

$($psitem.createdtime) -- $($psitem.objectname) -- $($psitem.fullformattedmessage)

"@
}
Else{
$body = $body.Trim()
set-Content $bodyfile $body
}
}
}
$count++
}
Until ([int]$count -eq [int]$endcount)
}

#################################################

$start = (get-date).adddays(-1).ToString('M/d/yyyy')
#$start = get-date -format M/d/yyyy
#$start = "1/1/2014"
$end = (get-date).adddays(+1).ToString('M/d/yyyy')
$start2 = get-date -format M.d.yyyy
set-content \\pathtoalogfile\nfs_logs_$start2.txt "Starting Run @ $(get-date)"
Add-PSSnapin VMware.VimAutomation.Core
Set-PowerCLIConfiguration -DefaultVIServerMode Multiple -Confirm:$false
cls
connect-viserver yourhostorvcenternumber1 -User youruseraccount -Password yourpassword
connect-viserver yourhostorvcenternumber2 -User youruseraccount -Password yourpassword

$pattern = "esx.problem.vmfs.nfs.server.disconnect|esx.problem.storage.apd.timeout"
$hostlist = (get-vmhost | sort).name

cls
$log = $null
$hostlist | % {
$psitem
$log += ,@(get-vmhost $psitem | get-vievent -start $start -Finish $end | ? { $psitem.EventTypeID -imatch ".*($pattern).*" })
}
$body1 = [IO.File]::ReadAllText('C:\vievents.txt').Trim()
processoutput C:\vievents.txt
$body2 = [IO.File]::ReadAllText('C:\vievents.txt').Trim()
disconnect-viserver * -Confirm:$false

If ($body1 -ne $body2){ sendmail $body2 }
Else { add-content \\pathtoalogfile\nfs_logs_$(get-date -format M.d.yyyy).txt "No NEW Errors as of $(get-date)" }

An Automated, Slam Poetry rendition of "Yellow Ledbetter"

Save this as with a .ps1 file extension.

While I appreciate the vocal and instrumental talent, the song makes even less sense, but is way more funny this way.

$voice = New-Object -ComObject SAPI.SPVoice
$voice.Rate = -3
function invoke-speech{
param([Parameter(ValueFromPipeline=$true)][string] $say )
process{
$voice.Speak($say) | out-null;
} }

$speech = @"
Unsealed on a porch a letter sat.
Then you said, I wanna leave it again.
Once I saw her on a beach of weathered sand.
And on the sand I wanna leave it again.
Yeah. On a weekend I wanna wish it all away, yeah.
And they called and I said that I want what I said and then I call out again.
And the reason oughta leave her calm, I know.
I said I know what I was the boxer or the bag.
Ah yeah, can you see them out on the porch?
Yeah, but they dont wave. I see them round the front way.
Yeah.
And I know, and I know I dont want to stay.
Make me cry...
I see...
Oh I dont know why theres something else.
I wanna drum it all away...
Oh, I said, I dont, I dont know whether I was the boxer or the bag.
Ah yeah, can you see them out on the porch?
Yeah, but they dont wave.
But I see them round the front way.
Yeah. And I know, and I know.
I dont wanna stay at all.
I dont wanna stay.
Yeah.
I dont wanna stay.
I dont...
Dont wanna, oh...
Yeah.
Ooh...
Ohh...
"@

$speech | invoke-speech

Use Powershell to Standardize NTFS and SMB (or share) ACLs

How many times have you wondered about whether your share permissions were standard and uniform between your file servers?

Fear not!  Now provided you are utilizing the Windows Server 2012 R2 DFSR module and DFS to manage your file shares you can run this little guy.  It will prompt the operator for the source namespace, share, and file server, then apply the ACL settings for the specified information to any partner servers within the specified namespace share.

Enjoy!

#DATA RETRIEVAL INPUT PROMPT SECTION

#Titles
$headers = @{title0 = "Run Again?";title1 = "Select DFS Namespace";title2 = "Select DFS Share";title3 = "Select Permissions Source";message0 = "Yes or No?";message1 = "Select the DFS Namespace";message2 = "Select the DFS Share";message3 = "Select the permissions source"}

#FUNCTION SECTIONS

#.NET Menu Function
Function dataselection($title,$message,$inputoption){
    [void] [System.Reflection.Assembly]::LoadWithPartialName("System.Windows.Forms")
    [void] [System.Reflection.Assembly]::LoadWithPartialName("System.Drawing")

    $objForm = New-Object System.Windows.Forms.Form
    $objForm.Text = $title
    $objForm.Size = New-Object System.Drawing.Size(335,400)
    $objForm.StartPosition = "CenterScreen"

    $objForm.KeyPreview = $True
    $objForm.Add_KeyDown({if ($_.KeyCode -eq "Enter")
        {$global:x=$objListBox.SelectedItem;$objForm.Close()}})
    $objForm.Add_KeyDown({if ($_.KeyCode -eq "Escape")
        {$objForm.Close()}})

    $OKButton = New-Object System.Windows.Forms.Button
    $OKButton.Location = New-Object System.Drawing.Size(75,335)
    $OKButton.Size = New-Object System.Drawing.Size(75,23)
    $OKButton.Text = "OK"
    #$OKButton.Add_Click({$global:$x=$objListBox.SelectedItem;$objForm.Close()})
    $OKButton.Add_Click({$global:x=$objListBox.SelectedItem;$objForm.Close()})
    $objForm.Controls.Add($OKButton)

    $CancelButton = New-Object System.Windows.Forms.Button
    $CancelButton.Location = New-Object System.Drawing.Size(150,335)
    $CancelButton.Size = New-Object System.Drawing.Size(75,23)
    $CancelButton.Text = "Cancel"
    $CancelButton.Add_Click({$objForm.Close()})
    $objForm.Controls.Add($CancelButton)

    $objLabel = New-Object System.Windows.Forms.Label
    $objLabel.Location = New-Object System.Drawing.Size(10,10)
    $objLabel.Size = New-Object System.Drawing.Size(280,20)
    $objLabel.Text = $message
    $objForm.Controls.Add($objLabel)

    $objListBox = New-Object System.Windows.Forms.ListBox
    $objListBox.Location = New-Object System.Drawing.Size(10,40)
    $objListBox.Size = New-Object System.Drawing.Size(300,300)
    $objListBox.Height = 300
    ForEach ($option in $inputoption){
        [void] $objListBox.Items.Add($option)
    }
   
    $objForm.Controls.Add($objListBox)

    $objForm.Topmost = $True

    $objForm.Add_Shown({$objForm.Activate()})
    [void] $objForm.ShowDialog()

    return $x
}

Function printresults($results){
switch ($($Results)){
        0 {$result = 'Success';break}
2 {$result = 'Access Denied';break}
8 {$result = 'Unknown Failure';break}
9 {$result = 'Invalid Name';break}
10 {$result = 'Invalid Level';break}
21 {$result = 'Invalid Parameter';break}
22 {$result = 'Duplicate Object';break}
23 {$result = 'Redirected Path';break}
24 {$result = 'Unknown Device or Directory';break}
25 {$result = 'Network Name Not Found';break}
default {$result = '*** Unknown Error ***';break}
}
    return $result
}

Function dataaction(){
    $actionobject = @{}
#Namespace Groups
$actionobject["namespace"] = @((get-dfsnroot).path)
$actionobject["activenamespace"] = dataselection $headers.title1 $headers.message1 $actionobject.namespace

#Namspace Share
$actionobject["sharename"] = @((dir $actionobject.activenamespace | select Name).Name)
$actionobject["activesharename"] = dataselection $headers.title2 $headers.message2 $actionobject.sharename

#Local Share Source
$actionobject["dfssharepath"] = $actionobject.activenamespace + '\' + $actionobject.activesharename
$actionobject["sourcename"] += @((get-dfsnfolderTarget -path $actionobject.dfssharepath).TargetPath)
$actionobject["activesourcename"] = dataselection $headers.title3 $headers.message3 $actionobject.sourcename
$actionobject["sourceserver"] = $actionobject.activesourcename.substring(2)
$actionobject["sourceserver"] = $actionobject.sourceserver.substring(0,$actionobject.sourceserver.indexof('\'))

#AUTOMATED DATA RETRIEVAL SECTIONS

#Local Share Target
$actionobject["dfsgroupname"] = $actionobject.dfssharepath.Substring(2)
$actionobject["targetservers"] = @((Get-DfsrConnection -g $actionobject.dfsgroupname -SourceComputerName $actionobject.sourceserver).DestinationComputerName)

#Get Local Share Paths
$actionobject["sourcefolder"] = ("\\$($actionobject.sourceserver)\" + (get-dfsrmembership -g $actionobject.dfsgroupname -computername $actionobject.sourceserver).contentpath) -replace ':','$'
$actionobject["targetfolder"] = @()
ForEach ($targetserver in $actionobject.targetservers){
$actionobject["targetfolder"] += ("\\$targetserver\" + (get-dfsrmembership -g $actionobject.dfsgroupname -computername $targetserver).contentpath) -replace ':','$'
}

    #AUTOMATED ACTION SECTIONS

#Set NTFS Permissions
    $aclobject = @{}
$aclobject["sourcentfsacl"] = Get-Acl $actionobject.sourcefolder
ForEach ($ntfstarget in $actionobject.targetfolder){
        Set-Acl $ntfstarget $aclobject.sourcentfsacl
        If ($? -eq $True){$aclobject["ntfsresult"] = "Success"}
        ElseIf ($? -eq $False){$aclobject["ntfsresult"] = "Unspecified Failure"}
        Write-Host "Object:  $ntfstarget, Location:  $ntfstarget, Action:  Set NTFS ACL, Outcome:  $($aclobject.ntfsresult)"
       
        #Catalogue NTFS ACL For Logging
 
        $aclobject["ntfscatalogue"] = @()
   ForEach ($ntfsaclentry in $aclobject.sourcentfsacl.Access){
            $trustee = $ntfsaclentry.IdentityReference
            $effectivepermissions = $ntfsaclentry.FileSystemRights
   $aclobject.ntfscatalogue += ,($trustee," - $effectivepermissions"," - $($aclobject.ntfsresult)")
            $trustee = $null
            $effectivepermissions = $null
   }
    }

#Get Source SMB Permissions
$aclobject["sourcesmbacl"] = Get-WMIObject win32_LogicalShareSecuritySetting -comp $actionobject.sourceserver | ?{($_.name -imatch $actionobject.activesharename)}
$aclobject["sourcesmbacl"] = $aclobject.sourcesmbacl.GetSecurityDescriptor().Descriptor.DACL
$aclobject["Class"] = 'Win32_Share'

    #Set SMB Permissions
ForEach ($smbtarget in $actionobject.targetservers){
        $aclobject["securitydescriptor"] = ([WMIClass] "\\$smbtarget\root\cimv2:Win32_SecurityDescriptor").CreateInstance()
        #Create a server specific ACL built off the source ACL
        ForEach ($acl in $aclobject.sourcesmbacl){
            $trustee = ([WMIClass] "\\$smbtarget\root\cimv2:Win32_Trustee").CreateInstance()
            $trustee.Name = $acl.Trustee.Name
            $trustee.Domain = $acl.Trustee.Domain
            $ace = ([WMIClass] "\\$smbtarget\root\cimv2:Win32_ACE").CreateInstance()
            $ace.AccessMask = $acl.AccessMask
            $ace.AceFlags = $acl.AceFlags
            $ace.AceType = $acl.AceType
            $ace.Trustee = $Trustee
            $aclobject.securitydescriptor.DACL += $ace.psObject.baseobject
        }
#The path variable is used strictly for posting a result of the SMB ACL application to the console window
        $aclobject["path"] = ("\\$smbtarget\" + (get-dfsrmembership -g $actionobject.dfsgroupname -computername $smbtarget).contentpath) -replace ':','$'
$aclobject["share"] = [Wmi]"\\$smbtarget\ROOT\CIMV2:$($aclobject.Class).Name='$($actionobject.activesharename)'"
#In this command the share name is passed in as the description as well.
$results = $aclobject.share.SetShareInfo([int32]::MaxValue,$actionobject.activesharename,$aclobject.securitydescriptor)
        $aclobject["smbresult"] = printresults $results.returnvalue
        Write-Host "Object:  $($actionobject.activesharename), Location:  $($aclobject.path), Action:  Set SMB ACL, Outcome:  $($aclobject.smbresult)"
 
        #Catalogue SMB ACL for Logging
 
        $aclobject["smbcatalogue"] = @()
   ForEach ($smbaclentry in $aclobject.sourcesmbacl){
   $trusteename = $smbaclentry.Trustee.Name
   $trusteedomain = $smbaclentry.Trustee.Domain
   $trustee = "$trusteedomain\$trusteename"
   Switch ($smbaclentry.AccessMask){
   2032127 {$effectivepermissions = "FullControl"}
   1179785 {$effectivepermissions = "Read"}
   1180063 {$effectivepermissions = "Read, Write"}
   1179817 {$effectivepermissions = "Read and Execute"}
   -1610612736 {$effectivepermissions = "Read and Execute Extended"}
   1245631 {$effectivepermissions = "Read and Execute, Modify, Write"}
   1180095 {$effectivepermissions = "Read and Execute, Write"}
   268435456 {$effectivepermissions = "FullControl (Sub Only)"}
   default {$effectivepermissions = $sharesecurity.AccessMask}
   }
   #Translates the access type to plain text
   Switch ($smbaclentry.AceType) {
   0 {$effectiveaccess = "Allow"}
   1 {$effectiveaccess = "Deny"}
   2 {$effectiveaccess = "Audit"}
   }
   $aclobject.smbcatalogue += ,($trustee," - $effectiveaccess"," - $effectivepermissions"," - $($aclobject.smbresult)")
   $trustee = $null
   $trusteename = $null
   $trusteedomain = $null
            $effectiveaccess = $null
            $effectivepermissions = $null
   }
}

#LOGGING SECTIONS

#Write to Log File
    $date = Get-Date
$logfile = '\\somesharepath\DFS_Share_and_NTFS_Permissions_change.log'
Add-Content $logfile ('#' * $actionobject.activenamespace.Length * 5)
Add-Content $logfile $date
Add-Content $logfile "Initiated By:  $env:USERNAME"
    Add-Content $logfile "Script Name:  $MyInvocation.MyCommand.Name"
Add-Content $logfile "Namespace:  $($actionobject.activenamespace)"
Add-Content $logfile "Namespace Share:  $($actionobject.activesharename)"
Add-Content $logfile "Permissions Source:  $($actionobject.sourcefolder)"
ForEach ($ntfstarget in $actionobject.targetfolder){Add-Content $logfile "Permissions Target:  $ntfstarget"}
Add-Content $logfile ('-' * $($actionobject.activenamespace.Length) * 1.5)+"NTFS PERMISSIONS APPLIED"+('-' * $($actionobject.activenamespace.Length) * 1.5)
ForEach ($ntfsaclentry in $aclobject.ntfscatalogue){Add-Content $logfile "Permission:  $ntfsaclentry"}
Add-Content $logfile ('-' * $($actionobject.activenamespace.Length) * 1.5)+"SMB PERMISSIONS APPLIED"+('-' * $($actionobject.activenamespace.Length) * 1.5) + '-'
ForEach ($smbaclentry in $aclobject.smbcatalogue){Add-Content $logfile "Permission:  $smbaclentry"}
Add-Content $logfile ('#' * $actionobject.activenamespace.Length * 5)

#PROMPT TO PROCESS ANOTHER OBJECT
$yes = New-Object System.Management.Automation.Host.ChoiceDescription "&Yes"
$no = New-Object System.Management.Automation.Host.ChoiceDescription "&No"
$options0 = [System.Management.Automation.Host.ChoiceDescription[]]($yes, $no)
$result = $host.ui.PromptForChoice($headers.title0, $headers.message0, $options0, 0)
If ($result -eq 0){dataaction}
}

dataaction

Use vbscript to edit local security policy

Recently I found the need to automate the addition of a user to the "logon as a service" local security policy of a windows system.  I found a lot of post about "how do I" and no definitive (start TO finish) summations to accomplish such.  So here it is:


Const ForReading = 1
Const ForAppending = 8
Const ForWriting = 2
strComputer = "."
Set objWMIService = GetObject("winmgmts:\\" & strComputer & "\root\cimv2")
Set objFSO = CreateObject("Scripting.FileSystemObject")
Set objShell = CreateObject("WScript.Shell")

'Grant logonasaservice right to the service user
strDomUser = "yourdomainuser"
strDom = "yourdomain"
'identifies the AD SID for the specified user
Set objAccount = objWMIService.Get("Win32_UserAccount.Name='" & strDomUser & "',Domain='" & strDom & "'")
'exports the user rights local policy
strComSec1 = "secedit /export /areas user_rights /cfg C:\temppath\~security.inf"
objShell.Run strComSec1, 0, True
'opens the unicode output to be parsed
Set strInpFile = objFSO.OpenTextFile("C:\temppath\~security.inf", ForReading, False, -1)
strInput = strInpFile.ReadAll
strInpFile.Close
objFSO.DeleteFile "D:\Scripts\Tcat\Service_Install\~security.inf"
'looks for the user SID - If the user had previously been given specific local policy rights you will have to refine this further.
If InStr(strInput, objAccount.SID) <= 0 Then
'build replacedment policy information - this is specific to "logon as a service" permissions - note the two values, one is a default nt services value, the other is the new SID
strData = "[Unicode]" & vbCrLf & "Unicode=yes" & vbCrLf & "[Version]" & vbCrLf & "signature=""$CHICAGO$""" & vbCrLf & "Revision=1" & vbCrLf & "[Privilege Rights]" & vbCrLf & "SeServiceLogonRight = *S-1-5-80-0,*" & objAccount.SID
Set strInpFile = objFSO.OpenTextFile("C:\temppath\~SecurityTemplate.inf", ForWriting, True, -1)
strInpFile.Write strData
strInpFile.Close
'edit the policy
strComSec2 = "secedit /configure /db C:\Windows\security\database\secedit.sdb /cfg ""C:\temppath\~SecurityTemplate.inf"" /areas user_rights /log seclog.loc"
objShell.Run strComSec2, 0, True
objFSO.DeleteFile "D:\Scripts\Tcat\Service_Install\~SecurityTemplate.inf"
End If

How to Alter Text Based Files en Mass

So I've spent the better part of a week trying to figure out how best to synchronize a production IT application with its Disaster Recovery counter part while editing the necessary configuration files in an automated fashion. The result is a pretty, dare I say it? BITCHIN set of scripts that will ease my administrative headaches by ten fold.

There are two methods that I am using. I call both methods from a batch script, but you could execute them straight from command line. I have used this to alter the following file types: .bat, .properties, and .xml. Essentially if it is a text based file that can be read with a text editor, this process can do the job.

Method 1: This method specifies the text file to be edited, while looping through it and replacing a specific text string with an alternate text string (which are both input as a variable).

Method 2: This method dumps a list of text based files into a text file, then reads that text file while combining the input from the text file with a file path location where a large number of text files reside. This then loops through the resultant text files and replacing a specific text string with an alternate text string (which are both input as a variable).

So here it is...

Method 1:

The command would be:

wscript.exe C:\replacespecific.vbs "pathoffiletoedit" "texttobereplaced" "newtextstring"

Simply paste the below content to the .vbs filename and location of your choice.

Const ForReading = 1

Const ForWriting = 2

strFilePath = Wscript.Arguments(0)

strSearchText = Wscript.Arguments(1)

strReplaceText = Wscript.Arguments(2)

Set objFSO = CreateObject("Scripting.FileSystemObject")

Set objSearchFile = objFSO.GetFile(strFilePath)

If objSearchFile.Size > 0 Then

Set objFile = objFSO.OpenTextFile(strFilePath, ForReading)

strText = objFile.ReadAll

objFile.Close

strNewText = Replace(strText, strSearchText, strReplaceText)

Set objFile = objFSO.OpenTextFile(strFilePath, ForWriting)

objFile.Write strNewText

objFile.Close

End If

Method 2:

The command sequence would be:

dir /B C:\Test\DirContent > C:\Test\list.txt

wscript.exe C:\replace.vbs C:\Test\list.txt "C:\Test\DirContent\" "texttobereplaced" "newtextstring"

or

dir /B "directorywheretextfileslive" > "pathtolistoftextfiles.txt"

wscript.exe C:\replace.vbs "pathtolistoftextfiles.txt" "directorywheretextfileslive/" "texttobereplaced" "newtextstring"

*Note: the "directorywheretextfileslive" entry must have a trailing backslash or the process will fail.

Simply paste the below content to the .vbs filename and location of your choice.

Const ForReading = 1

Const ForWriting = 2

strFileListFileName = Wscript.Arguments(0)

strFilePath = Wscript.Arguments(1)

strSearchText = Wscript.Arguments(2)

strReplaceText = Wscript.Arguments(3)

Set objFSO = CreateObject("Scripting.FileSystemObject")

Set objTextFile = objFSO.OpenTextFile(strFileListFileName, ForReading)

Do Until objTextFile.AtEndOfStream

fileToUpdate = objTextFile.Readline

Set objFSO2 = CreateObject("Scripting.FileSystemObject")

Set objSearchFile = objFSO.GetFile((strFilePath + fileToUpdate))

If objSearchFile.Size > 0 Then

Set objFile = objFSO2.OpenTextFile((strFilePath + fileToUpdate), ForReading)

strText = objFile.ReadAll

objFile.Close

strNewText = Replace(strText, strSearchText, strReplaceText)

Set objFile = objFSO2.OpenTextFile((strFilePath + fileToUpdate), ForWriting)

objFile.Write strNewText

objFile.Close

End If

Loop

Conclusion:

I have both tasks executing nightly where robocopy.exe pulls the files from production servers, edits the files based on the required parameters that I need and then saves them in the appropriate place. The batch script that I call these from actually cycles through the files several times each time replacing a different string of text...and it works BEAUTIFULLY.

So as I've had great success with this, I hope that you will likewise have great success with it for whatever your purpose may be.