Active Directory

What do we say to health checking Active Directory?

What do we say to health checking Active Directory?What do we say to health checking Active Directory?

Setting up a new Active Directory is an easy task. You download and install Windows Server, install required roles and in 4 hours or less have a basic Active Directory setup. In an ideal world that would be all and your only task would be to manage users, computers, and groups occasionally creating some Group Policies. Unfortunately, things with Active Directory aren't as easy as I've pictured it. Active Directory is a whole ecosystem and works well ranging from small companies with ten users to 500k users or more (haven't seen one myself – but so they say!). When you scale Active Directory adding more servers, more domains things tend to get complicated, and while things on top may look like they work correctly, in practice, they may not. That's why, as an Administrator, you need to manage Active Directory in terms of its Health and Security. Seems easy right? Not quite. While you may think you have done everything, checked everything, there's always something missing. Unless you have instructions for everything and can guarantee that things stay the same way as you left them forever, it's a bit more complicated. That's why Microsoft delivers you tools to the troubleshoot your Active Directory, such as dcdiag, repadmin and some others. They also sell monitoring solutions such as Microsoft SCOM which can help and detect when some things happen in your AD while you were gone. Surely there are some 3rd party companies give you some tools that can help with a lot of that as well. Finally, there is lo of folks within the community creating PowerShell scripts or functions that help with some Health Checks of your Active Directory.

Active Directory Health

For years I've been copying/pasting some stuff from different blogs trying to find what I've misconfigured or what I am missing. Alternatively, I used now discontinued Microsoft Active Directory Replication Status Tool that was excellent looking way telling me what doesn't work when it comes to Active Directory Replication. Do you remember it?

Some months ago, I saw on Twitter how Bill Kindle wrote Pester based Active Directory test. If you don't know Pester, it's a test framework for PowerShell scripts. He used it in the non-conventional way of testing infrastructure with it. It was a bit eye-opener for me even thou I know some people have done similar things before. At that time, I took his code, done some improvements, few small adjustments, and released to PowerShell Gallery under PesterInfrastructureTests name. This gave me the ability to use it freely without the need to copy/paste any code. I was using it for confirming that my or my Clients Active Directory works correctly. It covered some basic tests, but it was handy. It made me forget about the AD Replication Status Tool, and I could quickly check if everything is running correctly after Windows Updates. Let's see how PesterInfrastructureTests looks like and what it does, shall we? All we have to do is install it using Install-Module which downloads and install PesterInfrastructureTests. After that, we run a single command that targets Active Directory Forest scanning all Domains and all Domain Controllers doing some basics tests.

# First Install Module if you don't have one
# Install-Module PesterInfrastructureTests -Force -AllowClobber
Test-ADPester

What I love about it, is that with one command Test-ADPester it delivered me the status of my Active Directory. Not a lot of it, but enough to tell whether my Active Directory is healthy or not, with virtually zero effort on my side.

What I dislike about it is that errors make it hard to read. By default, PowerShell tends to throw the errors in a way that makes me uncomfortable. Don't get me wrong, the errors are useful to debug, but I don't want my screen to be red when I do my infrastructure tests.

See what I mean? So much red, some yellow warnings and some white ones. Oh well, when I see red, that means it's terrible, and I should jump in to fix things. I could probably do some try/catch, prettify it a bit and enjoy its functionality for more years to come. Unfortunately for me, a few months ago on a trip to PSConfEU (PowerShell Conference Europe), I've spent 8-hour driving with one of my colleagues Mateusz Czerniawski talking about Health Checks for Active Directory, and how cool would it be to have something similar to what DBAChecks offers to Microsoft SQL Community. Cool idea, right? He then followed thru by releasing Active Directory Health Check List on Github – a curated list of health checks that someone can take and build upon. It was just a list, but something one can rely on to develop their stuff.

Useful AD Health and Security Materials

Me, being me, and with my tendency to reinvent the wheel, I thought that I need something more then Pester checks and that my PesterInfrastructureTests is not going to cut it. I've decided to do a lot of research trying to find what others have done in the area and while there are lots of materials and approaches all require manual steps to start, or check an only small part of Active Directory. Following is a list of some things I found useful and things that gave me ideas to build something of my own.

And that's just a list of resources that cover at least 1 test. I've gone thru plenty of blog posts, websites, scripts, GitHub trying to asses what people think Health Check means for Active Directory. Arnaud Loos list is by far my most favorite, and the most comprehensive list I found. It's a gold mine!

Testimo - Basic features

Now all I have to do is reinvent the wheel and build something of my own. But what should it be? How should it work? Well, after a few weeks of working on it, changing how things are done, how things are working I've decided that it should at least cover these goals:

  • It should allow for testing Forest
  • It should allow for testing all domains within a forest
  • It should allow for testing all domain controllers in all domains
  • It should be easily extensible in adding more tests
  • It should allow changing settings on a global way (disable/enable test)
  • It should allow changing each test separately (think password policy differences between different companies)
  • It should allow excluding Domains and Domain Controllers from tests
  • It should allow easy installation
  • It should allow easy use, intuitive use out of the box
  • It should have tests based on settings rather then a snapshot/baseline comparison only
  • It should deliver Pass/Fail per test
  • It should allow for reporting with extended information and email status if required

Don't get me wrong, I did have those plans as above, but I've changed my mind probably 50 times while working on a solution. Today is just a day I am comfortable enough to share it with you and ask for feedback.

Testimo is PowerShell Module that I wrote to help me asses Active Directory Forests that I manage frequently or in-frequently. I often consult for MSP companies that manage multiple Clients, each with different Active Directory. Alternatively, I get ad-hoc requests for help, and so I need something that can help me quickly verify where the problem may be, without knowing how someone built their environment. It needs to be flexible enough to tell me where a possible problem may be without spending much time trying to find that problem. Usually, when I get such request you need to go thru lots of different areas by hand, and since I'm pretty lazy my happiness level goes down, and we don't want that right? That's why I've created Testimo, and I hope you will enjoy it as much as I do. It's by no means module that covers everything, but I hope it will be able to help me do my job better and faster. Let's see if it can help you? But before I jump into it, I wanted to add that Testimo is joining my two other AD-related modules that you may find useful.

Keep in mind that while the blog posts cover some functionality, I do work on those modules from time to time adding new features. It's best to track GitHub if you want to know what changes or what's added. I don't do blogs for every release.

So what Testimo can do? I've defined set of different tests with a very short overview what is the expectation from a test

  • Forest Backup – Verify last backup time should be less than X days
  • Forest Replication – Verify each DC in replication site can reach other replication members
  • Forest Optional Features – Verify Optional Feature Recycle Bin should be Enabled
  • Forest Optional Features- Verify Optional Feature Privileged Access Management Feature should be Enabled
  • Forest Optional Features – Verify Optional Feature Laps should be enabled Configured
  • Forest Sites Verification Verify each site has at least one subnet configured
  • Forest Sites Verification Verify each site has at least one domain controller configured
  • Forest Site Links – Verify each site link is automatic
  • Forest Site Links – Verify each site link uses notifications
  • Forest Site Links- Verify each site link does not use notifications
  • Forest Roles Verify each FSMO holder is reachable
  • Forest Orphaned/Empty Admins – Verify there are no Orphaned Admins (users/groups/computers)
  • Forest Tombstone Lifetime – Verify Tombstone lifetime is greater or equal 180 days
  • Domain Roles Verify each FSMO holder is reachable
  • Domain Password Complexity Requirements – Verify Password Complexity Policy should be Enabled
  • Domain Password Complexity Requirements – Verify Password Length should be greater than X
  • Domain Password Complexity Requirements – Verify Password Threshold should be greater than X
  • Domain Password Complexity Requirements – Verify Password Lockout Duration should be greater than X minutes
  • Domain Password Complexity Requirements – Verify Password Lockout Observation Window should be greater than X minutes
  • Domain Password Complexity Requirements – Verify Password Minimum Age should be greater than X
  • Domain Password Complexity Requirements – Verify Password History Count should be greater than X
  • Domain Password Complexity Requirements – Verify Password Reversible Encryption should be Disabled
  • Domain Trust Availability – Verify each Trust status is OK
  • Domain Trust Unconstrained TGTDelegation – Verify each Trust TGTDelegation is set to True
  • Domain Kerberos Account Age – Verify Kerberos Last Password Change Should be less than 180 days
  • Domain Groups: Account Operators – Verify Group is empty
  • Domain Groups: Schema Admins – Verify Group is empty
  • Domain User: Administrator – Verify Last Password Change should be less than 360 days or account disabled
  • Domain DNS Forwarders – Verify DNS Forwarders are identical on all DNS nodes
  • Domain DNS Scavenging Primary DNS Server – Verify DNS Scavenging is set to X days
  • Domain DNS Scavenging Primary DNS Server – Verify DNS Scavenging State is set to True
  • Domain DNS Scavenging Primary DNS Server – Verify DNS Scavenging Time is less than X days
  • Domain DNS Zone Aging – Verify DNS Zone Aging is set
  • Domain Well known folder – UsersContainer  Verify folder is not at it's defaults.
  • Domain Well known folder – ComputersContainer  Verify folder is not at it's defaults.
  • Domain Well known folder – DomainControllersContainer Verify folder is at it's defaults.
  • Domain Well known folder – DeletedObjectsContainer Verify folder is at it's defaults.
  • Domain Well known folder – SystemsContainer Verify folder is at it's defaults.
  • Domain Well known folder – LostAndFoundContainer Verify folder is at it's defaults.
  • Domain Well known folder – QuotasContainer Verify folder is at it's defaults.
  • Domain Well known folder – ForeignSecurityPrincipalsContainer Verify folder is at it's defaults.
  • Domain Orphaned Foreign Security Principals – Verify there are no orphaned FSP objects.
  • Domain Orphaned/Empty Organizational Units – Verify there are no orphaned Organizational Units
  • Domain Group Policy Missing Permissions – Verify Authenticated Users/Domain Computers are on each and every Group Policy
  • Domain DFSR Sysvol – Verify SYSVOL is DFSR
  • Domain Controller Information – Is Enabled
  • Domain Controller Information – Is Global Catalog
  • Domain Controller Service Status – Verify all Services are running
  • Domain Controller Service Status – Verify all Services are set to automatic startup
  • Domain Controller Service Status (Print Spooler) – Verify Print Spooler Service is set to disabled
  • Domain Controller Service Status (Print Spooler) – Verify Print Spooler Service is stopped
  • Domain Controller Ping Connectivity – Verify DC is reachable
  • Domain Controller Ports – Verify Following ports 53, 88, 135, 139, 389, 445, 464, 636, 3268, 3269, 9389 are open
  • Domain Controller RDP Ports – Verify Following ports 3389 (RDP) is open
  • Domain Controller RDP Security – Verify NLA is enabled
  • Domain Controller LDAP Connectivity – Verify all LDAP Ports are open
  • Domain Controller LDAP Connectivity – Verify all LDAP SSL Ports are open
  • Domain Controller Windows Firewall – Verify windows firewall is enabled for all network cards
  • Domain Controller Windows Remote Management – Verify Windows Remote Management identification requests are managed
  • Domain Controller Resolves internal DNS queries – Verify DNS on DC resolves Internal DNS
  • Domain Controller Resolves external DNS queries – Verify DNS on DC resolves External DNS
  • Domain Controller Name servers for primary domain zone Verify DNS Name servers for primary zone are identical
  • Domain Controller Responds to PowerShell Queries Verify DC responds to PowerShell queries
  • Domain Controller TimeSettings – Verify PDC should sync time to external source
  • Domain Controller TimeSettings – Verify Non-PDC should sync time to PDC emulator
  • Domain Controller TimeSettings – Verify Virtualized DCs should sync to hypervisor during boot time only
  • Domain Controller Time Synchronization Internal – Verify Time Synchronization Difference to PDC less than X seconds
  • Domain Controller Time Synchronization External – Verify Time Synchronization Difference to pool.ntp.org less than X seconds
  • Domain Controller Disk Free – Verify OS partition Free space is at least X %
  • Domain Controller Disk Free – Verify NTDS partition Free space is at least X %
  • Domain Controller Operating System – Verify Windows Operating system is Windows 2012 or higher
  • Domain Controller Windows Updates – Verify Last patch was installed less than 60 days ago
  • Domain Controller SMB Protocols – Verify SMB v1 protocol is disabled
  • Domain Controller SMB Protocols – Verify SMB v2 protocol is enabled
  • Domain Controller SMB Shares – Verify default SMB shares NETLOGON/SYSVOL are visible
  • Domain Controller DFSR AutoRecovery – Verify DFSR AutoRecovery is enabled
  • Domain Controller Windows Roles and Features – Verify Windows Features for AD/DNS/File Services are enabled

And that is just a starting point, something to expand on. I've tried to pick different tests so I can see how easy for me is to add new tests without changing how Testimo works. The goal is to mostly spend time on building new tests without touching core too much. Of course, I may have missed something or made an incorrect assumption, or even that in some cases, the test will fail for some reason. After all, this is an alpha product and something I've been developing for like three weeks on and off. Hopefully, with your help, I should be able to iron out the bugs, add new tests, and improve the quality of Testimo for different architectures. Maybe you can help out as well in other areas that need improvements? Feel free to reach out on GitHub/Twitter/Reddit/Discord. I'm in all those places.

Testimo - Running Tests

I gave you a little backstory, told the goals, shown some fancy feature list, but you're not here for that right? You want to see the action! Let's go straight for that! Meet Testimo! As with my earlier tests module again here, we have a single command that does all the magic.

Invoke-Testimo

What it does it executes what I call a data source (for example Get Last Backup Dates) and then it verifies each data source against defined tests (in this case checks whether last backup date is older than two days). Of course, the defaults are well, just defaults. Each Active Directory, each company has different needs.

As you can see above, it executed 420 tests, of which 50 failed. But while Invoke-Testimo is excellent in delivering quick, visual assessment, it doesn't play well if you would want to act on it. That's why there is ReturnResults switch. The usage is simple:

Invoke-Testimo -ReturnResults

What changes? Well, you get the same visual output, but also you get PowerShell object when done. That means you can process results and act on them if you need to. It's handy for automation where you would execute this daily/weekly and if any of the tests you've defined returns false you can act on it.

Easy? It gives you an option to easily filter out on failed tests only, tests that relate to a particular domain or domain controller.

Testimo - Sending summary to email

Of course, you can do much more with that data. For example, if you were to use Emailimo (my go-to PowerShell Module to create beautiful, functional emails with zero effort) you could do something like this:

$Results = Invoke-Testimo -ReturnResults -ExcludeDomains 'ad.evotec.pl'

Email {
    EmailHeader {
        EmailFrom -Address 'myemail@evotec.pl'
        EmailTo -Addresses "otheremail@evotec.pl"
        EmailServer -Server 'smtp.office365' -UserName 'myemail@evotec.pl' -Password 'C:\Support\Important\Password-Evotec.txt' -PasswordAsSecure -PasswordFromFile -Port 587 -SSL
        EmailOptions -Priority High -DeliveryNotifications Never
        EmailSubject -Subject '[Reporting Evotec] Summary of Active Directory Tests'
    }
    EmailBody -FontFamily 'Calibri' -Size 15 {
        EmailText -Text "Summary of Active Directory Tests" -Color None, Blue -LineBreak

        EmailTable -DataTable $Results {
            EmailTableCondition -ComparisonType 'string' -Name 'Status' -Operator eq -Value 'True' -BackgroundColor Green -Color White -Inline -Row
            EmailTableCondition -ComparisonType 'string' -Name 'Status' -Operator ne -Value 'True' -BackgroundColor Red -Color White -Inline -Row
        } -HideFooter
    }
} -AttachSelf -Supress $false

As you noticed, there's also a file attached. Since I've used AttachSelf switch, it attached an enhanced HTML version to email. When you open it up, you get a bit more options to play with.

You can export your content to Excel, filter things out, and play as needed. Cool right? Emailimo will be integrated into Testimo so that you can fill in your email address and some basic settings into configuration and whenever you run it with Invoke-Testimo -Email switch it would use email information to give you report as needed. Surely that's something I would love to get feedback on thou. This is planned for the next release.

Testimo - Advanced Configuration

As I mentioned it before every Active Directory is different so that rules can be different. While for ad-hoc tests that you do to asses health check of Active Directory it's fine to visually skip false-positives, using it daily with red all over the place isn't the best way to go. That's why Testimo allows you to export/edit configuration in three ways. To file/JSON:

Get-TestimoConfiguration -FilePath $PSScriptRoot\Configuration\TestimoConfiguration.json

Which delivers something that looks like configuration below where you can change your settings as you need them.

To JSON directly (in case you want to process data however you want to handle it):

Get-TestimoConfiguration -AsJson

And finally output to hashtable which you can edit while being in PowerShell:

$OutputOrderedDictionary = Get-TestimoConfiguration
$OutputOrderedDictionary.Forest.OptionalFeatures.Tests.RecycleBinEnabled.Enable = $false
$OutputOrderedDictionary.Forest.OptionalFeatures.Tests.LapsAvailable.Enable = $true
$OutputOrderedDictionary.Forest.OptionalFeatures.Tests.LapsAvailable.Parameters.ExpectedValue = $false

How you intend to use it, it's up to you. Exporting configuration and modifying it doesn't do much. You still need to import that configuration back to Testimo.

$ConfigurationFile = "$PSScriptRoot\Configuration\TestimoConfiguration.json"

$Sources = @(
    #'ForestFSMORoles'
    'ForestOptionalFeatures'
    'ForestBackup'
    #'ForestOrphanedAdmins'
    'DomainPasswordComplexity'
    #'DomainKerberosAccountAge'
    #'DomainDNSScavengingForPrimaryDNSServer'
    'DCWindowsUpdates'
)

$TestResults = Invoke-Testimo -ReturnResults -ExcludeDomains 'ad.evotec.pl' -ExtendedResults -Configuration $ConfigurationFile -Sources $Sources
$TestResults | Format-Table -AutoSize *

I hope that didn't scare you with that code above right? As you can see above I am loading configuration from JSON file, I've also decided that I want to partially limit the number of tests I want to run (even thou configuration has it all enabled) using Sources parameter, which means only four sources will be used in next run. I've also decided to skip one of my broken domains because it takes a lot more time to scan a broken domain (timeouts on every single test take time). Finally, I'm using ReturnResults but also with another switch ExtendedResults. This will give you a bit of individual output that I'll describe a bit later on. Alternatively, the Configuration parameter also takes Hashtables as input.

$OutputOrderedDictionary = Get-TestimoConfiguration
$OutputOrderedDictionary.Forest.OptionalFeatures.Tests.RecycleBinEnabled.Enable = $false
$OutputOrderedDictionary.Forest.OptionalFeatures.Tests.LapsAvailable.Enable = $true
$OutputOrderedDictionary.Forest.OptionalFeatures.Tests.LapsAvailable.Parameters.ExpectedValue = $false

$Sources = @(
    #'ForestFSMORoles'
    'ForestOptionalFeatures'
    'ForestBackup'
    #'ForestOrphanedAdmins'
    'DomainPasswordComplexity'
    #'DomainKerberosAccountAge'
    #'DomainDNSScavengingForPrimaryDNSServer'
    'DCWindowsUpdates'
)

$TestResults = Invoke-Testimo -ReturnResults -ExcludeDomains 'ad.evotec.pl' -ExtendedResults -Sources $Sources -Configuration $OutputOrderedDictionary
$TestResults | Format-Table -AutoSize *

It's up to you which one you prefer better. Keep in mind that what Sources do is they limit tests to a defined list, which is useful. However, there's also ExcludeSources which you can use if you want to skip one or two tests but leave everything else.

Testimo - Extended Report

As mentioned earlier next to ReturnResults, there's also ExtendedResults switch. When you run Testimo with that switch, the result is not impressive. To not spend the next 10 minutes waiting for all tests to finish, I'm going to limit the number of sources and use two types of tests, along with two mentioned switches.

Invoke-Testimo -Sources ForestRoles,DomainRoles -ReturnResults -ExtendedResults

See, the result is as usual displayed to screen, and additionally, PowerShell object is returned. This object is unique and something that is work in progress that I intend to use to build an excellent looking Overview of it. I'm returning it so you can take it and make something of your own if you want to. Below is a code I'm using, to create Overview of all tests that happened (that's PSWriteHTML in action).

$Sources = @(
    'ForestRoles'
    'ForestOptionalFeatures'
    'ForestOrphanedAdmins'
    'DomainPasswordComplexity'
    'DomainKerberosAccountAge'
    'DomainDNSScavengingForPrimaryDNSServer'
    'DomainSysVolDFSR'
    'DCRDPSecurity'
    'DCSMBShares'
    'DomainGroupPolicyMissingPermissions'
    'DCWindowsRolesAndFeatures'
    'DCNTDSParameters'
    'DCInformation'
    'ForestReplicationStatus'
)

$TestResults = Invoke-Testimo -ReturnResults  -ExtendedResults -Sources $Sources #-ExcludeDomains 'ad.evotec.pl' #-ExcludeDomainControllers $ExludeDomainControllers

New-HTML -FilePath $PSScriptRoot\Output\TestimoSummary.html -UseCssLinks -UseJavaScriptLinks {
    [Array] $PassedTests = $TestResults['Results'] | Where-Object { $_.Status -eq $true }
    [Array] $FailedTests = $TestResults['Results'] | Where-Object { $_.Status -ne $true }
    New-HTMLTab -Name 'Summary' -IconBrands galactic-senate {
        New-HTMLSection -HeaderText "Tests results" -HeaderBackGroundColor DarkGray {
            New-HTMLPanel {
                New-HTMLChart {
                    New-ChartPie -Name 'Passed' -Value ($TestResults['Summary'].Passed) -Color ForestGreen
                    New-ChartPie -Name 'Failed' -Value ($TestResults['Summary'].Failed) -Color OrangeRed
                    New-ChartPie -Name 'Failed' -Value ($TestResults['Summary'].Skipped) -Color LightBlue
                }
                New-HTMLTable -DataTable $TestResults['Summary'] -HideFooter -DisableSearch {
                    New-HTMLTableContent -ColumnName 'Passed' -BackGroundColor ForestGreen -Color White
                    New-HTMLTableContent -ColumnName 'Failed' -BackGroundColor OrangeRed -Color White
                    New-HTMLTableContent -ColumnName 'Skipped' -BackGroundColor LightBlue -Color White
                }
            }
            New-HTMLPanel {
                New-HTMLTable -DataTable $TestResults['Results'] {
                    New-HTMLTableCondition -Name 'Status' -Value $true -Color Green -Row
                    New-HTMLTableCondition -Name 'Status' -Value $false -Color Red -Row
                }
            }
        }
    }
    New-HTMLTab -Name 'Forest' -IconBrands first-order {
        foreach ($Source in $TestResults['Forest']['Tests'].Keys) {

            $Name = $TestResults['Forest']['Tests'][$Source]['Name']
            $Data = $TestResults['Forest']['Tests'][$Source]['Data']
            $SourceCode = $TestResults['Forest']['Tests'][$Source]['SourceCode']
            $Results = $TestResults['Forest']['Tests'][$Source]['Results']
            #$Details = $TestResults['Forest']['Tests'][$Source]['Details']
            [Array] $PassedTestsSingular = $TestResults['Forest']['Tests'][$Source]['Results'] | Where-Object { $_.Status -eq $true }
            [Array] $FailedTestsSingular = $TestResults['Forest']['Tests'][$Source]['Results'] | Where-Object { $_.Status -ne $true }

            New-HTMLSection -HeaderText $Name -HeaderBackGroundColor DarkGray -CanCollapse {
                New-HTMLContainer {
                    New-HTMLPanel {
                        New-HTMLChart {
                            New-ChartPie -Name 'Passed' -Value ($PassedTestsSingular.Count) -Color ForestGreen
                            New-ChartPie -Name 'Failed' -Value ($FailedTestsSingular.Count) -Color OrangeRed
                        }
                        New-HTMLCodeBlock -Code $SourceCode -Style 'PowerShell' -Theme enlighter
                    }
                }
                New-HTMLContainer {
                    New-HTMLPanel {
                        New-HTMLTable -DataTable $Data
                        New-HTMLTable -DataTable $Results {
                            New-HTMLTableCondition -Name 'Status' -Value $true -Color Green -Row
                            New-HTMLTableCondition -Name 'Status' -Value $false -Color Red -Row
                        }
                    }
                }
            }
        }
    }

    foreach ($Domain in $TestResults['Domains'].Keys) {
        New-HTMLTab -Name "Domain $Domain" -IconBrands deskpro {
            foreach ($Source in $TestResults['Domains'][$Domain]['Tests'].Keys) {
                $Name = $TestResults['Domains'][$Domain]['Tests'][$Source]['Name']
                $Data = $TestResults['Domains'][$Domain]['Tests'][$Source]['Data']
                $SourceCode = $TestResults['Domains'][$Domain]['Tests'][$Source]['SourceCode']
                $Results = $TestResults['Domains'][$Domain]['Tests'][$Source]['Results']
                # $Details = $TestResults['Domains'][$Domain]['Tests'][$Source]['Details']
                [Array] $PassedTestsSingular = $TestResults['Domains'][$Domain]['Tests'][$Source]['Results'] | Where-Object { $_.Status -eq $true }
                [Array] $FailedTestsSingular = $TestResults['Domains'][$Domain]['Tests'][$Source]['Results'] | Where-Object { $_.Status -ne $true }

                New-HTMLSection -HeaderText $Name -HeaderBackGroundColor DarkGray -CanCollapse {
                    New-HTMLContainer {
                        New-HTMLPanel {
                            New-HTMLChart {
                                New-ChartPie -Name 'Passed' -Value ($PassedTestsSingular.Count) -Color ForestGreen
                                New-ChartPie -Name 'Failed' -Value ($FailedTestsSingular.Count) -Color OrangeRed
                            }
                            New-HTMLCodeBlock -Code $SourceCode -Style 'PowerShell' -Theme enlighter
                        }
                    }
                    New-HTMLContainer {
                        New-HTMLPanel {
                            New-HTMLTable -DataTable $Data
                            New-HTMLTable -DataTable $Results {
                                New-HTMLTableCondition -Name 'Status' -Value $true -Color Green -Row
                                New-HTMLTableCondition -Name 'Status' -Value $false -Color Red -Row
                            }
                        }
                    }
                }
            }
            foreach ($DC in $TestResults['Domains'][$Domain]['DomainControllers'].Keys) {
                New-HTMLSection -HeaderText "Domain Controller - $DC" -HeaderBackGroundColor DarkSlateGray -CanCollapse {
                    New-HTMLContainer {
                        foreach ($Source in  $TestResults['Domains'][$Domain]['DomainControllers'][$DC]['Tests'].Keys) {
                            $Name = $TestResults['Domains'][$Domain]['DomainControllers'][$DC]['Tests'][$Source]['Name']
                            $Data = $TestResults['Domains'][$Domain]['DomainControllers'][$DC]['Tests'][$Source]['Data']
                            $SourceCode = $TestResults['Domains'][$Domain]['DomainControllers'][$DC]['Tests'][$Source]['SourceCode']
                            $Results = $TestResults['Domains'][$Domain]['DomainControllers'][$DC]['Tests'][$Source]['Results']
                            #$Details = $TestResults['Domains'][$Domain]['DomainControllers'][$DC]['Tests'][$Source]['Details']
                            [Array] $PassedTestsSingular = $TestResults['Domains'][$Domain]['DomainControllers'][$DC]['Tests'][$Source]['Results'] | Where-Object { $_.Status -eq $true }
                            [Array] $FailedTestsSingular = $TestResults['Domains'][$Domain]['DomainControllers'][$DC]['Tests'][$Source]['Results'] | Where-Object { $_.Status -ne $true }

                            New-HTMLSection -HeaderText $Name -HeaderBackGroundColor DarkGray {
                                New-HTMLContainer {
                                    New-HTMLPanel {
                                        New-HTMLChart {
                                            New-ChartPie -Name 'Passed' -Value ($PassedTestsSingular.Count) -Color ForestGreen
                                            New-ChartPie -Name 'Failed' -Value ($FailedTestsSingular.Count) -Color OrangeRed
                                        }
                                        New-HTMLCodeBlock -Code $SourceCode -Style 'PowerShell' -Theme enlighter
                                    }
                                }
                                New-HTMLContainer {
                                    New-HTMLPanel {
                                        New-HTMLTable -DataTable $Data
                                        New-HTMLTable -DataTable $Results {
                                            New-HTMLTableCondition -Name 'Status' -Value $true -Color Green -Row
                                            New-HTMLTableCondition -Name 'Status' -Value $false -Color Red -Row
                                        }
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }
} -ShowHTML

Which will give you the following HTML document

As you can see above, there are some tabs. The first tab has a summary of all tests, and I plan to expand it with some charts and more data that may be useful for quick assessment. Following pages give you a detailed overview of all tests for Forest, and each Domain separately. Please notice that on the left side, just under pie chart, it shows you a PowerShell command that is executed to get the data. On the right side, there is output from that command, and just below it tests results. The idea is that if something is wrong, you can see more information on the problem straight away or you can take the matter in your own hands and use the provided PowerShell commands and see for yourself. Of course, you may need to replace place holder variables with your own data. I may make it more comfortable in the next versions by filling that for you. This is just a very early concept of course, so I will make sure this gets some “I want to look good” treatment. I look forward to hearing your feedback on this. Since I know whatever I will build here, may not be not enough – all that data I used to create that output internally is available by using ReturnResults, ExtendedResults switches. The report from above is also available with just this little command. Keep in mind I'm limiting sources, so I don't spend 10 minutes waiting for it to execute everything. You can skip sources parameter and get full test output to HTML for all your Forest/Domain data.

Import-Module .\Testimo.psd1 -Force #-Verbose

$Sources = @(
    'ForestRoles'
    'ForestOptionalFeatures'
    'ForestOrphanedAdmins'
    'DomainPasswordComplexity'
    'DomainKerberosAccountAge'
    #'DomainDNSScavengingForPrimaryDNSServer'
    #'DomainSysVolDFSR'
    #'DCRDPSecurity'
    'DCSMBShares'
    'DomainGroupPolicyMissingPermissions'
    #'DCWindowsRolesAndFeatures'
    #'DCNTDSParameters'
    #'DCInformation'
    'ForestReplicationStatus'
)

Invoke-Testimo -Sources $Sources -ExcludeDomains 'ad.evotec.pl' -ShowReport

The report is available online for you to view – Testimo Example Report.

Testimo - Test Sources

While you may be wondering what Sources are available, I've decided to simplify this for you. Just run a command, and you get all the available sources. It's important because the same idea will apply to export new configuration. Keep in mind that this project is not yet written in stone and not everything is yet decided so some things may change, some typos will get fixed, or definitions changed. I'll try to make it as painless as possible thou.

Get-TestimoSources
  • DCNTDSParameters
  • DCDFSRAutoRecovery
  • DCDiskSpace
  • DCDnsNameServes
  • DCDnsResolveExternal
  • DCDnsResolveInternal
  • DCLDAP
  • DCOperatingSystem
  • DCInformation
  • DCPingable
  • DCPorts
  • DCRDPPorts
  • DCRDPSecurity
  • DCServices
  • DCSMBProtocols
  • DCSMBShares
  • DCTimeSettings
  • DCTimeSynchronizationExternal
  • DCTimeSynchronizationInternal
  • DCWindowsFirewall
  • DCWindowsRemoteManagement
  • DCWindowsRolesAndFeatures
  • DCWindowsUpdates
  • DomainDNSForwaders
  • DomainDNSScavengingForPrimaryDNSServer
  • DomainDnsZonesAging
  • DomainEmptyOrganizationalUnits
  • DomainGroupPolicyMissingPermissions
  • DomainKerberosAccountAge
  • DomainOrphanedForeignSecurityPrincipals
  • DomainPasswordComplexity
  • DomainRoles
  • DomainSecurityGroupsAccountOperators
  • DomainSecurityGroupsSchemaAdmins
  • DomainSecurityUsersAcccountAdministrator
  • DomainSysVolDFSR
  • DomainTrusts
  • DomainWellKnownFolders
  • ForestBackup
  • ForestOptionalFeatures
  • ForestOrphanedAdmins
  • ForestReplication # This uses newer approach
  • ForestReplicationStatus # This uses Repadmin
  • ForestRoles
  • ForestSiteLinks
  • ForestSiteLinksConnections
  • ForestSites
  • ForestTombstoneLifetime

Of course, when you are using Sources parameter, it will autocomplete as well. But sometimes it's faster to build array without using autocomplete. It's essential to understand that each Source can have multiple tests assigned to it. For example, you could get a list of all Domain Controllers and their availability. You could then do multiple tests checking different things based on one source. I have defined some criteria on each source, but it's possible to expand on it and establish more tests to the same source.

Testimo Requirements

Testimo to work requires PowerShell 5.1. It can be installed on Windows 2008 R2 however from my tests it misses a lot of commands I use for testing DNS or other parts of the system. Since 2008 R2 extended support ends in just a few months, it seems a bit unnecessary to add support for it. However, I will consider this if there will be interest to support it. A bigger problem is I don't have Windows 2008R2 to test with. Anything above Windows 2012 should work correctly. Feel free to let me know if it doesn't, and we can figure this out. If you're running this from Windows 10 machine, please make sure you have correct permissions to run it against your Active Directory and that you have RSAT installed. I'm doing most of the tests using Windows 10 1903.

Installing & Updating Testimo
Install-Module Testimo

For easy use and installation, all modules are available from PowerShellGallery. Installing it is as easy as it gets. Keep in mind that when you install Testimo, you get PSWinDocumentation.AD, PSWinDocumentation.DNS, ADEssentials, PSSharedGoods, PSWriteColor, Connectimo, PSWriteHTML and Emailimo installed by default, so you don't have to install it separately. You may be wondering why so many modules? Well, I already have some stuff written for different purposes, and while I could get the stuff I need from different modules, I wrote and keep everything integrated, it would kill my productivity.

Sometimes you may need additional steps

Install-Module Testimo -Force -AllowClobber

Since Testimo is available as PowerShell module new versions are also published there. When a new version is out, you can run

Update-Module Testimo

Using Force and AllowClobber is optional and should be only used if you experience issues in installation. For example, if you run Install-Module with Force switch while Testimo is already installed, it will overwrite/install the newest version, but it will also redownload and overwrite any dependencies it has. AllowClobber, on the other hand, is useful when I decide to move functions between modules. I may decide that function X should be in module Y instead of module Z. When you already have function X in module Z, and you move it in next version to module Y PowerShell will detect that there is already command with that name on disk and it will complain. That's where AllowClobber comes in where it tells PowerShell to ignore this problem for now. Since both module Z and Y will get updated function will be in only one of them, and everything will start to work.

Making Testimo Portable

Since Testimo aims to be useful also in scenarios where you don't want to install Testimo, and it's dependencies to PowerShell Modules path I've created a simple function that downloads Testimo and it's dependencies from PowerShellGallery, and finally uses Import-Module to import modules into memory so that you can use it as you like. I've written a short blog about it with code you can use to get that up and running.

Initialize-ModulePortable -Name 'Testimo' -Path $Env:UserProfile\Desktop\TestimoPortable -Download

After that Invoke-Testimo and all its features are available within-session, you're using it. Of course, you don't have to use the Download switch all the time, but only when you update Testimo and it's dependencies to a newer version. If you don't use the download switch all it does is preload all modules from a given path, and that's it.

Uninstalling & Removing Testimo

Since you can install it, you can also uninstall it. Of course it's not enought to just uninstall Testimo because all other dependencies will stay there. If you're not using any of those other modules that I've written, feel free to uninstall all of them.

$Modules = @('Testimo', 'PSWinDocumentation.AD','PSWinDocumentation.DNS','ADEssentials', 'PSSharedGoods','PSWriteColor', 'Connectimo', 'DSInternals' )
foreach ($Module in $Modules) {
    Uninstall-Module $Module -Force -AllVersions
}
Development Features Requests Bugs

Testimo and all other modules are open source and are available on GitHub. If you feel you can help with just about anything – from fixing typos to adding more tests – you're very welcome!

  • Testimo – Engine mostly, this is where main development happens
  • PSSharedGoods – my “glue” PowerShell module that holds a lot of functions that are used by my modules
  • PSWriteHTML – responsible for HTML reporting
  • ADEssentials – some commands I used are based here
  • PSWinDocumentation.AD – while it's a module to build AD Documentation I use some internal commands to build tests
  • PSWinDocumentation.DNS – same as above but for DNS (not ready for documentation, useful for tests – work in progress)
  • Emailimo – module for sending/building HTML based emails.

Have fun!

Przemyslaw Klys

System Architect with over 14 years of experience in the IT field. Skilled, among others, in Active Directory, Microsoft Exchange and Office 365. Profoundly interested in PowerShell. Software geek.

Share
Published by
Przemyslaw Klys
Tags: active directoryaddhcpdnshealth checkspowershellsecurity checkstestimo
6 years ago

Recent Posts

Mastering Active Directory Hygiene: Automating SIDHistory Cleanup with CleanupMonster

Security Identifier (SID) History is a useful mechanism in Active Directory (AD) migrations. It allows…

1 month ago

Upgrade Azure Active Directory Connect fails with unexpected error

Today, I made the decision to upgrade my test environment and update the version of…

4 months ago

Mastering Active Directory Hygiene: Automating Stale Computer Cleanup with CleanupMonster

Have you ever looked at your Active Directory and wondered, "Why do I still have…

8 months ago

Active Directory Replication Summary to your Email or Microsoft Teams

Active Directory replication is a critical process that ensures the consistent and up-to-date state of…

1 year ago

Syncing Global Address List (GAL) to personal contacts and between Office 365 tenants with PowerShell

Hey there! Today, I wanted to introduce you to one of the small but excellent…

1 year ago

Active Directory Health Check using Microsoft Entra Connect Health Service

Active Directory (AD) is crucial in managing identities and resources within an organization. Ensuring its…

2 years ago