3 Node Storage Spaces Direct Cluster Works!!!

I went through the following URL https://technet.microsoft.com/en-us/library/mt126109.aspx but instead of creating a 4 node Storage Spaces Direct cluster, I decided to try and see if a 3 node cluster would work. Microsoft documentation says that they will only support Storage Spaces Direct with 4 servers but I thought it can't hurt to try a 3 node... and it worked!!

Storage Spaces and Latent Sector Errors / Unrecoverable Read Errors

I emailed S2D_Feedback@microsoft.com to ask about storage spaces direct and how it handles Latent Sector Errors (LSE), otherwise known as Unrecoverable Read Errors. Here is the email I sent:

My company is in the process of evaluating different options for upgrading our production server environment. I’m tasked with finding a solution that meets our needs and is within our budget.

I’m trying to compare and contrast storage spaces direct with storage spaces utilizing JBOD enclosures. Data resiliency, integrity and availability are paramount. So I’m primary looking at both of these technologies from that perspective. Thus, if we go the JBOD route, we’re looking at implementing 3 enclosures and utilizing the enclosure awareness of storage spaces. This solution has existed longer then storage spaces direct and I would think has been tested more thoroughly. I like the scalability and elegance of storage space direct though. From a conceptual overview and a hardware setup perspective it just seems easier to grasp and it seems like a better solution.

My question is, how do both of these setups handle unrecoverable read errors/latent sector errors? Does one solution handle them better than the other?

There are horror stories about hardware RAID controllers evicting drives because of URE/LSE and then during RAID rebuilds encountering additional UREs/LSEs and bricking the storage. This is more worrisome when SATA disks are used (due to UREs/LSEs occurring more often and sooner with SATA disks compared to SAS disks.) How does storage spaces/S2D differ in this regard? I know one of the selling points of S2D is the use of SATA disks. I’m curious as to how this problem has been addressed since SATA disks are being promoted. What happens if there is a URE/LSE in end user data? What happens if there is a URE/LSE in the metadata used by storage spaces/S2D or the underlying file system?

Here is the response I received:

Both Spaces direct and Shared Spaces (with JBOD)  both rely on the same software raid implementation, difference is in the connectivity.  Software raid implementation does not throw away the entire drive on failure, we trigger activity to move the data out of the drive while keeping the copy till data is moved (if we have copies available).  On Write failure we try to move the impacted range right away while background activity is moving the untouched data out of the disk,  some of the disks fail to write but they can continue to support reads in which case the data on those drives can still be used to serve user requests.  Until the data on the failed drive is rebuilt on spare capacity the drive is not removed, user can still force but not automated.  On URE - we trigger rebuilt to recover lost copy, this is triggered both when reads errors detected while satisfying user error or by back ground scrub process.  Back ground scrub process detects URE by validating sector level checksum across copies and  validating. 

So it would appear that if you utilize storage spaces you don't have to worry about a LSE/URE taking out a drive and then a subsequent LSE/URE taking out another drive, thus taking down your array. 

IIS Application Initialization Quick Reference

1.  Need to install application initialization module, if IIS 7.5 it's a separate download or WPI, included in IIS 8

2. Set startMode to AlwaysRunning on application pool
1. Open configuration editor in IIS on server root
2. Select system.applicationHost/applicationPools
3. Click edit items
4. Find app pool, select and then change startMode in lower pane to alwaysRunning
5. Hit Apply This changes C:\Windows\System32\inetsrv\config\applicationHost.config

3. Set applicationDefault preloadEnabled on site
1. Open configuration editor in IIS on server root
2. Select system.applicationHost/sites
3. Click edit items
4. Find site and select
5. Expand applicationDefaults in lower pane
6. change preloadEnabled to true
7. Hit Apply This changes C:\Windows\System32\inetsrv\config\applicationHost.config
8. Note: This only changes the defaults for new apps, you may need to change the existing ones yet. You may not need step 3 if apps already exist…

4. Set preloadEnabled on site
1. Open C:\Windows\System32\inetsrv\config\applicationHost.config
2. Find site you're looking for <site name="www.domain.com" id=…
3. Add preloadEnabled="true"  to first application element tag
4. You should see a  <applicationDefaults preloadEnabled="true" /> under the site element if you performed step 3
5. Restart IIS

5. Set initalizationPage on site and doAppInitAfterRestart
1. Open configuration editor in IIS on desired site
2. Select system.webServer/applicationInitialization
3. Change from to ApplicationHost.config if you want to create a location element in C:\Windows\System32\inetsrv\config\applicationHost.config, otherwise the change will go in the web.config
4. Set doAppInitAfterRestart to true
5. Click edit items
6. Click add
7. Enter path for initializationPage ( /folder/page?param=something ), leave hostname blank (I think…)
8. Click apply

CHANGE IDLE TIME_OUT IN ADVANCED SETTINGS ON APPPOOL TO 0

DPM 2010 Slow when Selecting Roverypoint to Recover

It took almost an hour to select a date and time to recover in DPM 2010. It appeared the this was because of high cpu usage by SQL. The query that seemed to be responsible for this was:

SELECT Path, FileSpec, IsRecursive
    FROM tbl_RM_RecoverableObjectFileSpec
    WHERE RecoverableObjectId = @RecoverableObjectId AND 
          DatasetId = @DatasetId and
          iSgcED = 0

I didn't dig into things too much, but it appeared as though it was running this query for every single recovery point for the item selected and it was doing a clustered index scan for each recovery point. I created the following statistic and covering nonclustered index in the DPM db:

CREATE STATISTICS [_dta_stat_1042102753_9_2_3] ON [dbo].[tbl_RM_RecoverableObjectFileSpec]([IsGCed], [RecoverableObjectId], [DatasetId])

CREATE NONCLUSTERED INDEX [_dta_index_tbl_RM_RecoverableObjectFileSpec_7_1042102753__K2_K3_K9_5_6] ON [dbo].[tbl_RM_RecoverableObjectFileSpec] 
(
[RecoverableObjectId] ASC,
[DatasetId] ASC,
[IsGCed] ASC
)
INCLUDE ( [FileSpec],
[IsRecursive]) WITH (SORT_IN_TEMPDB = OFF, IGNORE_DUP_KEY = OFF, DROP_EXISTING = OFF, ONLINE = OFF) ON [PRIMARY]

Now the above query changed from a clustered index scan to a index seek and key lookup. The time it took to select a recovery point went from about an hour down to a minute.

Email When Scheduled Task Fails

1. Create a new scheduled task
2. On the actions tab, click new
3. Change the action to send an e-mail
4. Enter the to, from, subject, text and smtp server information
 For subject and text enter something along the lines of "A schedule task failed on server so and so"
5. Click OK

6. On the triggers tab click new
7. Change the "Begin the task" drop down to "On an event"
8. Click "Custom" under settings
9. Click "New Event Filter"
10. Click the XML tab
11. Check Edit query manually
12. Enter the following XML for the query:

<QueryList>
  <Query Id="0" Path="Microsoft-Windows-TaskScheduler/Operational">
    <Select Path="Microsoft-Windows-TaskScheduler/Operational">*[System[Provider[@Name='Microsoft-Windows-TaskScheduler'] and (EventID=201) ]]
and
*[EventData[Data[@Name='ResultCode'] and (Data='1')]] </Select>
  </Query>
</QueryList>

Now anytime a schedule task completes with a result code of 1, an email will go out letting you know. 1 in our case indicates an error. What ever task you have may return other codes to indicate errors. You could say were data > 0.

IIS and Certificates Random Musings

If you view a cert, windows goes out and downloads the root and puts it in the third-party trusted root certificate authorities store.

If you select a cert in IIS binding, windows goes out and downloads the root and puts it in the third-party trusted root certificate authorities store. It also downloads the intermediate certificates and places them in the intermediate certificate authorities store. (Assuming the cert has a proper AIA configured and accessible.) Thus, IIS will work and serve the intermediate certs even if you didn't explicitly install them into the intermediate cert store. I'm sure this was done to make things simple on admins.

I disabled the gateway and dns so that the windows box could not get out to the internet. Windows did not download the chain when viewing the cert by double clicking and IIS only served the one certificate.  IIS did not download the intermediate certs. (It couldn't.)

You can view the certificates IIS serves with openssl:

openssl s_client -showcerts -connect www.domainname.com:443

Of note, it appears as though the windows crypto APIs cache previous root certs. I viewed a cert while the server had internet access. Windows downloaded the root cert and put it in the third-party trusted root certificate authorities store. I removed the root cert from the third party store, disabled internet connectivity and viewed the same leaf cert again. Without an internet it placed the root cert in the third-party trusted root certificate authorities store again.

While disconnected, I manually installed intermediate certs in intermediate store. IIS did not server them until I touched the bindings of the site in IIS. You need to edit the binding in IIS and just hit ok for the new chain to be served. Just adding intermediate certs into the store will not do it.

It looks like IIS creates the cert chain when the binding is configured and then stores it somewhere for future access. Likely so that path resolution doesn't have to occur over and over. IIS just has to do it once on binding configuration. Restarting IIS/the website had no effect. The binding needed to be modified/re-applied. Restarting the server seems to have updated the certificates served by IIS. IIS's cert chain storage must not be persistent.

I tried placing the intermediate cert in the personal store and IIS did not serve it. I tried placing the intermediate cert in the root store and IIS did server it. So technically intermediate certs could go in the root or intermediate store. I would not put them in the root though as that has other implications.

The big take aways here are:

1. Windows/IIS will try to make things easy for you as an admin and download/configure cert chains for you
2. IIS will only update the certificates it send to clients when you update the binding or restart the server

In the future I'd like to test out certs with multiple paths and discover how IIS/path resolution determines path priority. Will one store take precedence? Does it base it solely on whatever window's path selection algorithm is? How will auto downloading play in? I personally think the way IIS serves certificates could use an overhaul. It would be nice to be able to server multiple certificates from IIS. For example, depending on client handshake capabilities, serve either RSA or ECDSA hashed cert for authentication to root CA. Then we could easily transition to ECC certs.

Find Custom Config Section in web.config by Type instead of by String

Instead of calling Configuration ConfigurationManager.GetSection("name of section"), call the following:

CustomSection section = GetConfigSection(typeof(CustomSection)) as CustomSection;

This way, you don't have to hard code the name of the section into your code and it allows who ever is consuming your code to change section names and group them as they see fit.

You'll need to add the following code for the above function. You could add it to your config class and add strong typing or you could add it to a utility class.

/// <summary>

/// Instead of getting the config section by name, use this to get the config section by type. It will return the first section with a matching type. 

/// Thus if there are multiple sections with the same type, it will only return the first one.

/// Call like CustomMailWebEventProviderSection section = GetConfigSection(typeof(CustomMailWebEventProviderSection)) as CustomMailWebEventProviderSection;

/// </summary>

/// <param name="configSectiontype">Type of config section you're looking for</param>

/// <returns>Config section of the specified type or null if no section of that type is found</returns>

public ConfigurationSection GetConfigSection( Type configSectiontype )

{

    // Set it up and call recursive function

    Configuration config = System.Web.Configuration.WebConfigurationManager.OpenWebConfiguration("~");

    return GetConfigSection(configSectiontype, config.Sections, config.SectionGroups);

}

// returns first config section that matchs the type we're looking for

public ConfigurationSection GetConfigSection(Type configSectiontype, ConfigurationSectionCollection sections, ConfigurationSectionGroupCollection groups)

{

    if (sections != null)

    {

        foreach (ConfigurationSection section in sections)

        {

            if (section.GetType() == configSectiontype)

            {

                return section;

            }

        }

    }

    if (groups != null)

    {

        foreach (ConfigurationSectionGroup group in groups)

        {

            ConfigurationSection section = GetConfigSection(configSectiontype, group.Sections, group.SectionGroups);

            if (section != null)

            {

                return section;

            }

        }

    }

    //section not found

    return null;

}