Monday, December 7, 2015

Clustered Instance Patch Failure - The Cluster Group Cannot Be Determined

I recently ran into an issue where I was unable to apply a cumulative update to one node of a Failover Clustered Instance (FCI) of SQL Server 2014.  The Summary log from the CU install had the following error:
The cluster group cannot be determined for the instance name 'MSSQLSERVER'.  This indicates there is a problem with the product registry setting for ClusterName, with product discovery, or the cluster resources.

This particular FCI was a DR cluster and we had renamed the network name resource for the instance to match the production server it was replacing.  I believe the instance name in the registry should automatically change when you change the network name, but in our case it did not do so.

To find the correct name, I launched Failover Cluster Manager and then expanded the cluster name and selected Roles in the left-hand pane.  From there I selected the Resources tab at the bottom of the center page.  
Note: I'm running Windows Server 2012 R2.  If you are on Windows Server 2008 (or 2008 R2), you'll need to expand Services and applications in the left-hand pane and then select the specific service you are looking for.  

In the Resources list, I found the Server Name resource and made note of the value specified there.

Next is just a simple matter of updating the registry value.  I launched Registry Editor (RegEdit.exe) and navigated to HKLM > Software > Microsoft > Microsoft SQL Server > MSSQL12.MSSQLSERVER > Cluster
Note: MSSQL12.MSSQLSERVER will vary based on SQL version (SQL 2012 is "MSSQL11") and instance name ("MSSQLSERVER" is the default instance and named instances are the actual instance name instead of "MSSQLSERVER").

Within the Cluster key I updated the value of ClusterName to match what was specified in Failover Cluster Manager:

I then ran the CU install again and it completed successfully!  So far this is a good Monday!

Saturday, January 17, 2015

Interview With A DBA - Why Do You Do What You Do?

I recently had a student conducting a career study/project contact me with questions about my career and the choices I made to get where I am.  I'm certainly not perfect by any stretch of the imagination, but I give it my all.  I hope that my answers to his questions helped inspire him to go after his dream, whatever it may be (even if it's not to be a DBA).

Student: What is the title of your career?
Me: My career title is essentially Database Administrator.  My current job title is Tier III Database Administrator (Senior DBA), and I specialize in architecture/infrastructure engineering for Microsoft SQL Server.

Student: How many years have you had on the job?
Me: I've been working with databases in general for about 15 years, and specifically SQL Server for over 10 years.

Student: What attracted you to this career? Did you have an experience that influenced your decision to choose this field?
Me:  I started out my professional career in a small call center providing phone support.  I gradually worked my way into more of an IT/Help Desk role and then up to Network Engineer.  When I started working as a Network Engineer, I happened to be the only one on the team that wasn't scared of the database servers.  As I learned more about SQL Server, I realized that I really liked working with it and I wanted to learn more about it.  There is a bit of an infinite cycle there, the more you learn about a subject, the more you realize how much more there is to learn about it.  Once I realized that I was more passionate about working with data and SQL Server than anything else, I made that my primary focus.
One thing that has really solidified my decision in my career is the tremendous sense of community and even family when you work with SQL Server.  There are plenty of people who are more than happy to take some time out of their day to help you out if you have a problem, or just want to chat.  Check out the #SQLHelp and #SQLFamily hash tags on Twitter for just a couple examples of what I'm referring to here.

Student: What type of education or training is required for your job?
Me: A strong foundation on math, and logic/analytical thinking comes into play when working with SQL Server.  There are multiple different job roles/responsibilities/career paths in working with SQL Server (and even more if you consider other database platforms like Oracle, MySQL, Hadoop, etc...).  Programming classes (specifically T-SQL classes would be ideal) would also help you to prepare for working with SQL Server.  Almost every aspect of SQL Server requires some familiarity with T-SQL code and the logic behind structuring a query and getting it to perform.  While it's pretty obvious that a SQL developer needs to have strong T-SQL coding skills, as a DBA, I also need be very familiar with T-SQL coding so that I can tune queries that no longer perform well.
One way to get free training is to attend local events or online training.  I am the president of the Rochester PASS SQL Server User Group and we have monthly meetings during most of the year (I skip Summer and December).
More online training can be found in the form of virtual chapters (periodic webinars/meetings online). More virtual chapter information can be found here: And finally, there is a one day SQL Server mini-conference (appropriately named SQL Saturday) coming to Rochester on Saturday, May 16th.

Student: What specific skills are needed to do the job?
I think my previous comments speak to this well, but I will also add that there are times that you need to have tenacity and focus in order to solve problems (this applies to many other careers than just a career in SQL Server).  I am a highly distractable person, and I find it hard to focus on a problem at times, but I still manage quite well.  It is just more difficult for me to go heads-down and focus solely on a specific issue.

Student: What are your major daily tasks?
Major daily tasks for me include reviewing alerts (backups, critical errors, performance issues, failed scheduled jobs, etc...) from the database servers (and other infrastructure alerts like storage and network). Typically our Junior and Mid-level DBAs handle those tasks, but I will also keep an eye on these so that I have a good understanding of the health of our servers.  I am an escalation point for major issues and incidents. And my typical role (outside of problem/incident response) is project based.  We recently migrated to a new SAN and I was responsible for migrating all the database files from the drives associated with the old SAN to the drives/paths on the new SAN.  I'm also working on a project to upgrade our infrastructure to SQL Server 2014 (our last production servers will upgrade this weekend and next week).  I also provide specifications for ordering new server hardware and balancing server loads and a host of other tasks.  No two DBA jobs/roles will be exactly the same, but the career is extremely rewarding.

Student: What do you like most about your job and why?
I love working with other people and learning new things.  I learned a long time ago that even after you graduate and you are "done" with school, you never stop learning.  The best employees make sure they're staying up to date on the technologies that they work with so that they can provide the best solutions to the problems that they face.  I love working with data!  I was a math and science geek in school and college, and that really helped me prepare for a career working with data, servers and code.

Student: What are some dislikes or frustrations you have and why:
One of my biggest frustrations is that when you work with people, you will always find that one person that just does not understand how things really work, and won't take no for an answer.  Like if someone puts in a request that would require a week worth of work (and that's assuming that you drop everything else and focus on just this one request, which is likely never the case), and then asks if it can be done in a day or two.  Poor planning on their part doesn't always necessitate an emergency on mine, but sometimes it does.  This gives me an opportunity to discuss the situation with the person requesting the task, and we can set proper expectations (that's a critical point, make sure that you understand what is required of you and that the person requesting it understands what and when they can expect a response).  However, you're always going to need to work with other people (management, team members, customers, etc...), so interpersonal skills are a must. 

Student: How do your perceive the future of this field in terms of expanding opportunities and security?
Technology is always changing, and you need to continually learn about the products you're working with (that way you don't become obsolete).  SQL Server is not going away any time soon, but it is adapting to fit more modern business needs.  I go to the premier SQL Server conference every year, the PASS Summit.  

There have been a lot of security improvements in SQL Server over the years, including integration with Windows Active Directory, encryption, and multiple methods to protect your data while in-flight (in memory) or at rest (on disk).  I am a big advocate of least privilege.  Only give the minimum amount of people the minimum permissions that they need.  However, depending on the situation, that is not always easy, or possible (there are exceptions to every rule).  If you work with any sensitive data (credit cards, Social Security numbers, Dates of Birth, etc...), you will be audited on how you handle and protect the data.  The more important the data, the more steps you need to take to protect it (and the more paperwork you need to fill out to prove that it is protected).

Student: Has the job changed since you began? if so please elaborate:
There isn't much in the technology field stays the same for very long, and SQL Server is no exception.  Apart from the product changes already mentioned, one of the biggest changes is that the amount of data that I work with has done nothing but grow.  When I first started working with Databases, they would fit on a floppy disk (just a few hundred KB).  Currently, I'm working with databases that are multiple terabytes (our largest single database is almost 4TB, but I know others that work with much larger databases), and this results in a lot more data flying around when you're querying the database.  You have to do a lot more tuning to queries that are hitting large amounts of data in order to get them to perform well.

Student: Is there anything I currently can do to prepare for this career?
I'd recommend you check out this blog series for the accidental DBA:  Gaining an understanding of relational databases and their structures will definitely help.  Personally, I had no clue when I first started working with SQL Server, but I was able to learn as I went along.

Student: Is there other information that may be helpful?
The bottom line is to find something you love to do.  To do that, you have to try a lot of things.  I didn't discover my passion for SQL Server until over a decade into my professional career.  Don't be afraid to try new things.  Don't be afraid to learn new things.  And don't be afraid to make mistakes.  That's how we all learn.

One of my favorite quotes by Albert Einstein is, "Anyone who has never made a mistake has never tried anything new."  Don't be afraid to give it all you've got to follow your dream.  You may make mistakes, but you'll never wonder, "What if I had just..."

No regrets!  I know I can certainly say that about my career.

To my new friend Hunter, I wish you the very best of luck in whatever you do.  Find your passion and give it your all!