Category: Consulting

Working With A SQL Server Consultant Checklist: Who Should Attend Meetings, And When?

Posted on by Erik Darling

This Week Only

I’m going to be running some posts about working with clients, which might be sort of specific to the way I tend to work with people.

I realize it’s not my usual technical content, but every once in a while the brain needs a break from writing about that stuff all the time.

Anyway, the first post up is about who should attend meetings.

Stakeholders

Usually there’s a mix of people who are potentially invested in SQL Server.

  • Technical staff, like DBAs, developers, and sysadmins
  • Executives and managers
  • Customer support to advocate for client issues
  • Third party developers who build the product

Sometimes there are other people involved, but that covers the most common mix.

I like working with people who work with the server and product directly, because they have the best domain knowledge about why things are the way they are, and where the bodies are buried.

Working In The Dark

I tell everyone I work with that I don’t like working in the dark. That means a couple things:

  • I don’t want my process to be a black box, I want to build self-sufficiency where I can
  • I don’t want to wander blindly through a server, I want to focus on the pieces where the pain is

That’s why I kick off every engagement by working via web meeting with anyone who’s interested. We look at everything all together.

If it touches the SQL Server, it’s in scope:

  • Hardware
  • Settings
  • Queries
  • Indexes
  • Monitoring data

Other stuff sometimes comes into the mix, like looking at application code and servers, but there’s not a ton I can do to fix those directly. It’s mostly just advice.

I save any scripts we use so folks can run them later, on any other SQL Server. That’s part of building self-sufficiency, but not working in the dark isn’t just for your benefit, it’s for me too.

Anyone can find a slow query, but I like to focus in directly on the queries that are causing the business pain I’m getting paid to solve.

Even with Query Store or a monitoring tool, you’re only guessing at exactly which queries users are complaining about.

Showtime

The thing is, not everyone needs to attend every minute of every meeting.

Executives and managers very rarely have the time and technical knowledge to make attending start to finish. They might care very much about the product, and making sure the engagement is successful, but I usually save their time for delivery of findings.

After the initial analysis, I write detailed findings about the issues that get uncovered during the initial meeting, and I start them off with an executive summary.

You know, for executives.

That gives the really busy, non-technical folks a chance to hear high level pain points, and to ask questions about them without having to sit through all the technical voodoo behind them

After that, I get into all the gory details with the folks who have the best chance of understanding them.

In tomorrow’s post, I’ll talk about how I get access to client SQL Servers in different circumstances.

Thanks for reading!

Going Further

If this is the kind of SQL Server stuff you love learning about, you’ll love my training. I’m offering a 75% discount to my blog readers if you click from here. I’m also available for consulting if you just don’t have time for that and need to solve performance problems quickly.

Darling Data Three Years In: Some Cool Stuff

Posted on by Erik Darling

Been A While

I wrote one of these after six months of independent consulting, and then never cared to do one again. You might find the April 1 timing of this weird, but there’s no additional Fools here, just me as usual.

Why bother now? Well, after three years, I figure I’m probably too much of a dandy to ever get a real job again. Since that’s the case, I’ve done a couple things:

  • I purchased the domain erikdarling.com for a stupid amount of money
  • I hired some nice gals from Brooklyn to make it look professional

Now, here’s the thing: many times in my life spending a significant amount of money on things signaled me about to quit doing that thing. I have many thousands of dollars worth of hobby equipment that sits forgotten in various bags, boxes, and closets.

But not this time! Unless this here PowerBall ticket is lined up with the stars.

When I first started getting set up to do this, I bought an easy domain, hooked it up to WordPress, designed some logos myself that I thought evinced a SQL coach aesthetic or some such, and let it ride.

Turns out that very few people want a SQL coach. They want someone to work out for them.

I never said I was a fast learner.

Number Signs

Since that first section isn’t much of a blog post, I’ll talk a little about how things are doing business-wise.

I’ve worked with over 500 consulting clients, and I’ve sold training to over 1,500 people. Most of the training sales are fairly low dollar value, but I’m hoping that changes with the move to Podia.

For consulting, I’ve got a couple basic products: a two-day performance analysis, or a bucket of hours. I still don’t do on-call or night/weekend work, but I found that a lot of shops out there want the insurance policy that comes with having time on the clock to use.

There’s no need to get into great detail here, because I often tweak things in response to individual clients, but this has been working out really well.

Anyway, keep an eye out for a SEAMLESS TRANSITION to the new website over the next month or so, and uh… I dunno.

Other stuff.

We’ll see.

Whatever.

Thanks for reading!

Going Further

If this is the kind of SQL Server stuff you love learning about, you’ll love my training. I’m offering a 75% discount to my blog readers if you click from here. I’m also available for consulting if you just don’t have time for that and need to solve performance problems quickly.

SQL Server Performance Problem Solving: Why I Love Consulting

Posted on by Erik Darling

30 Second Abs

I love helping people solve their problems. That’s probably why I stick around doing what I do instead of opening a gym.

That and I can’t deal with clanking for more than an hour at a time.

Recently I helped a client solve a problem with a broad set of causes, and it was a lot of fun uncovering the right data points to paint a picture of the problem and how to solve it.

Why So Slow?

It all started with an application. At times it would slow down.

No one was sure why, or what was happening when it did.

Over the course of looking at the server together, here’s what we found:

  • The server had 32 GB of RAM, and 200 GB of data
  • There were a significant number of long locking waits
  • There were a significant number of PAGEIOLATCH_** waits
  • Indexes had fill factor of 70%
  • There were many unused indexes on important tables
  • The error log was full of 15 second IO warnings
  • There was a network choke point where 10Gb Ethernet went to 1Gb iSCSI

Putting together the evidence

Alone, none of these data points means much. Even the 15 second I/O warnings could just be happening at night, when no one cares.

But when you put them all together, you can see exactly what the problem is.

Server memory wasn’t being used well, both because indexes had a very low fill factor (lots of empty space), and because indexes that don’t help queries had to get maintained by modification queries. That contributed to the PAGEIOLATCH_** waits.

Lots of indexes means more objects to lock, which generally means more locking waits.

Because we couldn’t make good use of memory, and we had to go to disk a lot, the poorly chosen 1Gb iSCSI connections were overwhelmed.

To give you an idea of how bad things were, the 1Gb iSCSI connections were only moving data at around USB 2 speeds.

Putting together the plan

Most of the problems could be solved with two easy changes: getting rid of unused indexes, and raising fill factor to 100. The size of data SQL Server would regularly have to deal with would drop drastically, so we’d be making better use of memory, and be less reliant on disk.

Making those two changes fixed 90% of their problems. Those are great short term wins. There was still some blocking, but user complaints disappeared.

We could make more changes, like adding memory to be even less reliant on disk, and replacing the 1Gb connections, sure. But the more immediate solution didn’t require any downtime, outages, or visits to the data center, and it bought them time to carefully plan those changes out.

If you’re hitting SQL Server problems that you just can’t get a handle on, drop me a line!

Thanks for reading!

Going Further

If this is the kind of SQL Server stuff you love learning about, you’ll love my training. I’m offering a 75% discount to my blog readers if you click from here. I’m also available for consulting if you just don’t have time for that and need to solve performance problems quickly.

Independent Consulting: Six Months And Running

Posted on by Erik Darling

Knock Knock

There’s nothing technical in this post.

It’s just sort of a general accounting of what six months of independent consulting has looked like for me.

Some things I’ve learned, things I’m happy with, things I’m unhappy with. I dunno.

We’ll see where this goes as I’m writing it. Don’t count on it being terribly insightful.

There are countless people who are owed great amounts of gratitude for the advice, encouragement, referrals, and all the other Spackle needed to get things off the ground.

Words in a blog post are insufficient. I’m plotting my thank yous carefully.

Caliente

I’ve had 23 clients in 6 months, not counting work that Mike Walsh has outsourced to me (insert largest possible thank you here).

The projects have varied quite a bit in length and price, so don’t go trying to reverse engineer my income.

I’m not as comfortable as some other people in talking about that, though I’m entirely willing to tell you I made $300 in January.

The only thing I’ll say is that I made just about my base salary last year at the beginning of May.

I’m very happy with that, but the thing I want to share is this: lump slums are really misleading.

I learned this from Brent.

An equally large thank you should be here, too, for probably the best education one could ask for in how to run a consulting business.

Bill Factor 100

Whenever money comes in, I pull 30% out to pay taxes with.

I stick it in a savings account, so maybe it’ll earn whatever pittance interest rates are these days.

Just to use a round number: If I charge someone $1000, $300 leaves immediately.

When you’re setting prices on your time, factoring in that 30% is really important.

The other things you need to consider are, obviously, your expenses.

I have rent, I have to fully pay for my own insurance now, and I have the ever-present work-harder motivation that is a wife and two kids.

Plus the usual phone, internet, utilities, and other recurring expenses.

To put $1000 in perspective: if I charge someone $1000, I still owe my insurance company $800.

To put it further in perspective, I’d still owe my landlord $2000.

And I owe Uncle Sam™ that $300. Whew.

If you ever wonder why consultants seem expensive, it’s because we have to factor a lot of stuff in.

Every price tag has that stuff (plus the constant fear of a slow month) built in to it.

Happy Happy

So what am I happy with? I have a thing that’s mine. I can do whatever I want with it.

I’ve found it fun building my own “brand”, even if I’m not terribly good at building a brand.

I’m happy with the work I’ve been doing, too. With Brent, it was very much hands off analysis.

I’ve been getting back into actually tuning stuff.

And you know what? It’s rewarding to know that the advice you give people actually works.

It’s not all fun and games, but what is?

At least I choose what I say yes or no to, so I only have myself to blame.

I’ve gotten to travel a bit more, though the locales haven’t exactly been exotic.

Downers

Stuff I’m unhappy with is largely cosmetic.

I hate the way my website looks. It’s dark, and it doesn’t sell anything very well.

All of the choices I made at the very beginning were very much hair-on-fire, oh-crap-I-gotta-make-this-work-quick.

Believe it or not, layoffs don’t come with a lot of advance notice.

Anyway, I feel like it shows when I look at my website. And I just don’t have time to do anything about it right now.

I’m also not great at marketing.

I mean, I’m pretty good at saying funny things about what I’m doing, but I have no idea how to get it in front of more faces, or get more people to say “yes”.

That psychology escapes me.

I’ve read books. I’ve seen doctors. I’ve talked to people. I just don’t think I have that gene.

Time Management

The one thing I’m really beating myself up about is training.

Writing it from scratch is hard.

And long.

Long and hard.

Like a morning without drinking.

It’s coming along, but all the “keep the lights on” work seems to eat away at the time I want to spend building it.

I’M A BIT OF A PERFECTIONIST (ha ha ha)

No but really. I have a lot to write down.

I’m plotting stuff that can take you from beginner to wherever I’m at currently. Hopefully that line keeps moving.

Futuristic

Alright, yeah. This thing.

My hope is that I can focus on getting training written and produced, even if it’s slower than I’d like.

I realized it would be daft to work on a set of my own data analysis scripts, so I’ll be working on the First Responder Kit as time allows.

When I’m comfortable with the amount of money in the bank, I’ll probably get the website a facelift.

Maybe I’ll magically get good at marketing.

Thanks for reading!

Going Further

If this is the kind of SQL Server stuff you love learning about, you’ll love my training. I’m offering a 75% discount to my blog readers if you click from here. I’m also available for consulting if you just don’t have time for that and need to solve performance problems quickly.

Building SQL Server Tools vs Building SQL Server Knowledge

Posted on by Erik Darling

Time Served

I’m gonna be honest with you: I’m not gonna build another First Responder Kit.

It’s established, it’s open source, and it’s badass. I poured a ton of work into it over the years, and starting from scratch seems like a bad use of my time.

I totally encourage you to help continue to build it. I learned a ton working on that stuff, and it was an incredibly valuable and rewarding experience.

I am going to build tooling that I think would be useful but that isn’t covered in there, for things like:

Any Tool Can See

Using any tool that returns a given set of information about wait stats, query plans, indexes, or whatever about SQL Server will show you roughly the same problems.

Roughly. Some better than others. Some I have no idea. Some not so much.

The point is, I can teach you to find problems with the tool you’re using, or help you find a tool that does.

I can also teach you how to solve them.

Nothing New But The Name

My thing over here is coaching. Helping you become better at whatever it is you wanna do.

Every tool is a wrapper for what’s inside SQL Server. Tools are interchangeable, mostly.

Knowing how use and interpret them in a meaningful way is not.

Thanks for reading!

Going Further

If this is the kind of SQL Server stuff you love learning about, you’ll love my training. I’m offering a 75% discount to my blog readers if you click from here. I’m also available for consulting if you just don’t have time for that and need to solve performance problems quickly.

Introductory

Posted on by Erik Darling

Testing, Yes, Yes

About a year and a half ago, when I started writing and presenting The Junior DBA Workout Plan, I was also thinking about the eventual inevitability of someday not working for Brent, and exactly what I’d wanna do afterwards.

It was a tough question at the time, because I felt like I’d get my butt handed to me trying to do the same thing, consulting-wise. One thing we always lamented was a general lack of follow-through on recommendations, which got me thinking about how I could do things differently, so as not to be handed my butt in the most thorough of ways.

A Bridge Appears

Let’s face it, you’re having trouble finding your SQL Server’s weak points.

Reading the wait stats hieroglyphics, understanding index crop circles, and those query plans that go on for days.

You’re gonna have a hard time fixing all that stuff on your own.

Even with explicit instructions and scripts, you’re gonna get gun shy. You’re gonna have other stuff to do, and your SQL Server tasks are gonna fall off.

I wanna bridge that gap, and make sure you stay on track.

Coaching, Not Consulting

I want to give you in-depth, personalized advice and training when I assess your server, but then follow up with you on a consistent basis to make sure you’re progressing and hitting milestones.

Look, there’s a million places to get a boilerplate. You need someone who can cut through the decades of meme advice out there, tell you what to do and how to do it, and make sure you stick to the plan.

That way, if you hit a wall, something doesn’t work, something stops working, you’re not stranded. We can adjust the plan together.

Going Further

If this is the kind of SQL Server stuff you love learning about, you’ll love my training. I’m offering a 75% discount to my blog readers if you click from here. I’m also available for consulting if you just don’t have time for that and need to solve performance problems quickly.