We Monitored Our Data Pipelines Manually for Months — Here's What Broke

Before DashGuard existed as a product, it existed as a messy collection of SQL scripts, cron jobs, and a shared Google Sheet that we optimistically called our "data quality tracker."

We were a data team at a mid-size company, and like most data teams, we'd been burned by bad data enough times that we decided to do something about it. Our plan was straightforward: write checks for our most important tables, run them regularly, and keep track of the results.

Simple, right?

It was — for about three weeks. Then reality set in.

This is the story of what we tried, what went wrong, and what we eventually built to fix it. If you're currently managing data quality with spreadsheets and good intentions, this might save you some pain.

Phase 1: The Honeymoon Period

We started small. Our analytics database had maybe 40 tables that mattered, feeding a couple dozen dashboards used across the company. We picked the ten most critical tables and wrote SQL checks for each one:

• Is the data fresh? (Check the max timestamp against the current time)
• Are row counts in a reasonable range? (Compare today vs. the 7-day average)
• Are there unexpected nulls in key columns?

We set up cron jobs to run these queries every morning at 6am, dump the results into a CSV, and email the file to our team Slack channel.

For the first few weeks, it felt great. We caught a stale table on day four — the pipeline had failed overnight and we knew about it before anyone else did. We felt like geniuses.

Phase 2: The Spreadsheet Era

After our early wins, the rest of the team wanted in. "Can you add checks for the marketing tables?" "What about the finance data?" "Can we monitor the customer churn model inputs?"

Reasonable requests. So we kept adding checks. Our ten-table script grew to 25 tables, then 40. The CSV email turned into a Google Sheet because the CSV was getting hard to read. Someone added conditional formatting. Someone else added a tab for "known issues."

At peak spreadsheet, we had:

• 63 individual SQL checks
• 4 different cron jobs running at different times
• A 12-tab Google Sheet with color-coded status indicators
• A Slack channel with so many automated messages that people muted it

We were spending roughly 45 minutes every morning just reviewing the results and deciding what needed attention. That's before actually fixing anything.

Phase 3: Things Start Breaking

The problems crept in slowly, then all at once.

The false alarm fatigue. Some checks were too sensitive. Row counts fluctuate naturally — weekends are slower, month-end is busier. We'd get flagged for "volume anomaly" when it was just a normal Saturday. After a while, people stopped paying attention to the alerts. Which meant they also missed the real ones.

The maintenance overhead. Every time a table schema changed, we had to update the checks. New column? Update the null check. Renamed table? Update three different scripts. Someone migrated a pipeline to a new schema? Time to rewrite everything. We were spending more time maintaining the monitoring than doing actual analysis.

The knowledge gaps. Only two people on the team really understood how the checks worked. When one of them went on vacation, we basically crossed our fingers and hoped nothing broke. (It did. Twice.)

The silent failures. This was the worst one. A cron job stopped running because the server it was on got restarted during maintenance. Nobody noticed for five days. Five days of unchecked data flowing into dashboards. When we finally caught it, we had to do a painful audit to figure out if anything had actually gone wrong during that window.

Spoiler: it had.

What We Learned (The Hard Way)

Looking back, the manual approach taught us a lot — mostly about what not to do, but also about what actually matters when monitoring data quality.

Lesson 1: Alerts Need Context

A check that just says "FAIL" isn't helpful. You need to know why it failed, how bad it is, and what the trend looks like. Is this the first time this check failed, or has it been flaky for weeks? Is the row count 5% below normal or 95% below normal? Context turns noise into signal.

Lesson 2: You Need History

Point-in-time checks are useful, but historical trends are way more valuable. If you can see that data freshness has been slowly degrading over the past two weeks, you can fix the root cause before it becomes an outage. Without history, every failure looks like a surprise.

Lesson 3: Ownership Matters

When everyone is responsible for data quality, nobody is. We learned to assign specific checks to specific people. "This table is your responsibility" is much more effective than "someone should probably look at this."

Lesson 4: The Tool Has to Be Simpler Than the Problem

Our monitoring setup became a system that itself needed monitoring. That's a sign you've gone wrong somewhere. The whole point of data quality checks is to reduce your workload, not add to it. If your monitoring solution requires significant ongoing maintenance, it's going to get abandoned eventually.

Lesson 5: Catch Issues in Minutes, Not Days

The difference between catching a pipeline failure at 6:05am and catching it at 2pm is enormous. At 6:05am, you fix it before anyone notices. At 2pm, you're doing damage control. Speed matters more than comprehensiveness.

From Scripts to Product

After about six months of the manual approach, we sat down and asked ourselves: what would we build if we could start over?

The answer was surprisingly focused:

1. Connect to the database directly. No more exporting CSVs or piping results through five different tools.
2. Schedule checks that actually run reliably. Not cron jobs on someone's laptop.
3. Track history automatically. Every check result stored, trends visible at a glance.
4. Alert through channels people actually check. Email and Slack, at minimum.
5. Make it easy to add new checks. If it takes more than two minutes to set up a new check, people won't do it.

That's what eventually became DashGuard. We built it because we needed it, and we figured other data teams were probably going through the same thing.

If You're Still Doing This Manually

No judgment — we did it for months. But if you're starting to feel the pain of manual monitoring, here are a few things you can do right now:

Start with your top 5 tables. Don't try to monitor everything at once. Pick the five tables that would cause the most damage if they broke, and focus there.

Check freshness first. It's the simplest check and catches the most common issue (stale data). Just compare the latest timestamp in the table against the current time.

Set reasonable thresholds. Not every fluctuation is a problem. Build in some tolerance for normal variation, especially on volume checks.

Actually look at the results. This sounds obvious, but the moment you stop reviewing your checks daily is the moment they become useless.

Or, if you'd rather skip the manual phase entirely, DashGuard does all of this out of the box. Free tier gets you started with one database and one check — enough to see if automated monitoring is worth it for your team.

Either way, the goal is the same: know about data problems before your stakeholders do. How you get there is up to you.