For data that has flat plateaus and canyons, like this cityscape. Derived from a CC-BY-2.0 image by abdallah.

I recently had to transform a set of data from typical date & measurement to a more compact date range & measurement format. The data in this case was very static: as the date incremented, the measurement was much more likely to be remain the same than it was to change. So storing the starting date and ending date for each measurement takes less space than storing each date’s measurement separately. Sure, it makes some subsequent  queries more convoluted, but let’s say that you found this post because you also need a similar transformation.

I stumbled at first by subconsciously assuming that a measurement would not repeat once its range was ended. This assumption works if your measurements never decrease (or if they never increase), say for the total number of copies of a book printed. They’re printed in batches, and most days no new copies are printed. If your data does meet that criterion, the query is simple (and should be portable from Microsoft Transact-SQL, where I wrote it):

SELECT d.[group],
       MIN(d.[date]) AS [start_date],
       MAX(d.[date]) AS [end_date],
       d.[measurement]
   FROM mydata d
   GROUP BY d.[group], d.[measurement]
   ORDER BY d.[group], MIN(d.[date])

So for data like this:

this generates the desired output:

Unfortunately, my data did not meet this criterion, and my results from that query had overlapping ranges, which was quite incorrect:

yields:

I solved it by counting off my start dates and end dates, and then for each group and measurement placing the first start date with the first end date, the second with the second, and so on. This still seems like overkill; if you know an optimization I missed, please add your comment below!

In Microsoft’s Transact SQL, counting off like that involves the OVER clause and CASE expressions against some LEFT JOINs. I LEFT JOIN the table against itself twice, once on the previous date and once on the subsequent date. Finding the NULLs on those joins (in the OVER + CASE constructs) allows me to count the starts and ends of each block of measurements. I also need a future date to move all of the “middle” dates out of order to the end—and they get thrown out by the WHERE COALESCE([start_date], [end_date]) IS NOT NULL part later.

/* any date well after all of the dates in the database will do */
DECLARE @futuredate DATE = '2100-01-01';

SELECT t.[group],
       MIN(t.[start_date]) AS [start_date],
       MIN(t.[end_date]) AS [end_date],
       t.[measurement]
   FROM (SELECT d.[group],
                CASE 
                   WHEN d2.[date] IS NULL THEN 
                      ROW_NUMBER()
                         OVER (PARTITION BY d.[group]
                               ORDER BY CASE
                                           WHEN d2.[date] IS NULL THEN d.[date]
                                           ELSE @futuredate
                                        END)
                   ELSE NULL
                END AS start_seq,
                CASE 
                   WHEN d3.[date] IS NULL THEN 
                      ROW_NUMBER()
                         OVER (PARTITION BY d.[group]
                               ORDER BY CASE
                                           WHEN d3.[date] IS NULL THEN d.[date]
                                           ELSE @futuredate
                                        END)
                   ELSE NULL
                END AS end_seq,
                CASE
                   WHEN d2.[date] IS NULL THEN d.[date]
                   ELSE NULL
                END AS [start_date],
                CASE
                   WHEN d3.[date] IS NULL THEN d.[date]
                   ELSE NULL
                END AS [end_date],
                d.[measurement]

            FROM mydata d
               LEFT JOIN mydata d2 ON d2.[group] = d.[group]
                                      AND d2.[date] = DATEADD(DD, -1, d.[date])
                                      AND d2.[measurement] = d.[measurement]
               LEFT JOIN mydata d3 ON d3.[group] = d.[group]
                                      AND d3.[date] = DATEADD(DD, 1, d.[date])
                                      AND d3.[measurement] = d.[measurement]

        ) t
   WHERE COALESCE([start_date], [end_date]) IS NOT NULL
   GROUP BY [group], COALESCE([start_seq], [end_seq]), [measurement]
   ORDER BY [group], [start_date]

which gives me the correct results:

I’m still making an assumption, that a measurement was stored in the data for every date. If the measurements are not taken that rigorously, you will need to accommodate that gap with a generated table of the appropriate dates to join against. If you’re up against that, let me know and I’ll do a follow-up post.

—jhunterj

decode(expression,
       search, result
       [, search , result]...
       [, default]
      )

seems functionally equivalent to

CASE expression 
   WHEN search THEN result
   [WHEN search THEN result]...
   [ELSE default]
END

By Steven Straiton (originally posted to Flickr as Gorilla) [CC-BY-2.0], via Wikimedia Commons

• has no limitation of “only” 255 total expression + search + result + default parameters
• is easier to read
• works in PL/SQL context
• is ANSI-compliant

The only “benefit” to decode I could find is that the decode function will attempt to convert all of the results to the type of the first result, while CASE just errors if you mix types. To me, that benefit would simply encourage sloppy coding and keep you from noticing buggy code as quickly.

So, what’s the point? Are their coders out there who “get” the function syntax more readily than the CASE syntax, so it helps their coding efficiency? Or is there a benefit I’m missing?

—jhunterj

It’s time to let the data tell you when it needs attention.

Get your data into a format (through a table, a view, or a subquery; we’ll say you have it in measurements) where you have records that have the thing you’re measuring (we’ll call it aspectID), when you measured it (we’ll use date, although this will work for monthly measurements, weekly, quarterly, yearly, hourly, or anything else), and what its measurement was then (value).

Linear regression

What management calls a trend, statistics calls a slope. If your data happens to fall in a line, the slope is how much is goes up (or down) each unit of time. −2 dollars per day. +100 page views per hour. +5 accounts per month. Probably your data doesn’t line up nicely for you though.

Some flavors of SQL will make this pretty easy to get, through linear regression functions. Oracle PL/SQL, for instance:

SELECT aspectID,
       REGR_SLOPE(value, TO_NUMBER(TO_CHAR(date, 'J')) AS slope
   FROM measurements;

while it’s a little less obvious in flavors without those built-in functions, such as Microsoft T‑SQL:

SELECT aspectID,
       ( (COUNT(1) * SUM(CAST([date] AS FLOAT) * [value])) -
         (SUM(CAST([date] AS FLOAT)) * SUM([value]))
       ) /
       ( (COUNT(1) * SUM(POWER(CAST([date] AS FLOAT), 2))) -
         (POWER(SUM(CAST([date] AS FLOAT)), 2))
       ) AS slope
   FROM measurements
   WHERE [value] IS NOT NULL
   GROUP BY aspectID;

Some additional things you may want to do:

• restrict (in the WHERE clause) the data to the recent past (30 days or 90 days for dailies, perhaps, or 72 hours for hourlies, etc.)
• scale the slopes (something that’s going from 1 to 100 is probably more interesting than something that’s going from 50,001 to 50,100), by dividing them by the MAX(value) for the period. This will give you a more useful sorting column than the “raw” slope, for some problem spaces.

The other thing you’ll often want to know is when is the current value “spiking” (up or down). A brief spike may not have a big impact on the linear regression, but it’s still a point of interest. Oracle PL/SQL first again:

SELECT aspectID,
       CASE
          WHEN STDDEV(value) = 0 THEN 0
          ELSE (MAX(value) KEEP (DENSE_RANK LAST ORDER BY date)
                   -  AVG(value)) /
               STDDEV(value)
       END AS spike
   FROM measurements
   GROUP BY aspectID;

“Spike” here is the number of standard deviations above (or below) average the most recent data point is. I like to look for instances that are more than two standard deviations from the current average (which should mean I’m getting hits 5% of the time for each measurement, or hits on 5% of the metrics with any given run) by adding

WHERE CASE
         WHEN STDDEV(value) = 0 THEN 0
         ELSE ABS(MAX(value) KEEP (DENSE_RANK LAST ORDER BY date)
                  -  AVG(value)) /
              STDDEV(value)
      END >= 2

(Note the addition of ABS there—you want sudden drops as well as sudden increases.) If you put that into a view, your WHERE clause will be a little prettier. Restrict the time scale as needed, just like with slopes.

Getting to the most recent value as well as aggregations of the values in Microsoft T-SQL is a little more cumbersome, but I don’t find either particularly more readable than the other:

SELECT m.aspectID,
       CASE
          WHEN STDEV(m.value) = 0 THEN 0
          ELSE (lastm.value - AVG(m.value)) / STDEV(m.value)
       END AS spike
   FROM measurements m
      JOIN (SELECT aspectID,
                   [value],
                   ROW_NUMBER()
                      OVER (PARTITION BY aspectID
                            ORDER BY MAX([date]) DESC) AS rank
               FROM #measurements
               GROUP BY aspectID,
                        [value]
           ) lastm ON lastm.aspectID = m.aspectID
   WHERE m.[value] IS NOT NULL
      AND lastm.rank = 1
   GROUP BY m.aspectID,
            lastm.value;

And those are the base tools. Get them into a view or schedule a stored procedure to run periodically and email you the metrics that trip your thresholds. And please let me know if you have better (more readable) approaches in these or other SQL flavors, or if you’ve put these kinds of measurements to interesting uses.

—jhunterj

The tech involved in this problem, although probably you know this if you reached this page: SSRS is a database reporting solution for Microsoft SQL Server—you can turn data into graphs! Mobi Reports Pro is an iOS application for view SSRS reports on your iPad or iPhone—you can see your graphs and check your reports on the go!

The long problem description and solution explanation:

Mobi Reports has some caveats and can be a bit ticklish with some SSRS features, though. The feature that caused me the most headaches is Drillthrough reports.

SSRS gives you several ways to keep from burying your report user: Drillthrough, Drilldown, Subreports, and Nested Data Regions. Drillthroughs are a simple way to allow the user to click on one element of one report and load another report—drilling through the first, hopefully down to a second that makes sense and expands on the part they clicked on. (See the SQL Server TechCenter article Drillthrough, Drilldown, Subreports, and Nested Data Regions (Report Builder 3.0 and SSRS) for full details.)

Mobi Reports supports Drillthroughs, as long as you keep all the reports in the same directory, or use SSRS Linked Reports to put a link to a remote report in the same directory as the base report. (See Mobi Weave Support Center’s Does Mobi Reports support Drill down and Drill through report navigation?)

So far so good. I found that you also had to pass all parameters all the time (you cannot omit some parameters some of the time from SSRS’s Actions properties), and passing NULL (or Nothing! in SSRS terms) as a parameter value seemed to cause problems too:

Mobi error message for omitted parameter.

I have no clever workaround here. Just don’t omit any parameters, and don’t try to pass NULL values (pass 'DUMMY' or −99 or something else that won’t show up in real data, and add code on the Drillthrough report to turn it back into NULL).

The real problem happened when trying to launch a Top-Level Report, then drilling through to a remote 2nd-Level Report, and then drilling through again to a remote 3rd-Level report. Creating linked reports in every directory to all the remote reports won’t help; you’ll still see this:

Mobi error message for 3rd-Level Linked Drillthrough

The problem’s source is is the handling of those Linked Reports. Yes, the Linked Report is in the same directory as the base report. Mobi Reports will dutifully load it. But go back and load it in Report Manager on a desktop web browser again, and drill through there. Notice what happens in the Report Server Toolbar as each report loads. Even though you’re using Linked Reports, all in the same directory, Report Manager recognizes when you’ve gone afield. This is why Mobi Reports complains when you try to drill through a 2nd-level report into a 3rd-level report.

You can keep Report Manager and Mobi Reports from straying out of the base directory by giving a full path to each drillthrough report in the Action properties. At first I simply passed the base directory as a parameter all the way down the line, but there’s an easier way: the built-in global Globals!ReportFolder. Everywhere you want to set a drillthrough, follow these steps:

• Properties → Action
• Enable as an action: Go to report
• Specify a report: [ƒ_x]
• Set expression for: Action.Drillthrough.ReportName = Globals!ReportFolder + "/LinkedReportName"

(replacing the “/LinkedReportName” with the name of your linked report, of course). That’s it! ReportManager will report that you never leave the starting directory, and Mobi Reports will happily load drillthrough after drillthrough.

—jhunterj