Learning Analytics: The Metrics That Matter and the Ones That Mislead

This is engineering guidance, not legal advice. Confirm specifics with qualified counsel.

Why this matters

If you run learning and development, found an EdTech product, or own a training platform's roadmap, the analytics you choose decide what you can prove and what you can fix. Pick vanity metrics and you will present a beautiful "engagement up 40%" slide while completion quietly sits at 41% and one broken module sheds half your learners. Pick the metrics that matter and the same data tells you exactly which module to re-cut and whether anyone actually learned anything. This article gives you the judgment to tell the two apart, the plain-language map of the system that generates the numbers, and enough standards grounding to brief an engineer without being misled by a vendor's dashboard demo. It is the opening piece of this course's measurement block; the deep dives on each metric follow in the articles it links to.

What learning analytics actually is

Start with the plain version. Learning analytics is the work of collecting the data a learning product generates and turning it into decisions that make learning better. That is the whole job: data in, better decisions out. Everything else — the standards, the storage, the dashboards — is plumbing in service of that loop.

The field has a formal definition, and it is worth reading because it changed in a way that matters for this article. The Society for Learning Analytics Research, the academic body that has organized the field since its first conference in 2011, originally defined learning analytics in 2011 as "the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning" [1]. In 2025, after fifteen years, the same body revised it to "the collection, analysis, interpretation and communication of data about learners and their learning that provides theoretically relevant and actionable insights to enhance learning and teaching" [1].

Notice what got added: actionable insights. The discipline spent its first decade learning that data and dashboards are not the point — acting on them is — and wrote that lesson into its own definition. That single edit is the thesis of this article. A metric earns its place only if someone can act on it.

It helps to separate two words people use as if they were the same. Reporting is showing what happened: 18,400 video plays last month. Analytics is interpreting what happened well enough to decide what to do next: plays are up but completion fell, because Module 3 loses 60% of viewers before the halfway mark, so re-cut Module 3. Reporting is a rear-view mirror; analytics is the steering wheel. Most "analytics dashboards" sold to training teams are really reporting dashboards wearing a nicer coat.

The trap: vanity metrics

The phrase vanity metric comes from startup measurement — a number that reliably goes up, looks impressive in a presentation, and changes no decision you make [2]. Its opposite is an actionable metric: one that ties a cause to an outcome and answers "what should we change, and did the change work?" [2]. Learning products are unusually prone to vanity metrics because the easy-to-collect numbers and the meaningful numbers are almost never the same.

Here is the gallery of misleading metrics, why each is seductive, and what it actually proves.

Total video views or plays. The number is large and always grows, which feels like momentum. It conflates one learner watching ten times with ten learners watching once, and it counts a play that was muted in a background tab. It proves the player loaded, not that anyone learned.

Logins and time-on-platform. A high "average 47 minutes per session" reads like deep engagement. Time is not learning — a learner stuck, lost, or idle with the tab open produces the same minutes as one who is absorbed. Activity is not achievement.

"Watched 100%." The most expensive confusion in the field: treating a video timeline reaching its end as a person finishing a course. The bytes played; the human may have left the room. We pull apart the five different things "complete" can mean in Learning Metrics 101; the short version is that raw playback is the weakest of the five.

Enrollments and registrations. Easy to grow with a marketing push, and they say nothing about outcomes. In open courses the gap is stark: large studies of massive open online courses (MOOCs) find that around 52% of registrants never even start the content, and completion runs roughly 5–15% of those who registered [3]. A registration is a hope, not a result.

Average score. A single "81% average" hides the shape of the data. It can mean everyone scored near 81, or that two-thirds aced it while a cluster failed one module — two situations needing opposite responses. An average without its distribution is a number that has been rounded until it stopped being useful.

Satisfaction and Net Promoter Score. The end-of-course "did you like it?" survey — known in training circles as the "smile sheet" — measures reaction, not learning. In the widely used Kirkpatrick model of training evaluation, that is Level 1 of four; learning is Level 2, on-the-job behavior change is Level 3, and business results are Level 4 [4]. Liking a course and learning from it are weakly correlated at best.

Figure 1. The same learning activity, measured two ways. The left column flatters; the right column drives a decision. Each misleading metric has an actionable counterpart that answers "so what?"

None of these numbers are evil. The failure is reporting them as if they were outcomes. Total views is a fine operational metric for capacity planning; it is a terrible metric for deciding whether your course works. The discipline is knowing which question each number can honestly answer.

The metrics that matter: the actionability test

A metric matters when it earns a decision. Four questions tell you whether it does. Run any number you are about to put on a dashboard through them.

1. Is it tied to a decision? Name the action someone takes when this number moves. "If first-attempt pass on Module 3 drops below 50%, we re-cut Module 3." If no sentence like that exists, the metric is decoration. This is the single most important filter, and most vanity metrics fail it instantly: there is no action you take when total views rises.

2. Can it be segmented? A single platform-wide number is almost never actionable, because you cannot fix "the platform." The same metric broken down by cohort, by module, by week, or by learner segment points at where the problem is. "Completion is 41%" is a headline; "completion is 41% because the Sales cohort stalls at Module 3" is a work order.

3. Does it pair a leading signal with a lagging outcome? Leading indicators — engagement, progress — move before the result and let you intervene; lagging indicators — completion, mastery — confirm what happened. (We define all four in Learning Metrics 101.) A dashboard of only lagging metrics tells you that you failed after the cohort has gone home. A dashboard of only leading metrics feels busy but never proves a result. Actionable analytics pairs them: watch-time drop predicts the completion dip you can still prevent.

4. Is it reproducible and defensible? If two analysts compute "completion" and get different numbers because the rule is fuzzy, the metric cannot survive a regulator, an auditor, or a board meeting. A metric that matters has a written definition — the exact rule, the exact denominator — that anyone can apply and get the same answer.

There is a useful ladder behind these questions, borrowed from data analytics generally and named explicitly in the learning-analytics field: descriptive, diagnostic, predictive, prescriptive [1]. Descriptive analytics says what happened (completion was 41%). Diagnostic says why (Module 3 drop-off). Predictive says what will happen (this cohort is tracking toward 38% unless we act). Prescriptive says what to do (re-cut Module 3, add a checkpoint). Vanity metrics are stuck on the bottom rung — they describe, weakly, and never climb. The metrics that matter are built to climb toward the top, where a number turns into an action.

Figure 2. The analytics ladder applied to learning. Value rises as you climb from "what happened" to "what to do." Vanity metrics never leave the bottom step.

A worked example: one cohort, two stories

Numbers make this concrete. Take a corporate course — five video modules plus a final assessment — rolled out to 2,000 employees over one month. The vendor dashboard greets you with four big numbers.

• 18,400 video plays.
• 142,000 minutes watched.
• 81% average quiz score.
• 1,250 completions.

Read as a vanity dashboard, this is a triumph. Read with the actionability test, the same data tells a different and more useful story.

Plays per learner. 18,400 plays ÷ 2,000 learners = 9.2 plays each — but the course has five modules, so 9.2 includes replays and re-opens. Unique starters were 1,500, meaning 500 enrolled employees (25%) never pressed play once. The "18,400" hid a quarter of the audience doing nothing.

Real completion. The dashboard's "1,250 completions" counts the video timeline reaching its end. Apply a defensible rule instead — all five modules watched to threshold and final score at least 80% — and only 820 learners qualify. Completion rate = 820 ÷ 2,000 = 41%, not the 62.5% the raw count implied. (Why "complete" is so slippery, and how to move the real number, is the subject of Completion Rate: Defining, Measuring, and Improving It.)

The average that lied. The 81% average score had a median of 88% — most people did well. But a cluster of 320 learners scored between 50% and 65%, and almost all of their low marks were on one assessment: Module 3, where first-attempt pass was just 38%. The average smeared a sharp, local failure into a comfortable global number.

The drop-off that explained it. Per-second watch data showed the Module 3 video — a single 14-minute lecture — losing roughly 60% of viewers before the seven-minute mark. The diagnostic and the assessment agreed: Module 3 is where learning breaks.

The vanity reading was "engagement is strong, scores are high, we are done." The actionable reading produced a work order: re-cut Module 3 into three sub-7-minute segments with a checkpoint after each, then re-measure first-attempt pass and drop-off on the next cohort. Same data, opposite conclusions. The difference was entirely in which metrics were trusted. (The watch-time and drop-off side of this lives in Video Engagement: Watch-Time, Drop-Off, and Re-Watch.)

A closing note on big numbers themselves: the global learning-analytics market is often quoted at figures that range, depending on who is counting and what they include, from a few billion dollars to over thirty billion for 2026, with one widely cited estimate putting the broader education-and-learning-analytics market near $35.5 billion and growing about 20% a year [5]. That spread of estimates is itself a vanity-metric lesson. A number large enough to impress and too loosely defined to act on is exactly the kind of metric this article warns against — cite it for context, never for a decision.

Where the numbers come from: the analytics stack

A trustworthy metric depends on a clean path from the learner's screen to your dashboard. Understanding that path in plain language is what lets you tell an engineer what to build and a vendor when their demo is hand-waving. Five stages, left to right.

Stage 1 — the player emits events. Every meaningful thing a learner does should fire an event: the video started, paused, was scrubbed back ten seconds, reached a checkpoint, the quiz was answered. If the player does not emit an event, that fact is gone forever — you cannot analyze what was never recorded. This is why the cheapest moment to decide what to track is before the player is built.

Stage 2 — a standard carries the events. Raw events need a shared language so any system can read them. Three standards dominate. SCORM (Sharable Content Object Reference Model) is the old, universal packaging standard; it records a fixed set — completion status, success status, score, time — inside a single learning-management-system launch, and treats the course as a black box [6]. xAPI (the Experience API) records open-ended "statements" shaped like sentences: actor – verb – object, as in "Maria completed Module 3" [7]. Its companion xAPI Video Profile defines the exact verbs a player should emit — initialized, played, paused, seeked, completed — plus result data including a progress value and played-segments, the list of which intervals were actually watched [8]. Caliper Analytics, from the standards body 1EdTech, defines a comparable MediaEvent for video interactions stamped with the current playback time, and is built so institutions can "collect learning data from digital resources to understand and visualise" it [9]. xAPI and Caliper are not interchangeable; the standards community itself frames the choice as "horses for courses," not one-or-the-other [9].

Stage 3 — a store receives them. xAPI statements are written into a Learning Record Store (LRS) — think of it as the notebook those sentences are recorded in. The LRS is a defined system with a web interface for receiving, storing, and returning statements; that interface was standardized as IEEE 9274.1.1-2023, the ratified form of xAPI [10]. SCORM data, by contrast, lands inside the LMS that launched the course. The store is where "an event happened" becomes "a fact we can query."

Stage 4 — a warehouse joins the data. A learning record on its own answers learning questions. The business question — did training move a sales number, a safety incident rate, a support-ticket volume — needs learning data sitting next to HR, CRM, or operations data. That join happens in a data warehouse, where LRS statements are combined with the rest of the organization's data so trends across systems become visible. This is the stage where Level 3 and 4 outcomes from the Kirkpatrick model become measurable at all [4].

Stage 5 — a dashboard turns facts into a decision. The final stage is where a human reads the result and acts. The recurring failure here has a name in the research: a dashboard can raise awareness without changing behavior, the "last mile" where analytics ends at insight instead of action [11]. A dashboard that matters does not stop at a chart; it surfaces the segment, the likely cause, and the recommended action.

Figure 3. The learning-analytics pipeline. The richness of every metric is capped at Stage 1 — a metric can be no better than the event the player emitted. Tracking-bearing stages are tinted.

The single most important consequence of this map: your metrics are capped by your weakest upstream stage. If the player emits only "completed," no warehouse or dashboard can reconstruct a drop-off curve. If you chose plain SCORM, per-second video analytics are off the table regardless of how good your reporting tool is. The table below shows what each standard can and cannot carry — read the standards-support columns before you promise a stakeholder a heatmap.

The reading is blunt: if your roadmap promises engagement heatmaps or per-second video insight, plain SCORM will not deliver them — you need the xAPI Video Profile or Caliper, and the player must emit those events from day one. We go deeper on the video-specific tracking in Tracking Video with xAPI, and on building the whole pipeline — schema, sampling, storage — in Building the Learning-Analytics Pipeline.

The metrics that matter, by who is asking

The same data serves three audiences, and each needs a different cut of it — which is why "one dashboard for everyone" is itself a mild vanity trap. A learner needs to see progress, current mastery, and the single next step — feedback that helps them self-correct. An instructor or instructional designer needs cohort health, per-module drop-off, and item-level question difficulty — the signals that say which content to fix. The business needs completion for compliance evidence, mastery as proof of capability, and, where the budget justifies the cost of joining systems, the behavior-change and results numbers that show training moved an outcome. The principle for all three is the same: report the metric tied to that reader's decision, and nothing else. Designing those reports so people actually use them is its own discipline, covered in Reporting to Stakeholders.

Common mistakes

E-learning measurement is a minefield, and the same few mistakes recur across almost every team.

The defining error is reporting the metric you can easily get instead of the one that matters — shipping total views because the player gives it away free, while the drop-off curve that would actually improve the course goes uncollected.

A close second is the dashboard that visualizes but never recommends. Teams buy a reporting tool, fill a screen with charts, and discover months later that nothing changed, because no chart ever named an action. The research calls this the gap between awareness and behavior; practitioners call it "lots of dashboards, not many decisions" [11].

Third is trusting the average and ignoring the distribution — the 81% that hid a failing Module 3. Always ask for the shape, not just the center.

Fourth is collecting nothing until launch. You cannot backfill events you never recorded; the data you skip this quarter is gone permanently. Storage is cheap, missing history is priceless, so instrument the player to emit rich events from the first release even before you have a dashboard to read them.

Fifth, and the reason for the disclaimer at the top of this article, is treating learner-data privacy as an afterthought. Learner records are personal data. In the European Union they fall under the General Data Protection Regulation (GDPR, Regulation (EU) 2016/679); US student records fall under FERPA; biometric signals, if you ever capture them, invoke laws like Illinois's BIPA. The engineering principle that keeps you out of trouble is data minimization: collect the least learner data that answers a real question, and decide retention before you collect. The privacy and legal landscape is covered properly in Proctoring Data, Privacy, and the Legal Landscape — treat that as required reading before you wire learner data into a warehouse.

Where Fora Soft fits in

Fora Soft has built video streaming, real-time WebRTC, and interactive-player software since 2005, and in e-learning the analytics work is rarely the chart at the end — it is wiring the player so the right events flow into the right standard so the metric you need can exist at all. The build-vs-buy trade-off is usually this: an off-the-shelf LMS hands you canned, mostly descriptive dashboards for free, but the moment you need diagnostic or prescriptive analytics on video — drop-off heatmaps, custom mastery rules, learning data joined to business outcomes — you are building an interactive player plus an xAPI or Caliper pipeline and a warehouse on top. We help teams decide which metrics genuinely justify that custom layer, then build the tracking so the analytics become possible. The same pattern shows up across the real-time and streaming work we do in conferencing, telemedicine, and OTT.

What to read next

• Learning Metrics 101: Completion, Progress, Mastery, Engagement
• Completion Rate: Defining, Measuring, and Improving It
• Building the Learning-Analytics Pipeline

Call to action

• Talk to a e-learning engineer — book a 30-minute scoping call to talk through your learning analytics plan.
• See our case studies — 250+ shipped projects across video streaming, WebRTC, OTT, telemedicine, e-learning, surveillance, and AR/VR.
• Download the Learning Analytics Metric Audit — A one-page worksheet that runs every metric on your dashboard through the four-question actionability test, with vanity metrics to demote and actionable metrics to promote.

References

1. Society for Learning Analytics Research (SoLAR). What is Learning Analytics? (2025 definition; original LAK 2011 definition; descriptive/diagnostic/predictive/prescriptive methodologies). https://www.solaresearch.org/about/what-is-learning-analytics/ — Tier 5 (field-defining institutional source). Field originated at the first LAK conference, 2011; 2025 definition adds "actionable insights."
2. Ries, E. (2011). The Lean Startup — vanity metrics vs actionable metrics. https://theleanstartup.com/ — Tier 6 (foundational concept). A vanity metric rises and flatters but drives no decision; an actionable metric links cause to outcome.
3. Reich, J., & Ruipérez-Valiente, J. A. (2019). The MOOC Pivot. Science, 363(6423). https://www.science.org/doi/10.1126/science.aav7958 — Tier 5 (peer-reviewed). ~52% of MOOC registrants never start; completion ~5–15% of registrants.
4. Kirkpatrick, D. L. The Kirkpatrick Model: Four Levels of Training Evaluation (Reaction, Learning, Behavior, Results). Kirkpatrick Partners. https://www.kirkpatrickpartners.com/the-kirkpatrick-model/ — Tier 5 (institutional/foundational). Satisfaction is Level 1; behavior and business results are Levels 3–4.
5. The Business Research Company. Education And Learning Analytics Global Market Report 2026. https://www.thebusinessresearchcompany.com/report/education-and-learning-analytics-global-market-report — Tier 7 (market-research estimate; figures vary widely by scope). Cited only to illustrate the wide spread of headline market figures.
6. ADL Initiative. SCORM 2004 4th Edition — Run-Time Environment (RTE). https://adlnet.gov/projects/scorm/ — Tier 1 (primary standard). Fixed data model: cmi.completion_status, cmi.success_status, score, time; course tracked as a black box.
7. ADL Initiative. Experience API (xAPI) Specification v1.0.3 — Part 2: Statements. https://github.com/adlnet/xAPI-Spec — Tier 1 (primary standard). Actor-verb-object statement model and the Learning Record Store interface.
8. ADL / xAPI Video Community Profile. xAPI Video Profile v1.0.3. https://github.com/adlnet/xapi-authored-profiles/tree/master/video — Tier 1 (primary profile). Verbs initialized/played/paused/seeked/completed; result extensions progress and played-segments.
9. 1EdTech (formerly IMS Global). Caliper Analytics® — Media Profile / MediaEvent, and Initial xAPI/Caliper Comparison. https://www.1edtech.org/standards/caliper — Tier 1 (primary standard). MediaEvent with playback time; xAPI and Caliper are complementary, "horses for courses."
10. IEEE. 9274.1.1-2023 — Standard for Learning Technology: JSON Data Model Format and RESTful Web Service for Learner Experience Data Tracking and Access (xAPI base standard; defines the LRS). https://standards.ieee.org/ieee/9274.1.1/7321/ — Tier 1 (primary standard). The ratified IEEE form of xAPI 1.0.3.
11. Jivet, I., Scheffel, M., Drachsler, H., & Specht, M. (2017). Awareness Is Not Enough: Pitfalls of Learning Analytics Dashboards in the Educational Practice. EC-TEL 2017. https://link.springer.com/chapter/10.1007/978-3-319-66610-5_7 — Tier 5 (peer-reviewed). Dashboards can raise awareness without changing behavior — the analytics "last mile."

Where sources disagreed, the official specifications were followed. Many vendor articles describe "SCORM tracks everything"; this article follows SCORM 2004's fixed, black-box data model [6] and notes that per-second video data requires the xAPI Video Profile [8] or Caliper [9]. Market-size figures [5] are presented as a deliberately wide, low-confidence range rather than a single authoritative number, because market-research estimates for this category differ by an order of magnitude depending on scope.