Buying Speed? What Canadians Pay for Broadband: Part 1 – The CRTC’s “Measuring Broadband Canada” report does not measure up

How fast is your Internet connection, really? How good is it, anyway? How can you tell?

The Canadian Radio-television and Telecommunications Commission (CRTC) is rightly interested in this question.  So the CRTC contracted with SamKnows, a “global internet measurement and analysis platform”, to collect data in October 2019 on the performance of broadband Internet services sold to Canadian consumers. The results were published in a report, “Measuring Broadband Canada,” released in June 2020, at the tail of the “first wave” of COVID-19 in Canada. The outcome according to the CRTC?:

“Canadian consumers are receiving maximum advertised Internet speeds”.  PIAC was suspicious.

The data were collected with “Whiteboxes”, which are hardware installed between a user’s device and their home modem or router to monitor broadband performance when no one in the home is using the Internet. You heard that right.  When no one is using the Internet in the home.

Another important limitation of these Whiteboxes: measurements are only taken from the service provider’s location to the Whitebox, not within the user’s home network. You heard that one right too.  Not accounting for your network setup, devices, or anything that uses or potentially slows down the Internet speed during customers actually “using” the Internet in a normal way.

Let’s give them the limited and “perfect” conditions, however.  Let’s examine what they measured: the “performance indicators”. Those measured were: download and upload speeds; latency; packet loss; and webpage loading time.  These are limited, but useful, indicators.  However, other parameters could have been included – ones like “jitter”: a/k/a “packet delay variation”, (where variation in IP packet arrival at nodes in the Internet can cause packet loss and dropouts and interruptions, especially for a user’s voice and videocall applications – which are essential during the pandemic, whew!).  Oh well.

The test results purported to show that all major Canadian ISPs are providing users with speeds meeting or exceeding the advertised speed, apparently to the point that users often were getting “additional” throughput, with very few instances of service falling below advertised speeds, for all performance indicators.  Wow, this seems great.

The Report also claimed that speeds also did not decrease significantly during peak hours. Really?

Now we are suspiciouser.

We believe a closer examination of this claim reveals that, for a Report that claims to be “designed to provide accurate data on the broadband performance experienced by the majority of Canadian fixed-line broadband users,” the study is actually extremely limited in scope, and the conclusions drawn from the results are tone-deaf to the real-world usage context of Canadians. Perhaps this was made easier by the significant scaling down of the sample size and diversity of the measured connections, compared to a similar SamKnows study conducted in 2016.  (This creates a risk of drawing conclusions from small sample sizes, or in short: the human cognitive bias to give too much credence to statistically insignificant  results, called by behavioural economics scientist Daniel Kahneman the “law of small numbers”.)

However, most of the conclusions in this Report appear to rely upon what was chosen to be studied, and not on what was deliberately excluded, and that these scope reduction choices made by the Report authors were justified on technical bases but not on social or actual real-world use bases – the real world being the point of studying consumers’ experiences of their broadband service (and, we might add, the authors’ choices and limitations were implicitly endorsed by the CRTC, which uncritically announced the Report’s results with an industry-boosting News Release).  We examine these critical limitations, and the sweeping conclusions reached, below.

The sample pool is heavily skewed towards higher tier plans and urban users

The first major limitation is in the service packages and demographics that the 2020 Report chooses to include, or rather, to exclude. The results were based on a pool of measurements from 2035 Whiteboxes deployed to customers of participating Internet Service Providers (ISPs), including the three largest ISPs: Bell; Rogers; and TELUS. Only Internet packages with the highest subscriber counts were included in the study in order to “represent the majority of Canadian fixed-line broadband users”. For comparison, the 2016 study used data from 3056 Whiteboxes, without the “highest subscriber count” condition.

The 2020 Report also excludes advertised download speeds of 10 Mbps or less, and packages that had less than 25 000 total subscribers. With few exceptions, sample sizes of less than 40 Whiteboxes per Internet package were excluded. The 10 Mbps cutoff is particularly concerning, as many rural Canadians only have these lower tier plans available to them. The study does not explain whether the exclusion of lower tier service packages was because of declining number of subscribers or otherwise. The exclusion is especially confounding considering the 2016 study did include Bell Canada’s 7/0.64 Mbps and 5/1 Mbps plans, and TELUS’ 6/1 Mbps plan, which respectively underperformed at 81%, 86%, and 81% of the advertised speeds. The CRTC’s choice to not re-evaluate these plans 3 years later calls into question whether the speed and quality of service has improved for Canadians still relying on these basic plans. 

The lack of evidence for lower-tier plans does a disservice to rural Canadians, who tend to only have access to lower speed broadband Internet. Based on the most recent Communications Monitoring Report (CMR), released by the CRTC earlier this year, the broadband coverage in rural communities in 2018 was only 40.8% for broadband speeds of 50/10 Mbps (31.3% in First Nations reserves), compared to 97.7% coverage for urban areas. 1.5 Mbps broadband was available to rural communities at a much higher coverage rate of 94.0%, and yet the SamKnows study does not address whether these users are getting reliable service at speeds that are already inadequate for modern usage needs even at the advertised benchmark.

Another major difference between the 2016 and 2020 Reports is that the 2016 Report explicitly took measurements that “covered all geographic regions of Canada in a mix of urban and rural settings,” and acknowledged variations in results stemming from rural and remote measurements. The 2020 Report makes no such claims – because it cannot – in fact, it does not mention a rural sample portion at all. We can only assume, based on how the data collection is skewed towards higher-tier services, that effectively only urban and suburban users were included in the study. Furthermore, the study excluded Northwestel from the results for webpage loading times because “their remote location would have an adverse impact on results compared to other ISPs.” This should raise the eyebrows of anyone familiar with selection bias. A fairer presentation would have been to include this information from Northwestel and then to explain those data’s effect on the bottom line number with an explanatory note.  In effect, the study cherry-picks results, limited to urban and suburban users, who typically enjoy greater reliability and more service choice than rural and remote users. 

Collecting data during periods of inactivity only measures speed, not user experience 

As we noted earlier, the “real-world” utility of Whitebox measurements is also limited by the fact that data are only collected when there is no end-user “cross-traffic” on the home network. In other words, the Whitebox only takes measurements when there is no one in the household actually using the Internet, apparently so that the “WhiteBox’s measurements are not “distorted” by end-user activity, and that the Whitebox’s measurement traffic does not interfere with the user’s experience of the Internet. Except, of course, the user’s actual household experience is always filtered by the fact that they must use their Internet connection, and some sort of consumer device, such as a smartphone, laptop, or connected TV to experience speed and to use the product, that is, the Internet.  Therefore the study only measures potentially available speeds on a household network, not how efficiently and reliably those broadband speeds stand up to normal and indeed, human, user activity. The study qualifies that the Whiteboxes only measure speeds to the “doorstep” (Whitebox) because factors like the number of devices in use at the same time, faulty equipment, and poor Wi-Fi connectivity can affect broadband performance inside the home. Well, duh.  We all live in the real world, with some “network overhead”: routers, WiFi, devices.  Why can the study not allocate and take into account a “typical” such level of overhead?

It is precisely the real-world factors that, together with the “to the door” delivered speed and quality, to “make or break” the utility of an Internet service for a household, especially during peak hours. Without more comprehensive research that accounts for these factors, or at the least some allowance for consumers to live in the real world, with a real network and real Internet devices, the study should recognize its results for what they are: merely the potential maximum speeds “available” to a household.

The 2020 Report, however, makes the very much larger, and, in the real world, at least confusing claim that “Canadian ISPs have mostly met or exceeded maximum advertised download and upload speeds across tiers and regions,” despite the Report’s partial and theoretical, rather than real-world, basis. But wait, there’s more: the report extrapolates that “quality of service is consistent across Canada,” and that the results were based on “the broadband performance experienced by the majority of Canadian fixed-line broadband users.” Firstly, the effective exclusions of rural areas by concentration on higher-tier packages completely undermines the assertion that service is consistently up to snuff across Canada. Secondly, the Report, by its own methodology explicitly excludes any “consumer experience” at all – since only the Whiteboxes’ “experience” is measured, not the experience of a real consumer on a real network using a real device – so any claim of “performance experienced by … Canadian … users” is manifestly false.  Lastly, the Report, despite the measurements being conducted prior to the COVID-19 pandemic (but released during it) now is of questionable utility in the real world context of the current pandemic. With more households working and going to school from home, resulting in longer peak periods and heavier traffic, more use of video and audio streaming and communications tools like videoconferencing, the need for faster and more reliable broadband is greater than ever. 

Let’s park our cynicism and assume for a moment, however, that the majority of Canadians do in fact have access to high speed Internet, the issue during the pandemic and well before, for many consumers, is not speed, but affordability of Internet service. In rural communities, household spending for Internet services is increasing despite slow deployment of high speed Internet. From 2013 to 2017, average monthly Internet access spending increased for rural households from $37.42 to $54.83, a whopping 46.5% increase.

The CMR directly acknowledges that rural households spend more than urban households because of “slightly higher prices offered in rural areas, where there are typically fewer service providers.” Instead of a simple Report examining largely the highest service tiers for the most easily-served demographics, the CRTC should at the least supplement this Report with a study that helps resolve the accessibility and affordability issues that have persisted for years, especially for vulnerable and underserved Canadians.

Conclusion: What’s wrong, why it matters, and how it can be fixed

The 2020 Report is flawed.  It presents an artificial “measurement” of selected networks, in selected locations, for selected users at selected speeds, in ideal conditions and a totally artificial context as far from “real world” Internet experience of users as we can imagine.

To then claim that Canadians’ experiences of the Internet are that it is fast is flatly wrong. It smacks of regulatory propaganda.  We are tired of this approach from our telecommunications regulator.

At the very least, the CRTC should rethink its methodological approach to make the next report more comprehensive and reliable. The CRTC should rethink its communications regarding this report and similar reports prepared for the CRTC such as the even more recent “Secret Shopper Project Report” – which has its own limitations, as examined in PIAC’s “We Fight for That” podcast, episode 2.

Next up: Part 2 – Traffic Cops on the Internet – Broadband Speed Advertising in Canada and Abroad

In part 2 of our “Buying Speed? What Canadians Pay for Broadband” series, our next blog post focuses on broadband speed advertising. PIAC notes that other countries view the broadband speed question much more pragmatically than Canada, and require advertised speeds to correspond to lived experiences of average users, at average times under average network loads.  Is Canada’s laissez-faire approach to this facilitating something like false advertising? You be the judge.

Comments on Ontario Privacy Consultation: PIAC Calls for Strengthening PIPEDA and Introducing Provincial Employment Privacy Legislation

OTTAWA– (16 October 2020) The Public Interest Advocacy Centre (“PIAC”) today responded to the government of Ontario’s Consultation to strengthen privacy protections in Ontario with PIAC’s recommendations for strengthening privacy protections in Ontario’s private sector, preferably by strengthening the federal Personal Information Protection and Electronic Documents Act (“PIPEDA”) and introducing provincial employment privacy legislation.
PIAC recommended bolstering privacy protections – whether in PIPEDA or a new Ontario statute – by making privacy a right, by widening the scope of privacy legislation and by greatly increasing compliance and enforcement powers of privacy authorities.
PIAC Executive Director, John Lawford stated: “We agree privacy of Ontarians and indeed all Canadians needs strengthening for a truly digital age. The best approach is to make privacy a consumer right and to enable consumers to enforce that right – whoever takes the lead.”
PIAC’s recommendations would help to align Canadian privacy legislation with other key jurisdictions, including the state of California and the European Union (“EU”). The EU General Data Protection Regulation  (“GDPR”), recognizes and balances individuals’ right to privacy with business and government interests in processing personal information.
“Whatever the outcome of Ontario’s efforts, we hope that they will enhance public trust in consumer privacy, allow businesses to innovate, increase participation in the digital economy and preserve democracy,” Lawford added.
Please click here for a copy of PIAC’s submission to the Government of Ontario.
For more information please contact:
 
John Lawford
Executive Director and General Counsel
Public Interest Advocacy Centre
Tel: 613-562-4002 ext. 125
Cell: 613-447-8125
jlawford@piac.ca

We Fight For That – Episode 2

A private sector privacy law for Ontario: a step in the right direction?

On August 13, the Ontario government launched a public consultation to solicit input on “creating a legislative framework for privacy in the province’s private sector,” citing longstanding public concern over data privacy intensified by increased reliance on digital platforms during the COVID-19 pandemic. The consultation and accompanying discussion paper outline key issues in data protection, many of which will be familiar to those following Canadian privacy legislative reform: broadly, the focus will be on increasing transparency around how information is gathered and used, strengthening consent and establishing an opt-in model for secondary uses of information, introducing a right to erasure of personal information (subject to limitations), introducing a right to data portability, increasing the enforcement powers of Ontario’s Privacy Commissioner, introducing requirements and protections for de-identified or derived data, enabling the establishment of data trusts for information sharing, and expanding the scope of the law to non-profit and non-commercial organizations, including political parties. The consultation comes two months after Quebec introduced a bill to update its data protection strategy along the lines of the European Union’s General Data Protection Regulation (GDPR). If a new law is passed, Ontario will join other provinces, namely BC, Alberta, and Quebec, in having its own private sector privacy legislation.

It may superficially appear that better privacy protection for Canadians at any jurisdictional level could only be a positive development. Privacy in Ontario’s private sector is currently governed by the 2000 Personal Information Protection and Electronic Documents Act (PIPEDA), a federal law crafted in the late 1990s that has long been groaning under the weight of various pressures, including paradigm-shifting developments in the media environment and the need to keep pace with international data-sharing norms, most notably the global standard-setting GDPR. Among the most significant gaps in PIPEDA are its lack of real enforcement mechanisms and its grey areas around consent, which have become muddier in the age of big data as increasingly complex information flows undermine people’s ability to fully understand what they’re agreeing to. Under the “substantial similarity” exemption to PIPEDA, provinces are allowed to establish their own private sector privacy law if it offers comparable privacy protection to the federal legislation.

While there have been rumblings of reform at the federal level, including the government’s May 2019 release of an aspirational “Digital Charter” and accompanying proposals for modernizing PIPEDA, it’s not clear how extensive the changes will be, or when Canadians can expect them, especially with the parliamentary schedule having been disrupted by the pandemic. The introduction of an Ontario data protection strategy might thus come as a welcome development to those eager for reform who are understandably frustrated being at the mercy of a slow-moving federal process.

But there are more reasons to be wary of further fragmenting privacy legislation along provincial or territorial lines. Without highly coordinated pan-provincial consistency and cooperation, a province-by-province enactment of privacy laws risks providing uneven protection to Canadians, whose personal information may be treated differently based on territorial factors like the residency of the consumer, the storage location of the data, or the locus of incorporation of the company that offers the service. There’s also a risk that the move will encourage legal gamesmanship, with companies simply transferring operations to weaker privacy jurisdictions.

A patchwork of provincial laws will also complicate the business environment and potentially exacerbate internal trade barriers. The movement of personal data across both national and international borders is essential to the internet economy, and some Ontario business leaders are already balking at the increased compliance burden posed by multiple, potentially inconsistent layers of regulation. These kinds of challenges are already playing out in the US, which has begun its own state-by-state introduction of consumer privacy laws in the void of a comprehensive national regime. Companies are seeing that even slight inconsistencies between laws—and even between rules that appear on the surface to grant the same rights, such as data portability—can lead to huge compliance costs, which may be passed onto consumers in the form of both higher prices and a shrunken market.

Some analysts have pointed to an emerging irony in the global privacy crackdown: rules that are outwardly pro-consumer may end up empowering the very tech monoliths whose abusive data practices they’re meant to target, since these companies have the deep pockets to absorb rising compliance costs and increased legal risk. While poll after poll shows that Canadians do have an interest in strong privacy protections, a robust federal law can avoid the unnecessary compliance burden posed by a proliferation of regional frameworks.

While promising to protect citizens within each province, a piecemeal approach to privacy may also pose challenges for federal and provincial regulators. Again, we can look for guidance to the international context, where traditional notions of territoriality and jurisdictional authority are being challenged by the nature of electronic data. Even as the EU’s equivalency requirement has put increasing pressure on countries to update their privacy laws, data privacy rights vary considerably across national borders, and the speed, ease, and complexity of global data circulation often severs the factual link between the location of data and the location of its user. This tension between bordered privacy regimes and borderless data has led to serious conflicts between countries seeking control over online information, including efforts by governments to set global privacy standards via their own domestic regulation. The result is that businesses, regulators, and consumers increasingly operate in an environment of uncertainty in which it’s unclear which country’s or region’s laws govern online data at any given time. A patchwork of provincial laws risks reproducing this uncertainty within Canada.

Managing these complexities will likely be pricey for provinces. As former federal privacy commissioner Jennifer Stoddart notes, Quebec’s recently tabled Bill 64, which proposes amendments to the province’s public and private sector privacy laws, intends to deal with the issue of cross-border transfers via a GDPR-style adequacy condition that requires assessment of the destination’s privacy regulations, but this process has proved cumbersome to even the EU’s large, experienced bureaucracy. In the EU, regulators are finding that the GDPR requires enormous investment and staffing resources in order to give it teeth. And in the US, state privacy laws are under near-constant amendment to close ambiguities and catch up to other jurisdictions. Even if Ontario’s rules would apply only to commercial activities within the province and not to interprovincial or international transfers, there are costs involved in reviewing and assessing compliance with any new regulatory regime.

Those impatient for change might be reassured by the rising urgency of federal privacy reform. Federal privacy commissioner Daniel Therrien has warned the federal government that the growing discrepancy between Canadian and European privacy law is increasingly threatening our trade relationship with the EU. Under the GDPR, EU citizens’ personal data can be transferred only to jurisdictions that have been determined by the European Commission (EC) to provide “adequate” privacy protection, unless the data subject’s valid consent has been obtained. With Canada’s adequacy status scheduled for review by 2022, privacy experts are calling for “serious, rather than cosmetic, reform to PIPEDA” to maintain the free flow of data Canada and European countries. A single, robust federal privacy regime is a more realistic road to adequacy and to ensuring that the EU is confident exchanging data with Canada.

PIAC’s preliminary view is that Canadians will be better protected by a robust, modernized federal data protection regime than by increasing province-by-province legislation. A regulatory patchwork risks putting Canadians in a worse position when it comes to understanding their privacy rights, increasing uncertainty around how data is handled and potentially enabling inconsistent treatment of personal information depending on the residency of the consumer. As digital surveillance has become more pervasive and intrusive and the risks to both individual and society more profound, it’s clear that PIPEDA has failed to keep up, but it’s unlikely that a proliferation of regional frameworks will be more effective in protecting consumers against the power of the multi-billion-dollar personal data industry. Federal lawmakers need to step in to protect Canadians’ interests as consumers and rights as citizens.