Archive for March 1, 2021

Google Voice Outage Caused By Expired Certificates…. REALLY?

Posted in Commentary with tags , on March 1, 2021 by itnerd

Back in mid February, Google Voice went down for about four hours. That left users unable to log in and use their Google Voice accounts. That’s a problem if you rely on Google Voice. And a lot of people and companies do given the times that we live in. Well, Google has released an incident report [Warning: PDF] and it is eyebrow raising. The outage was caused by expired TLS certificates:

Google Voice uses the Session Initiation Protocol (SIP) to control voice calls over Internet Protocol. During normal operation, Google Voice client devices aim to maintain continuous SIP connection to Google Voice services. When a connection breaks, the client immediately attempts to restore connectivity. All Google Voice SIP traffic is encrypted using Transport Layer Security (TLS). The TLS certificates and certificate configurations used by Google Voice frontend systems are rotated regularly.

Due to an issue with updating certificate configurations, the active certificate in Google Voice frontend systems inadvertently expired at 2021-02-15 23:51:00, triggering the issue. During the impact period, any clients attempting to establish or reestablish an SIP connection were unable to do so. These clients were unable to initiate or receive VoIP calls during the impact period. Client devices with an SIP connection that was established before the incident and not interrupted during the incident were unaffected.

And this is what they are going to do to stop this from happening again:

To guard against the issue recurring and to reduce the impact of similar events, we are taking the following actions:

  • Configure additional proactive alerting for upcoming certificate expiration events.
  • Configure additional reactive alerting for TLS errors in Google Voice frontend systems.
  • Improve automated tooling for certificate rotation and configuration updates.
  • Utilize more flexible infrastructure for rapid deployment of configuration changes.
  • Update resource allocation systems to more efficiently provision emergency resources during incidents.
  • Develop training and practice scenarios for emergency rollouts of Google Voice frontend systems and configurations.

Now I expect a small or medium company to have issues keeping track of when certificates that power their infrastructure expire. But for a company the size of Google to have this issue is mind blowing.

Chris Hickman, chief security officer at Keyfactor (www.keyfactor.com), a provider of cloud-first PKI as-a-Service and crypto-agility solutions has this to say:

An outage happens when expired certificates fail to authenticate or establish secure communication tunnels. A certificate expiration on its own is not necessarily a security response incident but is disruptive and can lead to outages like that experienced by Google Voice customers. Certificate expiration is an important mechanism to make sure certificates are still being issued to a valid system, similarly to why a driver’s license or passport needs to be renewed periodically. It offers a check and balance system, in the form of workflow and approvals, to maintain legitimacy and authorization. Changes implemented last year by the CA/B forum reduced the lifetime of an SSL/TLS certificate to 398 days and therefore has compounded the issue of keeping up with expiring certificates.

Recent research found that 73% of enterprise respondents experienced unplanned downtime and outages due to mismanaged digital certificates. More than half of those organizations said they experienced four or more certificate-related outages in the past two years. Service outages due to expired certificates are fairly common – and avoidable. Whether you’re a large enterprise or a small business, certificates expire. The key is maintaining visibility to every certificate on the network to stay ahead of expirations and renewals or better yet, using automation to ensure certificates are renewed prior to expiration without the need for human intervention.

These steps can help IT teams avoid similar outages and potential disruptions: 

  • Conduct an audit to understand how many digital certificates the organization has.
  • Build an inventory to identify where certificates live and what they’re used for. 
  • Document the hash algorithm they use and their overall health. 
  • Flag certificate expiration dates. 
  • Assign or note who owns every certificate.
  • Map the methods used to protect valuable code-signing certificates. 
  • Ensure a centralized method is used to securely update every certificate.”

Maybe Google should reach out to Keyfactor as clearly this is a weak point for them.

Kitchener Launches New Website & Customizable MyKitchener Portal

Posted in Commentary with tags on March 1, 2021 by itnerd

Today, the City of Kitchener launches its comprehensive new online experience for Kitchener residents, including a completely transformed website and new, cutting-edge online customer service portal called MyKitchener. The seamless integration between the new website, portal and City services reimagines how municipalities offer services online.

The MyKitchener portal is a home screen for every Kitchener resident, offering a personalized experience through a customizable collection of widgets – each with a modern design that scales to any size of device. These widgets let residents build a home screen with the information that matters to them – neighbourhood events, service notifications, opportunities to get more involved in the community. It’s a tool that becomes something unique for every family in Kitchener. With a secure MyKitchener account, residents can:

  • Customize their dashboard with content that is most important to the user
  • Receive notifications when new information is available on the portal, including snow events, skating and swimming events, road closures and more
  • View information about multiple properties in one place
  • View and pay property tax and utility bills
  • View road closures near their address
  • View upcoming swim times at their local pool
  • Find a time to skate at their local arena
  • View news and upcoming events
  • Watch a council meeting and download meeting minutes and agendas

The MyKitchener portal is the product of extensive public consultation with the local community. During the 2018 Customer Service Review, residents said they wanted to see more city services online and that they’d prefer to access and use them in a single place.

The City has also redeveloped its online presence from the ground up, incorporating best-in-class web design practices to give Kitchener residents the information they’re looking for right away. The MyKitchener experience includes a new website with plain language content. By lowering the reading level of the website from college to grade seven, the City is making their website more accessible and easier to understand. Content on the website is structured so that voice assistants like Siri, Alexa and Google can use website content to answer popular resident questions. The City hopes that their new website and MyKitchener portal will empower residents to self-serve on their schedule. 


The Digital Kitchener Lab at Communitech played a key role in supporting the design and user experience of MyKitchener. In the early days of the project, the Digital Kitchener lab hosted its first design sprint to explore ideas like voice to text search navigation, which is a new feature included in the website redesign based on user feedback. The design sprint offered a unique opportunity to prototype early concepts with residents before developing a full solution.

The website and MyKitchener portal are live, continuously evolving resources that will incorporate resident feedback, adapt to new programs and incorporate new features over time. The City is asking residents to participate in the ongoing development of the site and portal by registering an account and providing their feedback directly on the website or through the share feedback button inside the portal.

Residents can find the new website at www.kitchener.ca and create a MyKitchener account on the website or at mykitchener.kitchener.ca