When CyberRatings.org, an independent security product testing organisation, released their latest Cloud Network Firewall test results, one number stood out: zero. While industry leader Fortinet and its Next-Generation Firewall (NGFW) achieved a perfect 100, Microsoft’s Azure Firewall registered a jaw-dropping 0% effectiveness score. At face value, a score of 0 versus 100 seems almost too convenient, perhaps even a bit of marketing spin. Was this a fair comparison?
To understand the abysmal score, we looked past the headline and into the testing process itself. It quickly became clear this wasn’t a marketing stunt; CyberRatings.org was transparent about their methods. They detailed a methodology where firewalls were scored on their response to threat traffic, mapped against the OSI model. They also specified the use of a specialised tool from Keysight called Cyperf for the evaluation. This technical approach convinced us to replicate the test and see if the results held up under our own scrutiny.
Our Lab: Replicating a Real-World Enterprise Scenario
To put Azure Firewall to the test, we built a classic Hub-and-Spoke network in Azure. This is a standard, widely adopted architecture for most enterprise cloud environments, making our test highly relevant to real-world deployments.
In the hub, we deployed an Azure Firewall Premium instance. We wanted to give it the best possible chance, so we enabled all the advanced security features:
- Threat Intelligence: Switched from the default “alert only” mode to the much stricter “alert and deny”.
- IDPS (Intrusion Detection and Prevention System): Enabled to inspect traffic for malicious patterns.
- TLS Inspection: Enabled to allow the IDPS to see inside encrypted traffic.
Each spoke virtual network was configured with a default subnet for workloads and a separate management subnet. A route table was applied to force all outbound traffic (destined for 0.0.0.0/0) to the internal IP of the Azure Firewall. The goal here wasn’t to test basic access control lists, but to specifically examine the effectiveness of the premium security features. To facilitate this, we created an any-to-any allow rule between the two spoke networks.
The Tool for the Job: Keysight’s Cyperf
We used Keysight’s Cyperf testing tool, the same platform used by CyberRatings.org. Cyperf is designed to simulate both legitimate application traffic and malicious threat traffic between its agents, allowing you to evaluate the security effectiveness and performance of a network implementation under load.
Our Cyperf deployment in Azure consisted of:
- A controller, which we placed in the Hub VNet. This allowed direct communication with the agents rather than having the firewall inspect the orchestration traffic. The controller used a public IP for management access.
- Two agents. Agent 1 acted as the client, and Agent 2 as the server. Each agent was equipped with two network interfaces: one for the test traffic and one for management communication back to the controller.
Thanks to the Keysight team, we secured a trial license and got a helpful walkthrough to get our tests up and running quickly.
The Results Are In… And They’re a Perfect Echo
After configuring a test session to run for 10 minutes using a combination of application and threat traffic, we initiated the test and watched the real-time statistics. The final result? A resounding zero blocked threats.
While our test wasn’t an exact replica of the CyberRatings methodology, it didn’t need to be. Our findings clearly illuminated why Azure Firewall performed the way it did and solidified our long-held position on its use in enterprise environments.
Why the “Zero”? A Deeper Dive into Azure Firewall’s Architecture
The disappointing result isn’t a bug; it’s a feature of Azure Firewall’s design. The premium security capabilities that promise next-generation protection have specific and critical limitations.
Threat Intelligence
This feature operates as a simple blacklist of known malicious public IPs and FQDNs. Crucially, it is only applied to North-South traffic—that is, traffic moving in and out of the internet (the “untrusted interface”). Since our Cyperf test simulated East-West traffic between two spokes within our Azure VNet, this feature was never triggered. Even if the test had been run from the internet, the source IP would have needed to be on Microsoft’s specific blacklist to be blocked.
IDPS (Intrusion Detection and Prevention System)
Microsoft maintains an impressive list of over 80,000 threat signatures for its IDPS, covering everything from botnets to trojans. We were initially sceptical of the CyberRatings result because we knew this feature existed. Unfortunately, like Threat Intelligence, the IDPS only inspects traffic on the untrusted, internet-facing interface. Furthermore, it only works on Application Rules, which are designed for traffic heading to a specific FQDN or URL. Our test traffic was between two internal agents and allowed via a Network Rule, meaning the IDPS engine completely ignored it.
TLS Inspection
This feature is an extension of the IDPS, allowing it to decrypt and inspect encrypted traffic. As such, it inherits all the same limitations. Since the IDPS wasn’t evaluating our test traffic, TLS inspection was irrelevant.
The Verdict: Beyond the Hype and Back to Reality
This exercise demonstrates that while Azure Firewall has “next-generation” features, they are confined to the Premium SKU, are not enabled by default, and most importantly, are designed almost exclusively for North-South, internet-bound traffic.
We’ve long held the opinion that while Azure Firewall is better than nothing and suitable for an initial cloud pilot, customers should look to the same full-featured next-generation firewalls they are already using on-premises in the cloud. The “0-to-100” security gap is real, but the argument goes beyond just effectiveness. Using a consistent firewall platform like Fortinet across your on-premises and cloud sites provides a massive operational advantage. It allows IT teams to operationalise the cloud with familiar tools and shared experience, reducing complexity and the risk of misconfiguration.
Modern cloud migrations are an opportunity to adopt a Zero Trust security model, where you segment networks and inspect traffic between those segments. This exercise highlights that attempting to do so with Azure Firewall for your critical East-West traffic flows will not yield the security benefits you expect from a true next-generation solution.