With Matthew Bair
2026 marks the fourth consecutive year we’ve used ThousandEyes to monitor the Black Hat USA network for latency. Our first deployment in 2023 featured a small fleet of Raspberry Pi’s with (officially unsupported!) wireless NICs running the ThousandEyes Enterprise Agent in a custom build configuration. Over the years, the hardware has improved to beefier Orange Pi’s with their own built in wireless NICs, rather than add-on external antennas, the fleet of agents has multiplied in number, and we’ve added key control and automation capabilities.
As our latency monitoring mesh has expanded to cover more and more of the conference, we’ve also begun supplementing our ThousandEyes monitoring with on-demand Linux commands to track latency across different protocols. In this blog post, we’ll take a quick review of the deployment as it stands today, and show a few examples of the Linux based monitoring we’ve incorporated into our monitoring mesh of agents.
The Heart of ThousandEyes Visibility
The primary output of our entire ThousandEyes deployment is a dashboard that simultaneously tracks latency across multiple protocols, download speed, and internet availability.
This gives us an at-a-glance snapshot of connection quality spread across many of the rooms of the conference. As with the prior conference years, we reinvented our dashboard this year to provide better visuals and incorporate more data.
The Key ThousandEyes Tests
The automated ThousandEyes tests that inform our dashboards include HTTPS connectivity tests, the monitoring of cloud services, agent to agent tests, internal and external DNS resolution, and even file downloads. And going beyond the dashboards, the visualization that ThousandEyes provides can deliver unpararelled insight into traffic paths at scale, from one host to the entire deployment, either through automated test results or those performed on demand.
So where do Linux commands come into all of this? We’ve found that having rapid-fire tests and results at command line speed have a place in our monitoring when the need for immediate troubleshooting and quick verification is needed. Here are a few of the commands we’re using.
On-Demand Latency Tests from the Agent Mesh
Ping and traceroute are the go-tos for anyone who wants to do a quick test of their latency, but we’ve run into protocol specific latency at Black Hat and other conferences that required more dedicated troubleshooting. After some iterations, we built the curl command below that breaks an HTTPS connection into a few useful measurements, separating the DNS request from the TCP 3 way handshake and the TLS negotiation, before calculating time to first byte and then calculating the total time.
curl -sS -o /dev/null \
-w '%{time_namelookup} %{time_connect} %{time_appconnect} %{time_starttransfer} %{time_total}\n' \
https://example.com |
awk '{
printf "DNS lookup: %8.2f ms\n", $1 * 1000
printf "TCP connection: %8.2f ms\n", ($2 - $1) * 1000
printf "TLS handshake: %8.2f ms\n", ($3 - $2) * 1000
printf "Wait for first byte: %8.2f ms\n", ($4 - $3) * 1000
printf "Total: %8.2f ms\n", $5 * 1000
}'
DNS lookup: 73.30 ms
TCP connection: 37.77 ms
TLS handshake: 64.52 ms
Wait for first byte: 48.11 ms
Total: 224.94 ms
This breaks the connection down into components that can be compared to identify exactly where an example of high latency is occurring (e.g. DNS, or TLS negotiation). Note that the ‘wait for first byte’ includes the request, network transit time to and from the server, and the server’s own response time. However, there is a gotcha here for repeated tests: running the curl command multiple times will result in a cached DNS entry, greatly speeding up the DNS resolution component:
curl -sS -o /dev/null \
-w '%{time_namelookup} %{time_connect} %{time_appconnect} %{time_starttransfer} %{time_total}\n' \
https://example.com |
awk '{
printf "DNS lookup: %8.2f ms\n", $1 * 1000
printf "TCP connection: %8.2f ms\n", ($2 - $1) * 1000
printf "TLS handshake: %8.2f ms\n", ($3 - $2) * 1000
printf "Wait for first byte: %8.2f ms\n", ($4 - $3) * 1000
printf "Total: %8.2f ms\n", $5 * 1000
}'
DNS lookup: 4.86 ms
TCP connection: 43.89 ms
TLS handshake: 60.31 ms
Wait for first byte: 75.74 ms
Total: 185.82 ms
The above output can still be very useful, but we must account for the variance in DNS resolution time for repeat outputs, and the effect on the total. If DNS is the focus of our troubleshooting (which it was at times during this Black Hat), we found that the dig command was the most useful way to isolate the DNS component of the connection with consistent results. The below command forces a DNS resolution of example.com against one of the public Umbrella DNS IPs:
$ dig @208.67.220.220 example.com | grep -i "query time" ;; Query time: 61 msec
Running the command multiple times produces consistent DNS query times, unlike the curl command shown above.
$ dig @208.67.220.220 example.com | grep -i "query time" ;; Query time: 50 msec
With the right command configuration, the output from dig represents a rapid fire visual to compare against the DNS tests we automate or run on demand in ThousandEyes.
Conclusion
While ping and traceroute provide a quick test of network latency, sometimes greater protocol awareness is needed. A recent example we hit at one conference was an instance of NAT exhaustion predominantly affecting DNS resolution: the high volume, quick DNS resolutions made up the bulk of connection requests that were trying and failing to be mapped to a port. What on the surface looked like general latency issues and slow page load times was revealed to be specifically a problem with DNS. However, this wasn’t discovered until protocol specific commands were used to isolate DNS resolution failures, and packet captures confirmed the behavior.
ThousandEyes gives us both broad and targeted monitoring of network performance, including for DNS resolution. However, when speed is of the essence and a quick visual of the components of a connection are needed, tools like curl and dig can provide quick and helpful output to confirm troubleshooting direction.
Check out the other blogs from our team at Black Hat USA 2026.
-
- Building the Agentic SOC, One Live Event at a Time
- Safeguarding DNS With Secure Access at Black Hat
- Thrown into the SOC – A Black Hat First-Timer’s Story
- Building a Risk-Based Secure Network Analytics Detection with Splunk Detection Editor (Alpha)
- Troubleshooting Wi-Fi at Black Hat USA 2026 with ThousandEyes
About Black Hat
Black Hat is the cybersecurity industry’s most established and in-depth security event series. Founded in 1997, these annual, multi-day events provide attendees with the latest in cybersecurity research, development, and trends. Driven by the needs of the community, Black Hat events showcase content directly from the community through Briefings presentations, Trainings courses, Summits, and more. As the event series where all career levels and academic disciplines convene to collaborate, network, and discuss the cybersecurity topics that matter most to them, attendees can find Black Hat events in the United States, Canada, Europe, Middle East and Africa, and Asia. For more information, please visit www.BlackHat.com.