Profilerpedia: A map of the Software Profiling Ecosystem

Photo by William Warby / Unsplash

Summary: I mapped out over 100 Profilers, what Profile Data Formats they output, and what Profiler UIs can visualise those formats: Profilerpedia (Google Sheets link).

Profilers measure platforms, outputting Profile Data Formats.  Converters convert profile data formats to each other. Analysis UIs read Profile Data Formats.
Profilepedia catalogues Profilers, Profile Data Formats, Profile Converters, and Profile Analysis UIs.

I want to make software faster, so I do a lot of profiling. But often, I get stuck using only the profile analysis UIs built for a specific platform.

Profile Analysis UIs and Visualisers help people analyse profiles. Profile data files are usually large – many megabytes – too much information to read directly by humans. Visualisation and summarisation are necessary to understanding profile data.

I often want powerful visualisation features: symbolisation, deobfuscation, aggregated flamegraphs, time-series flamecharts, duration filtering, per-thread and per-process tracks, thread/process drilldown, fast search, stack pivoting, permalink-sharing, profile diffing, joining multiple profiles.

But these features are expensive to implement, and profilers are very siloed: different programming languages, runtimes, and OS's often have different profilers and analysis UIs. Most profilers have few analysis features.

Converting my platform-specific profile data to another profile format often unlocks much better profile visualisers – for example:

This happened to me enough times that I think it's worth mapping out what profile analysis UIs can read different profiler data formats, and how to convert between profile data formats. I suspect many engineers are suffering with poor analysis UIs, unaware that better tools are available.

Software profiling is a rich ecosystem, with hundreds of distinct profilers, profile data formats, and profile visualisers.

I hope this profiler ecosystem map will be useful for:

  • Performance engineers who are looking for better or complementary profile analysis tools. e.g. convert your language-specific profile to Firefox Profiler format to get link-sharing.
  • Profiler Engineers, who are looking to leverage pre-existing profile formats and analysis UIs. e.g. if you're writing a profiler for a new language runtime, output Folded Stacks format to leverage the FlameGraph toolkit, or Gecko Profile format to leverage Firefox Profiler.
  • Analysis UI Engineers, who are looking to accept different profile formats, or find pre-written parsers for those formats.
  • Profiler Product Managers, trying to bring new profilers to market.
  • Profiler Standardisers, who are looking to standardise on profiling formats. For example, OpenTelemetry is discussing profile formats. I anticipate a wave of evolution, then consolidation and  standardisation as the cloud profiler space rapidly heats up and then matures.

I did a quick look through Google Scholar and couldn't find a comparable literature review, so I think Profilepedia is the most complete map of the profiling ecosystem available on the web.

Structure

Profilerpedia is 6 tables, laid out like a normalised relational database:

  • Profilers: Profilers measure a system.
  • Formats: File formats output by Profilers and read in by UIs.
  • UIs: Analysis UIs for interpreting profiles
  • Outputs: which profilers output which formats. many:many Profiler:Format relationship.
  • Converters: which converters can change one format into another
  • Visualises: which UIs can read which Formats. many:many Format:UI relationship.

Terms

Profilers measure the performance of software and hardware, and help you narrow down the cause of performance problems, so you can optimise your system performance. Profilers generally run alongside some workload that's being measured.

Profilers most commonly measure time (latency), but can also measure memory allocation, memory usage, bandwidth usage, cache performance, disk performance, and many other resources.

Profilers can measure many targets:

Profilers typically output profiles, usually as files in some profile data format. For example, Go's CPU Profiler outputs profiles in pprof format, and Windows Performance Recorder outputs profiles in "Event Tracing for Windows" format.

There is a blurry line between Profilers and Tracers with much overlap. Usually, Profilers sample data at some frequency (e.g. sampling stacks on-CPU at 1000Hz). Tracers are usually code that's been instrumented to log when it hits a certain codepath. Many analysis toolkits combine sampling and instrumentation, so I've included both tracers and profilers in this analysis.

Many Profilers have both the measurement and analysis built-in: for example Chrome DevTools can profile code and analyse the profile. I've modelled these as both Profilers and UIs.

Next Steps

I'm definitely missing some entries. Please suggest additions/changes by leaving comments on the spreadsheet.

I hope to use this map to:

  • Write a blogpost about the strongly-connected nodes in the graph: the "lingua franca"/Rosetta-stone profilers, formats, and UIs.
  • Plot the dataflow in Graphviz, to identify popular, but disconnected data formats, or excellent but disconnected analysis UIs, and see if we can build converters to connect them.
  • Build converters between popular profilers and analysis UIs to unlock better analysis.
  • Update profiler documentation to point to great analysis UIs that can accept the output for that profiler.

Fingers-crossed, with relatively-small effort (building converters, documenting existing converters and analysis UIs) we can empower engineers profiling many different languages/runtimes, with far more powerful profiling tools.

Long-term (very speculative), this M*N mesh of profiling data formats might benefit from a conversion toolkit that turns M*N converters into M+N readers and writers.

Mark Hansen

Mark Hansen

Sydney, Australia