Binary Analysis and Reverse Engineering. A Security Team's Field Guide

Your scanners read text your engineers wrote. Somewhere between forty and ninety percent of what runs in production is code nobody at your company has ever read, and a large share of it arrives compiled.

Vendor agents. Container base layers. Firmware on the appliance in the rack. Statically linked libraries inside a binary whose lockfile you were never given.

Binary analysis is how you get a security opinion on that code. This is a working guide to the method, the tooling, the file formats, and the point at which the technique stops paying for itself.

Where this sits next to source-level tooling: CodeAnt AI covers the source side with AI code review and SAST, and the exploitation side with agentic pentesting and attack surface management. Binary analysis is the discipline for the middle, where you hold an artifact and no repository. The three answer different questions, and this guide is explicit about which question belongs to which.

What is Binary Analysis?

Binary analysis is the practice of determining what a compiled program does without access to its source code, by examining the executable file itself and by observing it run. It splits into two halves that answer different questions. Static analysis reads the file at rest, recovering structure, strings, imports, and control flow without executing anything.

Dynamic analysis runs the program under instrumentation and records what it actually does, including behaviour that only appears at runtime. Neither half is sufficient alone. Static analysis sees every path including ones that rarely execute, and it cannot resolve values computed at runtime.

Dynamic analysis sees ground truth for the paths it happens to take, and it is blind to everything else.

What is Reverse Engineering in Cyber Security?

Reverse engineering is the broader activity of recovering a system's design and behaviour from the finished product. In cyber security it has four distinct purposes, and confusing them wastes weeks.

• Vulnerability discovery: Finding exploitable defects in software you did not write, whether for a bug bounty, a product security assessment, or offensive research.

• Malware analysis: Determining what a hostile sample does, what it talks to, and how to detect it. The adversary here actively resists you.

• Interoperability and composition: Establishing what components a binary contains, which versions, and whether any carry known vulnerabilities.

• Patch analysis: Comparing a pre-patch and post-patch build to recover the vulnerability a vendor silently fixed.

The tooling overlaps heavily. The methodology does not. Deciding which of the four you are doing, before you open a disassembler, is the single highest-leverage decision in the engagement.

Why Source-Code Security Tools Cannot See the Shipped Binary

Every mainstream application security tool sits at a point in the pipeline where its input is text a developer wrote.

The gap is not an oversight. It is structural. A repository is a description of a program. A binary is the program, plus whatever the toolchain added, minus whatever the optimiser removed. Two things live only in the binary. The first is the actual composition, because a statically linked library leaves no lockfile entry.

The second is capability, because a repository holds no operating-system permission grants and a signed, installed application does.

How ELF, Mach-O, and PE Binaries Are Laid Out

You cannot analyse what you cannot parse. Three formats cover almost everything you will encounter.

ELF is the Linux and BSD format

An ELF file carries two overlapping views of the same bytes. Program headers describe segments, which is what the loader maps into memory. Section headers describe sections, which is what the linker used. The sections that matter early are .text for executable code and .rodata for read-only constants, which holds most string literals. Then .data and .bss for mutable globals, and .plt with .got for dynamic symbol resolution.

Mach-O is the macOS and iOS format

Instead of a section table it uses load commands, a sequence of variable-length records the kernel walks at load time. Segments are conventionally named __TEXT, __DATA, and __LINKEDIT. Mach-O has one property that catches people out. A universal binary is an archive containing several complete Mach-O files for different architectures, so your first task is often to extract the slice you care about before anything else parses correctly.

PE is the Windows format

A vestigial DOS header points at the real NT headers, followed by a section table with the familiar .text, .rdata, .data, and .rsrc. The Import Address Table is the highest-signal structure in the file, because it lists exactly which external functions the binary resolves at load.

What Stripping Removes, and What it Cannot

A stripped binary has had its symbol table discarded, so function names are gone. This is the single largest determinant of how long an engagement takes. What survives stripping is more than people expect. Dynamic symbols survive, because the loader needs them to resolve imports, so every library call is still named.

String literals survive, because they are data. Relocations survive. Exception-handling metadata survives on most platforms and can be used to recover function boundaries. Control flow survives, because it is the code. The practical consequence is that a stripped binary is harder, not opaque. You lose the author's names for things and keep everything the machine needs.

Static Binary Analysis, Step by Step

The order below is deliberate. Each step is cheap relative to the one after it, and each frequently answers the question before you reach the expensive part.

Step 1. Triage the file before opening a disassembler

Identify the format, the architecture, the bitness, the endianness, and whether it is stripped. Compute a hash and record it, because every finding you write will be attached to that hash.

file target                 # format, arch, stripped or not, dynamic or static
readelf -h target           # ELF header
otool -hv target            # Mach-O header

file target                 # format, arch, stripped or not, dynamic or static
readelf -h target           # ELF header
otool -hv target            # Mach-O header

file target                 # format, arch, stripped or not, dynamic or static
readelf -h target           # ELF header
otool -hv target            # Mach-O header

Check whether the binary is statically or dynamically linked. A statically linked binary means every dependency is inside the file with no external record, which makes composition analysis harder and more valuable.

Step 2. Read the strings, then measure the entropy

Strings are the cheapest high-value artifact in binary analysis. Endpoint paths, format specifiers, error messages, embedded configuration, version banners, and build paths all survive compilation intact.

strings -n 8 -t x target | less        # ASCII, with file offsets
strings -e l -n 8 target               # UTF-16LE, common in PE

strings -n 8 -t x target | less        # ASCII, with file offsets
strings -e l -n 8 target               # UTF-16LE, common in PE

strings -n 8 -t x target | less        # ASCII, with file offsets
strings -e l -n 8 target               # UTF-16LE, common in PE

Error and log strings are the richest source, because developers write messages that explain the security intent of the surrounding code. A literal reading certificate verification disabled tells you a control exists and roughly where to look for it.

Entropy tells you whether you are looking at real code. Compiled machine code typically measures around 6.0 to 6.5 bits per byte. Compressed or encrypted data approaches 8.0. A section at 7.9 with almost no strings means the payload is packed and your static pass will produce nothing until you unpack it. That is a finding in itself when the binary is not supposed to be packed.

Step 3. Read the import table as a capability manifest

Imports tell you what a binary is able to do, before you understand how it does it. This is the fastest route from a cold file to a threat model.

readelf --dyn-syms target      # ELF dynamic symbols
objdump -p target | grep NEEDED
otool -L target                # Mach-O shared library dependencies

readelf --dyn-syms target      # ELF dynamic symbols
objdump -p target | grep NEEDED
otool -L target                # Mach-O shared library dependencies

readelf --dyn-syms target      # ELF dynamic symbols
objdump -p target | grep NEEDED
otool -L target                # Mach-O shared library dependencies

Read the import list the way you would read an entitlement list. Network primitives mean it talks. dlopen means it loads code chosen at runtime, so your static call graph is incomplete by construction. Process creation primitives mean it can execute other programs. Cryptographic imports tell you which library and often which era.

The absence of an import is equally informative, because a binary with no networking imports and no dlopen cannot exfiltrate anything on its own.

Step 4. Disassemble, then decompile

Disassembly converts machine code to assembly instructions and is close to lossless. Decompilation attempts to reconstruct C-like source and is lossy, heuristic, and occasionally wrong.

Use the decompiler for speed and the disassembly for truth. When a finding depends on a specific comparison, an integer width, or a sign, read the instructions, because that is exactly the class of detail decompilers get wrong.

Two recovery techniques close most of the gap left by stripping. Library function identification matches known compiled forms of common library routines against your binary, so memcpy and friends get their names back automatically.

IDA calls these FLIRT signatures, and Ghidra ships an equivalent. Type propagation is the other. Once you name one structure correctly, the decompiler propagates that type through every function that touches it, and output quality improves sharply across the whole file.

Step 5. Recover control flow and reach the interesting code

A control flow graph turns a linear instruction listing into a navigable structure of basic blocks and branches. From there, cross-references are how you actually work. Take an interesting string, cross-reference to where it is used, cross-reference from that function to its callers, and walk upward until you reach an entry point that attacker-controlled data reaches.

Indirect calls through function pointers, virtual dispatch in C++ binaries, and anything resolved through dlopen will break the graph. Those breaks are where dynamic analysis earns its place.

Dynamic binary analysis, and when to reach for it

Run the program when static analysis stalls, when values are computed at runtime, or when you need proof rather than an argument.

• Debuggers give you breakpoints, memory inspection, and single-stepping. gdb with an enhancement like GEF or pwndbg on Linux, lldb on macOS, x64dbg on Windows.

• Dynamic instrumentation lets you hook functions in a running process and log or modify arguments without patching the binary. Frida is the common choice and is unusually good on mobile and desktop applications.

• System call tracing shows you every file opened, every socket, and every process spawned, which often answers the whole question. strace on Linux, dtruss on macOS, Process Monitor on Windows.

• Emulation runs a binary for an architecture you do not have hardware for, which is how most firmware analysis proceeds. QEMU user-mode emulation covers a surprising amount.

• Symbolic execution reasons about a program with unconstrained inputs and solves for the values that reach a target state. Tools like angr can produce a concrete input that reaches a specific instruction.

The honest limit on symbolic execution is path explosion. Branch counts multiply and real programs exceed any solver, so it is a scalpel for a specific function rather than a strategy for a whole binary.

How to Check Exploit Mitigations in a Compiled Binary

Before hunting for a memory-safety bug, establish what protections are compiled in. Mitigations decide whether a bug is exploitable and how hard.

On Linux, checksec --file=target reports most of these in one line.

On macOS the equivalent questions are whether the hardened runtime is enabled and whether library validation is set, which you read from codesign -dv --verbose=4. Library validation deserves its own note. It restricts loading to libraries signed by the same team identity, which closes the entire dylib-injection class in one flag. Its presence or absence should redirect hours of effort.

Binary Diffing and n-day Vulnerability Discovery

Patch diffing is the highest-yield technique in the whole discipline, and it is under-taught. A vendor ships a security fix. The advisory says a memory corruption issue was addressed. It names no function, no file, and no root cause.

The vulnerability is now public knowledge in the sense that a patch exists, and completely private in the sense that nobody outside the vendor knows what it was.

Diff the two builds and you recover it. The naive approach fails immediately. A recompile shifts addresses, reorders functions, and changes register allocation, so a byte-level diff reports that everything changed.

Structural diffing solves this by matching functions on properties that survive recompilation. Control flow graph shape, basic block count, call graph position, and string references. BinDiff and Diaphora both work this way. The workflow is short. Match functions across builds, sort by lowest similarity, discard the noise from inlining and register churn, and read the handful of functions with a genuine logic change.

A new bounds check on a length field is the vulnerability, described precisely.

Two operational implications follow.

• Defensively, the window between a patch shipping and a working exploit existing is now days, so patch deployment speed is a security control rather than an operations metric.

• Offensively, this is why an unreleased fix sitting in a public repository is more dangerous than an undiscovered bug. The diff is already available to anyone reading the commit log.

Binary Composition Analysis and the Binary SBOM

Software composition analysis reads your manifests. Binary composition analysis reads the artifact and infers what is inside it, which is the only option when you did not do the build.

The detection techniques are the same ones you use manually, applied at scale.

• Version strings. Most libraries embed a version banner. OpenSSL is the canonical example, and a single string frequently identifies the exact release.

• Function signatures. A compiled function has a recognisable shape. Matching against a corpus of known library builds identifies components with no strings at all.

• Constant tables. Cryptographic implementations contain distinctive constant arrays. S-boxes and initialisation vectors are effectively fingerprints.

• Build artifacts. Compiler version strings, embedded build paths, and debug metadata frequently survive and often disclose more than intended.

This is what makes a binary SBOM meaningfully different from a generated one. An SBOM produced by your build system lists what you declared. A binary SBOM lists what is actually in the artifact, and the two diverge whenever a dependency was vendored, statically linked, or copied into a container image without a package record.

For anything regulated, the binary-derived list is the one that matters. A vulnerable statically linked library is present whether or not any manifest mentions it.

Reverse Engineering Tools Compared

Tool choice matters less than method, but it matters. The realistic options split cleanly.

Ghidra is the correct default for almost everyone.

It is free, its decompiler is genuinely good, and its headless mode scripts cleanly, which matters the moment you want to analyse a hundred binaries rather than one.

The two commercial tools earn their price on daily professional use rather than on any single capability, and the free tooling is close enough that budget is rarely the real constraint.

Interpreted and Bundled Artifacts are not Binaries

A large share of what people call binary reverse engineering is not binary work at all, and treating it as such wastes effort. An Electron desktop application ships an archive of JavaScript. An Android APK ships DEX bytecode. A .NET assembly ships CIL. A Python application may ship bytecode or a bundled interpreter.

In every one of those cases the logic is in an intermediate or source-like form. You extract the archive, decompile the bytecode, or unminify the bundle, and you are reading something much closer to source than to assembly.

Opening these in a disassembler shows you the interpreter or the runtime, not the application. That is a genuinely common wasted afternoon. Where the analysis does become binary work is at the edges.

Native modules, cryptographic implementations, licensing checks, and anything deliberately pushed into compiled code to resist exactly this inspection.

Firmware, IoT, and Container Images

Three artifact types deserve their own note because their unwrapping differs.

• Firmware usually arrives as a monolithic image containing a bootloader, a kernel, and one or more filesystems. binwalk identifies and extracts the embedded filesystems, after which you are reading an ordinary Linux root filesystem with hardcoded credentials and an ancient BusyBox in it.

• Mobile applications are archives. An APK unzips to DEX bytecode plus native libraries, an IPA to a Mach-O plus resources. The composition question and the hardcoded-secret question are usually more productive than the memory-corruption question.

• Container images are layered tarballs. Package manager metadata gives you a partial inventory, and anything copied in with a COPY instruction has no package record at all, which is precisely the gap binary composition analysis fills.

Obfuscation, Packing, and Anti-Analysis

Some binaries actively resist you. The techniques are well catalogued.

• Packing compresses or encrypts the real payload and unpacks it in memory at runtime. High entropy plus a tiny import table is the signature. The general defeat is to run it and dump memory once it has unpacked itself.

• Control flow flattening replaces structured branches with a dispatch loop, so every basic block returns to a switch. It destroys decompiler output and is recognisable on sight.

• String encryption removes your cheapest source of signal. The decryption routine is in the binary by necessity, so the standard answer is to find it and script bulk decryption.

• Anti-debugging checks for a debugger and alters behaviour. Detection primitives are limited and well known, and hooking them is routine.

The strategic point is that obfuscation raises cost rather than providing security. Everything needed to execute the program is present in the artifact by definition. For defenders, that means anti-tamper and runtime application self-protection are commercially useful and are not a substitute for the code being correct. For analysts, it means the question is only ever whether the target is worth the hours.

Is Reverse Engineering Legal?

This is a genuine constraint on the work and not a footnote. What follows is general information rather than legal advice, and jurisdictions differ substantially.

• In the United States, the DMCA prohibits circumventing technical protection measures, with statutory exemptions including one for good-faith security testing, plus periodic rulemaking exemptions. Contract terms are a separate and often stricter constraint than copyright law, and end-user licence agreements frequently prohibit reverse engineering outright.

• In the European Union, the Software Directive expressly permits decompilation for interoperability under defined conditions, and contract terms cannot override that particular right.

Three practical rules keep researchers out of trouble. Analysing software you lawfully possess, for defensive purposes, without redistributing derived code, is the safest posture.

Anything touching a system you do not own needs written authorisation before it starts. And where the work is commercial or the vendor is litigious, involve counsel early rather than after publication.

Where AI Actually Changes Binary Analysis

One disambiguation first, because two different disciplines share the same words.

This section is about using models to assist analysis of compiled software. It is not about reverse-engineering AI models, which is a separate field.

Language models are genuinely useful at three narrow tasks in this workflow.

• Naming and summarising decompiled functions. Given decompiler output, a model produces a plausible function name and a summary. This is a real time saver across hundreds of unnamed functions, and it is a hypothesis rather than a fact.

• Explaining unfamiliar constructs. Compiler idioms, unusual instruction sequences, and platform-specific patterns get explained faster than a search does.

• Scripting the mechanical parts. Generating Ghidra scripts, parsing structures, and automating repetitive analysis passes.

What models are not reliable at is the part that matters, which is deciding whether a specific code path is exploitable. Decompiler output is already lossy, and a model reasoning over lossy output produces confident text with no execution behind it.

The rule that keeps this useful. Model output is a hypothesis to verify against the disassembly or against the running program, never a finding to report.

How CodeAnt AI Model Fits

Being precise here is more useful than being expansive. CodeAnt does not ship a disassembler. If your task is Ghidra work on a stripped ARM firmware image, this guide's tooling table is the answer and no platform replaces it.

What CodeAnt addresses is the reason binary analysis gets reached for in the first place. Teams lose the source-level view of what they ship and what they run.

On the source side, AI code review and SAST run against the code as it is written, which is the only point where a defect is cheap to fix. The record is public. CodeAnt has found 150+ CVEs, including a CVSS 10.0 vulnerability in pac4j that went undetected for six years, across projects representing 2B+ monthly downloads protected.

That pac4j finding is the relevant proof for this article.

A critical flaw sat for six years in reviewed, open-source, heavily depended-on code, and it was not found by looking harder at any single component.

• On the artifact side, the AI penetration testing pipeline already performs artifact-level analysis on the web surface. Production JavaScript bundles are pulled and statically mined for API keys, internal hostnames, OAuth client identifiers, and source-map artifacts that leak original paths. That is the same technique this guide describes for bundled applications, applied continuously rather than during an engagement.

• On the exposure side, attack surface management correlates more than thirty sources including the National Vulnerability Database, the CISA Known Exploited Vulnerabilities catalog, and EPSS scoring. That is exactly the cross-reference a binary composition finding needs, because knowing a vulnerable library version is embedded is only half the answer.

CVSS tells you how bad the flaw is in theory. EPSS tells you how likely it is to be exploited in practice, and the pair is how a hundred CVEs becomes a shortlist.

On the proof side, more than 500 agents chain findings to demonstrate what an attacker actually reaches, because a detected vulnerability is not a confirmed leak.

The honest division of labour looks like this.

Binary analysis is the tool for the case where you have an artifact and nothing else. The engineering goal is to be in that position as rarely as possible.

The Binary Analysis Checklist

Run in order. Most engagements resolve before step 6.

Triage

• Identify and pin. Format, architecture, bitness, stripped or not, static or dynamic. Record the hash.

• Check signing. Signature validity, signer identity, and on macOS the hardened runtime and library validation flags.

Cheap signal

• Extract strings in both ASCII and UTF-16, and read the error and log messages first.

• Measure section entropy. Above 7.5 with no strings means packed, and nothing static will work until it is unpacked.

• Read the imports as a capability list, and note whether dlopen or equivalent appears.

Composition

• Identify embedded components by version string, function signature, and constant table.

• Cross-reference each identified version against NVD, the CISA KEV catalog, and EPSS.

• Diff the binary inventory against your declared SBOM. Every discrepancy is an undeclared dependency.

Depth, only if warranted

• Check mitigations before hunting memory-safety bugs, since they determine exploitability.

• Cross-reference from interesting strings to callers, and walk up to an attacker-reachable entry point.

• Run it when static analysis stalls on runtime-computed values or indirect calls.

Rigour

• Verify every model-generated name against the disassembly before it enters a report.

• State what you proved and what you inferred as separate claims in every finding.

Binary Analysis: Understand What You Ship Before Attackers Do

Binary analysis gives security teams visibility into compiled software when source code, manifests, or build context are unavailable. It helps uncover hidden dependencies, embedded vulnerabilities, security controls, and attacker-relevant behaviour across binaries, firmware, containers, and shipped applications.

But for software you control, reverse engineering should be the fallback—not your first line of defence. Find vulnerabilities in source code before they become compiled artifacts, continuously understand what reaches production, and validate whether attackers can actually exploit what you find.

That is the gap CodeAnt AI is built to close.

CodeAnt AI combines AI code review, SAST, attack surface management, and agentic penetration testing to help security teams find vulnerabilities earlier, prioritize real exposure, and validate attack paths continuously.

Don't wait until you're reverse engineering your own production binary to discover what went wrong.

👉 Explore CodeAnt AI and see how continuous AI-powered application security can protect your code, artifacts, and attack surface.