Demangled type names reference missing types

[CouleeApps]

Version and Platform (required):

• Binary Ninja Version: 3.5.4288-dev
• OS: macOS
• OS Version: 12.4
• CPU Architecture: M1

Bug Description:
When the Demangler constructs a type from a name, it can create Named Type References to types that do not yet exist in the analysis. These types are not created even afterwards, leaving you with a bunch of broken type references to missing names. This is especially noticeable when loading a PDB with mangled names and stripped type information.

Steps To Reproduce:
Please provide all steps required to reproduce the behavior:

1. Grab the binary from PDB Can create DataVariable to types which don't exist #4235
2. In the load options, disable pdb.features.createMissingNamedTypes (it is a workaround for PDBs for this bug)
3. Go to 0x140056c08
4. Try to navigate to the type of that variable
5. Notice it does not exist

Expected Behavior:
I expected types used in the analysis to exist, even if just as forward declared empty structures or something.

Screenshots:

This type does not exist.

Additional Information:
Demangler plugins #467 is blocking on this Demangler plugins proceeded without fixing this haha

[0xdevalias]

I've run into another variant of this when reversing a macOS C++ binary. Asking on Slack, it was suggested that it's another case of this issue, so wanted to include the context here.

In the linear view, there are a bunch of struct vtable_for_* entries, but there are no corresponding types created for them:

On slack, @CouleeApps said (Ref): "those are an artifact of the binary having a symbol for "here's where the vtable is" but it doesn't actually define a type for it"

This also seems to apply to the typeinfo_for_* entries as well:

[0xdevalias]

This issue seems somewhat related (at least to the variant I mentioned in my previous comment above):

• Automatically recognize and create C++ structures #1163

Also this comment/references:

I'm not 100%, so correct me if i'm wrong, but I believe this may be related to automatically creating the structs for vtable_for_* and typeinfo_for_* and similar?

If so, I was wondering why this didn't seem to currently be a thing, I first found myself at this issue (due to the empty structs not being visible in the types):

• Demangled type names reference missing types #4551

And then found the following blog series, which helped me understand the in-memory layout of the vtables/etc in clang binaries a lot better:

• https://shaharmike.com/cpp/vtable-part1/

  • C++ vtables - Part 1 - Basics

• https://shaharmike.com/cpp/vtable-part2/

  • C++ vtables - Part 2 - Multiple Inheritance

• https://shaharmike.com/cpp/vtable-part3/

  • C++ vtables - Part 3 - Virtual Inheritance

• https://shaharmike.com/cpp/vtable-part4/

  • C++ vtables - Part 4 - Compiler-Generated Code

Or for something way more low-level:

• https://clang.llvm.org/doxygen/VTableBuilder_8cpp_source.html

This may also be of interest:

• https://blog.trailofbits.com/2017/02/13/devirtualizing-c-with-binary-ninja/

As well as the existing vtable docs:

• https://docs.binary.ninja/guide/cpp.html#virtual-function-tables

It would be awesome if Binary Ninja core was able to handle this automagically!

---

Also, since they don't seem to be cross-linked currently, here is the issue for MSVC for easier findability:

• MSVC RTTI analysis  #3930

Originally posted by @0xdevalias in #3857 (comment)

[ccarpenter04]

I've also ran into this issue when dealing with a x86-64 Linux ELF.

[CouleeApps]

So quick update on this from a couple weeks of attempting to resolve it:

Creating all of the referenced missing types from the demangler is not hard. A quick pass through all demangled names is pretty fast and easy and solves that problem. The issue is that it creates a bunch of new problems. Notably, it leaves you with a bunch of empty demangler-defined types that clog up later stages of analysis, like PDB/DWARF parsing. It leaves you with an analysis full of duplicate structures and an extremely confused type system. Trying to resolve that problem is considerably harder than just creating empty structs for missing names.

There's also a secondary problem of QualifiedName and The Great QualifiedName Flattening of 202X (v35: see internal wiki page) where the names returned by the demangler are structurally different than those entered by users, and need some post-processing to be correct. This is probably not required for this issue but it came up again because the demanglers started making types with those names that technically don't conflict with user types, but actually yes they do.

That's where this issue is at for now. There might be more progress made in Gallifrey but it's a sneakily difficult task to tackle.

[CouleeApps]

Also I will add, Demangler Plugins was originally blocking on this because we didn't want to expose user plugins to the mess that is all of these problems. For 4.2 we decided that allowing customization was more important than exposing the mildly broken system. We intend to fix these bugs eventually, and wrote the demangler API as assuming the bugs are fixed, so that when they are, the demanglers will not need to be updated.