[RFC] -fsanitize=vptr instrumentation

Hi!

This patch attempts to instrument C++ when a pointer might not point
to object of the right dynamic type.

clang++ apparently instruments:
1) member accesses
2) method calls
3) downcasts (both pointer and reference downcasts)

The vptr-1.C (and from there derived -2 and -3 testcases) attempts to
model compiler-rt ubsan/TestCases/TypeCheck/vptr.cpp test.
1) and 2) are instrumented during genericization, 3) is instrumented
much earlier (build_static_cast_1), because afterwards it is not possible
to differentiate between reinterpret_cast, static_cast etc., but so that
it works with constexpr it is expanded only partially (just a builtin
function call) and the rest of instrumentation around it is added during
genericization; the builtin is ignored during constexpr evaluation.

The instrumentation works by computing a hash from the mangled type name
of the type we expect, and vtable pointer value in the object, this hash
is first used for a quick check in __ubsan_vptr_type_cache 128 element
hash table (if (__ubsan_vptr_type_cache[hash & 127] == hash), we don't
call anything and assume the access is fine), otherwise we call a library
function with the hash value, vtable pointer and rtti info what we are
looking for.

I've tried to optimize away checks for the most simple cases,
like when a member function is called on an object, not through a function
pointer (because then we know the vptr and the compiler should check that at
compile time), or when doing member access through this or member calls
through this (but only if it is to the actual type of the current class);
the this optimizations aren't done in ctors because the vptr is changing
there and the caller doesn't check for obvious reasons the vptr pointer
before calling ctor.

Attached are 3 testcases I'd like to discuss:

vptr-5.C is one Jason mailed me yesterday, clang++ doesn't instrument this
and g++ right now doesn't either, build_static_cast_1 certainly isn't called
in that case, and I must say I have no idea what should be checked there,
where etc.

vptr-6.C shows where the this optimization is performed and where it isn't
(clang++ has 10 instrumentations in T::h and 1 in S::l, g++ has fewer than
that, but not 0 in T::h (1 in S::l is right and needed I think)).

For downcasts, I'm testing the downcasted pointer for non-NULL (clang++ does
too), but not so for member access.  vptr-8.C testcase is me trying to
understand what is and what is not considered valid C++ for member accesses.
clang++ through -fsanitize=null already complains in f[1-6], g++ does
not (it will only instrument for null actual accesses, not mere taking of
address), but with -fsanitize=undefined (what matters is null,vptr)
clang++ complains in f1, complains in f2 and then segfaults on the vptr
instrumentation trying to read vtable pointer.  g++ doesn't complain on f1,
but complains on f2 and segfaults there too.
I guess my question is what exactly is strictly valid code (offsetof
really shouldn't be written that way I believe), and as f6 testcase shows,
often the pointer even isn't NULL when already taking address of a field
in dereference of a NULL pointer.
I hope all of f[1-6] is invalid, I really don't see how we could instrument
member accesses otherwise (we'd need to limit to not taking address of it);
NULL pointer shouldn't point at a valid object.

Thoughts, comments, suggestions for how to improve this?

2014-09-16  Jakub Jelinek  <jakub@redhat.com>

	* flag-types.h (enum sanitize_code): Add SANITIZE_VPTR,
	include SANITIZE_VPTR in SANITIZE_UNDEFINED.
	* opts.c (common_handle_option): Add -fsanitize=vptr.
	* sanitizer.def (BUILT_IN_UBSAN_HANDLE_DYNAMIC_TYPE_CACHE_MISS,
	BUILT_IN_UBSAN_HANDLE_DYNAMIC_TYPE_CACHE_MISS_ABORT): New.
	* ubsan.h (enum ubsan_null_ckind): Add UBSAN_DOWNCAST_POINTER
	and UBSAN_DOWNCAST_REFERENCE.
cp/
	* config-lang.in (gtfiles): Add cp/cp-ubsan.c.
	* cp-gimplify.c (cp_genericize_r): Call
	cp_ubsan_maybe_instrument_member_call for member calls,
	cp_ubsan_maybe_instrument_member_access for member accesses
	and cp_ubsan_fixup_downcast_instrumentation for downcast
	instrumentation.
	* cp-tree.h (cp_ubsan_maybe_instrument_member_call,
	cp_ubsan_maybe_instrument_member_access,
	cp_ubsan_maybe_instrument_downcast,
	cp_ubsan_fixup_downcast_instrumentation): New prototypes.
	* cp-ubsan.c: New file.
	* Make-lang.in (CXX_AND_OBJCXX_OBJS): Add cp/cp-ubsan.o.
	* semantics.c (cxx_eval_call_expression): Handle 4 argument
	BUILT_IN_UBSAN_HANDLE_DYNAMIC_TYPE_CACHE_MISS{,_ABORT}.
	* typeck.c (build_class_member_access_expr): Provide locus
	for COMPONENT_REFs.
	(build_static_cast_1): Instrument downcasts.
gcc/testsuite/
	* g++.dg/ubsan/vptr-1.C: New test.
	* g++.dg/ubsan/vptr-2.C: New test.
	* g++.dg/ubsan/vptr-3.C: New test.
	* g++.dg/ubsan/vptr-4.C: New test.
	* g++.dg/ubsan/vptr-5.C: New test.

	Jakub
extern "C" void abort();                                                                                                                           

struct S { virtual void f() {} };                                                                                                                  
struct T: S { ~T(); };                                                                                                                             
struct U: S { };                                                                                                                                   
struct V: T, virtual U {};                                                                                                                         

U* up;                                                                                                                                             
V* vp;                                                                                                                                             

int main()                                                                                                                                         
{                                                                                                                                                  
  V v;                                                                                                                                             
  up = vp = &v;                                                                                                                                    
}                                                                                                                                                  

T::~T()                                                                                                                                            
{                                                                                                                                                  
  if (vp != up)                                                                                                                                    
    abort();                                                                                                                                       
}
struct T;

struct S
{
  S() : a(0) {}
  ~S() {}
  int a;
  int f() { return 0; }
  virtual int v() { return 0; }
  const T *l();
};

struct T : S
{
  T() : b(0) {}
  int b;
  int g() { return 0; }
  virtual int v() { return 1; }
  int h();
};

int
T::h()
{
  return a + b + this->a + this->b + f() + g() + v() + this->f() + this->g() + this->v();
}

const T *
S::l()
{
  return (const T *) this;
}
struct R
{
  int r;
};

struct S
{
  S() : a(0) {}
  ~S() {}
  int a;
  int f() { return 0; }
  virtual int v() { return 0; }
};

struct T : R, S
{
  T() : b(0) {}
  int b;
  int g() { return 0; }
  virtual int v() { return 1; }
};

struct U
{
  char u;
  T t;
};

int *
f1 (R *p)
{
  return &p->r;
}

int *
f2 (S *p)
{
  return &p->a;
}

int *
f3 (T *p)
{
  return &p->a;
}

int *
f4 (T *p)
{
  return &p->b;
}

int *
f5 (U *p)
{
  return &p->t.b;
}

int *
f6 (U *p)
{
  T *t = &p->t;
  return &t->b;
}

int
main ()
{
  int *a = f1 (__null);
  int *b = f2 (__null);
  int *c = f3 (__null);
  int *d = f4 (__null);
  int *e = f5 (__null);
  int *f = f6 (__null);
}

[RFC] -fsanitize=vptr instrumentation

Commit Message

Comments

Patch