A source of random perl crashes

Recently we rolled out a new version of code to production and started observing a lot of segfaults being recorded to do with freeing memory:

Dec  5 14:11:06 XXX kernel: [268032.637417] perl[62948]: segfault at 7ffc3e26dff8 ip 00007f296a21f6ef sp 00007ffc3e26e000 error 6 in libc-2.19.so[7f296a1a3000+19f000]

Researching a bit further, I found that this was caused by some code which was meant to automatically save an object to the database when it was destroyed, if it had not already been saved:

# Catch objects being destroyed that have not been correctly flushed
sub DESTROY {
  shift->flush_to_database
}

Researching further, I discovered that the issue is to do with the way perl handles global destruction. During a normal object destroy phase this code will work just fine; however when the process is exiting and objects are being destroyed you don’t know what order they will be destroyed in. In this case I assume that the database object that we tried sending the object to had already been DESTROY/garbage collected which is what was causing the crash. Fortunately there’s quite a simple work-around:

# Catch objects being destroyed that have not been correctly flushed.
sub DESTROY {
  if( ${^GLOBAL_PHASE} eq 'DESTRUCT' ) {
    warn "Unflushed objects cannot be flushed when process is exiting";
  } else {
    shift->flush_to_database
  }
}

If you want to support perl < 5.14 you should use the Devel::GlobalDestruction module. Moo provides this more easily via the DEMOLISH subroutine:

# Catch objects being destroyed that have not been correctly flushed.
sub DEMOLISH {
  my ($self, $in_destruction) = @_;
  if( $in_destruction ) {
    warn "Unflushed objects cannot be flushed when process is exiting";
  } else {
    $self->flush_to_database
  }
}

We had another similar issue when using Log4perl – if you try to access it from within the DESTROY scope the logger object may already have been destroyed and the process baloons in memory until it is OOM’d. Again just a check if you are in global destruction from within your logging function will handle this just fine.

Leave a Reply

Your email address will not be published. Required fields are marked *