Recently we rolled out a new version of code to production and started observing a lot of segfaults being recorded to do with freeing memory:
Dec 5 14:11:06 XXX kernel: [268032.637417] perl[62948]: segfault at 7ffc3e26dff8 ip 00007f296a21f6ef sp 00007ffc3e26e000 error 6 in libc-2.19.so[7f296a1a3000+19f000]
Researching a bit further, I found that this was caused by some code which was meant to automatically save an object to the database when it was destroyed, if it had not already been saved:
# Catch objects being destroyed that have not been correctly flushed sub DESTROY { shift->flush_to_database }
Researching further, I discovered that the issue is to do with the way perl handles global destruction. During a normal object destroy phase this code will work just fine; however when the process is exiting and objects are being destroyed you don’t know what order they will be destroyed in. In this case I assume that the database object that we tried sending the object to had already been DESTROY/garbage collected which is what was causing the crash. Fortunately there’s quite a simple work-around:
# Catch objects being destroyed that have not been correctly flushed. sub DESTROY { if( ${^GLOBAL_PHASE} eq 'DESTRUCT' ) { warn "Unflushed objects cannot be flushed when process is exiting"; } else { shift->flush_to_database } }
If you want to support perl < 5.14 you should use the Devel::GlobalDestruction
module. Moo
provides this more easily via the DEMOLISH subroutine:
# Catch objects being destroyed that have not been correctly flushed. sub DEMOLISH { my ($self, $in_destruction) = @_; if( $in_destruction ) { warn "Unflushed objects cannot be flushed when process is exiting"; } else { $self->flush_to_database } }
We had another similar issue when using Log4perl – if you try to access it from within the DESTROY scope the logger object may already have been destroyed and the process baloons in memory until it is OOM’d. Again just a check if you are in global destruction from within your logging function will handle this just fine.