Skip to content

PDI allocates new memory that is never freed everytime a variable is exposed

Report by Emilio J. Padrón on Slack:

@Romain and I are using PDI in a long numeric simulation and we have noticed an important memory leak. After a minimal mem profiling I think the issue comes from PDI allocating new memory every time a variable is exposed, memory that is not freed ever. Is it normal (I mean, the expected behavior)?

Just with a small proof of concept I can reproduce this memory leak: (our issue is with a Fortran code, but I obtain the same behavior in C)

#include <pdi.h>

int main(int argc, char* argv[]) {
  PDI_init(PC_parse_path("example6.yml"));

  int my_int = 0;
  float my_float = 0;
  int my_loop = 100000;

  while (my_loop > 0) {
    PDI_multi_expose("event_between",
                     "an_int", &my_int, PDI_OUT,
                     "a_float", &my_float, PDI_OUT,
                     NULL);
    my_int += 1;
    my_float += 0.1;
    my_loop -= 1;
  }

  PDI_finalize();

  return 0;
}

with this simple yaml file:

data:
  an_int: int
  a_float: float

plugins:
  trace: ~

This is a snapshot taken with Massif (Valgrind): screenshot-1634454987

And this is the report from Valgrind Memcheck, warning about the memory being allocated every PDI expose that is never freed:

==366206==
==366206== HEAP SUMMARY:
==366206==     in use at exit: 48,006,358 bytes in 300,048 blocks
==366206==   total heap usage: 7,300,528 allocs, 7,000,480 frees, 829,731,027 bytes allocated
==366206==
==366206== 160 bytes in 1 blocks are possibly lost in loss record 23 of 31
==366206==    at 0x4839F2F: operator new(unsigned long) (vg_replace_malloc.c:417)
==366206==    by 0x493C891: PDI::Scalar_datatype::clone_type() const (in /opt/pdi/lib/libpdi.so.1.4.0-alpha.2021-08-04.09fd084)
==366206==    by 0x48EF431: PDI::Data_descriptor_impl::share(void*, bool, bool) (in /opt/pdi/lib/libpdi.so.1.4.0-alpha.2021-08-04.09fd084)
==366206==    by 0x492C397: PDI_share (in /opt/pdi/lib/libpdi.so.1.4.0-alpha.2021-08-04.09fd084)
==366206==    by 0x492C95B: PDI_multi_expose (in /opt/pdi/lib/libpdi.so.1.4.0-alpha.2021-08-04.09fd084)
==366206==    by 0x109209: main (example6.c:12)
==366206==
==366206== 4,346 (104 direct, 4,242 indirect) bytes in 1 blocks are definitely lost in loss record 29 of 31
==366206==    at 0x48397B5: malloc (vg_replace_malloc.c:380)
==366206==    by 0x496D5A5: PC_parse_file (in /opt/pdi/lib/libparaconf.so.0.4.6)
==366206==    by 0x496D75A: PC_parse_path (in /opt/pdi/lib/libparaconf.so.0.4.6)
==366206==    by 0x109186: main (example6.c:4)
==366206==
==366206== 15,999,840 bytes in 99,999 blocks are definitely lost in loss record 30 of 31
==366206==    at 0x4839F2F: operator new(unsigned long) (vg_replace_malloc.c:417)
==366206==    by 0x493C891: PDI::Scalar_datatype::clone_type() const (in /opt/pdi/lib/libpdi.so.1.4.0-alpha.2021-08-04.09fd084)
==366206==    by 0x48EF431: PDI::Data_descriptor_impl::share(void*, bool, bool) (in /opt/pdi/lib/libpdi.so.1.4.0-alpha.2021-08-04.09fd084)
==366206==    by 0x492C397: PDI_share (in /opt/pdi/lib/libpdi.so.1.4.0-alpha.2021-08-04.09fd084)
==366206==    by 0x492C95B: PDI_multi_expose (in /opt/pdi/lib/libpdi.so.1.4.0-alpha.2021-08-04.09fd084)
==366206==    by 0x109209: main (example6.c:12)
==366206==
==366206== 32,000,000 bytes in 200,000 blocks are definitely lost in loss record 31 of 31
==366206==    at 0x4839F2F: operator new(unsigned long) (vg_replace_malloc.c:417)
==366206==    by 0x493C891: PDI::Scalar_datatype::clone_type() const (in /opt/pdi/lib/libpdi.so.1.4.0-alpha.2021-08-04.09fd084)
==366206==    by 0x48EF431: PDI::Data_descriptor_impl::share(void*, bool, bool) (in /opt/pdi/lib/libpdi.so.1.4.0-alpha.2021-08-04.09fd084)
==366206==    by 0x492C397: PDI_share (in /opt/pdi/lib/libpdi.so.1.4.0-alpha.2021-08-04.09fd084)
==366206==    by 0x492CB82: PDI_multi_expose (in /opt/pdi/lib/libpdi.so.1.4.0-alpha.2021-08-04.09fd084)
==366206==    by 0x109209: main (example6.c:12)
==366206==
==366206== LEAK SUMMARY:
==366206==    definitely lost: 47,999,944 bytes in 300,000 blocks
==366206==    indirectly lost: 4,242 bytes in 39 blocks
==366206==      possibly lost: 160 bytes in 1 blocks
==366206==    still reachable: 1,996 bytes in 7 blocks
==366206==         suppressed: 16 bytes in 1 blocks
==366206== Reachable blocks (those to which a pointer was found) are not shown.
==366206== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==366206==
==366206== For lists of detected and suppressed errors, rerun with: -s
==366206== ERROR SUMMARY: 4 errors from 4 contexts (suppressed: 0 from 0)

I am using PDI from the master branch in the repo, specifically commit 09fd0842 from August 4th

Edited by Julien Bigot
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information