PDI allocates new memory that is never freed everytime a variable is exposed
Report by Emilio J. Padrón on Slack:
@Romain and I are using PDI in a long numeric simulation and we have noticed an important memory leak. After a minimal mem profiling I think the issue comes from PDI allocating new memory every time a variable is exposed, memory that is not freed ever. Is it normal (I mean, the expected behavior)?
Just with a small proof of concept I can reproduce this memory leak: (our issue is with a Fortran code, but I obtain the same behavior in C)
#include <pdi.h>
int main(int argc, char* argv[]) {
PDI_init(PC_parse_path("example6.yml"));
int my_int = 0;
float my_float = 0;
int my_loop = 100000;
while (my_loop > 0) {
PDI_multi_expose("event_between",
"an_int", &my_int, PDI_OUT,
"a_float", &my_float, PDI_OUT,
NULL);
my_int += 1;
my_float += 0.1;
my_loop -= 1;
}
PDI_finalize();
return 0;
}
with this simple yaml file:
data:
an_int: int
a_float: float
plugins:
trace: ~
And this is the report from Valgrind Memcheck, warning about the memory being allocated every PDI expose that is never freed:
==366206==
==366206== HEAP SUMMARY:
==366206== in use at exit: 48,006,358 bytes in 300,048 blocks
==366206== total heap usage: 7,300,528 allocs, 7,000,480 frees, 829,731,027 bytes allocated
==366206==
==366206== 160 bytes in 1 blocks are possibly lost in loss record 23 of 31
==366206== at 0x4839F2F: operator new(unsigned long) (vg_replace_malloc.c:417)
==366206== by 0x493C891: PDI::Scalar_datatype::clone_type() const (in /opt/pdi/lib/libpdi.so.1.4.0-alpha.2021-08-04.09fd084)
==366206== by 0x48EF431: PDI::Data_descriptor_impl::share(void*, bool, bool) (in /opt/pdi/lib/libpdi.so.1.4.0-alpha.2021-08-04.09fd084)
==366206== by 0x492C397: PDI_share (in /opt/pdi/lib/libpdi.so.1.4.0-alpha.2021-08-04.09fd084)
==366206== by 0x492C95B: PDI_multi_expose (in /opt/pdi/lib/libpdi.so.1.4.0-alpha.2021-08-04.09fd084)
==366206== by 0x109209: main (example6.c:12)
==366206==
==366206== 4,346 (104 direct, 4,242 indirect) bytes in 1 blocks are definitely lost in loss record 29 of 31
==366206== at 0x48397B5: malloc (vg_replace_malloc.c:380)
==366206== by 0x496D5A5: PC_parse_file (in /opt/pdi/lib/libparaconf.so.0.4.6)
==366206== by 0x496D75A: PC_parse_path (in /opt/pdi/lib/libparaconf.so.0.4.6)
==366206== by 0x109186: main (example6.c:4)
==366206==
==366206== 15,999,840 bytes in 99,999 blocks are definitely lost in loss record 30 of 31
==366206== at 0x4839F2F: operator new(unsigned long) (vg_replace_malloc.c:417)
==366206== by 0x493C891: PDI::Scalar_datatype::clone_type() const (in /opt/pdi/lib/libpdi.so.1.4.0-alpha.2021-08-04.09fd084)
==366206== by 0x48EF431: PDI::Data_descriptor_impl::share(void*, bool, bool) (in /opt/pdi/lib/libpdi.so.1.4.0-alpha.2021-08-04.09fd084)
==366206== by 0x492C397: PDI_share (in /opt/pdi/lib/libpdi.so.1.4.0-alpha.2021-08-04.09fd084)
==366206== by 0x492C95B: PDI_multi_expose (in /opt/pdi/lib/libpdi.so.1.4.0-alpha.2021-08-04.09fd084)
==366206== by 0x109209: main (example6.c:12)
==366206==
==366206== 32,000,000 bytes in 200,000 blocks are definitely lost in loss record 31 of 31
==366206== at 0x4839F2F: operator new(unsigned long) (vg_replace_malloc.c:417)
==366206== by 0x493C891: PDI::Scalar_datatype::clone_type() const (in /opt/pdi/lib/libpdi.so.1.4.0-alpha.2021-08-04.09fd084)
==366206== by 0x48EF431: PDI::Data_descriptor_impl::share(void*, bool, bool) (in /opt/pdi/lib/libpdi.so.1.4.0-alpha.2021-08-04.09fd084)
==366206== by 0x492C397: PDI_share (in /opt/pdi/lib/libpdi.so.1.4.0-alpha.2021-08-04.09fd084)
==366206== by 0x492CB82: PDI_multi_expose (in /opt/pdi/lib/libpdi.so.1.4.0-alpha.2021-08-04.09fd084)
==366206== by 0x109209: main (example6.c:12)
==366206==
==366206== LEAK SUMMARY:
==366206== definitely lost: 47,999,944 bytes in 300,000 blocks
==366206== indirectly lost: 4,242 bytes in 39 blocks
==366206== possibly lost: 160 bytes in 1 blocks
==366206== still reachable: 1,996 bytes in 7 blocks
==366206== suppressed: 16 bytes in 1 blocks
==366206== Reachable blocks (those to which a pointer was found) are not shown.
==366206== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==366206==
==366206== For lists of detected and suppressed errors, rerun with: -s
==366206== ERROR SUMMARY: 4 errors from 4 contexts (suppressed: 0 from 0)
I am using PDI from the master branch in the repo, specifically commit 09fd0842 from August 4th
Edited by Julien Bigot