[unixODBC-support] Memory leak in SQLFetch?

Saunders, Steve ssaunders at docucorp.com
Tue Dec 14 19:40:57 GMT 2004


Cameron, 

If I understand it correctly, the system memory usage on Linux sometimes
includes memory that the OS holds onto even after a program free's it so
that it is in a cache ready to hand back to a program upon subsequent alloc
calls.   

Watching the specific memory usage of a process is the more accurate method
to use to detect memory leaks since this cached memory is not charged to the
process in these counters.  The data for these tools (ps, top, gtop,) comes
from the /proc filesystem which is in memory (like a ram-disk) and it keeps
counters for each process running.

By commenting out the Fetch's you probably just caused less alloc and free
calls to occur so there wasn't as much to cache in the first place.

There are much more savvy guru's that can comment on what the Linux memory
manager does and now it is reflected in the system memory usage counters.  

-Steve

> -----Original Message-----
> From: unixodbc-support-bounces at easysoft.com
> [mailto:unixodbc-support-bounces at easysoft.com]On Behalf Of Craddock,
> Richard C
> Sent: Tuesday, December 14, 2004 2:12 PM
> To: Support for the unixODBC project
> Subject: RE: [unixODBC-support] Memory leak in SQLFetch?
> 
> 
> Hmm...
> 
> I ran it for the full 20160 using the ODBC - ODBC bridge.  
> Here is why I
> concluded that there is a memory leak:
> 
> 1) during testing I noticed that my system memory continued to shrink
> 2) I noticed that the system memory shrinks by ~3 MB every time I run
> the program
> 3) If I comment out the SQLFetch then the system memory does not leak.
> 
> But now I am confused.
> 
> -Cameron
> 
> -----Original Message-----
> From: unixodbc-support-bounces at easysoft.com
> [mailto:unixodbc-support-bounces at easysoft.com] On Behalf Of Saunders,
> Steve
> Sent: Tuesday, December 14, 2004 2:02 PM
> To: 'Support for the unixODBC project'
> Subject: RE: [unixODBC-support] Memorleak in SQLFetch?
> 
> Cameron, 
> 
> Doesn't look like it is growing any at all after it initializes.  The
> RSS
> and VSZ counts stay steady.  
> 
> Did you run it for 101 integrations or more?
> 
> Looking at the system memory usage can be very misleading when trying
> find
> memory leaks.  ps, top and gtop and similar tools that get down to the
> process level are better.
> 
> 
> 
> 
> 
> 
> 
> > -----Original Message-----
> > From: unixodbc-support-bounces at easysoft.com
> > [mailto:unixodbc-support-bounces at easysoft.com]On Behalf Of Craddock,
> > Richard C
> > Sent: Tuesday, December 14, 2004 1:49 PM
> > To: Support for the unixODBC project
> > Subject: RE: [unixODBC-support] Memory leak in SQLFetch?
> > 
> > 
> > Steve,
> > 
> > I performed the test that you asked for and these are the 
> > results.  What
> > do they mean?
> > 
> > -Cameron
> > -------------------------
> > 
> > [root at localhost file_upload]# ps l 5423
> > F   UID   PID  PPID PRI  NI   VSZ  RSS WCHAN  STAT TTY        TIME
> > COMMAND
> > 4     0  5423  3825  16   0  4276 1640 -      S    pts/2      0:00
> > ./testfetch
> > [root at localhost file_upload]# ps l 5423
> > F   UID   PID  PPID PRI  NI   VSZ  RSS WCHAN  STAT TTY        TIME
> > COMMAND
> > 4     0  5423  3825  16   0  4276 1640 -      S    pts/2      0:00
> > ./testfetch
> > [root at localhost file_upload]# ps l 5423
> > F   UID   PID  PPID PRI  NI   VSZ  RSS WCHAN  STAT TTY        TIME
> > COMMAND
> > 4     0  5423  3825  15   0  4276 1668 667797 S    pts/2      0:00
> > ./testfetch
> > [root at localhost file_upload]# ps l 5423
> > F   UID   PID  PPID PRI  NI   VSZ  RSS WCHAN  STAT TTY        TIME
> > COMMAND
> > 4     0  5423  3825  15   0  4276 1668 667797 S    pts/2      0:00
> > ./testfetch
> > [root at localhost file_upload]# ps l 5423
> > F   UID   PID  PPID PRI  NI   VSZ  RSS WCHAN  STAT TTY        TIME
> > COMMAND
> > 4     0  5423  3825  15   0  4276 1668 667797 S    pts/2      0:00
> > ./testfetch
> > [root at localhost file_upload]# ps l 5423
> > F   UID   PID  PPID PRI  NI   VSZ  RSS WCHAN  STAT TTY        TIME
> > COMMAND
> > 4     0  5423  3825  15   0  4276 1668 667797 S    pts/2      0:00
> > ./testfetch
> > [root at localhost file_upload]# ps l 5423
> > F   UID   PID  PPID PRI  NI   VSZ  RSS WCHAN  STAT TTY        TIME
> > COMMAND
> > 4     0  5423  3825  15   0  4276 1668 667797 S    pts/2      0:00
> > ./testfetch
> > [root at localhost file_upload]# ps l 5423
> > F   UID   PID  PPID PRI  NI   VSZ  RSS WCHAN  STAT TTY        TIME
> > COMMAND
> > 4     0  5423  3825  15   0  4276 1668 667797 S    pts/2      0:00
> > ./testfetch
> > [root at localhost file_upload]# ps l 5423
> > F   UID   PID  PPID PRI  NI   VSZ  RSS WCHAN  STAT TTY        TIME
> > COMMAND
> > 4     0  5423  3825  15   0  4276 1668 667797 S    pts/2      0:00
> > ./testfetch
> > [root at localhost file_upload]# ps l 5423
> > F   UID   PID  PPID PRI  NI   VSZ  RSS WCHAN  STAT TTY        TIME
> > COMMAND
> > 4     0  5423  3825  15   0  4276 1668 667797 S    pts/2      0:00
> > ./testfetch
> > [root at localhost file_upload]# ps l 5423
> > F   UID   PID  PPID PRI  NI   VSZ  RSS WCHAN  STAT TTY        TIME
> > COMMAND
> > 4     0  5423  3825  15   0  4276 1668 667797 S    pts/2      0:01
> > ./testfetch
> > [root at localhost file_upload]# ps l 5423
> > F   UID   PID  PPID PRI  NI   VSZ  RSS WCHAN  STAT TTY        TIME
> > COMMAND
> > 4     0  5423  3825  15   0  4276 1668 667797 S    pts/2      0:01
> > ./testfetch
> > [root at localhost file_upload]# ps l 5423
> > F   UID   PID  PPID PRI  NI   VSZ  RSS WCHAN  STAT TTY        TIME
> > COMMAND
> > 4     0  5423  3825  15   0  4276 1668 667797 S    pts/2      0:01
> > ./testfetch
> > [root at localhost file_upload]# ps l 5423
> > F   UID   PID  PPID PRI  NI   VSZ  RSS WCHAN  STAT TTY        TIME
> > COMMAND
> > 4     0  5423  3825  15   0  4276 1668 667797 S    pts/2      0:01
> > ./testfetch
> > [root at localhost file_upload]# ps l 5423
> > F   UID   PID  PPID PRI  NI   VSZ  RSS WCHAN  STAT TTY        TIME
> > COMMAND
> > 4     0  5423  3825  15   0  4276 1668 667797 S    pts/2      0:01
> > ./testfetch
> > [root at localhost file_upload]# ps l 5423
> > F   UID   PID  PPID PRI  NI   VSZ  RSS WCHAN  STAT TTY        TIME
> > COMMAND
> > 4     0  5423  3825  15   0  4276 1668 667797 S    pts/2      0:01
> > ./testfetch
> > [root at localhost file_upload]# ps l 5423
> > F   UID   PID  PPID PRI  NI   VSZ  RSS WCHAN  STAT TTY        TIME
> > COMMAND
> > 4     0  5423  3825  15   0  4276 1668 667797 S    pts/2      0:01
> > ./testfetch
> > [root at localhost file_upload]# ps l 5423
> > F   UID   PID  PPID PRI  NI   VSZ  RSS WCHAN  STAT TTY        TIME
> > COMMAND
> > 4     0  5423  3825  15   0  4276 1668 667797 S    pts/2      0:01
> > ./testfetch
> > [root at localhost file_upload]# ps l 5423
> > F   UID   PID  PPID PRI  NI   VSZ  RSS WCHAN  STAT TTY        TIME
> > COMMAND
> > 4     0  5423  3825  15   0  4276 1668 667797 S    pts/2      0:01
> > ./testfetch
> > [root at localhost file_upload]# ps l 5423
> > F   UID   PID  PPID PRI  NI   VSZ  RSS WCHAN  STAT TTY        TIME
> > COMMAND
> > 4     0  5423  3825  15   0  4276 1668 -      S    pts/2      0:01
> > ./testfetch
> > [root at localhost file_upload]# ps l 5423
> > F   UID   PID  PPID PRI  NI   VSZ  RSS WCHAN  STAT TTY        TIME
> > COMMAND
> > 4     0  5423  3825  15   0  4276 1668 -      S    pts/2      0:01
> > ./testfetch
> > [root at localhost file_upload]# ps l 5423
> > F   UID   PID  PPID PRI  NI   VSZ  RSS WCHAN  STAT TTY        TIME
> > COMMAND
> > 
> > -----Original Message-----
> > From: unixodbc-support-bounces at easysoft.com
> > [mailto:unixodbc-support-bounces at easysoft.com] On Behalf Of 
> Saunders,
> > Steve
> > Sent: Tuesday, December 14, 2004 12:23 PM
> > To: 'Support for the unixODBC project'
> > Subject: RE: [unixODBC-support] Memory leak in SQLFetch?
> > 
> > This display's memory for the system.  
> > 
> > Better data would come from the 'ps l pid' (where pid is the 
> > process id
> > of
> > the program).  I would do it by getting the initial memory 
> > usage of the
> > program by having it pause for user input before it loops, 
> then get a
> > few
> > different samples while it is running, and then have it 
> pause for user
> > input
> > again before it exits and collect the memory usage of the 
> > program before
> > stopping it.  
> > 
> > This will give you the detail about how much memory the 
> > specific program
> > is
> > using.  
> > 
> > This is what I watch for leaks with during load testing our 
> > software on
> > various Unix/Linux platforms.
> > 
> > 
> > 
> > 
> > > -----Original Message-----
> > > From: unixodbc-support-bounces at easysoft.com
> > > [mailto:unixodbc-support-bounces at easysoft.com]On Behalf 
> Of Craddock,
> > > Richard C
> > > Sent: Tuesday, December 14, 2004 12:15 PM
> > > To: Support for the unixODBC project
> > > Subject: RE: [unixODBC-support] Memory leak in SQLFetch?
> > > 
> > > 
> > > Here is the latest information that I have discovered.
> > > 
> > > In order to detect memory leaks I run free -b before and 
> > > after executing
> > > my code.  I assume that the difference between the two was 
> > > allocated but
> > > not freed by the application.
> > > 
> > > If I execute the SQLfetch() loop 101 times I get the 
> following data:
> > > 
> > > [root at localhost file_upload]# free -b
> > >              total       used       free     shared    buffers
> > > cached
> > > Mem:    1326329856  604139520  722190336          0  281374720
> > > 231182336
> > > -/+ buffers/cache:   91582464 1234747392
> > > Swap:   4104372224          0 4104372224
> > > [root at localhost file_upload]# ./testfetch
> > > [root at localhost file_upload]# free -b
> > >              total       used       free     shared    buffers
> > > cached
> > > Mem:    1326329856  604270592  722059264          0  281391104
> > > 231198720
> > > -/+ buffers/cache:   91680768 1234649088
> > > Swap:   4104372224          0 4104372224
> > > 
> > > This shows a reproducible difference of 131072 bytes between runs.
> > > 
> > > If I allow the SQLfetch() loop execute 11 times I get the 
> following:
> > > 
> > > [root at localhost file_upload]# free -b
> > >              total       used       free     shared    buffers
> > > cached
> > > Mem:    1326329856  604532736  721797120          0  281628672
> > > 231239680
> > > -/+ buffers/cache:   91664384 1234665472
> > > Swap:   4104372224          0 4104372224
> > > [root at localhost file_upload]# ./testfetch
> > > [root at localhost file_upload]# free -b
> > >              total       used       free     shared    buffers
> > > cached
> > > Mem:    1326329856  604532736  721797120          0  281636864
> > > 231243776
> > > -/+ buffers/cache:   91652096 1234677760
> > > Swap:   4104372224          0 4104372224
> > > 
> > > There is no difference between the two.  Maybe there is a memory
> > > corruption involved?  Attached is the sql.log file for the 
> > > 100 run case.
> > > 
> > > Thanks,
> > > Cameron
> > > 
> > > 
> > _______________________________________________
> > unixODBC-support mailing list
> > unixODBC-support at easysoft.com
> > http://mail.easysoft.com/mailman/listinfo/unixodbc-support
> > 
> > 
> > 
> > _______________________________________________
> > unixODBC-support mailing list
> > unixODBC-support at easysoft.com
> > http://mail.easysoft.com/mailman/listinfo/unixodbc-support
> > 
> _______________________________________________
> unixODBC-support mailing list
> unixODBC-support at easysoft.com
> http://mail.easysoft.com/mailman/listinfo/unixodbc-support
> 
> 
> 
> _______________________________________________
> unixODBC-support mailing list
> unixODBC-support at easysoft.com
> http://mail.easysoft.com/mailman/listinfo/unixodbc-support
> 



More information about the unixODBC-support mailing list