java - interpreting jstack output -

-

i have java process loading lot of data bunch of .csv files neo4j database using batchinserter. using:

  • openjdk 7
  • ubuntu 12.04
  • neo4j 2.0 m3

after loading first 164 gb (according ls -lh) folder size stopped increasing process kept running, no memory released, , cpu still @ 100% (all according htop).

the loading process single threaded, jvm using more 1 thread - guess parallelgc.

i'm not sure how diagnose type of problem instructed try jstack, have included output below.

anyone has idea went wrong or has suggestions on how can go diagnosing this?

full thread dump openjdk 64-bit server vm (22.0-b10 mixed mode):  "attach listener" daemon prio=10 tid=0x00007fc3a4001000 nid=0x5636 runnable [0x0000000000000000]    java.lang.thread.state: runnable  "service thread" daemon prio=10 tid=0x00007fcf58123000 nid=0x4545 runnable [0x0000000000000000]    java.lang.thread.state: runnable  "c2 compilerthread1" daemon prio=10 tid=0x00007fcf58120800 nid=0x4544 waiting on condition [0x0000000000000000]    java.lang.thread.state: runnable  "c2 compilerthread0" daemon prio=10 tid=0x00007fcf5811d800 nid=0x4543 waiting on condition [0x0000000000000000]    java.lang.thread.state: runnable  "signal dispatcher" daemon prio=10 tid=0x00007fcf5811b800 nid=0x4542 waiting on condition [0x0000000000000000]    java.lang.thread.state: runnable  "finalizer" daemon prio=10 tid=0x00007fcf580c4800 nid=0x4541 in object.wait() [0x00007fc3f3cfb000]    java.lang.thread.state: waiting (on object monitor)         @ java.lang.object.wait(native method)         - waiting on <0x00007fc4165fc708> (a java.lang.ref.referencequeue$lock)         @ java.lang.ref.referencequeue.remove(referencequeue.java:135)         - locked <0x00007fc4165fc708> (a java.lang.ref.referencequeue$lock)         @ java.lang.ref.referencequeue.remove(referencequeue.java:151)         @ java.lang.ref.finalizer$finalizerthread.run(finalizer.java:177)  "reference handler" daemon prio=10 tid=0x00007fcf580c2000 nid=0x4540 in object.wait() [0x00007fc3f3dfc000]    java.lang.thread.state: waiting (on object monitor)         @ java.lang.object.wait(native method)         - waiting on <0x00007fc4165ffe08> (a java.lang.ref.reference$lock)         @ java.lang.object.wait(object.java:503)         @ java.lang.ref.reference$referencehandler.run(reference.java:133)         - locked <0x00007fc4165ffe08> (a java.lang.ref.reference$lock)  "main" prio=10 tid=0x00007fcf58007800 nid=0x452c runnable [0x00007fcf606b7000]    java.lang.thread.state: runnable         @ java.lang.string.substring(string.java:1960)         @ java.lang.string.subsequence(string.java:1993)         @ java.util.regex.pattern.split(pattern.java:1202)         @ com.ldbc.socialnet.neo4j.csvfilereader.parseline(csvfilereader.java:73)         @ com.ldbc.socialnet.neo4j.csvfilereader.nextline(csvfilereader.java:61)         @ com.ldbc.socialnet.neo4j.csvfilereader.hasnext(csvfilereader.java:33)         @ com.ldbc.socialnet.neo4j.csvfileinserter.bufferlines(csvfileinserter.java:62)         @ com.ldbc.socialnet.neo4j.csvfileinserter.insertallbuffered(csvfileinserter.java:93)         @ com.ldbc.socialnet.neo4j.ldbcsocialneworkneo4jimporter.load(ldbcsocialneworkneo4jimporter.java:79)         @ com.ldbc.socialnet.neo4j.ldbcsocialneworkneo4jimporter.main(ldbcsocialneworkneo4jimporter.java:49)  "vm thread" prio=10 tid=0x00007fcf580ba000 nid=0x453f runnable   "gc task thread#0 (parallelgc)" prio=10 tid=0x00007fcf58012800 nid=0x452d runnable   "gc task thread#1 (parallelgc)" prio=10 tid=0x00007fcf58014800 nid=0x452e runnable   "gc task thread#2 (parallelgc)" prio=10 tid=0x00007fcf58016000 nid=0x452f runnable   "gc task thread#3 (parallelgc)" prio=10 tid=0x00007fcf58018000 nid=0x4530 runnable   "gc task thread#4 (parallelgc)" prio=10 tid=0x00007fcf5801a000 nid=0x4531 runnable   "gc task thread#5 (parallelgc)" prio=10 tid=0x00007fcf5801b800 nid=0x4532 runnable   "gc task thread#6 (parallelgc)" prio=10 tid=0x00007fcf5801d800 nid=0x4533 runnable   "gc task thread#7 (parallelgc)" prio=10 tid=0x00007fcf5801f800 nid=0x4534 runnable   "gc task thread#8 (parallelgc)" prio=10 tid=0x00007fcf58021000 nid=0x4535 runnable   "gc task thread#9 (parallelgc)" prio=10 tid=0x00007fcf58023000 nid=0x4536 runnable   "gc task thread#10 (parallelgc)" prio=10 tid=0x00007fcf58025000 nid=0x4537 runnable   "gc task thread#11 (parallelgc)" prio=10 tid=0x00007fcf58026800 nid=0x4538 runnable   "gc task thread#12 (parallelgc)" prio=10 tid=0x00007fcf58028800 nid=0x4539 runnable   "gc task thread#13 (parallelgc)" prio=10 tid=0x00007fcf5802a800 nid=0x453a runnable   "gc task thread#14 (parallelgc)" prio=10 tid=0x00007fcf5802c800 nid=0x453b runnable   "gc task thread#15 (parallelgc)" prio=10 tid=0x00007fcf5802e000 nid=0x453c runnable   "gc task thread#16 (parallelgc)" prio=10 tid=0x00007fcf58030000 nid=0x453d runnable   "gc task thread#17 (parallelgc)" prio=10 tid=0x00007fcf58032000 nid=0x453e runnable   "vm periodic task thread" prio=10 tid=0x00007fcf5812d800 nid=0x4546 waiting on condition   jni global references: 175  heap  psyounggen      total 11211840k, used 7045440k [0x00007fcb95000000, 0x00007fcf3d160000, 0x00007fcf55000000)   eden space 7045440k, 100% used [0x00007fcb95000000,0x00007fcd43050000,0x00007fcd43050000)   space 4166400k, 0% used [0x00007fce39510000,0x00007fce39510000,0x00007fcf379d0000)     space 4035328k, 0% used [0x00007fcd43050000,0x00007fcd43050000,0x00007fce39510000)  psoldgen        total 31457280k, used 31300002k [0x00007fc415000000, 0x00007fcb95000000, 0x00007fcb95000000)   object space 31457280k, 99% used [0x00007fc415000000,0x00007fcb8b668b98,0x00007fcb95000000)  pspermgen       total 21248k, used 7487k [0x00007fc40aa00000, 0x00007fc40bec0000, 0x00007fc415000000)   object space 21248k, 35% used [0x00007fc40aa00000,0x00007fc40b14ff48,0x00007fc40bec0000)

in particular, can give little insight on these heap values or suggest read them?

heap  psyounggen      total 16348096k, used 12660905k [0x00007f833a560000, 0x00007f87639c0000, 0x00007f8765000000)   eden space 15282432k, 82% used [0x00007f833a560000,0x00007f863f18a688,0x00007f86df1a0000)   space 1065664k, 0% used [0x00007f86df1a0000,0x00007f86df1a0000,0x00007f8720250000)     space 1036288k, 0% used [0x00007f87245c0000,0x00007f87245c0000,0x00007f87639c0000)  paroldgen       total 34952576k, used 34903343k [0x00007f7ae5000000, 0x00007f833a560000, 0x00007f833a560000)   object space 34952576k, 99% used [0x00007f7ae5000000,0x00007f833754bd90,0x00007f833a560000)  pspermgen       total 21248k, used 7130k [0x00007f7adfe00000, 0x00007f7ae12c0000, 0x00007f7ae5000000)   object space 21248k, 33% used [0x00007f7adfe00000,0x00007f7ae04f6860,0x00007f7ae12c0000)

it looks process out of memory:

eden space 7045440k, 100% used object space 31457280k, 99% used

in case gc may run continiosly, trying free memory, consuming alomist cpu. application logic thread under heavy starvation. may either add more memory via -xmx jvm option or profile application. profiling via jvisualvm, jprofiler or other tool give important runtime data:

  • memory allocaton dynamics. if memory consumption increasing time you're have memory leak
  • heap contents find out occupies momory
  • gc statistics

based on data you'll certanly able resolve issue.


Comments

Post a Comment

Popular posts from this blog

css - Which browser returns the correct result for getBoundingClientRect of an SVG element? -

-
this svg contains rect overflows svg element: <svg id='svg' width='10' height='10'> <rect x='-10' y='-10' width='30' height='30'/> </svg> chrome 28 , opera 12 return getboundingclientrect() svg element width , height of 10. firefox 23 reports width , height of 30. correct? jsfiddle the relevant spec cssom , delegates svg spec if svg element not "have associated css layout box". haven't found definition of "having associated css layout box", correct result seem hinge on definition, getbbox returns 30x30 rect in browsers. this firefox bug fixed now, fix in firefox nightlies , should make through firefox 33 released on 14 october 2014. see bug 530985 details.
Read more

gcc - Calling fftR4() in c from assembly -

-
i trying asm fft implementation www.embeddedsignals.com work. running make gives error: undefined reference fftr4(short*, short*, int) at top of asm have .syntax unified .thumb .global fftr4 .type fftr4, %function and there #define's associate c functions arguments registers followed macros fftr4 function uses. after that, fftr4 called .thumb_func fftr4: stmfd sp!, {r4-r11, lr} mov tmp0, #0 // bit reversed counter mov tmp1, #0 .word 0xf3a2fa92 //rbit r3,r2 //rbit r,n lsl r,#3 and c code follows: void fftr4(short *y, short *x, int n); short x[512]; short y[512]; int i; void loop() { (i=0;i<512;i++) x[i]=0; (i=0;i<512;i=i+8) { x[i+0]=16384; x[i+2]=16384; x[i+4]=-16384; x[i+6]=-16384;} fftr4(y, x, 256); serialusb.println(y[128]); serialusb.println(y[129]); serialusb.println(y[384]); serialusb.println(y[385]); // expected results: //y[128]is 8191; y[129] -8190 y[...
Read more

.htaccess - Matching full URL in RewriteCond -

-
i'm trying block post access site (and return 403), except 2 ip's , 1 url ( http://mydomain.com/about-us/contact-us/ ). here's have: rewritecond %{request_method} post rewritecond %{remote_addr} !=1.2.3.4 rewritecond %{remote_addr} !=4.3.2.1 rewritecond %{request_uri} !^http://mydomain.com/about-us/contact-us/$ rewriterule ^ / [f] these rules allowing access ip properly, contact page still not accessible. i'm not sure i'm using request_uri correctly? i've tried variations relative urls rewritecond %{request_uri} !^/about-us/contact-us$ , rewritecond %{request_uri} !^/about-us/contact-us/$ without luck. what doing wrong please? it looks me rule on file name specific. if relax bit, should work. made few other esthetic tweaks. give try: rewritecond %{request_method} post rewritecond %{remote_addr} !^1.2.3.4 rewritecond %{remote_addr} !^4.3.2.1 rewritecond %{request_uri} !about-us/contact-us rewriterule ^ - [f]
Read more