tag:blogger.com,1999:blog-42663886009639913112024-03-05T23:42:54.746+00:00Collecting the garbage ...Thoughts, ramblings, and musings from a Java developer. Like all the cool kids, I update this embarrassingly rarely. My postings here reflect my personal views and do not necessarily represent those of my employer IBM.Holly Cumminshttp://www.blogger.com/profile/08367229042550909996noreply@blogger.comBlogger15125tag:blogger.com,1999:blog-4266388600963991311.post-43329291318060681492016-07-19T16:39:00.000+01:002016-07-19T16:39:09.323+01:00New presentation - "Building Stuff for Fun and Profit - Confessions from a Life in Cables and Code"I recently spoke at <a href="http://www.buildstuff.com.ua/odessa/">BuildStuff Odessa</a>, and had a great time. It's the only conference where I've seen developers on sun loungers, by the pool, working away at their laptop.
<div class="separator" style="clear: both; text-align: center;"><a href="https://pbs.twimg.com/media/Cm1ETWuWIAAylFt.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://pbs.twimg.com/media/Cm1ETWuWIAAylFt.jpg" width="320" height="240" /></a></div>
I've tried a new experiment with my slides, and are publishing them with speaker notes, so that they make more sense on Slideshare:
<div class="separator" style="clear: both; text-align: center;">
<iframe src="//www.slideshare.net/slideshow/embed_code/key/FYBGDIUPiQEVNc" width="595" height="485" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" style="border:1px solid #CCC; border-width:1px; margin-bottom:5px; max-width: 100%;" allowfullscreen> </iframe> <div style="margin-bottom:5px"> <strong> <a href="//www.slideshare.net/HollyCummins/making-stuff-for-fun-and-profit-confessions-from-a-life-in-code-and-cables" title="Building Stuff for Fun and Profit - confessions from a life in code and cables" target="_blank">Building Stuff for Fun and Profit - confessions from a life in code and cables</a> </strong> de <strong><a href="//www.slideshare.net/HollyCummins" target="_blank">Holly Cummins</a></strong> </div>
</div>
Unfortunately, this makes the actual slide tiny. I experimented with Slideshare's built-in speaker note view, and that hid the notes in a big list. Older versions of Keynote apparently have a more compact speaker note export, but unless that gets re-introduced, it seems I'm stuck with the rather inelegant format.Holly Cumminshttp://www.blogger.com/profile/02737477241094504981noreply@blogger.com0tag:blogger.com,1999:blog-4266388600963991311.post-24389688428495633992016-07-04T21:29:00.001+01:002016-07-04T21:35:19.234+01:00Six Myths and Paradoxes of Garbage Collection - MSc ThesisI'm not sure why I didn't do this until now, but I've made my MSc dissertation, <a href="https://drive.google.com/file/d/0Byc09jy2RF1YVGpZZDFfTDNwNlE/view">"Six Myths and Paradoxes of Garbage Collection,"</a> available online. It was written in 2007, so in tech terms, it can be considered quaintly antique - back when it was written, there was still a debate about whether garbage collection was a good idea. StackOverflow hadn't been invented yet, and C++ was the language to beat.<br />
<br />
What's the dissertation about? Well, this is the abstract:<br />
<br />
<div style="margin: 0cm 1cm;text-align: justify;">
Many myths and paradoxes surround garbage collection. The first myth is that garbage collection is
only suitable for the incompetent, unskilled, or lazy. In fact garbage collection offers many architec-
tural and software engineering advantages, even to the skilled developer. The second myth is that
garbage collection is all about about collecting garbage. Garbage collectors also include an allocation
component, which, along with their powers of object rearrangement, can make a significant difference
to application performance. Thirdly, criticisms of garbage collection often focus on the pause times,
and responses to these criticisms often focus exclusively on reducing pause times, in the mistaken belief
that small pause times guarantee good application response times. Pause times are also often used as
a metric of general application performance, and an increase in pause times is taken as an indicator
of worsened performance, when in fact the opposite the opposite is often true. Paradoxically, even
the total amount of time spent paused for garbage collection is not a good predictor of the impact of
garbage collection on application performance. Finally, the sixth myth is that garbage collection has
a disastrous performance impact. While garbage collection can hurt application performance, it can
also help application performance to the point where it exceeds the performance with manual memory
management."
</div>
<br />
Although some of the arguments it makes are no longer necessary, there's a lot I still like about the paper. It's got equations, it's got queueing theory, it's got dissections of usenet trolling about GC, it's got diagrams, and - the best bit - photos of garbage bins. You can read the full 84 pages <a href="https://drive.google.com/open?id=0Byc09jy2RF1YVGpZZDFfTDNwNlE">here</a>; it's licensed under <a href="https://creativecommons.org/licenses/by-nc-sa/4.0/">Creative Commons Non-Commercial Share Alike</a>.
Holly Cumminshttp://www.blogger.com/profile/02737477241094504981noreply@blogger.com0tag:blogger.com,1999:blog-4266388600963991311.post-591400726458488412015-02-25T14:38:00.000+00:002015-02-25T20:28:15.885+00:00Interacting with headless computers (or "How to not keep losing your raspberry pi on the network")I spend a fair amount of time playing with Raspberry Pis and other ARM-based boxes, like pcDuinos and Utilite. (If my boss is reading this, I'm not "playing", I'm "working".) I still find it amazing how small, cheap, and powerful these devices are.<br />
<br />
One thing it's taken me a while to work out is how to connect to the computer. It's kind of a prerequisite for doing anything more interesting, like embedding it in a hat or a ball, or just ... using it. It should be so easy, right? A lot of set-up guides start with you plugging the processor to an HDMI monitor and USB keyboard. This isn't so practical if I'm out and about with the computer, because I'm not going to be lugging an HDMI monitor with me. Even at home, it's not foolproof - my new pcDuino doesn't seem able to display a picture over HDMI to any of the (six) monitors, TVs, or projectors I tried. At that point it's not so much "headless" as "decapitated".<br />
<br />
Connecting by ssh requires less bulky hardware, but it has its own challenges: what on earth is the ip address of this thing? I used to ensure I always knew the ip address by nailing it down. I'd connect the pi to my laptop with an ethernet cable (no need for a fancy crossover one with a modern computer), configure the pi with a static ip address (for example, 192.168.2.10), and then configure my laptop's ethernet interface to think it was 192.168.2.1. This ensured I could easily connect to my pi, but it did require some initial set up of the network. For my totally inaccessible pcDuino, this is no good. Even for the pi, it's not ideal - the static ip address means then even if I plug it into my router, it's totally divorced from the "real" internet. More seriously, I've had a few panics where I've loaned the pi out and then found I was unable to connect to it when I got it back. (Curse you, nameless colleague who re-configured the pi to use dhcp and didn't tell me!) The static ip settings also left my laptop in a pretty messed up state.<br />
<br />
I've now found a new system which avoids some of these disadvantages. I don't touch the ethernet configuration of the arm box, so it can stay using dhcp. I connect it to my laptop with an ethernet cable but I configure my mac to share its internet connection from wi-fi to ethernet. This OS magic enables the two computers to see each other without either having to have crazy network settings.<br />
<br />
I then use nmap to find the pi on the network, as follows:<br />
<br />
<div style="background-color: #f2f1f0; color: #232323; font-family: Courier; font-size: 14px;">
<span style="letter-spacing: 0.0px;">sudo nmap -sn 192.168.1.0/24 </span></div>
<br />
To work out which ip address to scan, I run <span style="background-color: #f2f1f0; color: #232323; font-family: Courier; font-size: 14px;">ifconfig </span>and use the ethernet access of the <span style="background-color: #f2f1f0; color: #232323; font-family: Courier; font-size: 14px;">bridge100 </span>network interface (but change the last digit to 0). The <span style="background-color: #f2f1f0; color: #232323; font-family: Courier; font-size: 14px;">sudo</span> isn't necessary, but it makes nmap show the MAC addresses. This isn't so useful for the pcDuino, which doesn't have a fixed MAC address, but it really helps identify the pi, which will always have a MAC address in the b8:27:eb range.<br />
<br />
<b>Edit - other options: </b>I haven't tried these, but twitter has spoken! If you have a flock (or even a small handful) of identical devices, including <a href="https://gist.github.com/andysc/5605183">ipspeak</a> in the bootable image bypasses the need for any mmap sleuthing. You can plug headphones into the device and it will read out its ip address every thirty seconds. Alternatively, if you have individually configured devices, <a href="http://www.avahi.org/">avahi</a> allows them to be addressed using a memorable name.<br />
<br />
<b>Acknowledgements: </b>There are many similar guides on the internet. I've posted this here mostly in the hope it will be useful notes to my future self, since I've independently rediscovered my optimum flow about four times. I figured out the nmap commands from a relatively recent post at <a href="http://raspberrypi.stackexchange.com/questions/13936/find-raspberry-pi-address-on-local-network">stackoverflow</a>, and the idea of using network sharing to connect my pi to my laptop without wrecking my laptop's settings is from <a href="http://www.element14.com/community/thread/26532/l/quick-start-of-pcduino-without-a-hdmi-monitor-and-serial-debug-cable?displayFullThread=true">alvin_jin</a>.<br />
<br />
<br />Holly Cumminshttp://www.blogger.com/profile/02737477241094504981noreply@blogger.com0tag:blogger.com,1999:blog-4266388600963991311.post-44383216055109649052012-09-27T10:00:00.000+01:002012-09-27T12:30:03.228+01:00Java basics: converting a collection to a string<span style="font-family: inherit;">If you look up how to write a map or a list out as a string, you can find lots of complicated answers involving loops and even XMLEncoders. It doesn't have to be that hard. The easiest way to convert a Map into a nice string?</span><br />
<br />
<br />
<div>
<span style="font-family: Courier New, Courier, monospace;">String niceString = Arrays.toString(<span style="color: #0431c3;">map</span>.entrySet().toArray());</span></div>
<div>
<br /></div>
<span style="font-family: inherit;">It's even easier for a list:</span><br />
<div style="font-family: Monaco;">
<br /></div>
<div style="font-family: Monaco;">
</div>
<span style="font-family: Courier New, Courier, monospace;">String niceString = Arrays.toString(<span style="color: #0431c3;">list</span>.toArray());</span><br />
<br />
Ta-daaa!<br />
<br />
<br />
<br />
<br />Holly Cumminshttp://www.blogger.com/profile/02737477241094504981noreply@blogger.com1tag:blogger.com,1999:blog-4266388600963991311.post-15245039053717760442012-09-26T15:37:00.001+01:002012-09-26T15:37:08.357+01:00Enterprise OSGi in Action<div style="text-align: center;">
<br /></div>
<div style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: x-small;">We're getting really close now ... </span></div>
<div style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="text-align: center;">
<b><span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><a href="http://www.manning.com/cummins/">Enterprise OSGi in Action</a></span></b></div>
<b><br /></b>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://www.manning.com/cummins/cummins_cover150.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://www.manning.com/cummins/cummins_cover150.jpg" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div style="background-color: white; font-family: Arial, sans-serif; font-size: 13px; text-align: -webkit-auto;">
Are you looking for a way to write an OSGi web application? Or get JPA persistence working in OSGi? Or just group your OSGi bundles into something a bit more granular? Do you need a guide to best practices for OSGi build and test? </div>
<div style="background-color: white; font-family: Arial, sans-serif; font-size: 13px; text-align: -webkit-auto;">
<br /></div>
<div style="background-color: white; font-family: Arial, sans-serif; font-size: 13px; text-align: -webkit-auto;">
Modern enterprise applications must be scalable, maintainable, and modular. Unfortunately, by itself Java EE doesn't do modularity very well. The Enterprise OSGi model enforces simple rules to make Java better at modularity. And now, projects such as Apache Aries and Geronimo provide pluggable components that make it easier than ever to use OSGi's mature modularity system in your own enterprise applications.</div>
<div style="background-color: white; font-family: Arial, sans-serif; font-size: 13px; text-align: -webkit-auto;">
<br /></div>
<div style="background-color: white; font-family: Arial, sans-serif; font-size: 13px; text-align: -webkit-auto;">
<i>Enterprise OSGi in Action</i> is a hands-on guide for developers using OSGi to build the next generation of enterprise Java applications. By presenting relevant examples and case studies, it guides the reader through the maze of new standards and projects. Armed with this knowledge, readers will learn how to build and deploy enterprise OSGi applications, use automatic dependency provisioning, declaratively provide enterprise qualities of service to their business logic, and make use of the Java EE technologies they know and love in a familiar way.</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: x-small;">Manning have a <a href="http://www.manning-sandbox.com/forum.jspa?forumID=762">reader forum</a> where you can ask Tim and me questions (or I'm on twitter <a href="https://twitter.com/holly_cummins">@holly_cummins</a>).</span></div>
<b><br /></b>
<br />
<br />Holly Cumminshttp://www.blogger.com/profile/02737477241094504981noreply@blogger.com0tag:blogger.com,1999:blog-4266388600963991311.post-65864612803403523482009-06-15T17:23:00.002+01:002009-06-15T17:33:43.131+01:00Health Center 1.0 releasedLast week was a big week for the Health Center team. Version 1.0 of the Health Center was released. Full installation instructions are available from the <a href="http://www.ibm.com/developerworks/java/jdk/tools/healthcenter/">Health Center homepage</a>. It now works with IBM Support Assistant and is suitable for use in production (with recent IBM JVMs). A method profiler that's got a low enough overhead to be used in production and left on all the time is a pretty special thing, I think. One which is only part of a tool which also assesses system stability, triages a range of performance problems, and makes performance tuning accessible to non-experts is even more special, I think.<br /><br />To celebrate the whole team baked and brought in carrot cake, brownies, chocolate chip cookies, lemon cake, rocky road squares, and flapjacks. We even had balloons. It was very nice but I've never eaten so much at work and I think we were all ready to explode by the afternoon - in future I think we should delegate and have only <i>half</i> the team bake per release.Holly Cumminshttp://www.blogger.com/profile/02737477241094504981noreply@blogger.com0tag:blogger.com,1999:blog-4266388600963991311.post-33974177510385641312009-05-12T10:00:00.007+01:002009-05-12T15:08:53.682+01:00How to interpret a method profileIn a <a href="http://hollycummins.blogspot.com/2009/05/how-do-you-solve-performance-problem.html">previous post</a>, I described the general methodology I use to diagnose performance problems. <br /><br />Once an application has been identified as CPU-bound, either by using the <a href="https://www14.software.ibm.com/iwm/web/cc/earlyprograms/ibm/ibmmdtjhc/">Health Center</a> or CPU monitoring, the next step is to figure out <i>what</i> is eating CPU. In a Java application, this will usually be Java code, but it could be native code. Profiling native code usually requires platform specific tools; on linux, I use <a href="http://perfinsp.sourceforge.net/tprof.html">tprof</a>. Profiling Java code is a lot easier, and is more likely to yield big performance improvements, so I usually start with a Java profile and only profile native code if I didn't get anywhere with the Java profile. For Java profiling, I use the Health Center. It's got a few advantages, one of which is that there's no bytecode instrumentation needed, there's no need to specify only a few packages to profile, and the overhead is very low, so it won't affect the performance characteristics of what you're trying to profile.<br /><br />So what does a method profile tell you? Simply put, it tells you what your application is spending its time doing. More precisely, it tells you what code your application is spending its time running - it doesn't tell you when your application is waiting on a lock instead of running your code, and it doesn't tell you when the JVM is collecting garbage instead of running your code. Assuming locking and GC aren't the cause of the performance problem (see <a href="http://hollycummins.blogspot.com/2009/05/how-do-you-solve-performance-problem.html">triaging a performance problem</a>), the method profile will give you the information you need to make your application go faster. <br /><br />The application is doing too much work, and that's slowing it down. Your aim in performance tuning is to make the application do less work. There are lots of ways to make code more efficient. Sometimes people start performance tuning by code inspection - they read through the code base looking for obvious inefficiencies. I've done this myself lots of times, but it's not a particularly efficient technique. Say I find a method which is pretty carelessly implemented, and I double its speed with a bit of refactoring. Then I triumphantly re-run my application, only to discover nothing's changed. What's going on? The problem is that a big performance improvement on a method which is rarely called isn't going to change much of anything. For example, if I double the speed of a method which uses 0.5% of my CPU time, I've sped my application up by an imperceptible 0.25%. If, on the other hand, I shave 10% of the time of a method which is using 20% of my CPU, my application will go 2% faster. So the first rule of performance tuning is to optimise the methods at the top of the profile and ignore the ones near the bottom. <br /><br />This is example from a method profiler, in this case the one in the Health Center. One method is clearly using more CPU than the rest, and so it's coloured red. In this case, 60% of the time the JVM checked what the application was doing, it was executing the FireworkParticle.animate() method. This is what's shown by the left-hand 'Self' column. The 'Tree' column on the right shows how much time the application spent in both the animate() method and its descendants. Some profilers call this column 'descendants' instead. Usually the Self figures are more useful for optimising an application.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiHtwToNucasBE5U3m2-xk1SQI3uDiOYamBQv5Xh4N5kbZ_k4VBXtrGCrRMxg7kVJ3dZUvXfpoJWN9NrHIcINwGc4Kmlvc3H62hPNNQ916rB5yPjXcj8E11t_hR5CCOv8cvlnyUCADOqKkJ/s1600-h/healthcenterprofile.gif"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 500px; height: 210px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiHtwToNucasBE5U3m2-xk1SQI3uDiOYamBQv5Xh4N5kbZ_k4VBXtrGCrRMxg7kVJ3dZUvXfpoJWN9NrHIcINwGc4Kmlvc3H62hPNNQ916rB5yPjXcj8E11t_hR5CCOv8cvlnyUCADOqKkJ/s400/healthcenterprofile.gif" border="0" alt=""id="BLOGGER_PHOTO_ID_5334592138110185490" /></a><br /><br />What makes a method appear near the top of a method profile? It's taking up a lot of CPU time, but why? There are two reasons; either the method is being called too often, or the method is doing too much work when it's called. Sometimes it happens that a method really is doing the right amount of work the right number of times, but this is usually only the case after a fair amount of work. In their natural state, most programs can - and should - contain inefficiencies. (Remember that <a href="http://fresh.homeunix.net/~luke/misc/knuth-turingaward.pdf">premature optimisation is the root of all evil</a>.) <br /><br />Some profilers can distinguish between a method which is called several times, and one which is called once and then spends a long time executing, but many cannot. The reason is that some profilers operate by <i>tracing</i> - that is, recording every entry and exit of a method. This gives very precise information, but usually carries a fairly heavy performance cost. The IBM JVM can be <a href="http://www.ibm.com/developerworks/java/library/j-ibmjava5/">configured with launch parameters</a> to count or time method executions, but it's only advisable to do this for a restricted subset of methods. An alternate method of collecting profiling information is to <i>sample</i> - that is, check periodically what method is executing. This is much less expensive but doesn't give as much detail as tracing profilers. The Health Center uses method sampling already built into the JVM to allow profiling with extremely low overhead. <br /><br />Often it will be obvious when inspecting a hot method if it's being called frequently or is slow to run. Code with loops, particularly nested loops, is probably expensive to run. Code which doesn't seem to do much but which is at the top of a profile is probably being called a lot. This leads neatly to the next steps in optimization: eliminate loops and do less work inside loops for expensive methods, and call inexpensive method less frequently. <br /><br />How do you go about making sure a method is called less? Method profilers which also record stack traces can make calling method less pretty easy. For example, this is the output of the Health Center, showing where calls to one of the top methods in the profile have come from: <br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjMqn8L0iP1ZQBkTbQq9DxApftZIsK1WauXe-_NhDWYE-7JzEpmKH26P0N-oHHkhiTIv9DyzRlSMZyvdBRRBiBRk155IR0eug05U-BuMzXdLWWBrjOFzPKHhXXlHreFTAtU1OSL9s3lnBlp/s1600-h/healthcenterinvocationpaths.gif"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 400px; height: 148px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjMqn8L0iP1ZQBkTbQq9DxApftZIsK1WauXe-_NhDWYE-7JzEpmKH26P0N-oHHkhiTIv9DyzRlSMZyvdBRRBiBRk155IR0eug05U-BuMzXdLWWBrjOFzPKHhXXlHreFTAtU1OSL9s3lnBlp/s400/healthcenterinvocationpaths.gif" border="0" alt=""id="BLOGGER_PHOTO_ID_5334595042640234610" /></a><br /><br />In this case, 98% of the time the doSomeWork() method was sampled, it was animate() that called it. 2% of the time, it was draw() that called it. In this case, the next step is to inspect the animate() method and see why it's calling doSomeWork(). Often, at least in the first passes of optimisation, most of the calls to the top method are totally unnecessary and can be trivially eliminated.Holly Cumminshttp://www.blogger.com/profile/08367229042550909996noreply@blogger.com0tag:blogger.com,1999:blog-4266388600963991311.post-88793207558454047252009-05-11T14:41:00.007+01:002009-05-12T16:03:50.652+01:00How do you solve a performance problem?I was talking to a friend today about a performance problem he was seeing, and he explained that he wasn't entirely sure how to fix it. Performance tuning can be very skilled, but the basic principles are straightforward and should be accessible to every Java developer. <br /><br />This is the methodology I recommend. <br /><br />All performance problems are caused by a limited resource. Your job as a performance analyst (or developer who's suddenly required to be a performance analyst) is to identify what resource is limited - what's the bottleneck for this application? Often after fixing the first bottleneck, a second bottleneck will become apparent - the process of performance tuning is the process of eliminating bottlenecks, one by one, until the performance is good enough for you and your stakeholders.<br /><br />Computational resources fall into a few basic categories. Different people count them differently, but I like to think of four types of resource: the CPU, memory, I/O, and locks. <br /><br />A CPU-bound application can't get enough processor time to complete its work. A memory-bound application needs more memory than is available. An I/O-bound application is trying to do I/O faster than the system can handle. Finally, a lock-bound application is being held up by the fact that multiple threads are contending on the same locks. 'Lock-bound' isn't a terribly common term, but I think it's really important to consider lock contention when analyzing performance problems. As systems become more and more parallel, synchronization are increasingly a limiting factor in the scalability of the system. <br /><br />So how do you identify which of these resources is the cause of the hold-up? Some heuristics can help as a first step. If the CPU is at or near 100%, the CPU is likely to be the culprit. If the CPU isn't near 100%, it's probably locking or I/O. The rule of thumb becomes a bit muddier when it comes to memory. Sometimes memory problems can show up as low CPU usage, because the system is waiting on a heap lock (which is properly locking) or physical memory access - paging in the worst case (technically, I/O). However, in a garbage collected system, excessive memory consumption will often manifest as lots of processing time being spent in the garbage collector (CPU). <br /><br />Tools can help turn these fairly fuzzy heuristics into a more precise diagnosis. There are lots of tools available, both free and not-free. My favourite tool is <a href="https://www14.software.ibm.com/iwm/web/cc/earlyprograms/ibm/ibmmdtjhc/">IBM's Health Center</a>. This is probably because I'm the technical lead for the Health Center, so I think what it does is pretty sensible and cool. :) The Health Center is free, but it can only monitor IBM JVMs. I do think it's one of the best ways to investigate garbage collection, locking, and collect method profiles from an IBM JVM.<br /><br />The Health Center tries to automate the process of identifying the root cause of a performance problem. The front page shows a dashboard with a bunch of status indicators. If one of them is red or orange, that's a pretty good indicator of where to start tweaking the performance. The one area the Health Center doesn't cover at the moment is I/O. It can identify garbage collection and locking bottlenecks pretty accurately, and the method profiler can identify the root cause of much excessive CPU usage. <br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi081ehDoonjqsPVTDGoUhzbO1Qzw6GK3zm5Am7CR39mbBreBi8CIUXKJsxQPIkouZ_QFjFzID-ADx5fUyb6t_G7hEXikckv2JFdBeCRcEr0K3CnS-Bq751t5ADttUWPMX3kRWR1LWDjx7H/s1600-h/healthcenterfrontpage.gif"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 400px; height: 271px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi081ehDoonjqsPVTDGoUhzbO1Qzw6GK3zm5Am7CR39mbBreBi8CIUXKJsxQPIkouZ_QFjFzID-ADx5fUyb6t_G7hEXikckv2JFdBeCRcEr0K3CnS-Bq751t5ADttUWPMX3kRWR1LWDjx7H/s400/healthcenterfrontpage.gif" border="0" alt=""id="BLOGGER_PHOTO_ID_5334569732516554114" /></a><br /><br />So let's say you've used the Health Center (or your tool of choice) and you're seeing some red crosses. What next? In the Health Center, clicking on the link next to a red cross will bring up more information and more detailed recommendations about how to fix the problem. In later posts I'll give a bit more background about how to go about fixing locking issues, memory issues, and CPU issues.<br /><br />Forward links:<br /><ul><br /><li> <a href="http://hollycummins.blogspot.com/2009/05/how-to-interpret-method-profile.html">How to interpret a method profile</a><br /></li><br /></ul>Holly Cumminshttp://www.blogger.com/profile/08367229042550909996noreply@blogger.com0tag:blogger.com,1999:blog-4266388600963991311.post-3470071973480010682009-05-08T14:30:00.007+01:002009-05-08T14:44:44.802+01:00More Health Center featuresIn a <a href="http://hollycummins.blogspot.com/2008/09/ibm-monitoring-and-diagnostic-tools-for.html">previous post</a>, I described a new IBM tool, IBM Monitoring and Diagnostic Tools for Java - Health Center. We've added some new features to the <a href="https://www14.software.ibm.com/iwm/web/cc/earlyprograms/ibm/ibmmdtjhc/">Health Center beta</a>. <br /><br />Ever wondered what the classpath of that application which is behaving strangely is? Or why all the core files are truncated? The environment perspective provides details of the Java version, Java classpath, boot classpath, environment variables, and system properties. This can be really handy for debugging problems, particularly problems on remote systems or systems where you don't control the configuration. It also shows the ulimit, which is a common cause of strange behaviour on linux and unix systems. If the Health Center detects misconfigured applications, it will provide recommendations on how to fix it. <br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbMLo-8ChwYIdPuon0tPJ7Z07Kj-Bj9Gddeh3bSTBVeRsLYkemxeS9iEcj46BuD6CmTo0HiTKtXdNOHM27XbDpYaohweVRW0BwGAn0TJ3ezrPxHuoITX9SvudQ72pVeVu68AFpUrm23oMr/s1600-h/environment.gif"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 500px; height: 300px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbMLo-8ChwYIdPuon0tPJ7Z07Kj-Bj9Gddeh3bSTBVeRsLYkemxeS9iEcj46BuD6CmTo0HiTKtXdNOHM27XbDpYaohweVRW0BwGAn0TJ3ezrPxHuoITX9SvudQ72pVeVu68AFpUrm23oMr/s400/environment.gif" border="0" alt=""id="BLOGGER_PHOTO_ID_5333447542863314706" /></a><br /><br />Another new feature is the ability to export and import data. This means one person can collect the data, and if there's a problem, they can send it to someone else for more analysis. For example, if a method is unusually hot in the method profile, a system operator could send the exported data to the developer responsible for that area of the code. <br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhCgU2T6VeP3pChMaEZC9L2hyndOpYz_VXMxX1zkSGpZYHhk-9vENJXGjJpOKUtABd_a9JwpP2xnELeKdDCIzpzuomDjjx1TAwCPpehupqP1rn1kpchlgT4ZdGWpYMKo1zPLzb27fpVnLw3/s1600-h/export.gif"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 204px; height: 186px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhCgU2T6VeP3pChMaEZC9L2hyndOpYz_VXMxX1zkSGpZYHhk-9vENJXGjJpOKUtABd_a9JwpP2xnELeKdDCIzpzuomDjjx1TAwCPpehupqP1rn1kpchlgT4ZdGWpYMKo1zPLzb27fpVnLw3/s400/export.gif" border="0" alt=""id="BLOGGER_PHOTO_ID_5333448299976921874" /></a>Holly Cumminshttp://www.blogger.com/profile/08367229042550909996noreply@blogger.com0tag:blogger.com,1999:blog-4266388600963991311.post-45894220005106031992008-10-08T10:43:00.001+01:002008-10-08T10:43:00.455+01:00Garbage collection flavoursIn my previous post, I described the basic garbage collection algorithms. Within these basic categories, there are many variants, or flavours.<br /><h2>Generational collectors</h2>Generational collectors exploit the observed properties that most objects tend to die young, and that young objects are more likely to reference old objects than the reverse. Together these are known as the weak generational hypothesis.<br /><br />Generational collectors divide the heap up into multiple generations. Young generations can be collected without collecting old generations. These partial collections are quicker than full heap collections, and are likely to produce a good return of free space relative to the area collected, since most objects die young. At least the younger generations tend to employ copying collectors, since these collectors are very efficient in heaps with high attrition rates.<br /><br />Objects which survive a given number of collections are tenured and move up to an older generation. In order to avoid collecting objects in younger generations which are referenced by older generations, a remembered set is maintained of references from old generations to young generations. A write barrier between the generations is used to catch changes to these references.<br /><br /><h2>Incremental collection</h2><br /><br />Incremental collectors allow single collections to be divided into smaller collections. This allows pause times to be limited. Incremental techniques almost always require one of a write barrier or a read barrier to prevent changes to the object connectivity graph mid-way through a collection causing the reachability results to be incorrect.<br /><h2>Concurrent collection</h2>A concurrent collector is one which can execute concurrently with a mutator thread. Even on moderately multi-processor systems, it usually does not make sense to dedicate an entire processor to running garbage collection. Therefore the typical concurrent collection is perhaps more accurately described as a highly incremental collection; each thread is assigned small units of garbage collection work to do along with its application work, so the garbage collection work is finely interleaved with application work. One way of thinking of this garbage collection work is as a 'tax' - threads have to do some garbage collection work in exchange for being able to allocate. As with incremental techniques, concurrent collections need a write- or read-barrier.<br /><h2>Parallel collection</h2>A parallel collector is one which divides the collection work so that multiple collector threads are collecting concurrently. This is counter-productive on single-threaded processors, but increasingly necessary as the number of processors in a system increases. On systems with many processors a single-threaded collector is unable to keep up with the amount of garbage the processors can produce.<br /><br />Similar techniques are used to achieve incrementality, concurrency, and parallelism, so many collectors which have one of these properties also have some of the others.Holly Cumminshttp://www.blogger.com/profile/08367229042550909996noreply@blogger.com0tag:blogger.com,1999:blog-4266388600963991311.post-66972532286279289332008-10-07T18:44:00.002+01:002008-10-07T18:44:00.064+01:00Garbage collection algorithms<div style="text-align: justify;"><br /><p>Garbage collection has been the subject of much academic research . In particular, the volume of new techniques with various claimed properties is high enough that it is tempting to coin the phrase <a href="http://en.wikipedia.org/wiki/Another">YAGA (Yet Another GC Algorithm)</a>. For a pointer to many of these articles, see <a href="http://www.cs.kent.ac.uk/people/staff/rej/gcbib/gcbibP.html">Richard Jones' garbage collection bibliography.</a></p><br /><h2>Garbage collection algorithms</h2><br /><p>Despite the large number of variants, most garbage collection algorithms fall into a few simple categories.</p><br /><h2>Reference counting</h2><br /><p>Yes, reference counting is a form of garbage collection. It's sometimes seen as an alternative to garbage collection, and sometimes even as a morally superior alternative to garbage collection, but really, it's just a type of garbage collection algorithm.</p><br /><p>Reference counting garbage collectors track the number of references to each object. Often the count is maintained in and by the object itself. When a new reference to an object is added, the reference count is incremented. When a reference is removed, the count is decremented. When the count reaches zero, the object is destroyed and the memory released. Reference counting fails when objects reference one another in a cycle. In these cases each object will be seen to be referenced and will never be freed, even if nothing references the cyclic structure. </p><br /><p>Reference counting is unable to deal with cycles unless augmented with occasional invocations of another form of garbage collection. Reference counting has a number of other disadvantages, mostly to do with performance and bounding of work, and so it is rarely used in modern garbage collection systems. However, smart pointers in C++, while not traditionally considered a form of garbage collection, do make use of reference counting. Ad hoc garbage collectors introduced to manage object reachability in complex environments also tend to use reference counting since it is easily implemented in a hostile environment.<br /></p><br /><h2>Tracing collectors</h2><p>Most garbage collectors perform some sort of tracing to determine object liveness. Tracing does not have the same problem with collecting cycles as reference counting does. However, tracing collectors are more sensitive to changes in the object graph and usually require the suspension of the application threads or close monitoring of application activity to ensure correctness.</p><br /><h3>Mark-sweep</h3><br /><p>Mark-sweep collectors collect in two phases. In the first phase, all reachable objects are traced, begining from a set of roots. The roots are all objects which are guaranteed reachable, including objects referenced from the stack and static variables. Every object directly or indirectly reachable from a root is marked. Objects which do not end up marked are unreachable and therefore cannot be used again by the application, so they may safely be freed.<br /></p><br /><p>In the second phase of a mark-sweep collection, all unmarked objects are added to a free-list. Contiguous areas of free memory are merged on the free list.<br /></p><br /><h3>Mark-compact collectors</h3><br /><p>Mark-compact mark live objects using identical techniques to those used by mark-sweep collectors. However, mark-compact collectors do not use a free list. Instead, in the second phase all reachable objects are moved so that they are stored as compactly as possible. New allocation is then performed from the empty area of the heap.<br /></p><br /><p>Many collectors hybridise these two approaches, combining frequent sweeping with occasional compaction.</p><br /><h3>Copying collectors</h3><br /><p><br />Copying collectors divide the heap into two areas, a to-space and from-space. All objects are allocated in the to-space. When the new space is full, a collection is performed and the spaces are swapped. All reachable objects in the to-space are copied to the from-space, which is declared the new to-space. New allocation is then performed in the new to-space.<br /></p><br /><p>Copying collectors have a number of advantages over marking collectors. Because copying is done<br />in the same phase as tracing, it is not necessary to maintain a space-consuming list of which objects have been marked. Because no sweep is performed, the cost of the collection is proportional only to the amount of live data.<br /></p><br /><p>If most objects are unreachable, a copying collecton can be very efficient. The corollary is that if most objects remain reachable, or if some very large objects remain reachable, the collection will be woefully inefficient, because a large amount of memory will need to be copied every collection.<br /></p><br /><p style="text-align: justify;">Copying collectors also keep the heap very compact, and this can boost allocation and application performance. However, because a completely empty from-space must be maintained at all times, copying collectors are not space-efficient. Modern collectors tend to estimate an object death-rate and maintain less than half the heap for the from-space accordingly. Copying collectors are also known as semispace collectors.<br /></p><br /><p>For a good overview of how this relates to the IBM JVM's garbage collection, see Mattias Persson's <a href="http://www.ibm.com/developerworks/java/library/j-ibmjava2/">developerWorks article</a>. </p><br /></div>Holly Cumminshttp://www.blogger.com/profile/08367229042550909996noreply@blogger.com0tag:blogger.com,1999:blog-4266388600963991311.post-23431986524012966682008-10-05T18:29:00.004+01:002008-10-06T11:03:23.567+01:00Garbage collection mythsI've been lucky enough in my work to learn a fair bit about garbage collection. One of the things I've discovered is how many myths and half-truths exist about garbage collection. Part of the reason these myths have sprung up is that garbage collection has some pretty paradoxical properties. The first myth is that garbage collection is only suitable for the incompetent, unskilled, or lazy. In fact garbage collection offers many architectural and software engineering advantages, even to the skilled developer. The second myth is that garbage collection is all about about collecting garbage. It seems obvious from the name, but it's not true! Garbage collectors also include an allocation component, which, along with their powers of object rearrangement, can make a significant difference to application performance. Thirdly, criticisms of garbage collection often focus on the pause times, and responses to these criticisms often focus exclusively on reducing pause times, in the mistaken belief that small pause times guarantee good application response times. Pause times are also often used as a metric of general application performance, and an increase in pause times is taken as an indicator of worsened performance, when in fact the opposite the opposite is often true. Paradoxically, even the total amount of time spent paused for garbage collection is not a good predictor of the impact of garbage collection on application performance. Finally, the sixth myth is that garbage collection has a disastrous performance impact. While garbage collection can hurt application performance, it can also help application performance to the point where it exceeds the performance with manual memory management. I'll go through each of these in detail in later posts.<br /><br />But, to start off with, what is garbage collection? Garbage collection is a system of automatic memory management. Memory which has been dynamically allocated but which is no longer in use is reclaimed for future re-use without intervention by the application. Garbage collection solves the otherwise difficult problem of determining object liveness by freeing memory only when it becomes unreachable. <br /><br />Garbage collection is pretty ubiquitous in modern languages. Garbage collected languages include Java, the .Net languages, Lisp, Python, Perl, PHP, Ruby, Smalltalk, ML, Self, Modula-3, and Eiffel. Some languages which are not traditionally garbage collected offer garbage collection as a pluggable or configurable extension. For example, collectors are available for C++, and Objective-C was recently extended to allow garbage collection. Understanding the garbage collector is an important part of performance tuning in these languages.Holly Cumminshttp://www.blogger.com/profile/08367229042550909996noreply@blogger.com0tag:blogger.com,1999:blog-4266388600963991311.post-46356449914813752662008-09-24T20:13:00.001+01:002009-05-11T14:40:00.976+01:00IBM Monitoring and Diagnostic Tools for Java™ - Health Center™ is now availableI'm part of a team in IBM's Java Technology Centre which develops diagnostic tools, and we've just released a brand new tool. It's called IBM Monitoring and Diagnostic Tools for Java™ - Health Center™ and it's available as an early access beta <a href="https://www14.software.ibm.com/iwm/web/cc/earlyprograms/ibm/ibmmdtjhc/">download</a> from developerWorks. We're really excited about the diagnostic insights it can offer. It allows running JVMs to be observed, health-checked, and tuned. At the moment it gives insight into general system health, application activity, and garbage collection activity. There are visualizations and also recommendations, so you don't have to be an expert to improve the performance of your application.<br /><br />One of the nice things about the Health Center is that it presents different kinds of information in one place. There's a status page which shows at a glance what's going well and what might need attention:<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEggjwWIM7WRQVtHPUDOl2URb5i61ZHWqi6kvSFADXNNEO5z5kHUd1OxE3GdXC-UO-pYLQNuvjsxqiVuGurroEPlY-KDb9Mba1pjR_kr9VRjrSeScGivU2lCzpUS5sPl0qguFnBMYiEumj9a/s1600-h/frontpage.gif"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEggjwWIM7WRQVtHPUDOl2URb5i61ZHWqi6kvSFADXNNEO5z5kHUd1OxE3GdXC-UO-pYLQNuvjsxqiVuGurroEPlY-KDb9Mba1pjR_kr9VRjrSeScGivU2lCzpUS5sPl0qguFnBMYiEumj9a/s400/frontpage.gif" alt="" id="BLOGGER_PHOTO_ID_5249684603172018898" border="0" /></a><br />Garbage collection is one of the first and easiest things to check when tuning an application. Sometimes garbage collection can benefit from a bit of tuning, and sometimes looking at what the garbage collection is doing can highlight application problems. Is your application using more memory than seems reasonable? Is it leaking memory? The Health Center will suggest a command line, diagnose when garbage collection is causing performance problems, and point out memory leaks:<br /><br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgjRGtOJLqCNOZsBjjltXmdHhRmPmq2hdPUa1dgwTqwNwmXik7jbYQNWG2vs5YWmceqyO6o-F4qhPhvo-J6py-Nb7xrIeoIS3xX56vYdYuoptyxQKvMFerXVpzAa5DLHuoRmaBtUOjbwvBn/s1600-h/gc.gif"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgjRGtOJLqCNOZsBjjltXmdHhRmPmq2hdPUa1dgwTqwNwmXik7jbYQNWG2vs5YWmceqyO6o-F4qhPhvo-J6py-Nb7xrIeoIS3xX56vYdYuoptyxQKvMFerXVpzAa5DLHuoRmaBtUOjbwvBn/s400/gc.gif" alt="" id="BLOGGER_PHOTO_ID_5249684957172846290" border="0" /></a><br />Synchronization can be a big performance bottleneck for applications on multi-CPU systems, and it can be difficult to identify hot locks or even assess the impact of locking. The Health Center has always-on lock monitoring which identifies the objects which are most contended and provides guidance about whether synchronization is hurting performance:<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgdbIYkFieS97rOhGYPnLStUUzgPOCNW4o1zjN7xXEQVSmO5zBknZ4EXy3I632lds9eWUBqUZ5X2Lpvsyxy9LI-MAM0f3etYbm3wRCSKqnIWOkZW9w3T47aOwxX5Cv_e-KWh2kCXf-46DAx/s1600-h/locking.gif"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgdbIYkFieS97rOhGYPnLStUUzgPOCNW4o1zjN7xXEQVSmO5zBknZ4EXy3I632lds9eWUBqUZ5X2Lpvsyxy9LI-MAM0f3etYbm3wRCSKqnIWOkZW9w3T47aOwxX5Cv_e-KWh2kCXf-46DAx/s400/locking.gif" alt="" id="BLOGGER_PHOTO_ID_5249685155314280674" border="0" /></a><br />Last, but not least, the Health Center includes a nice sampling method profiler. This can be really valuable in diagnosing applications which are using high CPU. Because it's low overhead, there's no need to specify in advance which parts of the application should be profiled. It works without recompilation or byte code instrumentation. It shows where the application is spending its time and gives full call stacks for all sampled methods. Because of the call stacks, this kind of profiling isn't just for performance problems - sometimes it can give surprising functional insight:<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgROv_SH0yVaRHOsn6bw69k5JOhgX45ZAM7UfZZKrPMz_NfnEmx2rzIAOsBiifnCFfJxpKtJ_UdHSTpgo19NiC0Zhnrn-xn4OS3JAKSXh9hY7BJUNJy4DW9qAn2RbiyEKo_caAfK3Rb9ZqC/s1600-h/profile.gif"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgROv_SH0yVaRHOsn6bw69k5JOhgX45ZAM7UfZZKrPMz_NfnEmx2rzIAOsBiifnCFfJxpKtJ_UdHSTpgo19NiC0Zhnrn-xn4OS3JAKSXh9hY7BJUNJy4DW9qAn2RbiyEKo_caAfK3Rb9ZqC/s400/profile.gif" alt="" id="BLOGGER_PHOTO_ID_5249685344562824546" border="0" /></a><br />The way the Health Center is able to get so much information without much overhead is that it's closely linked to the JVM, so there are minimum system requirements. The absolute minimum requirement is an IBM JVM, version 5.0 or higher. This will give locking information and some GC information. To get profiling Java 5 SR8 or Java 6 SR1 is required. The fullest GC information is available in Java 6 service refresh 2. IBM JVMs can be downloaded from <a href="http://www.ibm.com/developerworks/java/jdk/">developerWorks</a>.Holly Cumminshttp://www.blogger.com/profile/08367229042550909996noreply@blogger.com0tag:blogger.com,1999:blog-4266388600963991311.post-2467102515025566502007-09-17T19:57:00.001+01:002008-10-06T11:03:48.279+01:00How to download EVTK?EVTK can be downloaded as a plug-in to the <a href="http://www-01.ibm.com/software/support/isa/">IBM Support Assistant</a>. Download ISA first, and then choose EVTK from the list of available tools. Both are free.<br /><br />An <a href="http://www.ibm.com/developerworks/forums/dw_thread.jsp?forum=1030&thread=158867&cat=10">EVTK forum</a> is available on developerWorks. The forum is pretty quiet at the moment but it has links to installation instructions for EVTK. It would also be great to have more traffic on the forum itself.<br /><br /><b>EDIT:</b> EVTK has has a name change! It's now known as IBM Monitoring and Diagnostic Tools for Java - Garbage Collection and Memory Visualizer. Different name, same tool.Holly Cumminshttp://www.blogger.com/profile/08367229042550909996noreply@blogger.com1tag:blogger.com,1999:blog-4266388600963991311.post-1342348811919946702007-09-16T21:41:00.000+01:002007-09-16T22:02:28.071+01:00I've just got back from speaking at JavaZone in Oslo. I'll post links to the presentation videos when they're available. The feedback on my presentation and on the Extensible Verbose Toolkit gc analysis tooling I demonstrated was really positive, which was nice.<br /><br />One of the main things I discussed in my talk is that care is required in interpreting garbage collection pauses. Even though pauses stop an application doing work, which is obviously bad, there's not necessarily any correlation between shorter garbage collection pauses and better application performance. Sometimes shorter pauses give better performance, sometimes longer pauses give better performance. This is pretty uncontroversial if the quality of service requirement is better throughput. It's a lot more contentious if response times are the critical performance metric. After my talk a few people told me about their applications, for which longer garbage collection pauses <span style="font-style: italic;">definitely</span> meant longer response times and <span style="font-style: italic;">definitely</span> mattered a lot. I certainly wasn't going to tell them they were wrong, either! I think the number of cases where there's a strong correlation between garbage collection pause times and response times - or, at least, maximum response times - is pretty high. What I was trying to get across is that there are also at least <span style="font-style: italic;">some</span> cases where the gc pause times are only a small contributor to the maximum response times. It would be unfortunate to minimise pause times without ever checking back against response times if your application happened to be one of the ones where pause times aren't so important.<br /><br />The whole subject of performance tuning is fascinating because there are so many different system configurations and so many different performance requirements that coming up with a single answer is impossible. Even coming up with a single coherent set of heuristics is pretty tricky.Holly Cumminshttp://www.blogger.com/profile/08367229042550909996noreply@blogger.com0