I have seen many bad Drupal sites built around complex modules and panels. For a years I have avoided of using Drupal Panels. I mean in 90% cases developer can get along using just views and good structured theme.
But there are 10% cases when you should consider of using Drupal panels. And those are cases when you want to have a large end-user style editors to have in your site. When we are dealing with end-users - everything must be created as for 5th graders (sometimes as for 1st graders).
So I got a project, which involved use of Drupal Panels. I tried to avoid them as long as possible, but then in one point I realised - hey - it's Drupal Panels what I need.
Basically I could still get avoid of Drupal Panels, but that would involve additional custom theme file creating and templating. This is not we are looking for. We still need to have a minimum loaded site, yet powerful content type display management.
This site has pretty complex content type architecture, so again I come back of using Drupal panels.
What is Drupal Panels
The Panels module allows a site administrator to create customized layouts for multiple uses. At its core it is a drag and drop content manager that lets you visually design a layout and place content within that layout. Integration with other systems allows you to create nodes that use this, landing pages that use this, and even override system pages such as taxonomy and the node page so that you can customize the layout of your site with very fine grained permissions.
In a Panel, you can create contexts, which represent the objects being displayed. For example, when displaying the node view, NID argument on the page is converted into a context through the 'arguments' system. You can then create a relationship from that node to, say, the node author, or if you have a node reference from CCK, a related node [as of CCK 2.3]. Once the contexts are in place, content specifically about those contexts can be placed. For the node context you can add CCK fields, the node body, attached files and a host of other information that can be provided by plugins. For the user context you can display things like the user picture or profile. Note that CCK 2.3 and later only has support for Panels 3.
In addition, these contexts can be checked for information and use that not only to make content available to be displayed, but to choose which layout to display! For example, if your site is international, you can use context to see if the node being viewed is set for a particular language and choose to display it one way if it is in French or another way if it is in English. You can also select on attributes like node type, whether or not the user has access to edit the node, and more. This system is also pluggable and you can add your own custom criteria with only a small amount of code. Want to display nodes differently based on how a custom CCK field you've added to a node type is set? That is very simple to write and you can use this to change the presentation entirely.
Panels also includes simpler applications of the drag and drop system. There is a node type (the 'panel' node) that can simply be added as content to your system. By being a node it loses a lot of the features that the more powerful page system has, but it does have the advantage of simplicity, and gaining all of the functionality that nodes normally get.
Panels can also be used for items smaller than pages. What if you have a normal sidebar, and you have two pieces of content that waste a bunch of space because they're really too narrow, and your design looks significantly better if they are side by side? That's complex to do in Drupal because what you end up having to do is create a custom block with custom code to display these two pieces of content. With Panels, you just create a "mini-panel" with a two column layout. Add one block to the left, one block to the right, and finish. This mini panel will then be available to your system as an ordinary block, or as panel content to go in the other panels.
What is my suggestion on when to Use Drupal Panels
Don't grab a panels right away, learn is Panels a module you really, really need. For me I have found I'm preferring to use Panels in cases, when complex theme rewrites are involved.
Complex rewrites in most cases means a bad site architecture, so plan your site structure first!