Discussion: View Thread

Rigor vs. simplicity (or, Consultants cause complications)

  • 1.  Rigor vs. simplicity (or, Consultants cause complications)

    Posted 04-29-2012 14:48
    I'm interested in thoughts or resources (e.g., articles, websites) about the following issue, which I suspect many consultants encounter: How do we and our clients decide on a satisfactory balance between rigor and simplicity when planning studies and analyzing data?  From another perspective, how can we consultants uphold our professional integrity without unnecessarily complicating clients' projects or wasting their resources?

    By "rigor" I loosely mean how appropriate, accurate, or valid a study design or analysis strategy is, given the client's situation (e.g., research problem, data attributes).  By "simplicity" I loosely mean how easily or quickly the study design or analysis strategy can be implemented or understood by parties involved (e.g., consultant, client, peer reviewers, readers).  Rigor and simplicity tend to be related inversely, though there are certainly exceptions.

    Consider this hypothetical situation: Your client considers your choice of a best approach for their situation too complicated, time-consuming, or otherwise excessive and would instead prefer something easier or faster.  It can be difficult to decide whether a simpler approach might be inappropriate and, if so, explain this convincingly.  This is further complicated when the appropriateness depends non-trivially on unknown data characteristics or when viable alternatives haven't been studied under realistic scenarios like the client's.

    Here's a real example where this is likely to arise: I've worked with a specific client for nearly 10 years on large NIH-funded quantitative research syntheses (i.e., systematic reviews).  Meta-analytic methods I've been using are reasonable -- often better than common practice -- but suboptimal for certain messy aspects of the data (e.g., non-independence of effect estimates, missing or otherwise troublesome covariates).  I plan to suggest that I spend part of my billable time finding or creating better methods, but I'm not sure I can make a strong case that any benefits from improving our methods will be worth my time -- at least tens of hours and possibly a couple hundred -- and the client's money.


    Cheers,

    Adam

    -------------------------------------------
    Adam Hafdahl
    Owner & Principal Consultant
    ARCH Statistical Consulting, LLC
    -------------------------------------------


  • 2.  RE:Rigor vs. simplicity (or, Consultants cause complications)

    Posted 04-29-2012 15:37
    Hi Adam and all,

    This is a good question and a very common situation in consulting. I think that this sometimes comes as a trade-off between the length of the methods + results section vs. the length of the discussion section of a peer-reviewed article or report. Sometimes there are not the resources to do the very optimal thing. If so, I feel it is fair to have part of the discussion section dedicated to what impact that has. So, the methods and results sections are shorter, but the discussion section is longer than for the "optimal" analysis.

    If there is not going to be a peer-reviewed article or report (with a discussion section such as I described), then I feel the consultant needs to write a memo to the client with appropriate caveats and limitations and a discussion of those.

    It probably happens some time that the client or the journal reader just takes away "the number", and all the discussion and caveats get lost. Nevertheless, if the issue becomes controversial, the back-up is there in print. 

    I do feel that we should try to accommodate clients and their budgets unless it is mission impossible. An eminent advisor that I had in grad. school noted that he didn't need to always be an optimizer; he could sometimes be a "satisficer." I.e., be satisfactory rather than optimal. 

    Best wishes,

    Nayak



    -------------------------------------------
    Nayak Polissar
    Principal Statistician
    The Mountain-Whisper-Light Statistics
    -------------------------------------------








  • 3.  RE:Rigor vs. simplicity (or, Consultants cause complications)

    Posted 04-29-2012 16:45

    Hi Adam, Based on the example that you provided, my feeling is that you should not charge the client for time spent finding and developing better methods to analyze the client's project.  I view this extra work on your part as a capital investment, similar to the situation when a building contractor needs to upgrade his/her equipment in order to satisfactorily complete the job for a given customer.  The new or improved methods that you find or develop will increase your skill set and could later be applied (partially or with minor adjustments) to other consulting jobs as well.  Regards, Edith
    -------------------------------------------
    Edith Zang
    Independent Consultant
    NYCASA
    -------------------------------------------








  • 4.  RE:Rigor vs. simplicity (or, Consultants cause complications)

    Posted 04-29-2012 22:36
    Thanks for your thoughts on this, Edith.  Your advice seems sound in some situations, but other situations are less clearcut.  For instance, in the situation I described some of the messy issues I'd like to handle better are fairly specific to that client: They're almost certain to arise in that client's future projects -- as they have in most of our dozen-ish past projects -- but (in my judgment) relatively unlikely to arise in other clients' projects.  That makes it harder to view my potential work to find or create better methods as a capital investment; it seems more to me like problem solving for a particular client's relatively unique type of data.

    At the risk of straying from my original topic, this seems to me like a grey area in stats consulting: When a client's problem requires the consultant to learn about or develop a new (to the consultant) method, it's unclear whether or how much to expect the client to pay for the consultant's problem-solving time and effort.  Should this be influenced by how transferable this "R&D" work is to future projects, or by other factors?  Certainly this should be explicit in any agreement about services and payment, but I wonder how other consultants navigate such decisions.


    Cheers,

    Adam
    -------------------------------------------
    Adam Hafdahl
    Owner & Principal Consultant
    ARCH Statistical Consulting, LLC
    -------------------------------------------








  • 5.  RE:Rigor vs. simplicity (or, Consultants cause complications)

    Posted 04-29-2012 23:01
    Perhaps you indicated in the earlier post and I did not see it, but what, if any, intellectual property clause do you have in your consulting agreement as a standard matter of course? In some agreements all IP remains with the client, and sometimes these clauses are worded so as to be very broad and encompassing to the advantage of the client and the disadvantage of the consultant.  I would certainly take the IP clause of the agreement into account in considering how much new development to undertake without compensation.  Regardless of the IP clause, I think it is the consultant's obligation to point out the weaknesses, limitations, and threats to valid statistical inferences of the methods that the client expects and is willing to pay for.  Like other professionals providing a service, e.g., doctors, lawyers, plumbers, electricians, mechanics, the client has a right to say the services for which  they are willing to pay the professional.  However, if the professional feels it is unwise to provide a certain kind of service, they shoudl tell the client so, and tell them why.  The professional can always recommend that the client hire someone else to provide that service becuase it is against their professional opinion to perform the service the client wants without the additional procedures for which the client is unwilling to pay.  The client can decide what they want and are willing to pay for, but you get to decide the conditions under which you're willing to provide that service and what other work must be done to accompany it if you are to sign off on it.

    -------------------------------------------
    David Francis
    Hugh Roy and Lillie Cranz Cullen Distinguished University Chair
    Department of Psychology
    Univ of Houston
    -------------------------------------------








  • 6.  RE:Rigor vs. simplicity (or, Consultants cause complications)

    Posted 04-29-2012 23:20
    Those are important points I hadn't considered, David.  Although I use contracts for all my paid consulting projects, I don't -- in my contract I originate -- have an IP clause.  There's often such a clause, however, in contracts originated by clients, and I should probably pay more attention to their stipulations when I might develop methods for the client's problem.


    Cheers,

    Adam

    -------------------------------------------
    Adam Hafdahl
    Owner & Principal Consultant
    ARCH Statistical Consulting, LLC
    -------------------------------------------








  • 7.  RE:Rigor vs. simplicity (or, Consultants cause complications)

    Posted 04-30-2012 07:56
    While I don't disagree with Edith in whole, I do believe that what we charge should reflect the time that is spent in research.  While a contractor does not charge a client when a piece of equipment needs to be replaced, s/he does work into the fee the cost of maintainance, upkeep and eventual replacement.  And to some extent, if a specific piece of equipment or tool is required for that job, there is often a supplemental charge attached.  I believe that a part of what our clients pay us for is research; finding the most cost effective and efficent solution to a problem and as such, our fees should both reflect and incorporate the cost of that research, both immediate and ongoing.

    Frank

    -------------------------------------------
    Frank Cohen
    Senior Analyst
    The Frank Cohen Group, LLC
    -------------------------------------------








  • 8.  RE:Rigor vs. simplicity (or, Consultants cause complications)

    Posted 04-30-2012 09:47


    This has been an interesting discussion.  I was curious how it would go.  Being retired from big pharma I tend to do things as I did in industry.  While employed in big pharma, if I had a new project and I needed to learn something about cancer or diabetes or a new stat technique I would do the research on company time, purchase books paid for by the company, and possibly attend seminars or training sessions paid for by the company.  Now it was in both my interest and the company's interest for me to become knowledgeable on these topics and the company would have to approve my time and expenses.  I use the same philosophy as a stat consultant.  If the additional expense is minor (an hour or two) I usually just add it to my charge.  If it is extensive I discuss it with the client.  Overall I try to be sensitive to the needs and budgets of the clients and also trying to provide high quality statistical consulting.  This seems to work for me.  Just a thought.



    -------------------------------------------
    Rocco Brunelle
    Senior Statistician
    Bowsher Brunelle Smith LLC
    -------------------------------------------








  • 9.  RE:Rigor vs. simplicity (or, Consultants cause complications)

    Posted 04-30-2012 09:58
    The work you describe doing on company time is certainly very appropriate.  Consulting though is a very different business.  As I have said billing of research after discussing it with the client and getting approval is reasonable.  The only thing you said that I didn't agree with is doing a few hours of research and just adding it into the bill.  I find that in the consulting business the client usually expects and they should get an accounting for what you did for all your billable hours.

    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------








  • 10.  RE:Rigor vs. simplicity (or, Consultants cause complications)

    Posted 04-30-2012 09:48
    Edith said research is not something a consultant should bill for and you said that you didn't totally disagree but went on to saying that we should bill for it.  That sounds like disagreeing completely to me.  I agree with Edith but I do think that there may be situations where some small amount of research could be warranted and billable but only if you can convince the client and put some well defined limits on the number of hours to be spent on the research.  I don't think comparing research to replacing a piece of equipment is a good analogy.  Research is a completely different animal.  We do research not knowing what we will find.  We cannot accurately predict how much time we will spend on it and we could find that the research time did not lead to a desired solution to the problem.  Also as has been said before the consultant works for the client.  it is not the place of the consultant to tell the client what he will spend his billable time on.  I don't think many clients would allow any billable time for research.  Generally the consultant is hired to provide a service based on his specific expertise.  The client expects the consultant to solve his specific problem.  If this cannot be done in a straightforward way the consultant should probably bow out of the job.
    Research is something we do for our own benefit in the way Edith has described.

    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------








  • 11.  RE:Rigor vs. simplicity (or, Consultants cause complications)

    Posted 04-29-2012 17:12
    To reply to Adam's example, I do not think you can charge a client for research time developing statistical methods.  This is consulting work not a research grant. 

    If suboptimal means correct but less efficient, i can easily agree to do it.  For a simple example suppose you are comparing the means of two groups.  Sample size is small and data are not normally distributed but the data are paired.  I could use the sign test or the sign rank test.  If I can reject with the sign test I might show that to the consultant. It is a little simpler to explain the sign test though it is less powerful.  But what if I can reject the null hypothesis with th sign rank test but not with the sign test.  I might attribute the fasilure to reject due to the lack of power.  I don't thin I should then allow for a different conclusion just to use the simpler method,  In reporting I think as Nayak states everything should be disclosed in the report or journal article. 

    Another situation that I was actually faced with was a case of comparing two survival curves using Kaplan-Meier.  The log rank test showed a significant difference between the curves because of differences near the cutoff time.  However the Wilcoxon test did not.  This is actually a problem with the log rank test when there are only a few cases not censored in the late time stage.  I report both tests and would suggest that the difference may not be significant because of the problem with the log rank test.  In either case the tests are difficult to explain to the client but these tests are generally excepted in the medical literature and my clients don't have a problem with my use of the methods.

    I think I would take a relatively tough line.  The method must be correct and appropriate but a less powerful test can be substituted for a known more powerful method as long as the lack of power does not affect the results.

    But if the method is not appropriate I can't use it just to keep thing simple for the client.  In my example the paired t test would not be appropriate because the data do not appear to be normal and the normal assumptions is critical to the applicability of the t test.  So I would not use the t test even if the client is more familiar and comfortable with the t test.  In the case of contingency table analysis if there are many sparse cells would you use the chi square test?  You know it may not be accurate since it is an asymptotic procedure.  I would use the Fisher exact test even if the client may not understand and would be more comfortable with the chi square test
    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------








  • 12.  RE:Rigor vs. simplicity (or, Consultants cause complications)

    Posted 04-29-2012 23:11
    Thanks for your thoughts, Michael.  In my reply to Edith I addressed your comment about charging a client for developing (or finding/learning) methods, so I won't say more about that here.

    Your examples about choosing between techniques seem relatively clearcut, so below I'll pose one that seems harder to resolve, based on some previous work I've done for a few different clients:

    I've had clients bring me large, messy data sets that I've spent considerable billable time checking for errors and anomalies before running primary analyses.  Although I've found several things to correct in these clients' data, it's not clear to me that these improvements to the data are worth the amount of time I spent on "cleaning," and I suspect that sometimes the client would rather not pay for these improvements (though only rarely has a client complained).  This is an example where being more "rigorous" (i.e., more fastidious about data cleaning) can be costly for a client, the client might prefer a "simpler" approach (involving less cleaning), and it's often unclear in advance -- before doing the checking -- whether any improvements will be worth the time and expense.

    What interests me about situations like this example is that it can be challenging to explain to some clients why an apparently costlier approach might be preferable.  I can often empathize with these clients: If I hire a contractor to build something for me and s/he offers options ranging in quality and price, I'll often avoid the most expensive options because (to me) their benefits aren't worth the additional expense.


    Cheers,

    Adam

    -------------------------------------------
    Adam Hafdahl
    Owner & Principal Consultant
    ARCH Statistical Consulting, LLC
    -------------------------------------------








  • 13.  RE:Rigor vs. simplicity (or, Consultants cause complications)

    Posted 04-30-2012 00:32
    I thnik that before you do work for a client that is not clearly an agreed part of the assignment you explain why you want to do it and get the client's approval.  That way there is no controversy about whether or not they should pay you for the work.  In the case of data cleaning I think data cleaning should always be part of data base management prior to analysis.  If you see errors in the data it should be corrected.  It shouldn't matter whether or not you foresee a problem with the analysis as a  consequence.  there is always the danger of unforeseen problems with the current work or future work.  The client benefits from data cleaning. Nevertheless it is the client's choice as to what he is willing to pay you to do and you can only charge for what is agreed upon.  That also goes to your question about doing research for a client.  That too would have to be agreed upon by both parties and I would think that if they would allow this they would limit the amount of time spent on the research. But I find it unlikely that a typical client would pay for research.  I agree with Edith that this is something you would do on your own initiative for your benefit and benefit to future clients.

    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------








  • 14.  RE:Rigor vs. simplicity (or, Consultants cause complications)

    Posted 04-30-2012 11:46
    Perhaps my experience is atypical, but I often bill clients for what I'll call here "R&D" -- learning about unfamiliar methods or developing new ones to meet the client's needs.  In fact, I consider this learning on the job a key perk of stats consulting.

    My perspective on this is that if something I spend time on during our contract period benefits the client, then I'm comfortable billing the client for it.  That includes R&D.  If a client isn't comfortable with that for some reason, we can either negotiate a compromise or decide not to work together.

    I completely agree, Michael, that any R&D a client pays for should be discussed and agreed upon in advance and handled transparently.  Maybe my clients are unusual, but they rarely object to my spending billable time on R&D for their projects.  In brief, here's how I typically handle this:

    * Before I'm hired I tell the client that a substantial part of my billable work will involve R&D to find or develop appropriate methods.  We discuss this, and it's specified in my contract, so the client has ample opportunity to negotiate (or hire someone else).

    * I sometimes offer a markedly reduced fee for either the entire project or the R&D component, especially if the method is something I'm especially interested in learning or developing.  I tell the client this, so s/he knows I'm sacrificing some income while spending time on activity that doesn't directly produce deliverables.

    * I usually include "approval intervals," whereby after each fixed amount of billable time (e.g., 5 hrs) I update the client on my progress and obtain his or her approval before continuing.  This gives the client a way to track my billable contributions and, if s/he's uncomfortable with the amount of R&D (or anything else), discuss alternative strategies with me.

    This has worked well so far.  For example, on a 120-hour project that involved investigating temporal precedence in bivariate panel data, I spent about 15 hours reading literature to find an appropriate method and well over 30 hours understanding details of a particular approach (J. J. McArdle and colleagues' bivariate dual change-score model) and implementing it in LISREL and Mplus.  Some of my considerable R&D time involved consulting with experts in this approach and running small Monte Carlo simulations to check its performance with data like my client's.  I learned a lot, and I think the client did, too -- she really wanted to understand the technique and its limitations.  As far as I know she was pleased with my contributions; she and her colleagues published the study in a top clinical psychology journal.


    Cheers,

    Adam

    -------------------------------------------
    Adam Hafdahl
    Owner & Principal Consultant
    ARCH Statistical Consulting, LLC
    -------------------------------------------








  • 15.  RE:Rigor vs. simplicity (or, Consultants cause complications)

    Posted 04-30-2012 12:01

    When i talked about research I was only talking about the part where you would develop new methods that are original and might be publishable in a peer reviewed journal.  This was the context set by your meta analysis example.  Spending time researching appropriate existing techniques is a different matter.  That does not have the uncertainty that would come from trying to develop a new technique on your own.  if this amounts to 3 or 4 hours of work I don't think many clients would object.  Some might though.  There are clients that are very strict about only allowing work directly connected to an agreed on deliverable.  You are paid as an expert consultant and so if there is an aspect of the problem that you don't know they may feel that it is your obligation to learn it on your own time.  In any case the imprtant thing is good communication with the client so that you don't get into the situation of billing them for several hours of work that they did not authorize and did not expect.
    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------








  • 16.  RE:Rigor vs. simplicity (or, Consultants cause complications)

    Posted 04-30-2012 00:32
    I thnik that before you do work for a client that is not clearly an agreed part of the assignment you explain why you want to do it and get the client's approval.  That way there is no controversy about whether or not they should pay you for the work.  In the case of data cleaning I think data cleaning should always be part of data base management prior to analysis.  If you see errors in the data it should be corrected.  It shouldn't matter whether or not you foresee a problem with the analysis as a  consequence.  there is always the danger of unforeseen problems with the current work or future work.  The client benefits from data cleaning. Nevertheless it is the client's choice as to what he is willing to pay you to do and you can only charge for what is agreed upon.  That also goes to your question about doing research for a client.  That too would have to be agreed upon by both parties and I would think that if they would allow this they would limit the amount of time spent on the research. But I find it unlikely that a typical client would pay for research.  I agree with Edith that this is something you would do on your own initiative for your benefit and benefit to future clients.

    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------








  • 17.  RE:Rigor vs. simplicity (or, Consultants cause complications)

    Posted 04-30-2012 08:06

    WRT data cleaning, in my experience many clients have to be closely involved with this.   It is helpful to guide them through doing the data cleaning with my supervision/support.  Very often clients are willing to do the nitty gritty so that they pay me less.  For example,  they may not know what goes on under the hood when one clicks the "Identify unusual cases" entry in a menu, but they certainly can understand that these are cases that need to be double checked.  Most people have heard the saying "Garbage in, garbage out".

    I explain during the up front discussions that 90 or 95% of the time needed for analysis is getting the data cleaned. That even a few wild codes can greatly distort results. For example,  In 1973 I had a file of 1200 cases.  Three had missing data codes of 999.  I was used to DEC systems where opening a binary file changing some stuff and saving it did not distort the last few bits.  But this job was in an IBM environment.  Opening a binary file and just saving it in a new name would often change the lowest order bits.   Correlations changed from the +.40s to -.60s on some runs  where both  variables were missing. Substantively this made no sense at all.  Took me a week to figure it out.

    WRT needing to develop expertise.  In my experience, I can hold brief discussions with other methods/stat people who have written about a topic as a professional courtesy. Often people can ask questions or make suggestions in a few minutes.  If it requires longer, I explain to the client that it would save money to have a part of the work done by someone else.  I have not had anyone object to this so I have not thought out what I would do if the client insisted that I do a particular task.  IIRC the last time I entered data for a client was 1974.

    Also, on my own time I find it helpful to look at materials from other disciplines.  Although meta-analysis started in psychology, for the last couple decades it has spread to many disciplines. 

    In terms of turning to someone else, despite stat culture picturing the lone ranger, I do not consider it unethical to get help for information or confirmation.  It may or may not be in the ethics code for stat, but in some other ethics codes it is an obligation.  For example, a psychotherapist who has a general practice but concentrates on working with children  is presented with an adult alcoholic client. (S)he is obliged to seek supervision or to turn the case over to someone with more expertise in working with adult alcoholics.

    Turning to lists like this can be a very effective in getting ideas to think about.
    -------------------------------------------
    Arthur Kendall
    Social Research Consultants
    -------------------------------------------








  • 18.  RE:Rigor vs. simplicity (or, Consultants cause complications)

    Posted 04-30-2012 09:30
    Since joining this eGroup I have been struck by the way the membership splits into two groups
    (1) the highly experienced and wise professional statistical consultants like Edith Zhang and Art Kendall among others and
    (2) the wanna be or very inexperienced consultants.

    I think that sometime those from group two would be much better off if they listened and learned from the advice and comment of the members in group1 rather than argue and make some outlandish remarks.
    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------








  • 19.  RE:Rigor vs. simplicity (or, Consultants cause complications)

    Posted 05-01-2012 08:42
    Now that was an arrogant, unkind and unnecessary response. Not a very good way to encourage an open and honest exchange of thoughts and ideas amongst members.

    -------------------------------------------
    Frank Cohen
    Senior Analyst
    TFCG, LLC
    -------------------------------------------








  • 20.  RE:Rigor vs. simplicity (or, Consultants cause complications)

    Posted 05-01-2012 10:03

    When I directed stat collaborating/consulting groups in academic medicine, we discussed this problem in terms like this.

    An "A" level project has major scientific importance. The quality of the stat science needs to be at least A-, meaning that the approach and intensity are the best we can provide using methods and software already available to us, even if those methods are not yet commonly found in the literature. However, if no good methods and software are available "off the shelf," then new/customized ones will need to be developed. This is "A" level statistical science; relatively few projects get "A" attention.

    Most "decent" studies in academic medicine have only marginal scientific importance. But they certainly contribute to our knowledge base, so they still need to be done decently. These "B" level projects should at least get B-level statistical science--traditional "textbook" approaches done competently and thoroughly. Any serious investigator should willingly fund such effort. BTW, "B" level stat science is better than much of the statistical work in evidence in the medical literature.

    If a project is clearly C level, then you may still have to consult anyway. Give it "minimally sufficient" attention, which is probably just what the investigator wants anyway. Think three times about accepting an authorship; I never do.

    BTW, never accept acknowledgements; they have no real benefits and yet carry potential risks.

    Of course, don't do anything that you consider scientifically dishonest. Withdrawing from such relationships can be tough and the knives in our backs may scar us a bit, but we must do it and we survive. I've done so twice in the last 15 months, but I'll spare you those stories. At nearly 63, I've got dozens of them.

    -------------------------------------------
    Ralph O'Brien
    Case Western Reserve University
    -------------------------------------------








  • 21.  RE:Rigor vs. simplicity (or, Consultants cause complications)

    Posted 05-01-2012 10:07
    I couldn't agree more.

    -------------------------------------------
    Matthew Mercurio
    -------------------------------------------








  • 22.  RE:Rigor vs. simplicity (or, Consultants cause complications)

    Posted 05-01-2012 10:49

    I was trying to make an important point but since I was so harsh I think some of you misunderstood.  This eGroup is very fortunate to have such wise and experienced statistical consultants like Art and Edith willing to take the time to post here. I could add Charles Barker to that list and several others.  There are also many of us who are rather naive about statistical consulting and may think that a degree in statistics is enough to go out and be a consultant.  This is one of my pet peeves.  Good consulting requires the ability to listen to other,  a willingness to learn enough subject matter to properly understand and address the clients problem and the ability to communicate the key ideas to the client at the clients level without statistical jargon.  These skills are more important than technical knowledge of statistical theory and methods.

    I was not being arrogant and I was not trying to discourage discussion.  Perhaps rather than saying "outlandish statements" I should have used the term "naïve."  Anyone wanting to agree or disagree with me is welcome to speak.  I welcome the debate.

    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------








  • 23.  RE:Rigor vs. simplicity (or, Consultants cause complications)

    Posted 05-01-2012 11:52
    Although I don't always agree with Michael, I often find his comments often thought provoking and reasonable.  I agree that his past statement was a little hard but not worth making a fuss over.  My biggest concern is which group am I in? (Hehe).  Personally I don't care.  I'm confident in my abilities and I hope I know when to get advice.  I think all consultants need to be confident, but not overly so.  Too bad web based sites don't have a Happy Hour.

    -------------------------------------------
    Rocco Brunelle
    Senior Statistician
    Bowsher Brunelle Smith LLC
    -------------------------------------------








  • 24.  RE:Rigor vs. simplicity (or, Consultants cause complications)

    Posted 05-01-2012 12:06
    Rocco's right.  Maybe I should buy a round of beers for everyone.  I could have gotten my message across better but i am not sorry that I said it.  I think the two group categorization is way too simplistic.  There probably should be several groups.  I put Art Kendall up at the top because he has been doing this kind of consulting for a long time.  I certainly don't put myself in the top group because I have only been doing this seriously for about 5 years and only very sporadically before that.  Another reason why I am humbled is because I chaired the Dixon Committee this year.  We had 25 nominations.  Prior to doing this I was thinking of myself as a good candidate for this award but boy was I wrong!  There were at least 5 and maybe more whose achievements as consultants far exceeded mine.  You may appreciate that when you hear who this year's winner is.

    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------








  • 25.  RE:Rigor vs. simplicity (or, Consultants cause complications)

    Posted 05-01-2012 12:51
    Michael, if you do buy a round of beers, please do it at a place that has non-alcoholic beer, too.

    I'll buy the pizza.

    Best wishes,

    Nayak



    -------------------------------------------
    Nayak Polissar
    Principal Statistician
    The Mountain-Whisper-Light Statistics
    -------------------------------------------








  • 26.  RE:Rigor vs. simplicity (or, Consultants cause complications)

    Posted 05-01-2012 14:27

    Where are we going to meet?  If you all come to Philadelphia, we can go to Gino's, have the beer and some non-alcoholic beverages and some of our famous philly cheesesteaks.


    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------








  • 27.  RE:Rigor vs. simplicity (or, Consultants cause complications)

    Posted 05-01-2012 17:00
    Sometimes, simplest and most rigorous are synonomous.  Take a randomized two-period crossover study (two sample approach).

    Let Yj=.5(Period2-Period1 response), regardless of treatment order.

    Compare the means of the two orders via, say, a Sattherthwaite corrected t-test (or a regular t-test if you prefer).  The difference in sample means is unbiased for the average effect size in period 1 and period2, (equally weighted) regardless of carryover effects or order effects or imbalance in randomization.  This method has greatrer power than the one sample approach, and is essentially assumption free, thanks to its approximation to the permutation test.  Methods from Senn's book, for example, require far more in te way of assumptions, and are no more powerful unless you believe unverifiable assumptions behind them.  No diagnostic test can prove assumptions are correct.  They can either reject assumptions or are inconclusive on assumptions. 

    So toss mixed models for crossover designs and be more rigorous and every bit as powerful in the real world.

    Jon Shuster
    -------------------------------------------
    Jon Shuster
    University of Florida
    -------------------------------------------








  • 28.  RE:Rigor vs. simplicity (or, Consultants cause complications)

    Posted 05-01-2012 18:16
    Jon just a few comments since you put so much out there and some can be challenged.  Rightly you mention that diagnostic tests can't be used to strictly verify assumptions but to be pragmatic when assumptions in models are not rejected people will use the model.  It is the same when a null hypothesis is not rejected.  We are careful to say that we can't reject rather than say that we accept.  Practically speaking it is a minor distinction.  In time series analysis stationarity (strict or second order) is not an assumption that anyone can possibly check with data.  But statisticians will apply an ARMA model if the sample autocorrelation and partial autocorrelation look like those from a stationary ARMA model.  This is a pragmatic approach that has work well for many years and was proposed by George Box no less.

    Now the statement "So toss mixed models for crossover designs and be more rigorous and every bit as powerful in the real world." seems a little bold but you made a decent case but would you say the same thing in the face of missing data, something that occurs in many practical problems?

     

    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------








  • 29.  RE:Rigor vs. simplicity (or, Consultants cause complications)

    Posted 05-02-2012 07:25
    Certainly people do use the default (i.e., a priori, null, pre-existing, presumed, prevailing, standard) model.  It is the one that is already in place, maybe for years.  This mode of reasoning is analogous to that in the legal situation (in countries with Common Law) finding after the trial there was not sufficient proof to reach a given standard of proof. The proposed alternative theory, policy, or practice has not been proven superior. It remains.

    -------------------------------------------
    Arthur Kendall
    Social Research Consultants
    -------------------------------------------








  • 30.  RE:Rigor vs. simplicity (or, Consultants cause complications)

    Posted 05-01-2012 10:57
    Hey everybody, peace! 

    Nayak

    -------------------------------------------
    Nayak Polissar
    Principal Statistician
    The Mountain-Whisper-Light Statistics
    -------------------------------------------








  • 31.  RE:Rigor vs. simplicity (or, Consultants cause complications)

    Posted 05-01-2012 11:51

    I think the "Peace" comment applies but it should have arrived immediately after Michael's original comment.

    His comment was unnecessary but maybe along with Michael's usually excellent commentary unintentially makes a good point about communication skills.

    -------------------------------------------
    James Baldwin
    Station Statistician
    US Forest Service
    -------------------------------------------








  • 32.  RE:Rigor vs. simplicity (or, Consultants cause complications)

    Posted 04-29-2012 19:17
    I had same exact experience.  I proposed an imputation strategy tailored to the specifics of their problem. Instead, they opted for canned software which I didn't think was well suited to their problem.

    -------------------------------------------
    Daniel Scharfstein
    Johns Hopkins School of Public Health
    -------------------------------------------








  • 33.  RE:Rigor vs. simplicity (or, Consultants cause complications)

    Posted 04-29-2012 19:29

    How did you explain this to the client?  Did you tell them what the possible consequences are for using an inappropriate method?  How far di you go along with it? Did you collaborator on a paper using a method that you thought was inappropriate?  Statisticians are often under pressures to do things the way the client wants.  But I think we need to step up and say no.  We are the ones who are in position to understand what can go wrong.  It is not simply a matter of just doing what is most easy and convenient.
    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------








  • 34.  RE:Rigor vs. simplicity (or, Consultants cause complications)

    Posted 04-30-2012 09:41
    They paid for my proposal.  They decided not to go with it. It is hard to quantify, without extensive work, to know the inferential consequences of their decision  Anyway, that's where our relationship (amicably) ended. I do not do work for clients that I cannot stand behind.  Ultimately, it is my reputation that is on the line.  Fortunately, I am in a position where I can say no.

    -------------------------------------------
    Daniel Scharfstein
    Johns Hopkins School of Public Health
    -------------------------------------------








  • 35.  RE:Rigor vs. simplicity (or, Consultants cause complications)

    Posted 04-30-2012 09:51
    That is what I like to hear!  I hope people will do that even if they depend heavily on their consulting work for income and have a hard time getting clients.

    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------








  • 36.  RE:Rigor vs. simplicity (or, Consultants cause complications)

    Posted 04-30-2012 10:03

    This issue is just as relevant to a statistician who works full-time for an employer.  In either case I try to think like an engineer and ask, "What does my employer need."  I often find myself telling my (non-statistician) supervisor, "This approach is good enough for our purposes.  A more rigorous approach would make a dandy masters thesis or PhD dissertation [whichever applies] but the cost would outweight the benefit for us."

    The tricky part is deciding what is "good enough" and what is not justified on a cost/benefit basis.  Those decisions are usually case by case.  It's hard to imagine general rules except for asking, "If I were my client (or supervisor) what approach would I take?"  It's slightly easier for me because my supervisor and I work for the same employer, but I did try to approach it that way back in my indepenent consulting days.
    -------------------------------------------
    Emil M Friedman, PhD
    emil.friedman@alum.mit.edu (forwards to day job)
    emilfrie@alumni.princeton.edu (home)
    http://www.statisticalconsulting.org


  • 37.  RE:Rigor vs. simplicity (or, Consultants cause complications)

    Posted 04-30-2012 10:10
    The only distinction I would make is that in industry your employer may very well expect you to make that decision on your own.  Consulting with the boss is always a good idea.  Part of the reason you were hired may very well have been becuase the employer thought you would have the expertise to make those decisions related to statistics.  In the consultant/client relatinship i think it should be strictly up to the client regarding what if any research work you should be paid for.

    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------








  • 38.  RE:Rigor vs. simplicity (or, Consultants cause complications)

    Posted 05-01-2012 17:05
    Good point.  I always wound up discussing those things more explicitly with clients because I never knew them as well as I knew my supervisors in industry. 

    I don't remember any cases where the client had me not go down the analytical path I recommended, but that's probably because my and their thinking evolved during the discussions until we reached consensus.  I do remember cases where someone used an experimental design that differed substantially from what I recommended.  When I'm an employee that usually means extra work for me, but when I'm a consultant it just means more billable hours.
    -------------------------------------------
    Emil M Friedman, PhD
    emil.friedman@alum.mit.edu (forwards to day job)
    emilfrie@alumni.princeton.edu (home)
    http://www.statisticalconsulting.org