Discussion: View Thread

Back to discussions
Expand all | Collapse all

a consult that was not ideal

  • 1.  a consult that was not ideal

    Posted 01-30-2012 12:28

    I want to share a recent consulting experience with you and hope to get some feedback as to how i might have approached this better.
    Here is a sketch of the problem:
    A principal investigator at Lankenau Medical Center is a knee replacement specialist.  He collected angle measurement estimates from four examiners of three Xrays for each of 31 patients.  The issue was to determine how well the raters agreed on the angle estimates.  One of the difficulties I have is that sometimes the PI is collaborating with one or more of his residents and he gives the research problem to the resident to do the analysis and the write-up.  Unfortunately the PI after giving initial direction to the resident stays out of the data analysis until the end when the paper is drafted.

    So in this case I consulted with the resident only and subsequent email commnucations were with the resident only.  The discussion started out something like this. "I have done the summary analysis of the data and we are practically ready to write up the report except that we want to compute Kappa statistics for interrater agreement and I don't know how to do that.  Can you compute the Kappas and their p-values so I can add that to the abstract and the report?"  I naturally followed this up with a lot of questions.  Most importantly I wanted to know how to categorize the level of agreement.  He told me that agreement within 1 degree was considered excellent, 1-2 good, 2-3 fair and 3 or higher poor.  So I took that and created the agreement tables to compute both the interrater agreement and the intrarater agreement. For all my correspondences and results I sent emails to both the resident and the PI.  I would get responses from the resident and followup phone calls.  Eventually the analysis was completed and the final results sent to both.  It is at this point that I get a call from the PI asking questions about the analysis.  In the course of the conversation I find out some aspects of the problem were not conveyed to me correctly by the resident.  Most significantly the PI wanted continuous measure of agreement rather than categorical measurements.  This meant that the work I did computing the Kappas was wasted.  The PI had a difficult time explaining what analysis he wanted but he clearly wanted to use methods that were commonly used in the medical journals for these types of data.  Eventually I realized that the data was well set up for an R&R gauge analysis and that correlations that measure agreement that were a biproduct of the special type of ANOVA was what he needed.  This work was done but was a little rushed due to the timing of the discussion.
    Perhaps there would be a better way to engage the PI in the analysis early on.  The emails were apparently overlooked until the end.  But that may be easier said than done because the PI is very busy with his medical practice.  He therefore leaves the bulk of the work to the resident to conserve his time and also because these projects are intended to be learning experiences for the resident.
    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------


  • 2.  RE:a consult that was not ideal

    Posted 01-30-2012 13:06

    Except for the waste of your time and the rush at the end, it doesn't sound like things ended up too badly. They got the test they needed.  

    The only thing I can suggest is to let the day-to-day contact (the resident in your case) know that you will ask for signoff from the person in charge (the PI) before final analysis is done.  If everyone knows that this will be a step in the process ("after I get signoff on the analysis, I'll...") then you are more likely to get attention at the right time.   

    -------------------------------------------
    Bridget Bly
    -------------------------------------------








  • 3.  RE:a consult that was not ideal

    Posted 01-30-2012 14:00

    Thank you for the good suggestion.  Yes things ended up fine I think.  The problem is the rush at the end and the waste of effort.  I am one who preaches clear communication between the statistician and the clients to avoid this.  So I don't like it that I let myself get into this trap.  I need to keep in mind based on past experience that the resident sometimes does not interpret what the PI wants and therefore I should always make sure that I include the PI in the dialogue sufficiently to catch such problems without being seen as being a pest.
    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------






    Original Message


  • 4.  RE:a consult that was not ideal

    Posted 01-30-2012 15:49

    I gave your story a little thought.  Before I retired and started my own consulting firm I was always asked to do more in less time.  So things like this I would tend to avoid by only working with the primary PI.  But now as a consultant I have a completely different perspective.  I charge by the hour and I point out that I could have been more efficient if I had more information up front.  Some times money, cost and time are good motivators for the PI.

    -------------------------------------------
    Rocco Brunelle
    Senior Statistician
    Bowsher Brunelle Smith LLC
    -------------------------------------------






    Original Message


  • 5.  RE:a consult that was not ideal

    Posted 01-30-2012 21:26


    -------------------------------------------
    Milton Goldsamt
    Consulting Research Psychologist and Statistician
    -------------------------------------------
    In most of the messages already written, all good advice, it's seemed to be one-way, that is, the consulting statistician carrying out work as he/she understands it or based on guidelines provided by the research assistant, not the PI.  I had a similar experience, and before I began, not only did I have questions that I posed to the PI and got answered, but also I put together a "memo of understanding" that stated what I would do to carry out the request, and how I would do it. I was careful to phrase the language in broad enough terms so that it was understandable to the "customer." I submitted that memo of understanding, and didn't begin until I got a confirmation that what I planned to do was what they wanted.  Perhaps such a two-way arrangement and mutual understanding would also help.





    Original Message


  • 6.  RE:a consult that was not ideal

    Posted 01-30-2012 21:36

    Milton, I think your approach is very sensible and should avoid a lot of wasted time and misunderstanding.  I will try to  follow this approach in the future - never start working on the project until a brief, written "statistical analysis plan" has been reviewed and approved. 
    -------------------------------------------
    Edith Zang
    Independent Consultant
    NYCASA
    -------------------------------------------






    Original Message


  • 7.  RE:a consult that was not ideal

    Posted 01-30-2012 21:51
    While I agree that Milton's approach is a good way to do things in the environment of private consulting, I don't think it would be prectical for me.  I am in an environmental where I represent the entire statistics department for the Mainline Health System.  I am juggling several consulting projects at the same time.  I am not in the position to set ultimata on our clients and I should not turn down any requests except in special circumstances where my supervisor decides that other projects have higher priority and I can afford the time.  I don't use shortcuts but on the other hand I can't afford to take the amount of time drafting up an agreement.  Also I am not in a contractual situation.  I think a reasonable solution forme would be to insist on getting the PI involved in one or two consulting sessions and make sure that he reviews any interim analyses.  but it would be inappropriate for me to hold off on a project which may have some important deadline in order to get all the I's dotted and T's crossed.

    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------






    Original Message


  • 8.  RE:a consult that was not ideal

    Posted 01-31-2012 10:07
    Even in the Pharma industry, the protocol and the Statistical Analysis Plan have to be signed off on by key team members before they can be implemented. 

    -------------------------------------------
    Edith Zang
    Independent Consultant
    NYCASA
    -------------------------------------------






    Original Message


  • 9.  RE:a consult that was not ideal

    Posted 01-31-2012 10:35

    These are all good points.  Most of my clients get me involved early in a project and we have a short project plan that outlines my involvement (i.e. study design, protocol, sample size, SAP, analyses, etc.) the timeline and the projected cost.  Along the way we put together a protocol and an SAP that are both reviewed and approved by the client.  I like these types of projects.  They are typically well defined and you know what you need to do.  However, even in these types of projects someone will decide to do a poster or paper at the last minute and ask for some not well defined analyses.  It's nice to have things well defined and this is very important in a clinial trial but for adhoc analyses or abstracts or papers this does not always happen.  I don't mind helping my clients even when the requests are not always well defined.  I'm happy they searched me out and asked me to help.  I feel it is sort of like being on a team and providing your expertise to the team. 

    Now this brings up another point - authorship.  At the current stage of my carreer I'm not concerned about being an author.  Actually I try to not let my clients know who my other clients are, but once in a while a client will want me to be a coauthor and right away all of my other clients find out.  I've worked for clients that are developing similar drugs/devices and it makes things awkward.  I sometimes try to politely ask not to be a co-author but my clients sometimes insist. 

    -------------------------------------------
    Rocco Brunelle
    Senior Statistician
    Bowsher Brunelle Smith LLC
    -------------------------------------------






    Original Message


  • 10.  RE:a consult that was not ideal

    Posted 01-31-2012 10:41

    They cannot make you a co-author against your will.  All reputable journals require you to sign off on becoming a co-author.  You may explain to your colleagues that if there are any statistical questions from the reviewers you will be willing to help them answer those questions.  Also, it may be OK for you to be included in the Acknowledgements.

    -------------------------------------------
    Edith Zang
    Independent Consultant
    NYCASA
    -------------------------------------------






    Original Message


  • 11.  RE:a consult that was not ideal

    Posted 01-31-2012 10:56
    Let's not forget that we have an ethical obligation to be listed as a co-author if we made a significant contribution to the manuscript. Leaving your name off such a publication is akin to putting your name on one to which you contributed too little to qualify. Economic considerations should not take precedence over ethical obligations to our profession and to scientific inquiry.

    -------------------------------------------
    Thomas Sexton
    Professor and Associate Dean
    Stony Brook University
    -------------------------------------------






    Original Message


  • 12.  RE:a consult that was not ideal

    Posted 01-31-2012 11:06

    Tom,

    I haven't heard about that type of obligation.  Where is that
    written? 

    -------------------------------------------
    Daniel Jeske
    Professor and Chair
    University of California
    Department of Statistics
    -------------------------------------------






    Original Message


  • 13.  RE:a consult that was not ideal

    Posted 01-31-2012 11:31
    I'm not an expert in ethical guidelines for publication but I do know that the research enterprise should be conducted in a manner in which everyone knows who is saying what. I know that a co-author may remove their name from a paper if they cannot agree with other co-authors about what to say or how to say it. But that is very different from trying to mask your involvement in a piece of work when that involvement constitutes co-authorship.

    -------------------------------------------
    Thomas Sexton
    Professor and Associate Dean
    Stony Brook University
    -------------------------------------------






    Original Message


  • 14.  RE:a consult that was not ideal

    Posted 02-01-2012 13:24
    I am not an expert in ethical guidelines for publication, either.  However, the ethics (or lack thereof) of "ghost writing" is a recent topic of conversation in healthcare.  There are times when a medical device or pharma manufacturer/marketer might prefer NOT to have their contributing employees listed as authors, feeling that the appearance of 3rd-party "objectivity" (by omitting names/affiliation of some potential authors) will strengthen the product's position.  This conflicts with Tom's clearly stated phrase "the research enterprise should be conducted in a manner in which everyone knows who is saying what."

    -------------------------------------------
    Mark Martin
    Siemens Healthcare Diagnostics
    -------------------------------------------






    Original Message


  • 15.  RE:a consult that was not ideal

    Posted 02-01-2012 19:44

    I follow the recommendation of the Institute for Mathematical Statistics New Researcher's Survival Guide regarding authorship and politely refuse Acknowledgments which have no rights like co-authorship.

    "4.2.3 Co-authorship
    If you write any part of the paper, contribute substantially to the analysis, or put in a substantial amount of
    time, you have a right to co-authorship. This should be established unequivocally from the time you start doing anything more than giving oral advice or after the third meeting. This should always be the case if the client is a researcher, but may be more problematic if the client is a graduate student doing dissertation research. In the latter case, ask for co-authorship on any papers coming from the dissertation which involve your part of the work.  If you are a co-author, insist on reviewing the final draft of the paper or report to ensure accurate representation of the statistical analyzes.

    If you are not a co-author, avoid acknowledgments or other recognition of your contribution unless the investigators did precisely what you suggested. Otherwise, you will have no say in what was written, and you will not receive credit for your work, but you may be blamed if something went wrong.

    As co-author, the statistician generally writes the statistical methods and results sections of the paper. This can be submitted to the client as a final report, prior to writing the paper, but should be in a format suitable for publication, including relevant tables and plots."

    http://www.imstat.org/publications/books/NewResearchersGuide.pdf


    the ASA Ethical Guidelines include some language on authorship

    "C. Responsibilities in Publications and Testimony


    1. Maintain personal responsibility for all work bearing your name; avoid undertaking work or coauthoring publications for which you would not want to acknowledge responsibility. Conversely, accept (or insist upon) appropriate authorship or acknowledgment for professional statistical contributions to research and the resulting publications or testimony.
    2. Report statistical and substantive assumptions made in the study.
    3. In publications or testimony, identify who is responsible for the statistical work if it would not otherwise be apparent.
    4. Make clear the basis for authorship order, if determined on grounds other than intellectual contribution. Preferably, authorship order in statistical publications should be by degree of intellectual contribution to the study and material to be published, to the extent that such ordering can feasibly be determined. When some other rule of authorship order is used in a statistical publication, the rule should be disclosed in a footnote or endnote. (Where authorship order by contribution is assumed by those making decisions about hiring, promotion, or tenure, for example, failure to disclose an alternative rule may improperly damage or advance careers.)
    5. Account for all data considered in a study and explain the sample(s) actually used.
    6. Report the sources and assessed adequacy of the data.
    7. Report the data cleaning and screening procedures used, including any imputation.
    8. Clearly and fully report the steps taken to guard validity. Address the suitability of the analytic methods and their inherent assumptions relative to the circumstances of the specific study. Identify the computer routines used to implement the analytic methods.
    9. Where appropriate, address potential confounding variables not included in the study.
    10. In publications or testimony, identify the ultimate financial sponsor of the study, the stated purpose, and the intended use of the study results.
    11. When reporting analyses of volunteer data or other data not representative of a defined population, include appropriate disclaimers.
    12. Report the limits of statistical inference of the study and possible sources of error. For example, disclose any significant failure to follow through fully on an agreed sampling or analytic plan and explain any resulting adverse consequences.
    13. Share data used in published studies to aid peer review and replication, but exercise due caution to protect proprietary and confidential data, including all data that might inappropriately reveal respondent identities.
    14. As appropriate, promptly and publicly correct any errors discovered after publication.
    15. Write with consideration of the intended audience. (For the general public, convey the scope, relevance, and conclusions of a study without technical distractions. For the professional literature, strive to answer the questions likely to occur to your peers.)"

    H. Responsibilities of Employers, Including Organizations, Individuals, Attorneys, or Other Clients Employing Statistical Practitioners

    6. Do not include statistical practitioners in authorship or acknowledge their contributions to projects or publications without their explicit permission."

    http://www.amstat.org/about/ethicalguidelines.cfm

    in addition, the International Committee of Medical Journal Editors has some very helpful material on their website regarding criteria for authorship.
    http://www.icmje.org/ethical_1author.html



    -------------------------------------------
    Stuart Gansky
    University of California, San Francisco
    -------------------------------------------






    Original Message


  • 16.  RE:a consult that was not ideal

    Posted 02-01-2012 20:26

    I think we can all agree that this is what should be done.  This is good advice but not really enforceable.
    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------






    Original Message


  • 17.  RE:a consult that was not ideal

    Posted 02-01-2012 20:10

    i don't know why people are saying that they are not experts in ethics.  To me anyone with a moral compass has a sense of what is ethical even though we might not be able to give a clear discriminating definition.  I think in discussing issues with publications we should differentiate between unethical behavior and illegal behavior.  Laws can be enforced ethics cannot. Organizations can make rules to encourage ethical behavior but compelling people to be ethical and detecting unethical behavior can be very difficult.   To use a ghostwriter to deceive a regulator authority that the work was done by a neutral party and on the other hand an author concealing his authorship because he is serving two competing masters are unethical.  I think most of us probably agree on that. In the case of the author who wants to refuse authorship, no one should compell him to become an author.  That should be his choice and we cannot be sure whether his motives are ethical or not.  Also do we have the right to tell an author that he can't hire writers to add clarity to the submission?  Again there is a possible unethical ulterior motive but a perfectly reasonable and ethical motive is also possible.  We can set rules of ethics and hope people will follow them.  But dictating to them is problematic.  For a journal I think the important thing is that the referees have enough information to detect fraud, and distinguish objectivity from bias.  If the information is lacking they have the authority to reject the paper.

    Of course the reader should know who is writing the paper to help them judge what is written.  But what is ideal is not always an enforcible outcome.

    I want to apologize for a post that wound up 4 or 5 times in your mailbox.  I tried submitting it and the submission timed out.  So it didn't go through and I tried a few more times until I got the check that indicated the message was received.  I did not know that the other submissions would enter a queue that could eventually get them submitted.  But apparently that happened.
    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------






    Original Message


  • 18.  RE:a consult that was not ideal

    Posted 02-02-2012 16:51
    It is quite possible to have a strong personal moral compass, yet still be unaware of many instances in our complicated world and market system where we could unknowingly break legal and/or ethical rules in certain circles.  It takes practice (and usually education and/or mentoring) to apply one's moral compass well across a variety of settings.

    Because I am still ignorant of many consulting situations, I do not declare to be "expert" because I hope that others will also chime in with their knowledge or experience.  I realize I am about to change the subject from "publishing" to "meals during consultations" -- to make a point about how easy it is to "not be an expert in ethics" in areas that may affect us personally.

    Here is an example where a personal moral compass might be insufficient, without knowledge of both the law and cultural etiquette norms:

    If I (being an employee with a healthcare company) host two experts on-site to consult for a day, it might seem fair and courteous to buy a modest lunch for the 3 of us during that workday.  However, if one of the two consultants is a government employee and the other is not, I can legally buy lunch only for the non-government employee.  Therefore, to obey the law I cannot buy lunch for everyone.  The remaining options are to tactfully offer the food to non-government employees (however that can best be accomplished, and allow the government employee to select/buy his own food), or "be fair" by clarifying up front that everyone will be on their own for lunch. 

    I would not have even known these legal/ethical rules without special on-the-job training.  Where else are such things discussed and taught?  Yes, I am a firm believer in conscience and personal responsibility.  Ongoing education and discussion in this area can strengthen personal resolve and appropriate actions.

    -------------------------------------------
    Mark Martin
    Siemens Healthcare Diagnostics
    -------------------------------------------






    Original Message


  • 19.  RE:a consult that was not ideal

    Posted 02-02-2012 17:25
    I agree with what you say but I think you may be misinterpeting what I said or at least the intended meaning of what I said.  I was just trying to say that I did not see the need for people to be apologetic about ethics.  If by expert you mean someone knowledgeable in all the rules and codes of ethics, I dare say very few if any of us satisfy that definition. My point was that unlike mathematics, physics or psychology which are disciplines which require some special training to be labelled an expert then without that training we are not experts and when commenting on those subjects we should qualify what we say by admitting our lack of expertise.  With ethics I think the situation is somewhat different.  I do not think we need a course in ethics to have a good sense of what is ethical. I do not think one needs say a degree in ethics to be qualified to speak about ethics.  Since so few of us would be qualified as an expert in ethics by the strict definition I gave above, it really is true that we are not that kind of expert in ethics and it doesn't need to be said.

    On the other hand if we are talking about something like a professional societies codes of ethics then we really should know that code to speak about it and if we are commenting on it without such expertise we probably should point out our lack of expertise.

    My statement was simple and easily misconstrued.  I just wanted to express my intent even though it was probably not something worth arguing about.

    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------






    Original Message


  • 20.  RE:a consult that was not ideal

    Posted 02-03-2012 11:17
    Michael,
    Thank you for the very clear clarification.  Points well taken.  I also appreciate your raising the topic which led to others sharing/quoting some of our professional society ethics guideline documents.  I printed those to keep for quick reference.

    -------------------------------------------
    Mark Martin
    Siemens Healthcare Diagnostics
    -------------------------------------------






    Original Message


  • 21.  RE:a consult that was not ideal

    Posted 01-31-2012 11:40
    At least in concern to medical journals, the International Committee of Medical Journal Editors provides authorship guidelines that can be found at:
    Uniform Requirements for Manuscripts Submitted to Biomedical Journals: Ethical Considerations in the Conduct and Reporting of Research: Authorship and Contributorship

    These guidelines state:
    Authorship credit should be based on 1) substantial contributions to conception and design, acquisition of data, or analysis and interpretation of data; 2) drafting the article or revising it critically for important intellectual content; and 3) final approval of the version to be published.

    -------------------------------------------
    Sid Ketchum
    Graduate Student
    Virginia Commonwealth Univeristy, Dept of Biostatistics
    Richmond, VA
    -------------------------------------------






    Original Message


  • 22.  RE:a consult that was not ideal

    Posted 01-31-2012 12:10

    James presents what we know and all probably agree on.  No one should be included as an author if they did not contribute substantially to the article.  These guidelines came as a reaction to the common practice in medicine to include twenty or more people as coauthors, many of whom are given courtesy authorship by the "real" authors and some who are sponsors or just recognized authorities in the field.  Hopefully these ethical guidelines will cut down on these practices but I do not think they will eliminate them.  i hope that people will not sign that they made a contribution when they didn't.  There are even some journals that require each author to spell out exactly what their contribution to the paper was.

    But contrary to what Thomas Sexton said I do not think that anyone should be forced to take authorship.  Consultants that contribute to a paper are providing a service and if they do not want to be an author that s their right.  I think the onus is on the authors to ask.  To not offer authorship to someone who made a significant contribution is unethical in my opinion and if the consultant declines authorship they should acknowledge the contribution.  Authors should be required to explain how their data was collected and analyzed and even make the data available to the readers to check their results if desired.  But I do not think they should be required to say who they paid to do the analysis.
    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------






    Original Message


  • 23.  RE:a consult that was not ideal

    Posted 01-31-2012 11:08

    The authorship issue is touchy.  I work at a research institute where publications have a high priority.  So my supervisor wants me to be a coauthor on all the papers I contribute on.  Whenever I am involved in a research project my role is significant.  I also consult with the institute's laboratory scientists and the President of our institute insists that they include me for at least 5% of mine time on all grants the submit.  Ther is actually pressure on me to insist on authorship and it only became an issue once.  Ethically i think that the statisticians contribution often deserves coauthorship when only an acknowledgment is given.  Even an acknowledgment is better than being ignored.  I have even had an investigator have the attitude that if I am a paid consultant I should not get authorship and that rationale is used to try to convince me to do the work pro bono with authorship as the enticement.  That was one of the few clients that I actually turned down.

    Now Rocco raises a point that I have not thought of before.  If private consulting is your full time job and there is no incentive to publish, you might prefer to not be included in the paper at all even through acknowledgment.  On the side of ethics though I think one should avoid working for competing clients at the same time.  There may be a conflict of interest and certainly knowledge of one clients proprietary data provides with information that you can use to the competitors advantage, even if you are trying to be ethical and fair to both parties.  The statistician does more than just objectively analyze the data.  He is often involved in strategic planning as well as planning the analysis.  these two forms of planning are difficult to separate and probably should not be separated.  The importance of Bayesian methods in statistics is that prior knowledge is incorporated to improve efficiency of the statistical plan and to play a role in the final decision process.

    The issue of competition is the reason that so many businesses insist on having consultants sign confidentiality agreements.

    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------






    Original Message


  • 24.  RE:a consult that was not ideal

    Posted 01-31-2012 10:45

    Yes.  When I was in the pharmaceutical industry that was wonderful.  Our job and our authority was mandated by the FDA.  But alas that is not where I am now and I think the same can be said for most private consultants also.  The case I gave was an investigator initiated study.  It involved no patients.  The PI probably didn't even need IRB approval.  Even the small investigator initiated studies usually have a protocol and when they do i look them over and correct them.  Here serving the hospital system when I work on clinical trials they are investigated initiated.  Some have internal foundation support others are funded by the NIH.  On some occasions we are involved supporting a biotech or pharma company.  There we tend to have oversight and can critique the protocols but are not the holders of the data.  As an example, for the RE-LY trial I am on the review committee for publications.  This trial has been completed and the drug approved and now marketed as Pradaxa.  We do substudies on the data for scientific value and postmarket work to identify commercial advantages over warfarin and the new competitors.  Some of the investigators are hostile to the statistics group but fortunately the sponsors wants us to be a watchdog to make sure that the protocols are up to all standards and that statistical plans are produced and reviewed by the committee.  That is great and works well but currently represents no more than 1% of my business activity.  Many investigations are similar to the one I described all this one was the first time I had the PI ask me to scrap the analysis to do something different.  These investigations sometimes only require statistics because the investigator knows that he will need boxplots and p-values to get his results published in the journal of his choice.  Too often I am brought in to rescue a paper that was criticized by the referees based on a statistical issue.  Situations are not ideal but I usually do good and often can salvage a paper when the experiment was done reasonably.

    In this particular case i really think I could have handled it better.  But still it was a difficult situation.  It is just totally unrealistic to suggest that in the environment that I work under I could invest the time to write up a detailed analysis plan for every project and insist that it be agreed upon before I start work.  With a staff consisting of me and one junior volunteer this idea is totally impractical.  I think there are other ways to sensibly handle these situations.
    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------






    Original Message


  • 25.  RE:a consult that was not ideal

    Posted 01-31-2012 10:52

    Completely sympathize with your situation. Can't think it is very satisfying or that the highest quality research can result. I would hope you have an advocate in a supervisor or someone else who can help you try to change the culture. Having to rescue manuscripts is really not the spirit of what I think statistical consulting and/or collaboration should be. All the best.

    -------------------------------------------
    Rhonda Rosychuk
    Univ of Alberta
    -------------------------------------------






    Original Message


  • 26.  RE:a consult that was not ideal

    Posted 01-31-2012 11:54

    In my situation I am not bothered by the prospect of salvaging a manuscript.  I actually enjoy it and I often get coauthorship when I do it.  If the design is terrible and there is a need for inference but no valid way to do formal statistical inference then there is nothing I can do for them.  I have freedom to make these decisions.  The culture here is excellent.  I have all the respect I want.  Often what happens is that the data are collected using appropriate randomization but no one did a power calculation upfront.  I do the formal analysis and find statistical significance.  Even if the sample size is small, if the difference is clinically significant and the p-value is small I think the analysis is valid and I can persuade the referees that the study is valid.  My thinking on this has been tempered by discussions we have had in this forum about interpretation of p-values when sample size is small.  In other cases I might think of a different way to analyze the data that adds value to the publication.  Of course I understand the dangers of post hoc analysis but all the assumptions and the strengths and weaknesses of the paper are spelled.  Some of the better medical journals actually require this.  Also some studies are intended to make inferential comparisons but do not or cannot use randomization.  The authors may not know about observational studies but fortunately they collect a wealth of medical data on the patients that covariates can be found to balance through case-control matching methods or by covariate adjustment in regression.
    I will describe a case where I refused to help with a publication.  An client that I will leave anonymous conducted a survey where respondent were self-selected.  They had to find the website and then have an interest in the survey to participate in responding to the survey.  The client was totally passive.  There was no control of who the respondents were.  They left this passive survey on the site for three years and simply collected the thousands of responses that they got.  I was asked to analyze the data.  In fact when i looked at the data, it was the first time anyone looked at it.  There were many problems with the data but i told them we could provide summary statistics and would have to state that this was a self-selected sample and because it was not selected by the client no statistical inferences could be drawn about broader populations.  This was just interesting summary information about subjects who would choose to respond.  Since it was on the web responses came in from the UK and Australia although predominantly from the US.  The only identifiers we had were zip code (or postal code) and email address when the respondent chose to give that information.  Sometimes fictitious emails were given.  We did some cleaning of the data.  Everytime I discussed the data with the client I emphasized that no inference could be drawn from this data.  Everything went fine and the client was very grateful to us and acknowledged that in almost all their emails. But they decided to submit the results to a conference for a poster.  So we agreed to provide summary information and a caveat that the sample was self-selected and hence not random.  This was agreed to.  But just before submission they told us that one of the investgiators told them that the poster would not be accepted without p-values.  I told them that we could not provide p-values. Then the investigator wrote that although the sample was not random we could present p-values and the statistical analysis would be valid.  At that point I got a little nasty with him.  But the bottom line was that we refused to provide p-values and would not allow our names to be included in the abstract and poster if p-values were put in.  The investigator was at Harvard and I think if he were determined enough he could find someone to give him p-values.  My current supervisor is an MD with experience in the pharmaceutical industry and contract research organizations.  he supported me 100%.  As a follow-up this week they decided to submit summary statistics only for abstracts that went out to two conferences.  We reviewed the abstracts which were acceptable and my assistant and I were included as coauthors.
    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------






    Original Message


  • 27.  RE:a consult that was not ideal

    Posted 01-31-2012 12:26
    I agree that the FDA does a good job enforcing adherence to scientific and statistical principles. It is time that scientific/medical journals and granting institutions also established mechanisms to enforce that studies are conducted according to an approved protocol and that all statistical analyses are carried out correctly by qualified statisticians.  There also has to be more of an oversight of the accuracy of the data.  Many years ago I was asked by a researcher to reanalyze his data of "40 patients" and see if I could elaborate on the results that he already wrote up in a draft manuscript.  I tried and tried but could only find 20 patients in the data.  When I finally confronted him with this problem, he admitted that he had "intended" to include 40 patients in his study but was only able to find 20.  Nonetheless, the results in his manuscript referred to 40 patients.  While I did not perform the analysis and the MS was never published, this is an extreme case that demonstrates the need for the closer oversight of published data.  
    -------------------------------------------
    Edith Zang
    Independent Consultant
    NYCASA
    -------------------------------------------






    Original Message


  • 28.  RE:a consult that was not ideal

    Posted 01-31-2012 14:06

    I understand your sentiments Edith.  I agree that publication standards and policy still can use improvement in the medical field.  But it is the editorial standards and policies that need to fetter out the bad papers.  There should be at least one person with statistical expertise reviewing every paper that is submitted with statistical content.  I don't think you should go so far as to require that all statistical analyses be carried out by qualified statisticians.  I don't think that we as a profession should be saying who must do data analysis for a paper.  We must only insist that the analysis be carried out properly.  The statistical judge of the paper will be able to determine if it is carried out properly and if the information is lacking to determine that the analysis was done properly he has the right to demand proof including adequate description of the problem, the data, the method of collection and the methods of inference.  Also the data should be made available to the referee to check the results as he sees fit.  There are still instances in the literature where investigators committed fraud by changing data or performing "data cleaning" methods without reporting them.  This would be very much like the way the FDA operates reviewing clinical trials and registration submissions.  They require that sound statistical practice is carried out and will check to see that their guidances are followed requiring justification when a guidance is not followed.  But they do not require "qualified statisticians" carry out the analysis.  In effect the sponsors must use qualified statisticians because they want to ensure that approvals are not sabotaged by faullty design or analysis.
    Even if we accept that statistical analyses for papers must be carried out by qualified statisticians, we must then define what a "qualified statistician" is.  Here I think it would be difficult to get agreement.  Is it anyone with at least a Bachelor's degree in statistics, or would it be enough to have a degree in a related field such as epidemiology, actuarial science, or bioinformatics?  Maybe a Bachelor's degree is not enough.  Do we require a masters or a PhD?  Maybe there is a certain amount of units in statistics courses that would be required without necessarily any degree in statistics. Maybe accreditation by a professional society would do?  I think there are pros and cons to any of these definitions and we could argue them for hours and hours.  The best thing to do is to leave the quality of the paper to be judged by appropriate referees just like the FDA leaves it to their reviewers and expert panels to decide the merits of an application for licensure.
    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------






    Original Message


  • 29.  RE:a consult that was not ideal

    Posted 01-31-2012 14:46

    I see Michael's point and take back my suggestion about using a qualified statistician to analyze the data.  However, as he says, there should always be a credible statistical review conducted by ....someone "qualified"?   At one point we need to be assured that the results are correct, and the question is how do we determine that?  Perhaps the only way out would be to establish a licensing process for practicing statisticians, much as medical treatment and legal representation can only be provided by licensed practicioners. What do you think?
    -------------------------------------------
    Edith Zang
    Independent Consultant
    NYCASA
    -------------------------------------------






    Original Message


  • 30.  RE:a consult that was not ideal

    Posted 01-31-2012 15:00

    The ASA has debated this for at least 20 years.  The original idea was rejected.  A few years ago the question resurfaced and a volunatry system was agreed on and instituted.  You pay a fee for a five year accreditation.  A committee of statisticians review your application and a decision is made.  I applied last year and got accredited.  Once you have a completed application with a few references (and they do contact references) the process does not take very long.  The professional statistician logo gets placed in your profile on the ASA website.  I got it because I feel it can help me get clients who can't simply judge my competence based on my resume.  The UK, Australia and Canada all have accreditation systems.  But for Edith's idea to work there would have to be a mandatory global system.  It is not practical and not desirable for many well-qualified statisticians.  Another approach was to have a standard test to be passed and this was rejected by ASA a long time ago.

    For more information check with Ron Wasserstein ASA Executive Director
    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------






    Original Message


  • 31.  RE:a consult that was not ideal

    Posted 02-01-2012 12:12

    I agree with Edith.  I have been consulting since 1984.  In the "early days" there were no plans for the analysis - it was more interactive.  Now all the analyses - no matter how trivial, are reflected back to the client for approval prior to the initiation o fthe work.  For a large project it would be a formal Statistical Analysis Plan (we do pharma as well); for small projects sometimes a short e-mail asking for the client's agreement.  Often there is clarification at this early stage - and that is what we need - so we can "do it right the first time".

    I think this actually saves time.  Most of us are juggling a lot (we have 23 active projects this month), and often clients put things on hold.  If you already have a brief description of the planned work, re-activating is a lot easier.  The first few "plans" may seem tedious, but once you get a template you like, it is really simple to execute - and if it really complex to write up - then I think you will be happy to have this "roadmap" to move forward and execute.
    -------------------------------------------
    Janet McDougall
    President
    McDougall Scientific Ltd
    -------------------------------------------






    Original Message


  • 32.  RE:a consult that was not ideal

    Posted 02-01-2012 13:38
    I asked for advice on this situation and eveeryone has given good advice that is worth thinking about.  But I was looking at this as a situation that was unusual for me.  I don't think increased formalization of my consulting services is necessary for me as the Director of Biostatistical Services.  I have an excellent working relationship with almost all of my clients.  This was the first time in three years that this problem has come up.  I feel that is it is not broke don't fix it.  The idea of formal agreements makes sense to me in a business situation that involves contracts and payment for specific services.  It would make sense for me to do this in my private consulting practice.  But at Lankenau I am paid for a general consulting service.  My organization has thrived because I have satisfied customers and management support that makes it possible for me to never sacrifice my principles.  The business continues to grow.  I am supported by general hospital and institutional funds as well as government and industry grants.  My clients are research scientist and clinical trial investigators.  They are not private businesses.  Such formality which would be looked at as necessary red tape for doing business could be looked at as wasted time and money by my clients.  To do this could possibly hurt my business more than it helps it.  Most of my clients know my reputation as a professional statistician and accept my suggestions without objection.  In the few cases where I get resistance I take a strong stand.  On some issues compromise and flexibility is appropriate. This is particularly true with tradeoffs between statistical accuracy and economic feasibility.

    I think your suggestion is good advice for the majority of consultants in this egroup.  They are either private consultants or part of a consulting business.  My situation is different and may be similar to internal consulting at universities.  Unlike baseball caps one size does not fit all in the arena of statistical consulting.

    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------






    Original Message


  • 33.  RE:a consult that was not ideal

    Posted 02-01-2012 14:27

    I asked for advice on this situation and everyone has given good advice that is worth thinking about.  But I was looking at this as a situation that was unusual for me.  I don't think increased formalization of my consulting services is necessary for me as the Director of Biostatistical Services.  I have an excellent working relationship with almost all of my clients.  This was the first time in three years that this problem has come up.  I feel that is it is not broke don't fix it.  The idea of formal agreements makes sense to me in a business situation that involves contracts and payment for specific services.  It would make sense for me to do this in my private consulting practice.  But at Lankenau I am paid for a general consulting service.  My organization has thrived because I have satisfied customers and management support that makes it possible for me to never sacrifice my principles.  The business continues to grow.  I am supported by general hospital and institutional funds as well as government and industry grants.  My clients are research scientist and clinical trial investigators.  They are not private businesses.  Such formality which would be looked at as necessary red tape for doing business could be looked at as wasted time and money by my clients.  To do this could possibly hurt my business more than it helps it.  Most of my clients know my reputation as a professional statistician and accept my suggestions without objection.  In the few cases where I get resistance I take a strong stand.  On some issues compromise and flexibility is appropriate. This is particularly true with tradeoffs between statistical accuracy and economic feasibility.

    I think your suggestion is good advice for the majority of consultants in this egroup.  They are either private consultants or part of a consulting business.  My situation is different and may be similar to internal consulting at universities.  Unlike baseball caps one size does not fit all in the arena of statistical consulting.



    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------






    Original Message


  • 34.  RE:a consult that was not ideal

    Posted 02-01-2012 14:53

    I asked for advice on this situation and everyone has given good advice that is worth thinking about.  But I was looking at this as a situation that was unusual for me.  I don't think increased formalization of my consulting services is necessary for me as the Director of Biostatistical Services.  I have an excellent working relationship with almost all of my clients.  This was the first time in three years that this problem has come up.  I feel that is it is not broke don't fix it.  The idea of formal agreements makes sense to me in a business situation that involves contracts and payment for specific services.  It would make sense for me to do this in my private consulting practice.  But at Lankenau I am paid for a general consulting service.  My organization has thrived because I have satisfied customers and management support that makes it possible for me to never sacrifice my principles.  The business continues to grow.  I am supported by general hospital and institutional funds as well as government and industry grants.  My clients are research scientist and clinical trial investigators.  They are not private businesses.  Such formality which would be looked at as necessary red tape for doing business could be looked at as wasted time and money by my clients.  To do this could possibly hurt my business more than it helps it.  Most of my clients know my reputation as a professional statistician and accept my suggestions without objection.  In the few cases where I get resistance I take a strong stand.  On some issues compromise and flexibility is appropriate. This is particularly true with tradeoffs between statistical accuracy and economic feasibility.

    I think your suggestion is good advice for the majority of consultants in this egroup.  They are either private consultants or part of a consulting business.  My situation is different and may be similar to internal consulting at universities.  Unlike baseball caps one size does not fit all in the arena of statistical consulting.


    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------






    Original Message


  • 35.  RE:a consult that was not ideal

    Posted 02-01-2012 15:16
    I asked for advice on this situation and everyone has given good advice that is worth thinking about.  But I was looking at this as a situation that was unusual for me.  I don't think increased formalization of my consulting services is necessary for me as the Director of Biostatistical Services.  I have an excellent working relationship with almost all of my clients.  This was the first time in three years that this problem has come up.  I feel that is it is not broke don't fix it.  The idea of formal agreements makes sense to me in a business situation that involves contracts and payment for specific services.  It would make sense for me to do this in my private consulting practice.  But at Lankenau I am paid for a general consulting service.  My organization has thrived because I have satisfied customers and management support that makes it possible for me to never sacrifice my principles.  The business continues to grow.  I am supported by general hospital and institutional funds as well as government and industry grants.  My clients are research scientist and clinical trial investigators.  They are not private businesses.  Such formality which would be looked at as necessary red tape for doing business could be looked at as wasted time and money by my clients.  To do this could possibly hurt my business more than it helps it.  Most of my clients know my reputation as a professional statistician and accept my suggestions without objection.  In the few cases where I get resistance I take a strong stand.  On some issues compromise and flexibility is appropriate. This is particularly true with tradeoffs between statistical accuracy and economic feasibility.

    I think your suggestion is good advice for the majority of consultants in this egroup.  They are either private consultants or part of a consulting business.  My situation is different and may be similar to internal consulting at universities.  Unlike baseball caps one size does not fit all in the arena of statistical consulting.


    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------






    Original Message


  • 36.  RE:a consult that was not ideal

    Posted 02-01-2012 15:42
    I asked for advice on this situation and everyone has given good advice that is worth thinking about.  But I was looking at this as a situation that was unusual for me.  I don't think increased formalization of my consulting services is necessary for me as the Director of Biostatistical Services.  I have an excellent working relationship with almost all of my clients.  This was the first time in three years that this problem has come up.  I feel that is it is not broke don't fix it.  The idea of formal agreements makes sense to me in a business situation that involves contracts and payment for specific services.  It would make sense for me to do this in my private consulting practice.  But at Lankenau I am paid for a general consulting service.  My organization has thrived because I have satisfied customers and management support that makes it possible for me to never sacrifice my principles.  The business continues to grow.  I am supported by general hospital and institutional funds as well as government and industry grants.  My clients are research scientist and clinical trial investigators.  They are not private businesses.  Such formality which would be looked at as necessary red tape for doing business could be looked at as wasted time and money by my clients.  To do this could possibly hurt my business more than it helps it.  Most of my clients know my reputation as a professional statistician and accept my suggestions without objection.  In the few cases where I get resistance I take a strong stand.  On some issues compromise and flexibility is appropriate. This is particularly true with tradeoffs between statistical accuracy and economic feasibility.

    I think your suggestion is good advice for the majority of consultants in this egroup.  They are either private consultants or part of a consulting business.  My situation is different and may be similar to internal consulting at universities.  Unlike baseball caps one size does not fit all in the arena of statistical consulting.


    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------






    Original Message


  • 37.  RE:a consult that was not ideal

    Posted 01-30-2012 13:24
    Michael:

      It seems to me that you ran into a common issue with interacting with a research group. The research is being done by more junior personnel and is only loosely supervised by the PI. The PI, who as you said is often busy, doesn't involve themselves until the paper is being reviewed for submission.  At that point the PI wants a different analysis.  This is really a "failure to communicate" problem that the PI has, but it's often hard to tell them that because they are senior personnel.  It might make sense to sit down with the PI and explain how a lot of time (and money) likely will be wasted paying you to re-do analyses if the communication problem isn't eliminated.

    The main thing I can suggest to alleviate some of the problem is that the statistical consultant needs to be very proactive at the design of the experiment stage and connect with both the junior researcher and the PI at the outset.  In this way, you can try to get everyone on the same page and lay out the statistical reasons for the experimental design and analysis.  If you are successful at this then your resulting analysis will be more powerful and subject to less "after the fact hand-waving". This is obviously much easier said than done as many PI's will either brush off this early intervention (because they just want the research done but don't want to think about it first) or you will not be involved at the outset (you know, the "Hey, we've done this experiment, now how do we apply statistics to it?").  If it is an ongoing consultation, then you have a chance to show, by example, how things are much better when you are involved in designing the experiment than when you are called up at the end.

    Good luck.

    -------------------------------------------
    Jeffrey Proehl
    -------------------------------------------








  • 38.  RE:a consult that was not ideal

    Posted 01-30-2012 13:54

    I'm sure many of us have had that problem. I won't meet the resident without the PI. I make sure we are all clear from the beginning. As a Prof my peer is the PI, so I feel I need to be at the same level as the PI and establish that from the beginning. May be easier for me to do that in my situation than others.

    The waste of time and money is something that can be very frustrating. Unfortunately, often we in consulting bear the majority that cost. I try to place that elsewhere and try to get fees upfront, which can be done if there is some monetary aspect to a academic medical consulting service.

    -------------------------------------------
    Rhonda Rosychuk
    Univ of Alberta
    -------------------------------------------






    Original Message


  • 39.  RE:a consult that was not ideal

    Posted 01-30-2012 14:17

    Rhonda:  In my case this was not a private consulting situation.  It is part of my full time service to the Mainline Heatlh System. In the beginning I guess I didn't think of it like a full scale consult or I would have done exactly what you and the others have suggested.  That is the advice I preach here all the time (maybe too often).  But I got caught up in the "We just need you to take this data and calculate Kappa statistics.  We have already done the rest of the analysis."  So I took the description of the problem and the requirements at face value and didn't worry that the resident was convey something different from what the PI intended.  In my private consulting practice when I make a mistake and need to spend added time to correct it I will not charge for the wasted effort.  However, if it is the clients fault I will charge for all the time that I spent on the problem.  Usually the client is very apologetic and accepts the charges.  I am fortunate that I geenrally have good clients that I know very well.
    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------






    Original Message


  • 40.  RE:a consult that was not ideal

    Posted 01-30-2012 19:56
    I took this as more experience sharing on your part than a request for advice. 

    Certainly the phrase ""We just need you to take this data and calculate XXX statistics." is one that sets off alarm bells for most of us.  This only works if the client is knowledgeable but either can't work the software or the analysis is tedious and they need someone to help.  They MIGHT just need to calculate XXX statistics, or their problems may run deeper.


    -------------------------------------------
    Michael Kruger
    Information Resources Inc
    -------------------------------------------






    Original Message


  • 41.  RE:a consult that was not ideal

    Posted 02-07-2012 13:39
    I was involved in many research projects first as a chemist and later as a statistician.  Early planning meetings involving all the stakeholders were crucial.  Otherwise we stand an excellent chance of solving the wrong problem.  It doesn't matter whether the manager tells me what the engineer needs or the engineer tells me what the manager wants.  Something is very apt to get lost in translation unless all of us get together at the same time.

    PS, sometimes one can get 10 engineers and scientists in a room and still disagree about the objective of the projects.  In that case it usually means that we to include the VP who commissioned the problem without defining it.
    -------------------------------------------
    Emil M Friedman, PhD
    emil.friedman@alum.mit.edu (forwards to day job)
    emilfrie@alumni.princeton.edu (home)
    http://www.statisticalconsulting.org