Hi Sudhir,
Thanks for sharing. That's help to understand the problem.
As El-ad reported, there are those unattended "\r" characters, but they are not very likely to fail the analysis. That would be interesting to know if you are working on Mac/windows/linux while running premessa.
The main problem for Cytofkit is that the channels are not column aligned. For example CD19 is initially linked to Cd142 in one batch and Nd142 in the other batch. After renaming the parameter is Cd142 in both batches and the associated name is also the same. But, CD19 is at column 4 in one batch and column 24 in the other. Cytofkit considers that all FCS have markers in the same columns/channels. It does not do any marker realignment while reading FCS files. I think that FlowJo and CytoBank (those you cited) have some kind of automatic alignment of markers, that makes you think that the panel have harmonized. Premessa's panel editor is also realigning the names in the interface, but does not change the position of the markers in the channel matrix of intensities. It just (but nicely) renames parameters. This is sufficient for many software, especially commercial ones, but not for cytofkit.
As El-ad already reported, cytutils is a good package for renaming channels in a batch fashion. I must admit that El-ad and I have contributed to that package (very minor contributions for my side). But this also mean that we read the code and approve what and how it is done. cytutils does the same job as premessa's panel editor (If I Remember Correctly, and premessa is working correctly as far as I know), so this will not realign columns, what is the main problem in re-conciliating the batches. This task is addressed by cytofcore. cytofcore could add and remove channels, but it mainly write the selected channels in the same order. This is what is needed by cytofkit.
In my lab, we have set up a procedure to achieve this. Please do follow the script at the end of the page
http://impact-cyto.inserm.fr/harmoniser-des-panels/ The rest of the page is in french and I didn't take time to translate it.
I did have apply this procedure to your files, leading to a correct alignment of columns. I will send you back the files.
To be noticed:
* in the cytutils step, the channels are ordered by mass, what eases the renaming. We renamed both the name and the description of each mass that duplicated. We use the typical convention <mass><metal>_<marker> because it is widely used and our pipeline is based on it. I added a formula to highlight the channels that need to be checked. This column has to be removed before saving the CSV file.
- cytutils CSV file
* in the cytofcore step, you have to add zero to missing channels, otherwise these unbalanced missing channels will create mis-alignment in the matrix of intensities. I will add this to my resource.
- cytofcore important notice
The cytofkit result is maybe better than the one you get, but I feel there is still a batch effect. I noticed differences in the range of CD4 and CD8, but I only checked a few.
- cytofkit setting
- cytofkit result
I thank you for reporting this problem in cytofkit. I will add a quick check in cytofkit in my fork and push it to the authors.
Hope this will improve your analysis. Now you have work to do on your side.
Stay safe,
Samuel