Possible bug with MPI and writing ancestral state trace

[ziad-khouri]

Hi,

I ran into a potential issue with how ancestral states are written with mnJointConditionalAncestralState() when using MPI: some sites/characters appear to be skipped like in the example below.

"1,1,1,1,1,1,0,1,1,0,1,0,0,1,0,0,0,1,0,1,1,1,1,0,1,0,1,1,1,1,0,1,1,1,1,1,0,1,1,0,,0,0,1,0,0,0,0,,0,1,1,1,0,,0,,1,0,0,1,0"

This does not occur when replicating the run without MPI. Additionally, when using MPI with this particular analysis, RevBayes freezes on or after the last MCMC iteration.

I'm attaching code to replicate the issue, as well as logs and output from two runs.

Environment:
OS: Ubuntu 18.04
GCC: 8.3.0
OpenMPI: 4.0.1
RevBayes: 59d42df
(development branch, Aug 7, 2019)

Best,

Ziad
mpi_bug_test.zip

[ziad-khouri]

Update: The issue does not show up when doing power posterior runs with MPI. I suspect it is linked specifically with parallelization across sites.