Maximum likelihood sequence detection for mitigating nonlinear effects