Motivated by the insulating behavior of α-(BEDT-TSeF)₂I₃ at low temperatures (T's), we first performed first-principles calculations based on the crystal structural data at 30 K under ambient pressure, and we constructed a two-dimensional effective model using maximally localized Wannier functions. As possible causes of the insulating behavior, we studied the effects of the on-site Coulomb interaction U and spin-orbit interaction (SOI) by investigating the electronic state and the transport coefficient using the Hartree approximation and the T-matrix approximation. The calculations at a finite T demonstrated that a spin-ordered massive Dirac electron (SMD) appeared due to the on-site Coulomb interaction. We had an interest in the anomalous competitive effect with U and SOI when the SMD phase is present in α-(BETS)₂I₃, and we investigated these contributions to the electronic state and conductivity. The SMD is not a conventional spin order, but it exhibits the spin-valley Hall effect. Direct current resistivity in the presence of a spin order gap increased divergently and exhibited negative magnetoresistance in the low T region with decreasing T. The charge density hardly changed below and above the T at which this insulating behavior appeared. However, when considering the SOI alone, the state changed to a topological insulator phase, and the electrical resistivity is saturated by edge conduction at quite low T. When considering both the SMD and the SOI, the spin order gap was suppressed by the SOI, and gaps with different sizes opened in the left and right Dirac cones. This phase transition leads to distinct changes in microwave conductivity, such as a discontinuous jump and a peak structure.

(Click figure for a larger image.)