The transmissibility of the structure can be defined as the ratio of spectrum of two measured response signals. The transmissibility is entirely unrelated to the excitation, which comprehensively reflects the physical characteristics of the structure, such as stiffness, damping and mass. Therefore, it is widely used in structural state detection. Here, signals observed by tri-axial accelerometer are treated as vector signals, and then transmissibilities for single channel signal are extended to quaternion-based three-channel joint transmissibility (QTJT). This method preserves the correlation among different channels, and overcomes the problem of signal distortion caused by separate processing for each single channel. Moreover, QTJT avoids the negative influence of sensor installation attitude on the final results in different detections. Thus the engineering practicability is improved. Based on the comprehensive test platform of ballastless track, state detection experiments of fastener loosening and temperature force changing were carried out by using QTJT, in which method of Karhunen-Loeve Transform (K-LT) and dual-tree quaternion wavelet transform (DQWT) were employed, respectively. For each experiment, detection result with the accuracy of over 90% was obtained.